Model training method and device, electronic equipment and computer readable medium

Abstract

The invention provides a model training method and device, electronic equipment and a computer readable medium, and the method comprises the steps: carrying out the feature extraction of a voice sample, and obtaining a voice feature; inputting the voice features into a coding network in a to-be-trained model for coding processing; decoding the intermediate coding feature to obtain additional loss; obtaining coding loss based on the coding features and the coding labels; obtaining a total coding loss based on the additional loss, the coding loss and a preset first loss weight; the coding features are input into a decoding network for decoding processing, and the total decoding loss is obtained; obtaining the total loss of a model based on the total loss of coding, the total loss of decoding and a preset second loss weight; and updating parameters in the model based on the total loss of the model, and continuing to train the to-be-trained model according to the updated parameters until the total loss of the model is converged, thereby obtaining the trained model. The embodiment of the invention can improve the accuracy of the model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a model training method and device, an electronic device, and a computer-readable medium. Background Art

[0002] Speech recognition is the process of converting human speech into text. End-to-end speech recognition uses a neural network model to replace traditional alignment models, acoustic models, and language models to directly convert audio sequences into text sequences without the need for pronunciation dictionaries and phoneme annotations.

[0003] The neural network-based speech recognition model includes an encoder and a decoder, where the encoder is responsible for mapping acoustic features and the decoder is responsible for modeling semantic information. The encoder uses Connectionist Temporal Classification (CTC) loss for sequence alignment and loss calculation. The output features of the encoder are input to the decoder, and the decoder uses sequence loss modeling. Due to the natural independence of CTC, the encoder cannot be modeled based on semantic information. Therefore, the performance of the encoder is low, resulting in low accuracy of the speech recognition model. Moreover, in order to improve the generalization ability of the speech recognition model, the speech recognition model has a large number of network layers, and the intermediate decoding layer is also unable to capture semantic information, which makes it difficult to improve the accuracy of the speech recognition model. Summary of the Invention

[0004] The present disclosure provides a model training method and device, an electronic device, and a computer-readable medium for improving the accuracy of model recognition.

[0005] In a first aspect, the present disclosure provides a model training method, comprising:

[0006] Extract features from speech samples to obtain speech features;

[0007] Inputting the speech features into a coding network in a model to be trained for coding processing, wherein the coding network includes cascaded coding layers, and the coding layers include an intermediate coding layer and an Nth coding layer;

[0008] Decoding an intermediate coding feature to obtain an additional loss, wherein the intermediate coding feature is a coding feature output by the intermediate coding layer;

[0009] Obtaining a coding loss based on the coding features and coding labels output by the Nth coding layer;

[0010] Obtaining a total encoding loss based on the additional loss, the encoding loss, and a preset first loss weight;

[0011] Inputting the coding features output by the Nth coding layer into the decoding network for decoding processing to obtain a total decoding loss;

[0012] Obtaining a total model loss based on the total encoding loss, the total decoding loss, and a preset second loss weight;

[0013] Based on the total loss of the model, the parameters in the encoding network and the decoding network are updated, and the model to be trained is trained according to the updated parameters until the total loss of the model converges, thereby obtaining a trained model.

[0014] In a second aspect, the present disclosure provides a speech recognition model, which is a model obtained by the above-mentioned model training method.

[0015] In a third aspect, the present disclosure provides a model training device, comprising:

[0016] Extraction module, used to extract features from speech samples to obtain speech features;

[0017] An encoding module, configured to input the speech features into an encoding network in a model to be trained for encoding processing, wherein the encoding network includes cascaded encoding layers, and the encoding layers include an intermediate encoding layer and an Nth encoding layer;

[0018] An additional module, configured to decode an intermediate coding feature to obtain an additional loss, wherein the intermediate coding feature is a coding feature output by the intermediate coding layer;

[0019] a calculation module, configured to obtain a coding loss based on the coding features and coding labels output by the Nth coding layer; and obtain a total coding loss based on the additional loss, the coding loss, and a preset first loss weight;

[0020] The decoding module is further configured to input the coding features output by the Nth coding layer into the decoding network for decoding processing to obtain a total decoding loss;

[0021] The calculation module is further configured to obtain a total model loss based on the total encoding loss, the total decoding loss, and a preset second loss weight;

[0022] An updating module is used to update the parameters in the encoding network and the decoding network based on the total loss of the model, and train the model to be trained according to the updated parameters until the total loss of the model converges to obtain a trained model.

[0023] In a fourth aspect, the present disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and one or more of the computer programs are executed by the at least one processor to enable the at least one processor to execute the above-mentioned model training method.

[0024] In a fifth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned model training method when executed by a processor.

[0025] In a sixth aspect, the present disclosure provides a computer program product comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned model training method.

[0026] In the model training method provided by the embodiment of the present disclosure, an additional decoding network obtains intermediate coding features from the intermediate coding layer of the coding network, and decodes the intermediate coding features to obtain additional decoding features, and obtains additional losses based on the additional decoding features and additional labels. Since the intermediate coding features contain more semantic information, the additional loss determined by the intermediate coding features contains more semantic information. Therefore, the total coding loss determined based on the additional loss, the coding loss and the preset first loss weight contains more semantic information, and the total model loss obtained by combining the total coding loss with the total decoding loss also contains semantic information. Therefore, the parameters of the speech recognition model to be trained are updated based on the total model loss until the total model loss converges, so that the trained speech recognition model can obtain more semantic information, thereby improving the accuracy of the speech recognition model.

[0027] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing detailed example embodiments with reference to the accompanying drawings. In the accompanying drawings:

[0029] Figure 1 Schematic diagram of the structure of the speech recognition model in the embodiment of the present disclosure;

[0030] Figure 2 Schematic diagram of a model training structure for training a speech recognition model in an embodiment of the present disclosure;

[0031] Figure 3 A flowchart of a model training method provided in an embodiment of the present disclosure;

[0032] Figure 4 A schematic diagram of the structure of a coding layer provided in an embodiment of the present disclosure;

[0033] Figure 5 A block diagram of an intelligent voice system provided in an embodiment of the present disclosure;

[0034] Figure 6 A block diagram of a model training device provided in an embodiment of the present disclosure;

[0035] Figure 7 A block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] To enable those skilled in the art to better understand the technical solutions of the present disclosure, exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0037] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0038] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0039] The terms used herein are only used to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof is not excluded. Similar words such as "connected" or "connected" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0040] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0041] CTC loss can be applied to end-to-end speech recognition models for recognizing streaming speech, directly converting speech sequences into label sequences. However, the independence of CTC prevents the encoding network from modeling based on semantic information. While the decoding network can impart some semantic information modeling capabilities to the output of the encoding network's final layer, the large number of encoding layers in the encoding network means that the final layer captures very limited semantic information, failing to improve the accuracy of the speech recognition model.

[0042] The model training method and apparatus, electronic device, and readable medium provided in the embodiments of the present disclosure can be executed by an electronic device such as a terminal device or a server. The terminal device can be an in-vehicle device, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, etc. The method can be implemented by a processor calling computer-readable program instructions stored in a memory. Alternatively, the method can be executed by a server.

[0043] In this disclosure, unless otherwise specified, the following technical terms should be understood as follows:

[0044] Transformer model: A time series model based on the self-attention mechanism. The encoding network can effectively encode time series information and has better processing capabilities for time series information than the Long Short-Term Memory (LSTM) network. It also has strong parallel computing capabilities and fast computing speed. It is widely used in natural language processing, computer vision, machine translation, and speech recognition.

[0045] Conformer model: It combines the Transformer model with Convolutional Neural Networks (CNN). The Transformer model is good at capturing content-based global interactions, and CNN can effectively utilize local features, making the Conformer model have better modeling capabilities for long-term global interaction information and local features.

[0046] The CTC model adds a classification layer to the final layer of a recurrent neural network (RNN) to predict sequence labels. Traditional sequence labeling algorithms require perfect alignment of input and output symbols at every moment. CTC, however, expands the label set by adding empty elements. After annotating the sequence with the expanded label set, any predicted sequence that can be converted to the true sequence using a mapping function is considered correct, meaning that no data alignment is required to obtain the predicted sequence.

[0047] In a first aspect, the embodiments of the present disclosure provide a model training method, wherein the accuracy of the model trained by the training method is higher. The embodiments of the present disclosure do not limit the model, for example, the model can be a speech recognition model.

[0048] Figure 1 Schematic diagram of the structure of the model in the embodiment of the present disclosure. Figure 1 As shown, the model includes an encoding network 10 and a decoding network 20, wherein the encoding network 10 includes N cascaded encoding layers, the encoding layers including an intermediate encoding layer and an N-th encoding layer. The decoding network 20 includes M cascaded decoding layers, the decoding layers including an intermediate decoding layer and an M-th decoding layer, wherein N and M are both integers greater than or equal to 2. The embodiment of the present disclosure does not limit the number of encoding layers of the encoding network 10. For example, the number of encoding layers is 12. The encoding layer can be encoded using a residual attention mechanism.

[0049] The disclosed embodiment does not limit the number of layers in the decoding layer of the decoding network 20. For example, the number of layers in the decoding layer is 6. The disclosed embodiment does not limit the type of decoder used in the decoding layer. For example, the decoder can be a transform decoder, and the decoder uses words as calculation targets.

[0050] When using the model for speech recognition, the speech to be recognized is input into the model, and the speech to be recognized passes through the encoding network 10 and the decoding network 20, and the recognition result is output.

[0051] Figure 2 This is a schematic diagram of the model training structure provided by the embodiment of the present disclosure. Figure 2 As shown, the model training structure includes an encoding network 10, a decoding network 20 and an additional decoding network 30. The encoding network 10 and the decoding network 20 are connected to Figure 1 The encoding network 10 and decoding network 20 are identical and are not described in detail here. The additional decoding network 30 includes L cascaded decoding layers, where L is an integer greater than or equal to 1. The present embodiment does not limit the number of decoding layers in the additional decoding network 30. For example, the number of decoding layers can be 6.

[0052] The input of the additional decoding network 30 comes from the intermediate coding features output by the intermediate coding layer of the encoding network 10. The additional decoding network 30 processes the intermediate coding features to obtain additional decoding features, and obtains additional loss based on the additional decoding features and the additional labels. Since the additional loss is determined based on the additional decoding features, and the additional decoding features are obtained based on the intermediate coding features of the encoding network 10, the intermediate coding features contain more semantic information, so that the additional decoding features capture more semantic information. Therefore, the parameters of the model determined based on the additional loss are more accurate.

[0053] In the model training structure, the number of additional decoding networks 30 can be one or more. When the model training structure includes multiple additional decoding networks 30, the number of decoding layers in each additional decoding network 30 can be the same or different.

[0054] The inputs of different additional decoding networks 30 come from different encoding layers in the encoding network 10, that is, each additional decoding network 30 processes the intermediate encoding features output by different encoding layers to capture the semantic information in different encoding layers, thereby making the parameters of the model more accurate.

[0055] When the model training structure includes an additional decoding network 30, the additional decoding network 30 can process any intermediate coding feature in the encoding network 10, for example, it can process the intermediate coding features at one-half, one-third, or two-thirds of the encoding network 10.

[0056] When the model training structure includes two additional decoding networks 30, the two additional decoding networks 30 can respectively process the intermediate coding features at one-third and two-thirds of the encoding network 10. For example, when the encoding network 10 includes 12 encoding layers, one additional decoding network 30 processes the intermediate coding features output by the 4th encoding layer in the encoding network 10, and the other additional decoding network 30 processes the intermediate coding features output by the 8th encoding layer in the encoding network 10.

[0057] A model training method provided by the embodiment of the present disclosure is to Figure 2 The training is performed in the model training structure shown in FIG. 1 , but when the trained model is used, the additional decoding network is removed, and only the encoding network 10 and the decoding network 20 are retained.

[0058] Figure 3 This is a flow chart of a model training method provided by an embodiment of the present disclosure. Figure 3 , the model training method includes:

[0059] Step S301: extract features from the speech sample to obtain speech features.

[0060] The voice samples can be conversations between customers and agents in a customer service system. For example, nearly 50,000 hours of conversational speech from the customer service system can be used as the initial sample. This initial sample is processed to remove invalid initial samples to obtain the voice sample. Invalid initial samples include short, insubstantial speech. For example, conversations shorter than 5 seconds can be removed.

[0061] Before performing feature extraction on the speech sample, the speech sample may also be indexed to obtain the corresponding text, and the text data may be annotated to obtain a sample label.

[0062] In some embodiments, the speech features are fbank features, and the steps of extracting features from speech samples to obtain the fbank features include: pre-emphasis, framing, windowing, discrete Fourier transform, and Mel filtering.

[0063] Pre-emphasis: Air is the carrier of voice signals, transmitting and dissipating sound wave energy. Given a given sound source size, the higher the frequency, the greater the loss. Pre-emphasis can compensate for the loss of high-frequency components and enhance the signal's high-frequency content.

[0064] This disclosed embodiment uses a high-pass filter for pre-emphasis processing. This filter controls the degree of high-frequency information in the speech sample, enhancing the high-frequency portion. This allows the spectrum to be calculated using the same signal-to-noise ratio across the entire frequency range, from low to high. High-frequency information contains a greater amount of linguistic information, allowing the model to capture more linguistic information during subsequent information fusion processing, thereby improving model accuracy.

[0065] The working principle of the high-pass filter is: O(n) = k*x(n)-m*x(n-1)

[0066] Where O(n) represents the result after high-pass filtering, n represents the sampling point, x represents the speech sample, k and m represent the filter coefficients, k represents the ability to preserve high-frequency information, and m represents the ability to suppress high-frequency information. The larger k is and the smaller m is, the weaker the high-pass filter's ability to suppress high-frequency information.

[0067] Framing: In order to obtain fine-grained features for the coordinated use of subsequent language information and acoustic information. The disclosed embodiment sets the value of the audio sampling point N after framing to 256, which includes a time of approximately 20ms. At the same time, in order to avoid the interval between two adjacent frames being too large, resulting in insufficient feature granularity when jointly modeling speech and emotion recognition, there is an overlapping area between two adjacent frames, and the overlapping area contains 128 audio sampling points. Through this framing scheme, the information confusion caused by insufficient feature granularity capture in the subsequent attention score matrix can be alleviated, thereby improving the information stability of the attention score matrix merger.

[0068] Windowing: Framing is essentially a discrete representation of speech samples. This discrete representation must maximize the representation of continuous information to minimize spectral leakage in the extracted speech features. When deep fusion of speech and acoustic information occurs, spectral leakage can lead to significant errors in speech interpretation, affecting the final representation of acoustic information. Therefore, continuous information from speech frames is required to ensure model accuracy.

[0069] Therefore, after the speech samples are framed, each speech frame needs to be windowed to increase the continuity between the left and right ends of the frame, reduce spectrum leakage, and ensure that the two ends of the window do not change sharply but transition smoothly to 0, so that the cut-off speech frame slowly drops to 0, reducing the truncation effect of the speech frame.

[0070] The embodiment of the present disclosure performs a windowing operation through a window function, and the expression of the window function is formula (1):

[0071]

[0072] In formula (1), N represents the number of sampling points, n represents the sampling point, and K(n) represents the result after windowing the sampling point. The windowing function can make the discrete information of each speech frame correlated. The reason is that the sin function with limited weights establishes a functional connection between each speech frame through a specific weighting method. This can enable the additional decoding network to improve the semantic information capability of the decoding network. When the attention score matrix in the encoding network is fused, the discrete information before and after the speech frame can be continuously represented, avoiding spectrum leakage, thereby improving the accuracy of the model.

[0073] Fourier transform: Frequency domain energy distribution can better observe speech characteristics. After each speech frame is transformed through Fourier transform, a spectrum signal is obtained. That is, the spectrum signal of each speech frame is obtained in the frequency domain.

[0074] Mel filtering: The spectrum signal is filtered through a Mel filter bank. By using the passband and stopband characteristics of different Mel filters in different frequency ranges, a mapping relationship between the human ear's perception of real audio features and discrete signals is established, thereby mapping the spectrum into a Mel nonlinear spectrum that conforms to human ear perception.

[0075] The Mel filtering in the disclosed embodiment uses a set of rectangular filters with adjustable coefficients, with center frequencies f(m) = 1, 2, 3…P, where P can be 22. The rectangular filter preserves the original information in low-amplitude regions as much as possible while transforming high-amplitude regions into a representation that is consistent with the human ear. This makes the feature more consistent with the prediction requirements of the additional decoding network and meets the low-amplitude acoustic information requirements for information fusion between attention score matrices.

[0076] The fine-grained speech features are obtained through step S301, and the speech features help the subsequent additional decoding network to obtain more semantic information, thereby making the trained model more accurate.

[0077] In some embodiments, feature extraction of speech samples is performed before obtaining speech features, including: obtaining speech samples, cutting the speech samples to obtain speech segments; labeling speech segments that belong to noise to obtain noise labels; and feature extraction of speech segments that do not belong to noise to obtain speech features.

[0078] Noise includes speech segments where sound is present but the specific words cannot be identified. Noise is labeled with an unknown word (UNK) label, and silent speech segments are labeled with a blank label. This trained model avoids learning both UNK and blank labels as blank labels, thereby improving the model's noise immunity.

[0079] The speech sample is cut into several speech segments, each of which can be between 0.8 and 2.0 seconds long. Then, the speech segments are recognized, the speech segments belonging to noise are labeled to obtain noise labels, and the features of the speech segments that do not belong to noise are extracted to obtain speech features.

[0080] The present disclosure cuts the speech samples into speech segments of 0.8-2.0s, which can identify the noise and thus mark the noise, so that the model can have the ability to identify noise, thereby improving the model's noise resistance.

[0081] Step S302: Input the speech features into the encoding network in the model to be trained for encoding processing.

[0082] The encoding network includes N cascaded encoding layers, where N is an integer greater than or equal to 2. The encoding layers include an intermediate encoding layer and an Nth encoding layer. Both the intermediate encoding layer and the Nth encoding layer can use encoders from the field of deep learning to map input speech features to low-dimensional vectors. In some embodiments, the encoding layer uses a residual attention module, which is an improvement on the multi-head attention mechanism.

[0083] Figure 4 This is a schematic diagram of the structure of the coding layer provided in the embodiment of the present disclosure. Figure 4 As shown, the encoding layer includes a feedforward network sublayer 41, a residual multi-head attention sublayer 42 and a convolution sublayer 43. The feedforward network sublayer 41, the residual multi-head attention sublayer 42 and the convolution sublayer 43 all adopt a residual structure.

[0084] The feedforward network sublayer 41 can use a feedforward network in the field of deep learning to extract features, and the convolution sublayer 43 can extract speech features through convolution operations. The present disclosure does not limit the structures of the feedforward network sublayer and the convolution sublayer.

[0085] The residual multi-head attention sub-layer 42 fuses the information from the current coding layer with the information from the previous coding layer to obtain the mutual attention relationship between a certain character and other characters. This is to achieve mutual attention relationships at different levels, which can increase the dynamic attention range of the current coding layer. The fused information is then input into the next coding layer. The previous coding layer refers to the previous coding layer adjacent to the current coding layer, and the next coding layer refers to the next coding layer adjacent to the current coding layer.

[0086] In some embodiments, the coding layers 2 through N in the coding network perform the following steps:

[0087] The previous layer score matrix is ​​obtained based on the query matrix and key value matrix in the previous coding layer; the current layer score matrix is ​​obtained based on the query matrix and key value matrix in the current coding layer; the previous layer score matrix and the current layer score matrix are merged to obtain the coding feature, and the coding feature is input into the next coding layer. The coding feature is the coding feature after the features of the current coding layer and the previous coding layer are fused, that is, the coding feature is the coding feature that fuses the information of different layers.

[0088] The query matrix and key-value matrix are matrices in the self-attention mechanism, and this disclosure does not limit the specific methods for obtaining the query matrix and key-value matrix. For example, the query matrix Q is calculated based on the input speech features or the output of the previous coding layer using the linear transformation matrix WQ, and the key-value matrix K is calculated based on the input speech features or the output of the previous coding layer using the linear transformation matrix WK.

[0089] After multiplying the query matrix Q and the key-value matrix K in any encoding layer, a softmax operation is performed to calculate the attention coefficient of each word to other words, and the score matrix of the encoding layer is obtained.

[0090] For example, the current coding layer obtains the query matrix Q and key value matrix K in the previous coding layer, and multiplies the query matrix Q and key value matrix K in the previous coding layer, and performs a normalization (softmax) operation on the multiplication result to obtain the previous layer score matrix; at the same time, the query matrix Q and key value matrix K in the current coding layer are multiplied, and the multiplication result is softmaxed to obtain the current layer score matrix. Then, the previous layer score matrix and the current layer score matrix are input into the merging sublayer 44 for merging, and the merging result is input into the linear sublayer 45 to realize the fusion of attention information of different layers. Finally, the fused coding features are sent to the next coding layer. Among them, the merging of the previous layer score matrix and the current layer score matrix can be a matrix addition operation.

[0091] It should be noted that for the Nth coding layer, after obtaining the fusion information, it is no longer input into the next coding layer, but is used as the output result of the coding network to calculate the coding loss.

[0092] The coding layer provided in the embodiment of the present disclosure adopts information fusion technology to fuse the attention information of different coding layers, which can improve the range and ability of the model to capture information and improve the accuracy of the model.

[0093] Step S303: decode the intermediate encoding features to obtain additional loss.

[0094] The intermediate coding features are coding features output by the intermediate coding layers. For example, when the coding network includes 12 coding layers, the intermediate coding features are coding features output by any one of the 1st to 11th coding layers. The additional labels can be pre-annotated by manual means or other methods.

[0095] In some embodiments, the intermediate coding features include intermediate coding features at one-third of the coding network, and / or intermediate coding features at two-thirds of the coding network. For example, when the coding network includes 12 coding layers, the intermediate coding features may be intermediate coding features output by the 4th coding layer, and / or intermediate coding features output by the 8th coding layer. By selecting the intermediate coding features at one-third and / or two-thirds as the input of the additional decoding network, additional loss can be calculated from semantic information at different levels, and then the total coding loss is calculated based on the additional loss to determine the total model loss, that is, the total model loss includes the additional loss, and the model parameters adjusted in this way can make the total model loss converge faster.

[0096] In some embodiments, decoding the intermediate coding features to obtain the additional loss includes: decoding the intermediate coding features to obtain additional decoding features, and obtaining the additional loss based on the additional decoding features and a preset additional decoding label.

[0097] The intermediate encoded features are input into an additional decoding network for decoding to obtain additional decoded features. The additional decoding network comprises K cascaded decoding layers, where K is an integer greater than or equal to 1. The decoding layers can employ transformer decoders or other decoders. The K decoding layers decode the intermediate encoded features to obtain additional decoded features. The additional decoding network then uses the additional decoded features and the additional labels to obtain an additional loss.

[0098] In some embodiments, the additional loss L add-middle It can be obtained by formula (2):

[0099] L add-middle =-logP(y / x (k / 3) ) (2)

[0100] In formula (2), L add-middle represents the additional loss, P represents the attention distribution probability, y represents the additional label, x represents the input, k / 3 represents the intermediate encoding layer, and k represents the number of encoding layers.

[0101] For example, if the input of the additional decoding network is the output of the 4th encoding layer in the encoding network, that is, x is the output of the 4th encoding layer in the encoding network as the input of the additional decoding network, and the additional label y is the pre-labeled additional label, the additional loss L can be obtained by formula (2): add-middle .

[0102] It should be noted that when the model training method uses multiple additional decoding networks, each additional decoding network generates an additional loss. In this case, the additional loss can be the sum of the additional losses of each additional decoding network. For example, when the model training method uses a first additional decoding network and a second additional decoding network, the input of the first additional decoding network can be the output of the fourth coding layer, and the input of the second additional decoding network can be the output of the eighth coding layer. The first additional loss is obtained by the first additional decoding network, and the second additional loss is obtained by the second additional decoding network. The additional loss is the sum of the first and second additional losses.

[0103] Step S304: Obtain coding loss based on the coding features and coding labels output by the Nth coding layer.

[0104] Among them, the output of the Nth coding layer is the output of the coding network, that is, the coding features output by the Nth coding layer are the coding features output by the coding network, and the speech features are encoded layer by layer by the N coding layers in the coding network to output the coding features.

[0105] The coding labels in this embodiment can be pre-labeled manually. The coding loss is obtained based on the output coding features and coding labels of the coding network.

[0106] In some embodiments, the coding loss is calculated using formula (3).

[0107]

[0108] In formula (3), L ctc represents the encoding loss, l represents the label sequence, l=(l1...l r ), r represents the length, P represents the attention distribution probability, δ() represents a many-to-one mapping, which is used to delete blank and repeated outputs generated during alignment, ε represents the encoding label, and y enc Represents input.

[0109] In some embodiments, the coding loss can be calculated by a backtracking algorithm. The backtracking algorithm can effectively obtain all sequence solutions and then obtain the optimal solution from all sequence solutions. Therefore, the coding loss using the backtracking algorithm can improve the accuracy of the model.

[0110] Step S305 : Obtaining a total coding loss based on the additional loss, the coding loss, and a preset first loss weight.

[0111] Among them, the first loss weight is pre-set.

[0112] In some embodiments, the total coding loss is calculated by formula (4).

[0113] L enc_total =(1-q)*L ctc +q*L add-middle (4)

[0114] In formula (4), L enc_total Denotes the total coding loss, L ctc represents the coding loss, L add-middle represents the additional loss, and q represents the first weight.

[0115] In step S306, the coding features output by the Nth coding layer are input into the decoding network for decoding processing to obtain the total decoding loss.

[0116] The decoding network includes M cascaded decoding layers, the decoding layers include an intermediate decoding layer and an M-th decoding layer, and M is an integer greater than or equal to 2.

[0117] The output encoding features of the last layer of the encoding network are input into the decoding network, that is, the encoding features output by the Nth encoding layer are input into the decoding network, the encoding features are decoded by the decoding network to obtain the decoding features, and then the total decoding loss is obtained based on the decoding features.

[0118] In some embodiments, the encoding features output by the Nth encoding layer are input into the decoding network for decoding processing to obtain the total decoding loss, including: obtaining the intermediate loss based on the decoding features and intermediate decoding labels output by the intermediate decoding layer; the decoding network includes the intermediate decoding layer and the Mth decoding layer; obtaining the decoding loss based on the decoding features and decoding labels output by the Mth decoding layer; and obtaining the total decoding loss based on the intermediate loss, the decoding loss and a preset third loss weight.

[0119] The intermediate decoding labels and the decoding labels are pre-labeled, and this embodiment does not limit the labeling method. For example, they can be labeled manually. The third loss weight is pre-set or can be determined based on experiments. For example, the third loss weight can be 0.7.

[0120] In some embodiments, the total decoding loss can be calculated using formula (5).

[0121] L ATT_total =β*L ATT +(1-β)*L ATT_middle (5)

[0122] In formula (5), L ATT_total Denotes the total decoding loss, L ATT represents the decoding loss, L ATT_middle represents the intermediate loss, and β represents the third loss weight.

[0123] For example, Figure 2 As shown, the decoding network includes 6 decoding layers, the third layer is the intermediate decoding layer, and the intermediate loss L is obtained based on the output of the third decoding layer and the intermediate decoding label. ATT_middle The decoding loss L is obtained based on the decoding features and decoding labels output by the Mth layer decoding layer. ATT Then, based on the intermediate loss, decoding loss and the preset third loss weight, the total decoding loss L is obtained ATT_total .

[0124] Step S307: Obtain the total model loss based on the total encoding loss, the total decoding loss, and a preset second loss weight.

[0125] Among them, the second loss weight is pre-set and the user can set it arbitrarily according to the actual training situation.

[0126] In some embodiments, the total model loss is calculated by formula (6).

[0127] L total =α*L enc_tatal +(1-α)*L ATT_total (6)

[0128] In formula (6), L total Represents the total loss of the model, L enc_tatal Denotes the total coding loss, L ATT_total represents the total decoding loss, and α represents the second loss weight.

[0129] Step S308: Update the parameters in the encoding network and the decoding network based on the total model loss, and train the model to be trained according to the updated parameters until the total model loss converges to obtain the trained model.

[0130] The parameters in the encoding network and decoding network are adjusted based on the total model loss. After the parameters are adjusted, the model is continued to be trained using speech samples according to the updated parameters until the total model loss converges to obtain the trained model.

[0131] In some embodiments, updating parameters in the encoding network and the decoding network based on the total model loss includes: updating parameters in the encoding network and the decoding network based on the total model loss and the regularization term.

[0132] During model training, the parameters of the encoding and decoding networks are updated based on the total model loss during the parameter update phase and the gradient update phase. During the parameter update phase, a regularization term is added to update the parameters of the encoding and decoding networks to improve the model's convergence speed and generalization ability. However, during the gradient update phase, no regularization term is added to update the parameters of the encoding and decoding networks. This prevents gradient accumulation and the resulting violent parameter fluctuations, thereby achieving rapid and accurate loss function minimization and improving model training results and generalization ability.

[0133] In some embodiments, the learning rate is adaptively optimized using an Adam (Adaptive Moment Estimation) optimizer. The regularization term may be an L2 regularization term.

[0134] The update rules of the Adam optimizer are as follows:

[0135] m t =β1m t-1 +(1-β1)g t

[0136]

[0137]

[0138]

[0139]

[0140] Among them, m t represents the first-order momentum, v t represents the second-order momentum, β1 and β2 represent the attenuation coefficients, g t represents the gradient of the parameters, and represents the bias-corrected moving average, θ t+1 represents the updated parameters, θ t represents the parameters before update, η represents the learning rate, and ε represents a constant.

[0141] In some embodiments, before the intermediate coding features are input into the additional decoding network for decoding, it also includes: obtaining a first coding loss based on the coding features and coding labels output by the Nth coding layer; inputting the coding features output by the Nth coding layer into the decoding network for decoding processing to obtain a first total decoding loss, the decoding network includes M cascaded decoding layers, M is an integer greater than or equal to 2; obtaining a first model total loss based on the first total coding loss, the first total decoding loss and a preset second loss weight; updating the parameters in the encoding network and the decoding network based on the first model total loss, and training the model to be trained according to the updated parameters until the preset conditions are met to obtain a pre-trained model; and using the parameters of the pre-trained model as the initial parameters of the encoding network and the decoding network.

[0142] During the pre-training phase, no additional loss needs to be calculated. The speech features are encoded based on the encoding network to obtain encoding features. The encoding features and encoding labels are then used to obtain a first encoding loss, i.e., a first encoding loss is obtained based on the encoding features and encoding labels output by the Nth encoding layer, and this first encoding loss is used as the total encoding loss. A first total decoding loss is obtained in the same manner as step 306. Then, a first total model loss is obtained based on the first total encoding loss, the first total decoding loss, and a preset second loss weight. The parameters in the encoding and decoding networks are updated based on the first total model loss, and the model to be trained continues to be trained until the preset conditions are met, thereby obtaining a pre-trained model.

[0143] After obtaining the pre-trained model, the network parameters in the decoding network are assigned to the additional decoding network, and then the pre-trained model is further trained. During the further training process, the intermediate encoding features are decoded using the additional decoding network, and additional losses are obtained. Then, steps S304 to S308 are executed until the trained model is obtained.

[0144] In the model training method provided by the embodiments of the present disclosure, an additional decoding network obtains intermediate coding features from the coding layer in the middle of the coding network, and decodes the intermediate coding features to obtain additional decoding features, and obtains additional losses based on the additional decoding features and additional labels. Since the intermediate coding features contain more semantic information, the additional loss determined by the intermediate coding features contains more semantic information. Therefore, the total coding loss determined based on the additional loss, the coding loss and the preset first loss weight contains more semantic information. The total model loss obtained by combining the total coding loss with the total decoding loss also contains semantic information. Therefore, the parameters of the model to be trained are updated based on the total model loss until the total model loss converges, so that the trained model can obtain more semantic information, thereby improving the accuracy of the model.

[0145] On the second aspect, the embodiment of the present disclosure provides a speech recognition model, which is a model obtained by the model training method provided by the embodiment of the present disclosure. The model training method will not be described in detail.

[0146] When using this speech recognition model, the speech to be recognized can be input into the speech recognition model, and the speech to be recognized can be encoded through the encoding network to obtain encoding features; the encoding features can be decoded through the decoding network to obtain the speech recognition results.

[0147] Since the speech recognition model is a model obtained through the model training method provided by the embodiment of the present disclosure, more semantic information is obtained during the training process. Therefore, the speech recognition model has more accurate speech recognition results.

[0148] Figure 5 This is a block diagram of an intelligent voice system provided by an embodiment of the present disclosure. The intelligent voice system can be used by an intelligent customer service system or an intelligent sales system.

[0149] like Figure 5 As shown, the intelligent voice system 500 includes a voice acquisition module 501 , a voice recognition module 502 , an intention understanding module 503 , a text generation module 504 and a voice synthesis module 505 .

[0150] The voice collection module 501 is used to collect voice, and the collected voice can be a voice stream signal transmitted in real time from the telephone user end.

[0151] The speech recognition module 502 can use the model trained by the model training method provided in the embodiment of the present disclosure to perform speech recognition on the collected speech, and obtain the decoding result by performing feature extraction, encoding and decoding steps through the model.

[0152] The intention understanding module 503 is used to determine the intention of the decoding result and obtain the intention corresponding to the voice stream signal.

[0153] The text generation module 504 is used to obtain a corresponding reply based on the intention and judgment logic, and obtain a response text.

[0154] The speech synthesis module 505 is used to perform speech synthesis on the response text to obtain the corresponding speech.

[0155] If there are multiple rounds of conversation, the voice stream information of any round will be generated into a corresponding response voice through the voice recognition module 502, the intention understanding module 503, the text generation module 504 and the voice synthesis module 505, and this cycle will be repeated to achieve intelligent response.

[0156] In a third aspect, an embodiment of the present disclosure provides a model training device.

[0157] Figure 6 A block diagram of a model training device provided in an embodiment of the present disclosure, such as Figure 6 As shown, the model training device 600 includes:

[0158] The extraction module 601 is used to extract features from the speech sample to obtain speech features.

[0159] The encoding module 602 is used to input the speech features into the encoding network in the model to be trained for encoding processing. The encoding network includes cascaded encoding layers, and the encoding layers include an intermediate encoding layer and an Nth encoding layer.

[0160] The additional module 603 is used to decode the intermediate coding features to obtain additional losses, where the intermediate coding features are coding features output by the intermediate coding layer.

[0161] The calculation module 604 is used to obtain the coding loss based on the coding features and coding labels output by the Nth coding layer; and to obtain the total coding loss based on the additional loss, the coding loss and the preset first loss weight.

[0162] The decoding module 605 is further configured to input the coding features output by the Nth coding layer into the decoding network for decoding processing to obtain the total decoding loss.

[0163] The calculation module 604 is further configured to obtain a total model loss based on the total encoding loss, the total decoding loss, and a preset second loss weight.

[0164] The updating module 606 is used to update the parameters in the encoding network and the decoding network based on the total model loss, and train the model to be trained according to the updated parameters until the total model loss converges to obtain the trained model.

[0165] In some embodiments, the additional module 603 is configured to perform decoding processing on the intermediate encoding features to obtain additional decoding features; and obtain additional loss based on the additional decoding features and a preset additional decoding label.

[0166] In some embodiments, the intermediate coding features include intermediate coding features at one-third of the coding network, and / or intermediate coding features at two-thirds of the coding network.

[0167] In some embodiments, the decoding module 605 is further used to perform decoding processing on the encoding feature decoding network output by the Nth encoding layer; obtain an intermediate loss based on the decoding features and intermediate decoding labels output by the intermediate decoding layer, and the intermediate decoding layer is a decoding layer other than the Mth decoding layer in the decoding network; obtain a decoding loss based on the decoding features and decoding labels output by the Mth decoding layer; obtain a total decoding loss based on the intermediate loss, the decoding loss and a preset third loss weight.

[0168] In some embodiments, the encoding module 602 is also used to obtain the previous layer score matrix based on the query matrix and key value matrix in the previous encoding layer; obtain the current layer score matrix based on the query matrix and key value matrix in the current encoding layer; merge the previous layer score matrix and the current layer score matrix to obtain encoding features, and input the encoding features into the next encoding layer.

[0169] In some embodiments, a pre-training model is also included, which is used to input speech features into the model to be trained, pre-train the model to be trained, and obtain initial parameters of the encoding network and the decoding network.

[0170] In some embodiments, the updating module 606 is further configured to update the parameters in the encoding network and the decoding network based on the total model loss and the regularization term during the parameter updating phase.

[0171] In some embodiments, the extraction module 601 is further configured to obtain speech samples, segment the speech samples to obtain speech segments, and label the speech segments that are noise to obtain noise labels.

[0172] In the model training device provided by the embodiment of the present disclosure, the additional decoding network obtains intermediate coding features from the encoding layer in the middle of the encoding network, and decodes the intermediate coding features to obtain additional decoding features, and obtains additional losses based on the additional decoding features and the additional labels. Since the intermediate coding features contain more semantic information, the additional loss determined by the intermediate coding features contains more semantic information. Therefore, the total coding loss determined based on the additional loss, the coding loss and the preset first loss weight contains more semantic information. The total model loss obtained by combining the total coding loss with the total decoding loss also contains semantic information. Therefore, the parameters of the model to be trained are updated based on the total model loss until the total model loss converges, so that the trained model can obtain more semantic information, thereby improving the accuracy of the model.

[0173] In a fourth aspect, an embodiment of the present disclosure provides an electronic device. Figure 7 This is a block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 7 , an embodiment of the present disclosure provides an electronic device, which includes: at least one processor 701; at least one memory 702, and one or more I / O interfaces 703, connected between the processor 701 and the memory 702; wherein the memory 702 stores one or more computer programs that can be executed by the at least one processor 701, and the one or more computer programs are executed by the at least one processor 701 to enable the at least one processor 701 to perform the above-mentioned model training method.

[0174] In some embodiments, the processor 701 is used to extract features from the speech sample to obtain speech features; input the speech features into the coding network in the model to be trained for coding processing, the coding network includes a cascaded coding layer, the coding layer includes an intermediate coding layer and an N-th coding layer; decode the intermediate coding features to obtain additional loss, the intermediate coding features are coding features output by the intermediate coding layer; obtain coding loss based on the coding features and coding labels output by the N-th coding layer; and obtain total coding loss based on the additional loss, coding loss and a preset first loss weight; output of the N-th coding layer The encoded features are input into the decoding network in the model to be trained for decoding processing to obtain a total decoding loss, the decoding network includes M cascaded decoding layers, M is an integer greater than or equal to 2, the decoding layer includes an intermediate decoding layer and an M-th decoding layer, and the intermediate decoding layer is a decoding layer other than the M-th decoding layer in the decoding network; the total model loss is obtained based on the total encoding loss, the total decoding loss and a preset second loss weight; the parameters in the encoding network and the decoding network are updated based on the total model loss, and the model to be trained is trained according to the updated parameters until the total model loss converges to obtain a trained model.

[0175] In some embodiments, the processor 701 is further configured to perform decoding processing on the intermediate encoding features to obtain additional decoding features; and obtain additional loss based on the additional decoding features and a preset additional decoding label.

[0176] In some embodiments, the intermediate coding features include intermediate coding features at one-third of the coding network, and / or intermediate coding features at two-thirds of the coding network.

[0177] In some embodiments, the processor 701 is further used to input the encoding features output by the Nth encoding layer into the decoding network in the model to be trained for decoding processing; obtain the intermediate loss based on the decoding features and intermediate decoding labels output by the intermediate decoding layer; obtain the decoding loss based on the decoding features and decoding labels output by the Mth decoding layer; obtain the total decoding loss based on the intermediate loss, decoding loss and a preset third loss weight.

[0178] In some embodiments, the processor 701 is also used to obtain a previous layer score matrix based on the query matrix and key value matrix in the previous coding layer; obtain a current layer score matrix based on the query matrix and key value matrix in the current coding layer; merge the previous layer score matrix and the current layer score matrix to obtain coding features, and input the coding features into the next coding layer.

[0179] In some embodiments, the processor 701 is further used to input speech features into the model to be trained, pre-train the model to be trained, and obtain initial parameters of the encoding network and the decoding network.

[0180] In some embodiments, the processor 701 is further configured to update the parameters in the encoding network and the decoding network based on the total model loss and the regularization term during the parameter updating phase.

[0181] In some embodiments, the processor 701 is further configured to obtain a speech sample, segment the speech sample to obtain speech segments, and label the speech segments that are noise to obtain a noise label.

[0182] Each module in the above-mentioned electronic device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0183] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned model training method when executed by a processor. The computer-readable storage medium can be volatile or non-volatile computer-readable storage medium.

[0184] An embodiment of the present disclosure also provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above-mentioned model training method.

[0185] It will be understood by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium).

[0186] As is well known to those skilled in the art, the term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information (such as computer-readable program instructions, data structures, program modules or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technology, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as is well known to those skilled in the art, communication media typically contains computer-readable program instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0187] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0188] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0189] The computer program product described herein may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0190] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0191] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0192] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0193] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0194] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A model training method, characterized in that: include: Extract features from speech samples to obtain speech features; Inputting the speech features into a coding network in a model to be trained for coding processing, wherein the coding network includes cascaded coding layers, and the coding layers include an intermediate coding layer and an Nth coding layer; Decoding an intermediate coding feature to obtain an additional loss, wherein the intermediate coding feature is a coding feature output by the intermediate coding layer; Obtaining a coding loss based on the coding features and coding labels output by the Nth coding layer; Obtaining a total encoding loss based on the additional loss, the encoding loss, and a preset first loss weight; Inputting the coding features output by the Nth coding layer into the decoding network for decoding processing to obtain a total decoding loss; Obtaining a total model loss based on the total encoding loss, the total decoding loss, and a preset second loss weight; Based on the total loss of the model, the parameters in the encoding network and the decoding network are updated, and the model to be trained is trained according to the updated parameters until the total loss of the model converges, thereby obtaining a trained model.

2. The method according to claim 1, characterized in that The decoding of the intermediate encoding features to obtain the additional loss includes: Using an additional decoding network, decoding the intermediate encoding features to obtain additional decoding features; The additional loss is obtained based on the additional decoding feature and a preset additional decoding label.

3. The method according to claim 1, characterized in that The intermediate coding features include intermediate coding features at one-third of the coding network and / or intermediate coding features at two-thirds of the coding network.

4. The method according to any one of claims 1 to 3, characterized in that Inputting the coding features output by the Nth coding layer into the decoding network for decoding processing to obtain the total decoding loss includes: Obtaining an intermediate loss based on the decoding features and intermediate decoding labels output by the intermediate decoding layer, wherein the decoding network includes the intermediate decoding layer and the Mth decoding layer; Obtaining a decoding loss based on the decoding features and decoding labels output by the M-th decoding layer; The total decoding loss is obtained based on the intermediate loss, the decoding loss and a preset third loss weight.

5. The method according to any one of claims 1 to 3, characterized in that The following steps are performed in the encoding network: Obtain the previous layer score matrix based on the query matrix and key value matrix in the previous encoding layer; Obtain the current layer score matrix based on the query matrix and key value matrix in the current encoding layer; The previous layer score matrix and the current layer score matrix are combined to obtain the coding features, and the coding features are input into the next coding layer.

6. The method according to claim 1, characterized in that Before decoding the intermediate encoding features to obtain the additional loss, the method further includes: Obtaining a first coding loss based on the coding features and coding labels output by the coding layer of the Nth layer; Inputting the encoding features output by the Nth encoding layer into the decoding network to be trained for decoding processing to obtain a first total decoding loss; Obtaining a first model total loss based on the first encoding total loss, the first decoding total loss, and a preset second loss weight; Updating parameters in the encoding network and the decoding network based on the total loss of the first model, training the model to be trained until a preset condition is met, and obtaining a pre-trained model; The parameters of the pre-trained model are used as the initial parameters of the encoding network and the decoding network.

7. The method according to claim 1, characterized in that The updating of parameters in the encoding network and the decoding network based on the total loss of the model includes: In the parameter updating stage, the parameters in the encoding network and the decoding network are updated based on the total loss of the model and the regularization term.

8. The method according to claim 1, characterized in that The feature extraction of the speech sample, before obtaining the speech features, includes: Acquire a voice sample, and segment the voice sample to obtain voice segments; Label the speech segments that belong to noise and obtain noise labels.

9. A speech recognition model, characterized in that The speech recognition model is a model obtained by the model training method described in any one of claims 1-8.

10. A model training device, characterized in that: include: Extraction module, used to extract features from speech samples to obtain speech features; An encoding module, configured to input the speech features into an encoding network in a model to be trained for encoding processing, wherein the encoding network includes cascaded encoding layers, and the encoding layers include an intermediate encoding layer and an Nth encoding layer; An additional module, configured to decode an intermediate coding feature to obtain an additional loss, wherein the intermediate coding feature is a coding feature output by the intermediate coding layer; A calculation module, configured to obtain a coding loss based on the coding features and coding labels output by the Nth coding layer; and obtaining a total coding loss based on the additional loss, the coding loss, and a preset first loss weight; The decoding module is further configured to input the coding features output by the Nth coding layer into the decoding network for decoding processing to obtain a total decoding loss; The calculation module is further configured to obtain a total model loss based on the total encoding loss, the total decoding loss, and a preset second loss weight; An updating module is used to update the parameters in the encoding network and the decoding network based on the total loss of the model, and train the model to be trained according to the updated parameters until the total loss of the model converges to obtain a trained model.

11. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to execute the model training method according to any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When executed by a processor, the computer program implements the model training method according to any one of claims 1 to 8.

13. A computer program product, characterized in that It includes a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the model training method as described in any one of claims 1 to 8.

Images