Speech recognition model training method and device, speech recognition method, and medium

Abstract

The embodiment relates to the technical field of artificial intelligence, and in particular to a speech recognition model training method and device, a speech recognition method and a medium. The speech recognition model training method comprises the following steps: acquiring a speech feature sequence and a frame-by-frame letter sequence with a blank flag; generating a global feature sequence according to the speech feature sequence; obtaining a connection time classification loss function according to the speech feature sequence, the global feature sequence and the frame-by-frame letter sequence; obtaining an attention loss function according to the speech feature sequence and the global feature sequence; and updating parameters of a speech recognition model according to the connection time classification loss function and the attention loss function to obtain a trained speech recognition model. The technical scheme of the embodiment of the application can simplify the training steps of the speech recognition model and improve the efficiency of generating a dialogue by introducing the connection time classification loss function and the attention loss function.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech recognition model training method and apparatus, a speech recognition method, and a medium. Background Technology

[0002] Automatic Speech Recognition (ASR) has gradually replaced traditional Deep Neural Networks (DNNs) and Hidden Markov Models (HMMs) due to its simple modeling process and good performance. ASR avoids the need for resources such as speech dictionaries, text alignment, and context dependency trees, thus greatly simplifying the model building process. There are two main end-to-end architectures for ASR: attention-based methods use attention mechanisms to perform alignment between acoustic frames and recognized symbols; while Connectionist Temporal Classification (CTC) uses Markov assumptions to efficiently solve sequence problems through dynamic programming.

[0003] Currently, end-to-end ASR joint decoding algorithms based on attention and CTC structures effectively utilize attention mechanisms and CTC modeling, achieving better model performance. However, when applied to long-term modeling, the large number of similar sequences in long-term sequences leads to excessive weight dependence of the attention mechanism, resulting in disorder of the attention mechanism. This complicates the training steps of the speech recognition model and affects the efficiency of the model in generating dialogue. Summary of the Invention

[0004] The main objective of this disclosure is to propose a speech recognition model training method and apparatus, a speech recognition method, and a medium, which can simplify the training steps of the speech recognition model and improve the efficiency of the model in generating dialogue.

[0005] To achieve the above objectives, a first aspect of this disclosure provides a speech recognition model training method for training a speech recognition model, the method comprising:

[0006] Obtain the speech feature sequence and the frame-by-frame letter sequence marked with whitespace;

[0007] Generate a global feature sequence based on the speech feature sequence;

[0008] Based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence, the connection time classification loss function is obtained;

[0009] Based on the speech feature sequence and the global feature sequence, the attention loss function is obtained;

[0010] The parameters of the speech recognition model are updated according to the connection time classification loss function and the attention loss function to obtain the trained speech recognition model.

[0011] In some embodiments, generating a global feature sequence based on the speech feature sequence includes:

[0012] The speech feature sequence is processed by several convolutional enhancement transformers within the encoder to obtain the global feature sequence. Each convolutional enhancement transformer contains two feedforward modules, and the convolutional layer and attention layer of the convolutional enhancement transformer are located between the two feedforward modules.

[0013] In some embodiments, obtaining the connection-temporal classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence includes:

[0014] Based on the probability chain rule, a conditional independence assumption is made on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence to obtain the connection time classification posterior distribution of the speech feature sequence and the global feature sequence;

[0015] Based on the connection time classification posterior distribution, obtain the connection time classification objective function;

[0016] Based on the connection-time classification posterior distribution and the connection-time classification objective function, obtain the connection-time classification loss function.

[0017] In some embodiments, obtaining a connection-time classification loss function based on the connection-time classification posterior distribution and the connection-time classification objective function includes:

[0018] The connection-time classification objective function is used to calculate the sequence result after processing the speech feature sequence and the frame-by-frame letter sequence using the connection-time classification posterior distribution;

[0019] Summing the sequence results yields the connection-time classification loss function.

[0020] In some embodiments, obtaining the attention loss function based on the speech feature sequence and the global feature sequence includes:

[0021] Based on the probability chain rule, a conditional independence assumption is made between the speech feature sequence and the global feature sequence to obtain the attention posterior distribution of the speech feature sequence and the global feature sequence;

[0022] The cross-entropy loss of the decoder is calculated based on the attention posterior distribution.

[0023] Calculate the first attention weight based on the similarity value between the encoder and the decoder;

[0024] The second attention weight is calculated based on the triples, where the triples include the start time, end time, and time span.

[0025] Based on the first attention weight and the second attention weight, the additional loss function is obtained;

[0026] The attention loss function is obtained based on the cross-entropy loss and the additional loss function.

[0027] In some embodiments, updating the parameters of the speech recognition model according to the connection-time classification loss function and the attention loss function to obtain the trained speech recognition model includes:

[0028] The final loss function is obtained by weighted summing of the connection-time classification loss function and the attention loss function.

[0029] The speech recognition model is trained using a gradient update algorithm, and the parameters of the speech recognition model are updated.

[0030] To achieve the above objectives, a second aspect of this disclosure provides a speech recognition method, the method comprising:

[0031] Obtain the current dialogue information of the dialogue system;

[0032] The current dialogue information is input into a speech recognition model for speech recognition processing to obtain the target dialogue; wherein the speech recognition model is trained according to the method described in any one of the first aspect embodiments.

[0033] To achieve the above objectives, a third aspect of this disclosure provides a speech recognition model training apparatus, the apparatus comprising:

[0034] The acquisition module is used to acquire speech feature sequences and frame-by-frame letter sequences marked with whitespace.

[0035] The generation module is used to generate a global feature sequence based on the speech feature sequence;

[0036] The processing module is configured to obtain a connection time classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence; and to obtain an attention loss function based on the speech feature sequence and the global feature sequence.

[0037] The training module is used to update the parameters of the speech recognition model according to the connection time classification loss function and the attention loss function to obtain the trained speech recognition model.

[0038] To achieve the above objectives, a fourth aspect of the present disclosure provides an electronic device including at least one memory;

[0039] At least one processor;

[0040] At least one computer program;

[0041] The computer program is stored in the memory, and the processor executes the at least one computer program to achieve:

[0042] The method as described in any of the embodiments of the first aspect; or

[0043] The method described in the second aspect embodiment.

[0044] To achieve the above objectives, a fifth aspect of this disclosure also provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform:

[0045] The method as described in any of the embodiments of the first aspect; or

[0046] The method described in the second aspect embodiment.

[0047] The speech recognition model training method and apparatus, speech recognition method, and medium proposed in this disclosure involve: acquiring a speech feature sequence and a frame-by-frame letter sequence marked with whitespace; inputting the speech feature sequence into an encoder to obtain a global feature sequence, wherein the encoder has at least two feedforward modules; obtaining a connection-temporal classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence; obtaining an attention loss function based on the speech feature sequence and the global feature sequence; and updating the parameters of the speech recognition model based on the connection-temporal classification loss function and the attention loss function to obtain a trained speech recognition model. This simplifies the training steps of the speech recognition model and improves the efficiency of the model in generating dialogue. Attached Figure Description

[0048] Figure 1 This is a flowchart of the speech recognition model training method provided in the embodiments of this application;

[0049] Figure 2 yes Figure 1 A flowchart illustrating the specific method of step S200;

[0050] Figure 3 yes Figure 2 A schematic diagram of the structure of the convolutional enhancement transformer;

[0051] Figure 4 yes Figure 1A flowchart illustrating the specific method of step S300;

[0052] Figure 5 yes Figure 4 A flowchart illustrating the specific method of step S330 in the middle section;

[0053] Figure 6 yes Figure 1 A flowchart illustrating the specific method of step S400;

[0054] Figure 7 yes Figure 1 A flowchart illustrating the specific method of step S500;

[0055] Figure 8 This is a flowchart of the speech recognition method provided in the embodiments of this application;

[0056] Figure 9 This is a block diagram of the speech recognition model training device provided in the embodiments of this application;

[0057] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0061] First, let's analyze some of the terms used in this application:

[0062] Artificial Intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0063] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0064] Embedding: The embedding layer is a word embedding jointly learned with a neural network model specific to natural language processing. This embedding method performs one-hot encoding on words in cleaned text, and the size or dimension of the vector space is specified as part of the model, such as 50, 100, or 300 dimensions. The vectors are initialized with small random numbers. The embedding layer is used as the front end of the neural network and is supervised by backpropagation. Encoded words are mapped to word vectors, which are concatenated before being input into the model if a multilayer perceptron (MLP) model is used. If a recurrent neural network (RNN) is used, each word can be used as an input to the sequence. This method of learning embedding layers requires a large amount of training data and can be slow, but it can train embedding models that are suitable for both text-specific data and NLP. Embedding is a vector representation that uses a low-dimensional vector to represent an object, such as a word, a product, or a movie. The property of an embedding vector is that vectors that are close in distance correspond to objects with similar meanings. For example, the embeddings "Avengers" and "Iron Man" are very close, but the embeddings "Avengers" and "Gone with the Wind" are farther apart. Essentially, embedding is a mapping from semantic space to vector space, while preserving as much of the semantic relationship as possible in the vector space. For instance, two semantically similar words are also relatively close in the vector space. Embedding can encode objects using low-dimensional vectors while retaining their meaning. It is commonly used in machine learning. In the process of building machine learning models, objects are encoded as low-dimensional dense vectors before being fed to a deep neural network (DNN) to improve efficiency.

[0065] Bellman equations: Bellman equations are equations used in control theory, also known as dynamic programming equations. Bellman equations are a necessary condition for dynamic programming and other mathematical optimization methods to achieve optimality. This equation expresses the "value of a decision problem at a specific time" as "the ratio of the reward from the initial choice to the value of the decision problem derived from the initial choice." In this way, the dynamic optimization problem is transformed into simpler subproblems that obey the "optimization principle" proposed by Bellman.

[0066] DQN: Similar to Q-learning, DQN is a value-iterative algorithm. However, in ordinary Q-learning, when the state and action spaces are discrete and of low dimensionality, a Q-table can be used to store the Q-values ​​of each state-action pair. But when the state and action spaces are high-dimensional and continuous, using a Q-table becomes very difficult due to the large size of the action and state spaces. Therefore, the Q-table update can be transformed into a function fitting problem. A function is fitted to replace the Q-table to generate Q-values, ensuring that similar states produce similar output actions. This leads us to consider the effectiveness of deep neural networks in extracting complex features. Therefore, combining Deep Learning with Reinforcement Learning results in DQN.

[0067] Currently, ASR employs industry-leading deep learning algorithms to convert various speech formats into text. It comprehensively meets the speech recognition needs of multiple scenarios, including voice navigation, voice search, real-time input, live streaming captions, and content moderation.

[0068] In related technologies, the end-to-end ASR joint decoding algorithm based on attention and CTC structure effectively utilizes the attention mechanism and CTC modeling, achieving better model performance. However, when applied to long-term modeling, the large number of similar sequences in the long-term sequence leads to excessive weight dependence of the attention mechanism, resulting in the disorder of the attention mechanism, which complicates the training steps of the speech recognition model and affects the efficiency of the model in generating dialogue.

[0069] Based on this, the present disclosure provides a speech recognition model training method and apparatus, a speech recognition method, and a medium, which can simplify the training steps of the speech recognition model and improve the efficiency of the model in generating dialogue.

[0070] This application provides a speech recognition model training method and apparatus, a speech recognition method, and an electronic device, which are specifically described through the following embodiments. First, the speech recognition model training method in the embodiments of this disclosure is described.

[0071] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0072] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0073] The speech recognition model training method and apparatus, speech recognition method, and medium provided in this application relate to the field of artificial intelligence technology, and particularly to the field of data mining technology. The speech recognition model training method or speech recognition method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc.; the server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application implementing the speech recognition model training method or speech recognition method, but is not limited to the above forms.

[0074] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0075] Firstly, please refer to Figure 1 , Figure 1This is an optional flowchart of a speech recognition model training method provided in some embodiments of this application. The speech recognition model training method includes steps S100, S200, S300, S400, and S500. It should be understood that the speech recognition model training method in the embodiments of this application includes, but is not limited to, steps S100 to S500. The following describes the process in conjunction with... Figure 1 These five steps will be explained in detail.

[0076] Step S100: Obtain the speech feature sequence and the frame-by-frame letter sequence marked with blanks.

[0077] Specifically, in step S100 of some embodiments, voice data is acquired through an audio input device and converted into a voice feature sequence that can be directly read by a voice recognition model. The audio input device can be a conference microphone, live streaming microphone, singing microphone, narration microphone, etc., and different audio input devices are used to collect voice data in different scenarios to ensure the stability and accuracy of voice data acquisition.

[0078] In step S100 of some embodiments of this application, the collected speech data is converted to obtain a speech feature sequence X = {x}. i ∈R D |i=1,...,T},where T is the length of the speech feature sequence, x i This is the D-dimensional speech feature vector at frame i. The process of converting speech data into a speech feature sequence is existing technology and will not be elaborated upon here.

[0079] In the CTC architecture, a letter sequence of length L, C = {c l Let {z ∈ U | l = 1, ..., L} to represent a set U with distinct letters. To efficiently solve the sequence problem, a frame-by-frame letter sequence Z = {z ∈ U | l = 1, ..., L} with an additional whitespace marker b is introduced. t ∈UU|t=1,...,T}.

[0080] In step S100 of some embodiments of this application, the speech feature sequence refers to the user's speech data obtained by the audio input device after conversion.

[0081] It should be noted that the content of the speech feature sequence includes multiple sets of speech data. For example, one minute of speech data collected by the current audio input device can be extracted and then converted into the corresponding speech feature sequence. The length T of the speech feature sequence varies according to the length of the speech data.

[0082] Step S200: Generate a global feature sequence based on the speech feature sequence.

[0083] Specifically, in step S200 of some embodiments, a convolution enhancement transformer is used as the encoder, including a feed forward module, a multi-headed self-attention module, and a convolution module. The feedforward module mainly consists of two linear layers and a nonlinear activation in the middle. After adding residual connections to the feedforward layers, layer normalization is performed, and Swish activation and dropout are applied to regularize the network parameters. The self-attention module adopts a relative sinusoidal positional encoding scheme from the Transformer model in this embodiment. Relative positional encoding allows the self-attention module to generalize better at different input lengths, and the generated encoder is more robust to changes in speech length. At the same time, dropout technology is applied to the residual units to accelerate model training convergence and increase network depth. The convolution module consists of eight parts and a residual summation operation. It is formed by layer normalization, then pointwise convolutional layers and gated linear units (GLUs), one-dimensional depth convolution, batch normalization, Swish activation, pointwise convolutional layers, dropout regularization, and finally residual summation operation.

[0084] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a specific method for step S200 in some embodiments of this application. In some embodiments of this application, step S200 includes, but is not limited to, steps S210, S220, and S230, which will be discussed below. Figure 2 These three steps will be explained in detail.

[0085] Step S210: The speech feature sequence is processed by several convolutional enhancement transformers within the encoder.

[0086] Specifically, in step S210 of some embodiments, a convolutional enhancement transformer is used as an encoder. First, the input is sampled twice by a convolution, then it goes through a linear layer and a dropout regularization, followed by several convolutional enhancement transformers.

[0087] Step S220: The convolution enhancement transformer has two feedforward modules, and the convolutional layer and attention layer of the convolution enhancement transformer are located between the two feedforward modules.

[0088] Specifically, in step S220 of some embodiments, the convolutional enhancement transformer acts as an encoder, replacing the original feedforward layer in the Transformer model with two half-step feedforward layers located before the convolutional layer and after the attention layer, respectively. The feedforward modules utilize half-step residual weights. A convolutional enhancement transformer resembles a macaron structure, mainly consisting of four parts: a first feedforward module, a self-attention module, a convolutional module, and a second feedforward module. The structure of the convolutional enhancement transformer is as follows: Figure 3 As shown.

[0089] Step S230: Obtain the global feature sequence.

[0090] Specifically, in step S230 of some embodiments, after the speech feature sequence X is processed by the encoder, a global feature sequence Y is output, and the processing is shown in the following formula:

[0091]

[0092]

[0093] x″ i =x′ i +Conv(x′ i )

[0094] y i =Layernorm(x″) i +1 / 2FFN(x″ i ))

[0095] Among them, the speech feature sequence x i The input Conformer block is passed through the feedforward module to obtain... After passing through the self-attention module, x′ is obtained. i After passing through the convolution module, x″ is obtained. i All three operations described above employ residual block computation. Finally, through a half-step feedforward module and a layer normalization module, the convolution enhancement transformer outputs the result y. i This forms the global feature sequence Y.

[0096] Here, FFN refers to the feedforward module, MHSA refers to the multi-head self-attention module, and Conv refers to the output of the last block of the convolution module being equivalent to the hidden parameters.

[0097] It should be noted that in the Conformer architecture, there are two macaron network-style feedforward layers with half-step residual connections, sandwiching the attention and convolutional modules in the middle. The multi-head self-attention module is good at capturing dependencies in long sequences, while the convolutional module is good at capturing local information. The Conformer block combines the two for modeling, which can capture global information better than a standalone multi-head self-attention module or convolutional model.

[0098] Step S300: Obtain the connection time classification loss function based on the speech feature sequence, global feature sequence, and frame-by-frame letter sequence.

[0099] In step S300 of some embodiments, in the CTC architecture, the computation formula follows Bayesian decision theory, thus estimating the most likely word sequence among all possible word sequences V. Satisfy the following formula:

[0100]

[0101] Where W = {w n ∈V|n=1,…,N} is a word sequence of length N, w n It is the word at position n in vocabulary list V.

[0102] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating a specific method of step S300 in some embodiments of this application. In some embodiments of this application, step S300 includes, but is not limited to, steps S310, S320, and S330, which will be further described below. Figure 4 These three steps will be explained in detail.

[0103] Step S310: Based on the probability chain rule, make conditional independence assumptions on the speech feature sequence, global feature sequence, and frame-by-frame letter sequence to obtain the connection time classification posterior distribution of the speech feature sequence and the global feature sequence.

[0104] Specifically, in step S310 of some embodiments, conditional independence assumptions are made for the speech feature sequence X, global feature sequence Y, and frame-by-frame letter sequence Z according to the probability chain rule, and the connection time classification posterior distribution is obtained as shown in the following formula:

[0105]

[0106] Wherein, p(z) t |X) is the posterior distribution of the frame, p(z) t |z t-1 C) represents the transition probability. Additionally, p in the above formula... ctc (C|X) is the objective function for connection-time classification.

[0107] Step S320: Obtain the objective function for connection time classification based on the posterior distribution of connection time classification.

[0108] Specifically, in step S320 of some embodiments, the connection time classification objective function p ctc (C|X) is obtained from the formula in step S310 above.

[0109] Step S330: Obtain the connection-time classification loss function based on the connection-time classification posterior distribution and the connection-time classification objective function.

[0110] Specifically, in step S330 of some embodiments, it is necessary to process the sum of all possible sequences for p(C|X) in the formula of step S310, that is, to calculate the connection-time classification loss function L. CTC Due to the Markov property, dynamic programming can be used for efficient computation.

[0111] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating the specific method of step S330 in some embodiments of this application. In some embodiments of this application, step S330 includes, but is not limited to, steps S331 and S332, which are described below in conjunction with... Figure 5 These two steps will be explained in detail.

[0112] Step S331: Calculate the sequence results after processing the speech feature sequence and frame-by-frame letter sequence using the connection time classification objective function.

[0113] Specifically, in step S331 of some embodiments, in order to calculate the complete connection-time classification loss function, the connection-time classification posterior distribution needs to process the sum of all possible sequences. Here, the processing results of the speech feature sequence and the frame-by-frame letter sequence are selected as the result of p(C|X) processing all possible sequences.

[0114] Step S332: Sum the sequence results to obtain the connection-time classification loss function.

[0115] Specifically, in step S332 of some embodiments, the sequence results obtained in step S331 are summed to obtain the connection time classification loss function L. CTC .

[0116] Step S400: Obtain the attention loss function based on the speech feature sequence and the global feature sequence.

[0117] In step S400 of some embodiments, compared with the CTC architecture, the attention-based architecture does not make a conditional independence assumption, but directly estimates the posterior probability of the speech feature sequence based on the probability chain rule. Furthermore, to align the speech feature sequence with the global feature sequence, an additional loss function is introduced, treating the correspondence between attention weights and alignment as a learning problem.

[0118] Please refer to Figure 6 , Figure 6 This is a flowchart illustrating a specific method for step S400 in some embodiments of this application. In some embodiments of this application, step S400 includes, but is not limited to, steps S410, S420, S430, S440, S450, and S460, which are described below in conjunction with… Figure 6 These six steps will be explained in detail.

[0119] Step S410: Based on the probability chain rule, make a conditional independence assumption on the speech feature sequence and the global feature sequence to obtain the attention posterior distribution of the speech feature sequence and the global feature sequence.

[0120] In step S410 of some embodiments, a conditional independence assumption is made on the speech feature sequence X and the global feature sequence Y according to the probability chain rule, and the attention posterior distribution is obtained as shown in the following formula:

[0121]

[0122] Where, p att (C|X) is the attention objective function.

[0123] Step S420: Calculate the cross-entropy loss of the decoder based on the attention posterior distribution.

[0124] In step S420 of some embodiments, the cross-entropy loss L of the decoder is obtained based on the attention posterior distribution obtained in step S410. ce Cross-entropy loss is one of the most widely used loss functions in deep learning. The calculation of cross-entropy loss during decoding is a current technique and will not be elaborated here.

[0125] Step S430: Calculate the first attention weight based on the similarity value between the encoder and decoder.

[0126] In step S430 of some embodiments, a similarity value between the encoder and decoder, i.e., a first attention weight α, is introduced. Each element α within the first attention weight α... k，i Used to characterize the encoder's hidden vector h i and decoder hidden vector d kThe similarity score is calculated. Calculating the similarity value between the encoder and decoder is a common method for calculating the loss function in deep learning. Obtaining the first attention weight α is an existing technique and will not be elaborated here.

[0127] Step S440: Calculate the second attention weight based on the triples, where the triples include the start time, end time, and time span.

[0128] In step S440 of some embodiments, the alignment is a triple (s1, e1, y1), ..., (s k e k y k ), where s k It is the start time, e k It is the end time, y k This represents the time span; this application converts alignment into a second attention weight α. * The calculation process, which involves assigning uniform weights to the vector at the corresponding start and end times, is shown in the following formula:

[0129]

[0130] Where, if c is true, then l c The value is 1 if the value is positive and 0 otherwise. When double sampling is involved, the corresponding weights need to be added together using the coefficient of the attention weight value γ. Specifically, the attention weights for k and t′ after double sampling are:

[0131] Step S450: Obtain the additional loss function based on the first attention weight and the second attention weight.

[0132] In step S450 of some embodiments, the first attention weight α and the second attention weight α are obtained. * After the attention weight value γ, the Frobenius norm is used to measure α and α'. * The distance between them is calculated, and the additional loss function is calculated. The specific calculation is shown in the following formula:

[0133]

[0134] Among them, L attn For α and α * Additional loss function between them.

[0135] Step S460: Obtain the attention loss function based on the cross-entropy loss and the additional loss function.

[0136] In step S460 of some embodiments, after obtaining the additional loss function, the cross-entropy loss and the additional loss function are weighted and summed using the attention weight value γ, as shown in the following formula:

[0137] L = L ce +γL attn

[0138] Where L is the attention loss function.

[0139] Step S500: Update the parameters of the speech recognition model according to the connection time classification loss function and the attention loss function to obtain the trained speech recognition model.

[0140] In step S500 of some embodiments, after obtaining the connection-temporal classification loss function and the attention loss function, it is necessary to combine CTC and supervised attention mechanism for joint decoding. In this application, CTC and attention-based sequence probabilities are combined during the process of obtaining the connection-temporal classification loss function and the attention loss function in the above steps.

[0141] Please refer to Figure 7 , Figure 7 This is a flowchart illustrating a specific method for step S500 in some embodiments of this application. In some embodiments of this application, step S500 includes, but is not limited to, steps S510 and S520, which will be discussed below. Figure 7 These two steps will be explained in detail.

[0142] Step S510: Weighted summation of the connection time classification loss function and attention loss function to obtain the final loss function.

[0143] In step S510 of some embodiments, by weighted summation of the connection-time classification loss function and the attention loss function, the sequence probabilities of CTC and the attention mechanism can be combined to model the local and global dependencies of the speech feature sequence in a parameter-efficient manner. The final calculation process of the loss function is shown in the following formula:

[0144] L all =λL CTC +(1-λ)L=λL CTC +(1-λ)(L ce +γL attn )

[0145] Where the hyperparameter λ: 0≤λ≤1, L all This is the final loss function.

[0146] Step S520: Train the speech recognition model using the gradient update algorithm and update the parameters of the speech recognition model.

[0147] In step S520 of some embodiments, the above-mentioned network model is trained using the gradient update algorithm (Adam). The decoding target of the speech recognition model is defined as shown in the following formula:

[0148]

[0149] in, The goal is to find the most likely letter sequence. It's understandable that CTC probabilities enforce monotonic alignment, disallowing large jumps or loops within the same frame. Therefore, without relying on overlay terms, length penalties, or minimum / maximum lengths, we can choose hypotheses with better alignment and exclude irrelevant ones.

[0150] The speech recognition model training method proposed in this application involves: acquiring a speech feature sequence; inputting the speech feature sequence into an encoder to obtain a global feature sequence, wherein the encoder has two feedforward modules; acquiring a frame-by-frame letter sequence with blank markers; obtaining a connection-temporal classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence; obtaining an attention loss function based on the speech feature sequence and the global feature sequence; and updating the parameters of the speech recognition model based on the connection-temporal classification loss function and the attention loss function to obtain a trained speech recognition model. This method simplifies the training steps of the speech recognition model and improves the efficiency of the model in generating dialogue.

[0151] Secondly, please refer to Figure 8 , Figure 8 This is a flowchart of a speech recognition method provided in some embodiments of this application. The speech recognition method includes, but is not limited to, steps S600 and S700. The following is in conjunction with… Figure 8 These two steps will be explained in detail.

[0152] Step S600: Obtain the current dialogue information of the dialogue system;

[0153] Step S700: Input the current dialogue information into the speech recognition model for speech recognition processing to obtain the target dialogue; wherein, the speech recognition model is trained according to any one of the methods in the first aspect embodiment.

[0154] Specifically, in some embodiments of this application, the current dialogue information of the dialogue system is first obtained, and the current dialogue information is input into the speech recognition model for speech recognition processing to obtain the target dialogue.

[0155] Specifically, the speech feature sequence is input into the speech recognition model to obtain the global feature sequence. The global feature sequence is then input into the already trained speech recognition model for speech recognition processing, and the target dialogue is output. This simplifies the training steps of the speech recognition model and improves the efficiency of the model in generating dialogue.

[0156] The speech recognition method of this application embodiment obtains a speech feature sequence; inputs the speech feature sequence into an encoder to obtain a global feature sequence, wherein the encoder has two feedforward modules; obtains a frame-by-frame letter sequence with blank markers, and obtains a connection time classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence; obtains an attention loss function based on the speech feature sequence and the global feature sequence; and updates the parameters of the speech recognition model based on the connection time classification loss function and the attention loss function to obtain a trained speech recognition model. This method simplifies the training steps of the speech recognition model and improves the efficiency of the model in generating dialogue.

[0157] It should be noted that in the embodiments of this application, the speech recognition model is updated in real time. For example, in an audio conferencing application scenario, the conference microphone continuously inputs speech feature sequences into the speech recognition model. The previously input speech feature sequences iteratively update the speech recognition model, updating its parameters to obtain a speech recognition model used to generate the next target dialogue. Alternatively, there may be other settings, for which this application does not impose specific limitations.

[0158] Please refer to Figure 9 , Figure 9 This is a block diagram of a speech recognition model training apparatus provided in some embodiments of this application. In some embodiments, the speech recognition model training apparatus 800 includes an acquisition module 810, a generation module 820, a processing module 830, and a training module 840.

[0159] The acquisition module 810 is used to acquire speech feature sequences and frame-by-frame letter sequences marked with whitespace.

[0160] The generation module 820 is used to generate a global feature sequence based on the speech feature sequence;

[0161] The processing module 830 is used to obtain the connection time classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence; and to obtain the attention loss function based on the speech feature sequence and the global feature sequence.

[0162] The training module 840 is used to update the parameters of the speech recognition model according to the connection time classification loss function and the attention loss function, so as to obtain the trained speech recognition model.

[0163] The speech recognition model training device of this application embodiment acquires a speech feature sequence; inputs the speech feature sequence into an encoder to obtain a global feature sequence, wherein the encoder is provided with two feedforward modules; acquires a frame-by-frame letter sequence with blank markers, and obtains a connection time classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence; obtains an attention loss function based on the speech feature sequence and the global feature sequence; updates the parameters of the speech recognition model based on the connection time classification loss function and the attention loss function, and obtains a trained speech recognition model. This simplifies the training steps of the speech recognition model and improves the efficiency of the model in generating dialogue.

[0164] It should be noted that the speech recognition model training device in this application corresponds to the aforementioned speech recognition model training method. For the specific training process, please refer to the aforementioned speech recognition model training method, which will not be described in detail here.

[0165] This application also provides an electronic device, including:

[0166] At least one memory;

[0167] At least one processor;

[0168] At least one program;

[0169] The program is stored in memory, and the processor executes at least one program to implement the speech recognition model training method or speech recognition method described above in this disclosure. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), in-vehicle computers, etc.

[0170] The electronic device of this application embodiment is used to execute the above-described speech recognition model training method or speech recognition method. It acquires a speech feature sequence; inputs the speech feature sequence into an encoder to obtain a global feature sequence, wherein the encoder has two feedforward modules; acquires a frame-by-frame letter sequence with blank markers, and obtains a connection-temporal classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence; obtains an attention loss function based on the speech feature sequence and the global feature sequence; and updates the parameters of the speech recognition model based on the connection-temporal classification loss function and the attention loss function to obtain a trained speech recognition model. This simplifies the training steps of the speech recognition model and improves the efficiency of the model in generating dialogue.

[0171] The following is combined Figure 10 The electronic devices described in the embodiments of this application will be described in detail.

[0172] like Figure 10 , Figure 10The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0173] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0174] The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to execute the speech recognition model training method or speech recognition method of the embodiments of this disclosure.

[0175] The input / output interface 1800 is used to implement information input and output.

[0176] The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0177] Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900);

[0178] The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0179] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described speech recognition model training method or speech recognition method.

[0180] The storage medium of this application embodiment is used to execute the above-described speech recognition model training method or speech recognition method. This involves acquiring a speech feature sequence; inputting the speech feature sequence into an encoder to obtain a global feature sequence, wherein the encoder has two feedforward modules; acquiring a frame-by-frame letter sequence with blank markers; and obtaining a connection-temporal classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence; obtaining an attention loss function based on the speech feature sequence and the global feature sequence; and updating the parameters of the speech recognition model based on the connection-temporal classification loss function and the attention loss function to obtain a trained speech recognition model. This simplifies the training steps of the speech recognition model and improves the efficiency of the model in generating dialogue.

[0181] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0182] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0183] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0186] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0187] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0191] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] The preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present disclosure shall be within the scope of the claims of the present disclosure.

Claims

1. A speech recognition model training method, used to train a speech recognition model, characterized in that, The method includes: Obtain the speech feature sequence and the frame-by-frame letter sequence marked with whitespace; The speech feature sequence is processed by several convolutional enhancement transformers in the encoder to obtain a global feature sequence. The convolutional enhancement transformer includes a first feedforward module, a self-attention module, a convolution module, and a second feedforward module. The feedforward module uses half-step residual weights. Based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence, the connection time classification loss function is obtained; Based on the probability chain rule, a conditional independence assumption is made between the speech feature sequence and the global feature sequence to obtain the attention posterior distribution of the speech feature sequence and the global feature sequence; The cross-entropy loss of the decoder is calculated based on the attention posterior distribution. Calculate the first attention weight based on the similarity value between the encoder and the decoder; The second attention weight is calculated based on the triples, which include the start time, end time, and time span, to convert alignment into second attention weights by uniformly assigning weights to the vector at the corresponding start and end times. Based on the first attention weight and the second attention weight, the additional loss function is obtained; Based on the cross-entropy loss and the additional loss function, the attention loss function is obtained; The parameters of the speech recognition model are updated according to the connection time classification loss function and the attention loss function to obtain the trained speech recognition model.

2. The method according to claim 1, characterized in that, The step of obtaining the connection-time classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence includes: Based on the probability chain rule, a conditional independence assumption is made on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence to obtain the connection time classification posterior distribution of the speech feature sequence and the global feature sequence; Based on the connection time classification posterior distribution, obtain the connection time classification objective function; Based on the connection-time classification posterior distribution and the connection-time classification objective function, obtain the connection-time classification loss function.

3. The method according to claim 2, characterized in that, The step of obtaining the connection-time classification loss function based on the connection-time classification posterior distribution and the connection-time classification objective function includes: The connection-time classification objective function is used to calculate the sequence result after processing the speech feature sequence and the frame-by-frame letter sequence using the connection-time classification posterior distribution; Summing the sequence results yields the connection-time classification loss function.

4. The method according to claim 3, characterized in that, The step of updating the parameters of the speech recognition model according to the connection time classification loss function and the attention loss function to obtain the trained speech recognition model includes: The final loss function is obtained by weighted summing of the connection-time classification loss function and the attention loss function. The speech recognition model is trained using a gradient update algorithm, and the parameters of the speech recognition model are updated.

5. A speech recognition method, characterized in that, The method includes: Obtain the current dialogue information of the dialogue system; The current dialogue information is input into a speech recognition model for speech recognition processing to obtain the target dialogue; wherein the speech recognition model is trained according to the method described in any one of claims 1 to 4.

6. A model training device, characterized in that, The device includes: The acquisition module is used to acquire speech feature sequences and frame-by-frame letter sequences marked with whitespace. The generation module is used to process the speech feature sequence through several convolutional enhancement transformers in the encoder to obtain a global feature sequence. The convolutional enhancement transformer includes a first feedforward module, a self-attention module, a convolution module, and a second feedforward module. The feedforward module uses half-step residual weights. The processing module is configured to: obtain a connection-temporal classification loss function based on the speech feature sequence, the global feature sequence, and the frame-by-frame letter sequence; make a conditional independence assumption on the speech feature sequence and the global feature sequence according to the probability chain rule to obtain the attention posterior distribution of the speech feature sequence and the global feature sequence; calculate the cross-entropy loss of the decoder based on the attention posterior distribution; calculate a first attention weight based on the similarity value between the encoder and the decoder; calculate a second attention weight based on triples, wherein the triples include a start time, an end time, and a time span, to convert alignment into second attention weights by uniformly assigning weights to the vector at the corresponding start and end times; obtain an additional loss function based on the first attention weight and the second attention weight; and obtain an attention loss function based on the cross-entropy loss and the additional loss function. The training module is used to update the parameters of the speech recognition model according to the connection time classification loss function and the attention loss function to obtain the trained speech recognition model.

7. An electronic device, characterized in that, include: At least one memory; At least one processor; At least one computer program; The computer program is stored in the memory, and the processor executes the at least one computer program to achieve: The method as described in any one of claims 1 to 4; or The method as described in claim 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform: The method as described in any one of claims 1 to 4; or The method as described in claim 5.

Images