Attention-based tracking of large intervals for end-to-end speech recognition

Abstract

An attention-based end-to-end (E2E) automatic speech recognition (ASR) training method includes: performing cross-entropy training on a model based on one or more input features of a speech signal; performing beam search using the model to generate a list of n best hypotheses for output hypotheses; and determining the best hypothesis from the generated list of n best hypotheses. The method further includes: determining character-based gradients and word-based gradients based on the model and a loss function that maximizes the distance between a reference sequence and the determined best hypothesis; and performing backpropagation on the model based on the determined character-based gradients and word-based gradients to update the model.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 16 / 276,081, filed on February 14, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to speech processing technology, and more particularly to attention-based large interval tracking for end-to-end speech recognition. Background Art

[0004] Minimum Bayesian risk (MBR) training, such as minimum word error rate (MWER) training, aims to minimize the expected risk on the output hypothesis. The risk or loss L can be expressed as follows:

[0005]

[0006] where %, .s) represents samples in the training set D, x represents input features, s represents their corresponding sequence labels, s' represents output hypotheses generated during training, and . represents model parameters. The difference between s and s' is denoted as / ts',s), which is the word or character level edit distance.

[0007] Accordingly, the output sequence during evaluation can be generated as follows:

[0008]

[0009] in, represents a candidate output sequence, and $un represents the output sequence selected by MBR decoding.

[0010] The search space of grows exponentially with its length, making MBR decoding mechanisms such as Recognition Output Voting Error Reduction (ROVER) inefficient. Although methods based on n-best lists or confusion networks can improve efficiency, beam search decoding based on maximum a posteriori (MAP) is still one of the most commonly used evaluation methods in practice. In MAP decoding, the output hypothesis with the highest (logarithmic) posterior is directly used for evaluation as follows:

[0011]

[0012] The mismatch between MBR training and MAP decoding indicates that there may be training schemes with better efficiency than MBR training and comparable performance to MBR training. Summary of the invention

[0013] According to an embodiment, a method for attention-based end-to-end (E2E) automatic speech recognition (ASR) training includes: performing cross-entropy training of a model based on one or more input features of a speech signal; performing beam search of the model on which the cross-entropy training is performed to generate an n-best hypothesis list of output hypotheses; and determining a best hypothesis in the generated n-best hypothesis list. The method also includes: determining a character-based gradient and a word-based gradient based on the model on which the cross-entropy training is performed and a loss function that maximizes the distance between a reference sequence and the determined best hypothesis; and performing back-propagation of the determined character-based gradient and the determined word-based gradient to the model to update the model.

[0014] According to an embodiment, a device for attention-based E2EASR training includes: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to the program code instructions. The program code includes: a first execution code configured to cause at least one processor to perform cross-entropy training of a model based on one or more input features of a speech signal; a second execution code configured to cause at least one processor to perform a beam search of a model in which cross-entropy training is performed to generate an n-best hypothesis list of output hypotheses; and a first determination code configured to cause at least one processor to determine an optimal hypothesis in the generated n-best hypothesis list. The program code also includes: a second determination code and a third determination code, wherein the second determination code and the third determination code are configured to cause at least one processor to determine a character-based gradient and a word-based gradient based on a model on which cross-entropy training is performed and a loss function that maximizes the distance between a reference sequence and a determined optimal hypothesis; and a third execution code configured to cause at least one processor to perform back propagation of the determined character-based gradient and the determined word-based gradient to the model to update the model.

[0015] According to an embodiment, a non-transitory computer-readable medium stores instructions that, when executed by at least one processor of a device, cause the at least one processor to: perform cross-entropy training of a model based on one or more input features of a speech signal; perform beam search of the model on which the cross-entropy training is performed to generate an n-best hypothesis list of output hypotheses; and determine a best hypothesis in the generated n-best hypothesis list. The instructions also cause the at least one processor to: determine a character-based gradient and a word-based gradient based on the model on which the cross-entropy training is performed and a loss function that maximizes the distance between a reference sequence and the determined best hypothesis; and perform back propagation of the determined character-based gradient and the determined word-based gradient to the model to update the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a diagram of an environment in which the methods, apparatuses, and systems described herein may be implemented, according to an embodiment.

[0017] Figure 2 yes Figure 1 A diagram of example components of one or more devices.

[0018] Figure 3 is a flowchart of a method for attention-based E2EASR training according to an embodiment.

[0019] Figure 4 is a diagram of an apparatus for attention-based E2EASR training according to an embodiment. DETAILED DESCRIPTION

[0020] Attention-based E2E ASR systems have a simplified training pipeline compared to conventional recognition systems, but their performance still needs to be improved. Many studies have devoted themselves to various training strategies to improve recognition accuracy.

[0021] Embodiments described herein include a sequence-by-sequence large-margin training scheme for an attention-based E2E ASR system. Instead of minimizing expected loss, this training scheme maximizes the margin. In other words, when the best hypothesis is not the reference, the model is trained to maximize the score difference between the best hypothesis and the reference, making the model more discriminative. Unlike the MBR-based MWER standard, this new approach focuses on only one hypothesis per utterance during training, but still achieves comparable performance. When both approaches employ character-level edit distance, the large-margin approach consistently outperforms MWER training. The large-margin training scheme is also a more concise formulation of the concept of large margins. It maintains the original model structure and is therefore easier to apply than current support vector machine (SVM)-based systems. The new model is tested on the benchmark SWB300h dataset. It achieves the same performance as the results of MWER training.

[0022] In detail, attention-based E2E ASR systems map input audio features into text sequences through three steps: encoding, attention, and decoding. The most commonly used decoder is a recurrent network trained with a point-wise cross-entropy loss. Recently, sequence-discriminative optimization criteria such as MBR-based MWER have been applied to improve model performance. MWER requires multiple assumptions during training. Because decoding is based on the commonly used MAP evaluation, large-margin-based training criteria may match MAP decoding better than MBR.

[0023] The large margin concept is often fused with SVM. By enlarging the margin between the reference sequence and the incorrect sequence, the upper bound of the generalization error can be minimized. In recent years, structured SVM (SSVM) has been combined with various deep neural networks (DNN) for speech recognition tasks. In these models, the softmax layer is replaced by the SSVM layer, and the training process consists of two stages. In the first stage, the weights of the SSVM layer are calculated using the cutting-plane algorithm for all training samples. The parameters in the DNN are then updated using the back-propagation algorithm. Regarding tasks such as phone text message dictation, the sequence-level implementation of the deep neural support vector machine (DNSVM) shows better performance than the corresponding sequence-discriminatively trained DNN. The method described in this paper can maintain the original deep neural network structure and is therefore easier to apply than DNSVM.

[0024] Definitions of abbreviations and terms appearing in the detailed description include the following:

[0025] Speech Recognition System: A computer program that can recognize and translate speech signals into written characters / words.

[0026] Encoder-Decoder: A model architecture where the encoder network maps raw input to feature representations and the decoder takes the feature representations as input and produces output.

[0027] Attention-based end-to-end (E2E) model: A model with an encoder-decoder architecture coupled with an attention scheme that enables the model to learn to focus on specific parts of the input sequence during decoding.

[0028] Large Margin: Large margin or maximum margin is a learning principle that trains a model to maximize the distance to boundary examples.

[0029] Support Vector Machine (SVM): A discriminative training method that exploits the large margin principle to learn the optimal hyperplane for classifying new examples.

[0030] Minimum Bayesian Risk (MBR): A training / decoding principle that aims to minimize the expected error in classification.

[0031] Figure 1 1 is a diagram of an environment 100 in which the methods, apparatuses, and systems described herein may be implemented, according to an embodiment. Figure 1 As shown, environment 100 may include user device 110, platform 120, and network 130. The devices of environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0032] The user device 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with the platform 120. For example, the user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or the like. In some implementations, the user device 110 may receive information from the platform 120 and / or send information to the platform 120.

[0033] The platform 120 includes one or more devices as described elsewhere herein. In some implementations, the platform 120 may include a cloud server or a cloud server group. In some implementations, the platform 120 may be designed to be modular so that software components can be swapped in or out depending on specific needs. In this way, the platform 120 can be easily and / or quickly reconfigured for different uses.

[0034] In some implementations, as shown, the platform 120 can be hosted in a cloud computing environment 122. It is worth noting that while the implementations described herein describe the platform 120 as being hosted in a cloud computing environment 122, in some implementations, the platform 120 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0035] Cloud computing environment 122 includes an environment hosting platform 120. Cloud computing environment 122 can provide computing, software, data access, storage, etc. services that do not require end users (e.g., user devices 110) to be aware of the physical location and configuration of the systems and / or devices hosting platform 120. As shown, cloud computing environment 122 can include a set of computing resources 124 (collectively, “computing resources 124” and individually, “computing resource 124”).

[0036] The computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, the computing resources 124 can host the platform 120. Cloud resources may include computing instances executed in the computing resources 124, storage devices provided in the computing resources 124, data transmission devices provided by the computing resources 124, etc. In some implementations, the computing resources 124 can communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0037] If further Figure 1As shown in FIG. 1 , the computing resources 124 include a set of cloud resources, such as one or more applications (“APP”) 124 - 1 , one or more virtual machines (“VM”) 124 - 2 , a virtualized storage device (“VS”) 124 - 3 , one or more hypervisors (“HYP”) 124 - 4 , and the like.

[0038] The applications 124-1 include one or more software applications that can be provided to or accessed by the user device 110 and / or the platform 120. The applications 124-1 can eliminate the need to install and execute software applications on the user device 110. For example, the applications 124-1 can include software associated with the platform 120 and / or any other software that can be provided via the cloud computing environment 122. In some implementations, one application 124-1 can send information to or receive information from one or more other applications 124-1 via the virtual machine 124-2.

[0039] The virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) such as a physical machine that executes a program. The virtual machine 124-2 can be a system virtual machine or a process virtual machine, depending on the use and correspondence of the virtual machine 124-2 to any real machine. The system virtual machine can provide a complete system platform that supports the execution of a complete operating system ("OS"). The process virtual machine can execute a single program and can support a single process. In some implementations, the virtual machine 124-2 can execute on behalf of a user (e.g., a user device 110) and can manage the infrastructure of the cloud computing environment 122, such as data management, synchronization, or long-duration data transfer.

[0040] The virtualized storage device 124-3 includes one or more storage systems and / or one or more devices using virtualization technology within the storage system or device of the computing resource 124. In some implementations, within the context of the storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the extraction (or separation) of logical storage from physical storage so that the storage system can be accessed without considering physical storage or heterogeneous structures. Separation may allow administrators of the storage system flexibility in terms of administrator management of storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and the location where the file is physically stored. This may enable optimization of storage usage, server consolidation, and / or performance of non-disruptive file migration.

[0041] Hypervisor 124-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute simultaneously on a host computer such as computing resource 124. Hypervisor 124-4 may present a virtual operating platform to the guest operating systems and may manage the execution of the guest operating systems. Multiple instances of various operating systems may share virtualized hardware resources.

[0042] The network 130 includes one or more wired and / or wireless networks. For example, the network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LEE) network, or a wireless network.

[0043] The present invention relates to a network that is a telecommunication network, a third generation (3G) network, a code division multiple access (CDMA) network, a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-optic-based network, etc., and / or a combination of these or other types of networks.

[0044] Figure 1 The number and arrangement of devices and networks shown are provided as examples. Figure 1 The devices and / or networks shown may contain additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks. Figure 1 Two or more of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one or more devices of one set of devices (eg, one or more devices) of environment 100 may perform one or more functions described as being performed by another set of devices of environment 100.

[0045] Figure 2 yes Figure 1 Device 200 may correspond to user device 110 and / or platform 120. Figure 2As shown, device 200 may include a bus 210 , a processor 220 , a memory 230 , a storage component 240 , an input component 250 , an output component 260 , and a communication interface 270 .

[0046] The bus 210 includes components that allow communication between components of the device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 220 includes one or more processors that can be programmed to perform functions. The memory 230 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 220.

[0047] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-volatile computer-readable medium and a corresponding drive.

[0048] Input components 250 include components that allow device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 260 include components that provide output information from device 200 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0049] The communication interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 270 can allow the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0050] Device 200 can perform one or more processes described herein. Device 200 can perform these processes in response to processor 220 executing software instructions stored by non-volatile computer-readable media (e.g., memory 230 and / or storage component 240). Computer-readable media is defined herein as non-volatile memory devices. Memory devices include memory space within a single physical storage device or memory space distributed on multiple physical storage devices.

[0051] The software instructions may be read into the memory 230 and / or storage component 240 from another computer-readable medium or from another device via the communication interface 270. When executed, the software instructions stored in the memory 230 and / or storage component 240 may cause the processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the implementations described herein are not limited to any particular combination of hardware circuitry and software.

[0052] Figure 2 The number and arrangement of components shown are provided as examples. Figure 2 Components shown, device 200 may include additional components, fewer components, different components, or differently arranged components. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0053] The embodiments described herein include a training method based on a sequence-by-sequence loss function. Like other sequence-based training methods, this training method can be applied after training the model with a point-by-point cross entropy loss. This training method is not limited to speech recognition, but can also be applied to other sequence-to-sequence tasks.

[0054] Figure 3 3 is a flow chart of a method 300 for attention-based E2E ASR training according to an embodiment. In some implementations, the platform 120 may perform Figure 3 In some implementations, the processing may be performed by another device or group of devices (eg, user device 110) separate from or including platform 120. Figure 3 One or more processing blocks of .

[0055] like Figure 3As shown, in operation 310, method 300 includes performing cross entropy training of a model based on one or more input features of a speech signal. Based on the cross entropy training being performed, method 300 includes performing large interval training of the model in which the cross entropy training is performed in operations 320 to 360.

[0056] Large margin training refers to a sequence-level training criterion that enlarges the distance between the reference sequence and the most competitive incorrect sequence. The following equation (1) shows the distance between the reference sequence s and the most competitive incorrect sequence The interval and the loss function LO defined on the entire training data set DD:

[0057]

[0058] Referring to the following equation (2), the threshold controls the desired distance between the reference sequence and the most competing incorrect sequence. To filter out samples that already satisfy the threshold constraint, a hinge function [·]+ is applied. The loss function is squared so that it can affect not only the sign but also the value of its gradient.

[0059] In operation 320 , the method includes performing a beam search of the model on which cross entropy training is performed to generate an n-best hypothesis list having, for example, the highest a posteriori output hypothesis or sequence.

[0060] In operation 330 , method 300 includes determining a best hypothesis in the generated list of n-best hypotheses.

[0061] Unlike Hidden Markov Model (HMM) based systems, attention-based E2EASR systems cannot generate a complete posterior graph due to the explicit dependencies between the output tokens. A common approximation of the posterior graph is a list of n-best hypotheses. In the list of n-best hypotheses, one best hypothesis is selected as the most competing incorrect hypothesis. Therefore, the loss function is as follows in equation (2), where, This represents a best hypothesis:

[0062]

[0063] In the attention-based E2E system, the log-posterior of the hypothesis can be interpreted as a score. Therefore, the above equation (2) can be written as the following equation (3), where score() is unnormalized by the sequence length:

[0064]

[0065] In operation 340 , the method 300 includes determining a character-based gradient based on the model on which cross entropy training is performed and a loss function that maximizes a distance between a reference sequence and the determined one best hypothesis.

[0066] In operation 350 , the method includes determining a word-based gradient based on a model in which cross entropy training is performed and a loss function that maximizes a distance between a reference sequence and the determined one best hypothesis.

[0067] The threshold / (s,s) can be chosen to be the word and / or character level edit distance. Each of the following equations (4) gives the gradient at the top layer, where δ(·) is the Kronecker delta (indicator) function, and By γ + γ + express:

[0068]

[0069]

[0070] exist Repeated training on the correct beginning segment may make the training process unstable and prone to overfitting. To avoid training instability and overfitting, The first misclassified word in starts to apply the gradient in equation (4) above, which is shown in equation (5) as follows:

[0071]

[0072] By manually assigning gradients with respect to the log-posteriori, gradients of other parameters can be automatically derived by deep learning platforms such as PyTorch and Chainer.

[0073] In operation 360, the method includes performing back propagation of the determined character-based gradients and the determined word-based gradients to the model to update the model. The method 300 returns to operation 320 to continue large-margin training.

[0074] Method 300 can be implemented with different loss functions. In the above embodiments, character-based and word-based loss functions are used. When applied to other sequence learning tasks such as translation, the loss function can be based on bilingual evaluation underscore (BLEU) scores, etc. Even if the goal of method 300 is to maximize the interval, multiple boundary examples can be applied. In addition to those described in the embodiments, there are various ways to select a set of hypotheses where the hypothesis with the highest posteriori is beam searched. Finally, the gradient calculations on the hypotheses may also be different. Gradients in all positions or only gradients at special positions can be used.

[0075] although Figure 3 An example block diagram of method 300 is shown, but in some implementations, Figure 3 Compared to the blocks depicted in the method 300, the method 300 may include additional blocks, fewer blocks, different blocks, or differently arranged blocks. Additionally or alternatively, two or more of the blocks in the method 300 may be executed in parallel.

[0076] The embodiments provide similar performance to that of the MWER method on the same benchmark dataset. The embodiments combine the large margin principle with deep neural network learning, which is easier to apply than the current SVM deep learning combination method.

[0077] Models according to embodiments are compared with the benchmark SWB300h dataset. The settings are the same as those in MWER, where the input is 40 dim log melfbank features and the output is 49 characters. The E2E framework is fed with label attachment scores (LAS) as input to 6 bidirectional long short-term memories (BiLSTM) as encoders and 2 LSTMs as decoders. The baseline cross entropy model is the same as that of MWER, which is trained with the cross entropy criterion of the predetermined sampling authorization. The Adam optimizer is used for training, and the initial learning rate is 7.5*10 -7 The dropout rate is chosen to be 0.2 and the mini-batch size is 8. For large-margin training, results using multiple hypotheses are reported in addition to large-margin training based on one best hypothesis. The many-hypothesis and multi-hypothesis large-margin criteria in MWER are identical. Results without using an external language model are denoted “w / o LM” and results using a language model are denoted “w / LM”.

[0078] Compared to the baseline WER of 13.3%, the results of large-interval training based on one best hypothesis achieved a relative improvement of 6.8%, and the results of large-interval training based on four best hypotheses achieved a relative improvement of 8.3%, as shown in Table 1:

[0079] Table 1. Results and comparisons of long-interval training

[0080]

[0081] Table 3 shows a comparison with other published results:

[0082] Table 3. Comparison with previously proposed attention-based end-to-end systems

[0083]

[0084] Figure 4 4 is a diagram of an apparatus 400 for attention-based E2E ASR training according to an embodiment. Figure 4 As shown, the apparatus 400 includes a first execution code 410 , a second execution code 420 , a first determination code 430 , a second determination code 440 , a third determination code 450 , and a third execution code 460 .

[0085] The first execution code 410 is configured to perform cross entropy training of a model based on one or more input features of a speech signal.

[0086] The second execution code 420 is configured to perform a beam search of the model on which cross entropy training is performed to generate an n-best hypothesis list of output hypotheses.

[0087] The first determination code 430 is configured to determine a best hypothesis in the generated n-best hypothesis list.

[0088] The second determination code 440 and the third determination code 450 are respectively configured to determine a character-based gradient and a word-based gradient based on a model on which cross entropy training is performed and a loss function that maximizes a distance between a reference sequence and the determined one best hypothesis.

[0089] The third execution code is configured to perform back-propagation of the determined character-based gradients and the determined word-based gradients to the model to update the model.

[0090] The output hypothesis included in the generated n-best hypothesis list may have the highest posterior among the output sequences of the model on which the cross entropy training is performed.

[0091] The second execution code 420 may also be configured to re-perform beam search of the model being executed with cross entropy training based on back propagation of the determined character-based gradients and the determined word-based gradients to the model being executed to regenerate the n-best hypothesis list.

[0092] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed.Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0093] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0094] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software codes, and it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0095] Even though particular combinations of features are defined in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically defined in the claims and / or disclosed in the specification. Although each of the attached dependent claims may directly refer to only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim group.

[0096] Unless explicitly described herein, the elements, actions or instructions used herein should not be interpreted as critical or necessary. Moreover, as used herein, the articles "a kind of" and "an" are intended to include one or more items, and can be used interchangeably with "one or more". In addition, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more". In the case of meaning only one item, the term "one" or similar language is used. Moreover, as used herein, the terms "have", "have", "contain" etc. are intended to be open terms. In addition, unless otherwise expressly stated, the phrase "based on" is intended to mean "based at least in part on".

Claims

1. A method for attention-based end-to-end (E2E) automatic speech recognition (ASR) training, characterized in that: The method comprises: performing cross entropy training on the model based on one or more input features of the speech signal; performing a beam search using the model to generate an n-best hypothesis list of output hypotheses; Determine the best hypothesis in the generated n-best hypothesis list, wherein in the n-best hypothesis list, select a most competitive incorrect hypothesis as the best hypothesis; Determining a character-based gradient and a word-based gradient based on the model and a loss function that maximizes a distance between a reference sequence and the determined best hypothesis; and performing back-propagation on the model based on the determined character-based gradient and the determined word-based gradient to update the model; The loss function is expressed as follows: Where (χ, s) represents the samples in the training set D, χ represents the input features, and s represents the reference samples. represents the best hypothesis, represents the distance between the reference sequence and the best hypothesis, and θ represents the model parameters.

2. The method according to claim 1, characterized in that The output hypothesis in the n-best hypothesis list has the highest posterior in the output sequence of the model.

3. The method according to claim 1, characterized in that The loss function can also be expressed as follows: Where (χ, s) represents the sample in the training set D, X represents the input feature, and s represents the reference sample. represents the best hypothesis, represents the distance between the reference sequence and the best hypothesis, θ represents the model parameters, and score represents the log posterior.

4. The method according to claim 3, characterized in that Each of the character-based gradient and the word-based gradient is expressed as follows: where δ(·) represents the Kronecker delta function and y+ represents 5. The method according to claim 3, characterized in that: Each of the character-based gradient and the word-based gradient is expressed as follows: Where δ(·) represents the Kronecker delta function, y+ represents And ω represents the first wrong participle.

6. The method according to claim 1, characterized in that The method further includes: performing the beam search again using the model subjected to the back-propagation to regenerate the n-best hypothesis list.

7. An apparatus for attention-based end-to-end (E2E) automatic speech recognition (ASR) training, characterized in that The device comprises: at least one memory configured to store program code; and At least one processor, the at least one processor is configured to read the program code and operate according to the instructions of the program code, the program code comprising: a first execution code configured to cause the at least one processor to perform cross entropy training on a model based on one or more input features of a speech signal; second execution code configured to cause the at least one processor to perform a beam search using the model to generate an n-best hypothesis list of output hypotheses; a first determination code configured to cause the at least one processor to determine a best hypothesis in the generated n-best hypothesis list, wherein in the n-best hypothesis list, a most competitive incorrect hypothesis is selected as the best hypothesis; second determining code and third determining code configured to cause the at least one processor to determine a character-based gradient and a word-based gradient based on the model and a loss function that maximizes a distance between a reference sequence and the determined best hypothesis; and a third execution code configured to cause the at least one processor to perform back propagation on the model based on the determined character-based gradient and the determined word-based gradient to update the model; The loss function is expressed as follows: Where (χ, s) represents the samples in the training set D, X represents the input features, and s represents the reference samples. represents the best hypothesis, represents the distance between the reference sequence and the best hypothesis, and θ represents the model parameters.

8. The device according to claim 7, characterized in that The output hypothesis in the n-best hypothesis list has the highest posterior in the output sequence of the model.

9. The device according to claim 7, characterized in that The loss function can also be expressed as follows: Where (χ, s) represents the sample in the training set D, χ represents the input feature, and s represents the reference sample. represents the best hypothesis, represents the distance between the reference sequence and the best hypothesis, θ represents the model parameters, and score represents the log posterior.

10. The device according to claim 9, characterized in that Each of the character-based gradient and the word-based gradient is expressed as follows: where δ(·) represents the Kronecker delta function and γ+ represents 11. The device according to claim 9, characterized in that Each of the character-based gradient and the word-based gradient is expressed as follows: Among them, δ(·) represents the Kronecker delta function, γ+ represents And ω represents the first wrong participle.

12. The device according to claim 7, characterized in that The beam search is performed again using the model that has been back-propagated to regenerate the n-best hypothesis list.

13. A non-transitory computer readable medium, characterized in that The computer readable medium stores instructions that, when executed by at least one processor of a device, cause the at least one processor to: performing cross entropy training on the model based on one or more input features of the speech signal; performing a beam search using the model to generate an n-best hypothesis list of output hypotheses; Determine the best hypothesis in the generated n-best hypothesis list, wherein in the n-best hypothesis list, select a most competitive incorrect hypothesis as the best hypothesis; determining a character-based gradient and a word-based gradient based on the model and a loss function that maximizes a distance between a reference sequence and the determined best hypothesis; as well as performing back-propagation on the model based on the determined character-based gradient and the determined word-based gradient to update the model; The loss function is expressed as follows: Where (χ, s) represents the samples in the training set D, χ represents the input features, and s represents the reference samples. represents the best hypothesis, represents the distance between the reference sequence and the best hypothesis, and θ represents the model parameters.

14. The non-transitory computer readable medium of claim 13, wherein: The output hypothesis in the n-best hypothesis list has the highest posterior in the output sequence of the model.

15. The non-transitory computer readable medium of claim 13, wherein: The loss function can also be expressed as follows: Where (χ, s) represents the sample in the training set D, χ represents the input feature, and s represents the reference sample. represents the best hypothesis, represents the distance between the reference sequence and the best hypothesis, θ represents the model parameters, and score represents the log posterior.

16. The non-transitory computer readable medium of claim 15, wherein: Each of the character-based gradient and the word-based gradient is expressed as follows: where δ(·) represents the Kronecker delta function and γ+ represents 17. The non-transitory computer readable medium of claim 15, wherein: Each of the character-based gradient and the word-based gradient is expressed as follows: Among them, δ(·) represents the Kronecker delta function, γ+ represents And ω represents the first wrong participle.

Images