Method and device for end-to-end automatic speech recognition training based on attention

Abstract

A method for attention-based end-to-end (A‑E2E) automatic speech recognition (ASR) training, comprising: performing cross-entropy training on a model based on one or more input features of a speech signal; determining a posterior probability vector at the time of a first erroneous token in one or more output tokens of the model performing the cross-entropy training; and determining the loss of the first erroneous token at the time based on the determined posterior probability vector. The method also includes: determining a total loss of a training set of the model performing the cross-entropy training based on the determined loss of the first erroneous token; and updating the model performing the cross-entropy training based on the determined total loss of the training set.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 16 / 275,971, filed on February 14, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present application relates to the technical field of speech recognition, and in particular to a method and device for attention-based end-to-end automatic speech recognition training. Background Art

[0004] Attention-based end-to-end (A-E2E) automatic speech recognition (ASR) systems differ from conventional hidden Markov model (HMM) systems in that speech signals are directly mapped to tokens (characters / subwords / words). This enables A-E2E ASR systems to use training criteria that are close to evaluation metrics such as character error rate (CER) and word error rate (WER).

[0005] The policy gradient of the A-E2E ASR model can be used to back-propagate the CER and / or WER losses to the model parameters. First, the expected CER and / or WER can be calculated by a sampling-based method. Then, the policy gradient can be applied to directly optimize the expected CER and / or WER. Inspired by the minimum Bayes risk (MBR) training of conventional HMM-based models, the minimum word error rate (MWER) training can be used for the A-E2E ASR system. The loss of MWER training is the expected number of word errors. Because it is inefficient to calculate such losses accurately, two approximate methods can be used to calculate. The first method is based on sampling, and the second method uses n-best hypotheses generated during training. Experimental results show that the loss estimation based on n-best hypotheses is better than the loss estimation based on sampling. MWER training can be improved by using softmax smoothing during n-best hypothesis generation. Competitive results can be obtained using the Switchboard (SWBD) 300h corpus.

[0006] In A-E2E ASR systems, correction of the first few errors can be very important due to the dependencies of the output tokens. Fine-grained partial error can be seen as an attempt in this direction. Using the length-normalized edit distance, more weight is assigned to the tokens at the beginning of the output sequence in the loss. Experimental results show that using fine-grained partial error can be important for performance improvement. One extreme of the weighted approach is to completely mask the loss after the first erroneous token. Such training schemes are widely used in machine translation, spoken dialogue processing, and annotation and parsing problems in natural language processing (NLP) under the name of early update. When applied to A-E2E ASR models, standard early update methods can assign non-zero gradients to the correct tokens at the beginning.

[0007] Therefore, how to train the A-E2E ASR model to minimize the errors in the output tokens (characters / subwords / words) is still a problem that needs to be further studied and solved in this field. Summary of the invention

[0008] According to an embodiment, a method for A-E2E ASR training is provided, the method comprising: performing cross entropy training on a model based on one or more input features of a speech signal; determining a posterior probability vector at the time of a first erroneous token in one or more output tokens of the model performing cross entropy training; and determining the loss of the first erroneous token at the time based on the determined posterior probability vector. The method also includes: determining the total loss of a training set of the model performing cross entropy training based on the determined loss of the first erroneous token; and updating the model performing cross entropy training based on the determined total loss of the training set.

[0009] According to an embodiment, a device for A-E2E ASR training is provided, the device comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to the instructions of the program code. The program code comprises: an execution code configured to cause the at least one processor to perform cross entropy training on a model based on one or more input features of a speech signal; a first determination code configured to cause the at least one processor to determine a posterior probability vector at the time of the first erroneous token in one or more output tokens of the model performing cross entropy training; and a second determination code configured to cause the at least one processor to determine the loss of the first erroneous token at the time based on the determined posterior probability vector. The program code also comprises: a third determination code configured to cause the at least one processor to determine the total loss of the training set of the model performing cross entropy training based on the determined loss of the first erroneous token; and an update code configured to cause the at least one processor to update the model performing cross entropy training based on the determined total loss of the training set.

[0010] According to an embodiment, a non-transitory computer-readable medium storing instructions is provided, and when at least one processor of a device executes the instructions, the instructions cause the at least one processor to perform the following operations: perform cross-entropy training on a model based on one or more input features of a speech signal; determine a posterior probability vector at the time of the first erroneous token in one or more output tokens of the model performing cross-entropy training; and determine the loss of the first erroneous token at the time based on the determined posterior probability vector. The instructions also cause the at least one processor to determine the total loss of the training set of the model performing cross-entropy training based on the determined loss of the first erroneous token, and update the model performing cross-entropy training based on the determined total loss of the training set.

[0011] According to an embodiment, a device for A-E2E ASR training is provided, the device comprising: an execution module configured to perform cross-entropy training on a model based on one or more input features of a speech signal; a first determination module configured to determine a posterior probability vector at the time of a first erroneous token in one or more output tokens of the model performing the cross-entropy training; a second determination module configured to determine the loss of the first erroneous token at the time based on the determined posterior probability vector; a third determination module configured to determine the total loss of a training set of the model performing the cross-entropy training based on the determined loss of the first erroneous token; and an update module configured to update the model performing the cross-entropy training based on the determined total loss of the training set.

[0012] According to an embodiment, a computer device is provided, the device comprising a processor and a memory. The memory is used to store program code and transmit the program code to the processor; the processor is used to execute the above-mentioned attention-based end-to-end automatic speech recognition training method according to the instructions in the program code.

[0013] According to the above technical solution provided in the embodiment of the present application, first, cross-entropy training is performed on the model based on one or more input features of the speech signal, and then the posterior probability vector at the time of the first erroneous token in one or more output tokens of the model performing cross-entropy training is determined. And, the loss of the first erroneous token at the time is determined based on the determined posterior probability vector, the total loss of the training set of the model performing cross-entropy training is determined based on the loss of the determined first erroneous token, and the model performing cross-entropy training is updated based on the determined total loss of the training set. In view of this, through the token-wise training (TWT) method in the present application that only acts on the first erroneous token in the hypothesis, once the first erroneous token is corrected, subsequent errors can also be corrected without additional model parameter adjustments. Compared with sequence-based MWER training, the TWT scheme is more conservative for training and therefore converges more stably. The TWT scheme is also very flexible and can be combined with various loss functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a diagram of an environment in which the methods, devices, and systems described herein may be implemented, according to an embodiment.

[0015] Figure 2 yes Figure 1 A diagram of example components of one or more devices.

[0016] Figure 3 is a flow chart of a method of A-E2E ASR training according to an embodiment.

[0017] Figure 4 is a graph of the first incorrect token in the hypothesis with the longest correct prefix, according to an embodiment.

[0018] Figure 5 is a diagram of an apparatus for A-E2E ASR training, according to an embodiment.

[0019] Figure 6 is a diagram of the difference between a regular sequence based training scheme and an early update training scheme. DETAILED DESCRIPTION

[0020] In an A-E2E ASR system, since the output tokens have dependencies, it is very important to correct the errors in the first few output tokens. To address this problem, there are currently two main approaches: a training scheme based on a rule sequence and an early update training scheme. Figure 6 A diagram showing the difference between a regular sequence based training scheme and an early update training scheme. In both cases (a) and (b), the gradient is calculated in the same way, but is applied to different positions of the hypothesis-reference (Hyp-Ref) pair.

[0021] Based on this, the present application proposes a method and device for attention-based end-to-end automatic speech recognition training, which trains an attention-based end-to-end speech recognition model to directly minimize errors in output tokens (characters / subwords / words).

[0022] Assuming that each output token depends on the previous output, decoding errors can easily propagate along the output sequence. The implementation described in this article includes a token-wise training (TWT) scheme that only acts on the first erroneous token in the hypothesis. Once the first erroneous token is corrected, subsequent errors can also be corrected without additional model parameter adjustments. Compared with sequence-based MWER training, the TWT scheme is more conservative in training and therefore converges more stably. The TWT scheme is also flexible and can be combined with various loss functions.

[0023] To extend TWT for multiple hypotheses, embodiments described herein also include a TWT in beam (TWTiB) training scheme. The hypothesis with the longest correct prefix is ​​selected, rather than the hypothesis with the largest a posteriori probability as in most training algorithms. TWTiB outperforms the MWER criterion on the benchmark SWBD 300h dataset.

[0024] The above schemes provide new learning principles for sequence-to-sequence learning and produce state-of-the-art results on benchmark speech recognition datasets.

[0025] Definitions of abbreviations and terms appearing in the detailed description include the following:

[0026] Speech Recognition System: A computer program that can recognize speech signals and translate them into written characters / words.

[0027] Encoder-decoder: A model architecture in which the encoder network maps the raw input to feature representations and the decoder takes the feature representations as input and produces output.

[0028] Attention-based end-to-end (A-E2E) models: models with an encoder-decoder architecture coupled with an attention scheme that enables focusing on learning specific parts of the input sequence during decoding.

[0029] Point-wise Loss: A loss function calculated based on two tokens, regardless of whether the two tokens are the same.

[0030] Sequence-wise Loss: A loss function calculated based on the similarity between two sequences. Usually the best alignment of the two sequences is given and then the difference is added.

[0031] Figure 1 1 is a diagram of an environment 100 in which the methods, devices, and systems described herein may be implemented, according to an embodiment. Figure 1 As shown, environment 100 may include user device 110, platform 120, and network 130. Devices in environment 100 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0032] The user device 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with the platform 120. For example, the user device 110 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch), or a similar device. In some implementations, the user device 110 may receive information from the platform 120 and / or send information to the platform 120.

[0033] The platform 120 includes one or more devices as described elsewhere herein. In some implementations, the platform 120 may include a cloud server or a cloud server group. In some implementations, the platform 120 may be designed to be modular so that software components can be swapped in or out depending on specific needs. In this way, the platform 120 can be easily and / or quickly reconfigured for different uses.

[0034] In some implementations, as shown, the platform 120 can be hosted in a cloud computing environment 122. It is worth noting that while the implementations described herein describe the platform 120 as being hosted in a cloud computing environment 122, in some implementations, the platform 120 is not cloud-based (i.e., can be implemented outside of a cloud computing environment) or can be partially cloud-based.

[0035] Cloud computing environment 122 includes an environment that hosts platform 120. Cloud computing environment 122 can provide computing, software, data access, storage, and other services that do not require end users (e.g., user devices 110) to be aware of the physical location and configuration of the systems and / or devices that host platform 120. As shown, cloud computing environment 122 can include a set of computing resources 124 (collectively, “computing resources 124” and individually, “computing resource 124”).

[0036] The computing resources 124 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, the computing resources 124 can host the platform 120. Cloud resources can include computing instances executed in the computing resources 124, storage devices provided in the computing resources 124, data transmission devices provided by the computing resources 124, etc. In some implementations, the computing resources 124 can communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0037] Further Figure 1 As shown, the computing resources 124 include a group of cloud resources, such as one or more applications ("APP") 124-1, one or more virtual machines ("VM") 124-2, virtualized storage ("VS") 124-3, one or more hypervisors ("HYP") 124-4, etc.

[0038] The applications 124-1 include one or more software applications that can be provided to or accessed by the user device 110 and / or the platform 120. The applications 124-1 can eliminate the need to install and execute software applications on the user device 110. For example, the applications 124-1 can include software associated with the platform 120 and / or any other software that can be provided via the cloud computing environment 122. In some implementations, one application 124-1 can send information to or receive information from one or more other applications 124-1 via the virtual machine 124-2.

[0039] Virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 124-2 can be a system virtual machine or a process virtual machine, depending on the use and correspondence of virtual machine 124-2 to any real machine. System virtual machines can provide a complete system platform that supports the execution of a complete operating system ("OS"). Process virtual machines can execute a single program and can support a single process. In some implementations, virtual machine 124-2 can execute on behalf of a user (e.g., user device 110) and can manage the infrastructure of cloud computing environment 122, such as data management, synchronization, or long-duration data transfer.

[0040] Virtualized storage 124-3 includes one or more storage systems and / or one or more devices using virtualization technology within the storage system or device of computing resource 124. In some implementations, within the context of a storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to abstracting (or separating) logical storage from physical storage so that the storage system may be accessed without regard to physical storage or heterogeneous structures. Separation may allow administrators of storage systems to have flexibility in terms of administrator management of storage for end users. File virtualization may eliminate the dependency between data accessed at the file level and the location of the physical storage file. This may enable optimization of storage usage, server consolidation, and / or performance of non-disruptive file migration.

[0041] Hypervisor 124-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute simultaneously on a host computer such as computing resource 124. Hypervisor 124-4 may present a virtual operating platform to the guest operating systems and may manage the execution of the guest operating systems. Multiple instances of various operating systems may share virtualized hardware resources.

[0042] The network 130 includes one or more wired and / or wireless networks. For example, the network 130 may include a cellular network (e.g., a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0043] supply Figure 1 The number and arrangement of devices and networks shown are examples. In practice, except Figure 1 In addition to the devices and / or networks shown, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or differently arranged devices and / or networks. Figure 1 Two or more of the devices shown may be implemented in a single device, or Figure 1 The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, a collection of devices (eg, one or more devices) of environment 100 may perform one or more functions described as being performed by another collection of devices of environment 100.

[0044] Figure 2 yes Figure 1 Device 200 may correspond to user device 110 and / or platform 120. Figure 2 As shown, the apparatus 200 may include a bus 210 , a processor 220 , a memory 230 , a storage component 240 , an input component 250 , an output component 260 , and a communication interface 270 .

[0045] The bus 210 includes components that allow communication between components of the computing device 200. The processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 220 includes one or more processors that can be programmed to perform functions. The memory 230 includes a random access memory (RAM), a read only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 220.

[0046] The storage component 240 stores information and / or software related to the operation and use of the device 200. For example, the storage component 240 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-transitory computer-readable medium and a corresponding drive.

[0047] Input components 250 include components that allow device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 250 may include sensors for sensing information (e.g., global positioning system (GPS) components, accelerometers, gyroscopes, and / or actuators). Output components 260 include components that provide output information from device 200 (e.g., a display, a speaker, and / or one or more light-emitting diodes (LEDs)).

[0048] The communication interface 270 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 270 can allow the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0049] Device 200 may perform one or more processes described herein. Device 200 may perform these processes in response to processor 220 executing software instructions stored by non-transitory computer-readable media such as memory 230 and / or storage component 240. Computer-readable media is defined herein as non-transitory memory devices. Memory devices include storage space within a single physical storage device or storage space distributed across multiple physical storage devices.

[0050] The software instructions may be read into the memory 230 and / or storage component 240 from another computer-readable medium or from another device via the communication interface 270. The software instructions stored in the memory 230 and / or storage component 240, when executed, may cause the processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0051] Figure 2 The number and arrangement of components shown are provided as examples. Figure 2 Compared to the components shown, the device 200 may include additional components, fewer components, different components, or components arranged differently. Additionally or alternatively, a collection of components of the device 200 (e.g., one or more components) can perform one or more functions described as being performed by another collection of components of the device 200.

[0052] Embodiments described herein include training schemes for sequence models in which the output at time t has a significant impact on the output at time t+1. Rather than applying gradients to the entire sequence as in conventional training schemes, the new training scheme targets the first incorrectly decoded position of the sequence. Additionally, rather than selecting one or more hypotheses with the highest a posteriori probability, the new training scheme selects the hypothesis with the longest correctly decoded prefix. Like other sequence-based training schemes, this new training scheme is applied after training the model with a pointwise cross entropy loss. This training scheme is not limited to speech recognition and can also be applied to other sequence-to-sequence tasks.

[0053] Figure 3 is a flow chart of an example method 300 of A-E2E ASR training according to an embodiment. In some implementations, Figure 3 One or more processing blocks of may be executed by platform 120. In some implementations, Figure 3 One or more processing blocks of may be performed by another device or group of devices (eg, user device 110 ) that is separate from platform 120 or includes platform 120 .

[0054] like Figure 3 As shown, in operation 310, method 300 includes performing cross entropy training on a model based on one or more input features of a speech signal. Based on the cross entropy training being performed, method 300 includes performing token-wise training on the model performing cross entropy training in operations 320 to 350.

[0055] In operation 320 , method 300 includes determining a posterior probability vector at the time of a first erroneous token in one or more output tokens of a model performing cross entropy training.

[0056] During evaluation, the inference process of the A-E2E ASR system can be expressed as the following equations (1)-(3):

[0057] H enc =Encoder(X) (1)

[0058] p t =Decoder(y t-1 , enc ) (2)

[0059]

[0060] Among them, matrix X represents the input audio features, H enc represents the encoded feature, y t represents the output token at time t, pt represents the posterior probability vector at time t, and p t，i represents the probability of the token over dimension i at time t. Note that the decoder state and attention mechanism are omitted in these equations only for the sake of clarity.

[0061] During the training process without scheduled sampling, the reference token r at time t-1 is used in the golden transcription. t-1 to replace the output token y at time t-1 in equation (2) t-1 , as shown in the following formula (4):

[0062] p t =Decoder(r t-1 , H enc ) (4)

[0063] To adjust the model towards the evaluation scenario, we use a mixture of output tokens y t-1 With reference token r t-1 The sampling is scheduled in the following way, as shown in the following formula (5):

[0064] p t =Decoder(s t-1 ∈{r t-1 ,y t-1}, H enc ) (5),

[0065] Among them, s t-1 is from {r t-1 ,y t-1} randomly selected tokens. t For t-1 , an error at time t-1 can easily lead to the subsequent output token y t ,y t+1 , .... Assume that at time t ω The first error token appears at Then, in the extreme case, the remaining output tokens y may not overlap with the remaining reference tokens r. In this case, when the first error is corrected, less optimization measures can be used to avoid subsequent errors.

[0066] In operation 330, method 300 includes determining a loss for a first erroneous token at the time based on the determined posterior probability vector.

[0067] In operation 340 , method 300 includes determining a total loss for the training set of the model performing cross entropy training based on the determined loss for the first erroneous token.

[0068] In detail, in the following equation (6), l θ (y t , r t ) represents the loss of the output token y at time t, and L(θ) represents the total loss of the training set. The model parameters are denoted as θ. The loss function can be expressed as the following equation (6):

[0069]

[0070] where (Y, R) represents the hypothesis-reference pairs in the entire training set, and T is usually chosen to be min(M, N). This loss function treats all positions in the output sequence equally, which makes it difficult to model output token dependencies in A-E2E ASR models. In addition, repeated training on correctly recognized tokens may also lead to oscillations. To ensure zero gradient on correctly decoded tokens, TWT is used, as shown in Equation (9):

[0071]

[0072] According to equation (9), TWT can be combined with various l θ (y t , r t ) value. In this specification, two l are used θ (y t , r t ) value, Ref and Ref+Err. Ref is the first error token At time t ω The cross entropy (CE) loss at is shown in the following formula (10):

[0073]

[0074] in, As shown in equation (5), Ref+Err is different from Ref in that it also reduces the first error token The probability of the wrong dimension is as shown in the following formula (11):

[0075]

[0076] In most cases, it is assumed that (including the first error token ) is selected as the hypothesis with the highest posterior probability (i.e., the best hypothesis). When TWT is applied to multiple hypotheses, the embodiments described herein include a TWTiB method by selecting the hypothesis with the longest correct prefix. Let y jis the output token of hypothesis j in the bundle, and its t ω t j，ω . With maximum t jl ，ω The hypothetical output token is y jl The TWTiB loss can be expressed as shown in the following equation (12):

[0077]

[0078] Figure 4 is a graph of the first incorrect token in the hypothesis with the longest correct prefix according to an embodiment. Figure 4 As shown, the first incorrect token is the assumed word “wreck” in “It is not easy to wreck a nice beach”.

[0079] Refer again Figure 3 , in operation 350 , method 300 includes updating the model performing cross entropy training based on the determined total loss of the training set.

[0080] The implementation includes two points. The first point is a gradient update strategy in which only the first position where an error occurs is selected to apply the gradient. This can be applied to another sequence translation or generation task with strong dependencies between outputs. The second point is the selection of hypotheses for the loss function. Usually the hypothesis with the highest posterior probability is selected, but it can be more advantageous in the new training scheme to select the hypothesis with the longest correct prefix. Finally, even though only two types of likelihood-based loss functions are described in the implementation, other loss functions can also be combined with the training scheme described in this article.

[0081] although Figure 3 Example blocks of method 300 are shown, but in some implementations, Figure 3 Compared to the blocks depicted in , method 300 may include additional blocks, fewer blocks, different blocks, or differently arranged blocks. Additionally or alternatively, two or more of the blocks of method 300 may be executed in parallel.

[0082] On the same benchmark dataset, the implemented methods provide better performance than the state-of-the-art MWER methods. They provide a new training principle for sequence generation related tasks by focusing on the first-occurrence errors. When training an A-E2E ASR system with scheduling samples, the output of the A-E2E ASR system at time t is greatly influenced by the previous outputs and thus benefits from this early update principle.

[0083] The model according to the embodiment is compared with the benchmark SWB300h dataset. The setting is the same as in MWER, with 40-dim log mel fbank features as input and 49 characters as output. The E2E framework is input fed label attachment score (LAS), with 6 bidirectional-Long Short Term Memory (BiLSTM) as encoder and 2 LSTM as decoder. The baseline cross entropy model is the same as that of MWER, which is trained with the scheduled sampling weighted cross entropy criterion. Adam optimizer is used for training and the initial learning rate is 7.5*10 -7 A dropout rate of 0.2 and a mini-batch size of 8 were chosen. For large margin training, in addition to large margin training on the best hypothesis, results using multiple hypotheses are also reported. The number of hypotheses in MWER and the multi-hypothesis large margin criterion are the same. Results without using an external language model are denoted “w / o LM”, while results with a language model are denoted “w / LM”.

[0084] The results and comparison of TWT are shown in Table 1. Note that TWT uses only a single assumption, while MWER and TWTiB use multiple assumptions and the number of assumptions is 4.

[0085] Table 1. Results and comparison of TWT

[0086]

[0087] A comparison of the representative TWT standard TWTiB with previously proposed systems is shown in Table 3 below.

[0088] Table 3. Comparison with other A-E2E speech recognition systems

[0089]

[0090] Figure 5 is a diagram of an apparatus 500 for A-E2E ASR training according to an embodiment. Figure 5 As shown, the device 500 includes an execution code 510 , a first determination code 520 , a second determination code 530 , a third determination code 540 , and an update code 550 .

[0091] The execution code 510 is configured to perform cross entropy training on the model based on one or more input features of the speech signal.

[0092] The first determination code 520 is configured to determine a posterior probability vector at the time of a first erroneous token in one or more output tokens of a model performing cross entropy training.

[0093] The second determination code 530 is configured to determine the loss of the first erroneous token at the time based on the determined posterior probability vector.

[0094] The third determining code 540 is configured to determine a total loss of a training set of a model performing cross entropy training based on the determined loss of the first erroneous token.

[0095] The update code 550 is configured to update the model performing cross entropy training based on the determined total loss of the training set.

[0096] The device 500 may also include a selection code configured to select a hypothesis with the longest correct prefix from a plurality of hypotheses of the model performing cross entropy training. The first determination code 520 may also be configured to determine a posterior probability vector at the time of the first erroneous token included in the selected hypothesis.

[0097] The disclosed embodiment also provides a device for A-E2E ASR training. The device includes: an execution module, which is configured to perform cross-entropy training on a model based on one or more input features of a speech signal; a first determination module, which is configured to determine a posterior probability vector at the time of the first erroneous token in one or more output tokens of the model performing the cross-entropy training; a second determination module, which is configured to determine the loss of the first erroneous token at the time based on the determined posterior probability vector; a third determination module, which is configured to determine the total loss of the training set of the model performing the cross-entropy training based on the determined loss of the first erroneous token; and an update module, which is configured to update the model performing the cross-entropy training based on the determined total loss of the training set.

[0098] In some examples, the posterior probability vector at the time is determined as follows:

[0099] p t =Decoder(s t-1 ∈{r t-1 ,y t-1}, H enc ),

[0100] Wherein, t represents the time, p t represents the posterior probability vector at time t, H enc represents the one or more features to be encoded, y t-1 represents the output token at the previous time t-1, r t-1represents the reference token at the previous time t-1, and s t-1 Indicates that from {r t-1 ,y t-1} randomly selected tokens.

[0101] In some examples, the total loss for the training set is determined as follows:

[0102]

[0103] Where L(θ) represents the total loss of the training set, (Y, R) represents the hypothesis-reference pair in the training set, and t ω represents the time, represents the first error token at the time, represents a reference token at said time, and Represents the loss of the first erroneous token.

[0104] In some examples, the loss of the first erroneous token is determined as follows:

[0105]

[0106] in, represents the posterior probability of the reference token at said time.

[0107] In some examples, the loss of the first erroneous token is determined as follows:

[0108]

[0109] in, represents the posterior probability of the reference token at that time, and represents the posterior probability of the first erroneous token at the time.

[0110] In some examples, the device also includes a selection module configured to select a hypothesis with the longest correct prefix from multiple hypotheses of the model performing the cross-entropy training, wherein the first determination module is also configured to determine the posterior probability vector at the time of the first erroneous token included in the selected hypothesis.

[0111] In some examples, the total loss for the training set is determined as follows:

[0112]

[0113] Where L(θ) represents the total loss of the training set, (Y, R) represents the hypothesis-reference pair in the training set, and t jl，ω represents the time, represents the first erroneous token at the time, represents a reference token at said time, and Represents the loss of the first erroneous token.

[0114] The embodiment of the present disclosure also provides a computer device. The device includes: a processor and a memory. The memory is used to store program code and transmit the program code to the processor; the processor is used to execute according to the instructions in the program code: perform cross-entropy training on the model based on one or more input features of the speech signal; determine the posterior probability vector at the time of the first erroneous token in one or more output tokens of the model performing the cross-entropy training; determine the loss of the first erroneous token at the time based on the determined posterior probability vector; determine the total loss of the training set of the model performing the cross-entropy training based on the determined loss of the first erroneous token; and update the model performing the cross-entropy training based on the determined total loss of the training set.

[0115] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

[0116] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0117] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0118] Although particular combinations of features are referenced in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically referenced in the claims and / or disclosed in the specification. Although each dependent claim listed below may directly reference only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

[0119] Unless explicitly stated, any element, action or instruction used in this article should not be interpreted as a key or necessary element, action or instruction. In addition, as used herein, the words "one" and "an" are intended to include one or more items, and can be used interchangeably with "one or more (one or more)". In addition, as used herein, the term "set" is intended to include one or more items (for example, related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more". In the case of meaning only one item, the term "one" or similar language is used. In addition, as used in this article, the terms "have", "have", "contain" etc. are intended to be open terms. In addition, unless explicitly stated otherwise, the phrase "based on" is intended to represent "based at least in part on".

Claims

1. A method for attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, the method comprising: performing cross entropy training on the model based on one or more input features of the speech signal; determining a posterior probability vector at the time of a first erroneous token in one or more output tokens of a model performing the cross entropy training; determining a loss for the first erroneous token at the time based on the determined posterior probability vector; determining a total loss for a training set of a model performing the cross entropy training based on the determined loss for the first erroneous token; as well as The model performing the cross entropy training is updated based on the determined total loss of the training set.

2. The method according to claim 1, characterized in that The posterior probability vector at that time is determined as follows: p t =Decoder(s t-1 ∈{r t-1 ,y t-1 },H enc ), Wherein, t represents the time, p t represents the posterior probability vector at time t, H enc represents the one or more features to be encoded, y t-1 represents the output token at the previous time t-1, r t-1 represents the reference token at the previous time t-1, and s t-1 Indicates that from {r t-1 ,y t-1 } randomly selected tokens.

3. The method according to claim 1, characterized in that The total loss for the training set is determined as follows: Where L(θ) represents the total loss of the training set, (Y, R) represents the hypothesis-reference pair in the training set, and t ω represents the time, represents the first erroneous token at the time, represents a reference token at said time, and Represents the loss of the first erroneous token.

4. The method according to claim 3, characterized in that The loss of the first erroneous token is determined as follows: in, represents the posterior probability of the reference token at said time.

5. The method according to claim 3, characterized in that: The loss of the first erroneous token is determined as follows: in, represents the posterior probability of the reference token at that time, and represents the posterior probability of the first erroneous token at the time.

6. The method according to claim 1, further comprising: selecting the hypothesis with the longest correct prefix from among the multiple hypotheses of the model performing said cross entropy training, Wherein, determining the posterior probability vector at the time includes: A posterior probability vector is determined at the time of the first erroneous token included in the selected hypothesis.

7. The method according to claim 6, characterized in that The total loss for the training set is determined as follows: Where L(θ) represents the total loss of the training set, (Y, R) represents the hypothesis-reference pair in the training set, and t jl，ω represents the time, represents the first erroneous token at the time, represents a reference token at said time, and Represents the loss of the first erroneous token.

8. A device for attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, the device comprising: an execution module configured to cause at least one processor to perform cross entropy training on a model based on one or more input features of a speech signal; a first determination module configured to cause the at least one processor to determine a posterior probability vector at the time of a first erroneous token in one or more output tokens of the model performing the cross entropy training; a second determination module configured to cause the at least one processor to determine a loss of the first erroneous token at the time based on the determined posterior probability vector; a third determination module configured to cause the at least one processor to determine a total loss of a training set of a model performing the cross entropy training based on the determined loss of the first erroneous token; as well as An updating module is configured to cause the at least one processor to update the model performing the cross entropy training based on the determined total loss of the training set.

9. The device according to claim 8, characterized in that The posterior probability vector at that time is determined as follows: p t =Decoder(s t-1 ∈{r t-1 ,y t-1 },H enc ), Wherein, t represents the time, p t represents the posterior probability vector at time t, H enc represents the one or more features to be encoded, y t-1 represents the output token at the previous time t-1, r t-1 represents the reference token at the previous time t-1, and s t-1 Indicates that from {r t-1 ,y t-1 } randomly selected tokens.

10. The device according to claim 8, characterized in that The total loss for the training set is determined as follows: Where L(θ) represents the total loss of the training set, (Y, R) represents the hypothesis-reference pair in the training set, and t ω represents the time, represents the first error token at the time, represents a reference token at said time, and Represents the loss of the first erroneous token.

11. The device according to claim 10, characterized in that The loss of the first erroneous token is determined as follows: in, represents the posterior probability of the reference token at said time.

12. The device according to claim 10, characterized in that The loss of the first erroneous token is determined as follows: in, represents the posterior probability of the reference token at that time, and represents the posterior probability of the first erroneous token at the time.

13. The apparatus of claim 8, further comprising a selection module configured to cause the at least one processor to select a hypothesis having a longest correct prefix from a plurality of hypotheses of the model performing the cross entropy training, in, The first determination module is further configured to cause the at least one processor to determine a posterior probability vector at the time of a first erroneous token included in the selected hypothesis.

14. The device according to claim 13, characterized in that The total loss for the training set is determined as follows: Where L(θ) represents the total loss of the training set, (Y, R) represents the hypothesis-reference pair in the training set, and t jl，ω represents the time, represents the first erroneous token at the time, represents a reference token at said time, and Represents the loss of the first erroneous token.

15. A non-transitory computer-readable medium storing instructions, which, when executed by at least one processor of a device, cause the at least one processor to perform the method for attention-based end-to-end automatic speech recognition training according to any one of claims 1-7.

16. A computer device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the method for attention-based end-to-end automatic speech recognition training according to any one of claims 1-7 according to the instructions in the program code.

Images