Optimal path change rate for weight selection in unsupervised language models
Through the unsupervised method based on the optimal path change rate, selecting the optimal language model weight in the speech recognition system, solving the problems of low selection efficiency of LMW and long calculation time in the prior art, achieving rapid adaptation and high LMW Accuracy selection improves voice recognition performance.
Patent Information
- Application Number
- CN202080015407.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-02-19
- Filing Date
- 2020-02-14
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-02-14
AI Technical Summary
When selecting language model weights (LMW), existing speech recognition systems need to adjust for each task, and it is difficult to select the optimal LMW when the development set is similar to the evaluation set data distribution, especially in the absence of similar development data sets. , resulting in low LMW selection efficiency and long time to calculate WER.
Using an unsupervised method based on the optimal path change rate (BPCR), the LMW with the smallest BPCR is selected as the optimal language model weight to achieve fast adaptation and efficient selection of LMWs.
This method can select the optimal LMW with high precision without real reference, improves the performance of the speech recognition system, reduces dependence on development data sets, and is faster than the prior art when calculating WER.
Smart Images

Figure CN113939869B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to U.S. Application No. 16 / 279,491, filed on February 19, 2019, the disclosure of which is incorporated herein by reference in its entirety. Background Art
[0003] In the speech recognition system of the prior art, the acoustic model P(A|W) and the language model P(W) are trained independently. During decoding, the likelihood score from the acoustic model and the probability from the language model are on different scales, so they need to be combined with a specific scale: score(W)=log(P(A|W))+LMW×log(P(W)), where LMW is a scaled value called the language model weight.
[0004] In many content-word applications, a trained acoustic model can be used together with a domain-specific language model for different recognition tasks. The LMW needs to be tuned for each task. Summary of the invention
[0005] In one embodiment, a method for selecting an optimal language model weight (LMW) for performing automatic speech recognition is provided, the method comprising: decoding a test audio into a word lattice using a language model; analyzing the word lattice using a first LMW of a plurality of LMWs to determine a first plurality of optimal paths; analyzing the word lattice using a second LMW of the plurality of LMWs to determine a second plurality of optimal paths; determining a first best path change rate (BCPR) corresponding to the first LMW based on a number of best path changes between the first plurality of optimal paths and the second plurality of optimal paths; and determining the first LMW to be an optimal LMW based on the first BCPR being a minimum BCPR among a plurality of BCPRs corresponding to the plurality of LMWs.
[0006] In one embodiment, a device for selecting an optimal LMW for performing automatic speech recognition is provided, the device comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to instructions of the program code, the program code comprising: a decoding code configured to cause the at least one processor to decode a test audio into a word lattice using a language model; a first analysis code configured to cause the at least one processor to analyze the word lattice using a first LMW among a plurality of LMWs to determine a first plurality of optimal paths; a second analysis code configured to cause the at least one processor to analyze the word lattice using a second LMW among a plurality of LMWs to determine a second plurality of optimal paths; a first determination code configured to cause the at least one processor to determine a first best path change rate (BCPR) corresponding to the first LMW based on the number of best path changes between the first plurality of best paths and the second plurality of best paths; and a second determination code configured to cause the at least one processor to determine the first LMW as the optimal LMW based on the first BCPR being the minimum BCPR among a plurality of BCPRs corresponding to the plurality of LMWs.
[0007] In one embodiment, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium storing a plurality of instructions, the plurality of instructions including one or more instructions, which, when executed by one or more processors of a device for selecting optimal language model weights (LMWs) for performing automatic speech recognition, cause the one or more processors to: decode test audio into word lattices using a language model; analyze the word lattices using a first LMW of a plurality of LMWs to determine a first plurality of optimal paths; analyze the word lattices using a second LMW of the plurality of LMWs to determine a second plurality of optimal paths; determine a first best path change rate (BCPR) corresponding to the first LMW based on a number of best path changes between the first plurality of optimal paths and the second plurality of optimal paths; and determine the first LMW to be an optimal LMW based on the first BCPR being a minimum BCPR among a plurality of BCPRs corresponding to the plurality of LMWs. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] Figure 1 is a diagram of a method for selecting language model weights.
[0009] Figure 2 is a diagram of an environment in which the methods, apparatuses, and systems described herein may be implemented, according to an embodiment.
[0010] Figure 3 yes Figure 2 A diagram of example components of one or more devices.
[0011] Figure 4 is a flowchart of a method for selecting optimal language model weights according to an embodiment.
[0012] Figure 5 is a flow chart demonstrating the change of the optimal path change rate according to an embodiment.
[0013] Figure 6 is a flow chart of a method for selecting optimal language model weights for performing automatic speech recognition according to an embodiment. DETAILED DESCRIPTION
[0014] The language model weight (LMW) is used in speech recognition systems as a scaling factor for combining the acoustic model and the language model. The LMW used for evaluation is usually chosen to be the one that produces the best performance on the development set. It is assumed that the data distribution of the development set is similar to that of the evaluation set. In practice, it may not be easy to meet this assumption and keep the development set small. Because the LMW is chosen for evaluation, it is best to find the optimal LMW directly on the evaluation set.
[0015] The embodiments described herein include an unsupervised LMW selection method based on the best path change rate (BPCR). The basic idea is based on the observation that the 1-best decoding result measured by WER is usually convex when the language model weights move within a certain range. In addition, when the LMW is close to the optimal point, the best path change rate is very small. Experimental results on various corpora show that the BPCR-based method can find the optimal LMW with high accuracy. The proposed method can be used for many practical applications, such as the rapid adaptation of LMW to different fields.
[0016] Definitions of abbreviations and terms appearing in the detailed description include the following:
[0017] Speech Recognition System: A computer program that is able to recognize speech signals and convert them into written characters / words.
[0018] Language Model Weight (LMW): The language model weight is used as a scaling factor for combining the acoustic model and the language model in a speech recognition system.
[0019] Lattice: A lattice is a directed graph representing the variants of recognition. The decoding lattice contains hundreds or thousands of possible decoding paths and their acoustic scores and language model scores.
[0020] Best path: The path in the lattice with the maximum score, which is a combination of the acoustic score and the language model score balanced by the language model weights. The best path is usually used as the decoding hypothesis.
[0021] Word Error Rate (WER): A measure of the performance of a speech recognition system. It is the minimum number of operations (insertions, deletions, and substitutions) required to achieve a hypothesis for a given word divided by the length of the hypothesis.
[0022] Figure 1 A prior art LMW selection method 100 is shown, which selects LMWs by the performance of different development sets for different tasks. This usually requires a development dataset of human transcriptions that is very similar to the real test data. The model then needs to be used to decode with different LM weights. The decoding results are then compared with the real reference to obtain the word error rate (WER). Finally, the LMW with the minimum WER is selected as the optimal LMW.
[0023] This prior art method of selecting the optimal LMW requires a development data set that is very similar to the test data. This development data may be difficult to collect and is not large enough to provide a reliable WER. In addition, this prior art method requires calculating the WER for each LMW.
[0024] Figure 2 2 is a diagram of an example environment 200 in which the systems and / or methods described herein may be implemented. Figure 2 As shown, environment 200 may include user device 210, platform 220, and network 230. The devices of environment 200 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.
[0025] The user device 210 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with the platform 220. For example, the user device 210 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a radio phone, etc.), a wearable device (e.g., a pair of smart glasses or a smart watch), or the like. In some implementations, the user device 210 may receive information from the platform 220 and / or send information to the platform 220.
[0026] The platform 220 includes one or more devices capable of selecting an optimal LMW for performing automatic speech recognition, as described elsewhere herein. In some implementations, the platform 220 may include a cloud server or a group of cloud servers. In some implementations, the platform 220 may be designed as a modular platform so that certain software components may be transferred in or out according to specific needs. Therefore, the platform 220 may be easily and / or quickly reconfigured for different purposes.
[0027] In some implementations, as shown, the platform 220 can reside in a cloud computing environment 222. It is worth noting that although the implementations described herein describe the platform 220 as residing in the cloud computing environment 222, in some implementations, the platform 220 is not cloud-based (i.e., can be implemented outside of a cloud computing environment) or can be partially cloud-based.
[0028] Cloud computing environment 222 includes an environment hosting platform 220. Cloud computing environment 222 may provide computing, software, data access, storage, and other services that do not require end users (e.g., user devices 210) to be aware of the physical location and configuration of the systems and / or devices hosting platform 220. As shown, cloud computing environment 222 may include a set of computing resources 224 (collectively, “computing resources 224” and individually, “computing resource 224”).
[0029] The computing resources 224 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, the computing resources 224 may control the platform 220. Cloud resources may include computing instances running in the computing resources 224, storage devices provided in the computing resources 224, data transmission devices provided by the computing resources 224, etc. In some implementations, the computing resources 224 may communicate with other computing resources 224 via wired connections, wireless connections, or a combination of wired and wireless connections.
[0030] like Figure 2 As further shown, the computing resources 224 include a set of cloud resources, such as one or more application programs ("APP") 224-1, one or more virtual machines ("VM") 224-2, virtualized storage ("VS") 224-3, one or more hypervisors ("HYP") 224-4, etc.
[0031] The applications 224-1 include one or more software applications that can be provided to or accessed by the user device 210 and / or the sensor device 220. The applications 224-1 can eliminate the need to install and run software applications on the user device 210. For example, the applications 224-1 can include software associated with the platform 220 and / or any other software that can be provided via the cloud computing environment 222. In some implementations, one application 224-1 can send information to / receive information from one or more other applications 224-1 via the virtual machine 224-2.
[0032] Virtual machine 224-2 includes a software implementation of a machine (e.g., a computer) that runs a program similar to a physical machine. Depending on the degree of correspondence and purpose of virtual machine 224-2 to any real machine, virtual machine 224-2 can be a system virtual machine or a process virtual machine. System virtual machines can provide a complete system platform that supports the operation of a complete operating system ("OS"). Process virtual machines can run a single program and can support a single process. In some implementations, virtual machine 224-2 can run on behalf of a user (e.g., user device 210) and can manage the infrastructure of cloud computing environment 222, such as data management, synchronization, or long-term data transfer.
[0033] Virtualized memory 224-3 includes one or more storage systems and / or one or more devices using virtualization technology within the storage system or device of computing resource 224. In some implementations, in the context of a storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage, so that the storage system can be accessed without considering physical storage or heterogeneous structures. Separation allows administrators of storage systems to have flexibility in how administrators manage storage for end users. File virtualization can eliminate the dependency between data accessed at the file level and the location where the file is physically stored. This can optimize storage usage, server consolidation, and / or non-disruptive file migration performance.
[0034] Hypervisor 224-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer such as computing resource 224. Hypervisor 224-4 may present a virtual operating platform to the guest operating systems and may manage the operation of the guest operating systems. Multiple instances of various operating systems may share virtualized hardware resources.
[0035] The network 230 includes one or more wired networks and / or wireless networks. For example, the network 230 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these networks or other types of networks.
[0036] Figure 2 The number and arrangement of devices and networks shown are provided as examples. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different Figure 2 The devices and / or networks shown are arranged differently. Figure 2 Two or more of the devices shown may be implemented in a single device, or Figure 2 The single device shown may be implemented as multiple distributed devices. Additionally, or alternatively, one set of devices (eg, one or more devices) of environment 200 may perform one or more functions described as being performed by another set of devices of environment 200.
[0037] Figure 3300 is a diagram of example components of device 300. Device 300 may correspond to user device 210 and / or platform 220. Figure 3 As shown, device 300 may include a bus 310 , a processor 320 , a memory 330 , a storage component 340 , an input component 350 , an output component 360 , and a communication interface 370 .
[0038] The bus 310 includes components that allow communication between components of the device 300. The processor 320 is implemented in hardware, firmware, or a combination of hardware and software. The processor 320 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 320 includes one or more processors that can be programmed to perform functions. The memory 330 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 320.
[0039] Storage component 340 stores information and / or software related to the operation and use of device 300. For example, storage component 340 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cartridge, a magnetic tape, and / or another type of non-transitory computer-readable medium, and a corresponding drive.
[0040] Input components 350 include components that allow device 300 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). In addition, or alternatively, input components 350 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 360 include components that provide output information from device 300 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).
[0041] The communication interface 370 includes components similar to a transceiver (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 300 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 370 may allow the device 300 to receive information from another device and / or provide information to another device. For example, the communication interface 370 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.
[0042] Device 300 may perform one or more processes described herein. Device 300 may perform these processes in response to processor 320 executing software instructions stored by a non-transitory computer-readable medium such as memory 330 and / or storage component 340. Computer-readable media is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.
[0043] The software instructions may be read into the memory 330 and / or storage component 340 from another computer-readable medium or from another device via the communication interface 370. When executed, the software instructions stored in the memory 330 and / or storage component 340 may cause the processor 320 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.
[0044] Figure 3 The number and arrangement of components shown are provided as examples. In practice, device 300 may include additional components, fewer components, different components, or different Figure 3 Additionally, or alternatively, one set of components (eg, one or more components) of device 300 may perform one or more functions described as being performed by another set of components of device 300.
[0045] Embodiments of the present disclosure relate to unsupervised LMW selection methods based on Best Path Change Rate (BPCR). According to embodiments, these methods select LMW based on statistics of decoding results of an evaluation set without knowing the true reference. Figure 4An exemplary LMW selection method 400 is shown. Test audio is decoded into a word lattice with a predetermined scale. The BRCR calculation requires multiple forward passes on the word lattice, but the WER is not calculated. Experimental results on various standard Kaldi speech recognition systems show that these methods can obtain LMWs comparable to or even better than the prior art LMW selection methods. In addition, for each LMW, the WER is calculated, and when calculating the true WER number, the embodiments of the present disclosure can operate faster than the prior art methods.
[0046] The embodiments of the present disclosure are applicable to all traditional speech recognition tasks, where the acoustic model and the language model are trained independently, and the scores of the acoustic model and the language model need to be balanced during decoding. For example, when it is difficult to collect a large amount of development data similar to the test data with high-quality human transcription, the embodiments described herein may be useful.
[0047] The example BPCR method according to the embodiment first decodes the test audio into a decoding word grid with initial weights. Then, starting with weight w and a fixed step size ε, Viterbi decoding is performed for each time t, where w+t*ε, t=0...n. For each utterance in the test audio, the best path is compared with the previous best path, and a value of 1 is given if the best path changes, otherwise a value of 0 is given. At each time t, the total number of best path changes is collected, which is recorded as r(w). The difference between adjacent r(w)s is then calculated. The weight with the smallest |r(w)-r(w+ε)| is selected as the optimal language model weight.
[0048] The best path may refer to the output sequence with the highest posterior probability in the lattice. For each utterance, when the LMW changes, the best path ps(w) may change. When the LMW changes from w to w+ε, the number of changes in the best path over the entire dataset may be expressed as c(w), as shown in Equation 1:
[0049]
[0050] where S represents the utterances in the entire dataset, δ(·) represents the Kronecker δ (indicator) function. w ranges from w0 to w n , with a step size of ε. Note that we only care whether the best path changes, regardless of how many words change in that path. BPCR is defined as the number of best path changes divided by the total number of utterances in the dataset. In Equation 2, BPCR is denoted as r(w) and the total number of utterances is denoted as N:
[0051]
[0052] Let R(w) be a continuous function defined by connecting the values of r(w) at discrete points [w, w+ε, w+2ε...w+(n-1)ε] with linear functions. Therefore, R(w) is confined to the continuous region [w0, w n ] on. Similarly, Ir(w) and Dr(w) are defined. Ir(w) represents the normalized number of best paths that increase WER after a LMW change. Dr(w) represents the normalized number of best paths that decrease WER after a LMW change. It should be noted that some best path changes do not result in a change in WER. These quantities are divided in half and added to Ir(w) and Dr(w) respectively, so that the sum of Ir(w) and Dr(w) is R(w), as shown in Equation 3:
[0053] R(w c )=I r (w c )+D r (w c ) (Equation 3)
[0054] where wc is a continuous variable in the range [w0, Wn]. The intersection of the curves Ir(w) and Dr(w) indicates that when w changes from w to w+ε, an equal number of optimal paths change to increase or decrease WER.
[0055] When the language model weight is wc, let E(wc) be WER. In many ASR systems, E(wc) is convex and has a minimum point in the range [w0,wn]. Because E(wc) decreases on the left side of the optimal LMW and increases on the right side, it is assumed that Ir(wc) and Dr(wc) are symmetric around wc = wopt. Without loss of generality, the linear functions in Equations 4 and 5 can represent Ir(wc) and Dr(wc) close to wopt, respectively:
[0056]
[0057]
[0058] Where k (k ≥ 0) represents the slope of the line. bI and bD are the intercepts of Ir(wc) and Dr(wc), respectively. Based on Equation 3, R(wc) close to wopt can be written as Equation 6:
[0059]
[0060]
[0061] Equation 6 can then be used to derive Equation 7. Note that it may not be possible to find such a wopt, so the LMW corresponding to the minimum value of R'(wopt) is selected for evaluation. In other words, if shifting w by ε results in the minimum optimal path change rate, then w is selected as the optimal LMW:
[0062]
[0063] Figure 5 An example of the relationship between Dr(w), Ir(w), and BPCR(w) is given, assuming a reference is given. BPCR(w) is R(w). Moving w to w+ε on the left side of the x-axis representing the language model weights generally improves the WER so that more best path changes fall into the D set rather than the I set. After some point, more path changes fall into the I set rather than the D set. The minimum R(w) is chosen, which is the sum of the best path changes, regardless of whether the best path change is from D or I.
[0064] Figure 6 6 is a flow chart of a method 600 for selecting an optimal language model weight (LMW) for performing automatic speech recognition according to an embodiment. In some implementations, the platform 220 may execute Figure 6 In some implementations, the processing may be performed by another device or group of devices separate from the platform 220 or another device or group of devices including the platform 220 (e.g., the user device 210). Figure 6 One or more processing blocks.
[0065] like Figure 6 As shown, in operation 610, method 600 may include decoding test audio into word lattices using a language model.
[0066] like Figure 6 As further shown, at operation 620 , method 600 may include analyzing the lattice using a first LMW of the plurality of LMWs to determine a first plurality of optimal paths.
[0067] like Figure 6 As further shown, at operation 620 , method 600 may include analyzing the lattice using a second LMW of the plurality of LMWs to determine a second plurality of optimal paths.
[0068] like Figure 6 As further shown, at operation 620, the method 600 may include determining a first best path change rate (BCPR) corresponding to the first LMW based on a number of best path changes between the first plurality of best paths and the second plurality of best paths.
[0069] like Figure 6As further shown, at operation 650, method 600 may include determining whether the first BCPR is a minimum BCPR among a plurality of BCPRs corresponding to a plurality of LMWs. If the answer at operation 650 is determined to be yes, then at operation 660, the first LMW may be determined to be an optimal LMW. If the answer at operation 650 is determined to be no, then at operation 670, method 600 may be repeated using another LMW among the plurality of LMWs. Operation 600 may be performed in this manner until an optimal LMW is found.
[0070] In one embodiment, the plurality of LMWs may include a plurality of scaling factors for combining the language model with the acoustic model.
[0071] In one embodiment, each LMW of the plurality of LMWs may be separated from an adjacent LMW of the plurality of LMWs by a fixed step size.
[0072] In one embodiment, each score of the plurality of scores corresponding to the first plurality of best paths may include a combination of an acoustic score and a language model score balanced by a first LMW, and each best path of the first plurality of best paths may include a path in the lattice having a maximum score.
[0073] In one embodiment, the first BCPR among the multiple BCPRs may be closest to R'(w opt ), and determine the first LMW as the optimal LMW.
[0074] Although Figure 6 Exemplary blocks of method 600 are shown, but in some implementations, method 600 may include additional blocks, fewer blocks, different blocks, or different blocks. Figure 6 Depicted blocks are blocks that are arranged differently. Additionally, or alternatively, two or more blocks of method 600 may be performed in parallel.
[0075] In the above embodiment, BPCR may be based on path changes, that is, the value of each utterance is 1 or 0. However, the present disclosure is not limited thereto. For example, in an embodiment, the number of word changes may be considered to compare the edit distance between the new best path and the previous best path.
[0076] Embodiments of the present disclosure are tested on four benchmark datasets: Switchboard Switch-1 Second Edition (SWBD), Wall Street Journal (WSJ), CHiME-4, and LibriSpeech. The SWBD corpus is a collection of two-way telephone conversations with speakers from all regions of the United States. The WSJ corpus consists mainly of read speech from Wall Street Journal news. CHiME-4 can be seen as a multi-channel extension of WSJ for speech recognition of long-distance speaking in noisy environments. LibriSpeech is a relatively large corpus containing approximately 1,000 hours of English read speech from audio books. On these four corpora, six acoustic models are trained using the corresponding Kaldi method. The language model (LM) is constructed using interpolated Kneser-Ney smoothing using the SRILM toolkit.
[0077] The SWBD corpus contains approximately 260 hours of English telephone conversations between 543 speakers. Two deep neural network (DNN) based acoustic models are trained on MFCC features. One acoustic model uses the frame-by-frame cross entropy training criterion and the other acoustic model is further trained using the state-level minimum Bayesian risk (sMBR) criterion. During decoding, a trigram LM trained on 3M words of the SWBD training transcript is applied to both acoustic models. For the sMBR trained acoustic model, additional experiments are performed to interpolate the SWBD trigram LM using a trigram LM trained on 11M words of the Fisher English Part 1 transcript. The Hub5'00 collection containing 4458 utterances is used as the test set in the experiments.
[0078] The WSJ training set (i.e., WSJ0 SI-84) contains approximately 80 hours of recordings from 84 speakers. The acoustic model is a DNN trained with a cross-entropy loss. A ternary LM with a vocabulary size of 146K is used during decoding. The Nov'92ARPA WSJ set containing 333 utterances is used for evaluation. Experiments on the CHiME-4 corpus follow the Kaldi method of Chen et al. A generalized eigenvalue (GEV) beamformer based on a bidirectional LSTM is applied. For acoustic modeling, a subsampled time-delay neural network (TDNN) is trained using lattice-free maximum mutual information (LF-MMI). A 5-gram LM is used for decoding. These models are evaluated on 330×6 actual recordings of the ChiME-4 challenge. The acoustic model on the LibriSpeech corpus is trained using the sMBR criterion and decoded using a large 4-gram LM. For evaluation, 2939 utterances from the "test-other" set are used.
[0079] Using the acoustic model and language model according to an embodiment of the present disclosure, a word grid on the evaluation set can be generated using a scaling value of 10. It was found in the experiment that for a given word grid, the LM cost change always corresponds to the best path change, rather than using the true best path change by comparing the 1-best path extracted from the word grid. Therefore, the LM cost change can be used as an indicator of the best path change ps(w) in Equation 1. In an embodiment, the LM cost is normalized by the length of the estimated transcription. Table 1 shows the theoretical optimal values (oracle), conventional and BPCR LMW on various corpora and models. The cross entropy training criterion is denoted as CE.
[0080] Table 1 Theoretical optimal values, conventional and BPCR LMW
[0081]
[0082] Even without using the development set, BPCR according to an embodiment obtains LMW estimates comparable to or better than prior art methods. This demonstrates the promise of BPCR as an alternative to prior art LMW selection methods in tasks where fast LMW selection and adaptability are important.
[0083] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise forms disclosed. Modifications and variations may be made in light of the foregoing disclosure or may be acquired from practice of the implementations.
[0084] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, or a combination of hardware and software.
[0085] Obviously, the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.
[0086] Even though particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each of the dependent claims listed below may only be directly dependent on one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.
[0087] Any element, action or instruction used herein should not be understood as critical or necessary unless so clearly described. In addition, as used herein, constraints "one" and "an" are intended to include one or more projects, and can be used interchangeably with "one or more". In addition, as used herein, the term "set" is intended to include one or more projects (e.g., related projects, non-related projects, combinations of related projects and non-related projects, etc.), and can be used interchangeably with "one or more". In the case where it is intended to be only one project, the term "one" or similar language is used. In addition, as used herein, the term "having", "containing", "having" or similar terms are intended to be open terms. In addition, the phrase "based on" is intended to mean "based at least in part on", unless otherwise explicitly stated.
Claims
1. A method for selecting an optimal language model weight (LMW) for performing automatic speech recognition, characterized in that The method comprises: Decode the test audio into word lattices using a language model; analyzing the word lattice using a first LMW of a plurality of LMWs to determine a first plurality of optimal paths; analyzing the word lattice using a second LMW of the plurality of LMWs to determine a second plurality of optimal paths; A first best path change rate (BCPR) corresponding to the first LMW is determined based on a number of best path changes between the first plurality of best paths and the second plurality of best paths, the number of best path changes being expressed as follows: Where c(w) represents the number of optimal path changes, s is the index of the utterance, S represents the utterance in the dataset corresponding to the test audio, δ() represents the Kronecker delta function, and p s () represents an optimal path, w represents the first LMW, ε represents a fixed step size, and w+ε represents the second LMW; and The first LMW is determined to be an optimal LMW based on the first BCPR being a minimum BCPR among a plurality of BCPRs corresponding to the plurality of LMWs.
2. The method according to claim 1, wherein: The plurality of LMWs include a plurality of scaling factors for combining the language model with an acoustic model.
3. The method according to claim 1, wherein: Each LMW of the plurality of LMWs is separated from an adjacent LMW of the plurality of LMWs by a fixed step size.
4. The method according to claim 1, wherein: Each score of the plurality of scores corresponding to the first plurality of best paths comprises a combination of an acoustic score and a language model score balanced by the first LMW, and wherein each best path of the first plurality of best paths comprises a path in the lattice having a maximum score.
5. The method according to claim 1, wherein: The first BCPR is expressed as follows: Where r(w) represents the first BCPR, and N represents the total number of utterances in the dataset.
6. The method according to claim 5, wherein: The continuous function corresponding to the BCPR is expressed as follows: R(w c )=Ir(w c )+D r (w c ), Where R(w c ) represents the continuous function corresponding to the BCPR, where I r (w c ) represents the normalized number of best paths that increase word error rate after LMW change, and D r (w c ) represents the normalized number of best paths that reduce the word error rate after the LMW change.
7. The method according to claim 6, wherein: I r (w c ) and D r (w c ) is expressed as follows: as well as where w c is the weight of the language model, w opt represents the optimal LMW, k represents (k≥0) the slope, b I Indicates I r (w c ), and b D Indicates D r (w c )’s intercept.
8. The method according to claim 7, wherein: R(w c ) is expressed as follows: And among them, R'(w opt ) is expressed as follows:
9. The method according to claim 8, wherein: Based on the fact that the first BCPR is closest to R'(w opt ), and determining the first LMW as the optimal LMW.
10. A device for selecting an optimal language model weight (LMW) for performing automatic speech recognition, characterized in that The device comprises: at least one memory configured to store program code; and at least one processor configured to read the program code and operate according to instructions of the program code, the program code comprising: decoding code configured to cause the at least one processor to decode the test audio into word lattices using the language model, first analysis code configured to cause the at least one processor to analyze the word lattice using a first LMW of a plurality of LMWs to determine a first plurality of optimal paths, second analysis code configured to cause the at least one processor to analyze the word lattice using a second LMW of the plurality of LMWs to determine a second plurality of optimal paths, A first determining code configured to cause the at least one processor to determine a first best path change rate (BCPR) corresponding to the first LMW based on a number of best path changes between the first plurality of best paths and the second plurality of best paths, the number of best path changes being expressed as follows: Where c(w) represents the number of optimal path changes, s is the index of the utterance, S represents the utterance in the dataset corresponding to the test audio, δ() represents the Kronecker delta function, and p s () represents an optimal path, w represents the first LMW, ε represents a fixed step size, and w+ε represents the second LMW; and The second determination code is configured to cause the at least one processor to determine that the first LMW is an optimal LMW based on the first BCPR being a minimum BCPR among a plurality of BCPRs corresponding to the plurality of LMWs.
11. The device according to claim 10, wherein: The plurality of LMWs include a plurality of scaling factors for combining the language model with an acoustic model.
12. The device according to claim 10, wherein: Each LMW of the plurality of LMWs is separated from an adjacent LMW of the plurality of LMWs by a fixed step size.
13. The apparatus according to claim 10, wherein: The first BCPR is expressed as follows: Where r(w) represents the first BCPR, and N represents the total number of utterances in the dataset.
14. The device according to claim 13, wherein: The continuous function corresponding to the BCPR is expressed as follows: R(w c )=I r (w c )+D r (w c ), Where R(w c ) represents the continuous function corresponding to the BCPR, where I r (w c ) represents the normalized number of best paths that increase word error rate after LMW change, and D r (w c ) represents the normalized number of best paths that reduce the word error rate after the LMW change.
15. The device according to claim 14, wherein: I r (w c ) and D r (w c ) is expressed as follows: as well as where w c is the weight of the language model, w opt represents the optimal LMW, k represents (k≥0) the slope, b I Indicates I r (w c ), and b D Indicates D r (w c )’s intercept.
16. The device according to claim 15, wherein: R(w c ) is expressed as follows: And among them, R'(w opt ) is expressed as follows:
17. The device according to claim 16, wherein: Based on the fact that the first BCPR is closest to R'(w opt ), and determining the first LMW as the optimal LMW.
18. A non-transitory computer-readable medium storing instructions, characterized in that: The instructions include: one or more instructions that, when executed by one or more processors of a device for selecting optimal language model weights LMW for performing automatic speech recognition, cause the one or more processors to: Use the language model to decode the test audio into word lattices, analyzing the word lattice using a first LMW of a plurality of LMWs to determine a first plurality of optimal paths, analyzing the word lattice using a second LMW of the plurality of LMWs to determine a second plurality of optimal paths, A first best path change rate (BCPR) corresponding to the first LMW is determined based on a number of best path changes between the first plurality of best paths and the second plurality of best paths, the number of best path changes being expressed as follows: Where c(w) represents the number of optimal path changes, s is the index of the utterance, S represents the utterance in the dataset corresponding to the test audio, δ() represents the Kronecker delta function, and p s () represents an optimal path, w represents the first LMW, ε represents a fixed step size, and w+ε represents the second LMW; and Based on the first BCPR being a minimum BCPR among a plurality of BCPRs corresponding to the plurality of LMWs, the first LMW is determined to be an optimal LMW.
19. An apparatus for selecting an optimal language model weight (LMW) for performing automatic speech recognition, characterized in that include: a decoding module configured to decode the test audio into word lattices using a language model; a first analysis module configured to analyze the word lattice using a first LMW of the plurality of LMWs to determine a first plurality of optimal paths; a second analysis module configured to analyze the word lattice using a second LMW of the plurality of LMWs to determine a second plurality of optimal paths; A first determining module is configured to determine a first best path change rate (BCPR) corresponding to the first LMW based on a number of best path changes between the first plurality of best paths and the second plurality of best paths, wherein the number of best path changes is expressed as follows: Where c(w) represents the number of optimal path changes, s is the index of the utterance, S represents the utterance in the dataset corresponding to the test audio, δ() represents the Kronecker delta function, and p s () represents the optimal path, w represents the first LMW, ε represents a fixed step size, and w+ε represents the second LMW; as well as The second determination module is configured to determine that the first LMW is an optimal LMW based on the first BCPR being a minimum BCPR among a plurality of BCPRs corresponding to the plurality of LMWs.
