Speech recognition model training method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to a method and device for training a speech recognition model, equipment and a storage medium. The method provided by the invention comprises the following steps: generating a voice feature sequence of a voice sample by using a coding model; processing the speech feature sequence using a language model to generate probability information; providing the speech feature representation to a reference model corresponding to the language model to obtain a set of recognition texts corresponding to the speech sample; determining training loss based on the probability information, the group of recognition texts and a labeling text corresponding to the voice sample; and adjusting parameters of the speech recognition model based on the training loss. In this way, according to the embodiment of the invention, the voice recognition model can be trained based on a reinforcement learning mode, so that the training efficiency of the voice recognition model is improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for training speech recognition models. Background Technology

[0002] With the development of internet and computer technology, speech recognition has become an important fundamental capability. For example, some solutions can utilize machine learning-based speech recognition models to perform speech recognition tasks. The training process of a speech recognition model directly affects its recognition accuracy. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for training a speech recognition model is provided. The method includes: generating a speech feature sequence of a speech sample using an encoding model; processing the speech feature sequence using a language model to generate probability information; providing a speech feature representation to a reference model corresponding to the language model to obtain a set of recognition text corresponding to the speech sample; determining a training loss based on the probability information, the set of recognition text, and the labeled text corresponding to the speech sample; and adjusting the parameters of the speech recognition model based on the training loss.

[0004] In a second aspect of this disclosure, an apparatus for training a speech recognition model is provided. The apparatus includes: a generation module configured to generate a speech feature sequence of speech samples using an encoding model; a prediction module configured to process the speech feature sequence using a language model to generate probability information; a providing module configured to provide speech feature representations to a reference model corresponding to the language model to obtain a set of recognized text corresponding to the speech samples; a determining module configured to determine a training loss based on the probability information, the set of recognized text, and labeled text corresponding to the speech samples; and an adjustment module configured to adjust the parameters of the speech recognition model based on the training loss.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A block diagram is shown in which an example speech recognition model according to embodiments of the present disclosure may be implemented;

[0010] Figure 2 A flowchart illustrating an example process for training a speech recognition model according to some embodiments of the present disclosure is shown;

[0011] Figure 3 Example training block diagrams according to some embodiments of the present disclosure are shown;

[0012] Figure 4 A schematic structural block diagram of an example apparatus for training a speech recognition model according to some embodiments of the present disclosure is shown; and

[0013] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0018] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0019] As mentioned above, the training process of a speech recognition model directly affects its recognition accuracy. For example, traditional training can be performed end-to-end based on the difference between the decoding and annotation results; however, due to the long decoding process, such training is relatively inefficient.

[0020] Embodiments of this disclosure propose a scheme for training a speech recognition model. The scheme includes: generating a speech feature sequence of a speech sample using an encoding model; processing the speech feature sequence using a language model to generate probability information; providing a speech feature representation to a reference model corresponding to the language model to obtain a set of recognition text corresponding to the speech sample; determining a training loss based on the probability information, the set of recognition text, and the labeled text corresponding to the speech sample; and adjusting the parameters of the speech recognition model based on the training loss.

[0021] In this way, the embodiments of this disclosure can train speech recognition models based on reinforcement learning, thereby improving the training efficiency of speech recognition models.

[0022] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0023] Example speech recognition model

[0024] Figure 1 A block diagram is shown in which an example speech recognition model 100, in which embodiments of the present disclosure may be implemented, is illustrated.

[0025] like Figure 1 As shown, the speech recognition model 100 may include three sub-models: an encoding model 110, a conversion model 120, and a language model 130. For example... Figure 1 As shown, the encoding model 110 can be configured to acquire speech content 135 to encode it as an intermediate feature representation.

[0026] Furthermore, the transformation model 120 can convert the intermediate feature representation into a speech feature sequence 140, also known as a speech embedding representation or speech token. For example, the transformation model 120 can use a linear layer to map the intermediate feature representation to the feature dimension corresponding to the language model 130.

[0027] Accordingly, the language model 130 can be configured to generate a speech recognition result 150 corresponding to the speech content 135 based on the received input feature sequence. Such an input feature sequence may include, for example, a cue feature sequence 145 and a speech feature sequence 140. The cue feature sequence 145 may correspond to a preset cue item to instruct the language model 130 to perform a speech recognition task.

[0028] In some embodiments, the language model 130 can output speech recognition results based on NTP (next token prediction). For example... Figure 1 As shown in the figure <bos>(beginning of sentence) indicates the start of a sentence. <eos>(end of sentence) represents the end identifier of a sentence.

[0029] As Figure 1 shown, when predicting the output token, the language model 130 can predict the next output token based on the existing token sequence. For example, the language model 130 can sequentially output text tokens corresponding to the speech content 135, such as "the", "weather", "is", "nice", and "ah".

[0030] Thus, the speech recognition model 100 can utilize the language model 130 to achieve the recognition of the speech content 135. The training process of the speech recognition model 100 will be further described below.

[0031] Example process

[0032] Figure 2 shows a flowchart of an example process 200 for training a speech recognition model according to some embodiments of the present disclosure. The process 200 can be implemented at a suitable electronic device. The following will refer to Figure 1 to describe the process 200.

[0033] As Figure 2 shown, at block 210, the electronic device uses an encoding model to generate a speech feature sequence of a speech sample.

[0034] The following will refer to Figure 3 to describe a training framework 300 according to some embodiments of the present disclosure. As Figure 3 shown, the policy model 310 can be deployed at a first device, also referred to as a training device, for example. The reference model 325 can be deployed at a second device, also referred to as a decoding device, for example.

[0035] As Figure 3 shown, the policy model 310 can include, for example, a speech recognition model to be trained, which can include an encoder 320 and a language model 315. In some examples, the policy model 310 can also include, for example, a Figure 1 conversion model as shown.

[0036] In some embodiments, the reference model 325 can include a language model 330. In some embodiments, the parameters of the language model 315 in the policy model 310 can be used to initialize the language model 330. During the reinforcement learning process, the parameters of the language model 330 can remain unchanged, for example.

[0037] In some embodiments, the policy model 310 can include, for example, the speech recognition model 100 trained through a self-supervised training process and a supervised training process. Figure 2 The described reinforcement learning process can further optimize such speech recognition model 100.

[0038] As shown in Figure 3 The training device can process the speech sample 305 with the encoder 320 and a conversion model (not shown in the figure) to generate a sequence of speech features, as shown in

[0039] At block 220, the training device processes the sequence of speech features with a language model to generate probability information. Specifically, the training device can process an input feature sequence constructed based on the sequence of speech features with the language model 315 and generate probability information (also referred to as logits), as shown in Figure 3 The construction process of the input feature sequence and the processing process of the language model can refer to the description of Figure 1 The described content, which will not be repeated here.

[0040] At block 230, the training device provides the speech feature representation to a reference model corresponding to the language model to obtain a set of recognized texts corresponding to the speech sample.

[0041] Specifically, the training device can provide the generated sequence of speech features to the language model 330. Further, the language model 330 can process an input feature sequence constructed based on the sequence of speech features and can perform a decoding process to generate a set of recognized texts 340 (also referred to as nbest, i.e., n best recognition results).

[0042] In some embodiments, the language model 330 can determine the set of recognized texts 340 based on a beam search process, for example, n best recognition results.

[0043] With reference back to Figure 2 At block 240, the training device determines a training loss based on the probability information, the set of recognized texts, and the annotated text corresponding to the speech sample.

[0044] Specifically, the training device can determine a set of probabilities corresponding to the set of recognized texts 340 based on the probability information output by the language model 315. Further, the training device can also determine evaluation information of the set of recognized texts 340 based on the annotated text.

[0045] In some embodiments, such evaluation information may, for example, include a word error rate WER (Word Error Rate) and / or a weighted word error rate WWER (Weighted Word Error Rate) determined based on the annotated text.

[0046] Furthermore, the training device can determine the training loss 345 based on a set of probabilities and corresponding evaluation information. For example, the above process can be represented as:

[0047]

[0048] in, W(y) represents the posterior probability of the group of recognized texts 340 determined based on the probability information output by language model 315, where x represents the speech feature sequence, N represents the number of recognized texts output by language model 330, and Beam represents the beam search process; i ,y ★ ) indicates the recognition of text y i and the annotation text y ★ WER or WWER; This represents the average WER or average WWER of the identified text in this group.

[0049] In box 250, the parameters of the speech recognition model are adjusted based on the training loss.

[0050] In some embodiments, during the process of adjusting the policy model 310 based on the training loss 345 determined by formula (1), the training device may, for example, adjust at least the parameters of the encoding model 320. In some embodiments, the parameters of the transformation model may, for example, be fixed.

[0051] In some embodiments, during reinforcement learning, the training device may, for example, fix the parameters of the language model 315. Alternatively, the training device may collaboratively fine-tune the parameters of the language model 315. As another example, the training device may also adjust the parameters of a fine-tuning module associated with the language model 315. For example, the training device may adjust the parameters of a low-rank adaptation (Lora) module associated with the language model 315.

[0052] In some embodiments, as mentioned above, to improve the decoding efficiency of the reference model 325, the reference model 325 may be deployed at a different device (e.g., a decoding device) than the training device. In some embodiments, considering that decoding requires a relatively longer time, the computing power of the decoding device may, for example, be higher than that of the training device.

[0053] In this way, the embodiments of this disclosure can train speech recognition models based on reinforcement learning, thereby improving the training efficiency of speech recognition models.

[0054] Example devices and equipment

[0055] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an example apparatus 400 for training a speech recognition model according to certain embodiments of the present disclosure is shown. Apparatus 400 may be implemented as or included in an electronic device. Various modules / components in apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0056] like Figure 4 As shown, the apparatus 400 includes a generation module 410 configured to generate a speech feature sequence of a speech sample using an encoding model; a prediction module 420 configured to process the speech feature sequence using a language model to generate probability information; a providing module 430 configured to provide a speech feature representation to a reference model corresponding to the language model to obtain a set of recognition text corresponding to the speech sample; a determining module 440 configured to determine a training loss based on the probability information, a set of recognition text, and labeled text corresponding to the speech sample; and an adjusting module 450 configured to adjust the parameters of the speech recognition model based on the training loss.

[0057] In some embodiments, the speech recognition model further includes a transformation model, and the generation module 410 is further configured to: process speech samples using an encoding model to generate intermediate feature representations; and convert the intermediate feature representations into a speech feature sequence using the transformation model.

[0058] In some embodiments, a set of identified texts includes a set of identified texts determined by a reference model based on a beam search process.

[0059] In some embodiments, the determining module 440 is further configured to: determine a set of probabilities corresponding to a set of identified text based on probability information; determine evaluation information for a set of identified text based on labeled text; and

[0060] The training loss is determined based on a set of probabilities and corresponding evaluation information.

[0061] In some embodiments, the determining module 440 is further configured to: determine a set of word error rates and / or weighted word error rates of the identified text based on the labeled text.

[0062] In some embodiments, the adjustment module 450 is further configured to: adjust the parameters of the encoding model in the speech recognition model based on the training loss.

[0063] In some embodiments, the adjustment module 450 is further configured to: fix the parameters of the language model; fine-tune the parameters of the language model; or adjust the parameters of the fine-tuning module associated with the language model.

[0064] In some embodiments, the language model is deployed on a first device, and the reference model is deployed on a second device.

[0065] In some embodiments, the computing power of the second device is higher than that of the first device.

[0066] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example, and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0067] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Electronic devices.

[0068] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0069] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0070] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0071] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0072] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0073] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0074] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0075] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0076] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0077] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0078] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.< / eos> < / bos>

Claims

1. A method for training a speech recognition model, the speech recognition model comprising an encoding model and a language model, the method comprising: The encoding model is used to generate speech feature sequences of speech samples; The language model is used to process the speech feature sequence to generate probability information; The speech feature representation is provided to a reference model corresponding to the language model to obtain a set of recognized text corresponding to the speech sample; Based on the probability information, the set of recognized text, and the labeled text corresponding to the speech sample, the training loss is determined; as well as The parameters of the speech recognition model are adjusted based on the training loss.

2. The method according to claim 1, wherein the speech recognition model further includes a transformation model, and generating a speech feature sequence of a speech sample using the encoding model includes: The speech samples are processed using the coding model to generate intermediate feature representations; as well as The intermediate feature representation is converted into the speech feature sequence using the transformation model.

3. The method of claim 1, wherein the set of identified texts includes a set of identified texts determined by the reference model based on the beam search process.

4. The method according to claim 1, wherein determining the training loss based on the probability information, the set of recognized texts, and the labeled texts corresponding to the speech samples includes: Based on the probability information, a set of probabilities corresponding to the set of identified text is determined; Based on the labeled text, the evaluation information of the set of identified texts is determined; as well as The training loss is determined based on the set of probabilities and the corresponding evaluation information.

5. The method according to claim 4, wherein determining the evaluation information of the set of identified texts based on the labeled text includes: Based on the labeled text, determine the word error rate and / or weighted word error rate of the set of identified texts.

6. The method according to claim 1, wherein adjusting the parameters of the speech recognition model based on the training loss comprises: Based on the training loss, the parameters of the encoding model in the speech recognition model are adjusted.

7. The method according to claim 6, wherein adjusting the parameters of the speech recognition model based on the training loss further comprises: Fix the parameters of the language model; Fine-tune the parameters of the language model; or Adjust the parameters of the fine-tuning module associated with the language model.

8. The method of claim 1, wherein the language model is deployed at the first device and the reference model is deployed at the second device.

9. The method according to claim 8, wherein the computing power of the second device is higher than that of the first device.

10. An apparatus for training a speech recognition model, the speech recognition model comprising an encoding model and a language model, the apparatus comprising: The generation module is configured to generate a speech feature sequence of speech samples using the coding model; The prediction module is configured to process the speech feature sequence using the language model to generate probability information; A providing module is configured to provide the speech feature representation to a reference model corresponding to the language model in order to obtain a set of recognized text corresponding to the speech sample; The determination module is configured to determine the training loss based on the probability information, the set of recognized text, and the labeled text corresponding to the speech sample; as well as The adjustment module is configured to adjust the parameters of the speech recognition model based on the training loss.

11. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 9.

Images