Speech recognition model training method and device
By generating and filtering pseudo-labels with a teacher model, the method improves speech recognition accuracy and robustness in complex environments, addressing errors and resource inefficiencies in existing systems.
Patent Information
- Application Number
- CN202410955523.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-17
AI Technical Summary
Existing speech recognition systems face challenges in improving recognition accuracy in complex environments due to issues like background noise and speaker variations, and current pseudo-labeling methods introduce errors that degrade model performance.
A method involving a speech recognition teacher model to generate benchmark and reference pseudo-labels, constructing data pairs with low error rates, and training a student model using these pairs to enhance accuracy and robustness.
Enhances the robustness and generalization ability of speech recognition models by reducing errors in pseudo-labels, requiring fewer computing resources and improving performance in diverse acoustic conditions.
Smart Images

Figure CN118506770B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computers, and in particular to a speech recognition model training method and device. Background Art
[0002] In recent years, speech recognition has been increasingly used in voice assistants, smart homes, voice translation and other fields. Existing speech recognition systems have a lot of room for improvement in recognition accuracy when faced with complex background noise, speaker changes, accent differences and other issues. With the advancement of deep neural networks, automatic speech recognition (ASR) technology has made significant progress. Like other models based on deep neural networks, the performance of ASR models depends largely on the amount of available labeled speech data. Summary of the invention
[0003] According to an embodiment of the present invention, the speech recognition model training method includes: using a speech recognition teacher benchmark model to generate benchmark pseudo-labels for multiple unlabeled speech data segments, using a speech recognition teacher reference model to generate reference pseudo-labels for multiple unlabeled speech data segments, and calculating the word error rate of the reference pseudo-label of each unlabeled speech data segment in the multiple unlabeled speech data segments relative to the benchmark pseudo-label; constructing multiple unlabeled speech data-pseudo-label training data pairs based on multiple unlabeled speech data segments, the benchmark pseudo-labels of multiple unlabeled speech data segments, and the reference pseudo-labels of the unlabeled speech data segments whose reference pseudo-labels have a word error rate lower than a predetermined value relative to the benchmark pseudo-labels in the multiple unlabeled speech data segments; and training the speech recognition student model based on the multiple unlabeled speech data-pseudo-label training data pairs and their corresponding word error rates.
[0004] According to an embodiment of the present invention, the speech recognition model training device includes: a processor; and a memory on which computer executable instructions are stored, wherein these computer executable instructions, when executed by the processor, prompt the processor to execute the above-mentioned speech recognition model training method.
[0005] According to the computer-readable storage medium of an embodiment of the present invention, computer-executable instructions are stored thereon, wherein these computer-executable instructions, when executed by a processor, prompt the processor to execute the above-mentioned speech recognition model training method.
[0006] The computer program product according to an embodiment of the present invention includes computer executable instructions, wherein when these computer executable instructions are executed by a processor, they prompt the processor to execute the above-mentioned speech recognition model training method. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] The present invention can be better understood from the following description of specific embodiments of the present invention in conjunction with the accompanying drawings, in which:
[0008] Figure 1 A schematic block diagram of a speech recognition model training device according to an embodiment of the present invention is shown;
[0009] Figure 2 A schematic flow chart of a speech recognition model training method according to an embodiment of the present invention is shown;
[0010] Figure 3 A schematic diagram of a computer system that can implement the speech recognition model training device and method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0011] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In the detailed description below, many specific details are proposed to provide a comprehensive understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention. The present invention is by no means limited to any specific configuration and algorithm proposed below, but covers any modification, replacement and improvement of elements, parts and algorithms without departing from the spirit of the present invention. In the accompanying drawings and the following description, known structures and technologies are not shown to avoid unnecessary ambiguity to the present invention.
[0012] At present, considering the high cost of manual labeling, the ASR model is usually trained by using a model training method based on self-supervised learning or pseudo-labeling, where:
[0013] The model training method based on self-supervised learning first pre-trains the ASR neural network with unlabeled speech data, and then fine-tunes the pre-trained ASR neural network with labeled speech data to generate an ASR model. When the ASR model needs to be enhanced with newly acquired unlabeled speech data, the ASR neural network must be re-pre-trained with both the original unlabeled speech data and the newly acquired unlabeled speech data, and then the pre-trained ASR neural network must be re-fine-tuned with the original labeled speech data to generate an enhanced ASR model. It can be seen that the above ASR model training process requires a lot of computing resources, and the cost of model enhancement training is high.
[0014] The pseudo-label-based model training method first uses the ASR teacher model to generate pseudo labels for unlabeled speech data, and then uses the unlabeled speech data and its pseudo labels, as well as the labeled speech data, to train the ASR student model. When the ASR student model needs to be enhanced with newly acquired unlabeled speech data, the ASR teacher model is first used to generate pseudo labels for the newly acquired unlabeled speech data, and then the ASR student model is enhanced with the newly acquired unlabeled speech data and its pseudo labels.
[0015] Compared with the model training method based on self-supervised learning, the model training method based on pseudo-labels requires less computing resources and is more convenient for model enhancement training. However, pseudo-labels inevitably contain errors, and training the ASR student model based on erroneous pseudo-labels may result in low recognition performance of the ASR student model. Existing methods to mitigate the negative impact of noisy pseudo-labels include: estimating the correctness of pseudo-labels based on confidence scores and excluding pseudo-labels with low confidence scores; regularly using the latest ASR student model to improve the ASR teacher model to improve the quality of pseudo-labels generated by the ASR teacher model. Although the above methods can mitigate the negative impact of noisy pseudo-labels to a certain extent, they still cannot eliminate the errors of pseudo-labels.
[0016] In view of the above situation, a speech recognition model training method and device according to an embodiment of the present invention are proposed, which can improve the robustness and generalization ability of the speech recognition model.
[0017] Figure 1 A schematic block diagram of a speech recognition model training device according to an embodiment of the present invention is shown. Figure 2 A schematic flow chart of a speech recognition model training method according to an embodiment of the present invention is shown. Figure 1 and Figure 2 , a speech recognition model training device and method according to an embodiment of the present invention are described in detail.
[0018] like Figure 1As shown, the speech recognition model training device 100 according to an embodiment of the present invention includes a pseudo-label generating unit 102, a training data pair constructing unit 104, and a recognition model training unit 106, wherein: the pseudo-label generating unit 102 is configured to generate a plurality of reference pseudo-labels of unlabeled speech data segments using a speech recognition teacher reference model, generate a plurality of reference pseudo-labels of unlabeled speech data segments using a speech recognition teacher reference model, and calculate the word error rate of the reference pseudo-label of each unlabeled speech data segment in the plurality of unlabeled speech data segments relative to the reference pseudo-label (i.e., execute S202). The training data pair construction unit 104 is configured to construct a plurality of unlabeled speech data-pseudo-label training data pairs based on a plurality of unlabeled speech data segments, a plurality of benchmark pseudo-labels of the unlabeled speech data segments, and a plurality of unlabeled speech data segments whose benchmark pseudo-labels have a word error rate relative to the benchmark pseudo-labels lower than a predetermined value, (i.e., execute S204); the recognition model training unit 106 is configured to train a speech recognition student model based on a plurality of unlabeled speech data-pseudo-label training data pairs and their corresponding word error rates (i.e., execute S206). Here, the word error rate corresponding to a certain unlabeled speech data-pseudo-label training data pair refers to the word error rate of the benchmark pseudo-label of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the benchmark pseudo-label.
[0019] In some embodiments, the pseudo-label generation unit 102 can be configured to generate baseline pseudo-labels for multiple unlabeled speech data segments by the following processing: by using a speech recognition teacher baseline model to recognize multiple unlabeled speech data segments, generating baseline speech recognition results for multiple unlabeled speech data segments; and by performing text correction on the baseline speech recognition results of multiple unlabeled speech data segments (for example, text correction based on a generative pre-trained transformer (GPT)), generating baseline pseudo-labels for multiple unlabeled speech data segments.
[0020] In some embodiments, the pseudo-label generation unit 102 can be configured to generate reference pseudo-labels for multiple unlabeled speech data segments by: generating reference speech recognition results for multiple unlabeled speech data segments by identifying multiple unlabeled speech data segments using a speech recognition teacher reference model; and generating reference pseudo-labels for multiple unlabeled speech data segments by performing text correction on the reference speech recognition results of the multiple unlabeled speech data segments (for example, GPT-based text correction).
[0021] In some embodiments, the training data pair construction unit 104 can be configured to construct multiple unlabeled speech data-pseudo-label training data pairs by the following processing: by using a speech data sliding window to perform cross-segment interception and splicing of multiple unlabeled speech data segments, generating multiple spliced unlabeled speech data segments; by using a pseudo-label sliding window corresponding to the speech data sliding window to perform cross-label interception and splicing of the reference pseudo-labels of multiple unlabeled speech data segments and the reference pseudo-labels of the unlabeled speech data segments whose error rate of the reference pseudo-labels relative to the reference pseudo-labels is lower than a predetermined value in the multiple unlabeled speech data segments, generating multiple spliced pseudo-labels; and based on the multiple spliced unlabeled speech data segments and the multiple spliced pseudo-labels, construct part or all of the multiple unlabeled speech data-pseudo-label training data pairs. Through the above processing, the diversity of training data can be increased, and the problem of high error rate caused by too short training data can be reduced.
[0022] In some embodiments, the reference pseudo-labels of multiple unlabeled speech data segments and the reference pseudo-labels of unlabeled speech data segments whose error rates of reference pseudo-labels relative to the reference pseudo-labels are lower than a predetermined value in the multiple unlabeled speech data segments can be collectively referred to as class truth pseudo-labels. The training data pair construction unit 104 can be configured to generate a spliced pseudo-label by the following process: using a pseudo-label sliding window of a predefined size (the size of the pseudo-label sliding window refers to the length of the text segment processed each time) to select a text segment from continuous class truth pseudo-labels as the initial text segment; check whether the selected text segment meets the requirements (for example, whether the segment length is greater than a predetermined length); if the requirements are met, the selected text segment is used as a spliced pseudo-label, otherwise the pseudo-label sliding window is slid backward to select the next text segment; repeat the above process until all class truth pseudo-labels are traversed. It should be understood that the training data pair construction unit 104 can also be configured to generate spliced unlabeled speech data segments using similar processing.
[0023] In some embodiments, the recognition model training unit 106 can be configured to update the model parameters of the speech recognition student model by the following processing during the process of training the speech recognition student model: for each unlabeled speech data-pseudo-label training data pair in a plurality of unlabeled speech data-pseudo-label training data pairs, generate a speech recognition result for the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair by using the speech recognition student model; and based on the word error rates corresponding to the plurality of unlabeled speech data-pseudo-label training data pairs, calculate the multi-task loss function of the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-label in the unlabeled speech data-pseudo-label training data pair, and use the calculated multi-task loss function to update the model parameters of the speech recognition student model.
[0024] In some embodiments, the recognition model training unit 106 can be configured to generate a speech recognition result of an unlabeled speech data segment in a plurality of unlabeled speech data-pseudo-label training data pairs by the following processing during the training of the speech recognition student model: extracting features of the unlabeled speech data segment X (e.g., a 16kHz speech sampling point set of the audio input) in the unlabeled speech data-pseudo-label training data pair by using a feature extractor in the speech recognition student model to generate a shallow acoustic feature vector Z of the unlabeled speech data segment X (e.g., Mel-Frequency Cepstral Coefficients (MFCC), Mel filter bank coefficients (FBank), linear prediction coefficients (Linear Prediction Coefficient, LPC) etc.); by using the acoustic encoder in the speech recognition student model to perform acoustic feature encoding on the shallow acoustic feature vector Z, a deep acoustic feature vector C of the unlabeled speech data segment X is generated; by using the decoder in the speech recognition student model to multiply each component in the deep acoustic feature vector C by the word error rate corresponding to the component, a weighted feature vector of the unlabeled speech data segment X is generated; and by using the decoder in the speech recognition student model to convert the weighted feature vector of the unlabeled speech data segment X into a speech recognition result of the unlabeled speech data segment X. It should be noted that, in the case where a certain unlabeled speech data-pseudo-label training data pair is constructed by concatenating unlabeled speech data segments and concatenated pseudo-labels, the word error rate corresponding to the unlabeled speech data-pseudo-label training data pair includes the word error rate of the control pseudo-label of each unlabeled speech data segment involved in the concatenated unlabeled speech data segment relative to the benchmark pseudo-label, and the word error rate corresponding to multiple components in the deep acoustic feature vector of the concatenated unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair may be the word error rate of the control pseudo-label of the same unlabeled speech segment involved in the concatenated unlabeled speech data segment relative to the benchmark pseudo-label.
[0025] In some embodiments, the recognition model training unit 106 can be configured to, during the process of training the speech recognition student model, for each unlabeled speech data-pseudo-label training data pair in the plurality of unlabeled speech data-pseudo-label training data pairs, calculate the multi-task loss function L of the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-label in the unlabeled speech data-pseudo-label training data pair by the following processing: M: Calculate the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the connection temporal classification loss L of the pseudo-label in the unlabeled speech data-pseudo-label training data pair CTC ; Based on the maximum and minimum word error rates among the error rates corresponding to multiple unlabeled speech data-pseudo-label training data pairs and the average of the error rates corresponding to multiple unlabeled speech data-pseudo-label training data pairs, calculate the word-level loss L of the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-label in the unlabeled speech data-pseudo-label training data pair AED ; and the connection temporal classification loss L based on the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-label in the unlabeled speech data-pseudo-label training data pair CTC and word-level loss L AED , calculate the multi-task loss function L of the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-label in the unlabeled speech data-pseudo-label training data pair M .For example, , where the parameter The value range of is (0,1). , WER mean is the average of the word error rates corresponding to multiple unlabeled speech data-pseudo-label training data pairs, WER max is the maximum word error rate among the word error rates corresponding to multiple unlabeled speech data-pseudo-label training data pairs, WER min It is the minimum word error rate among the word error rates corresponding to multiple unlabeled speech data-pseudo-label training data pairs.
[0026] In some embodiments, the recognition model training unit 106 can be configured to calculate the connection temporal classification loss L of the unlabeled speech data segment in any unlabeled speech data-pseudo-label training data pair relative to the pseudo-label in the unlabeled speech data-pseudo-label training data pair during the process of training the speech recognition student model by the following processing: CTC:By segmenting the speech recognition results of the unlabeled speech data segments in the unlabeled speech data-pseudo-label training data pair, a discrete symbol sequence T1 corresponding to the speech recognition results of the unlabeled speech data segments in the unlabeled speech data-pseudo-label training data pair is generated; by processing the discrete symbol sequence T1 using a word embedding vector, a recognition result tensor Y1 corresponding to the speech recognition results of the unlabeled speech data segments in the unlabeled speech data-pseudo-label training data pair is generated; by segmenting the pseudo-labels in the unlabeled speech data-pseudo-label training data pair, a discrete symbol sequence T2 corresponding to the pseudo-labels in the unlabeled speech data-pseudo-label training data pair is generated; by processing the discrete symbol sequence T2 using a word embedding vector, a pseudo-label tensor Y2 corresponding to the pseudo-labels in the unlabeled speech data-pseudo-label training data pair is generated; and a continuous temporal classification loss of the recognition result tensor T1 relative to the pseudo-label tensor T2 is calculated as a connection temporal classification loss L of the speech recognition results of the unlabeled speech data segments in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-labels in the unlabeled speech data-pseudo-label training data pair. CTC .
[0027] Figure 3 FIG. 1 is a schematic diagram showing a computer system that can implement the speech recognition model training apparatus and method according to an embodiment of the present invention. It should be understood that Figure 3 The computer system 300 shown is only an example and should not bring any limitation to the functions and scope of use of the speech recognition model training apparatus and method according to the embodiments of the present invention.
[0028] like Figure 3 As shown, the computer system 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303. Various programs and data required for the operation of the computer system 300 are also stored in the RAM 303. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.
[0029] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a camera, an accelerometer, a gyroscope, a sensor, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, a motor, an electronic speed controller, etc.; storage devices 308 including, for example, a flash memory (FlashCard), etc.; and communication devices 309. The communication devices 309 may allow the computer system 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The computer system 300 is shown with various devices, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.
[0030] In particular, according to some embodiments of the present invention, the process described above with reference to the flowchart may be implemented as a computer program. For example, a computer readable medium is provided on which a computer program is stored, the computer program comprising a method for executing Figure 2 The program code of the speech recognition model training method shown in the figure. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functional units defined in the speech recognition model training device according to the embodiment of the present invention are implemented.
[0031] It should be noted that the computer-readable medium according to the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium according to the embodiment of the present invention may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device, or device. In addition, the computer-readable signal medium according to the embodiment of the present invention may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0032] Computer program code for performing operations according to embodiments of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0033] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0034] The present invention can be implemented in other specific forms without departing from its spirit and essential features. For example, the algorithms described in the specific embodiments can be modified, and the system architecture does not depart from the basic spirit of the present invention. Therefore, the current embodiments are regarded as exemplary and non-restrictive in all aspects, and the scope of the present invention is defined by the appended claims rather than the above description, and all changes falling within the meaning and equivalent scope of the claims are thus included in the scope of the present invention.
Claims
1. A speech recognition model training method, include: Generate a plurality of benchmark pseudo-labels for unlabeled speech data segments using a speech recognition teacher benchmark model, generate a plurality of benchmark pseudo-labels for the plurality of unlabeled speech data segments using a speech recognition teacher comparison model, and calculate a word error rate of the comparison pseudo-label for each of the plurality of unlabeled speech data segments relative to the benchmark pseudo-label; Based on the multiple unlabeled speech data segments, the reference pseudo-labels of the multiple unlabeled speech data segments, and the reference pseudo-labels of the unlabeled speech data segments whose reference pseudo-labels have a word error rate lower than a predetermined value relative to the reference pseudo-labels in the multiple unlabeled speech data segments, a plurality of unlabeled speech data-pseudo-label training data pairs are constructed, wherein the process of constructing the multiple unlabeled speech data-pseudo-label training data pairs includes: performing cross-segment interception and splicing of the multiple unlabeled speech data segments by using a speech data sliding window to generate a plurality of spliced unlabeled speech data segments, performing cross-label interception and splicing of the reference pseudo-labels of the multiple unlabeled speech data segments and the reference pseudo-labels of the unlabeled speech data segments whose reference pseudo-labels have a word error rate lower than the predetermined value relative to the reference pseudo-labels by using a pseudo-label sliding window corresponding to the speech data sliding window to generate a plurality of spliced pseudo-labels, and constructing part or all of the multiple unlabeled speech data-pseudo-label training data pairs based on the multiple spliced unlabeled speech data segments and the multiple spliced pseudo-labels; and Based on the multiple unlabeled speech data-pseudo-labeled training data pairs and their corresponding word error rates, a speech recognition student model is trained.
2. The speech recognition model training method according to claim 1, in, The process of training the speech recognition student model includes, for each unlabeled speech data-pseudo-labeled training data pair in the plurality of unlabeled speech data-pseudo-labeled training data pairs: Generate a speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair by using the speech recognition student model to recognize the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair; as well as Based on the word error rates corresponding to the multiple unlabeled speech data-pseudo-label training data pairs, a multi-task loss function of the speech recognition results of the unlabeled speech data segments in the unlabeled speech data-pseudo-label training data pairs relative to the pseudo-labels in the unlabeled speech data-pseudo-label training data pairs is calculated, and the model parameters of the speech recognition student model are updated using the calculated multi-task loss function.
3. The speech recognition model training method according to claim 2, in, The process of calculating the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the multi-task loss function of the pseudo-label in the unlabeled speech data-pseudo-label training data pair includes: Calculate the connection temporal classification loss of the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-label in the unlabeled speech data-pseudo-label training data pair; Based on the maximum word error rate and the minimum word error rate among the word error rates corresponding to the multiple unlabeled speech data-pseudo-label training data pairs and the average of the word error rates corresponding to the multiple unlabeled speech data-pseudo-label training data pairs, calculate the word level loss of the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-label in the unlabeled speech data-pseudo-label training data pair; and Based on the connection temporal classification loss and word level loss of the speech recognition results of the unlabeled speech data segments in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-labels in the unlabeled speech data-pseudo-label training data pair, a multi-task loss function of the speech recognition results of the unlabeled speech data segments in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-labels in the unlabeled speech data-pseudo-label training data pair is calculated.
4. The speech recognition model training method according to claim 3, in, The process of calculating the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the connection temporal classification loss of the pseudo-label in the unlabeled speech data-pseudo-label training data pair includes: By segmenting the speech recognition results of the unlabeled speech data segments in the unlabeled speech data-pseudo-label training data pair, a discrete symbol sequence corresponding to the speech recognition results of the unlabeled speech data segments in the unlabeled speech data-pseudo-label training data pair is generated; The recognition result tensor corresponding to the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair is generated by processing the discrete symbol sequence corresponding to the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair by using the word embedding vector; By segmenting the pseudo-labels in the unlabeled speech data-pseudo-label training data pair, a discrete symbol sequence corresponding to the pseudo-labels in the unlabeled speech data-pseudo-label training data pair is generated; Generate a pseudo-label tensor corresponding to the pseudo-label in the unlabeled speech data-pseudo-label training data pair by processing the discrete symbol sequence corresponding to the pseudo-label in the unlabeled speech data-pseudo-label training data pair using the word embedding vector; and Calculate the continuous temporal classification loss of the recognition result tensor corresponding to the speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair relative to the pseudo-label tensor corresponding to the pseudo-label in the unlabeled speech data-pseudo-label training data pair.
5. The speech recognition model training method according to claim 1, in, The process of generating the reference pseudo labels for the plurality of unlabeled speech data segments includes: Generate benchmark speech recognition results for the plurality of unlabeled speech data segments by using the speech recognition teacher benchmark model to recognize the plurality of unlabeled speech data segments; By performing text error correction on the reference speech recognition results of the plurality of unlabeled speech data segments, reference pseudo labels of the plurality of unlabeled speech data segments are generated.
6. The speech recognition model training method according to claim 1, in, The process of generating the comparison pseudo labels for the plurality of unlabeled speech data segments includes: Generate comparative speech recognition results of the plurality of unlabeled speech data segments by using the speech recognition teacher comparative model to recognize the plurality of unlabeled speech data segments; By performing text error correction on the comparison speech recognition results of the plurality of unlabeled speech data segments, comparison pseudo labels of the plurality of unlabeled speech data segments are generated.
7. The speech recognition model training method according to claim 1, in, The process of training the speech recognition student model includes, for each unlabeled speech data-pseudo-labeled training data pair in the plurality of unlabeled speech data-pseudo-labeled training data pairs: Extracting features of the unlabeled speech data segments in the unlabeled speech data-pseudo-labeled training data pair by using a feature extractor in the speech recognition student model to generate shallow acoustic feature vectors of the unlabeled speech data segments in the unlabeled speech data-pseudo-labeled training data pair; By using the acoustic encoder in the speech recognition student model to perform acoustic feature encoding on the shallow acoustic feature vector of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair, a deep acoustic feature vector of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair is generated; Generate a weighted feature vector of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair by multiplying each component in the deep acoustic feature vector of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair by the word error rate corresponding to the component using the decoder in the speech recognition student model; as well as The decoder in the speech recognition student model is used to convert the weighted feature vector of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair into a speech recognition result of the unlabeled speech data segment in the unlabeled speech data-pseudo-label training data pair.
8. A speech recognition model training device, include: processor; as well as A memory having computer executable instructions stored thereon, wherein the computer executable instructions, when executed by the processor, prompt the processor to execute the speech recognition model training method described in any one of claims 1 to 7.
9. A computer-readable storage medium having computer-executable instructions stored thereon, in, When the computer executable instructions are executed by a processor, they cause the processor to execute the speech recognition model training method described in any one of claims 1 to 7.
10. A computer program product comprising computer executable instructions, in, When the computer executable instructions are executed by a processor, they cause the processor to execute the speech recognition model training method described in any one of claims 1 to 7.
Citation Information
Patent Citations
Voice recognition acoustic model training method, system and device and medium
CN111883110A
Model generation method and device, speech recognition method and device, medium and equipment
CN116072108A
