Speech recognition model training method, speech recognition method, device and storage medium

Abstract

The present disclosure relates to a speech recognition model training method, speech recognition method, device and storage medium. The method comprises: obtaining training audio; determining a first target pseudo-label of the training audio based on the acoustic information and semantic information of the training audio, the first target pseudo-label indicating the target text corresponding to the training audio; training an initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model. According to an embodiment of the present disclosure, by obtaining unlabeled training audio and determining the first target pseudo-label of the training audio based on the acoustic information and semantic information of the training audio, labels with more reasonable semantics and greater training value can be selected, thereby training the initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model, which can save the labeling cost while improving the effect of model training and speech recognition, and obtain a speech recognition model with higher accuracy.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a speech recognition model training method, a speech recognition method, a device, and a storage medium. Background Art

[0002] Speech recognition technology is widely used in various fields, but its performance is limited by the quality and diversity of training data. Current speech recognition methods often need to cope with complex situations such as various accents, speaking speeds, and noise. Therefore, accurate, diverse, and sufficient training data is crucial for building high-performance speech recognition systems. Furthermore, for different scenarios, data related to these scenarios must be labeled to improve the model's recognition capabilities in a specific vertical field.

[0003] In current speech recognition technology, building high-quality training datasets often requires extensive manual annotation. However, the high human and time costs associated with annotation hinder the further development and application of speech recognition technology. In reality, a vast amount of unlabeled, rich audio exists, and leveraging this unlabeled audio to improve speech recognition is crucial. Summary of the Invention

[0004] In view of this, the present disclosure proposes a speech recognition model training method, a speech recognition method, a device and a storage medium.

[0005] According to one aspect of the present disclosure, a method for training a speech recognition model is provided. The method comprises:

[0006] Get training audio;

[0007] Determining a first target pseudo label of the training audio based on acoustic information and semantic information of the training audio, where the first target pseudo label indicates a target text corresponding to the training audio;

[0008] An initial speech recognition model is trained based on the training audio and the first target pseudo label to obtain a trained speech recognition model.

[0009] In one possible implementation, the acoustic information includes a first confidence level and a second confidence level, and the semantic information includes a third confidence level. The method further includes:

[0010] Inputting the training audio into the initial speech recognition model, obtaining a plurality of candidate pseudo-labels, and a first confidence and a second confidence respectively associated with the plurality of candidate pseudo-labels, wherein the candidate pseudo-labels indicate candidate texts corresponding to the training audio, the first confidence indicates the possibility that the phoneme sequence corresponding to the associated candidate pseudo-label is the phoneme sequence of the target pseudo-label, and the second confidence indicates the possibility that the pinyin sequence corresponding to the associated candidate pseudo-label is the pinyin sequence of the target pseudo-label;

[0011] The plurality of candidate pseudo-labels are respectively input into the large language model to obtain third confidences respectively associated with the plurality of candidate pseudo-labels, where the third confidences represent the semantic fluency of the associated candidate pseudo-labels.

[0012] In a possible implementation, the method further includes:

[0013] Get scene information of training audio;

[0014] Inputting multiple candidate pseudo labels into the large language model respectively, obtaining third confidences associated with the multiple candidate pseudo labels respectively, including:

[0015] Multiple candidate pseudo labels are concatenated with the scene information of the training audio and then input into the large language model to obtain third confidence levels associated with the multiple candidate pseudo labels.

[0016] In one possible implementation, determining a first target pseudo-label for the training audio according to acoustic information and semantic information of the training audio includes:

[0017] For any candidate pseudo-label, calculate the total confidence of the candidate pseudo-label according to the first confidence, the second confidence and the third confidence associated with the candidate pseudo-label;

[0018] A first target pseudo label is determined from the plurality of candidate pseudo labels according to the total confidence of each of the plurality of candidate pseudo labels.

[0019] In a possible implementation, the method further includes:

[0020] Perform masking on the frequency domain graph of the training audio to obtain a masked frequency domain graph;

[0021] Performing random dropout processing on a predetermined proportion of neurons in the initial speech recognition model, and training the initial speech recognition model after the random dropout processing based on the masked frequency domain map corresponding to the training audio and the first target pseudo label of the training audio to obtain a first intermediate speech recognition model;

[0022] The initial speech recognition model is updated with the first intermediate speech recognition model.

[0023] In one possible implementation, training an initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model includes:

[0024] Based on the training audio and the first target pseudo label, the initial speech recognition model is iteratively trained until the text change output by the initial speech recognition model is less than a preset threshold, and the iterative training is stopped to obtain the trained speech recognition model.

[0025] In one possible implementation, training an initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model includes:

[0026] Post-processing the first target pseudo label using a large language model to obtain a second target pseudo label;

[0027] Based on the training audio and the second target pseudo label, the initial speech recognition model is fine-tuned to obtain a trained speech recognition model.

[0028] In one possible implementation, post-processing includes one or more of the following:

[0029] Punctuation marks that are consistent with semantics are added to the first target pseudo-label, named entities in the first target pseudo-label are identified and named entity identifiers are added to the first target pseudo-label, and text smoothing is performed on the first target pseudo-label.

[0030] In a possible implementation, the method further includes:

[0031] Add preset tokens associated with post-processing to the vocabulary of the initial speech recognition model.

[0032] According to one aspect of the present disclosure, a speech recognition method is provided. The method comprises:

[0033] Get the audio to be recognized;

[0034] Input the audio to be recognized into the trained speech recognition model to obtain the text corresponding to the audio to be recognized;

[0035] The trained speech recognition model is obtained based on the above-mentioned speech recognition model training method.

[0036] In a possible implementation, the method further includes:

[0037] In response to a preset configuration, the vocabulary of the trained speech recognition model is processed so that the trained speech recognition model turns on or off one or more of the following functions: output punctuation, text smoothing, and named entity recognition.

[0038] According to another aspect of the present disclosure, a speech recognition model training device is provided. The device includes:

[0039] A first acquisition module is used to acquire training audio;

[0040] A first determination module is configured to determine a first target pseudo label of the training audio based on acoustic information and semantic information of the training audio, where the first target pseudo label indicates a target text corresponding to the training audio;

[0041] The first training module is used to train an initial speech recognition model based on the training audio and the first target pseudo label to obtain a trained speech recognition model.

[0042] In a possible implementation, the acoustic information includes a first confidence level and a second confidence level, the semantic information includes a third confidence level, and the apparatus further includes:

[0043] a third determination module, configured to input the training audio into the initial speech recognition model to obtain a plurality of candidate pseudo-labels, and a first confidence level and a second confidence level respectively associated with the plurality of candidate pseudo-labels, wherein the candidate pseudo-labels indicate candidate text corresponding to the training audio, the first confidence level indicates the likelihood that the phoneme sequence corresponding to the associated candidate pseudo-label is the phoneme sequence of the target pseudo-label, and the second confidence level indicates the likelihood that the pinyin sequence corresponding to the associated candidate pseudo-label is the pinyin sequence of the target pseudo-label;

[0044] The fourth determination module is configured to input the plurality of candidate pseudo-labels into the large language model respectively to obtain third confidences respectively associated with the plurality of candidate pseudo-labels, where the third confidences represent the semantic fluency of the associated candidate pseudo-labels.

[0045] In a possible implementation, the device further includes:

[0046] The third acquisition module is used to obtain scene information of the training audio;

[0047] The fourth determining module is configured to:

[0048] Multiple candidate pseudo labels are concatenated with the scene information of the training audio and then input into the large language model to obtain third confidence levels associated with the multiple candidate pseudo labels.

[0049] In a possible implementation, the first determining module is configured to:

[0050] For any candidate pseudo-label, calculate the total confidence of the candidate pseudo-label according to the first confidence, the second confidence and the third confidence associated with the candidate pseudo-label;

[0051] A first target pseudo label is determined from the plurality of candidate pseudo labels according to the total confidence of each of the plurality of candidate pseudo labels.

[0052] In a possible implementation, the device further includes:

[0053] A mask processing module is used to perform mask processing on the frequency domain image of the training audio to obtain a masked frequency domain image;

[0054] A second training module is configured to perform random deactivation on a predetermined proportion of neurons in the initial speech recognition model, and train the initial speech recognition model after the random deactivation based on the masked frequency domain map corresponding to the training audio and the first target pseudo-label of the training audio to obtain a first intermediate speech recognition model;

[0055] An updating module is used to update the initial speech recognition model with the first intermediate speech recognition model.

[0056] In a possible implementation, the first training module is configured to:

[0057] Based on the training audio and the first target pseudo label, the initial speech recognition model is iteratively trained until the text change output by the initial speech recognition model is less than a preset threshold, and the iterative training is stopped to obtain the trained speech recognition model.

[0058] In a possible implementation, the first training module is configured to:

[0059] Post-processing the first target pseudo label using a large language model to obtain a second target pseudo label;

[0060] Based on the training audio and the second target pseudo label, the initial speech recognition model is fine-tuned to obtain a trained speech recognition model.

[0061] In one possible implementation, post-processing includes one or more of the following:

[0062] Punctuation marks that are consistent with semantics are added to the first target pseudo-label, named entities in the first target pseudo-label are identified and named entity identifiers are added to the first target pseudo-label, and text smoothing is performed on the first target pseudo-label.

[0063] In a possible implementation, the device further includes:

[0064] The token adding module is used to add preset tokens associated with post-processing to the vocabulary of the initial speech recognition model.

[0065] According to another aspect of the present disclosure, a speech recognition device is provided. The device includes:

[0066] A second acquisition module is used to acquire the audio to be recognized;

[0067] The second determination module is used to input the audio to be recognized into the trained speech recognition model to obtain the text corresponding to the audio to be recognized;

[0068] The trained speech recognition model is obtained based on the above-mentioned speech recognition model training method.

[0069] In a possible implementation, the device further includes:

[0070] The vocabulary processing module is used to process the vocabulary of the trained speech recognition model in response to a preset configuration so that the trained speech recognition model can turn on or off one or more of the following functions: output punctuation, text smoothing, and named entity recognition.

[0071] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0072] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0073] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0074] According to the embodiment of the present disclosure, by obtaining unlabeled training audio and determining the first target pseudo-label of the training audio based on the acoustic information and semantic information of the training audio, labels with more reasonable semantics and greater training value can be selected, thereby training the initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model. This can save the labeling cost while improving the effect of model training and speech recognition, and obtain a speech recognition model with higher accuracy.

[0075] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0077] Figure 1 A schematic diagram illustrating an application scenario according to an embodiment of the present disclosure.

[0078] Figure 2 A flowchart of a speech recognition model training method according to an embodiment of the present disclosure is shown.

[0079] Figure 3 A flowchart of a speech recognition model training method according to an embodiment of the present disclosure is shown.

[0080] Figure 4 A schematic structural diagram of an RNN-T model according to an embodiment of the present disclosure is shown.

[0081] Figure 5 A flowchart of a speech recognition model training method according to an embodiment of the present disclosure is shown.

[0082] Figure 6 A flowchart of a speech recognition method according to an embodiment of the present disclosure is shown.

[0083] Figure 7 A structural diagram of a speech recognition model training device according to an embodiment of the present disclosure is shown.

[0084] Figure 8 A structural diagram of a speech recognition device according to an embodiment of the present disclosure is shown.

[0085] Figure 9 is a block diagram of an electronic device 1900 according to an exemplary embodiment. DETAILED DESCRIPTION

[0086] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0087] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0088] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0089] Speech recognition technology is widely used in multiple fields, but its performance is limited by the quality and diversity of training data. Current speech recognition methods often need to cope with complex situations such as various accents, speaking speeds, and noise. Therefore, accurate, diverse, and sufficient training data is crucial for building high-performance speech recognition systems. Moreover, for different scenarios, data related to these scenarios also needs to be labeled to improve the model's recognition capabilities in a specific vertical field. In current speech recognition technology, building high-quality training datasets often requires a large amount of manual labeling work. However, the process of labeling speech data is costly in terms of human resources and time, which restricts the further development and application of speech recognition technology. In real life, there is actually a large amount of unlabeled and rich audio. How to fully utilize this unlabeled audio to improve speech recognition performance is extremely important.

[0090] In view of this, the present disclosure provides a speech recognition model training method, speech recognition method, device, and storage medium. The speech recognition model training method of the embodiments of the present disclosure obtains unlabeled training audio, determines a first target pseudo-label for the training audio based on the acoustic and semantic information of the training audio, and selects labels with more reasonable semantics and greater training value. The trained speech recognition model is then trained based on the training audio and the first target pseudo-label to obtain a trained speech recognition model. This method can improve the effectiveness of model training and speech recognition while saving labeling costs, thereby obtaining a more accurate speech recognition model.

[0091] Figure 1 The schematic diagram of an application scenario according to an embodiment of the present disclosure is shown. The embodiment of the present disclosure can be applied to a scenario where a speech recognition system is used for speech recognition, such as Figure 1 The speech recognition system shown can deploy a trained speech recognition model obtained based on the embodiment of the present disclosure. The speech recognition system can obtain the audio to be recognized in relevant scenarios (such as a graphics processing unit (GPU), graphics card, smart hardware, etc.), use the trained speech recognition model to recognize the audio to be recognized, and output the text corresponding to the audio to be recognized.

[0092] The speech recognition system can be deployed on a terminal device or server. The terminal device can be any one or more of a mobile phone, a foldable electronic device, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), or an in-vehicle device. The embodiments of the present disclosure do not impose any particular restrictions on the specific type of terminal device, which can have wired or wireless communication capabilities.

[0093] The server can be located locally or in the cloud, and can be a physical device or a virtual device, such as a virtual machine or container, and has wireless communication capabilities, wherein the wireless communication capabilities can be set in the chip (system) or other parts or components of the server. The wireless communication function can be implemented, for example, through mobile communication technologies such as 2G / 3G / 4G / 5G, as well as Wi-Fi, Bluetooth, frequency modulation (FM), digital radio, satellite communication, etc. Communication can also be carried out through wired connections to achieve interaction with other devices.

[0094] The following Figures 2 to 5 First, the speech recognition model training method of the embodiment of the present disclosure is introduced.

[0095] Figure 2 A flow chart of a method for training a speech recognition model according to an embodiment of the present disclosure is shown. The method can be used in the above-mentioned speech recognition system, such as Figure 2 As shown, the method includes:

[0096] Step S201: Obtain training audio.

[0097] The training audio can be unlabeled training audio. The training audio can be predetermined (for example, accumulated by business or crawled online) and can be associated with a preset scenario (such as a GPU, graphics card, smart hardware, etc.).

[0098] Figure 3 FIG. 1 is a flow chart showing a method for training a speech recognition model according to an embodiment of the present disclosure. Figure 3 As shown, after step S201 and before step S202, the method may further include:

[0099] Step S301: Obtain scene information of training audio.

[0100] The scene information may include, for example, the source information of the training audio and the scene associated with the training audio (such as GPU, graphics card, smart hardware, etc.). i The corresponding scene information can be called P i .

[0101] Optionally, the method may further include:

[0102] Step S302: Input the training audio into the initial speech recognition model to obtain a plurality of candidate pseudo labels, and a first confidence level and a second confidence level respectively associated with the plurality of candidate pseudo labels.

[0103] The initial speech recognition model may be a basic speech recognition model obtained by pre-training using existing labeled audio, for example, an RNN-T (recurrent neural network transducer). Figure 4 FIG. 1 shows a schematic diagram of the structure of the RNN-T model according to an embodiment of the present disclosure. Figure 4 As shown, the RNN-T model can include a shallow decoder (shallow encoder), a deep decoder (deep encoder), a prediction network (prediction network), and a joint network (joint network).

[0104] Among them, the input of the shallow encoder can be the training audio X i , the output can be X i The corresponding N candidate phoneme sequences and the first confidence {P ce1 ,P ce2 …,P cej ,…,P ceN}, the first confidence level indicates the possibility that each candidate phoneme sequence is the phoneme sequence corresponding to the target pseudo label;

[0105] The input of the deep encoder can be X i The corresponding N phoneme sequence, the output can be X i The corresponding N candidate pinyin sequences and the second confidence {P ctc1 ,P ctc2 …,P ctcj ,…,P ctcN}, the second confidence level indicates the possibility that each candidate pinyin sequence is the pinyin sequence corresponding to the target pseudo-label;

[0106] The input of the prediction network can include the prediction result of the initial speech recognition model of the previous frame, and the output can include X i The conditional probability distribution of the corresponding target pseudo label;

[0107] The input of the joint network can be the output of the deep encoder and the output of the prediction network, and the output can include X i The joint conditional probability distribution of the corresponding target pseudo label, where the above conditional probability distribution and the joint conditional probability distribution both represent the probability that each candidate pseudo label in the output result is the correct label. Based on the joint conditional probability distribution, the softmax function is used to process the training audio X i N candidate pseudo labels {Y i1 ,Yi2 ,…,Y ij ,…,Y iN}, Y ij Respectively with P cej 、P ctcj The candidate pseudo-label may indicate the candidate text corresponding to the training audio. For example, the candidate pseudo-label may be the candidate text of the training audio itself.

[0108] It should be noted that in the process of pre-training the basic speech recognition model using existing annotated audio, the shallow encoder can calculate the cross-entropy loss (CE Loss) based on the output phoneme sequence for training; the deep encoder can calculate the CTC (connectionist temporal classification) Loss based on the output pinyin sequence for training; the joint network can calculate the RNN-T Loss based on the output text sequence (ie, candidate pseudo-label) for training.

[0109] Step S303: Input the plurality of candidate pseudo labels into the large language model respectively to obtain third confidences respectively associated with the plurality of candidate pseudo labels.

[0110] The large language model (LLM) may be a pre-obtained model (such as Llama2, etc.) The third confidence level may represent the semantic fluency of the corresponding candidate pseudo-label.

[0111] For example, you can i N candidate pseudo labels {Y i1 ,Y i2 ,…,Y ij ,…,Y iN} Input LLM and get the third confidence {L s1 ,L s2 ,…,L sj ,…,L sN}, the method for determining the third confidence level can be implemented based on existing technology.

[0112] In order to make the LLM consider the scene to which the training audio belongs when determining the third confidence level, so as to select a target pseudo-label whose semantics is more consistent with the corresponding scene, in step S303, the following steps may be performed:

[0113] Multiple candidate pseudo labels are concatenated with the scene information of the training audio and then input into the large language model to obtain third confidence levels associated with the multiple candidate pseudo labels.

[0114] The scene information of the training audio can be the scene information obtained in step S301 above.

[0115] For example, you can i N candidate pseudo labels {Y i1 ,Y i2 ,…,Y ij ,…,Y iN} respectively with P i After splicing, the data are input into LLM to determine the third confidence level corresponding to each of the N candidate pseudo labels.

[0116] After determining the first confidence level, the second confidence level, and the third confidence level, you can return to Figure 2 , the method comprising:

[0117] Step S202: Determine a first target pseudo label of the training audio based on the acoustic information and semantic information of the training audio.

[0118] The first target pseudo-label may indicate the target text corresponding to the training audio. The acoustic information may be information output by the initial speech recognition model, and the semantic information may be information output by the large language model.

[0119] The acoustic information may include a first confidence level corresponding to a phoneme sequence and a second confidence level corresponding to a pinyin sequence, and the semantic information may include a third confidence level. i The acoustic information may include the first confidence {P ce1 ,P ce2 …,P cej ,…,P ceN} and the second confidence {P ctc1 ,P ctc2 …,P ctcj ,…,P ctcN}, training audio X i The semantic information may include the third confidence {L s1 ,L s2 ,…,L sj ,…,L sN}.

[0120] In order to select target pseudo labels with more reasonable semantics and greater training value, in step S202, you can:

[0121] For any candidate pseudo-label, calculate the total confidence of the candidate pseudo-label according to the first confidence, the second confidence and the third confidence associated with the candidate pseudo-label;

[0122] A first target pseudo label is determined from the plurality of candidate pseudo labels according to the total confidence of each of the plurality of candidate pseudo labels.

[0123] For example, the first, second, and third confidences associated with the candidate pseudo-label can be weighted and summed using preset weights to calculate the total confidence of the candidate pseudo-label. The preset weights corresponding to the first, second, and third confidences are, for example, 1 / 3. The candidate label with the highest total confidence among the multiple candidate pseudo-labels can be used as the first target pseudo-label.

[0124] Therefore, it is possible to clean and filter data by combining acoustic information and semantic information at the same time.

[0125] After obtaining the first target pseudo label, the first target pseudo label can be used as the label of the training audio during the training process. First, the initial speech recognition model is updated. Figure 5 , shows a flow chart of a method for training a speech recognition model according to an embodiment of the present disclosure. Figure 5 As shown, step S203 may include:

[0126] Step S501 : Masking is performed on the frequency domain image of the training audio to obtain a masked frequency domain image.

[0127] The frequency domain graph of the training audio can be obtained based on existing technology. The frequency domain graph of the training audio can be masked in both the time domain and the frequency domain. In the frequency domain dimension, it can include masking the frequency of the training audio, for example, the frequency below or above a certain frequency threshold can be set to zero, or the amplitude of the training audio at the corresponding frequency can be adjusted by multiplying by a mask matrix. The masking of the time domain dimension can be achieved by masking the frequency domain graph of the training audio for different time periods (for example, masking the frequency domain graph of a certain time period).

[0128] The masked frequency domain image and the first target pseudo label are then used to train the initial speech recognition model.

[0129] Masking can be used to achieve speech enhancement, for example, by enhancing the audio signal within a specific frequency range to improve the quality and clarity of speech.

[0130] Step S502: Perform random inactivation on a predetermined proportion of neurons in the initial speech recognition model, and train the initial speech recognition model after random inactivation based on the masked frequency domain image corresponding to the training audio and the first target pseudo-label of the training audio to obtain a first intermediate speech recognition model.

[0131] For example, random dropout can be used during training to randomly set a portion of neurons to zero at a predetermined ratio to avoid overfitting. Iterative training can be performed using a noisy student training method, during which parameters are updated using a preset loss function and optimizer. When the text output by the model converges, i.e., when the change is less than a preset threshold, training can be stopped to obtain a first intermediate language recognition model.

[0132] Step S503: Update the initial speech recognition model with the first intermediate speech recognition model.

[0133] The obtained first intermediate speech recognition model replaces the initial speech recognition model.

[0134] Since the first target pseudo-label may differ from the actual text corresponding to the training audio, the first target pseudo-label can be updated using the updated initial speech recognition model, and training can be performed again based on the updated initial speech recognition model. Steps S302, S303, and S202 can be re-executed to update the first target pseudo-label.

[0135] Step S203: Train an initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model.

[0136] The initial speech recognition model may be the initial speech recognition model obtained by updating the initial speech recognition model with the first intermediate speech recognition model. Correspondingly, the first target pseudo-label may be the updated first target pseudo-label.

[0137] According to the embodiment of the present disclosure, by obtaining unlabeled training audio and determining the first target pseudo-label of the training audio based on the acoustic information and semantic information of the training audio, labels with more reasonable semantics and greater training value can be selected, thereby training the initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model. This can save the labeling cost while improving the effect of model training and speech recognition, and obtain a speech recognition model with higher accuracy.

[0138] In step S203, you can:

[0139] Based on the training audio and the first target pseudo label, the initial speech recognition model is iteratively trained until the text change output by the initial speech recognition model is less than a preset threshold, and the iterative training is stopped to obtain the trained speech recognition model.

[0140] During iterative training of the initial speech recognition model, masking of the frequency domain graph of the training audio and random dropout of the initial speech recognition model may not be performed. The initial speech recognition model may be iteratively trained using a noisy student training method, with parameters updated using a preset loss function and optimizer.

[0141] In order to enable the trained speech recognition model to have specific functions, such as enabling the trained speech recognition model to output punctuation marks, improve text readability, or perform named entity recognition, further fine-tuning training can be performed based on the initial speech recognition model to obtain a trained speech recognition model. This fine-tuning training can be performed in the last iteration of the above iterative training process. In step S203, you can:

[0142] The first target pseudo label is post-processed using a large language model to obtain a second target pseudo label; the initial speech recognition model is fine-tuned based on the training audio and the second target pseudo label to obtain a trained speech recognition model.

[0143] The first target pseudo label may be the updated first target pseudo label.

[0144] Post-processing may include one or more of the following:

[0145] Punctuation marks that are consistent with semantics are added to the first target pseudo-label, named entities in the first target pseudo-label are identified and named entity identifiers are added to the first target pseudo-label, and text smoothing is performed on the first target pseudo-label.

[0146] Semantically correct punctuation marks may be {,.?!}, etc.).

[0147] Named entities can include categories such as time, place, and person's name, and named entity identifiers can be used to indicate entities of the corresponding category. For example, the named entity identifier corresponding to time can be "<time start>" or "<time end>", which can be used to indicate the beginning and end of a named entity of the time category, respectively; the named entity identifier corresponding to place can be "<place start>" or "<place end>", which can be used to indicate the beginning and end of a named entity of the place category, respectively; the named entity identifier corresponding to a person's name can be "<person start>" or "<person end>", which can be used to indicate the beginning and end of a named entity of the person category, respectively.

[0148] Smoothing can be used to increase the readability of text, for example, by converting Chinese numerals in the text into Arabic numerals.

[0149] Fine-tuning can be performed on the same large language model to achieve different post-processing of the first target pseudo-label.

[0150] For example, the first target pseudo-label is "How is the weather in Beijing at 3 pm?". After adding semantically correct punctuation to the first target pseudo-label, the second target pseudo-label obtained can be "How is the weather in Beijing at 3 pm?"; after identifying the named entities in the first target pseudo-label and adding named entity identifiers to the first target pseudo-label, the second target pseudo-label obtained can be "How is the weather in Beijing at 3 pm?"; after performing text smoothing on the first target pseudo-label, the second target pseudo-label obtained can be "How is the weather in Beijing at 3 pm?"

[0151] In the process of fine-tuning the initial speech recognition model, the method may further include:

[0152] Add pre-set tokens (special tokens) associated with post-processing to the vocabulary of the initial speech recognition model.

[0153] The vocabulary can be a collection of all possible words or phrases that the model can recognize and understand. The preset tokens associated with adding semantically consistent punctuation marks to the first target pseudo-label may include corresponding punctuation marks, such as {,. ? !}, etc.; the preset tokens associated with identifying named entities in the first target pseudo-label and adding named entity identifiers to the first target pseudo-label may include named entity identifiers; the preset tokens associated with text smoothing of the first target pseudo-label may include Arabic numerals. The preset tokens can also be indicated by other symbols in the vocabulary. By adding preset tokens associated with post-processing, relevant information can be learned during the fine-tuning training of the initial speech recognition model.

[0154] Therefore, the initial speech recognition model is fine-tuned based on the training audio and the second target pseudo-label. The trained speech recognition model can have functions that match different post-processing, such as output punctuation, text smoothing, named entity recognition, etc.

[0155] After the trained speech recognition model is obtained, speech recognition can be implemented based on the model. The embodiment of the present disclosure also proposes a speech recognition method, see Figure 6 , shows a flow chart of a speech recognition method according to an embodiment of the present disclosure. The method can be used in the above-mentioned speech recognition system, such as Figure 6 As shown, the method may include:

[0156] Step S601: Acquire audio to be recognized.

[0157] Step S602: input the audio to be recognized into the trained speech recognition model to obtain the text corresponding to the audio to be recognized.

[0158] The trained speech recognition model can be obtained based on the above-mentioned speech recognition model training method.

[0159] In this way, the effect of speech recognition can be improved.

[0160] The method may further include:

[0161] In response to a preset configuration, the vocabulary of the trained speech recognition model is processed so that the trained speech recognition model turns on or off one or more of the following functions: output punctuation, text smoothing, and named entity recognition.

[0162] Wherein, a user may pre-configure whether to turn functions on or off. In the case where one or more functions are turned off in response to the pre-configured configuration, processing the vocabulary of the trained speech recognition model may include masking tokens in the vocabulary associated with the one or more functions.

[0163] Figure 7 FIG. 1 shows a structural diagram of a speech recognition model training device according to an embodiment of the present disclosure. Figure 7 As shown, the device includes:

[0164] A first acquisition module 701 is used to acquire training audio;

[0165] A first determining module 702 is configured to determine a first target pseudo label of the training audio based on acoustic information and semantic information of the training audio, where the first target pseudo label indicates a target text corresponding to the training audio;

[0166] The first training module 703 is used to train the initial speech recognition model based on the training audio and the first target pseudo label to obtain a trained speech recognition model.

[0167] In a possible implementation, the acoustic information includes a first confidence level and a second confidence level, the semantic information includes a third confidence level, and the apparatus further includes:

[0168] a third determination module, configured to input the training audio into the initial speech recognition model to obtain a plurality of candidate pseudo-labels, and a first confidence level and a second confidence level respectively associated with the plurality of candidate pseudo-labels, wherein the candidate pseudo-labels indicate candidate text corresponding to the training audio, the first confidence level indicates the likelihood that the phoneme sequence corresponding to the associated candidate pseudo-label is the phoneme sequence of the target pseudo-label, and the second confidence level indicates the likelihood that the pinyin sequence corresponding to the associated candidate pseudo-label is the pinyin sequence of the target pseudo-label;

[0169] The fourth determination module is configured to input the plurality of candidate pseudo-labels into the large language model respectively to obtain third confidences respectively associated with the plurality of candidate pseudo-labels, where the third confidences represent the semantic fluency of the associated candidate pseudo-labels.

[0170] In a possible implementation, the device further includes:

[0171] The third acquisition module is used to obtain scene information of the training audio;

[0172] The fourth determining module is configured to:

[0173] Multiple candidate pseudo labels are concatenated with the scene information of the training audio and then input into the large language model to obtain third confidence levels associated with the multiple candidate pseudo labels.

[0174] In a possible implementation, the first determining module 702 is configured to:

[0175] For any candidate pseudo-label, calculate the total confidence of the candidate pseudo-label according to the first confidence, the second confidence and the third confidence associated with the candidate pseudo-label;

[0176] A first target pseudo label is determined from the plurality of candidate pseudo labels according to the total confidence of each of the plurality of candidate pseudo labels.

[0177] In a possible implementation, the device further includes:

[0178] A mask processing module is used to perform mask processing on the frequency domain image of the training audio to obtain a masked frequency domain image;

[0179] A second training module is configured to perform random deactivation on a predetermined proportion of neurons in the initial speech recognition model, and train the initial speech recognition model after the random deactivation based on the masked frequency domain map corresponding to the training audio and the first target pseudo-label of the training audio to obtain a first intermediate speech recognition model;

[0180] An updating module is used to update the initial speech recognition model with the first intermediate speech recognition model.

[0181] In a possible implementation, the first training module 703 is configured to:

[0182] Based on the training audio and the first target pseudo label, the initial speech recognition model is iteratively trained until the text change output by the initial speech recognition model is less than a preset threshold, and the iterative training is stopped to obtain the trained speech recognition model.

[0183] In a possible implementation, the first training module 703 is configured to:

[0184] Post-processing the first target pseudo label using a large language model to obtain a second target pseudo label;

[0185] Based on the training audio and the second target pseudo label, the initial speech recognition model is fine-tuned to obtain a trained speech recognition model.

[0186] In one possible implementation, post-processing includes one or more of the following:

[0187] Punctuation marks that are consistent with semantics are added to the first target pseudo-label, named entities in the first target pseudo-label are identified and named entity identifiers are added to the first target pseudo-label, and text smoothing is performed on the first target pseudo-label.

[0188] In a possible implementation, the device further includes:

[0189] The token adding module is used to add preset tokens associated with post-processing to the vocabulary of the initial speech recognition model.

[0190] Figure 8 FIG. 1 shows a structural diagram of a speech recognition device according to an embodiment of the present disclosure. Figure 8 As shown, the device includes:

[0191] The second acquisition module 801 is used to acquire the audio to be recognized;

[0192] The second determination module 802 is configured to input the audio to be recognized into the trained speech recognition model to obtain the text corresponding to the audio to be recognized;

[0193] The trained speech recognition model is obtained based on the above-mentioned speech recognition model training method.

[0194] In a possible implementation, the device further includes:

[0195] The vocabulary processing module is used to process the vocabulary of the trained speech recognition model in response to a preset configuration so that the trained speech recognition model can turn on or off one or more of the following functions: output punctuation, text smoothing, and named entity recognition.

[0196] According to the embodiments of the present disclosure, by obtaining unlabeled training audio and determining the first target pseudo-label of the training audio based on the acoustic information and semantic information of the training audio, labels with more reasonable semantics and greater training value can be selected, thereby training the initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model. This can save the labeling cost while improving the effects of model training and speech recognition, and obtain a speech recognition model with higher accuracy.

[0197] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0198] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0199] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0200] An embodiment of the present disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0201] Figure 9 1 is a block diagram of an electronic device 1900 according to an exemplary embodiment. For example, the device 1900 may be provided as a server or a terminal device. Figure 9 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-mentioned speech recognition model training method or speech recognition method.

[0202] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0203] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the apparatus 1900 to perform the above-described method.

[0204] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0205] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0206] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0207] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0208] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0209] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0210] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0211] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0212] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A speech recognition model training method, characterized in that: The method comprises: Get training audio; Determining a first target pseudo label of the training audio according to the acoustic information and semantic information of the training audio, where the first target pseudo label indicates a target text corresponding to the training audio; Training an initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model; The first target pseudo-label is determined from a plurality of candidate pseudo-labels according to a total confidence of each of the plurality of candidate pseudo-labels, wherein the total confidence is calculated according to a first confidence, a second confidence, and a third confidence associated with the candidate pseudo-labels; The first confidence level indicates the possibility that the phoneme sequence corresponding to the associated candidate pseudo-label is the phoneme sequence of the target pseudo-label, the second confidence level indicates the possibility that the pinyin sequence corresponding to the associated candidate pseudo-label is the pinyin sequence of the target pseudo-label, and the third confidence level is obtained by splicing the multiple candidate pseudo-labels with the scene information of the training audio and then inputting them into the large language model.

2. The method according to claim 1, characterized in that The acoustic information includes a first confidence level and a second confidence level, the semantic information includes a third confidence level, and the method further includes: Inputting the training audio into an initial speech recognition model to obtain a plurality of candidate pseudo-labels, and a first confidence level and a second confidence level respectively associated with the plurality of candidate pseudo-labels, wherein the candidate pseudo-labels indicate candidate texts corresponding to the training audio; The plurality of candidate pseudo-labels are respectively input into a large language model to obtain third confidences respectively associated with the plurality of candidate pseudo-labels, where the third confidences represent semantic fluency of the associated candidate pseudo-labels.

3. The method according to claim 2, characterized in that The method further comprises: Get scene information of training audio; Inputting the plurality of candidate pseudo labels into the large language model respectively to obtain third confidences associated with the plurality of candidate pseudo labels respectively includes: The multiple candidate pseudo labels are respectively concatenated with scene information of the training audio and then input into the large language model to obtain third confidences respectively associated with the multiple candidate pseudo labels.

4. The method according to claim 2 or 3, characterized in that Determining a first target pseudo label of the training audio according to acoustic information and semantic information of the training audio includes: For any candidate pseudo-label, calculate the total confidence of the candidate pseudo-label according to the first confidence, the second confidence and the third confidence associated with the candidate pseudo-label; The first target pseudo label is determined from the plurality of candidate pseudo labels according to the total confidence of each of the plurality of candidate pseudo labels.

5. The method according to claim 1, wherein The method further comprises: Performing masking on the frequency domain graph of the training audio to obtain a masked frequency domain graph; Performing random deactivation on a predetermined proportion of neurons in the initial speech recognition model, and training the initial speech recognition model after the random deactivation based on the masked frequency domain graph corresponding to the training audio and the first target pseudo-label of the training audio to obtain a first intermediate speech recognition model; The initial speech recognition model is updated with the first intermediate speech recognition model.

6. The method according to claim 1, characterized in that The step of training an initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model includes: Based on the training audio and the first target pseudo-label, the initial speech recognition model is iteratively trained until the text change output by the initial speech recognition model is less than a preset threshold, and the iterative training is stopped to obtain the trained speech recognition model.

7. The method according to claim 1, characterized in that Training an initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model includes: Post-processing the first target pseudo-label using a large language model to obtain a second target pseudo-label; Based on the training audio and the second target pseudo-label, the initial speech recognition model is fine-tuned to obtain the trained speech recognition model.

8. The method according to claim 7, characterized in that The post-processing includes one or more of the following: Add semantically consistent punctuation marks to the first target pseudo label, identify named entities in the first target pseudo label and add named entity identifiers to the first target pseudo label, and perform text smoothing on the first target pseudo label.

9. The method according to claim 7, characterized in that The method further comprises: Preset tokens associated with the post-processing are added to the vocabulary of the initial speech recognition model.

10. A speech recognition method, characterized in that: The method comprises: Get the audio to be recognized; Input the audio to be recognized into the trained speech recognition model to obtain the text corresponding to the audio to be recognized; The trained speech recognition model is obtained based on the speech recognition model training method described in any one of claims 1 to 9.

11. The method according to claim 10, characterized in that The method further comprises: In response to a preset configuration, the vocabulary of the trained speech recognition model is processed so that the trained speech recognition model turns on or off one or more of the following functions: output punctuation, text smoothing, and named entity recognition.

12. A speech recognition model training device, characterized in that: The device comprises: A first acquisition module is used to acquire training audio; A first determining module is configured to determine a first target pseudo label of the training audio according to acoustic information and semantic information of the training audio, where the first target pseudo label indicates a target text corresponding to the training audio; A first training module is configured to train an initial speech recognition model based on the training audio and the first target pseudo-label to obtain a trained speech recognition model; The first target pseudo-label is determined from a plurality of candidate pseudo-labels according to a total confidence of each of the plurality of candidate pseudo-labels, wherein the total confidence is calculated according to a first confidence, a second confidence, and a third confidence associated with the candidate pseudo-labels; The first confidence level indicates the possibility that the phoneme sequence corresponding to the associated candidate pseudo-label is the phoneme sequence of the target pseudo-label, the second confidence level indicates the possibility that the pinyin sequence corresponding to the associated candidate pseudo-label is the pinyin sequence of the target pseudo-label, and the third confidence level is obtained by splicing the multiple candidate pseudo-labels with the scene information of the training audio and then inputting them into the large language model.

13. A speech recognition device, characterized in that: The device comprises: A second acquisition module is used to acquire the audio to be recognized; A second determination module is configured to input the audio to be recognized into a trained speech recognition model to obtain a text corresponding to the audio to be recognized; The trained speech recognition model is obtained based on the speech recognition model training method described in any one of claims 1 to 9.

14. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 9, or the method described in any one of claims 10-11, when executing the instructions stored in the memory.

15. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented, or the method according to any one of claims 10-11 is implemented.

Images