Speech recognition method, speech recognition model training method, device and apparatus

Abstract

The present disclosure relates to a speech recognition method, a training method, device and equipment of a speech recognition model, the method comprising: obtaining sample audio and labeled text corresponding to a first sample audio in the sample audio; based on the first sample audio and the labeled text, jointly pre-training an initial speech recognition model to obtain a first speech recognition model; based on the sample audio, performing parameter adjustment training on the first speech recognition model to obtain a trained speech recognition model. The embodiment of the present disclosure uses only the labeled text corresponding to part of the sample audio as a supervision signal of the intermediate layer of the model for supervised pre-training, and simultaneously performs self-supervised pre-training on the speech recognition model. Such joint pre-training helps the model to learn different dimensional pronunciation features at the same time, so that the model learns audio features faster and better in formal training, improves the training efficiency of the speech recognition model, and improves the accuracy of the speech recognition model in the application side speech recognition.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a speech recognition method, a training method for a speech recognition model, an apparatus, and a device. Background Technology

[0002] Automatic speech recognition (ASR) is a technology that automatically transcribes speech into corresponding text. With the development of computer technology and artificial intelligence, ASR is widely used in scenarios such as search, recommendation, customer service, voice assistants, and automatic captioning.

[0003] In related technologies, automatic speech recognition models rely on a large amount of labeled data for supervised training, typically requiring tens of thousands of hours of labeled speech data.

[0004] In related technologies, the production cost of speech annotation data is high and the cycle is long, resulting in low training efficiency of speech recognition models. Summary of the Invention

[0005] This disclosure provides a speech recognition method, a speech recognition model training method, apparatus, and device to at least solve the problems of high cost and long cycle in producing speech annotation data and low training efficiency of speech recognition models in related technologies. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a speech recognition method is provided, the method comprising:

[0007] Obtain the target audio;

[0008] The target audio is input into a trained speech recognition model for speech recognition processing to obtain the transcribed text corresponding to the target audio.

[0009] The speech recognition model is obtained by training a first speech recognition model after joint pre-training based on sample audio. The joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. The supervised pre-training refers to supervised training based on the first sample audio and labeled text. The labeled text is the supervision signal corresponding to the audio features output by the intermediate layer in the speech recognition model. The self-supervised pre-training refers to self-supervised training based on the first sample audio. The first sample audio refers to the sample audio in which the labeled text is annotated.

[0010] In some possible designs, the speech recognition model includes a feature extraction network, a downsampling layer, an upsampling layer, and a text prediction network, with the intermediate layer being the downsampling layer. The step of inputting the target audio into the trained speech recognition model for speech recognition processing to obtain the transcribed text corresponding to the target audio includes:

[0011] The target audio is input into the feature extraction network for feature extraction processing to obtain a first audio feature sequence;

[0012] The first audio feature sequence is input into the downsampling layer for downsampling processing to obtain the second audio feature sequence. The first frame rate corresponding to the first audio feature sequence is less than the second frame rate corresponding to the second audio feature sequence. The first frame rate and the second frame rate correspond to different pronunciation durations.

[0013] The second audio feature sequence is input into the upsampling layer for upsampling processing to obtain a third audio feature sequence, which corresponds to the first frame rate.

[0014] The third audio feature sequence is input into the text prediction network for text prediction processing to obtain the transcribed text.

[0015] According to a second aspect of the present disclosure, a method for training a speech recognition model is provided, the method comprising:

[0016] Obtain sample audio and the annotation text corresponding to the first sample audio in the sample audio, wherein the first sample audio refers to the audio that has been annotated in the sample audio;

[0017] Based on the first sample audio and the labeled text, the initial speech recognition model is jointly pre-trained to obtain the first speech recognition model. The joint pre-training refers to the joint training that integrates supervised pre-training and self-supervised pre-training. The supervised pre-training refers to supervised training based on the first sample audio and the labeled text. The labeled text is the supervision signal corresponding to the audio features output by the intermediate layer in the speech recognition model. The self-supervised pre-training refers to self-supervised training based on the first sample audio.

[0018] The first speech recognition model is trained by adjusting the parameters based on the sample audio to obtain a trained speech recognition model.

[0019] In some possible designs, the initial speech recognition model includes a feature extraction network and an audio representation network, the intermediate layer being a downsampling layer in the audio representation network, and the initial speech recognition model being jointly pre-trained based on the first sample audio and the labeled text to obtain a first speech recognition model, including:

[0020] The first sample audio is input into the feature extraction network for feature extraction processing to obtain the fourth audio feature sequence;

[0021] The audio features at preset positions in the fourth audio feature sequence are masked to obtain the fifth audio feature sequence;

[0022] The fifth audio feature sequence is input into the audio representation network for feature extraction processing to obtain the audio representation result. The audio representation result includes the sixth audio feature sequence output by the audio representation network and the seventh audio feature sequence output by the downsampling layer in the audio representation network.

[0023] Self-supervised loss information is obtained by comparing and learning the sixth audio feature sequence and the fourth audio feature sequence.

[0024] Based on the seventh audio feature sequence and the labeled text, comparative learning is performed to obtain supervised loss information;

[0025] Based on the self-supervised loss information and the supervised loss information, the parameters of the initial speech recognition model are adjusted to obtain the first speech recognition model.

[0026] In some possible designs, the audio representation network further includes an upsampling layer. The step of inputting the fifth audio feature sequence into the audio representation network for feature extraction to obtain the audio representation result includes:

[0027] The fifth audio feature sequence is input into the downsampling layer for downsampling processing to obtain the seventh audio feature sequence. The first frame rate corresponding to the fifth audio feature sequence is less than the second frame rate corresponding to the seventh audio feature sequence. The first frame rate and the second frame rate correspond to different pronunciation durations.

[0028] The seventh audio feature sequence is input into the upsampling layer for upsampling processing to obtain the sixth audio feature sequence, which corresponds to the first frame rate.

[0029] In some possible designs, the self-supervised loss information obtained by comparing the sixth audio feature sequence and the fourth audio feature sequence includes:

[0030] Obtain the cluster center features corresponding to each of the multiple preset pronunciation feature identifiers;

[0031] The audio features at the preset positions in the fourth audio feature sequence are compared with the cluster center features corresponding to each of the multiple pronunciation feature identifiers to obtain the first pronunciation feature identifier corresponding to the audio features at the preset positions in the fourth audio feature sequence;

[0032] The sixth audio feature sequence is input into the classification layer of the initial speech recognition model for classification processing, and the second pronunciation feature identifier corresponding to the audio feature at the preset position in the sixth audio feature sequence is output.

[0033] The self-supervised loss information is obtained by comparing the first pronunciation feature identifier and the second pronunciation feature identifier.

[0034] In some possible designs, obtaining the cluster center features corresponding to each of the preset multiple pronunciation feature identifiers includes:

[0035] Extract the audio spectral features corresponding to each audio frame in the sample audio;

[0036] Clustering is performed on the audio spectral features corresponding to each audio frame to obtain multiple cluster centers;

[0037] The identifier corresponding to each cluster center is determined as the pronunciation feature identifier corresponding to each cluster center;

[0038] The cluster center feature corresponding to each cluster center is determined as the cluster center feature corresponding to the corresponding pronunciation feature identifier.

[0039] In some possible designs, the step of training the first speech recognition model by adjusting the parameters based on the sample audio to obtain a trained speech recognition model includes:

[0040] The first speech recognition model is trained in a self-supervised manner based on the sample audio to obtain the second speech recognition model.

[0041] Based on the first sample audio and the labeled text, the second speech recognition model is fine-tuned and trained to obtain the speech recognition model.

[0042] According to a third aspect of the present disclosure, a voice recognition device is provided, the device comprising:

[0043] The audio acquisition module is configured to acquire the target audio.

[0044] The speech recognition module is configured to perform speech recognition processing on the target audio input trained by the speech recognition model to obtain the transcribed text corresponding to the target audio;

[0045] The speech recognition model is obtained by training a first speech recognition model after joint pre-training based on sample audio. The joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. The supervised pre-training refers to supervised training based on the first sample audio and labeled text. The labeled text is the supervision signal corresponding to the audio features output by the intermediate layer in the speech recognition model. The self-supervised pre-training refers to self-supervised training based on the first sample audio. The first sample audio refers to the sample audio in which the labeled text is annotated.

[0046] In some possible designs, the speech recognition model includes a feature extraction network, a downsampling layer, an upsampling layer, and a text prediction network, with the intermediate layer being the downsampling layer. The speech recognition module includes:

[0047] An audio feature extraction unit is configured to input the target audio into the feature extraction network for feature extraction processing to obtain a first audio feature sequence;

[0048] The downsampling unit is configured to perform downsampling processing on the downsampling layer to obtain a second audio feature sequence, wherein the first frame rate corresponding to the first audio feature sequence is less than the second frame rate corresponding to the second audio feature sequence, and the first frame rate and the second frame rate correspond to different pronunciation durations.

[0049] The upsampling unit is configured to perform upsampling processing on the upsampling layer to obtain a third audio feature sequence, the third audio feature sequence corresponding to the first frame rate;

[0050] The text prediction unit is configured to perform text prediction processing by inputting the third audio feature sequence into the text prediction network to obtain the transcribed text.

[0051] According to a third aspect of the present disclosure, a training apparatus for a speech recognition model is provided, the apparatus comprising:

[0052] The sample acquisition module is configured to acquire sample audio and the annotation text corresponding to the first sample audio in the sample audio, wherein the first sample audio refers to the audio that has been annotated in the sample audio;

[0053] The joint pre-training module is configured to perform joint pre-training on the initial speech recognition model based on the first sample audio and the labeled text to obtain a first speech recognition model. The joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. The supervised pre-training refers to supervised pre-training based on the first sample audio and the labeled text. The labeled text is the supervision signal corresponding to the audio features output by the intermediate layer in the speech recognition model. The self-supervised pre-training refers to self-supervised training based on the first sample audio.

[0054] The model training module is configured to perform parameter adjustment training on the first speech recognition model based on the sample audio to obtain a trained speech recognition model.

[0055] In some possible designs, the initial speech recognition model includes a feature extraction network and an audio representation network, the intermediate layer is a downsampling layer in the audio representation network, and the joint pre-training module includes:

[0056] The audio feature extraction submodule is configured to input the first sample audio into the feature extraction network for feature extraction processing to obtain a fourth audio feature sequence.

[0057] The audio feature masking submodule is configured to mask audio features at preset positions in the fourth audio feature sequence to obtain a fifth audio feature sequence.

[0058] The audio representation submodule is configured to perform feature extraction processing by inputting the fifth audio feature sequence into the audio representation network to obtain an audio representation result, wherein the audio representation result includes the sixth audio feature sequence output by the audio representation network and the seventh audio feature sequence output by the downsampling layer in the audio representation network.

[0059] The self-supervised loss determination submodule is configured to perform comparative learning based on the sixth audio feature sequence and the fourth audio feature sequence to obtain self-supervised loss information.

[0060] The supervised loss determination submodule is configured to perform comparative learning based on the seventh audio feature sequence and the labeled text to obtain supervised loss information;

[0061] The parameter adjustment submodule is configured to perform parameter adjustment processing on the initial speech recognition model based on the self-supervised loss information and the supervised loss information to obtain the first speech recognition model.

[0062] In some possible designs, the audio representation network further includes an upsampling layer, and the audio representation submodule includes:

[0063] The downsampling unit is configured to perform downsampling processing by inputting the fifth audio feature sequence into the downsampling layer to obtain the seventh audio feature sequence. The first frame rate corresponding to the fifth audio feature sequence is less than the second frame rate corresponding to the seventh audio feature sequence. The first frame rate and the second frame rate correspond to different pronunciation durations.

[0064] The upsampling unit is configured to perform upsampling processing on the seventh audio feature sequence input into the upsampling layer to obtain the sixth audio feature sequence, the sixth audio feature sequence corresponding to the first frame rate.

[0065] In some possible designs, the self-supervised loss determination submodule includes:

[0066] The cluster center feature acquisition unit is configured to acquire the cluster center features corresponding to each of the preset multiple pronunciation feature identifiers;

[0067] The feature identifier determination unit is configured to compare the audio feature at the preset position in the fourth audio feature sequence with the cluster center features corresponding to each of the plurality of pronunciation feature identifiers, and obtain the first pronunciation feature identifier corresponding to the audio feature at the preset position in the fourth audio feature sequence;

[0068] The feature classification unit is configured to perform classification processing by inputting the sixth audio feature sequence into the classification layer of the initial speech recognition model, and output the second pronunciation feature identifier corresponding to the audio feature at the preset position in the sixth audio feature sequence.

[0069] The self-supervised loss determination unit is configured to compare the first pronunciation feature identifier and the second pronunciation feature identifier to obtain the self-supervised loss information.

[0070] In some possible designs, the cluster center feature acquisition unit includes:

[0071] The spectral feature extraction subunit is configured to extract the audio spectral features corresponding to each audio frame in the sample audio;

[0072] The spectral feature clustering subunit is configured to perform clustering of the audio spectral features corresponding to each of the audio frames to obtain multiple cluster centers.

[0073] The feature identifier determination subunit is configured to perform the action of determining the identifier corresponding to each cluster center as the pronunciation feature identifier corresponding to each cluster center;

[0074] The cluster center feature determination subunit is configured to perform the task of determining the cluster center feature corresponding to each cluster center as the cluster center feature corresponding to the corresponding pronunciation feature identifier.

[0075] In some possible designs, the model training module includes:

[0076] The self-supervised training unit is configured to perform self-supervised training on the first speech recognition model based on the sample audio to obtain the second speech recognition model.

[0077] The fine-tuning unit is configured to perform fine-tuning training on the second speech recognition model based on the first sample audio and the labeled text, to obtain the speech recognition model.

[0078] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the speech recognition method as described in any one of the first aspects above.

[0079] According to a sixth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a training method for a speech recognition model as described in any of the second aspects above.

[0080] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the speech recognition method described in any one of the first aspects of the present disclosure.

[0081] According to an eighth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a training method for a speech recognition model as described in any of the second aspects of the present disclosure.

[0082] According to a ninth aspect of the present disclosure, a computer program product including instructions is provided that, when run on a computer, causes the computer to perform the speech recognition method described in any one of the first aspects of the present disclosure.

[0083] According to a tenth aspect of the present disclosure, a computer program product including instructions is provided, which, when run on a computer, causes the computer to perform a training method for a speech recognition model as described in any of the second aspects of the present disclosure.

[0084] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0085] This method uses audio samples with labeled text as training samples for joint pre-training of a speech recognition model. Specifically, by using the labeled text as the supervision signal corresponding to the audio features output by the intermediate layer of the speech recognition model, supervised pre-training is performed. Simultaneously, self-supervised pre-training is also conducted based on the aforementioned audio samples, achieving a joint approach of self-supervised and supervised pre-training. This allows the model to learn different dimensions of pronunciation features during joint pre-training, before formal training is conducted based on the audio samples. After joint pre-training, the model can learn audio features faster and better during formal training, reducing its dependence on large-scale labeled speech data, improving training efficiency, and enhancing the accuracy of speech recognition in applications.

[0086] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0087] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0088] Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment;

[0089] Figure 2 This is a flowchart of a speech recognition method according to an exemplary embodiment. Figure 1 ;

[0090] Figure 3 An exemplary diagram of the structure of a speech recognition model is shown.

[0091] Figure 4 This is a flowchart of a speech recognition method according to an exemplary embodiment. Figure 2 ;

[0092] Figure 5 An example is shown in the network structure diagram of an audio representation network;

[0093] Figure 6 This is a flowchart illustrating a training method for a speech recognition model according to an exemplary embodiment. Figure 1 ;

[0094] Figure 7 This is a flowchart illustrating a training method for a speech recognition model according to an exemplary embodiment. Figure 2 ;

[0095] Figure 8 An example of a training flowchart for a speech recognition model is shown;

[0096] Figure 9 This is a block diagram of a speech recognition device according to an exemplary embodiment;

[0097] Figure 10 This is a block diagram of a training apparatus for a speech recognition model according to an exemplary embodiment;

[0098] Figure 11 This is a block diagram illustrating an electronic device for speech recognition or training a speech recognition model, according to an exemplary embodiment. Detailed Implementation

[0099] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0100] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0102] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment according to an exemplary embodiment, such as... Figure 1 As shown, the application environment may include terminal 100 and server 200.

[0103] Terminal 100 can be used to provide voice recognition services to any user. Specifically, terminal 100 can be, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices, or software running on the aforementioned electronic devices, such as applications. Optionally, the operating system running on the electronic device can be, but is not limited to, Android, iOS, Linux, Windows, etc.

[0104] In an optional embodiment, server 200 can provide background services to terminal 100, such as sending transcribed text to terminal 100. Specifically, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0105] In addition, it should be noted that, Figure 1 The example shown is merely one application environment provided by this disclosure. In practical applications, other application environments may also be included, such as more terminals.

[0106] In the embodiments described in this specification, the terminal 100 and the server 200 can be directly or indirectly connected through wired or wireless communication, and this disclosure does not impose any restrictions.

[0107] Figure 2 This is a flowchart of a speech recognition method according to an exemplary embodiment. Figure 1 Optionally, this speech recognition method is used in an electronic device. Optionally, the electronic device can be a terminal or a server. For example... Figure 2 As shown, the method may include the following steps (210-220).

[0108] Step 210: Obtain the target audio.

[0109] Optionally, the target audio is the audio to be identified. This application embodiment does not limit the method of acquiring the target audio; it can be audio uploaded over a network, audio recorded by a microphone in a device, audio read from device storage, or audio from a video.

[0110] Step 220: Input the target audio into the trained speech recognition model for speech recognition processing to obtain the transcribed text corresponding to the target audio.

[0111] The speech recognition model is trained on a first speech recognition model that has been jointly pre-trained based on sample audio. This speech recognition model is used to identify transcribed text in the target audio.

[0112] The aforementioned joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. During the joint pre-training phase, a batch of labeled data is used to simultaneously perform self-supervised pre-training and supervised training. The labeled data refers to the first sample audio in the sample audio. The first sample audio refers to the sample audio with labeled text, and the number of the first sample audio is less than a preset threshold; that is, the first sample audio refers to the labeled audio in the sample audio. The aforementioned first sample audio is a small number of sample audios with labeled text among all sample audio, and the number of the first sample audio is much smaller than the total number of sample audios.

[0113] Therefore, the aforementioned trained speech recognition model can be trained using low-resource speech annotation data, making it particularly suitable for speech recognition scenarios involving dialects, less commonly spoken languages, or other specific scenarios. Related technologies rely on large amounts of labeled data for supervised training of speech recognition models. However, in practical applications, industrial-grade speech recognition systems require a wide range of use cases, typically necessitating tens of thousands of hours of labeled speech data. Besides the insufficient amount of labeled speech data allocated to various scenarios, the availability of labeled speech data for most languages ​​is also low. There are over 7,000 languages ​​worldwide, with over 200 spoken in China alone. Chinese is spoken by 90% of the population, with the vast majority of the remainder being less commonly spoken languages. Statistics show that less commonly spoken languages, spoken by fewer than one million people worldwide, account for 80% of all languages ​​globally.

[0114] For domestic dialects, overseas minority languages, and other specific scenarios (application scenarios with insufficient speech annotation data), building a speech recognition model faces at least the following difficulties: (1) scarcity of speech data; (2) high cost of manual annotation; and (3) long construction period. The cost of manual transcription and annotation of minority languages ​​is usually more than 5 times that of Mandarin, and the construction period usually requires more than 6 months. However, the above-mentioned joint pre-training only requires a small amount of speech annotation data to train the speech recognition model, effectively solving the above problems.

[0115] The aforementioned supervised pre-training refers to supervised training based on the first sample audio and labeled text. The labeled text is the supervision signal corresponding to the audio features output by the intermediate layer in the speech recognition model. The labeled text serves as both the supervision signal for supervised training and the supervision signal for joint pre-training. The location where the supervision signal is added is the intermediate layer in the speech recognition model, where the frame rate is lower than that of the input layer. For example, the frame rate of the intermediate layer is half that of the input layer. By adding supervision signals to the intermediate layer with a lower frame rate in the model, the speech recognition model can learn coarse-grained articulation units from the input signal, enabling the model to learn context modeling faster and better in subsequent self-supervised learning.

[0116] Self-supervised pre-training refers to self-supervised training based on the first sample audio. The essence of meaningful speech is a regular sequence of articulatory units. Self-supervised learning allows the model to model this sequence entirely through fine-grained acoustic sequence mining. The self-supervised signal is added to the top layer of the model, and its frame rate is consistent with the input layer.

[0117] The loss from supervised learning and the loss from self-supervised learning, when added together with certain weights, can be used as the loss for joint pre-training.

[0118] In one example, such as Figure 3 As shown, it exemplarily illustrates a model structure diagram of a speech recognition model. Figure 3 The speech recognition model shown includes a convolutional downsampling layer 31 and an audio representation layer 32. The convolutional downsampling layer 31 is the input layer of the speech recognition model, used to convert the original speech into an audio feature sequence. For example, the original speech is converted into a normalized vector sequence of 20 milliseconds / frame, with each frame corresponding to 25 milliseconds of original speech. Using a 25-millisecond sampling window and a sampling stride of 20 milliseconds / frame, the original speech is downsampled to obtain the output normalized vector sequence (i.e., the first audio feature sequence described below). The audio representation layer 32 further extracts features from the feature sequence output by the convolutional downsampling layer 31. The features output by the audio representation layer 32 are used to identify the transcribed text corresponding to the original speech. Optionally, the audio representation layer 32 consists of several self-attention layers. On the training side, the features output by the convolutional downsampling layer 31 can be labeled with their corresponding self-supervised signals by a discretization preprocessing module 33. Comparing the features output by the audio representation layer 32 with the self-supervised signals enables self-supervised pre-training of the speech recognition model.

[0119] In an exemplary embodiment, the speech recognition model includes a feature extraction network, a downsampling layer, an upsampling layer, and a text prediction network; the audio representation network includes a downsampling layer and an upsampling layer, with an intermediate layer being a downsampling layer; correspondingly, as Figure 4 As shown, step 220 above may include the following steps (221-224), Figure 4 This is a flowchart of a speech recognition method according to an exemplary embodiment. Figure 2 .

[0120] Step 221: Input the target audio into the feature extraction network for feature extraction processing to obtain the first audio feature sequence.

[0121] The aforementioned feature extraction network can be a convolutional downsampling feature extraction network, serving as the input layer of the speech recognition model.

[0122] In one possible implementation, the above feature extraction process can be as follows: a sampling window of a preset length is driven to slide and sample in the target audio according to the first frame rate to obtain each audio frame; audio feature extraction processing is performed on each audio frame to obtain the first audio features corresponding to each audio frame, thereby constituting the above-mentioned first audio feature sequence. For example, by using a 25-millisecond sampling window and downsampling the original speech with a sampling sliding step of 20 milliseconds / frame, a normalized vector sequence (i.e., the first audio feature sequence) with an output of 20 milliseconds / frame can be obtained, where each frame corresponds to 25 milliseconds of the original speech.

[0123] Taking an original speech duration of 200 milliseconds as an example, using a 25-millisecond sampling window and a sampling stride of 20 milliseconds per frame, we obtain 10 audio frames of 25 milliseconds each, with a 5-millisecond overlap between each frame. Performing convolutional feature extraction on these 10 audio frames yields 10 primary audio features.

[0124] Step 222: Input the first audio feature sequence into the downsampling layer for downsampling processing to obtain the second audio feature sequence.

[0125] The first frame rate corresponding to the first audio feature sequence is less than the second frame rate corresponding to the second audio feature sequence. The first frame rate and the second frame rate correspond to different pronunciation durations. For example, the first frame rate is 20 milliseconds / frame, and the second frame rate is 40 milliseconds / frame.

[0126] After downsampling the first audio feature sequence, the number of second audio features in the resulting second audio feature sequence is less than the number of first audio features in the first audio feature sequence. This means that the second frame rate corresponding to the second audio feature sequence is higher than the first frame rate, and the pronunciation duration of the portion of the original speech corresponding to the second audio feature, i.e., the duration of the second speech, is also higher than the duration of the first speech corresponding to the first audio feature.

[0127] After the downsampling process described above, the second audio feature and the first audio feature can correspond to different pronunciation durations, with the pronunciation duration corresponding to the first audio feature being shorter than that corresponding to the second audio feature. That is, the second audio feature and the first audio feature correspond to different pronunciation granularities. The pronunciation granularity corresponding to the first audio feature is finer, while the pronunciation granularity corresponding to the second audio feature is coarser and closer to the actual pronunciation granularity. This allows the model to extract features of pronunciation units with coarser pronunciation granularity (longer pronunciation duration) in the audio. Furthermore, the supervisory signal from the supervised pre-training process is used to supervise the second audio feature with coarser pronunciation granularity, enabling the model to quickly learn the features of pronunciation units with coarser pronunciation granularity using limited speech annotation data. Simultaneously, the model can also learn the features of pronunciation units with finer pronunciation granularity through self-supervised pre-training. Therefore, it is evident that the model can learn pronunciation features of different duration dimensions during the joint pre-training stage, improving the speech recognition accuracy of the speech recognition model in application scenarios.

[0128] Step 223: Input the second audio feature sequence into the upsampling layer for upsampling processing to obtain the third audio feature sequence.

[0129] The second audio feature sequence mentioned above can be first input into a self-attention layer between the downsampling layer and the upsampling layer for feature extraction, at which point the dimensionality of the feature sequence remains unchanged. This autoencoder structure allows the model to achieve better performance and effectively reduces the computational cost of the intermediate layers, because the computational cost of self-attention in the time dimension for a 40 ms / frame feature sequence is only 1 / 4 of that for a 20 ms / frame feature sequence.

[0130] The third audio feature sequence corresponds to the first frame rate.

[0131] By upsampling as described above, the feature sequence output by the audio representation network can be restored to the original frame rate, thereby ensuring that the output granularity of the audio representation network is consistent with the input granularity, which facilitates the model to perform self-supervised pre-training of the input feature sequence (i.e., the first audio feature sequence mentioned above) by "masking-restoration".

[0132] Alternatively, both downsampling and upsampling utilize convolutional networks to perform one-dimensional convolution operations in the time dimension.

[0133] Step 224: Input the third audio feature sequence into the text prediction network for text prediction processing to obtain the transcribed text.

[0134] The aforementioned text prediction network is a trained neural network that can output the transcribed text corresponding to the audio after receiving the third audio feature sequence.

[0135] In one example, please refer to Figure 5 , Figure 5An example is shown in the network structure diagram of an audio representation network. For example... Figure 5 As shown, the audio representation network adopts an autoencoder structure. Let the audio representation network include N (N greater than 0 and a multiple of 3) self-attention layers. The downsampling layers are the first N / 3 layers, downsampling the original 20 milliseconds / frame speech feature sequence to 40 milliseconds / frame (i.e., halving the time dimension). The middle self-attention layers are also N / 3 layers, maintaining the same 40 milliseconds / frame speech length during feature extraction. The last N / 3 layers are upsampling layers, upsampling the 40 milliseconds / frame speech feature sequence to 20 milliseconds / frame. This ensures the final model output granularity matches the input granularity, facilitating self-supervised pre-training of the model using a "mask-restore" approach. The locations where the supervision signal for joint pre-training is added are as follows: Figure 5 As shown, the joint pre-training supervision signal corresponds to a speech feature sequence of 40 milliseconds / frame: the self-supervised signal is added to the top layer of the model, and its frame rate is consistent with that of the input layer.

[0136] In summary, the technical solution provided by the embodiments of this disclosure uses a portion of the audio samples annotated with labeled text as training samples for joint pre-training, thereby jointly pre-training the speech recognition model. Specifically, by using the labeled text as the supervision signal corresponding to the audio features output by the intermediate layer of the speech recognition model, supervised pre-training of the speech recognition model can be performed. Simultaneously, self-supervised pre-training of the speech recognition model is also performed based on the aforementioned portion of the audio samples, achieving a joint approach of self-supervised and supervised pre-training. This helps the model learn pronunciation features from different dimensions simultaneously during joint pre-training, and then proceeds to formal training based on the audio samples. After joint pre-training, the model can learn audio features faster and better during formal training, reducing the speech recognition model's dependence on large-scale labeled speech data, improving the training efficiency of the speech recognition model, and enhancing the accuracy of the speech recognition model in application scenarios.

[0137] Figure 6 This is a flowchart illustrating a training method for a speech recognition model according to an exemplary embodiment. Figure 1 Optionally, the training method for this speech recognition model is used in an electronic device. Optionally, the electronic device can be a terminal or a server. Figure 6 As shown, the method may include the following steps (610-630).

[0138] Step 610: Obtain the sample audio and the annotation text corresponding to the first sample audio in the sample audio.

[0139] Optionally, the first sample audio refers to the labeled audio in the sample audio. Optionally, the number of the first sample audio is less than a preset threshold.

[0140] Step 620: Based on the first sample audio and the labeled text, the initial speech recognition model is jointly pre-trained to obtain the first speech recognition model.

[0141] Optionally, joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. Supervised pre-training refers to supervised training based on the first sample audio and labeled text. The labeled text is the supervision signal corresponding to the audio features output by the intermediate layer in the speech recognition model. Self-supervised pre-training refers to self-supervised training based on the first sample audio.

[0142] The explanation of joint pre-training has been provided in the previous embodiment and will not be repeated here.

[0143] In an exemplary embodiment, the initial speech recognition model described above includes a feature extraction network and an audio representation network, and the intermediate layer can be a downsampling layer in the audio representation network; correspondingly, as... Figure 7 As shown, step 620 above may include the following steps (621-626), Figure 7 This is a flowchart illustrating a training method for a speech recognition model according to an exemplary embodiment. Figure 2 .

[0144] Step 621: Input the first sample audio into the feature extraction network for feature extraction processing to obtain the fourth audio feature sequence.

[0145] The fourth audio feature sequence mentioned above is the feature sequence output by the feature extraction network after performing feature extraction on the first sample audio. The feature extraction process is the same as that on the model application side, and will not be described again here.

[0146] Step 622: Mask the audio features at preset positions in the fourth audio feature sequence to obtain the fifth audio feature sequence.

[0147] In joint pre-training, self-supervised pre-training is "masking-restoration" training, which involves masking some audio features in the audio feature sequence input to the audio representation network. This allows the audio representation network to learn the masked audio features through other unmasked audio features, and then output the represented audio features.

[0148] Step 623: Input the fifth audio feature sequence into the audio representation network for feature extraction processing to obtain the audio representation result.

[0149] The audio representation results described above include the sixth audio feature sequence output by the audio representation network and the seventh audio feature sequence output by the downsampling layer in the audio representation network. Optionally, the intermediate layer is a downsampling layer.

[0150] In an exemplary embodiment, the audio representation network further includes an upsampling layer. The feature extraction process in the audio representation network in step 623 is as follows:

[0151] The fifth audio feature sequence is input into a downsampling layer for downsampling processing to obtain the seventh audio feature sequence. The first frame rate corresponding to the fifth audio feature sequence is lower than the second frame rate corresponding to the seventh audio feature sequence, and the first and second frame rates correspond to different pronunciation durations. The seventh audio feature sequence is input into an upsampling layer for upsampling processing to obtain the sixth audio feature sequence, which corresponds to the first frame rate.

[0152] The downsampling and upsampling processes described above are consistent with those on the application side of the model, although the data processed on the application side differs from that on the training side. The model can extract features of coarser-grained (longer-duration) articulators from the sample audio through the downsampling layer, and learn features of finer-grained articulators through the upsampling layer. Therefore, the model can learn articulator features of different duration dimensions during the joint pre-training stage, improving the speech recognition accuracy of the speech recognition model on the application side.

[0153] Step 624: Comparative learning is performed based on the sixth audio feature sequence and the fourth audio feature sequence to obtain self-supervised loss information.

[0154] The fourth audio feature sequence is the audio feature sequence before occlusion, and the sixth audio feature sequence is the complete audio feature sequence predicted by the audio representation network based on the fifth audio feature sequence after occlusion. By comparing the sixth audio feature sequence and the fourth audio feature sequence, the loss information of self-supervised pre-training in joint pre-training can be determined.

[0155] In an exemplary embodiment, the processing procedure of step 624 described above is as follows:

[0156] 1. Obtain the cluster center features corresponding to each of the preset multiple pronunciation feature identifiers. The different pronunciation features corresponding to the multiple pronunciation feature identifiers can be obtained through clustering and do not require labeling. In an exemplary embodiment, the process of obtaining the above-mentioned cluster center features is as follows:

[0157] Extract the audio spectral features corresponding to each audio frame in the sample audio. Optionally, extract the MFCC (Mel Frequency Cepstrum Coefficient) features corresponding to each audio frame.

[0158] Clustering is performed on the audio spectral features corresponding to each audio frame to obtain multiple cluster centers. Optionally, k-means clustering is used for MFCC at the frame level to obtain multiple cluster centers. Each cluster center obtained through clustering represents different speech unit features.

[0159] The identifier corresponding to each cluster center is determined as the pronunciation feature identifier for each cluster center. Optionally, the identifier corresponding to each cluster center is the index of the cluster center, and the index of the cluster center is recorded as a discretized symbol as the pronunciation feature identifier for each cluster center. The above pronunciation feature identifier can then serve as the self-supervised signal corresponding to the self-supervised pre-training in the joint pre-training.

[0160] The cluster center features corresponding to each cluster center are identified as the cluster center features corresponding to the corresponding pronunciation feature labels. This clustering method automatically clusters different pronunciation features and labels them accordingly, which can then be used as self-supervised signals for self-supervised pre-training, thus improving the accuracy of self-supervised pre-training.

[0161] 2. Compare the audio features at preset positions in the fourth audio feature sequence with the cluster center features corresponding to each of the multiple pronunciation feature identifiers to obtain the first pronunciation feature identifier corresponding to the audio features at preset positions in the fourth audio feature sequence.

[0162] Optionally, the feature distances between the audio features at preset positions in the fourth audio feature sequence and the features of each cluster center are determined, and the pronunciation feature identifier corresponding to the cluster center feature with the closest feature distance is determined as the first pronunciation feature identifier. Since the audio features at the preset positions are occluded audio features, the first pronunciation feature identifier is the self-supervised signal corresponding to the self-supervised pre-training.

[0163] 3. Input the sixth audio feature sequence into the classification layer of the initial speech recognition model for classification processing, and output the second pronunciation feature identifier corresponding to the audio feature at the preset position in the sixth audio feature sequence.

[0164] The fourth audio feature sequence mentioned above is the audio feature sequence before masking, and the sixth audio feature sequence is the complete audio feature sequence predicted by the audio representation network based on the fifth audio feature sequence after masking. Inputting the sixth audio feature sequence into a classification layer yields the probability of each audio feature in the sixth audio feature sequence corresponding to multiple pronunciation feature identifiers. From this, the maximum probability value corresponding to the sixth audio feature at a preset position (i.e., the restored feature corresponding to the masked feature) can be determined, and the pronunciation feature identifier corresponding to this maximum probability value is identified as the second pronunciation feature identifier. The second pronunciation feature identifier, predicted based on the restored feature, is the object of supervision for the self-supervised signal.

[0165] 4. Compare the first pronunciation feature identifier and the second pronunciation feature identifier to obtain the self-supervised loss information.

[0166] By comparing the first and second pronunciation feature identifiers, the loss information corresponding to the self-supervised pre-training can be accurately determined, thereby precisely measuring whether the audio representation network can accurately restore the masked audio features and provide corresponding feedback, which helps to improve the accuracy of the speech recognition model.

[0167] Step 625: Comparative learning is performed based on the seventh audio feature sequence and the labeled text to obtain supervised loss information.

[0168] The seventh audio feature sequence mentioned above is similar to the second audio feature sequence mentioned above. It corresponds to a pronunciation unit with a coarser granularity and is also the object of supervision in the supervised pre-training. The labeled text mentioned above is the supervision signal. By comparing the seventh audio feature sequence mentioned above and the labeled text, the loss information of supervised pre-training in joint pre-training can be obtained, that is, the aforementioned supervised loss information.

[0169] In an exemplary embodiment, the processing procedure of step 624 above is as follows: obtain the text representation data corresponding to the labeled text; compare the text representation data with the seventh audio feature sequence to obtain supervised loss information.

[0170] Each character in the above-mentioned annotated text has corresponding text representation data. By comparing the text representation data with the seventh audio feature sequence, it can be determined whether the audio features detected and predicted by the model are consistent with the corresponding text representation data. This can accurately measure whether the intermediate layer of the model has accurately learned the pronunciation unit features with coarser granularity, which helps to improve the accuracy of the speech recognition model.

[0171] Step 626: Based on the self-supervised loss information and the supervised loss information, the parameters of the initial speech recognition model are adjusted to obtain the first speech recognition model.

[0172] Based on the aforementioned self-supervised loss information and supervised loss information, the loss information for joint pre-training can be determined. The model parameters of the initial speech recognition model can then be adjusted according to the loss information for joint pre-training until the loss information for joint pre-training meets preset conditions, such as being below a threshold or reaching the maximum number of iterations. This completes the joint pre-training and yields the first speech recognition model after joint pre-training.

[0173] The model can extract features of coarse-grained (long-duration) articulators from sample audio samples through downsampling layers. The supervisory signals from the aforementioned supervised pre-training are used to supervise the audio features with coarse-grained articulation and calculate the supervisory loss. The supervisory loss can constrain the model to quickly learn the features of coarse-grained articulators with limited speech annotation data. At the same time, the model can also learn the features of fine-grained articulators through self-supervised pre-training and calculate the self-supervised loss. Thus, it can be seen that under the joint constraints of self-supervised loss and supervisory loss during the joint pre-training stage, the model can learn articulation features of different duration dimensions, thereby improving the speech recognition accuracy of the speech recognition model in application.

[0174] Step 630: Based on the sample audio, the parameters of the first speech recognition model are adjusted and trained to obtain the trained speech recognition model.

[0175] In an exemplary embodiment, such as Figure 7 As shown, step 630 above may include the following steps (631-632).

[0176] Step 631: Perform self-supervised training on the first speech recognition model based on sample audio to obtain the second speech recognition model.

[0177] Based on joint pre-training, the model undergoes self-supervised training using a large amount of unlabeled data from the target domain. The self-supervised training phase does not use supervision signals related to downstream tasks, resulting in strong generalization ability. However, direct self-supervised pre-training has drawbacks such as a lack of directionality and low training efficiency. Introducing the aforementioned joint pre-training effectively addresses this issue.

[0178] Step 632: Based on the first sample audio and labeled text, fine-tune and train the second speech recognition model to obtain the speech recognition model.

[0179] Finally, the model was further trained in a supervised manner on labeled data. With only a small amount of labeled data, the self-supervised model could be fine-tuned, which further improved the accuracy of the speech recognition model.

[0180] In one example, please refer to Figure 8 The example illustrates a training flowchart for a speech recognition model. Figure 8As shown, the training process of the speech recognition model is divided into three stages. The joint pre-training stage utilizes a small batch of labeled data, simultaneously performing self-supervised pre-training and supervised training. Since a speech representation of 40 milliseconds / frame is closer to the granularity of articulation, the supervised learning signal in this scheme is added to the 40 milliseconds / frame portion of the middle layer of the audio representation network; the self-supervised signal is added to the top layer of the model, with its frame rate consistent with the input layer. Self-supervised learning involves the model completely mining these articulation unit sequences from fine-grained acoustic sequences for modeling. The purpose of joint pre-training is to help the model learn coarse-grained articulation units from the input signal by adding supervised signals in the middle of the model, enabling the model to learn context modeling faster and better in subsequent self-supervised learning. Furthermore, supervised learning and self-supervised learning are performed in different output layers of the model, thus avoiding mutual interference. The loss from supervised learning and the loss from self-supervised learning are added together with certain weights to obtain the final loss.

[0181] Following joint pre-training, self-supervised training and supervised fine-tuning are performed. Based on the joint pre-training, the model undergoes self-supervised training using a large amount of unlabeled data from the target domain; finally, the model is fine-tuned through supervised training on labeled data. The initial goal of joint training in this process is to fully utilize the limited amount of labeled data for supervised learning, helping the model to train faster and better in the target domain through self-supervised pre-training (customized pre-training). Practice shows that through the joint pre-training stage, the model can significantly reduce the cycle of subsequent self-supervised training and achieve significantly better performance.

[0182] In summary, the technical solution provided by the embodiments of this disclosure uses a portion of the audio samples annotated with labeled text as training samples for joint pre-training, thereby jointly pre-training the speech recognition model. Specifically, by using the labeled text as the supervision signal corresponding to the audio features output by the intermediate layer of the speech recognition model, supervised pre-training of the speech recognition model can be performed. Simultaneously, self-supervised pre-training of the speech recognition model is also performed based on the aforementioned portion of the audio samples, achieving a joint approach of self-supervised and supervised pre-training. This helps the model learn pronunciation features from different dimensions simultaneously during joint pre-training, and then proceeds to formal training based on the audio samples. After joint pre-training, the model can learn audio features faster and better during formal training, reducing the speech recognition model's dependence on large-scale labeled speech data, improving the training efficiency of the speech recognition model, and enhancing the accuracy of the speech recognition model in application scenarios.

[0183] Figure 9 This is a block diagram illustrating a speech recognition device according to an exemplary embodiment. (Refer to...) Figure 9 The device 900 includes:

[0184] The audio acquisition module 910 is configured to acquire the target audio.

[0185] The speech recognition module 920 is configured to perform speech recognition processing on the target audio input trained by the speech recognition model to obtain the transcribed text corresponding to the target audio.

[0186] The speech recognition model is obtained by training a first speech recognition model after joint pre-training based on sample audio. The joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. The supervised pre-training refers to supervised training based on the first sample audio and labeled text. The labeled text is the supervision signal corresponding to the audio features output by the intermediate layer in the speech recognition model. The self-supervised pre-training refers to self-supervised training based on the first sample audio. The first sample audio refers to the sample audio in which the labeled text is annotated.

[0187] In some possible designs, the speech recognition model includes a feature extraction network, a downsampling layer, an upsampling layer, and a text prediction network, with the intermediate layer being the downsampling layer. The speech recognition module includes:

[0188] An audio feature extraction unit is configured to input the target audio into the feature extraction network for feature extraction processing to obtain a first audio feature sequence;

[0189] The downsampling unit is configured to perform downsampling processing on the downsampling layer to obtain a second audio feature sequence, wherein the first frame rate corresponding to the first audio feature sequence is less than the second frame rate corresponding to the second audio feature sequence, and the first frame rate and the second frame rate correspond to different pronunciation durations.

[0190] The upsampling unit is configured to perform upsampling processing on the upsampling layer to obtain a third audio feature sequence, the third audio feature sequence corresponding to the first frame rate;

[0191] The text prediction unit is configured to perform text prediction processing by inputting the third audio feature sequence into the text prediction network to obtain the transcribed text.

[0192] In summary, the technical solution provided by the embodiments of this disclosure uses a portion of the audio samples annotated with labeled text as training samples for joint pre-training, thereby jointly pre-training the speech recognition model. Specifically, by using the labeled text as the supervision signal corresponding to the audio features output by the intermediate layer of the speech recognition model, supervised pre-training of the speech recognition model can be performed. Simultaneously, self-supervised pre-training of the speech recognition model is also performed based on the aforementioned portion of the audio samples, achieving a joint approach of self-supervised and supervised pre-training. This helps the model learn pronunciation features from different dimensions simultaneously during joint pre-training, and then proceeds to formal training based on the audio samples. After joint pre-training, the model can learn audio features faster and better during formal training, reducing the speech recognition model's dependence on large-scale labeled speech data, improving the training efficiency of the speech recognition model, and enhancing the accuracy of the speech recognition model in application scenarios.

[0193] Figure 10 This is a block diagram of a training apparatus for a speech recognition model according to an exemplary embodiment. (Refer to...) Figure 10 The device 1000 includes:

[0194] The sample acquisition module 1010 is configured to acquire sample audio and the annotation text corresponding to the first sample audio in the sample audio, wherein the first sample audio refers to the annotated audio in the sample audio.

[0195] The joint pre-training module 1020 is configured to perform joint pre-training on the initial speech recognition model based on the first sample audio and the labeled text to obtain a first speech recognition model. The joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. The supervised pre-training refers to supervised pre-training based on the first sample audio and the labeled text. The labeled text is the supervision signal corresponding to the audio features output by the intermediate layer in the speech recognition model. The self-supervised pre-training refers to self-supervised training based on the first sample audio.

[0196] The model training module 1030 is configured to perform parameter adjustment training on the first speech recognition model based on the sample audio to obtain a trained speech recognition model.

[0197] In some possible designs, the initial speech recognition model includes a feature extraction network and an audio representation network, the intermediate layer is a downsampling layer in the audio representation network, and the joint pre-training module includes:

[0198] The audio feature extraction submodule is configured to input the first sample audio into the feature extraction network for feature extraction processing to obtain a fourth audio feature sequence.

[0199] The audio feature masking submodule is configured to mask audio features at preset positions in the fourth audio feature sequence to obtain a fifth audio feature sequence.

[0200] The audio representation submodule is configured to perform feature extraction processing by inputting the fifth audio feature sequence into the audio representation network to obtain an audio representation result, wherein the audio representation result includes the sixth audio feature sequence output by the audio representation network and the seventh audio feature sequence output by the downsampling layer in the audio representation network.

[0201] The self-supervised loss determination submodule is configured to perform comparative learning based on the sixth audio feature sequence and the fourth audio feature sequence to obtain self-supervised loss information.

[0202] The supervised loss determination submodule is configured to perform comparative learning based on the seventh audio feature sequence and the labeled text to obtain supervised loss information;

[0203] The parameter adjustment submodule is configured to perform parameter adjustment processing on the initial speech recognition model based on the self-supervised loss information and the supervised loss information to obtain the first speech recognition model.

[0204] In some possible designs, the audio representation network further includes an upsampling layer, and the audio representation submodule includes:

[0205] The downsampling unit is configured to perform downsampling processing by inputting the fifth audio feature sequence into the downsampling layer to obtain the seventh audio feature sequence. The first frame rate corresponding to the fifth audio feature sequence is less than the second frame rate corresponding to the seventh audio feature sequence. The first frame rate and the second frame rate correspond to different pronunciation durations.

[0206] The upsampling unit is configured to perform upsampling processing on the seventh audio feature sequence input into the upsampling layer to obtain the sixth audio feature sequence, the sixth audio feature sequence corresponding to the first frame rate.

[0207] In some possible designs, the self-supervised loss determination submodule includes:

[0208] The cluster center feature acquisition unit is configured to acquire the cluster center features corresponding to each of the preset multiple pronunciation feature identifiers;

[0209] The feature identifier determination unit is configured to compare the audio feature at the preset position in the fourth audio feature sequence with the cluster center features corresponding to each of the plurality of pronunciation feature identifiers, and obtain the first pronunciation feature identifier corresponding to the audio feature at the preset position in the fourth audio feature sequence;

[0210] The feature classification unit is configured to perform classification processing by inputting the sixth audio feature sequence into the classification layer of the initial speech recognition model, and output the second pronunciation feature identifier corresponding to the audio feature at the preset position in the sixth audio feature sequence.

[0211] The self-supervised loss determination unit is configured to compare the first pronunciation feature identifier and the second pronunciation feature identifier to obtain the self-supervised loss information.

[0212] In some possible designs, the cluster center feature acquisition unit includes:

[0213] The spectral feature extraction subunit is configured to extract the audio spectral features corresponding to each audio frame in the sample audio;

[0214] The spectral feature clustering subunit is configured to perform clustering of the audio spectral features corresponding to each of the audio frames to obtain multiple cluster centers.

[0215] The feature identifier determination subunit is configured to perform the action of determining the identifier corresponding to each cluster center as the pronunciation feature identifier corresponding to each cluster center;

[0216] The cluster center feature determination subunit is configured to perform the task of determining the cluster center feature corresponding to each cluster center as the cluster center feature corresponding to the corresponding pronunciation feature identifier.

[0217] In some possible designs, the supervised loss determination submodule includes:

[0218] The text representation data acquisition unit is configured to acquire the text representation data corresponding to the labeled text;

[0219] The supervised loss determination unit is configured to compare the text representation data with the seventh audio feature sequence to obtain the supervised loss information.

[0220] In some possible designs, the model training module includes:

[0221] The self-supervised training unit is configured to perform self-supervised training on the first speech recognition model based on the sample audio to obtain the second speech recognition model.

[0222] The fine-tuning unit is configured to perform fine-tuning training on the second speech recognition model based on the first sample audio and the labeled text, to obtain the speech recognition model.

[0223] In summary, the technical solution provided by the embodiments of this disclosure uses a portion of the audio samples annotated with labeled text as training samples for joint pre-training, thereby jointly pre-training the speech recognition model. Specifically, by using the labeled text as the supervision signal corresponding to the audio features output by the intermediate layer of the speech recognition model, supervised pre-training of the speech recognition model can be performed. Simultaneously, self-supervised pre-training of the speech recognition model is also performed based on the aforementioned portion of the audio samples, achieving a joint approach of self-supervised and supervised pre-training. This helps the model learn pronunciation features from different dimensions simultaneously during joint pre-training, and then proceeds to formal training based on the audio samples. After joint pre-training, the model can learn audio features faster and better during formal training, reducing the speech recognition model's dependence on large-scale labeled speech data, improving the training efficiency of the speech recognition model, and enhancing the accuracy of the speech recognition model in application scenarios.

[0224] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0225] Figure 11 This is a block diagram illustrating an electronic device for speech recognition or training a speech recognition model according to an exemplary embodiment. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a speech recognition method or a speech recognition model training method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0226] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the electronic device to which the present disclosure is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0227] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the speech recognition method as described in the embodiments of this disclosure.

[0228] In an exemplary embodiment, an electronic device is also provided, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a training method for a speech recognition model as described in the embodiments of this disclosure.

[0229] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the speech recognition method of the present disclosure embodiments.

[0230] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein when the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the training method of the speech recognition model in the embodiments of this disclosure.

[0231] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the speech recognition method of the present disclosure embodiments.

[0232] In an exemplary embodiment, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the training method of the speech recognition model in the embodiments of this disclosure.

[0233] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0234] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0235] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A speech recognition method, characterized in that, The method includes: Obtain the target audio; The target audio is input into a trained speech recognition model for speech recognition processing to obtain the transcribed text corresponding to the target audio. The speech recognition model includes a feature extraction network and an audio representation network. The audio representation network includes a downsampling layer and an upsampling layer. The speech recognition model is obtained by first performing self-supervised training on a first speech recognition model after joint pre-training based on sample audio to obtain a second speech recognition model. Then, the second speech recognition model is fine-tuned based on the first sample audio and labeled text. The joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. The supervised pre-training refers to supervised training based on the first sample audio and labeled text. The labeled text is the supervision signal corresponding to the audio features output by the downsampling layer in the audio representation network. The self-supervised pre-training in the joint pre-training refers to self-supervised training based on the first sample audio. The first sample audio refers to the sample audio in the sample audio that is labeled with the labeled text. The frame rate corresponding to the audio features output by the downsampling layer is lower than the frame rate corresponding to the audio features output by the speech recognition model feature extraction network. In the joint pre-training, features are extracted from the first sample audio, and the extracted audio features are masked and then input into the audio representation network. Supervised loss information is obtained based on the audio features output by the downsampling layer and the labeled text. Self-supervised loss information is obtained based on the audio features output by the upsampling layer and the audio features before masking. Parameters are adjusted based on the weighted result of the supervised loss information and the self-supervised loss information to obtain the first speech recognition model.

2. The method according to claim 1, characterized in that, The audio representation network includes a downsampling layer, an upsampling layer, and a text prediction network. The step of inputting the target audio into a trained speech recognition model for speech recognition processing to obtain the transcribed text corresponding to the target audio includes: The target audio is input into the feature extraction network for feature extraction processing to obtain a first audio feature sequence; The first audio feature sequence is input into the downsampling layer for downsampling processing to obtain the second audio feature sequence. The first frame rate corresponding to the first audio feature sequence is greater than the second frame rate corresponding to the second audio feature sequence. The first frame rate and the second frame rate correspond to different pronunciation durations. The second audio feature sequence is input into the upsampling layer for upsampling processing to obtain a third audio feature sequence, which corresponds to the first frame rate. The third audio feature sequence is input into the text prediction network for text prediction processing to obtain the transcribed text.

3. A method for training a speech recognition model, characterized in that, The method includes: Obtain sample audio and the annotation text corresponding to the first sample audio in the sample audio, wherein the first sample audio refers to the audio that has been annotated in the sample audio; Based on the first sample audio and the labeled text, an initial speech recognition model is jointly pre-trained to obtain a first speech recognition model. The speech recognition model includes a feature extraction network and an audio representation network. The joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. The supervised pre-training refers to supervised training based on the first sample audio and the labeled text. The labeled text is the supervision signal corresponding to the audio features output by the downsampling layer in the audio representation network. The self-supervised pre-training in the joint pre-training refers to self-supervised training based on the first sample audio. The initial speech recognition model includes a feature extraction network and an audio representation network. The audio representation network includes a downsampling layer and an upsampling layer. The labeled text is the supervision signal corresponding to the audio features output by the downsampling layer in the audio representation network. The frame rate corresponding to the audio features output by the downsampling layer is lower than the frame rate corresponding to the audio features output by the feature extraction network. In the joint pre-training, features are extracted from the first sample audio, and the extracted audio features are masked and then input into the audio representation network. Supervised loss information is obtained based on the audio features output by the downsampling layer and the labeled text. Self-supervised loss information is obtained based on the audio features output by the upsampling layer and the audio features before masking. Based on the weighted result of the supervised loss information and the self-supervised loss information, the parameters of the initial speech recognition model are adjusted to obtain the first speech recognition model. The first speech recognition model is trained in a self-supervised manner based on the sample audio to obtain the second speech recognition model. The second speech recognition model is fine-tuned and trained based on the first sample audio and the labeled text to obtain a trained speech recognition model.

4. The method according to claim 3, characterized in that, The step of jointly pre-training the initial speech recognition model based on the first sample audio and the labeled text to obtain the first speech recognition model includes: The first sample audio is input into the feature extraction network for feature extraction processing to obtain the fourth audio feature sequence; The audio features at preset positions in the fourth audio feature sequence are masked to obtain the fifth audio feature sequence; The fifth audio feature sequence is input into the audio representation network for feature extraction processing to obtain the audio representation result. The audio representation result includes the sixth audio feature sequence output by the audio representation network and the seventh audio feature sequence output by the downsampling layer in the audio representation network. Self-supervised loss information is obtained by comparing and learning the sixth audio feature sequence and the fourth audio feature sequence. Based on the seventh audio feature sequence and the labeled text, comparative learning is performed to obtain supervised loss information; Based on the self-supervised loss information and the supervised loss information, the parameters of the initial speech recognition model are adjusted to obtain the first speech recognition model.

5. The method according to claim 4, characterized in that, The step of inputting the fifth audio feature sequence into the audio representation network for feature extraction processing to obtain the audio representation result includes: The fifth audio feature sequence is input into the downsampling layer for downsampling processing to obtain the seventh audio feature sequence. The first frame rate corresponding to the fifth audio feature sequence is greater than the second frame rate corresponding to the seventh audio feature sequence. The first frame rate and the second frame rate correspond to different pronunciation durations. The seventh audio feature sequence is input into the upsampling layer for upsampling processing to obtain the sixth audio feature sequence, which corresponds to the first frame rate.

6. The method according to claim 4, characterized in that, The self-supervised loss information obtained by comparing and learning based on the sixth audio feature sequence and the fourth audio feature sequence includes: Obtain the cluster center features corresponding to each of the multiple preset pronunciation feature identifiers; The audio features at the preset positions in the fourth audio feature sequence are compared with the cluster center features corresponding to each of the multiple pronunciation feature identifiers to obtain the first pronunciation feature identifier corresponding to the audio features at the preset positions in the fourth audio feature sequence; The sixth audio feature sequence is input into the classification layer of the initial speech recognition model for classification processing, and the second pronunciation feature identifier corresponding to the audio feature at the preset position in the sixth audio feature sequence is output. The self-supervised loss information is obtained by comparing the first pronunciation feature identifier and the second pronunciation feature identifier.

7. The method according to claim 6, characterized in that, The step of obtaining the cluster center features corresponding to each of the preset multiple pronunciation feature identifiers includes: Extract the audio spectral features corresponding to each audio frame in the sample audio; Clustering is performed on the audio spectral features corresponding to each audio frame to obtain multiple cluster centers; The identifier corresponding to each cluster center is determined as the pronunciation feature identifier corresponding to each cluster center; The cluster center feature corresponding to each cluster center is determined as the cluster center feature corresponding to the corresponding pronunciation feature identifier.

8. A voice recognition device, characterized in that, The device includes: The audio acquisition module is configured to acquire the target audio. The speech recognition module is configured to perform speech recognition processing on the target audio input trained by the speech recognition model to obtain the transcribed text corresponding to the target audio; The speech recognition model includes a feature extraction network and an audio representation network. The audio representation network includes a downsampling layer and an upsampling layer. The speech recognition model is obtained by first performing self-supervised training on a first speech recognition model after joint pre-training based on sample audio to obtain a second speech recognition model. Then, the second speech recognition model is fine-tuned based on the first sample audio and labeled text. The joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. The supervised pre-training refers to supervised training based on the first sample audio and labeled text. The labeled text is the supervision signal corresponding to the audio features output by the downsampling layer in the audio representation network. The self-supervised pre-training in the joint pre-training refers to self-supervised training based on the first sample audio. The first sample audio refers to the sample audio in the sample audio that is labeled with the labeled text. The frame rate corresponding to the audio features output by the downsampling layer is lower than the frame rate corresponding to the audio features output by the speech recognition model feature extraction network. In the joint pre-training, features are extracted from the first sample audio, and the extracted audio features are masked and then input into the audio representation network. Supervised loss information is obtained based on the audio features output by the downsampling layer and the labeled text. Self-supervised loss information is obtained based on the audio features output by the upsampling layer and the audio features before masking. Parameters are adjusted based on the weighted result of the supervised loss information and the self-supervised loss information to obtain the first speech recognition model.

9. A training device for a speech recognition model, characterized in that, The device includes: The sample acquisition module is configured to acquire sample audio and the annotation text corresponding to the first sample audio in the sample audio, wherein the first sample audio refers to the audio that has been annotated in the sample audio; The joint pre-training module is configured to perform joint pre-training on the initial speech recognition model based on the first sample audio and the labeled text to obtain the first speech recognition model. The joint pre-training refers to joint training that integrates supervised pre-training and self-supervised pre-training. The supervised pre-training refers to supervised pre-training based on the first sample audio and the labeled text. The labeled text is the supervision signal corresponding to the audio features output by the downsampling layer in the audio representation network. The self-supervised pre-training in the joint pre-training refers to self-supervised training based on the first sample audio. The model training module is configured to perform: self-supervised training of the first speech recognition model based on the sample audio to obtain a second speech recognition model; and fine-tuning training of the second speech recognition model based on the first sample audio and the labeled text to obtain a trained speech recognition model. The initial speech recognition model includes a feature extraction network and an audio representation network. The audio representation network includes a downsampling layer and an upsampling layer. The labeled text is the supervision signal corresponding to the audio features output by the downsampling layer in the audio representation network. The frame rate corresponding to the audio features output by the downsampling layer is less than the frame rate corresponding to the audio features output by the feature extraction network. The joint pre-training module is further configured to perform the following: in the joint pre-training, feature extraction is performed on the first sample audio, and the extracted audio features are masked and then input into the audio representation network; supervised loss information is obtained based on the audio features output by the downsampling layer and the labeled text; self-supervised loss information is obtained based on the audio features output by the upsampling layer and the audio features before masking; and parameter adjustment processing is performed on the initial speech recognition model based on the weighted result of the supervised loss information and the self-supervised loss information to obtain the first speech recognition model.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the speech recognition method as described in claim 1 or 2, or the training method for the speech recognition model as described in any one of claims 3 to 7.

11. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the speech recognition method as described in claim 1 or 2, or the training method of the speech recognition model as described in any one of claims 3 to 7.

Images