Speech recognition method, device, equipment, medium and product

Abstract

The invention provides a voice recognition method and device, equipment, a medium and a product, and the method comprises the steps: obtaining original audio data to be subjected to voice recognition, and enabling the original audio data to comprise voice data and non-voice data; determining audio structure information in the original audio data, wherein the audio structure information represents different types of audios in the original audio data; the different types of audios comprise voice audios and at least one non-voice audio, and the at least one non-voice audio is at least one of multiple different types of non-voice audios; and based on the audio structure information in the original audio data, distinguishing voice data and non-voice data in the original audio data, and performing voice recognition on the voice data to obtain a voice recognition result. According to the invention, various types of non-voice data are accurately distinguished through the audio structure information, voice and non-voice can be accurately distinguished, and thus the voice recognition accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method, device, equipment, medium and product. Background Art

[0002] Speech recognition technology is a key technology that converts human speech into text or executes specific instructions. It plays an important role in smart assistants, voice search and various voice interaction systems.

[0003] Currently, speech recognition technology is usually implemented using speech recognition models. Speech recognition models are usually trained by using paired speech and corresponding text sequences as training data. This model can show good recognition performance when processing acoustic environments similar to the training data. However, when faced with new acoustic environments that have never appeared in the training set, the model often makes recognition errors. Therefore, a new speech recognition method is urgently needed to improve the accuracy of speech recognition, especially in complex and changing acoustic environments. Summary of the invention

[0004] Based on the above technical status, the present application proposes a speech recognition method, device, equipment, medium and product, which can improve the accuracy of speech recognition.

[0005] In order to achieve the above technical objectives, this application specifically proposes the following technical solutions:

[0006] The first aspect of the present application proposes a speech recognition method, comprising: obtaining original audio data to be subjected to speech recognition, wherein the original audio data includes speech data and non-speech data; determining audio structure information in the original audio data, wherein the audio structure information characterizes different categories of audio in the original audio data; the different categories of audio include speech audio and at least one non-speech audio, wherein the at least one non-speech audio is at least one of multiple different types of non-speech audio; based on the audio structure information in the original audio data, distinguishing the speech data and non-speech data in the original audio data, and performing speech recognition on the speech data to obtain a speech recognition result.

[0007] In some implementations, determining the audio structure information in the original audio data includes: determining the audio structure information in the original audio data through a speech recognition model; wherein, based on the audio structure information in the original audio data, distinguishing between speech data and non-speech data in the original audio data, and performing speech recognition on the speech data to obtain a speech recognition result, includes: distinguishing between speech data and non-speech data in the original audio data based on the audio structure information in the original audio data through a speech recognition model, and performing speech recognition on the speech data to obtain a speech recognition result; wherein the speech recognition model is a model trained based on a first discrete representation and annotated text corresponding to a first audio sample, and a second discrete representation corresponding to a second audio sample.

[0008] In some implementations, the training process of the speech recognition model includes the following steps: predicting the discrete representations of the first audio sample and the second audio sample respectively through the speech recognition model to be trained to obtain a first predicted discrete representation and a second predicted discrete representation; determining a first discrete representation loss based on the difference between the first predicted discrete representation and the first discrete representation, and determining a second discrete representation loss based on the difference between the second predicted discrete representation and the second discrete representation; converging the speech recognition model to be trained based on the first discrete representation loss and the second discrete representation loss to obtain a trained speech recognition model.

[0009] In some implementations, the training process of the speech recognition model also includes the following steps: predicting the speech recognition result of the first audio sample through the speech recognition model to be trained to obtain a predicted text; determining the text loss based on the difference between the predicted text and the annotated text; wherein, based on the first discrete representation loss and the second discrete representation loss, the speech recognition model to be trained is converged to obtain a trained speech recognition model, including: converging the speech recognition model to be trained based on the first discrete representation loss, the second discrete representation loss and the text loss to obtain a trained speech recognition model.

[0010] In some implementations, each first audio frame of the first audio sample is a speech sample, and each second audio frame of the second audio sample is a speech sample or a non-speech sample; the first discrete representation includes the classification result of the audio category of each first audio frame; and the second discrete representation includes the classification result of the audio category of each second audio frame.

[0011] In some implementations, the first discrete representation is obtained by discretizing the first audio frames through an audio discrete representation model; the second discrete representation is obtained by discretizing the second audio frames through an audio discrete representation model; wherein the audio discrete representation model is a model obtained by discrete representation learning based on a third audio sample, and is used to classify different categories of audio content in an audio sample, and each third speech frame in the third audio sample is a speech sample or a non-speech sample.

[0012] In some implementations, the speech recognition model includes an end-to-end speech recognition model or a non-end-to-end speech recognition model.

[0013] The second aspect of the present application proposes a speech recognition device, including: an acquisition unit, used to acquire original audio data to be subjected to speech recognition, wherein the original audio data includes speech data and non-speech data; a determination unit, used to determine audio structure information in the original audio data, wherein the audio structure information characterizes different categories of audio in the original audio data; the different categories of audio include speech audio and at least one non-speech audio, wherein the at least one non-speech audio is at least one of multiple different types of non-speech audio; a speech recognition unit, used to distinguish speech data and non-speech data in the original audio data based on the audio structure information in the original audio data, and perform speech recognition on the speech data to obtain a speech recognition result.

[0014] A third aspect of the present application provides an electronic device, including a memory and a processor;

[0015] The memory is connected to the processor and is used to store programs;

[0016] The processor is used to implement the speech recognition method described in any one of the first aspect and the implementation manner of the first aspect by running the program in the memory.

[0017] In a fourth aspect, the present application proposes a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the speech recognition method described in the first aspect and any one of the implementation methods of the first aspect is implemented.

[0018] In a fifth aspect, the present application proposes a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the speech recognition method described in the first aspect and any one of the implementation methods of the first aspect.

[0019] The speech recognition method, device, equipment, medium and product proposed in the present application obtain the original audio data to be speech recognized, and determine the audio structure information characterizing different categories of audio in the original audio data, wherein the different categories of audio include speech audio and at least one non-speech audio, and the at least one non-speech audio is at least one of multiple different types of non-speech audio, and then distinguish the speech data and non-speech data in the original audio data based on the audio structure information, and perform speech recognition on the speech data to obtain the speech recognition result. Since the audio structure information can distinguish different categories of audio data, which can be speech data or at least one of multiple different categories of non-speech data, more types of non-speech data can be more accurately recognized, thereby more finely and accurately distinguishing speech and non-speech, avoiding the interference of non-speech data on speech data, and improving the accuracy of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0021] Figure 1 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of a speech recognition model training process provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the structure of a speech recognition model to be trained provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of the training process of the VQ-VAE codec model provided in an embodiment of the present application;

[0025] Figure 5 A flowchart of another speech recognition model training process provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The technical solution proposed in the embodiment of the present application is applicable to various speech recognition application scenarios. For example, in the intelligent customer service scenario, speech recognition technology allows users to quickly communicate with customer service personnel to solve user problems, thereby improving the efficiency and quality of customer service. In the smart home scenario, speech recognition technology allows users to control home appliances through voice, improving the convenience of life. In the in-car entertainment scenario, speech recognition technology can help users control the operation of music playback, navigation and other functions through voice, improving driving safety.

[0029] The above examples are some application scenarios of speech recognition technology. In practice, the application scenarios of speech recognition technology are very wide, and they are not listed one by one here. The speech recognition technology of the embodiment of the present application can be applied to any speech recognition application scenario to improve the accuracy of speech recognition in any application scenario.

[0030] The technical solution provided in the embodiments of the present application can be exemplarily applied to hardware devices such as processors, electronic devices, servers (including cloud servers), or packaged into software programs to be run. When the hardware device executes the processing of the technical solution of the embodiments of the present application, or the above-mentioned software program is run, the automatic splitting of the target task and the automatic calling of the application program interface required for the task can be achieved to complete the purpose of the target task. The embodiments of the present application only exemplarily introduce the specific processing of the technical solution of the present application, and do not limit the specific implementation form of the technical solution of the present application. Any technical implementation form that can execute the processing of the technical solution of the present application can be adopted by the embodiments of the present application.

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] Before introducing this application solution, the relevant technologies are first introduced:

[0033] Speech recognition technology, also known as Automatic Speech Recognition (ASR), aims to convert the vocabulary content in human speech into computer-readable input forms, such as keystrokes, binary codes, or character sequences. The core of this technology lies in the speech recognition model, which can be divided into two categories: traditional speech recognition models (also called cascade speech recognition models) and end-to-end speech recognition models. Traditional speech recognition models independently model the acoustic model and language model. The end-to-end speech recognition model jointly models the two. In both modeling methods, the acoustic model needs to learn the pronunciation in the audio explicitly or implicitly so that the model can "understand" the content of the pronunciation.

[0034] Before the speech recognition model can be used, it needs to be trained. During the training process of the speech recognition model, paired speech and text sequences are usually collected first so that the speech recognition model can learn the mapping from speech to text sequences. These text sequences represent the content that the model is expected to recognize, that is, the labels corresponding to the speech, which are usually obtained through manual annotation. Such labels only contain information directly related to the task. For example, for speech recognition tasks, only the semantic text of the speech in the audio needs to be annotated; for sound event recognition, only the corresponding sound event labels need to be annotated.

[0035] This kind of manually labeled label can be understood as containing only the positive use cases and their labels required for the task. The model trained by these positive use cases has a good recognition effect when dealing with acoustic scenes similar to those in the training set. However, it is currently impossible to accurately label all audio in nature through manual labeling. Therefore, when faced with complex acoustic environments or new scenes that are not covered by the training set, the model often makes acoustic recognition errors, such as mistakenly identifying "horse neighing" as "hahahaha" and other words, or identifying incomprehensible noise as a string of meaningful words.

[0036] In order to solve this problem, two main strategies are currently adopted: one is to enhance the recognition effect of positive examples, and the other is to improve the ability to reject negative examples.

[0037] The positive example recognition effect refers to the transcription effect of the human voice part in a complex scene. In order to improve the positive example recognition effect, given the difficulty and time-consuming labeling of real scene audio data, simulation is usually used to simulate the real scene as much as possible to expand the model's coverage of complex scene audio, thereby improving the robustness of the model. Specifically, by simulating various complex scenes, a large amount of simulation data is generated and mixed with the original data to train the model. Common simulation techniques include adding noise, reverberation, speed change, etc. to the original audio. For far-field scenes, environmental simulation is also performed, such as simulating the impact response of different geometric spaces, different noise sources, noise positions, and motion states. The text labels of the simulated audio are the same as those before the simulation processing, which are all text contents that the speech recognition model should transcribe. These audios are then all used as positive example samples for training the speech recognition model. In this way, the positive example recognition effect in complex scenes can be effectively improved.

[0038] However, the audio data obtained by this simulation method is often difficult to fully match the real scene data, and there will always be some uncovered scenes.

[0039] The counterexample rejection effect refers to the rejection effect of parts that do not contain human voices in complex scenes. In order to improve the counterexample rejection effect, the voice activity detection (VAD) module is usually used for processing. The VAD module is located at the front end of the speech recognition model, and is used to detect whether the audio data contains human voices, and when the human voice is detected, it outputs the front and back boundary information of the human voice segment. The speech recognition model extracts the corresponding human voice audio segment in the audio data based on these boundary information for transcription. However, the VAD module may make misjudgments, resulting in the noise segment being mistakenly sent to the speech recognition model for transcription. In this case, the speech recognition model itself needs to have a certain noise rejection ability. The usual practice is to add non-human voice audio data for training when training the speech recognition model. For example, pure noise audio can be given a specific label, such as assigning a "SIL" label to each frame of audio at the phoneme level, or assigning a " <noise>"Tag. During speech recognition, if the audio recognition result is " <noise>” (or every frame is "SIL"), it will be rejected. However, this method can only mark non-speech content as " <noise>" or other content. This unified label granularity is too coarse to make a finer-grained distinction between audio other than speech. This means that the speech recognition model does not really understand the structure of sounds other than speech, but instead classifies them into one category, which easily leads to some human-like sounds being more likely to be judged as human voices.

[0040] In view of this, the embodiments of the present application are dedicated to providing a speech recognition method, apparatus, device and storage medium, which determines the audio structure information representing different categories of audio in the original audio data, distinguishes the speech data and non-speech data in the original audio data based on the audio structure information, and performs speech recognition on the speech data to obtain a speech recognition result, so as to improve the speech recognition effect. The following embodiments will be described in detail one by one.

[0041] Exemplary Methods

[0042] Figure 1 A flow chart of a speech recognition method provided in an embodiment of the present application. Figure 1 As shown, the speech recognition method provided in this embodiment includes steps S101 to S103:

[0043] S101: Acquire original audio data to be subjected to speech recognition.

[0044] The original audio data refers to the audio data collected from the real environment. The audio data can be any audio that can be collected in nature. The original audio data includes voice data and non-voice data. The voice data refers to the voice signal emitted by the target user, that is, the content to be voice recognized. The target user is the user to be voice recognized. The non-voice data covers all other sounds except the voice signal of the target user.

[0045] For example, in an office environment, non-speech data may include other people's conversations, keyboard tapping sounds, mobile phone ringing sounds, and door opening and closing sounds. In an amusement park environment, it may include other people's conversations or playing sounds other than the target user, the operation sounds of amusement facilities, and background music. In a park environment, non-speech data may include birdsong, insect calls, and natural wind sounds. These non-speech elements together constitute the background noise part of the original audio data.

[0046] S102: Determine audio structure information in original audio data.

[0047] Among them, the audio structure information characterizes different categories of audio in the original audio data. The audio structure information can reflect the audio features of different categories in the original audio data. By performing frequency domain conversion on the original audio data, the different frequency components contained in it can be observed from its spectrum graph, and these different frequency components constitute the audio structure information of the original audio data. It can be observed from the spectrum graph that different categories of audio each have a clear texture structure. For example, a piece of original audio data containing human speech, horse calls and bird calls, after converting it to the frequency domain, it can be observed that human speech, horse calls and bird calls each have a clear texture structure, and these different texture structures are the audio structure information.

[0048] Different types of audio include voice audio and at least one non-voice audio. The voice audio refers to the audio data corresponding to the voice signal emitted by the target user; and the at least one non-voice audio is any one or more of a plurality of different types of non-voice audio, which include but are not limited to the background noise or other non-voice signals described above.

[0049] In some embodiments, determining the audio structure information in the original audio data includes: determining the audio structure information in the original audio data through a speech recognition model; wherein the speech recognition model is a model trained based on a first discrete representation and annotated text corresponding to the first audio sample, and a second discrete representation corresponding to the second audio sample.

[0050] The speech recognition model may be an end-to-end speech recognition model or a non-end-to-end speech recognition model. For example, an end-to-end speech recognition model based on Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), Transformer (a sequence model based on attention mechanism), etc. A non-end-to-end speech recognition model may be a traditional cascade speech recognition model.

[0051] Before applying the speech recognition model to determine the audio structure information in the original audio data, the speech recognition model needs to be trained. The following describes the training process of the speech recognition model:

[0052] Figure 2 The following is a flow chart of a training process of a speech recognition model provided in an embodiment of the present application. Figure 2 As shown, the training process of the speech recognition model includes the following steps S201 to S203:

[0053] S201. Predict discrete representations of a first audio sample and a second audio sample respectively by using a speech recognition model to be trained to obtain a first predicted discrete representation and a second predicted discrete representation.

[0054] The first audio sample corresponds to the first predicted discrete representation, and the second audio sample corresponds to the second predicted discrete representation.

[0055] The first audio sample includes each first audio frame. Each first audio frame is a speech sample. The first predicted discrete representation includes a predicted classification result of an audio category of each first audio frame.

[0056] The second audio sample includes each second audio frame. Each second audio frame is a speech sample or a non-speech sample. The second predicted discrete representation includes a predicted classification result of the audio category of each second audio frame.

[0057] Taking the first audio sample as an example, the speech recognition model to be trained uses the audio frame as a prediction unit, predicts the audio category corresponding to each first audio frame in the first audio sample, and obtains the predicted classification results of the audio category of each first audio frame. The audio category refers to the audio attribute corresponding to each first audio frame, which can not only distinguish speech from non-speech, but also recognize the audio content of different acoustic scenes to distinguish different categories of audio in non-speech, such as distinguishing sound events or noise in different acoustic scenes.

[0058] The first predicted discrete representation and the second predicted discrete representation are both obtained through the speech recognition model to be trained. The following takes the end-to-end speech recognition model as an example and introduces the training process of the speech recognition model in conjunction with the accompanying drawings:

[0059] Figure 3 This is a schematic diagram of the structure of the speech recognition model to be trained provided in the embodiment of the present application. Figure 3 As shown, the speech recognition model to be trained includes: a first encoder module 31, an attention module 32 and a first decoder module 33. The first encoder module 31 includes an encoder 311, a first linear layer 312 and a first classifier 313; the first decoder module 33 includes a decoder 331, a second linear layer 332 and a second classifier 333.

[0060] Wherein, the first audio sample is represented as x s , and denote the second audio sample as x u .

[0061] During the training process, we first target the first audio sample x s and the second audio sample x u The acoustic features (ASR features) are extracted respectively to obtain the first audio features and the second audio features respectively. The first audio features and the second audio features can be Filter Bank (Fbank) features or Mel-scale Frequency Cepstral Coefficients (MFCC) features. Among them, the Fbank feature can use a 40-dimensional Fbank feature, the feature extraction window length is 25ms, and the window shift is 10ms, so 100-dimensional features can be extracted per second.

[0062] Afterwards, the first audio feature and the second audio feature are respectively encoded by the encoder 311 to obtain a first encoding result and a second encoding result; and the first encoding result and the second encoding result are respectively input into the first linear layer 312 for linear processing to obtain a first linear result and a second linear result; then, the first linear result and the second linear result are respectively input into the first classifier 313 for discrete representation classification to obtain a first predicted discrete representation and a second predicted discrete representation.

[0063] Continue reading Figure 2 , after step S201, step S202 may also be included.

[0064] S202: Determine a first discrete representation loss based on a difference between the first predicted discrete representation and the first discrete representation, and determine a second discrete representation loss based on a difference between the second predicted discrete representation and the second discrete representation.

[0065] The first discrete representation loss and the second discrete representation loss may adopt cross entropy loss.

[0066] The process of determining the cross entropy loss can be found in the introduction of related technologies, which will not be described in detail in this embodiment.

[0067] The first discrete representation is the discrete representation label corresponding to the first audio sample, denoted as represents the classification results of the audio of each category in the first audio sample. The second discrete representation is the discrete representation label corresponding to the second audio sample, recorded as The classification results of the audio of each category in the second audio sample are represented.

[0068] Continue reading Figure 3 , in the figure represents a second predicted discrete representation, that is, a probability of predicting the discretized representation of the second audio sample as the second discrete representation; represents a first predicted discrete representation, that is, a probability of predicting the discretized representation of the first audio sample as the first discrete representation; Represents the manually annotated character that is predicted for the t-th audio frame based on the first audio sample and the historical prediction text (the predicted character of the t-1 audio frame before the t-th first audio frame) probability.

[0069] The first discrete representation and the second discrete representation can be obtained by discretizing each first audio frame through an audio discrete representation model.

[0070] The audio discrete representation model is a model obtained by performing discrete representation learning based on the third audio sample and used to classify different types of audio content in the audio sample. Each third speech frame in the third audio sample is a speech sample or a non-speech sample.

[0071] The first discrete representation and the second discrete representation are audio classification labels obtained through unsupervised pre-training, which are similar to manually annotated semantic labels (such as text, pronunciation, etc.), and are also a kind of analysis of audio. The difference from manually annotated semantic labels is that manually annotated labels usually only include task-related labels. For example, for speech recognition tasks, it is only necessary to manually annotate the semantic text corresponding to the speech data in the audio; for sound events, it is only necessary to manually annotate the sound event labels corresponding to the sound events in the audio. In this embodiment, the first discrete representation and the second discrete representation are annotations of the audio categories to which each first audio frame in the first audio sample and each second audio frame in the second audio sample belong. This audio classification not only classifies task-related audio frames, but also classifies irrelevant audio frames, that is, fine audio categories are also given for irrelevant audio frames. For example, if the non-related audio frames include bird singing, horse neighing, dog barking, etc., then compared with the method of uniformly classifying the bird singing, horse neighing, and dog barking sounds as noise, the present embodiment can classify the bird singing, horse neighing, and dog barking sounds into bird singing, horse neighing, and dog barking sounds respectively, so as to accurately distinguish the audio categories corresponding to the non-related audio frames.

[0072] Before applying the audio discrete representation model to obtain the first discrete representation and the second discrete representation, the audio discrete representation model needs to be trained. The audio discrete representation model can be a VQ-VAE codec model or an EnCodec codec model. The following takes the VQ-VAE codec model as an example to introduce the training process of the audio discrete representation model:

[0073] Figure 4 Schematic diagram of the training process of the VQ-VAE codec model provided in the embodiment of the present application. Figure 4 As shown, the VQ-VAE encoding and decoding model includes: a second encoder module (Encoder) 41, a quantizer module (Quantiser) 42 and a second decoder module (Decoder) 43.

[0074] During the training process, the collected original third audio samples need to be converted to a sampling rate of 16K, and the sampling point values ​​need to be normalized to between 0 and 1, so as to obtain the third audio samples applied to the model training. The third audio samples include various categories of audio such as speech and non-speech. The third audio samples are unsupervised audio data, that is, the first audio samples do not have manually annotated text labels. It should be noted that when the original third audio samples are segmented by the VAD model, both the speech segment and the non-speech segment need to be retained.

[0075] The training process of the VQ-VAE codec model can include three parts: encoding, quantization, and decoding. The following is an introduction to these three parts:

[0076] (1) Coding

[0077] The third audio sample is encoded by the second encoder module 41 to obtain a third encoding result. Assume that the third audio sample is x and the third encoding result is z e (x), the encoding process can be expressed as follows:

[0078] z e (x)=Encoder(x)=CNN(x); (1)

[0079] After the third audio sample is converted to a 16K sampling rate, one second of audio may correspond to 16,000 values. The second encoder module 41 downsamples the third audio sample at a target frequency through a plurality of convolutional layers with steps to obtain a third audio sample at a target sampling rate, that is, a third encoding result.

[0080] Among them, the downsampling sampling rate of the second encoder module 41 can be determined according to multiple combinations of the kernel size and step size of the convolution module. For example, the second encoder module 41 includes 8 convolution layers, the convolution kernel sizes are 10-4-4-4-4-4-4-4, and the step sizes are 5-2-2-2-2-2-2-2-2. Thus, the target frequency obtained is 25Hz, and the number of discrete variables after quantization is 25 per second. In other words, the third audio sample per second is divided into 25 third audio frames, and each third audio frame is 25ms. The audio segment corresponding to each numerical value is a third audio frame. It can also be understood that after passing through the second encoder module 41, the third audio sample per second includes 25 audio segments, and each audio segment is a third audio frame.

[0081] (2) Quantification

[0082] The quantizer module 42 matches each third audio frame in the third encoding result with the vector in the codebook frame by frame, and uses the id (identifier) ​​corresponding to the successfully matched codebook vector as the discretization value corresponding to each third audio frame.

[0083] Wherein, each third audio frame in the third encoding result is matched with the vector in the codebook frame by frame, including: for each third audio frame in each third audio frame of the third encoding result, calculating the distance between the third encoding result corresponding to the third audio frame and each codebook vector in the codebook, and taking the codebook vector corresponding to the minimum distance as the successfully matched codebook vector corresponding to the third audio frame. The distance calculation method here can be a method for calculating the similarity between two vectors, such as Euclidean distance, cosine similarity, Hamming distance, etc.

[0084] Assume that the codebook size is K, that is, the codebook contains K codebook vectors, and the codebook vector (Embedding) matrix corresponding to the codebook is e∈R K*D , then the matching process can be expressed as the following formula:

[0085] q(x)=argmin j ||z e (x)―e j ||2; (2)

[0086] In the formula, e j is the jth codebook vector in the codebook; z e (x) is the third encoding result.

[0087] Among them, each codebook vector in the codebook can represent a category, and the quantizer module 42 can map the second encoding results corresponding to each third audio frame to each vector in the codebook by quantizing the second encoding results, so as to realize the classification of each third audio frame.

[0088] (3) Decoding

[0089] The second decoder module 43 includes several layers of Transpose convolutional layers. Its model structure is symmetrical with that of the second encoder module, and is intended to restore the discretized values ​​to the original third audio samples.

[0090] The codebook vector z corresponding to each third audio frame in the third audio sample is q (x) is used as the input of the second decoder module 43, and y is the output of the second decoder module. The decoding process of the second decoder module can be expressed as follows:

[0091] z q (x) = e(q(x)); (3)

[0092] In formula (3), q(x) represents the discretized value; e(q(x)) represents the corresponding codebook vector z determined in the codebook based on the discretized value. q (x).

[0093] p(x│z q )=Decoder(z q (x)) = TransCNN(z q (x)); (4)

[0094] In formula (4), Decoder(z q (x)) represents the value based on z q (x) decoding; TransCNN (z q (x)) represents decoding based on the TransCNN network in the third decoder, and obtains p(x│z q ); p(x│z q ) represents the codebook vector z q (x) reconstructs the probability distribution of the original third audio sample x.

[0095] y = argmax(p(x|z q ));(5)

[0096] Formula (5) represents the probability distribution p(x|z q ) takes the maximum probability as the reconstructed third audio sample.

[0097] After the decoding result is obtained, it is necessary to determine the reconstruction loss based on the difference between the reconstructed third audio sample and the original third audio sample; and determine the codebook loss based on the difference between the second encoding result and the codebook vector.

[0098] The reconstruction loss is intended to make the input of the second encoder module and the output of the second decoder as similar as possible. In this way, the accuracy of the quantization process of the quantizer can be guaranteed, that is, the audio category of each third audio frame can be accurately classified. The reconstruction loss can be determined by using a cross entropy loss.

[0099] The codebook loss can be represented by l2 error or l1 error. Taking l2 error as an example, l2 error can be used to represent the gap between the second encoding result and the codebook vector. The training goal of the model is to reduce this gap as much as possible so that the discrete representation can fully represent the output of the second encoder module. It is also the second decoder module that converts z q The key to recover the third audio sample. The mathematical representation of the specific training criterion is as follows:

[0100]

[0101] In the formula, logp(x│z q (x)) represents the reconstruction loss; zhongsg[z e (x)] represents the second encoding result z e The gradient of (x) stops so that the codebook vector e is close to z e (x); Where sg(e) means stopping the gradient of e so that the code vector e is close to the second encoding result z e (x).

[0102] The training goal of the VQ-VAE model is to minimize the above L. When L is minimized, such as less than the preset L, the VQ-VAE model converges.

[0103] In summary, the VQ-VAE model is unsupervised pre-trained through unsupervised third audio samples. The trained VQ-VAE model can use its second encoder module and quantizer module to label the audio data with discrete labels. The discrete labels come from the summary of the distribution of a large amount of training data during the training process of the VQ-VAE model. Through unsupervised pre-training, the VQ-VAE model can learn the audio structure information in a large amount of audio data to classify third audio frames with similar structures into the same audio category, and summarize the corresponding classification labels for non-speech audio frames. The classification label contains the classification of detailed information in various aspects of speech and non-speech data.

[0104] After the VQ-VAE model is trained, the second encoder module and the quantizer module in the VQ-VAE model can be used as an audio discrete representation model. The audio discrete representation model is used to generate corresponding discrete representation labels for a large amount of supervised audio data (audio data with text annotations) or unsupervised audio data (audio data without text annotations). For example, a first discrete representation is generated for a first audio sample, and a second discrete representation is generated for a second audio sample.

[0105] The first discrete representation serves as a discrete representation label for the first audio sample, and the second discrete representation serves as a discrete representation label for the second audio sample. At the same time, the first audio sample also corresponds to an annotated text, i.e., a text label. Therefore, the first audio sample can also be called supervised audio, and the second audio sample can also be called unsupervised audio. In this embodiment, the speech recognition model to be trained is trained by jointly using supervised audio and unsupervised audio, so that the first encoder module in the speech recognition model to be trained can understand and distinguish a large amount of structural information such as speech or non-speech during the training process.

[0106] Among them, the annotated text corresponding to the first audio sample is used for training the speech recognition task. The discrete representation labels corresponding to the first audio sample and the second audio sample are used for training the unsupervised annotation prediction task, even if the speech recognition model to be trained has the function of discretely annotating audio data. This embodiment can improve the robustness of the speech recognition model through the multi-task joint training of the speech recognition task and the unsupervised annotation prediction task, thereby improving the accuracy of speech recognition, especially the accuracy of speech recognition in complex noise scenarios.

[0107] Continue reading Figure 2 , after step S202, step S203 may also be included.

[0108] S203: Based on the first discrete representation loss and the second discrete representation loss, converge the speech recognition model to be trained to obtain a trained speech recognition model.

[0109] The purpose of the first discrete representation loss and the second discrete representation loss is to enable the speech recognition model to understand the audio structure information of the audio sample and have a more fine-grained understanding of various types of audio, so as to distinguish different types of audio. For example, for different bird calls in nature, the categories of bird calls can be distinguished and the type of bird making the bird call can be identified.

[0110] Among them, the training target of the speech recognition model is the Cross Entropy training criterion. That is, by minimizing the first discrete representation loss and minimizing the second discrete representation loss, the speech recognition model to be trained converges. When the discrete representation loss after the accumulation of the first discrete representation loss and the second discrete representation loss is less than the preset discrete representation loss, or the number of training iterations reaches the preset number of iterations, or the model accuracy reaches the preset accuracy, the training ends. Based on the model parameters at the end of the training, a trained speech recognition model can be obtained.

[0111] As described above, the first discrete representation loss and the second discrete representation loss enable the speech recognition model to learn discrete representations, thereby understanding the audio in a more fine-grained manner, especially the non-speech content in the audio, and accurately classifying the non-speech content in the audio to enhance acoustic robustness. On this basis, the speech recognition model also needs to have speech recognition capabilities. Therefore, the speech recognition capabilities of the speech recognition model also need to be trained.

[0112] Figure 5 A flowchart of the training process of another speech recognition model provided in an embodiment of the present application. Figure 5 by Figure 2 Based on Figure 2 On the basis of step S201 and step S202, the training process of the speech recognition model further includes the following steps S204 to S206:

[0113] S204: Predicting a speech recognition result of the first audio sample by using the speech recognition model to be trained to obtain a predicted text.

[0114] Continue reading Figure 3 During the training process, the first audio sample x is encoded by the encoder 311. s After encoding and obtaining a first encoding result, the first encoding result is input into the attention module 32, and the attention module 32 extracts the context information of the current predicted character based on the first encoding result and the decoding result of the previous character; and the first decoder module 33 decodes based on the context information of the current predicted character to obtain the predicted character.

[0115] Among them, the attention module 32 and the first decoder module 33 adopt an autoregressive decoding method, that is, the current character is decoded with reference to the historical decoding results (at least one predicted character), and the predicted text corresponding to the first audio sample can be obtained by repeating this cycle.

[0116] S205: Determine text loss based on the difference between the predicted text and the annotated text.

[0117] The purpose of text loss is to enable the speech recognition model to transcribe the speech content into corresponding text, thereby realizing the speech recognition function. The text loss can also be cross entropy loss.

[0118] On the basis of step S204 and step S205, step S203 may further include the following step S206:

[0119] S206 , converging the speech recognition model to be trained based on the first discrete representation loss, the second discrete representation loss and the text loss to obtain a trained speech recognition model.

[0120] In some embodiments, the speech recognition model to be trained is converged based on the first discrete representation loss, the second discrete representation loss and the text loss to obtain a trained speech recognition model, including: accumulating the first discrete representation loss, the second discrete representation loss and the text loss to obtain a comprehensive loss, and converging the speech recognition model to be trained based on the comprehensive loss. The convergence conditions include that the comprehensive loss is less than the preset loss, or the number of training iterations reaches the preset number of iterations, or the model accuracy reaches the preset accuracy, then the training ends. Based on the model parameters at the end of the training, a trained speech recognition model can be obtained.

[0121] Through the first discrete representation loss and text loss, the first discrete representation and the annotated text related to the speech sample can be associated, so that the speech recognition model can recognize the speech data in the audio data, thereby effectively distinguishing the speech data and non-speech data in the audio data.

[0122] Continue reading Figure 3 , the thinner lines in the figure represent the training process of the second audio sample. The second audio sample is also called unsupervised audio, that is, an audio sample that is not manually labeled. For the second audio sample, only the unsupervised label needs to be predicted, that is, the second predicted discrete representation. The thicker lines in the figure represent the training process of the first audio sample. The first audio sample is also called supervised audio, that is, a manually labeled audio sample. For the first audio sample, there are two corresponding training objectives, namely predicting the unsupervised label and the supervised text label. The unsupervised label is the first predicted discrete representation, and the supervised text label is the predicted text.

[0123] The trained speech recognition model includes an encoder 311 in the first encoder module, an attention module 32 and a first decoder module 33. That is to say, after the original audio data to be speech recognized is input into the trained speech recognition model, it passes through the encoder 311, the attention module 32 and the first decoder module 33 in the first encoder module in sequence to obtain the speech recognition result. Among them, since the encoder 311 in the first encoder module has learned a large number of discrete representations of supervised audio and unsupervised audio during the training process, it can accurately distinguish the audio structure information in these audios, that is, it can not only distinguish the speech in the audio data, but also distinguish different categories of non-speech in the audio data. When the audio structure information of the original audio data is input into the attention module 32, the attention module 32 can extract the speech data in the original audio data based on the audio structure information for decoding, so as to obtain the speech recognition result.

[0124] Of course, in some embodiments, the encoder 311 in the first encoder module can also extract the voice data in the original audio data based on the audio structure information and input it into the attention module 32 and the first decoder module 33 for decoding, so as to obtain the voice recognition result.

[0125] The decoding process adopts an autoregressive decoding method. The specific implementation process of the autoregressive decoding method can be found in the introduction of the relevant end, and this embodiment will not be described in detail.

[0126] Continue reading Figure 1 , after step S102, step S103 may also be included.

[0127] S103: based on the audio structure information in the original audio data, distinguish the speech data from the non-speech data in the original audio data, and perform speech recognition on the speech data to obtain a speech recognition result.

[0128] After the speech recognition model is trained, the trained speech recognition model can be used to distinguish speech data from non-speech data in the original audio data based on the audio structure information in the original audio data, and to perform speech recognition on the speech data to obtain a speech recognition result.

[0129] In summary, this embodiment performs unsupervised pre-training on the audio discrete representation model based on the third audio sample, so that the audio discrete representation model can automatically classify different categories of audio in the audio data by learning the audio structure in the unsupervised audio data, and automatically give a classification label, which contains the classification of various detailed information of speech and non-speech, so as to realize the discretization of audio representation. After that, the speech recognition model is allowed to learn the unsupervised discrete representation during the training process, so that the speech recognition model can have a further understanding of various categories of audio, learn more fine-grained audio information, and have a more fine-grained understanding of the audio data, so as to accurately distinguish various categories of audio, thereby identifying various categories of non-audio, so as to avoid misidentifying incomprehensible audio as human voice, resulting in speech recognition errors. In addition, based on the unsupervised audio data collected from the real scene, such as the second audio sample (including speech data and non-speech data), the speech recognition model predicts the classification label of such unsupervised audio data. Thereby, the speech recognition model can have a more fine-grained understanding of the speech and non-speech of the real scene, so as to improve the acoustic robustness of speech recognition.

[0130] In some embodiments, the first encoder module in the speech recognition model to be trained can use the second encoder module in the trained VQ-VAE model to train the speech recognition task, or can use other encoder modules other than the second encoder module to train the speech recognition task. This embodiment does not make specific limitations on this.

[0131] Corresponding to the above-mentioned speech recognition method, the present application embodiment also proposes a speech recognition device, such as Figure 6 As shown, the device comprises:

[0132] An acquisition unit 61 is used to acquire original audio data to be subjected to speech recognition, wherein the original audio data includes speech data and non-speech data;

[0133] A determination unit 62 is configured to determine audio structure information in the original audio data, wherein the audio structure information represents different types of audio in the original audio data; the different types of audio include speech audio and at least one type of non-speech audio, wherein the at least one type of non-speech audio is at least one of multiple different types of non-speech audio;

[0134] The speech recognition unit 63 is used to distinguish speech data and non-speech data in the original audio data based on the audio structure information in the original audio data, and perform speech recognition on the speech data to obtain a speech recognition result.

[0135] In some embodiments, the determination unit 62 determines the audio structure information in the original audio data, including: determining the audio structure information in the original audio data through a speech recognition model; wherein the speech recognition unit 63 distinguishes between speech data and non-speech data in the original audio data based on the audio structure information in the original audio data, and performs speech recognition on the speech data to obtain a speech recognition result, including: distinguishing between speech data and non-speech data in the original audio data based on the audio structure information in the original audio data through a speech recognition model, and performs speech recognition on the speech data to obtain a speech recognition result; wherein the speech recognition model is a model trained based on a first discrete representation and annotated text corresponding to a first audio sample, and a second discrete representation corresponding to a second audio sample.

[0136] In some embodiments, the training process of the speech recognition model includes the following steps: predicting the discrete representations of the first audio sample and the second audio sample respectively through the speech recognition model to be trained to obtain a first predicted discrete representation and a second predicted discrete representation; determining a first discrete representation loss based on the difference between the first predicted discrete representation and the first discrete representation, and determining a second discrete representation loss based on the difference between the second predicted discrete representation and the second discrete representation; converging the speech recognition model to be trained based on the first discrete representation loss and the second discrete representation loss to obtain a trained speech recognition model.

[0137] In some embodiments, the training process of the speech recognition model also includes the following steps: predicting the speech recognition result of the first audio sample by the speech recognition model to be trained to obtain a predicted text; determining the text loss based on the difference between the predicted text and the annotated text; wherein, based on the first discrete representation loss and the second discrete representation loss, the speech recognition model to be trained is converged to obtain a trained speech recognition model, including: converging the speech recognition model to be trained based on the first discrete representation loss, the second discrete representation loss and the text loss to obtain a trained speech recognition model.

[0138] In some embodiments, each first audio frame of the first audio sample is a speech sample, and each second audio frame of the second audio sample is a speech sample or a non-speech sample; the first discrete representation includes the classification result of the audio category of each first audio frame; and the second discrete representation includes the classification result of the audio category of each second audio frame.

[0139] In some embodiments, the first discrete representation is obtained by discretizing the first audio frames through an audio discrete representation model; the second discrete representation is obtained by discretizing the second audio frames through an audio discrete representation model; wherein the audio discrete representation model is a model obtained by learning the discrete representation based on the third audio sample, and is used to classify different categories of audio content in the audio sample, and each third speech frame in the third audio sample is a speech sample or a non-speech sample.

[0140] In some embodiments, the speech recognition model comprises an end-to-end speech recognition model or a non-end-to-end speech recognition model.

[0141] The speech recognition device provided in this embodiment belongs to the same application concept as the speech recognition method provided in the above embodiments of this application, can execute the speech recognition method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the speech recognition method. For technical details not fully described in this embodiment, please refer to the specific processing content of the speech recognition method provided in the above embodiments of this application, which will not be repeated here.

[0142] The functions implemented by the various units in the above speech recognition device can be implemented by the same or different processors, which is not limited in the embodiments of the present application.

[0143] It should be understood that the units in the above speech recognition device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the unit in the device can be implemented in the form of a hardware circuit, and the functions of some or all units can be realized by designing the hardware circuit. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all of the above units. All units of the above device can be implemented in the form of a processor calling software, or in the form of a hardware circuit, or in the form of a processor calling software, and the remaining part can be implemented in the form of a hardware circuit.

[0144] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0145] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0146] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0147] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to a data input component via the interface circuit of the control device.

[0148] The data input component specifically refers to a functional component capable of inputting or collecting voice data, such as a microphone, etc.

[0149] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0150] The processor in the control device is also a circuit with signal processing capability, which executes the speech recognition method introduced in the above embodiment. The specific implementation of the processor can refer to the above processor implementation, and the embodiment of the present application is not strictly limited.

[0151] Another embodiment of the present application also provides an electronic device, see Figure 7 As shown, the device includes:

[0152] Memory 200 and processor 210;

[0153] The memory 200 is connected to the processor 210 and is used to store programs;

[0154] The processor 210 is used to implement the speech recognition method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0155] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0156] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0157] A bus may include a pathway that transfers information between components of a computer system.

[0158] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0159] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0160] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0161] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0162] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0163] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0164] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any speech recognition method provided in the above embodiments of the present application.

[0165] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the speech recognition method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be found in the introduction to the above-mentioned embodiment of the speech recognition method.

[0166] In addition to the above methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the speech recognition method described in any of the above embodiments of this specification.

[0167] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0168] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is used by a processor to execute the steps of the speech recognition method described in any of the above embodiments of this specification.

[0169] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0170] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0172] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0174] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0176] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0177] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0178] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0179] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / noise> < / noise> < / noise>

Claims

1. A speech recognition method, characterized in that: include: Acquire original audio data to be subjected to speech recognition, wherein the original audio data includes speech data and non-speech data; Determine audio structure information in the original audio data, wherein the audio structure information represents different categories of audio in the original audio data; the different categories of audio include speech audio and at least one non-speech audio, wherein the at least one non-speech audio is at least one of a plurality of different types of non-speech audio; Based on the audio structure information in the original audio data, the voice data and non-voice data in the original audio data are distinguished, and voice recognition is performed on the voice data to obtain a voice recognition result.

2. The method according to claim 1, characterized in that The determining of the audio structure information in the original audio data comprises: Determine the audio structure information in the original audio data by using a speech recognition model; The method further comprises: distinguishing speech data from non-speech data in the original audio data based on the audio structure information in the original audio data, and performing speech recognition on the speech data to obtain a speech recognition result, including: Distinguishing speech data from non-speech data in the original audio data based on the audio structure information in the original audio data by a speech recognition model, and performing speech recognition on the speech data to obtain a speech recognition result; The speech recognition model is a model trained based on a first discrete representation and annotated text corresponding to a first audio sample, and a second discrete representation corresponding to a second audio sample.

3. The method according to claim 2, characterized in that The training process of the speech recognition model includes the following steps: Predicting discrete representations of the first audio sample and the second audio sample respectively by using a speech recognition model to be trained to obtain a first predicted discrete representation and a second predicted discrete representation; determining a first discrete representation loss based on a difference between the first predicted discrete representation and the first discrete representation, and determining a second discrete representation loss based on a difference between the second predicted discrete representation and the second discrete representation; Based on the first discrete representation loss and the second discrete representation loss, the speech recognition model to be trained is converged to obtain a trained speech recognition model.

4. The method according to claim 3, characterized in that: The training process of the speech recognition model also includes the following steps: Predicting a speech recognition result of the first audio sample by using a speech recognition model to be trained to obtain a predicted text; Determining text loss based on a difference between the predicted text and the annotated text; The method further comprises: converging the speech recognition model to be trained based on the first discrete representation loss and the second discrete representation loss to obtain a trained speech recognition model, including: The speech recognition model to be trained is converged based on the first discrete representation loss, the second discrete representation loss and the text loss to obtain a trained speech recognition model.

5. The method according to any one of claims 2 to 4, characterized in that: Each first audio frame of the first audio sample is a speech sample, and each second audio frame of the second audio sample is a speech sample or a non-speech sample; The first discrete representation includes a classification result of an audio category of each of the first audio frames; The second discrete representation includes classification results of audio categories of the respective second audio frames.

6. The method according to claim 5, characterized in that The first discrete representation is obtained by discretizing each of the first audio frames using an audio discrete representation model; The second discrete representation is obtained by discretizing each second audio frame through an audio discrete representation model; The audio discrete representation model is a model obtained by performing discrete representation learning based on the third audio sample and is used to classify different categories of audio content in the audio sample. Each third speech frame in the third audio sample is a speech sample or a non-speech sample.

7. The method according to any one of claims 2 to 4, characterized in that: The speech recognition model includes an end-to-end speech recognition model or a non-end-to-end speech recognition model.

8. An electronic device, characterized in that: including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method according to any one of claims 1 to 7 by running the program in the memory.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, cause the processor to implement the method as claimed in any one of claims 1 to 7.

Images