Training methods for audio text recognition models and audio text recognition methods

Abstract

This application relates to a training method for an audio-text recognition model, an audio-text recognition method, a computer device, and a computer program product. The method involves inputting an audio feature sequence into an audio-text recognition model to be trained. The model then obtains a language sequence recognition result based on the audio feature sequence, and a text sequence recognition result based on the language sequence recognition result and the audio feature sequence. The model parameters are adjusted according to a first loss function and a second loss function until the model training conditions are met, resulting in an audio-text recognition model. This model is then used to perform text recognition on audio. Compared to traditional methods that use hybrid models for recognition, this approach trains the audio-text recognition model using both audio features and language sequences. This allows the trained model to combine language recognition with audio text recognition, improving the accuracy of audio-text recognition.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a training method for an audio text recognition model, an audio text recognition method, an apparatus, a computer device, a storage medium, and a computer program product. Background Technology

[0002] With the development of computer technology, people can now listen to songs and other audio content on mobile phones and other devices. As time goes on, the number of songs stored on these devices will increase, necessitating the management of song libraries. Each song has corresponding lyrics, and managing these libraries requires transcribing and recognizing the lyrics. Currently, the common method for recognizing song lyrics is through hybrid model speech recognition. However, hybrid model speech recognition fails to recognize lyrics in songs containing multiple languages, leading to decreased accuracy.

[0003] Therefore, current methods for recognizing lyrics in song audio suffer from low accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a training method, audio text recognition method, device, computer equipment, computer-readable storage medium, and computer program product for an audio text recognition model that can improve recognition accuracy, in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for training an audio text recognition model, the method comprising:

[0006] Obtain audio samples and the corresponding audio text;

[0007] Obtain the audio feature sequence corresponding to the audio sample, and obtain the text sequence sample and language sequence sample corresponding to the audio text;

[0008] The audio feature sequence is input into the audio-text recognition model to be trained. The audio-text recognition model to be trained obtains a language sequence recognition result based on the audio feature sequence, and a text sequence recognition result based on the language sequence recognition result and the audio feature sequence. The model parameters of the audio-text recognition model to be trained are adjusted according to the first loss function corresponding to the language sequence recognition result and the language sequence sample, and the second loss function corresponding to the text sequence recognition result and the text sequence sample, until the model training conditions are met to obtain the audio-text recognition model.

[0009] In one embodiment, obtaining the audio feature sequence corresponding to the audio sample includes:

[0010] The spectrograms corresponding to the audio samples are converted into Mel spectrograms using a Mel-scale filter bank.

[0011] Based on the Mel spectrum features of each frame in the Mel spectrum, a Mel spectrum feature sequence corresponding to the audio sample is generated as the audio feature sequence.

[0012] In one embodiment, the step of inputting the audio feature sequence into the audio-text recognition model to be trained, obtaining a language sequence recognition result from the audio feature sequence by the audio-text recognition model to be trained, and obtaining a text sequence recognition result from the language sequence recognition result and the audio feature sequence includes:

[0013] The audio feature sequence is input into the audio text recognition model to be trained. The language sequence recognition result corresponding to the audio feature sequence is identified by the shared encoder and attention decoder in the audio text recognition model to be trained. The text sequence recognition result is obtained by the attention decoder based on the language sequence recognition result and the audio feature sequence.

[0014] In one embodiment, the step of identifying the language sequence recognition result corresponding to the audio feature sequence through the shared encoder and attention decoder in the audio-text recognition model to be trained, and obtaining the text sequence recognition result through the attention decoder based on the language sequence recognition result and the audio feature sequence recognition, includes:

[0015] The shared encoder generates a latent vector corresponding to each audio feature group based on each audio feature group in the audio feature sequence; the audio feature group contains the audio features in the audio feature sequence per unit time.

[0016] The latent vectors corresponding to each audio feature group are input into the attention decoder. The attention decoder obtains the language sequence recognition result based on the first attention corresponding to the latent vectors corresponding to each audio feature group, and obtains the text sequence recognition result based on the language sequence recognition result and the second attention corresponding to the latent vectors corresponding to each audio feature group.

[0017] In one embodiment, the step of obtaining the language sequence recognition result by the attention decoder based on the first attention corresponding to the latent vectors of each audio feature group includes:

[0018] For each unit of time, if the unit of time is the starting unit of time, the language recognition result corresponding to the unit of time is determined as the starting label; if the unit of time is the ending unit of time, the language recognition result corresponding to the unit of time is determined as the ending label; if the unit of time is an intermediate unit of time, the attention decoder determines the language recognition result corresponding to the unit of time based on the first attention of the latent vector within the unit of time and the language recognition result corresponding to the unit of time before the unit of time.

[0019] Based on the starting label corresponding to the starting unit time, the language identification results corresponding to multiple intermediate unit times, and the ending label corresponding to the ending unit time, the language sequence identification results are obtained.

[0020] In one embodiment, obtaining the text sequence recognition result based on the language sequence recognition result and the second attention corresponding to the latent vectors of each audio feature group includes:

[0021] For each unit of time, if the unit of time is the starting unit of time, the text recognition result corresponding to the unit of time is determined as the starting label; if the unit of time is the ending unit of time, the text recognition result corresponding to the unit of time is determined as the ending label; if the unit of time is an intermediate unit of time, the attention decoder determines the text recognition result corresponding to the unit of time from the text library corresponding to the language recognition result corresponding to the unit of time based on the second attention of the latent vector in the unit of time and the text recognition result corresponding to the unit of time before the unit of time.

[0022] The text sequence recognition result is obtained based on the start label corresponding to the start unit time, the text recognition results corresponding to multiple intermediate unit times, and the end label corresponding to the end unit time.

[0023] In one embodiment, the language sequence recognition result includes the language recognition result corresponding to each unit time, and the text sequence recognition result includes the text recognition result corresponding to each unit time; adjusting the model parameters of the audio-text recognition model to be trained according to the first loss function corresponding to the language sequence recognition result and the language sequence sample, and the second loss function corresponding to the text sequence recognition result and the text sequence sample, until the model training conditions are met to obtain the audio-text recognition model includes:

[0024] Obtain the first position of each language recognition result in the language sequence recognition result, and obtain the preceding language sequence sample corresponding to each language recognition result based on the language samples before the first position in the language sequence sample.

[0025] Based on the language sequence recognition results and the preceding language sequence samples corresponding to each language recognition result, a first conditional probability function corresponding to each language recognition result is constructed, and a first loss function is constructed based on multiple first conditional probability functions.

[0026] Obtain the second position of each text recognition result in the text sequence recognition result, and obtain the preceding text sequence sample corresponding to each text recognition result based on the text sample before the second position in the text sequence sample;

[0027] Based on each text recognition result and the preceding text sequence sample corresponding to each text recognition result in the text sequence recognition results, a second conditional probability function corresponding to each text recognition result is constructed, and a second loss function is constructed based on multiple second conditional probability functions;

[0028] If the first loss function is greater than the first loss function threshold or the second loss function is greater than the second loss function threshold, the model parameters of the audio text recognition model to be trained are adjusted according to the first loss function and the second loss function until the first loss function is less than or equal to the first loss function threshold and the second loss function is less than or equal to the second loss function threshold, thus obtaining the audio text recognition model.

[0029] Secondly, this application provides an audio text recognition method, the method comprising:

[0030] Obtain the audio feature sequence corresponding to the audio to be identified;

[0031] The above audio feature sequence is input into the audio text recognition model, which identifies the language tags corresponding to each frame of audio features in the audio feature sequence, and identifies the text sequence corresponding to the audio feature sequence from the text library corresponding to the language tags based on the audio feature sequence and the language tags corresponding to each frame of audio features; the audio text recognition model is trained according to the method described above.

[0032] The audio text corresponding to the audio to be identified is determined based on the text sequence.

[0033] In one embodiment, obtaining the audio feature sequence corresponding to the audio to be identified includes:

[0034] Obtain the spectrogram corresponding to the audio to be identified;

[0035] The spectrogram is converted into a Mel spectrum using a Mel-scale filter bank.

[0036] Based on the Mel spectrum features of each frame in the Mel spectrum, a Mel spectrum feature sequence corresponding to the audio to be identified is generated as the audio feature sequence.

[0037] In one embodiment, the audio text recognition model includes a shared encoder and an attention decoder; the step of identifying language tags corresponding to each frame of audio features in the audio feature sequence by the audio text recognition model, and identifying the text sequence corresponding to the audio feature sequence from the text library corresponding to the language tags based on the audio feature sequence and the language tags corresponding to each frame of audio features, includes:

[0038] The audio text recognition model generates latent vectors corresponding to the audio features of each frame based on the audio features of each frame in the audio feature sequence through the shared encoder;

[0039] The latent vectors corresponding to the audio features of each frame are input into the attention decoder. The attention decoder obtains the language sequence based on the first attention corresponding to the latent vectors corresponding to the audio features of each frame, and obtains the text sequence corresponding to the audio feature sequence based on the language sequence and the second attention corresponding to the latent vectors corresponding to the audio features of each frame. The language sequence includes the language label corresponding to the audio features of each frame.

[0040] Thirdly, this application provides a training apparatus for an audio text recognition model, the apparatus comprising:

[0041] The first acquisition module is used to acquire audio samples and the audio text corresponding to the audio samples;

[0042] The second acquisition module is used to acquire the audio feature sequence corresponding to the audio sample, and to acquire the text sequence sample and language sequence sample corresponding to the audio text;

[0043] The training module is used to input the audio feature sequence into the audio text recognition model to be trained, and the audio text recognition model to be trained obtains a language sequence recognition result based on the audio feature sequence, and obtains a text sequence recognition result based on the language sequence recognition result and the audio feature sequence; according to the first loss function corresponding to the language sequence recognition result and the language sequence sample, and the second loss function corresponding to the text sequence recognition result and the text sequence sample, the model parameters of the audio text recognition model to be trained are adjusted until the model training conditions are met to obtain the audio text recognition model.

[0044] Fourthly, this application provides an audio text recognition device, the device comprising:

[0045] The sequence acquisition module is used to acquire the audio feature sequence corresponding to the audio to be identified;

[0046] The recognition module is used to input the above-mentioned audio feature sequence into the audio text recognition model, which identifies the language tags corresponding to each frame of audio features in the audio feature sequence, and identifies the text sequence corresponding to the audio feature sequence from the text library corresponding to the language tags based on the audio feature sequence and the language tags corresponding to each frame of audio features; the audio text recognition model is trained according to the method described above.

[0047] The output module is used to determine the audio text corresponding to the audio to be identified based on the text sequence.

[0048] Fifthly, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0049] Sixthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0050] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0051] The aforementioned audio-to-text recognition model training method, device, computer equipment, storage medium, and computer program product acquire audio feature sequences corresponding to audio samples, as well as text sequence samples and language sequence samples corresponding to audio text. The audio feature sequences are input into the audio-to-text recognition model to be trained. The model then obtains language sequence recognition results based on the audio feature sequences, and text sequence recognition results based on the language sequence recognition results and audio feature sequences. The model parameters are adjusted according to a first loss function based on the language sequence recognition results and language sequence samples, and a second loss function based on the text sequence recognition results and corresponding text sequence samples, until the model training conditions are met, resulting in an audio-to-text recognition model. This model is then used to perform text recognition on audio. Compared to traditional methods using hybrid models, this approach trains the audio-to-text recognition model using audio features and language sequences, enabling the trained model to perform text recognition on audio by combining language, thus improving the accuracy of audio-to-text recognition. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the training method of an audio text recognition model in one embodiment;

[0053] Figure 2 This is a flowchart illustrating the audio text recognition steps in one embodiment;

[0054] Figure 3 This is a schematic diagram of the structure of an audio text recognition model in one embodiment;

[0055] Figure 4 This is a flowchart illustrating an audio text recognition method in one embodiment;

[0056] Figure 5 This is a structural block diagram of a training device for an audio text recognition model in one embodiment;

[0057] Figure 6 This is a structural block diagram of an audio text recognition device in one embodiment;

[0058] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0060] In one embodiment, such as Figure 1 As shown, a training method for an audio text recognition model is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server, including the following steps:

[0061] Step S202: Obtain the audio sample and the corresponding audio text.

[0062] The audio samples can be used to train an audio-text recognition model. Each audio sample can have a corresponding audio text. For example, if the audio is a song, the audio text could be the song's lyrics. The audio samples can contain multiple languages, and therefore their corresponding audio texts can also contain multiple languages. The terminal can use the audio samples to train the audio-text recognition model, thereby obtaining audio-text recognition results that can recognize audio containing multiple languages. The terminal can acquire audio samples used for training and the corresponding audio texts. The audio text corresponding to the aforementioned audio text can be a sample text containing the text information corresponding to the audio sample and the language information corresponding to each piece of text information. Therefore, the terminal can train the audio-text recognition model to be trained based on this audio text.

[0063] Step S204: Obtain the audio feature sequence corresponding to the audio sample, and obtain the text sequence sample and language sequence sample corresponding to the audio text.

[0064] The audio samples can be used to train the audio-text recognition model. During training, the terminal can acquire the audio feature sequence corresponding to the audio samples. This audio feature sequence can be a sequence composed of multiple audio features, each of which represents a feature corresponding to each frame of audio. The length of each frame can be set according to actual conditions, for example, 25ms per frame. The aforementioned audio feature sequence can be a Mel-spectrum feature sequence, which can contain the Mel-spectrum feature vectors of each frame in the audio samples.

[0065] For example, in one embodiment, obtaining the audio feature sequence corresponding to the audio sample includes: converting the spectrogram corresponding to the audio sample into a Mel spectrum using a Mel scale filter bank; and generating a Mel spectrum feature sequence corresponding to the audio sample based on the Mel spectrum features of each frame in the Mel spectrum, as the audio feature sequence.

[0066] In this embodiment, before training the audio text recognition model using audio samples, the terminal can convert the audio samples into audio feature sequences. The terminal can then use these audio feature sequences to train the audio text recognition model. These audio feature sequences can be Mel-scale feature sequences. When converting audio samples, the terminal can obtain the spectrogram corresponding to the audio samples, convert it into a Mel spectrum using a Mel-scale filter bank, and generate a Mel-scale feature sequence corresponding to the audio samples based on the Mel-scale features of each frame. Audio typically corresponds to a spectrogram, which is often a very large image. To obtain appropriately sized sound features, it is often transformed into a Mel spectrum using a Mel-scale filter bank. The terminal can then use the Mel-scale feature sequence corresponding to the audio samples to train the audio text recognition model.

[0067] In addition, the terminal can also acquire text sequence samples and language sequence samples corresponding to the aforementioned audio text. The terminal can acquire each character contained in the audio text corresponding to the aforementioned audio sample, and form a text sequence sample containing each character in the audio text based on each character. The terminal can also determine the language of each character in the text sequence sample, such as Chinese, English, or other languages. Thus, the terminal can obtain the language label of each character in the aforementioned text sequence sample, and form the language sequence sample corresponding to the aforementioned audio text based on the language labels of each character. Furthermore, the aforementioned audio text recognition model to be trained can also identify the language of the text contained in the audio, and the terminal can use the aforementioned language sequence samples to train the language recognition of the audio text recognition model to be trained.

[0068] Step S206: Input the audio feature sequence into the audio text recognition model to be trained. The audio text recognition model to be trained obtains the language sequence recognition result based on the audio feature sequence, and obtains the text sequence recognition result based on the language sequence recognition result and the audio feature sequence. Adjust the model parameters of the audio text recognition model to be trained according to the first loss function corresponding to the language sequence recognition result and the language sequence sample, and the second loss function corresponding to the text sequence recognition result and the text sequence sample, until the model training conditions are met to obtain the audio text recognition model.

[0069] The audio feature sequence can be the sequence corresponding to the audio sample. After obtaining the audio feature sequence of the audio sample, the terminal inputs the audio feature sequence into the audio text recognition model to be trained. The audio text recognition model to be trained can be an end-to-end speech recognition model based on the ASR (Automatic Speech Recognition) algorithm. The terminal can use the audio text recognition model to identify the language sequence corresponding to the audio feature sequence based on the above-mentioned input audio feature sequence.

[0070] The terminal can identify the language corresponding to audio features based on a preset number of frames of audio features in the audio feature sequence, using an audio text recognition model to be trained. This preset number of frames can be the number of audio features corresponding to a single character, or the number of frames within a preset time interval. After identifying the language features corresponding to the audio feature sequence, the terminal can obtain the language sequence recognition result.

[0071] The terminal can also use the audio-text recognition model to identify the text sequence recognition result corresponding to the audio sample based on the language sequence recognition result and the audio feature sequence. For example, the terminal can use the audio-text recognition model to determine the character recognition result corresponding to the audio features of the preset number of frames in the language sequence recognition result within the language corresponding to the preset number of frames. Thus, the terminal can obtain a text sequence recognition result containing multiple characters by recognizing multiple characters in the audio feature sequence based on the audio-text recognition model to be trained.

[0072] The aforementioned preset number of frames can be the number of frames corresponding to a single character, or the number of frames within a preset time interval. Specifically, the terminal can pre-convert the aforementioned audio samples into an audio feature sequence. This audio feature sequence can be a Mel-spectrum feature sequence, with the following specific form: X = [x1, x2, ...], which contains multiple Mel-spectrum feature vectors, where x... i Let be the Mel-spectral feature vector of the i-th frame of the audio. The specific form of the text sequence sample corresponding to the above audio sample can be Y = [y1, y2, ...], where y i This represents the i-th lyric character in the audio.

[0073] Taking a song as an example, the text sequence sample could be a lyric sequence. The terminal can assign a label z based on the number of languages, indicating which language it belongs to. Assuming the audio contains N languages, the aforementioned lyric text y... i The corresponding z must be one of N languages. Therefore, the terminal can make the specific form of the language sequence sample corresponding to the above text sequence sample as: Z = [z1, z2, ...], where z i Let y be the language type corresponding to the i-th character. During training, the terminal can identify each audio feature in the audio-text feature sequence and obtain the language sequence recognition result. Then, the terminal uses the audio-text recognition model to determine the language type of each audio feature in the language sequence recognition result, and identifies the text from the corresponding language type, thus obtaining the text sequence recognition result. Furthermore, when determining each language in the language sequence recognition result and each character in the text sequence recognition result using the audio-text recognition model, the terminal can determine the specific language information based on that language and other language information preceding it, and determine the specific character information based on that character and other character information preceding it, thereby enabling language and text recognition based on context.

[0074] After obtaining the language sequence recognition results and text sequence recognition results from the audio-text recognition model to be trained, the terminal can determine a first loss function based on the language sequence recognition results and the language sequence samples, and determine a second loss function based on the text sequence recognition results and the text sequence samples. The terminal can adjust the model parameters of the audio-text recognition model to be trained according to the first and second loss functions until the model training conditions are met, thus obtaining a trained audio-text recognition model. For example, when the values ​​of each loss function reach their respective target values, the model training conditions are considered met. Therefore, the terminal can identify the language information of each feature in the audio based on the trained audio-text recognition model, and determine the corresponding character for each feature based on the language information of each feature.

[0075] In the training method of the aforementioned audio-to-text recognition model, audio feature sequences corresponding to audio samples, as well as text sequence samples and language sequence samples corresponding to audio text, are obtained. The audio feature sequences are input into the audio-to-text recognition model to be trained. The model then obtains language sequence recognition results based on the audio feature sequences, and text sequence recognition results based on the language sequence recognition results and audio feature sequences. The model parameters are adjusted according to the first loss function of the language sequence recognition results and language sequence samples, and the second loss function of the text sequence recognition results and corresponding text sequence samples, until the model training conditions are met, resulting in the audio-to-text recognition model. This model is then used to perform text recognition on audio. Compared to traditional methods that use hybrid models for recognition, this approach trains the audio-to-text recognition model using audio features and language sequences, enabling the trained model to perform text recognition on audio by combining language, thus improving the accuracy of audio-to-text recognition.

[0076] In one embodiment, an audio feature sequence is input into an audio-text recognition model to be trained, and the audio-text recognition model to be trained obtains a language sequence recognition result based on the audio feature sequence, and obtains a text sequence recognition result based on the language sequence recognition result and the audio feature sequence, including: inputting the audio feature sequence into the audio-text recognition model to be trained, identifying the language sequence recognition result corresponding to the audio feature sequence through the shared encoder and attention decoder in the audio-text recognition model to be trained, and obtaining the text sequence recognition result through the attention decoder based on the language sequence recognition result and the audio feature sequence.

[0077] In this embodiment, the audio-to-text recognition model to be trained may include multiple modules, such as a shared encoder module and an attention decoder module. Each of these modules can be used to process the audio features in different ways. The terminal can input the audio feature sequence into the audio-to-text recognition model to be trained, and use the shared encoder and attention encoder in the model to identify the language sequence recognition result corresponding to the audio feature sequence. The attention decoder then uses the language sequence recognition result and the audio feature sequence to identify the text sequence recognition result. Specifically, as shown... Figure 2 As shown, Figure 2 This is a flowchart illustrating the audio text recognition steps in one embodiment. Wherein, s i To identify the i-th feature, a i For the recognition process at time i, the terminal can first identify the language z of the feature corresponding to that time by using a shared encoder and attention decoder. i And through an attention decoder based on language z i And the character y is obtained by recognizing the audio features corresponding to this language. i The terminal can perform language and text recognition on multiple features in the audio feature sequence to obtain language sequence recognition results and text sequence recognition results. The attention decoder mentioned above can be an attention decoder that includes language and character recognition functions. Alternatively, in some embodiments, the terminal can also construct independent attention decoders for language and character recognition respectively, and perform language recognition training and character recognition training separately through their respective attention decoders.

[0078] The shared encoder and attention decoder mentioned above can identify languages ​​and characters using latent vectors. For example, in one embodiment, the shared encoder and attention decoder in the audio-text recognition model to be trained identify the language sequence recognition result corresponding to the audio feature sequence, and the attention decoder obtains the text sequence recognition result based on the language sequence recognition result and the audio feature sequence recognition result. This includes: the shared encoder generating latent vectors corresponding to each audio feature group in the audio feature sequence, and obtaining multiple latent vectors based on multiple audio feature groups in the audio feature sequence; the audio feature group contains the audio features in the audio feature sequence within a unit time; inputting the latent vectors corresponding to each audio feature group into the attention decoder, and obtaining the language sequence recognition result based on the first attention corresponding to the latent vectors corresponding to each audio feature group, and obtaining the text sequence recognition result based on the language sequence recognition result and the second attention corresponding to the latent vectors corresponding to each audio feature group.

[0079] In this embodiment, the audio text recognition model to be trained may include a shared encoder and an attention decoder, and the structure of the audio text recognition model may be as follows: Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an audio text recognition model in one embodiment. Figure 3 It is known that the terminal can use the shared encoder and attention decoder in the audio text recognition model to identify the language and audio characters through latent vectors, respectively. The terminal can generate latent vectors corresponding to each audio feature group based on the shared encoder in the audio feature sequence, and obtain multiple latent vectors based on multiple audio feature groups in the audio feature sequence. Each audio feature group contains audio features within a unit of time in the audio feature sequence, and each audio feature group can include audio features from multiple frames. Specifically, as shown... Figure 3 As shown, the audio feature sequence can be X = [x1, x2, ...]. The terminal obtains the latent vector h corresponding to a unit time based on multiple audio features within a unit time through a shared encoder. i This yields the latent vector sequence H = [h1, h2, ...]. Here, the latent vectors represent features hidden in the original matrix, which are extracted through matrix factorization. After obtaining the latent vectors corresponding to each audio feature group, the terminal inputs these latent vectors into the attention decoder. The attention decoder, based on the first attention corresponding to the latent vectors of each audio feature group, obtains the language sequence recognition result. Furthermore, based on the language sequence recognition result and the second attention corresponding to the latent vectors of each audio feature group, it obtains the text sequence recognition result.

[0080] Specifically, such as Figure 3 As shown, the terminal can input the latent vector sequence h from the shared encoder into the attention decoder, and the attention decoder can then process the data based on unit time a. u The latent vectors corresponding to the audio features are used to identify the language corresponding to the audio features by calculating the first attention method. Thus, the terminal can identify the latent vectors corresponding to audio features from multiple time points, for example... Figure 3 The latent vector sequence h contains latent vectors for multiple audio features. The terminal divides the latent vector sequence h into latent vectors corresponding to the audio features at time u. At each time, the terminal performs language identification on the latent vectors of each audio feature, obtaining the language sequence identification result Z = [z1, z2, ...]. After obtaining the language corresponding to the latent vector at each time, the terminal can obtain the text sequence identification result Y = [y1, y2, ...] based on the language sequence identification result and the second attention corresponding to multiple latent vectors. For example, the terminal can use the attention decoder based on the language sequence identification result at unit time a. uThe latent vectors corresponding to the audio features, and the corresponding languages, are used to perform character recognition based on a text database of the aforementioned languages ​​using a second attention mechanism. Thus, the terminal can obtain the text sequence recognition result Y = [y1, y2, ...] based on the latent vectors corresponding to the audio features at multiple times and the language recognition. Figure 3 In this context, sos represents the starting label corresponding to the beginning unit time in the sequence, and eos represents the ending label corresponding to the end unit time in the sequence.

[0081] Through the above embodiments, the terminal can use the shared encoder and attention decoder in the audio text recognition model to recognize the text in the audio based on the latent vectors corresponding to the audio features and the language information corresponding to the audio features, thereby improving the accuracy of multilingual audio text recognition.

[0082] In one embodiment, the attention decoder obtains the language sequence recognition result based on the first attention corresponding to the latent vectors of each audio feature group, including: for each unit time, if the unit time is the starting unit time, determining the language recognition result corresponding to the unit time as the starting label; if the unit time is the ending unit time, determining the language recognition result corresponding to the unit time as the ending label; if the unit time is an intermediate unit time, the attention decoder determines the language recognition result corresponding to the unit time based on the first attention of the latent vectors in the unit time and the recognized language recognition results corresponding to the unit time before the unit time; and obtains the language sequence recognition result based on the starting label corresponding to the starting unit time, the language recognition results corresponding to multiple intermediate unit times, and the ending label corresponding to the ending unit time.

[0083] In this embodiment, depending on the time of the latent vector corresponding to the audio feature in the latent vector sequence, the terminal can use different methods to determine the recognition result corresponding to that time. Each of the above sequences includes a start time, a middle time, and an end time. For each unit of time, if the terminal detects that the unit of time is the start unit of time, the terminal can determine the language recognition result corresponding to that unit of time as the start label; if the terminal detects that the unit of time is the end unit of time, the terminal can determine the language recognition result corresponding to that unit of time as the end label. That is, the terminal can add start labels and end labels to the start and end positions of the final recognized language sequence recognition result, respectively.

[0084] If the terminal detects that the unit time is neither the start unit time nor the end unit time, i.e., it is an intermediate unit time, the terminal can use the aforementioned attention decoder to determine the language recognition result corresponding to the unit time based on the first attention of the latent vector within that unit time and the language recognition results corresponding to the unit time preceding that unit time. Thus, the terminal can combine this with the above information to determine the recognition result corresponding to that unit time. After recognizing the language for all unit times other than the start and end unit times, the terminal can obtain the language sequence recognition result based on the aforementioned start label, multiple language recognition results, and end label. For example, the terminal can use the start label as the beginning label of the language sequence recognition result, sort multiple language recognition results in chronological order after the start label, and use the end label as the ending label of the language sequence recognition result. Specifically, it can be as follows: Figure 3 As shown, Figure 3 In this context, "sos" represents the starting label for the initial unit of time, multiple intermediate units of time have corresponding language recognition results and text recognition results, and "eos" represents the ending label for the ending unit of time.

[0085] Through this embodiment, the terminal can determine the processing result of the latent vector based on the position of the unit time corresponding to the latent vector in the sequence. Furthermore, when it is determined that the unit time is not the start or end time, the terminal can determine the language information corresponding to each unit time based on the attention mechanism and by referring to the preceding text, thereby improving the accuracy of audio text recognition.

[0086] In one embodiment, the text sequence recognition result is obtained based on the language sequence recognition result and the second attention corresponding to the latent vectors of each audio feature group. This includes: for each unit of time, if the unit of time is the starting unit of time, determining the text recognition result corresponding to the unit of time as the starting label; if the unit of time is the ending unit of time, determining the text recognition result corresponding to the unit of time as the ending label; if the unit of time is an intermediate unit of time, the attention decoder determines the text recognition result corresponding to the unit of time from the text library corresponding to the language recognition result corresponding to the unit of time based on the second attention of the latent vectors in the unit of time and the recognized text recognition results corresponding to the unit of time before the unit of time; and obtaining the language sequence recognition result based on the starting label corresponding to the starting unit of time, the text recognition results corresponding to multiple intermediate unit of time, and the ending label corresponding to the ending unit of time.

[0087] In this embodiment, depending on the time of the latent vector corresponding to the audio feature in the latent vector sequence, the terminal can use different methods to determine the recognition result corresponding to that time. Each of the above sequences includes a start time, a middle time, and an end time. For each unit of time, if the terminal detects that the unit of time is the start unit of time, the terminal can determine the language recognition result corresponding to that unit of time as the start label; if the terminal detects that the unit of time is the end unit of time, the terminal can determine the language recognition result corresponding to that unit of time as the end label. That is, the terminal can add start labels and end labels to the start and end positions of the final recognized language sequence recognition result, respectively.

[0088] If the terminal detects that the unit time is neither the start unit time nor the end unit time, i.e., it is an intermediate unit time, the terminal can use the aforementioned attention decoder to determine the text recognition result corresponding to the unit time from the text library corresponding to the language recognition result of the unit time based on the second attention of the latent vector within that unit time and the recognized text recognition results corresponding to the unit time preceding that unit time. Thus, the terminal can combine the above information to determine the recognition result corresponding to that unit time. After recognizing the text for all unit times other than the start and end unit times, the terminal can obtain the text sequence recognition result based on the aforementioned start label, multiple text recognition results, and end label. For example, the terminal can use the start label as the beginning label of the language sequence recognition result, sort multiple language recognition results in chronological order after the start label, and use the end label as the ending label of the language sequence recognition result. Specifically, it can be as follows: Figure 3 As shown, Figure 3 In this context, "sos" represents the starting label for the initial unit of time, multiple intermediate units of time have corresponding language recognition results and text recognition results, and "eos" represents the ending label for the ending unit of time.

[0089] This embodiment allows for the determination of the processing result of the latent vector based on its position in the sequence within a given unit of time. When the unit of time is determined not to be the start or end time, an attention mechanism is used, along with contextual information, to identify the corresponding text character for each unit of time. Furthermore, language tags are explicitly included as training targets for the neural network. During prediction, the corresponding text is selected based on the language. This enhances the ASR end-to-end model's ability to distinguish audio features from different languages ​​and allows the model to remember whether the text context might switch languages, thus improving the accuracy of audio text recognition.

[0090] In one embodiment, adjusting the model parameters of the audio-text recognition model to be trained based on the first loss function corresponding to the language sequence recognition result and the language sequence sample, and the second loss function corresponding to the text sequence recognition result and the text sequence sample, includes: obtaining the first position of each language recognition result in the language sequence recognition result; obtaining the preceding language sequence sample corresponding to each language recognition result based on the language samples preceding the first position in the language sequence sample; constructing a first conditional probability function corresponding to each language recognition result based on each language recognition result in the language sequence recognition result and the preceding language sequence sample corresponding to each language recognition result; constructing a first loss function based on multiple first conditional probability functions; and obtaining the first position of each text recognition result in the text sequence recognition result. In the results, the second position is obtained by taking the text samples before the second position in the text sequence samples and obtaining the preceding text sequence samples corresponding to each text recognition result. Based on each text recognition result and the preceding text sequence samples corresponding to each text recognition result, a second conditional probability function is constructed for each text recognition result. Based on multiple second conditional probability functions, a second loss function is constructed. If the first loss function is greater than the first loss function threshold or the second loss function is greater than the second loss function threshold, the model parameters of the audio text recognition model to be trained are adjusted according to the first loss function and the second loss function until the first loss function is less than or equal to the first loss function threshold and the second loss function is less than or equal to the second loss function threshold, thus obtaining the audio text recognition model.

[0091] In this embodiment, the language sequence recognition result includes the language recognition result corresponding to each unit of time, and the text sequence recognition result includes the text recognition result corresponding to each unit of time. The terminal can adjust the audio-text recognition model to be trained based on the first loss function corresponding to the language and the second loss function corresponding to the text. The terminal can obtain the position of each language recognition result in the language sequence recognition result as the first position, and obtain the preceding language sequence sample corresponding to each language recognition result based on the language samples in the language sequence sample before the position corresponding to the first position. The preceding language sequence sample represents the samples of each language in the language sequence sample before the first position. The terminal can construct the first conditional probability function corresponding to each language recognition result based on each language recognition result in the language sequence recognition result and the preceding language sequence sample corresponding to each language recognition result. Thus, the terminal can construct the first loss function based on multiple first conditional probability functions. The specific formula of the first loss function corresponding to the above language can be as follows:

[0092] Among them, L Language The value of the first loss function. Let be the first conditional probability function. For the first language sequence sample, Let be the language output by the audio-text recognition model to be trained at time u. The terminal can use the above formula to determine the accuracy of the language sequence recognition result output by the audio-text recognition model to be trained.

[0093] The terminal can also obtain the position of each text recognition result in the text sequence recognition result as the second position, and obtain the preceding text sequence sample corresponding to each text recognition result based on the text sample before the position corresponding to the second position in the text sequence sample. The preceding text sequence sample represents the sample composed of all texts before the second position in the text sequence sample. The terminal can construct a second conditional probability function corresponding to each text recognition result based on each text recognition result in the text sequence recognition result and the preceding text sequence sample corresponding to each text recognition result. Therefore, the terminal can construct a second loss function based on multiple second conditional probability functions. Specifically, taking the above audio as a song as an example, the audio text is lyrics. Taking the lyrics as a mix of English and Chinese as an example, a lyrics text label y... i It can represent a Chinese character or an English word. Depending on the language, the terminal can separate all possible lyric text tags. For example, if y i For Chinese characters, the terminal can be marked as y. i ∈T chn If y i If it's in English, we can mark it as y. i ∈T eng The second loss function for text recognition can be shown below:

[0094] Among them, L Attention The value of the second loss function. This is the second conditional probability function. This is a sample of the preceding text sequence. The text output by the audio-text recognition model to be trained at time u is the text that needs to be derived from the language corresponding to time u. The corresponding text library determines that, i.e. Specifically, when language types Once determined, the terminal can recognize the text from the corresponding language. Recognized as Chinese, i.e. The terminal can make It must be in Chinese, that is The same principle applies to English. The terminal can use the above formula to determine the accuracy of the text sequence recognition results output by the audio text recognition model to be trained.

[0095] After determining the first and second loss functions, the terminal can compare the first loss function with its threshold and the second loss function with its threshold, and determine whether training is complete based on the comparison results. For example, if the terminal detects that the first loss function is greater than its threshold or the second loss function is greater than its threshold, the terminal can determine that the language recognition accuracy or text recognition accuracy is insufficient, and training is not complete. At this time, the terminal can adjust the model parameters of the audio-text recognition model to be trained according to the first and second loss functions, return to the step of inputting the audio feature sequence into the audio-text recognition model to be trained, and perform the next training until the first loss function is less than or equal to its threshold and the second loss function is less than or equal to its threshold, thus obtaining a trained audio-text recognition model.

[0096] Through this embodiment, the terminal can judge the language recognition accuracy and text recognition accuracy of the audio text recognition model to be trained based on the first loss function and the second loss function, and use the trained audio text recognition model to recognize text information in audio containing multiple languages, thereby improving the recognition accuracy of text in audio.

[0097] In one embodiment, such as Figure 4 As shown, an audio text recognition method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. The method includes the following steps:

[0098] Step S302: Obtain the audio feature sequence corresponding to the audio to be identified.

[0099] After training, the terminal can use the trained audio-text recognition model to perform text recognition. The terminal can obtain the audio feature sequence corresponding to the audio to be recognized. This audio feature sequence can be a sequence composed of multiple audio features, each of which represents a feature corresponding to each frame of audio. The length of each frame can be set according to actual conditions, for example, 25ms per frame. This audio feature sequence can be a Mel-spectrum feature sequence, which can contain the Mel-spectrum feature vectors of each frame in the audio sample. Specifically, the terminal can calculate the Mel-spectrum feature sequence of the audio to be recognized, obtaining X = [x1, x2, ...], and input it into the neural network audio-text recognition model.

[0100] Step S304: Input the above audio feature sequence into the audio text recognition model. The audio text recognition model identifies the language tags corresponding to each frame of audio features in the audio feature sequence. Based on the audio feature sequence and the language tags corresponding to each frame of audio features, the model identifies the text sequence corresponding to the audio feature sequence from the text library corresponding to the language tags. The audio text recognition model is trained according to the method described above.

[0101] The audio feature sequence can be the sequence corresponding to the audio sample. After obtaining the audio feature sequence of the audio sample, the terminal inputs the audio feature sequence into the audio text recognition model to be trained. This audio text recognition model can be an end-to-end speech recognition model based on the ASR (Automatic Speech Recognition) algorithm. The terminal can use this audio text recognition model to identify the language sequence corresponding to the audio feature sequence based on the input audio feature sequence. The language sequence contains language tags corresponding to each frame of audio features. The terminal can use the audio text recognition model to identify the text sequence corresponding to the audio feature sequence from a text library corresponding to the language tags. Specifically, the terminal can use a neural network to calculate the language sequence Z = [z1, z2, ...] based on Mel-spectrum features. Furthermore, the terminal can predict the audio text Y = [y1, y2, ...] based on the above language sequence Z, where each frame of the audio text represents a character y1. i ∈T zi Then the terminal can select text from the corresponding language to obtain the text sequence Y = [y1, y2, ...].

[0102] Step S306: Determine the audio text corresponding to the audio to be identified based on the text sequence.

[0103] The text sequence can be a sequence Y = [y1, y2, ...] containing multiple characters, identified based on the aforementioned audio text recognition model. The terminal can determine the audio text corresponding to the audio to be recognized based on the text sequence. For example, the terminal can extract each character from the text sequence to obtain the audio text composed of multiple characters, thus enabling the recognition of text in audio containing multiple languages.

[0104] In the aforementioned audio-to-text recognition method, audio feature sequences corresponding to audio samples, as well as text sequence samples and language sequence samples corresponding to audio text, are obtained. The audio feature sequences are then input into the audio-to-text recognition model to be trained. The model obtains language sequence recognition results based on the audio feature sequences, and text sequence recognition results based on the language sequence recognition results and audio feature sequences. The model parameters are adjusted according to a first loss function based on the language sequence recognition results and language sequence samples, and a second loss function based on the text sequence recognition results and corresponding text sequence samples, until the model training conditions are met, resulting in an audio-to-text recognition model. This model is then used to perform text recognition on the audio. Compared to traditional methods using hybrid models, this approach trains the audio-to-text recognition model using audio features and language sequences, enabling the trained model to perform text recognition on audio by combining language, thus improving the accuracy of audio-to-text recognition.

[0105] In one embodiment, obtaining the audio feature sequence corresponding to the audio to be identified includes: converting the spectrogram corresponding to the audio to be identified into a Mel spectrum using a Mel scale filter bank; and generating a Mel spectrum feature sequence corresponding to the audio to be identified based on the Mel spectrum features of each frame in the Mel spectrum, as the audio feature sequence.

[0106] In this embodiment, the aforementioned audio feature sequence can be a Mel-scale feature sequence, which may contain Mel-scale feature vectors for each frame of the audio to be recognized. Before using the audio-text recognition model to perform text recognition on the audio to be recognized, the terminal can convert the audio to be recognized into an audio feature sequence, so that the terminal can use the audio-text recognition model to perform text recognition on the audio feature sequence corresponding to the audio to be recognized. The aforementioned audio feature sequence can be a Mel-scale feature sequence. When the terminal converts the audio to be recognized, it can obtain the spectrogram corresponding to the audio to be recognized, and convert the spectrogram into a Mel spectrum using a Mel-scale filter bank. Based on the Mel-scale features of each frame in the Mel spectrum, a Mel-scale feature sequence corresponding to the audio to be recognized is generated as the audio feature sequence corresponding to the audio to be recognized. Audio usually corresponds to a spectrogram, which is often a very large image. To obtain sound features of a suitable size, it is often transformed into a Mel spectrum using a Mel-scale filter bank, so that the terminal can use the audio-text recognition model to perform text recognition on the audio based on the Mel-scale feature sequence corresponding to the audio to be recognized.

[0107] Through this embodiment, the terminal can improve the accuracy of audio text recognition by converting the audio to be recognized into a Mel-spectrum feature sequence and then using an audio text recognition model to recognize the text in the audio.

[0108] In one embodiment, an audio text recognition model identifies language tags corresponding to each frame of audio features in an audio feature sequence, and identifies a text sequence corresponding to the audio feature sequence from a text library corresponding to the language tags based on the audio feature sequence and the language tags corresponding to each frame of audio features. This includes: the audio text recognition model generating latent vectors corresponding to each frame of audio features in the audio feature sequence using a shared encoder; inputting the latent vectors corresponding to each frame of audio features into an attention decoder; obtaining a language sequence based on a first attention based on the latent vectors corresponding to each frame of audio features; and obtaining a text sequence corresponding to the audio feature sequence based on a second attention based on the language sequence and the latent vectors corresponding to each frame of audio features; the language sequence includes language tags corresponding to each frame of audio features.

[0109] In this embodiment, the structure of the audio-text recognition model can be as follows: Figure 3 As shown, the audio-text recognition model includes a shared encoder and an attention decoder. The terminal can identify the latent vectors of each frame's audio features using the shared encoder, and identify the language and text of each frame's audio features based on the attention encoder. The terminal can generate latent vectors corresponding to each frame's audio features based on the shared encoder. These latent vectors can then be input into the attention decoder. The attention decoder, based on the first attention of the latent vectors corresponding to each frame's audio features, obtains the language sequence. The terminal can then determine the language label corresponding to each frame's audio features from the language sequence. Finally, the attention decoder, based on the language sequence containing the language labels of each frame's audio features and the latent vectors corresponding to each frame's audio features, obtains the text sequence corresponding to the audio feature sequence. Thus, the terminal can obtain the text recognition result corresponding to each frame's audio features from the text sequence. Specifically, the latent vectors of each frame's audio features can form a latent vector sequence H = [h1, h2, ...]. The identified language sequence can be Z = [z1, z2, ...]. Here, z1 represents the language label of the first frame's audio features. The final text sequence can be Y = [y1, y2, ...], where y1 represents the text recognition result of the audio features of the first frame.

[0110] In this embodiment, the terminal uses the shared encoder and attention decoder in the audio text recognition model to recognize text in audio based on the latent vectors corresponding to the audio features and the language information corresponding to the audio features, thereby improving the accuracy of multilingual audio text recognition.

[0111] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0112] Based on the same inventive concept, this application also provides a training method for the audio text recognition model, a training device for the audio text recognition model, and an audio text recognition device for implementing the aforementioned audio text recognition model. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of the one or more audio text recognition model training devices and audio text recognition devices provided below can be found in the limitations of the audio text recognition model training method and audio text recognition method described above, and will not be repeated here.

[0113] In one embodiment, such as Figure 5 As shown, an audio text recognition model training method and an audio text recognition device are provided, including: a first acquisition module 500, a second acquisition module 502, and a training module 504, wherein:

[0114] The first acquisition module 500 is used to acquire audio samples and the corresponding audio text of the audio samples.

[0115] The second acquisition module 502 is used to acquire the audio feature sequence corresponding to the audio sample, as well as the text sequence sample and language sequence sample corresponding to the audio text.

[0116] The training module 504 is used to input the audio feature sequence into the audio text recognition model to be trained, and the audio text recognition model to be trained obtains the language sequence recognition result based on the audio feature sequence, and obtains the text sequence recognition result based on the language sequence recognition result and the audio feature sequence. According to the first loss function corresponding to the language sequence recognition result and the language sequence sample, and the second loss function corresponding to the text sequence recognition result and the text sequence sample, the model parameters of the audio text recognition model to be trained are adjusted until the model training conditions are met to obtain the audio text recognition model.

[0117] In one embodiment, the first acquisition module 500 is specifically used to convert the spectrogram corresponding to the audio sample into a Mel spectrum through a Mel scale filter bank; and to generate a Mel spectrum feature sequence corresponding to the audio sample based on the Mel spectrum features of each frame in the Mel spectrum, as an audio feature sequence.

[0118] In one embodiment, the training module 504 is specifically used to input the audio feature sequence into the audio text recognition model to be trained, identify the language sequence recognition result corresponding to the audio feature sequence through the shared encoder and attention decoder in the audio text recognition model to be trained, and obtain the text sequence recognition result through the attention decoder based on the language sequence recognition result and the audio feature sequence recognition.

[0119] In one embodiment, the training module 504 is specifically used to generate latent vectors corresponding to each audio feature group by the shared encoder based on each audio feature group in the audio feature sequence; the audio feature group contains the audio features in the audio feature sequence per unit time; the latent vectors corresponding to each audio feature group are input into the attention decoder, and the attention decoder obtains the language sequence recognition result based on the first attention corresponding to the latent vectors corresponding to each audio feature group, and obtains the text sequence recognition result based on the language sequence recognition result and the second attention corresponding to the latent vectors corresponding to each audio feature group.

[0120] In one embodiment, the training module 504 is specifically configured to, for each unit of time, determine the language recognition result corresponding to that unit of time as the starting label if the unit of time is the starting unit of time; determine the language recognition result corresponding to that unit of time as the ending label if the unit of time is the ending unit of time; if the unit of time is an intermediate unit of time, the attention decoder determines the language recognition result corresponding to that unit of time based on the first attention of the latent vector in that unit of time and the language recognition results corresponding to the unit of time before that unit of time; and obtain the language sequence recognition result based on the starting label corresponding to the starting unit of time, the language recognition results corresponding to multiple intermediate unit of time, and the ending label corresponding to the ending unit of time.

[0121] In one embodiment, the training module 504 is specifically used for each unit of time: if the unit of time is the starting unit of time, determine the text recognition result corresponding to the unit of time as the starting label; if the unit of time is the ending unit of time, determine the text recognition result corresponding to the unit of time as the ending label; if the unit of time is an intermediate unit of time, the attention decoder determines the text recognition result corresponding to the unit of time from the text library corresponding to the language recognition result corresponding to the unit of time based on the second attention of the latent vector in the unit of time and the text recognition results corresponding to the unit of time before the unit of time; and obtain the text sequence recognition result based on the starting label corresponding to the starting unit of time, the text recognition results corresponding to multiple intermediate unit of time, and the ending label corresponding to the ending unit of time.

[0122] In one embodiment, the training module 504 is specifically used to obtain the first position of each language recognition result in the language sequence recognition result; obtain the preceding language sequence samples corresponding to each language recognition result based on the language samples preceding the first position in the language sequence samples; construct a first conditional probability function corresponding to each language recognition result based on each language recognition result in the language sequence recognition result and the preceding language sequence samples corresponding to each language recognition result; and construct a first loss function based on multiple first conditional probability functions; obtain the second position of each text recognition result in the text sequence recognition result; and obtain the preceding language sequence samples corresponding to each language recognition result based on the text samples preceding the second position in the text sequence samples. The preceding text sequence samples corresponding to each text recognition result are used; based on each text recognition result and the preceding text sequence samples corresponding to each text recognition result, a second conditional probability function is constructed for each text recognition result, and a second loss function is constructed based on multiple second conditional probability functions; if the first loss function is greater than the first loss function threshold or the second loss function is greater than the second loss function threshold, the model parameters of the audio text recognition model to be trained are adjusted according to the first loss function and the second loss function until the first loss function is less than or equal to the first loss function threshold and the second loss function is less than or equal to the second loss function threshold, thus obtaining the audio text recognition model.

[0123] In one embodiment, such as Figure 6 As shown, an audio text recognition model training method and an audio text recognition device are provided, including: a sequence acquisition module 600, a recognition module 602, and an output module 604, wherein:

[0124] The sequence acquisition module 600 is used to acquire the audio feature sequence corresponding to the audio to be identified.

[0125] The recognition module 602 is used to input the above audio feature sequence into the audio text recognition model, which identifies the language tags corresponding to each frame of audio features in the audio feature sequence, and identifies the text sequence corresponding to the audio feature sequence from the text library corresponding to the language tags based on the audio feature sequence and the language tags corresponding to each frame of audio features; the audio text recognition model is trained according to the above method.

[0126] Output module 604 is used to determine the audio text corresponding to the audio to be recognized based on the text sequence.

[0127] In one embodiment, the sequence acquisition module 600 is specifically used to convert the spectrogram corresponding to the audio to be identified into a Mel spectrum through a Mel scale filter bank; and to generate a Mel spectrum feature sequence corresponding to the audio to be identified based on the Mel spectrum features of each frame in the Mel spectrum, as an audio feature sequence.

[0128] In one embodiment, the recognition module 602 is specifically used to generate latent vectors corresponding to each frame of audio features by the audio text recognition model through a shared encoder based on the audio features of each frame in the audio feature sequence; input the latent vectors corresponding to each frame of audio features into the attention decoder, and obtain the language sequence based on the first attention corresponding to the latent vectors corresponding to each frame of audio features, and obtain the text sequence corresponding to the audio feature sequence based on the language sequence and the second attention corresponding to the latent vectors corresponding to each frame of audio features; the language sequence includes the language label corresponding to each frame of audio features.

[0129] The training method of the aforementioned audio text recognition model and the various modules in the audio text recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0130] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a training method for an audio-text recognition model and an audio-text recognition method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0131] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0132] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described training method for the audio text recognition model and the audio text recognition method.

[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the above-described training method for the audio text recognition model and the audio text recognition method.

[0134] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the training method for the audio text recognition model and the audio text recognition method described above.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0136] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0137] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for an audio text recognition model, characterized in that, The method includes: Obtain audio samples and the corresponding audio text; Obtain the audio feature sequence corresponding to the audio sample, and obtain the text sequence sample and language sequence sample corresponding to the audio text; The audio feature sequence is input into the audio text recognition model to be trained. The shared encoder and attention decoder in the audio text recognition model to be trained identify the language sequence recognition result corresponding to the audio feature sequence. The attention decoder then obtains the text sequence recognition result based on the language sequence recognition result and the audio feature sequence. Based on the first loss function corresponding to the language sequence recognition result and the language sequence sample, and the second loss function corresponding to the text sequence recognition result and the text sequence sample, the model parameters of the audio text recognition model to be trained are adjusted until the model training conditions are met to obtain the audio text recognition model.

2. The method according to claim 1, characterized in that, The step of obtaining the audio feature sequence corresponding to the audio sample includes: The spectrograms corresponding to the audio samples are converted into Mel spectrograms using a Mel-scale filter bank. Based on the Mel spectrum features of each frame in the Mel spectrum, a Mel spectrum feature sequence corresponding to the audio sample is generated as the audio feature sequence.

3. The method according to claim 1, characterized in that, The step of identifying the language sequence recognition result corresponding to the audio feature sequence through the shared encoder and attention decoder in the audio-text recognition model to be trained, and obtaining the text sequence recognition result through the attention decoder based on the language sequence recognition result and the audio feature sequence recognition, includes: The shared encoder generates a latent vector corresponding to each audio feature group based on each audio feature group in the audio feature sequence; the audio feature group contains the audio features in the audio feature sequence per unit time. The latent vectors corresponding to each audio feature group are input into the attention decoder. The attention decoder obtains the language sequence recognition result based on the first attention corresponding to the latent vectors corresponding to each audio feature group, and obtains the text sequence recognition result based on the language sequence recognition result and the second attention corresponding to the latent vectors corresponding to each audio feature group.

4. The method according to claim 3, characterized in that, The process of obtaining the language sequence recognition result by the attention decoder based on the first attention corresponding to the latent vectors of each audio feature group includes: For each unit of time, if the unit of time is the starting unit of time, the language recognition result corresponding to the unit of time is determined as the starting label; if the unit of time is the ending unit of time, the language recognition result corresponding to the unit of time is determined as the ending label; if the unit of time is an intermediate unit of time, the attention decoder determines the language recognition result corresponding to the unit of time based on the first attention of the latent vector within the unit of time and the language recognition result corresponding to the unit of time before the unit of time. Based on the starting label corresponding to the starting unit time, the language identification results corresponding to multiple intermediate unit times, and the ending label corresponding to the ending unit time, the language sequence identification results are obtained.

5. The method according to claim 3, characterized in that, The process of obtaining text sequence recognition results based on the language sequence recognition results and the second attention corresponding to the latent vectors of each audio feature group includes: For each unit of time, if the unit of time is the starting unit of time, the text recognition result corresponding to the unit of time is determined as the starting label; if the unit of time is the ending unit of time, the text recognition result corresponding to the unit of time is determined as the ending label; if the unit of time is an intermediate unit of time, the attention decoder determines the text recognition result corresponding to the unit of time from the text library corresponding to the language recognition result corresponding to the unit of time based on the second attention of the latent vector in the unit of time and the text recognition result corresponding to the unit of time before the unit of time. The text sequence recognition result is obtained based on the start label corresponding to the start unit time, the text recognition results corresponding to multiple intermediate unit times, and the end label corresponding to the end unit time.

6. The method according to claim 1, characterized in that, The language sequence recognition result includes the language recognition result for each unit of time, and the text sequence recognition result includes the text recognition result for each unit of time. The step of adjusting the model parameters of the audio-text recognition model to be trained based on the first loss function corresponding to the language sequence recognition result and the language sequence sample, and the second loss function corresponding to the text sequence recognition result and the text sequence sample, until the model training conditions are met, to obtain the audio-text recognition model includes: Obtain the first position of each language recognition result in the language sequence recognition result, and obtain the preceding language sequence sample corresponding to each language recognition result based on the language samples before the first position in the language sequence sample. Based on the language sequence recognition results and the preceding language sequence samples corresponding to each language recognition result, a first conditional probability function corresponding to each language recognition result is constructed, and a first loss function is constructed based on multiple first conditional probability functions. Obtain the second position of each text recognition result in the text sequence recognition result, and obtain the preceding text sequence sample corresponding to each text recognition result based on the text sample before the second position in the text sequence sample; Based on each text recognition result and the preceding text sequence sample corresponding to each text recognition result in the text sequence recognition results, a second conditional probability function corresponding to each text recognition result is constructed, and a second loss function is constructed based on multiple second conditional probability functions; If the first loss function is greater than the first loss function threshold or the second loss function is greater than the second loss function threshold, the model parameters of the audio text recognition model to be trained are adjusted according to the first loss function and the second loss function until the first loss function is less than or equal to the first loss function threshold and the second loss function is less than or equal to the second loss function threshold, thus obtaining the audio text recognition model.

7. An audio text recognition method, characterized in that, The method includes: Obtain the audio feature sequence corresponding to the audio to be identified; The above audio feature sequence is input into the audio text recognition model, which identifies the language tags corresponding to each frame of audio features in the audio feature sequence, and identifies the text sequence corresponding to the audio feature sequence from the text library corresponding to the language tags based on the audio feature sequence and the language tags corresponding to each frame of audio features; the audio text recognition model is trained according to the method described in any one of claims 1 to 6. The audio text corresponding to the audio to be identified is determined based on the text sequence.

8. The method according to claim 7, characterized in that, The step of obtaining the audio feature sequence corresponding to the audio to be identified includes: The spectrogram corresponding to the audio to be identified is converted into a Mel spectrum using a Mel-scale filter bank. Based on the Mel spectrum features of each frame in the Mel spectrum, a Mel spectrum feature sequence corresponding to the audio to be identified is generated as the audio feature sequence.

9. The method according to claim 7, characterized in that, The audio text recognition model includes a shared encoder and an attention decoder; the step of identifying language tags corresponding to each frame of audio features in the audio feature sequence by the audio text recognition model, and identifying the text sequence corresponding to the audio feature sequence from the text library corresponding to the language tags based on the audio feature sequence and the language tags corresponding to each frame of audio features, includes: The audio text recognition model generates latent vectors corresponding to the audio features of each frame based on the audio features of each frame in the audio feature sequence through the shared encoder; The latent vectors corresponding to the audio features of each frame are input into the attention decoder. The attention decoder obtains the language sequence based on the first attention corresponding to the latent vectors corresponding to the audio features of each frame, and obtains the text sequence corresponding to the audio feature sequence based on the language sequence and the second attention corresponding to the latent vectors corresponding to the audio features of each frame. The language sequence includes the language label corresponding to the audio features of each frame.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images