A low-resource language adaptive speech recognition method based on AdLoRA Plus

Abstract

The present application belongs to the field of speech recognition technology. The present application provides a low-resource language adaptive speech recognition method based on AdLoRA Plus. The disclosed embodiment significantly improves the speech recognition accuracy of the Whisper model in a low-resource language environment by introducing the AdLoRA Plus fine-tuning technology. Through the low-rank adapter layer and dynamic learning rate adjustment, efficient parameter updates are achieved under limited training data. By adopting the strategy of freezing basic parameters, only the AdLoRA Plus adapter layer is trained, which significantly reduces the computational overhead and training time, while maintaining pre-training knowledge and preventing overfitting. It has wide adaptability and scalability to ensure excellent performance in practical applications. It effectively improves the speech recognition accuracy, reasoning efficiency and training efficiency, solves many challenges of existing technologies in small languages ​​and specific field applications, and has important practical application value.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of speech recognition technology, and in particular to a low-resource language adaptive speech recognition method based on AdLoRA Plus. Background Art

[0002] With the advancement of artificial intelligence (AI), voice interaction is becoming increasingly important across industries. In a linguistically diverse country like Indonesia, the food delivery industry is undergoing a voice-driven transformation. Consumers are increasingly using voice to communicate with customer service to obtain order status, delivery details, or learn about refunds. However, traditional manual customer service struggles to efficiently handle the massive volume of customer requests, especially when voice recognition inaccuracies affect response speed and quality.

[0003] Existing speech recognition systems perform poorly with low-resource languages ​​like Indonesian and its dialects because they are primarily designed for rich-resource languages ​​like English or Chinese. Indonesian has complex sound inflection patterns and a wide variety of dialects. This, combined with the background noise and varied conversational patterns found in food delivery customer service scenarios, further complicates recognition.

[0004] Related technologies suffer from the following specific flaws: 1. Insufficient training data: This makes it difficult for the model to fully learn language features. 2. Traditional fine-tuning methods are inefficient: They cannot achieve high performance with limited data. 3. Poor performance in specific areas: Under complex business logic and language requirements, such as food delivery customer service, the model performs poorly.

[0005] Therefore, it is necessary to improve one or more problems existing in the above-mentioned related technical solutions.

[0006] It should be noted that this section is intended to provide background or context for the technical solutions of the present disclosure stated in the claims. The description herein is not admitted to be prior art by virtue of being included in this section. Summary of the Invention

[0007] The purpose of the embodiments of the present disclosure is to provide a low-resource language adaptive speech recognition method based on AdLoRA Plus, thereby overcoming one or more problems caused by the limitations and defects of related technologies to at least a certain extent.

[0008] According to an embodiment of the present disclosure, a low-resource language adaptive speech recognition method based on AdLoRA Plus is provided, the method comprising:

[0009] Acquire speech data and perform preliminary transcription on the speech data using the Whisper model to obtain a text dataset;

[0010] Processing the text dataset using a large language model to filter out conversation texts in common scenarios;

[0011] Accurately transcribing the common scene dialogue text to obtain transcribed speech data, and combining the common scene dialogue text to obtain a labeled speech dataset;

[0012] Preprocessing the labeled speech dataset, and dividing the preprocessed labeled speech dataset into a training dataset and a test dataset according to a preset ratio;

[0013] Adding the AdLoRA Plus fine-tuning module to the Whisper model, training using the training dataset and the test dataset, and combining the weights of the Whisper model to obtain a trained optimized model;

[0014] The speech to be recognized is input into the trained optimization model to obtain predicted text data.

[0015] Furthermore, the step of obtaining voice data and performing preliminary transcription on the voice data using the Whisper model to obtain a text dataset includes:

[0016] Determining a target application scenario, data standard, and acquisition path for the voice data to obtain the voice data and perform normalization processing on the voice data;

[0017] Performing preliminary transcription on the speech data using the pre-trained Whisper model to obtain the text dataset;

[0018] The text data set is associated with the voice data, and the consistency between the text data set and the voice data is checked.

[0019] Furthermore, the step of processing the text dataset using a large language model to filter out dialogue texts in common scenarios includes:

[0020] Inputting the text data set and the instructional prompt into the large language model for analysis;

[0021] The large language model outputs the common scenario dialogue text.

[0022] Furthermore, the steps of accurately transcribing the common scene dialogue text to obtain transcribed voice data, and combining the common scene dialogue text to obtain a labeled voice dataset include:

[0023] Recording each of the common scenario conversation texts and recording the duration of each recording to obtain the transcribed voice data;

[0024] All the commonly used scene dialogue texts and their corresponding transcribed speech data are collated to obtain the labeled speech dataset.

[0025] Furthermore, the step of preprocessing the labeled speech dataset and dividing the preprocessed labeled speech dataset into a training dataset and a test dataset according to a preset ratio includes:

[0026] Using a data filler to perform data segmentation and filling on the labeled speech dataset, and performing data allocation for multi-card training on the labeled speech dataset;

[0027] Processing the labeled speech dataset into a preset input format, performing feature extraction on the labeled speech dataset, and standardizing the extracted features to obtain standardized features;

[0028] The standardized features are filled with data and batch sorted, and divided into the training data set and the test data set according to a preset ratio.

[0029] Furthermore, the step of adding an AdLoRA Plus fine-tuning module to the Whisper model, training using the training dataset and the test dataset, and combining the weights of the Whisper model to obtain a trained optimized model includes:

[0030] Configuring the task type and language of the Whisper model, and setting the forced_decoder_ids or suppress_tokens of the Whisper model; wherein the Whisper model includes a feature extraction module, an encoder, and a decoder;

[0031] Adding the AdLoRAPlus fine-tuning module to the query layer and value layer of the encoder and the query layer and value layer of the decoder;

[0032] Use loraplus_lr_ratio to control the learning rate ratio of the low-rank matrix A and the low-rank matrix B in the AdLoRA Plus fine-tuning module, use loraplus_lr_embedding to control the learning rate of the embedding layer, and use RankAllocator to dynamically adjust the ranks of the low-rank matrix A and the low-rank matrix B;

[0033] Freeze the original parameters of the Whisper model;

[0034] Training the Whisper model using the training data set to obtain an AdLoRA Plus model;

[0035] Merging the weights of the AdLoRA Plus model and the Whisper model to obtain an optimized model;

[0036] The optimization model is tested using the test data set to obtain a trained optimization model.

[0037] Furthermore, the step of training the Whisper model using the training data set to obtain the AdLoRAPlus model includes:

[0038] Setting hyperparameters according to the configuration; wherein the hyperparameters include at least learning rate, batch size, and number of gradient accumulation steps;

[0039] Enable half-precision training. If fp16 is set to True, the Whisper model is optimized using the compiler.

[0040] Use the training dataset to train, check the performance of the model regularly according to the set number of evaluation steps, and output the log to tensorboard;

[0041] Resume or continue training based on the saved checkpoint, saving the final state of the Whisper model and the optimal model to the output directory.

[0042] Furthermore, the step of merging the weights of the AdLoRA Plus model and the Whisper model to obtain an optimized model includes:

[0043] Check whether the AdLoRA Plus model path exists. If so, load the configuration parameters of the AdLoRA Plus model from the path. Use the Whisper model path obtained from the configuration to load the Whisper model and load it into the device. If the configuration specifies that the model should only be loaded locally, set local_files_only=True.

[0044] Use the merge_and_unload() method to merge the parameters of the AdLoRA Plus model and the weights of the Whisper model into an overall model, and clear the AdLoRA Plus model itself to ensure that the parameters of the AdLoRA Plus model are correctly merged into the Whisper model to form the optimized model; wherein, after the merger is completed, the optimized model is set to evaluation mode.

[0045] Furthermore, the step of inputting the speech to be recognized into the trained optimization model to obtain predicted text data includes:

[0046] Preprocessing the speech to be recognized;

[0047] The pre-processed speech to be recognized is input into the feature extraction module of the trained optimization model. After converting the one-dimensional audio waveform data into a two-dimensional spectrogram through short-time Fourier transform, a mel filter is used to generate a mel spectrogram. The mel spectrogram is subjected to dimensionality reduction processing using a convolutional layer to extract local features in time and frequency to generate audio features.

[0048] The audio features are input into the encoder of the trained optimization model, position codes are added to the audio features, and the audio features are passed through a multi-layer encoder to generate global semantic information containing speech features; wherein each layer of the encoder includes a multi-head self-attention module, a residual connection and normalization module, and a multi-layer perceptron module. The multi-head self-attention module is used to capture long-range dependencies by calculating the global correlation of features; the residual connection and the normalization module are used to ensure the stability of feature distribution; and the multi-layer perceptron module is used to learn high-order nonlinear relationships between features.

[0049] Inputting historical text data and the global semantic information into the multi-layer decoder of the trained optimization model, adding learnable positional encoding to the historical text data and the global semantic information, modeling the positional information of the text sequence, and outputting a probability distribution to represent the tag sequence of the current time step; wherein, the decoder includes a self-attention module, a cross-attention module, and a multi-layer perceptron module, the self-attention module is used to capture the global correlation of the decoder input and model the dependency between historical generated texts; the cross-attention module is used to combine the information of the encoder output and the decoder input to achieve alignment of audio features with text semantics through the cross-attention mechanism;

[0050] The tag sequence is decoded into text, restored to human-readable sentences using a predefined vocabulary or subword decomposition method, spliced ​​together according to chronological order, and punctuation and format correction are added. The text results are verified using grammar and spelling check tools to generate predicted text data.

[0051] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0052] In the embodiments of the present disclosure, through the above-mentioned low-resource language adaptive speech recognition method based on AdLoRA Plus, on the one hand, by introducing the AdLoRA Plus fine-tuning technology, the speech recognition accuracy of the Whisper model is significantly improved in a low-resource language environment. Through the low-rank adapter layer and dynamic learning rate adjustment, efficient parameter updates are achieved with limited training data. By adopting the strategy of freezing the basic parameters, only the AdLoRA Plus adapter layer is trained, which significantly reduces the computational overhead and training time, while maintaining pre-training knowledge and preventing overfitting. On the other hand, the method has wide adaptability and scalability, ensuring excellent performance in practical applications. It effectively improves the speech recognition accuracy, reasoning efficiency and training efficiency, solves many challenges of existing technologies in small languages ​​and specific field applications, and has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0054] Figure 1 A diagram showing the steps of a low-resource language adaptive speech recognition method based on AdLoRA Plus in an exemplary embodiment of the present disclosure is shown;

[0055] Figure 2 A specific flow chart of a low-resource language adaptive speech recognition method based on AdLoRA Plus in an exemplary embodiment of the present disclosure is shown;

[0056] Figure 3 A schematic diagram illustrating an attention mechanism in an exemplary embodiment of the present disclosure;

[0057] Figure 4 A flowchart of inference of a trained optimization model in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0058] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0059] In addition, the accompanying drawings are merely schematic illustrations of embodiments of the present disclosure and are not necessarily drawn to scale. Like reference numerals in the figures represent like or similar parts, and thus repeated descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically separate entities.

[0060] This example implementation provides a low-resource language adaptive speech recognition method based on AdLoRA Plus. Figure 2 As shown in , the low-resource language adaptive speech recognition method based on AdLoRA Plus may include: steps S101 to S106.

[0061] Step S101: Acquire speech data and perform preliminary transcription on the speech data using the Whisper model to obtain a text dataset;

[0062] Step S102: Processing the text dataset using a large language model to filter out dialogue texts in common scenarios;

[0063] Step S103: accurately transcribing the common scene dialogue text to obtain transcribed voice data, and combining the common scene dialogue text to obtain a labeled voice dataset;

[0064] Step S104: preprocessing the labeled speech dataset, and dividing the preprocessed labeled speech dataset into a training dataset and a test dataset according to a preset ratio;

[0065] Step S105: Adding an AdLoRA Plus fine-tuning module to the Whisper model, training using the training dataset and the test dataset, and combining the weights of the Whisper model to obtain a trained optimized model;

[0066] Step S106: inputting the speech to be recognized into the trained optimization model to obtain predicted text data.

[0067] The above-mentioned low-resource language adaptive speech recognition method based on AdLoRA Plus significantly improves the speech recognition accuracy of the Whisper model in low-resource language environments by introducing AdLoRA Plus fine-tuning technology. Through the low-rank adapter layer and dynamic learning rate adjustment, efficient parameter updates are achieved with limited training data. By adopting a strategy of freezing basic parameters and training only the AdLoRA Plus adapter layer, the computational overhead and training time are significantly reduced, while preserving pre-trained knowledge and preventing overfitting. On the other hand, this method has wide adaptability and scalability, ensuring excellent performance in practical applications. It effectively improves speech recognition accuracy, inference efficiency, and training efficiency, solves many challenges of existing technologies in small language and specific field applications, and has important practical application value.

[0068] Below, we will refer to Figures 1 to 4 Each step of the above-mentioned low-resource language adaptive speech recognition method based on AdLoRA Plus in this example implementation is described in more detail.

[0069] In steps S101 and S102, voice data is acquired and initially transcribed using the Whisper model to obtain a text dataset, which is then processed using a large language model to filter out text from common conversation scenarios.

[0070] For example, we collected customer service call logs from multiple Indonesian food delivery platforms (such as GoFood and GrabFood) to ensure that the data covers common scenarios such as order inquiries, delivery status inquiries, refund processing, and complaint handling. By collecting data from multiple platforms, we ensure data diversity and representativeness.

[0071] Preprocessing and transcription: Use the pre-trained Whisper model to convert raw speech data into high-quality text data. This data is then processed by the Large Language Model (LLM) to generate multi-turn long conversation scenarios, providing rich annotation information for subsequent fine-tuning.

[0072] Accurate transcription: High-precision transcription of the obtained text data ensures that the speech and text content are completely consistent, providing a reliable foundation for model training.

[0073] Time alignment. Marking the duration of each sentence ensures that the model can leverage time alignment information for deeper learning, especially in multi-turn conversations.

[0074] In step S103 and step S104, the common scene dialogue text is accurately transcribed to obtain transcribed voice data, and the common scene dialogue text is combined to obtain a labeled voice data set.

[0075] For example, the labeled speech dataset is preprocessed and the preprocessed labeled speech dataset is divided into a training dataset and a test dataset according to a preset ratio.

[0076] Load the Whisper model. Download and load a suitable Whisper pre-trained model (such as the medium version) to ensure that the model can efficiently process speech data in multiple languages ​​and scenarios.

[0077] Preprocess the voice data. Perform preprocessing on the collected voice data, including format conversion and audio normalization, to ensure that the data meets the input requirements of the Whisper model. This includes noise removal, sampling rate adjustment, and volume normalization.

[0078] Extract high-level features. Input the preprocessed audio data into the loaded Whisper model to generate feature representations such as mel-spectrograms, improving the model's recognition accuracy.

[0079] Text label conversion: Convert annotated text labels into token IDs to ensure consistency between text data and audio data, facilitating joint model training.

[0080] Data alignment and padding: Use data padding to unify audio and text sequences of different lengths into a fixed length to ensure the efficiency and stability of the model during training.

[0081] In step S105, an AdLoRA Plus fine-tuning module is added to the Whisper model, and the training data set and the test data set are used for training, and the weight of the Whisper model is combined to obtain a trained optimization model.

[0082] For example, we define the AdLoRA+ module (i.e., the AdLoRA Plus fine-tuning module): by introducing a low-rank matrix to adjust the model weights, we can reduce the number of parameters and computational overhead while improving the adaptability and generalization ability of the model.

[0083] Insert the AdLoRA+ module: Load the pre-trained Whisper model and embed it into the AdLoRA+ module, ready for fine-tuning. Ensure the model can run on the CPU or GPU and supports local file loading to avoid unnecessary network downloads.

[0084] Set the optimizer and learning rate scheduler: Select the AdamW optimizer and the linear learning rate scheduler to ensure smooth convergence of the model during training.

[0085] Freeze basic parameters. During the fine-tuning phase, freeze the original parameters of the Whisper model and only update the parameters of the AdLoRA+ adapter layer to speed up training and prevent overfitting.

[0086] Monitor key indicators such as the loss function in real time, regularly evaluate the performance of the model on the validation set, and save the best model to ensure the smooth progress of the training process.

[0087] Merge the adapter layer with the base model. After fine-tuning, merge the AdLoRA+ adapter layer with the base Whisper model to generate a complete optimized model, simplifying the structure and improving inference speed.

[0088] Save the final model. Save the optimized model and its auxiliary components (such as the feature extractor, tokenizer, and processor) to a specified directory to ensure that the model can be easily loaded and used in a production environment.

[0089] The optimized model is evaluated using the test set, covering scenarios such as order inquiry, delivery status inquiry, refund processing, and complaint handling, to verify the robustness and generalization ability of the model.

[0090] Iterative optimization: Adjust the model's hyperparameters or retrain based on the evaluation results to ensure the model achieves optimal performance in all scenarios.

[0091] In step S106 , the speech to be recognized is input into the trained optimization model to obtain predicted text data.

[0092] For example, the process of inferring the speech to be recognized by the trained optimization model includes core steps such as audio data preprocessing, feature extraction, encoder processing, decoder generation, and text post-processing.

[0093] In a specific embodiment, a fine-tuning method based on the AdLoRA+ and Whisper models of the present application is described in detail below with reference to examples and drawings.

[0094] This application relates to the field of speech recognition, and aims to improve the accuracy and robustness of speech recognition in low-resource languages ​​(such as Indonesian) and specific fields (such as the food delivery industry). Through the AdLoRA+ fine-tuning method, this application uses a small amount of domain data to optimize the Whisper model, combines low-rank adapters with multiple learning rate adjustment strategies, and gives full play to the advantages of large-scale pre-trained models, so that the model can still achieve high-precision speech recognition in a low-resource environment. Especially in the food delivery customer service scenario, the optimized model can accurately recognize voice data in different dialects, accents and background noise, thereby improving service quality. By freezing the basic model parameters and updating only the adapter layer, this application accelerates the fine-tuning process, prevents overfitting, and ensures the stability and generalization ability of the model.

[0095] like Figure 2 As shown, Figure 2The specific steps of collecting domain-specific speech data include:

[0096] Clarify target application scenarios and data standards.

[0097] Identify the target application scenario (e.g., customer service voice recognition for Indonesian food delivery) and determine the required voice data standards: 16kHz sampling rate, mono, WAV format, 10 to 30 seconds in length, covering common customer service scenario conversations such as order inquiries, delivery status, refund processing, and complaints.

[0098] Select a data acquisition channel.

[0099] Cooperate with food delivery platforms or customer service companies to obtain desensitized real conversation data to ensure practicality and representativeness; use public voice datasets (such as Mozilla Common Voice or OpenSLR) to screen Indonesian data that meets the requirements and increase data volume and diversity.

[0100] Data normalization processing.

[0101] Use Python's librosa or ffmpeg libraries to unify audio formats, remove severely distorted or noisy samples, and manually check to ensure audio clarity. This includes different regional accents, gender differences, and various noise backgrounds. Categorize and store audio by scene category for easy annotation and processing.

[0102] Figure 2 The specific steps for using the Whisper model to perform preliminary transcription of the collected speech data include:

[0103] Prepare a Whisper model, choosing the appropriate version (such as base, small, or medium) to balance accuracy and efficiency. Use the Python whisper library to load the model, ensuring that necessary tools such as ffmpeg are installed. Convert the audio files to 16kHz, mono WAV format and transcribe them one by one using the Whisper model to generate text output. Select multilingual support or language-specific parameters as needed. After transcription, associate the text with the original audio and check for consistency, providing a foundation for data annotation.

[0104] Figure 2 A large language model is used to process the transcribed text and filter out the common scene dialogue text. The specific steps include:

[0105] Import and build tips.

[0106] The resulting transcribed text is fed into the interactive interface of a large language model (such as GPT-4 or a similar model). To process these texts efficiently, they can be fed into the model in batches. The input consists of the original transcribed text and a prompt that clearly states the model's task. For example, "Please analyze the following text to determine whether it represents a common conversation scenario and extract the key content. Common scenarios include food delivery order confirmation, delivery time discussion, and complaint handling. Output concise and clear results."

[0107] Batch processing and result acquisition.

[0108] In operation, all text is passed to the large language model for processing by building a loop script or calling the model's API. The model returns structured data containing the screening results, such as text labeled as "common scenarios" and the specific scenario type label. If some text cannot be categorized as common scenarios, it can be marked as "other" or stored for later analysis.

[0109] Figure 2 The conversation texts collected in common scenarios are accurately transcribed into labeled speech data. The specific steps include:

[0110] Text preparation.

[0111] Prepare conversational transcripts that have been screened and annotated for common scenarios, ensuring that each transcript is labeled with the relevant scenario (e.g., takeout order confirmation, delivery time discussion, complaint handling, etc.). These transcripts will serve as the basis for recordings, ensuring that each piece of voice data is highly relevant to the specific scenario.

[0112] Precise recording by professionals.

[0113] During the recording process, ensure that the voice recording of each conversation meets certain quality standards to avoid background noise or unclear pronunciation that may affect data quality. To ensure accuracy, the pronunciation of the recorded text must be strictly consistent with the transcript. The duration of each recording should be recorded and associated with the voice data. During this step, professional recording equipment is used to capture clear voice, and the software automatically records the duration of each recording.

[0114] Furthermore, to ensure the timing and semantic consistency of the conversation, the recording staff should try to simulate the rhythm of real-life conversations, ensuring that the duration of each voice data line is consistent with the actual duration of the conversation. During the recording process, the recording software automatically generates start and end timestamps for each recording, accurately recording the duration of the recording.

[0115] Data organization and storage.

[0116] Ultimately, all recorded speech data will be organized into a labeled speech dataset along with corresponding text and scene labels. This data will be stored in a structured format (such as WAV audio files and corresponding text labels) and provide high-quality training data for subsequent feature extraction and model fine-tuning.

[0117] Figure 2 Load the high-quality dataset in . The specific steps include:

[0118] Data loading.

[0119] To load the dataset, use the custom CustomDataset class. This class reads a file containing the path to the speech data and matches it to the annotation file. Initializing the CustomDataset class requires passing in several parameters, such as the path to the speech data file, a text processor (such as WhisperProcessor), audio length, and language settings. Data augmentation configuration can also be configured if necessary. Using these parameters, the dataset is automatically processed and prepared for training.

[0120] Data segmentation and filling.

[0121] During the loading process, ensure that the speech data is segmented and padded as required. This is typically accomplished by using a data padding function (such as DataCollatorSpeechSeq2SeqWithPadding). Padding ensures that speech data of varying lengths are uniformly sized during batch processing, avoiding errors caused by length inconsistencies. The dataset is filtered based on preset minimum and maximum audio lengths to remove samples that do not meet the requirements and ensure data quality.

[0122] Data allocation for multi-card training.

[0123] When training on multiple GPUs, CustomDataset automatically distributes data based on the training configuration (such as local_rank), ensuring that each device is loaded with the appropriate data shard. This step ensures the efficiency and stability of distributed training, fully utilizing the resources of multiple GPUs and accelerating the training process.

[0124] Passed to the trainer.

[0125] The loaded training and test datasets are passed to the Seq2SeqTrainer during the training process for model training and evaluation. This high-quality data lays the foundation for subsequent model fine-tuning and performance improvement, ensuring that the model performs well in real-world applications.

[0126] Figure 2 Preprocess the loaded dataset in . The specific steps include:

[0127] Load the processor and perform data preprocessing.

[0128] First, we load the processor from WhisperProcessor and convert the audio data and text labels into an input format acceptable to the model. The audio data will be converted to a 16kHz sampling rate, and the text labels will be tokenized using the WhisperProcessor tokenizer. If timestamps are enabled, the processor will automatically extract and add timestamp information to the audio data.

[0129] During preprocessing, the CustomDataset class processes each sample individually to ensure that the audio data conforms to the model input format. Audio samples that do not meet the required length are automatically filtered. If data augmentation is configured, the audio data will be enhanced, such as by adding background noise or cropping the audio, to improve the model's generalization capabilities.

[0130] Feature extraction.

[0131] In addition, the preprocessing step also includes extracting high-level audio features such as Mel-spectrogram and MFCC (Mel-frequency cepstral coefficients), and normalizing the extracted features to ensure that they meet the model's input requirements. These high-level features help the model better understand the content of the speech signal.

[0132] Data filling and batch organization.

[0133] The processed data is padded using the DataCollatorSpeechSeq2SeqWithPadding class to ensure that the audio and text lengths of each batch are consistent. During the batch collation process, any samples that do not meet the standard length will be padded to a uniform length to ensure efficient training.

[0134] Figure 2 Load the pre-trained Whisper model in . The specific steps include:

[0135] Load the base model.

[0136] In the implementation, first load the Whisper model from the specified base model path (e.g., openai / whisper-large). Use the WhisperFor ConditionalGeneration.from_pretrained() method to load the model, and set relevant configurations as needed, such as whether to use 8-bit quantization (use_8bit) and whether to load the local model file (local_files_only).

[0137] Configuring task type and language

[0138] When loading a model, you can automatically configure it for a specific task. WhisperFor ConditionalGeneration supports specifying the task type (e.g., `transcribe` for transcription, `translate` for translation) via the `task` parameter. Additionally, you can specify the language used during training via the `language` parameter, ensuring that the model correctly recognizes and generates text in the corresponding language when processing input audio.

[0139] Decoder settings.

[0140] After loading the model, you can further adjust its configuration, such as setting forced_decoder_ids or suppress_tokens, to ensure the decoding process meets the task requirements. If fine-tuning or enhancement is required, you can load the adapter layer (AdLoRA+) to further improve the model's performance. This will be explained in detail in the following steps.

[0141] Figure 2 In

[15] , the AdLoRA+ low-rank matrix is ​​added to the original model (i.e., the Whisper model). The core idea is to enhance the adaptability of the Whisper model in specific low-resource speech recognition tasks by introducing a low-rank matrix, especially when processing low-resource languages ​​such as Indonesian, significantly improving the model's performance. In specific implementations, the AdLoRA+ method dynamically adjusts the rank of the low-rank matrix and uses different learning rate adjustment mechanisms (such as loraplus_lr_ratio and loraplus_lr_embedding), enabling the adapter to accurately optimize for specific tasks, thereby improving task performance. Specifically, the following steps are included:

[0142] Low-rank matrix is ​​introduced.

[0143] The AdLoRA+ adapter is inserted into key layers of the Whisper model, specifically between the self-attention mechanism and the feedforward network in the Transformer module. By adding low-rank matrix layers to these modules, AdLoRA+ is able to inject task-specific information without changing most model parameters, improving the model's accuracy on low-resource speech data. The introduction of low-rank matrices enables the model to flexibly adapt to the demands of new tasks while retaining pretrained knowledge.

[0144] Adjustment of learning rate.

[0145] The learning rate adjustment mechanism of AdLoRA+ is implemented by two main parameters: loraplus_lr_ratio and loraplus_lr_embedding.

[0146] loraplus_lr_ratio controls the learning rate ratio of the low-rank matrices A and B in the LoRA adapter. Matrix A is used to inject task-specific features, while matrix B is used to enhance the representation of these features. By adjusting the learning rates of A and B, loraplus_lr_ratio makes the learning rate of matrix B larger, which can quickly adapt to task requirements, while matrix A maintains the representation of the pre-trained model at a lower learning rate. This ratio is usually set at 2 3 to 2 4 To ensure the reasonable distribution of learning rates of different matrices and improve the adaptability and stability of the model.

[0147] loraplus_lr_embedding is used to control the learning rate of the embedding layer. The embedding layer contains a large number of parameters and is crucial for natural language processing tasks. During the fine-tuning process, in order to avoid drastic changes in the embedding layer parameters, loraplus_lr_embedding is usually set to a smaller value (such as 1e-6) to ensure its stability and avoid hindering the model's learning of task-specific features. Based on these parameters, the optimizer divides the model's parameters into multiple groups and assigns a different learning rate to each group. For example, the parameters of the main model use the basic learning rate (lr), the matrices A and B in the LoRA adapter use an adjusted learning rate, and the embedding layer uses a specially set learning rate. In this way, the model can flexibly adjust the learning rate of each module while maintaining pre-training capabilities, avoiding overfitting or underfitting, and improving the performance of low-resource speech recognition tasks.

[0148] Dynamically adjust the rank.

[0149] RankAllocator is an innovation in the AdLoRA+ method, mainly used to dynamically adjust the rank of low-rank matrices. This mechanism determines the rank of low-rank matrices in real time based on training error, gradient changes, and task complexity. When the training error is high, RankAllocator increases the rank of the low-rank matrix to improve the expressiveness of the model; when the error tends to stabilize, the rank is reduced to reduce computational overhead and prevent overfitting. When the gradient of the low-rank matrix changes significantly, RankAllocator increases the rank to help the model better capture complex features; when the gradient changes smoothly, the rank is reduced to avoid wasting resources. RankAllocator also adjusts the rank based on the complexity of the task. In simple tasks, the rank is reduced to prevent overfitting; in complex tasks, the rank is appropriately increased to ensure sufficient expressiveness.

[0150] In the early stages of training, the model needs more capacity to learn the complexity of the task. Therefore, RankAllocator increases the rank of the low-rank matrix to help the model converge quickly. As training progresses and the model approaches convergence, RankAllocator gradually reduces the rank of the low-rank matrix to improve computational efficiency and avoid overfitting.

[0151] The dynamic adjustment mechanism of RankAllocator works in conjunction with the learning rate adjustment strategy to ensure that the rank of the low-rank matrix and the learning rate can be flexibly adjusted according to task requirements. Through this collaborative optimization, the AdLoRA+ method can effectively improve performance in low-resource speech recognition tasks, especially in low-resource languages ​​such as Indonesian, fully leveraging the advantages of large-scale pre-trained models.

[0152] The Whisper model is an end-to-end speech recognition model based on the Transformer architecture. Its core components include an encoder and a decoder, which interact to extract features from audio signals and generate text output. Whisper's encoder utilizes a multi-layer self-attention mechanism, with each layer comprising multi-head self-attention, a feedforward network, residual connections, and layer normalization. The decoder relies on a similar structure to the encoder to generate speech output, and the attention mechanism plays a crucial role in the model.

[0153] In the Transformer architecture, such as Figure 3Figure 2 shows a schematic diagram of the attention mechanism. The core self-attention mechanism computes the relationship between different positions in the input sequence using three representations: query, key, and value. Specifically, each input unit participates in the calculation of self-attention weights by generating corresponding queries, keys, and values. During this process, the query layer (Q), key layer (K), and value layer (V) combine their respective linear transformations with the input data representation to calculate the attention weight for each input. Ultimately, a weighted average is used to obtain a representation for each input.

[0154] In the Whisper model, Q, K, and V layers are typically generated from input features via linear transformations within each layer of the multi-head self-attention mechanism. These transformations are performed using parameter matrices learned during training. Typically, Q, K, and V are mapped using different matrices, and their sizes determine the model's expressiveness and computational complexity. Within each self-attention layer, the Q, K, and V vectors are used to calculate a self-attention score, which determines how each position attends to information at other positions. Specifically, Q and K are dot-producted to calculate relevance (i.e., attention score). These are then normalized using Softmax to generate weighted coefficients, which are then multiplied by the V vector and combined into the output.

[0155] When fine-tuning the Whisper model, the AdLoRAPlus method introduces a low-rank matrix adjustment strategy to effectively reduce the scale of parameter updates while retaining sufficient learning capacity. During Whisper fine-tuning, AdLoRAPlus optimizes the model by adjusting only the adapter layer in the network without directly changing the original weights. This approach significantly reduces training time and storage requirements, and performs particularly well in low-resource environments.

[0156] Specifically for the fine-tuning of Whisper, the main location where AdLoRAPlus is inserted is the Q and V matrices in the self-attention mechanism. The traditional approach is to directly adjust the weight parameters of these matrices, while AdLoRAPlus performs fine-tuning by inserting low-rank adapter layers into these matrices. Specifically, AdLoRAPlus adds a low-rank matrix after the Q and V matrices in each self-attention layer. The advantage of this is that through the training of the low-rank matrix, the adaptive adjustment of the model can be achieved without significantly increasing the computational overhead. The learning ability of the low-rank matrix is ​​sufficient to capture the specific patterns required for different tasks, while avoiding the need for large-scale updates to the entire model.

[0157] In terms of implementation, the adapter layer of AdLoRAPlus is implemented by adding a pair of matrices (usually W A and WB ) is accomplished. For each matrix in Q and V, AdLoRAPlus multiplies the original weight matrix with a low-rank matrix. This low-rank matrix does not directly affect the original model parameters. During training, only the adapter layer parameters are updated, while the original Q and V matrices remain unchanged. This approach enables AdLoRAPlus to achieve superior results for fine-tuning specific datasets while using a smaller number of parameters.

[0158] In summary, the AdLoRA Plus method avoids the need to fully train the entire Whisper model by inserting low-rank matrices into the Q and V matrices, and only adjusts the parameters of the adapter layer. This method effectively reduces computational complexity while still retaining sufficient flexibility to adapt to new task requirements.

[0159] Figure 2 Freeze the original parameters of the Whisper model in . During the specific implementation process, you first need to ensure that the original parameters in the Whisper model are frozen to prevent them from being updated during the training process. These frozen parameters usually include pre-trained parameters such as the encoder and decoder weights, position embeddings, etc. By setting the requires_grad attribute of these parameters to False, you can prevent their gradient calculation and update, thereby avoiding modification. This can effectively reduce video memory usage and computational overhead, and focus computing resources on the parts that need to be optimized. After freezing the original parameters, ensure that the parameters of the newly added AdLoRA+ module are in a trainable state, allowing them to be updated during training. Finally, check the model to confirm that the original parameters have been frozen and the parameters of the newly added modules are still trainable.

[0160] Figure 2 Model training. Specifically includes the following steps:

[0161] Hyperparameter configuration.

[0162] First, hyperparameters are set according to the configuration, including learning rate, batch size, and number of gradient accumulation steps. During training, the model saves checkpoints and performs regular evaluations based on the set number of steps to monitor training performance.

[0163] Enable half-precision training and compiler optimizations.

[0164] To speed up training and reduce video memory consumption, half-precision training (fp16) is usually enabled, which can reduce memory usage and improve computational efficiency. If fp16 is set to True, the Pytorch 2.0 compiler will also be used, which helps further improve training efficiency, especially when processing large-scale models.

[0165] Save and evaluate regularly.

[0166] At regular intervals, the training process saves the current model state and evaluates its performance on the validation set. If the model performs well in the evaluation, the optimal model is saved to ensure that the best training results are preserved.

[0167] During training, Seq2SeqTrainer uses the provided training and evaluation sets and periodically checks the model's performance based on the specified number of evaluation steps. As training progresses, the model continuously adjusts its parameters, learns task-specific features, and gradually optimizes its weights. To ensure training stability, training logs are output to Tensorboard, making it easier to review training progress and debug.

[0168] After training is completed, the model will be restored or continue training based on the saved checkpoint, and the final state of the model and the optimal model will be saved in the specified output directory for subsequent use or further tuning.

[0169] Figure 2 The trained AdLoRA Plus model is merged with the original model weights. The specific steps include:

[0170] Load the fine-tuned AdLoRA Plus model.

[0171] Checks if the AdLoRA Plus model path exists and loads the AdLoRA Plus model configuration parameters from that path. Then, using the base model path obtained from the configuration, loads Whisper's base model and loads it into the device (such as the CPU or GPU). If the configuration specifies that the model should only be loaded locally, set local_files_only=True.

[0172] Merge the AdLoRA Plus model with the base model.

[0173] Use the merge_and_unload() method to merge the parameters of the AdLoRA Plus model with the weights of the base model into an overall model and clear the LoRA module itself to reduce memory usage. This step ensures that the parameters of the LoRA module are correctly merged into the base model to form the final optimized model.

[0174] Sets the evaluation mode.

[0175] After the merge is complete, use the model.eval() method to set the model to evaluation mode to ensure that it will not undergo any further training after the merge. This step ensures that the model is in the best state during inference and avoids unnecessary gradient calculations.

[0176] Save the merged model.

[0177] Saves the combined model, feature extractor, tokenizer, and processor to the specified directory. The save directory is dynamically generated based on the base model path in the LoRA configuration. If the base model path ends with a slash, it will be stripped. The saved folder name includes the base model name and a "finetune" suffix to indicate that it is a fine-tuned model. To ensure that large models are stored correctly, the save operation will split the file into 4GB chunks.

[0178] Evaluate the combined model,

[0179] After the model merge is complete, the final model is evaluated to verify its performance. The evaluation process involves loading the merged model and processor, switching it to evaluation mode, and performing inference on the test set. By comparing the model's predictions with the true labels of the test set, evaluation metrics (such as character error rate (CER) or word error rate (WER)) are calculated to measure the model's performance on the actual task. The evaluation results help confirm whether the merged model retains the ability of the fine-tuned model while maintaining the generalization performance of the base model. This step is critical to ensuring model quality and verifying the correctness of the merge operation.

[0180] In a specific embodiment, Figure 4 As shown in Figure 1, it is the inference flowchart of the trained optimization model.

[0181] The Whisper model's audio data inference process includes core steps such as audio data preprocessing, feature extraction, encoder processing, decoder generation, and text post-processing. Specifically, it includes the following steps:

[0182] Audio data preprocessing.

[0183] Specifically, the raw audio input data requires uniform preprocessing. Specifically, all audio is resampled to 16kHz to meet the model input requirements, and denoising is performed to reduce background noise interference. Longer audio data is segmented into fixed-length segments (e.g., 30 seconds) to facilitate model processing while maintaining semantic continuity. Furthermore, the audio is amplitude normalized to maintain consistent signal strength, thereby enhancing the model's inference stability.

[0184] Feature extraction.

[0185] After the speech data is preprocessed, the audio is converted into a logarithmic Mel spectrogram for feature extraction.

[0186] Specifically, after audio preprocessing is complete, audio feature extraction is performed. The one-dimensional audio waveform data is converted into a two-dimensional spectrogram using a short-time Fourier transform (STFT). A mel-spectrogram is then generated using a mel filter. To enhance feature robustness, data augmentation techniques such as time and frequency masking can be applied to the mel-spectrogram to simulate real-world speech variations. Furthermore, a convolutional layer is used to reduce the dimensionality of the mel-spectrogram, extracting local features in time and frequency to generate a high-dimensional, dense feature representation that meets the model input requirements.

[0187] Encoder processing.

[0188] The audio features are then fed into a Transformer encoder for processing. Positional encoding is first added to the features, enabling the model to capture the time-step information in the audio sequence. The audio features pass through a multi-layer Transformer encoder, each of which contains a multi-head self-attention module, a residual connection and normalization module, and a multi-layer perceptron module. The multi-head self-attention module captures long-range dependencies by calculating the global correlation of features, while the residual connection and normalization module ensure the stability of the feature distribution. The multi-layer perceptron module further learns high-order nonlinear relationships between features. Ultimately, the encoder generates a high-dimensional representation that contains the global semantic information of the speech features, providing contextual information for the decoder.

[0189] Decoder generation.

[0190] During the decoding phase, the Transformer decoder receives both the audio feature representation and historical text data from the encoder as input. The historical text data includes the text tags generated at the previous moment, and is semantically fused with the audio features through a cross-attention mechanism. The decoder first adds a learnable positional encoding to the input data to model the positional information of the text sequence. The multi-layer decoder consists of a self-attention module, a cross-attention module, and a multi-layer perceptron module. The self-attention module is used to capture the global correlation of the decoder input and model the dependency between historically generated texts; the cross-attention module combines the information of the encoder output and the decoder input to achieve alignment of audio features with text semantics through a cross-attention mechanism. The final output of the decoder is a probability distribution that represents the tag sequence at the current time step.

[0191] Text post-processing.

[0192] The token sequence generated by the decoder is restored to the complete text through a post-processing step.

[0193] First, the token sequence is decoded into text and restored to human-readable sentences using a predefined vocabulary or subword decomposition method (such as Byte Pair Encoding, BPE). For multiple text segments generated from long audio, they are spliced ​​together according to chronological order, and punctuation and formatting corrections are added. In addition, grammar and spelling check tools are applied to verify the text results to correct possible spelling or grammatical errors. Finally, the generated text results are formatted into the required output form (such as JSON or plain text) and saved to the specified location, while supporting real-time display or export. This complete process gradually converts audio data from raw signals into high-precision text representations and ensures the accuracy and usability of the generated results.

[0194] Through the above-mentioned low-resource language adaptive speech recognition method based on AdLoRA Plus, by introducing the AdLoRA+ fine-tuning technology, the speech recognition accuracy of the Whisper model is significantly improved in a low-resource language environment, especially in the Indonesian takeaway customer service scenario, showing better robustness and accuracy. Traditional models face the problems of insufficient data and poor generalization ability when dealing with low-resource languages. However, this application achieves efficient parameter updates with limited training data through low-rank adapter layers and dynamic learning rate adjustment. By adopting the strategy of freezing basic parameters and training only the AdLoRA+ adapter layer, the computational overhead and training time are significantly reduced, while maintaining pre-training knowledge and preventing overfitting. This method is not only suitable for Indonesian takeaway customer service, but also has wide adaptability and scalability, ensuring excellent performance in practical applications. In summary, this application effectively improves speech recognition accuracy, reasoning efficiency and training efficiency, solves many challenges of existing technologies in small languages ​​and specific field applications, and has important practical application value.

[0195] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification.

[0196] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

Claims

1. A low-resource language adaptive speech recognition method based on AdLoRA Plus, characterized in that: The method includes: Acquire speech data and perform preliminary transcription on the speech data using the Whisper model to obtain a text dataset; Processing the text dataset using a large language model to filter out conversation texts in common scenarios; Accurately transcribing the common scene dialogue text to obtain transcribed speech data, and combining the common scene dialogue text to obtain a labeled speech dataset; Preprocessing the labeled speech dataset, and dividing the preprocessed labeled speech dataset into a training dataset and a test dataset according to a preset ratio; Adding the AdLoRA Plus fine-tuning module to the Whisper model, training using the training dataset and the test dataset, and combining the weights of the Whisper model to obtain a trained optimized model; The speech to be recognized is input into the trained optimization model to obtain predicted text data.

2. The low-resource language adaptive speech recognition method based on AdLoRA Plus according to claim 1, characterized in that The steps of obtaining speech data and performing preliminary transcription on the speech data using the Whisper model to obtain a text dataset include: Determining a target application scenario, data standard, and acquisition path for the voice data to obtain the voice data and perform normalization processing on the voice data; Performing preliminary transcription on the speech data using the pre-trained Whisper model to obtain the text dataset; The text data set is associated with the voice data, and the consistency between the text data set and the voice data is checked.

3. The low-resource language adaptive speech recognition method based on AdLoRA Plus according to claim 1, characterized in that The step of processing the text dataset using a large language model to filter out conversation texts in common scenarios includes: Inputting the text data set and the instructional prompt into the large language model for analysis; The large language model outputs the common scenario dialogue text.

4. The low-resource language adaptive speech recognition method based on AdLoRA Plus according to claim 1, characterized in that The steps of accurately transcribing the common scene dialogue text to obtain transcribed speech data, and combining the common scene dialogue text to obtain a labeled speech dataset include: Recording each of the common scenario conversation texts and recording the duration of each recording to obtain the transcribed voice data; All the commonly used scene dialogue texts and their corresponding transcribed speech data are collated to obtain the labeled speech dataset.

5. The low-resource language adaptive speech recognition method based on AdLoRA Plus according to claim 1, characterized in that The step of preprocessing the labeled speech dataset and dividing the preprocessed labeled speech dataset into a training dataset and a test dataset according to a preset ratio includes: Using a data filler to perform data segmentation and filling on the labeled speech dataset, and performing data allocation for multi-card training on the labeled speech dataset; Processing the labeled speech dataset into a preset input format, performing feature extraction on the labeled speech dataset, and standardizing the extracted features to obtain standardized features; The standardized features are filled with data and batch sorted, and divided into the training data set and the test data set according to a preset ratio.

6. The low-resource language adaptive speech recognition method based on AdLoRA Plus according to claim 1, characterized in that The step of adding the AdLoRA Plus fine-tuning module to the Whisper model, training using the training dataset and the test dataset, and combining the weights of the Whisper model to obtain a trained optimized model includes: Configuring the task type and language of the Whisper model, and setting the forced_decoder_ids or suppress_tokens of the Whisper model; wherein the Whisper model includes a feature extraction module, an encoder, and a decoder; Add the AdLoRA Plus fine-tuning module to the query layer and value layer of the encoder and the query layer and value layer of the decoder; Use loraplus_lr_ratio to control the learning rate ratio of the low-rank matrix A and the low-rank matrix B in the AdLoRA Plus fine-tuning module, use loraplus_lr_embedding to control the learning rate of the embedding layer, and use RankAllocator to dynamically adjust the ranks of the low-rank matrix A and the low-rank matrix B; Freeze the original parameters of the Whisper model; Training the Whisper model using the training data set to obtain an AdLoRA Plus model; Merging the weights of the AdLoRA Plus model and the Whisper model to obtain an optimized model; The optimization model is tested using the test data set to obtain a trained optimization model.

7. The low-resource language adaptive speech recognition method based on AdLoRA Plus according to claim 6, characterized in that The step of training the Whisper model using the training data set to obtain the AdLoRA Plus model includes: Setting hyperparameters according to the configuration; wherein the hyperparameters include at least learning rate, batch size, and number of gradient accumulation steps; Enable half-precision training. If fp16 is set to True, the Whisper model will be optimized using the compiler. Use the training dataset to train, check the performance of the model regularly according to the set number of evaluation steps, and output the log to tensorboard; Resume or continue training based on the saved checkpoint, saving the final state of the Whisper model and the optimal model to the output directory.

8. The low-resource language adaptive speech recognition method based on AdLoRA Plus according to claim 7, characterized in that: The step of merging the weights of the AdLoRA Plus model and the Whisper model to obtain an optimized model includes: Check whether the AdLoRA Plus model path exists. If so, load the configuration parameters of the AdLoRA Plus model from the path. Use the Whisper model path obtained from the configuration to load the Whisper model and load it into the device. If the configuration specifies that the model should only be loaded locally, set local_files_only=True. Use the merge_and_unload() method to merge the parameters of the AdLoRA Plus model and the weights of the Whisper model into an overall model, and clear the AdLoRA Plus model itself to ensure that the parameters of the AdLoRA Plus model are correctly merged into the Whisper model to form the optimized model; wherein, after the merger is completed, the optimized model is set to evaluation mode.

9. The low-resource language adaptive speech recognition method based on AdLoRA Plus according to claim 8, characterized in that Inputting the speech to be recognized into the trained optimization model to obtain predicted text data includes: Preprocessing the speech to be recognized; The pre-processed speech to be recognized is input into the feature extraction module of the trained optimization model. After converting the one-dimensional audio waveform data into a two-dimensional spectrogram through short-time Fourier transform, a mel filter is used to generate a mel spectrogram. The mel spectrogram is subjected to dimensionality reduction processing using a convolutional layer to extract local features in time and frequency to generate audio features. The audio features are input into the encoder of the trained optimization model, position codes are added to the audio features, and the audio features are passed through a multi-layer encoder to generate global semantic information containing speech features; wherein each layer of the encoder includes a multi-head self-attention module, a residual connection and normalization module, and a multi-layer perceptron module. The multi-head self-attention module is used to capture long-range dependencies by calculating the global correlation of features; the residual connection and the normalization module are used to ensure the stability of feature distribution; and the multi-layer perceptron module is used to learn high-order nonlinear relationships between features. Inputting historical text data and the global semantic information into the multi-layer decoder of the trained optimization model, adding learnable positional encoding to the historical text data and the global semantic information, modeling the positional information of the text sequence, and outputting a probability distribution to represent the tag sequence of the current time step; wherein, the decoder includes a self-attention module, a cross-attention module, and a multi-layer perceptron module, the self-attention module is used to capture the global correlation of the decoder input and model the dependency between historical generated texts; the cross-attention module is used to combine the information of the encoder output and the decoder input to achieve alignment of audio features with text semantics through the cross-attention mechanism; The tag sequence is decoded into text, restored to human-readable sentences using a predefined vocabulary or subword decomposition method, spliced ​​together according to chronological order, and punctuation and format correction are added. The text results are verified using grammar and spelling check tools to generate predicted text data.

Images