A method and device for constructing a low-resource speech recognition model

Abstract

The application discloses a kind of construction method and device of low resource speech recognition model, method includes: preprocessing preparation: extracting linear layer weight from pre-training model and configuring LoRA parameter;Initialization construction: construct structured low rank adapter, generate the initial value of A / B matrix that fuses frequency spectrum and phase information, residual processing is carried out to basic weight;Training execution: using low resource dataset, update LoRA parameter by parameter efficient fine-tuning framework;Weight integration and output final model.The application solves the technical difficulties such as low model training efficiency, phase information loss and poor cross-link adaptation in traditional low resource scene by spectrum-phase residual initialization and parameter efficient fine-tuning technology without changing the original model architecture, significantly improves the precision of dialect recognition, rare vocabulary recognition and hotword adaptation.

Description

Technical Field

[0001] This invention relates to the field of automatic speech recognition, and in particular to a method and apparatus for constructing a low-resource speech recognition model based on low-rank adaptive fine-tuning. Background Technology

[0002] In the context of the rapid development of artificial intelligence technology, large pre-trained models have demonstrated outstanding performance and generalization capabilities in various fields such as natural language processing, computer vision, and speech recognition. However, when applying these massive models to specific downstream tasks, fine-tuning is usually required. Traditional full-scale fine-tuning methods require updating all parameters of the model, which incurs enormous computational and storage overhead. This overhead becomes unbearable, especially as model sizes continue to increase. For example, full-scale fine-tuning of a model with billions or even hundreds of billions of parameters may require several terabytes of GPU memory and weeks of training time, which is a huge challenge for resource-constrained individual researchers or small and medium-sized enterprises.

[0003] To address this challenge, Parametric Efficient Fine-Tuning (PEFT) techniques have emerged. The core idea of ​​PEFT is to achieve performance comparable to full fine-tuning on downstream tasks by adjusting only a small subset of parameters in the model, or introducing a few additional parameters. This significantly reduces the computational and storage costs of fine-tuning, making the application of large models more feasible and efficient. Among the various PEFT techniques, Low-Rank Adaptation (LoRA) has become one of the preferred methods for tuning large models due to its characteristics of not modifying the original model structure, ease of implementation, and efficient training. LoRA parameterizes weight updates by injecting a pair of low-rank matrices into the linear layers of a pre-trained model, thereby significantly reducing the number of trainable parameters while maintaining high performance.

[0004] While LoRA performs well in many tasks, its performance often lags behind full-scale fine-tuning in low-resource Automatic Speech Recognition (ASR) tasks. Low-resource ASR refers to speech recognition tasks performed with very limited training data. This is prevalent in many ASR applications involving dialects, minority languages, or specific vertical domains (such as medical and legal fields). Existing LoRA initialization strategies, such as random initialization or Xavier initialization based on a standard normal distribution, are generally generic and do not consider the inherent spectral and phase structure characteristics of speech signals. This causes LoRA to fail to effectively capture the essential features of speech data in the early stages of training, affecting its performance in low-resource ASR tasks. Especially when dealing with dialects or domain-specific hot words with unique phonological features, existing initialization strategies struggle to quickly adapt to these special speech patterns.

[0005] In addition, existing low-resource ASR solutions have the following limitations: First, existing fine-tuning solutions are mostly optimized for single stages (such as dialect conversion, speech recognition, and text correction), lacking a cross-stage parameter collaborative fine-tuning mechanism. Although they ensure the basic performance of a single stage in general scenarios, their ability to adapt to customized scenarios is limited when dealing with dialects with unique phonological characteristics (such as morphological changes in agglutinative languages ​​and tonal differences in dialects) or hot words with domain characteristics. Second, in extremely low-resource scenarios (such as dialect data with less than 3 hours of training data), existing models are unable to achieve rapid convergence through efficient parameter fine-tuning, and cannot accurately capture effective speech and text patterns in scarce data. The overall processing effect is limited, and it is difficult to meet the comprehensive needs of dynamic hot word adaptation, dialect conversion recognition, and efficient error correction in low-resource scenarios.

[0006] Therefore, there is an urgent need for a low-resource speech recognition model construction scheme that can fully utilize the characteristics of speech signals and achieve efficient parameter fine-tuning in order to overcome the above-mentioned technical bottlenecks. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, this invention aims to provide a method and apparatus for constructing a low-resource speech recognition model based on low-rank adaptive fine-tuning. The purpose of this invention is to introduce a spectrum-phase residual initialization strategy, utilizing the natural spectral basis and phase information from signal processing to guide the low-rank subspace of LoRA, thereby enabling faster convergence of the fine-tuning process and greater robustness under low-resource conditions, thus significantly improving the performance of LoRA in low-resource ASR tasks.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: a method for constructing a low-resource speech recognition model, comprising the following steps:

[0009] S1, Preprocessing preparation steps: Based on the pre-trained speech recognition model, perform model preprocessing, complete the configuration of basic data and parameters, and establish the basis for low-rank adaptive fine-tuning of the model.

[0010] S2, Initialization construction steps: Based on the aforementioned low-rank adaptive fine-tuning foundation, perform low-rank adapter structure initialization and residual processing on the model weights;

[0011] S3, Training execution steps: Using a low-resource speech recognition dataset, update the low-rank adapter parameters through a parameter fine-tuning framework, and monitor the training process to optimize the model;

[0012] S4, Weight Integration Step: Integrate the low-rank adapter parameters obtained from step S3 with the model weights after residualization in step S2 to output the final speech recognition model.

[0013] Preferably, the S1 preprocessing preparation step specifically includes:

[0014] S1-1, Model Weight Extraction: Load the pre-trained speech recognition model; locate and select the target linear layer; extract the basic weight matrix W of the target linear layer;

[0015] S1-2, configure key parameters for Low-Rank Adaptive (LoRA);

[0016] S1-3, Resource preparation for Discrete Fourier Transform basis vector (DFT basis) computation.

[0017] Preferably, the S2 initialization construction step specifically includes:

[0018] S2-1, Generate a DFT basis that conforms to the spectral characteristics of speech;

[0019] S2-2, Calculate the initial values ​​of the low-rank matrices A and B of LoRA;

[0020] S2-3, Calculate the residualized basic weight matrix.

[0021] Preferably, the S3 training execution step specifically includes:

[0022] S3-1, Training set data preprocessing; the preprocessing includes at least audio processing, feature extraction, text processing, and data partitioning;

[0023] S3-2, PEFT training framework setup;

[0024] S3-3, Training Iteration and Monitoring.

[0025] Preferably, step S4, the weight integration step, specifically includes:

[0026] S4-1, After training is complete, calculate the final weight increment of LoRA;

[0027] S4-2, Calculate the final weights of the linear layer after fine-tuning;

[0028] S4-3, Update the weights of the linear layers of the model, generate a complete fine-tuned model weight file, and output the final speech recognition model.

[0029] The present invention also provides an apparatus for constructing a low-resource speech recognition model, comprising:

[0030] The preprocessing preparation module is used to preprocess the model based on the pre-trained speech recognition model, complete the configuration of basic data and parameters, and establish the basis for low-rank adaptive fine-tuning of the model.

[0031] The initialization module is used to initialize the low-rank adapter structure based on the low-rank adaptive fine-tuning and to perform residual processing on the model weights.

[0032] The training module is used to update the low-rank adapter parameters through a parameter fine-tuning framework using a low-resource speech recognition dataset, and to monitor the training process to optimize the model.

[0033] The output module integrates the trained low-rank adapter parameters with the residual model weights to output the final speech recognition model.

[0034] Compared with the prior art, the advantages of the present invention are as follows:

[0035] First, this invention proposes an initial parameter construction method based on the magnitude and phase spectra of DFT basis vectors for LoRA fine-tuning training, which extracts the spectral features of speech signals and solves the limitation of existing ASR algorithms that are difficult to fully match the inherent features of speech signals due to the generalization of initialization strategies.

[0036] Secondly, this invention proposes a LoRA optimization method that removes the spectral prior interpretable components from the pre-trained weights. By using residual processing, the training is focused on mid- and low-frequency features, which solves the problem of many parameters and long training time in ASR models under low resource conditions.

[0037] Finally, the convergence monitoring method using gradient signals proposed in this invention can effectively focus on the impact of speech features on the loss, solving the problems of slow convergence and easy overfitting of ASR models under low resource conditions.

[0038] In summary, this invention, through the aforementioned spectrum-phase residual initialization and efficient parameter fine-tuning techniques, effectively solves the technical pain points of low model training efficiency, loss of phase information, and poor cross-stage adaptation in traditional low-resource scenarios without altering the original model architecture, and significantly improves the accuracy of dialect recognition, niche word recognition, and hot word adaptation. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of the method of the present invention;

[0041] Figure 2 This is a schematic diagram of the device architecture of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] This invention proposes a novel LoRA initialization strategy, SPaRLoRA (Spectral-Phase Residual LoRA Initialization), which utilizes a Discrete Fourier Transform (DFT) basis to construct a structured low-rank adapter, explicitly fusing spectral amplitude and phase information, and introducing a residual correction mechanism. This method requires no modification to the model architecture, introduces no inference overhead, and its initial low-rank subspace is highly aligned with the time-frequency characteristics of the speech signal.

[0044] The method flow of the present invention is as follows: Figure 1 As shown, the fine-tuning process of SPaRLoRA's low-resource ASR model mainly includes four stages, each closely linked to the next, from basic data and parameter preparation to the implementation of initialization strategies, and then to training optimization and weight integration, forming a complete technical process.

[0045] This invention provides a method for constructing a low-resource speech recognition model based on low-rank adaptive fine-tuning, comprising the following steps:

[0046] S1, Preprocessing Preparation Steps. The main goal of this step is to provide the basic data and parameter configuration for SPaRLoRA initialization and training, ensuring that subsequent processes can proceed accurately. Specifically, it includes three sub-steps: weight extraction (S1-1), parameter configuration (S1-2), and DFT base resource preparation (S1-3), which are executed sequentially.

[0047] S1-1, Extract weights from the pre-trained speech recognition model. For the selected pre-trained model in the low-resource ASR task, extract the weights of its linear layers. These linear layers are the core targets of LoRA. The specific implementation is as follows:

[0048] First, load the pre-trained model by using a deep learning framework to load the weight file of the pre-trained model and ensure that the data type of the weights is consistent with the subsequent calculations.

[0049] Subsequently, the linear layers were located. By analyzing the model structure, the linear layers that needed to be fine-tuned using LoRA (low-rank adaptive) were selected, and the input dimension (d) and output dimension (k) of each layer were recorded.

[0050] Finally, weight storage is implemented by extracting and storing the basic weight matrix W of the target linear layer separately, providing data support for subsequent residual processing. The output of this step is the basic weight matrix W, which is used for residual calculation in stage S2.

[0051] S1-2, Configure key parameters for low-rank adaptive LoRA. The main goal of this step is to configure key parameters based on the scale of low-resource ASR data and model complexity, while verifying the rationality of the parameters and balancing training efficiency and model recognition performance.

[0052] For example, as shown in the table below, configure the core parameters of LoRA based on the data size and model complexity of low-resource ASR tasks.

[0053]

[0054] Table 1 - Parameter Configuration Table

[0055] First, based on the scale of low-resource ASR data and model complexity, configure the target rank *r*. This controls the dimension of the low-rank subspace of LoRA, affecting the number of parameters and expressive power. It is typically set to 16 or 32, with 32 being preferred in low-resource scenarios. Configure the scaling factor *γ*, which adjusts the contribution of LoRA weight increments. *γ* is determined according to the formula *γ=α / r*, where *α* is the learning rate factor. For example, when the target rank *r*=32, *α* is usually set to 64, thus *γ*=2 according to the formula *γ=α / r*. Configure the imaginary part enable flag, which controls whether to concatenate the imaginary part of the DFT basis to preserve phase information. *use_imag* is set to True by default to preserve phase information. Configure the normalization method, which ensures the stability of the initial norm of the A matrix and avoids gradient anomalies. L2 normalization is configured and applied to each column of the A matrix.

[0056] S1-3: Prepare resources for Discrete Fourier Transform (DFT) basis vector calculation. Select a numerical computation library that supports complex number operations, and pre-calculate the size of the DFT matrix based on the input dimension of the linear layer to ensure accurate and efficient subsequent DFT basis generation.

[0057] The DFT basis is the core carrier for SPaRLoRA initialization, and numerical computation tools and parameters for generating the DFT matrix need to be prepared in advance. Preferably, this step is implemented as follows:

[0058] First, select a numerical library that supports complex number operations to ensure that the real and imaginary parts of the DFT basis vectors can be accurately extracted, which facilitates the calculation of amplitude and phase information;

[0059] Then, dimension matching is performed. Based on the input dimension d of the linear layer recorded in S1-1, the size of the DFT matrix (d×d complex matrix) is determined. Each column corresponds to a basis vector of a frequency component, and the matrix size must be completely consistent with the input dimension of the linear layer.

[0060] Finally, pre-computation optimization is implemented. For linear layers with the same input dimension, the DFT matrix is ​​pre-computed and cached to avoid redundant computation. For example, if multiple linear layers have the same input dimension d=256, a 256×256 DFT matrix is ​​pre-computed and reused, significantly improving efficiency.

[0061] In summary, the S1 preprocessing preparation stage provides the necessary data and parameter foundation for the subsequent initialization construction (S2) through three sub-steps: weight extraction, parameter configuration, and resource preparation. The outputs of each step (such as the weight matrix W, key parameters, and DFT computation resources) are used in the S2 stage.

[0062] S2, Initialization Construction Step. This stage is the core of the SPaRLoRA method. Its main goal is to complete the core operation of Spectrum-Phase Residual LoRA initialization (SPaRLoRA) using the basic weights and parameters prepared in stage S1. Specifically, this stage aims to generate initialization parameters that fit the characteristics of the speech spectrum, introduce the speech spectrum-phase prior through the DFT basis, construct the A / B matrix of LoRA, and perform residual processing on the basic weights, thereby solving the problem of "generalization and lack of signal awareness" in traditional LoRA initialization. The input of this stage depends on the output of stage S1: the basic weight matrix W, the target rank r, the scaling factor γ (calculated from α and r), and the DFT basis computation resources, etc. The entire process includes three key sub-steps in sequence: S2-1 DFT basis generation and selection, S2-2 A / B matrix initial value construction, and S2-3 residual basic weight calculation.

[0063] The initialization and construction phase is described in detail below.

[0064] S2-1, DFT basis generation conforming to speech spectrum characteristics.

[0065] This step generates a DFT basis that fits the spectral characteristics of speech based on the linear layer input dimension d determined in the preprocessing preparation stage, and selects the basis vectors corresponding to the core frequency components.

[0066] First, DFT basis generation is performed. Based on the input dimension d of the linear layer of the pre-trained model (e.g., d=256), a d×d identity matrix I is constructed. d×d The elements on the diagonal of this matrix are 1s, and all other elements are 0s. Performing a Fast Fourier Transform (FFT) on it yields the complete complex form DFT matrix F, which is mathematically represented as:

[0067]

[0068] The element in the m-th row and n-th column (m, n = 0, 1, ..., d-1) of matrix F is... , (where i is the imaginary unit), represents the DFT basis vector element in the m-th row and n-th column. This type of basis vector can capture specific frequency components of the speech signal, conforming to the spectral characteristics of the speech signal.

[0069] Subsequently, core basis vector selection is performed: the first r columns (where r is the target rank configured in stage S1, e.g., r=32) are selected from the DFT matrix F to form the core basis vector set C=Topr{F}. The selection is based on the fact that the first r columns of matrix F correspond to the low-frequency to mid-frequency components where the speech signal energy is most concentrated. This design ensures that the initialized low-rank subspace preferentially aligns with the main features of the speech, thereby maximizing its ability to capture speech features.

[0070] S2-2, Calculate the initial values ​​of the A / B matrix of LoRA.

[0071] This step, based on the core basis vector set C obtained in S2-1, and combined with the requirement to preserve speech phase information (controlled by the use_imag parameter in S1), constructs the initial values ​​of the low-rank matrices A and B of LoRA, ensuring that the initial parameters carry spectral-phase priors.

[0072] In the LoRA framework, for a pre-trained linear layer weight matrix W, its weight update ΔW is represented by a low-rank decomposition, i.e., ΔW = B. A.

[0073] The function of matrix A is dimensionality reduction and feature extraction. It projects the original high-dimensional input features (e.g., dimension d) into a predefined subspace with lower dimensions (e.g., dimension r). It can be understood as an "information filter" or "feature encoder," responsible for extracting the core features most relevant to the downstream task from the input data while ignoring redundant information. In the SPaRLoRA initialization of this invention, the initial value of matrix A is not randomly generated, but constructed from the amplitude and phase spectra of the DFT basis. This allows the "filter" to perceive the speech spectrum and phase structure from the beginning of training, laying the foundation for rapid convergence.

[0074] The function of matrix B is dimensionality enhancement and feature reconstruction. It remaps the low-dimensional features (e.g., dimension r) compressed by matrix A back to the original high-dimensional space (e.g., dimension d) to generate weight updates that match the dimensions of the original weight matrix W. Matrix B acts like a "decoder," reconstructing the compressed task-specific features into weight increments that can be integrated with the existing knowledge of the pre-trained model. In this invention, the initial value of matrix B is set to the transpose of the initial value of matrix A. This symmetric initialization strategy ensures that the low-rank approximation (B1A1) in the initial state can effectively represent the portion of the basic weights that can be explained by the prior speech.

[0075] The specific implementation of step S2-2 includes two sub-steps:

[0076] S2-2-1, Construction of initial values ​​for matrix A.

[0077] First, extract the spectral-phase information of the DFT basis vectors. Preferably, separate the real and imaginary parts of each column in the core basis vector set C, and calculate their magnitude matrices separately. (Characterizing signal strength) and phase matrix ∠C (Characterizing signal timing structure).

[0078] Next, construct and normalize the initial matrix A of matrix A. raw Preferably, the amplitude matrix is... Concatenate the phase matrix ∠C with the phase matrix, then transpose to obtain the initial matrix. Subsequently, regarding A raw Perform L2 normalization to obtain the initial value of matrix A. This normalization operation effectively controls the initial norm of matrix A, avoids gradient anomalies (such as explosion or vanishing) in the early stages of training, and ensures that its column space is well aligned with the frequency components of the speech spectrum.

[0079] S2-2-2, Construction of initial values ​​for matrix B.

[0080] Preferably, to achieve accuracy in the low-rank approximation and simplify calculations, the initial value B1 of matrix B is constructed symmetrically to the initial value A1 of matrix A, and is directly set as the initial value of matrix A. The transpose of a matrix, i.e.:

[0081] B1=A1 T

[0082] This approach maintains the correlation between the column spaces of matrices A and B, making the initial low-rank approximation more accurate and laying the foundation for subsequent low-rank approximation calculations.

[0083] In one specific implementation, assuming a linear layer has an input dimension d=256, an output dimension k=256, and a target rank r=32, then: Matrix A has a dimension of 32×256, and its initial values ​​are constructed from the amplitude and phase information of the first 32 columns (core frequency components) of the 256-dimensional DFT basis. Matrix B has a dimension of 256×32, and its initial values ​​are the transpose of matrix A. The number of trainable parameters is only (256×32)+(32×256)=16,384, far fewer than the 256×256=65,536 parameters required for full fine-tuning of this layer, fully demonstrating parameter efficiency.

[0084] S2-3, Calculate the residualized basic weight matrix.

[0085] This step is another innovation of the SPaRLoRA invention, aiming to remove components in the basic weight matrix that can be interpreted a priori by the spectrum through an initial low-rank approximation. This allows subsequent LoRA training to focus on residual information processing, thereby narrowing the parameter search range and improving training stability in low-resource scenarios. A preferred implementation is as follows:

[0086] First, calculate the initial low-rank matrix. Constructed using S2-2 Using the B1 matrix and the scaling factor γ determined in stage S1 (e.g., γ=2 when r=32, α=64), calculate the initial low-rank matrix:

[0087]

[0088] The matrix The dimension of the matrix is ​​consistent with that of the basic weight matrix W. Its physical meaning is that it represents the part of the basic weight W that can be explained by the DFT basis and corresponds to the low-frequency features in the speech signal.

[0089] Subsequently, residual processing is performed: the initial low-rank matrix is ​​subtracted from the original basic weight matrix W (from stage S1). The residual basic weights W are obtained. res And freeze, the calculation formula is:

[0090] W res =W-

[0091] W res The complex components in the basic weights that cannot be easily explained by spectral priors are retained. In subsequent training phases, these weights will be frozen, and LoRA only needs to learn the residuals, significantly reducing the optimization difficulty.

[0092] Finally, perform bias zeroing: set the bias term of the linear layer to 0. This operation is a standard step in LoRA fine-tuning, ensuring that the output of the model after initialization is consistent with the original pre-trained model, and avoiding the introduction of initial bias that could affect training convergence.

[0093] In summary, the S2 initialization construction phase, through three sub-steps—DFT basis generation, A / B matrix construction, and residual calculation—successfully injected the spectral and phase prior information of the speech signal into the LoRA initialization parameters. This ensured the initial low-rank subspace was aligned with the speech's time-frequency characteristics and prepared the initial model state for the S3 training phase, which focuses on residual learning. The output of this phase is the residual-based weights W. res The initialized A and B matrices will be directly used as inputs for the S3 training execution phase.

[0094] S3, Training execution steps.

[0095] This stage is a crucial implementation step in the Spectrum-Phase Residual Low-Rank Adaptive (SPaRLoRA) method. Its core objective is to efficiently fine-tune the model using a low-resource speech recognition dataset based on the Parameter Efficient Fine-Tuning (PEFT) framework. Only the A and B matrix parameters of the low-rank adaptive (LoRA) method are updated, while all other parameters are frozen. This reduces computational overhead and improves the model's performance in low-resource automatic speech recognition (ASR) tasks. The entire training process consists of three sub-steps: training data preprocessing (S3-1), PEFT training framework setup and execution (S3-2), and training iteration and monitoring (S3-3). These steps are executed sequentially to ensure efficient and controllable training and to provide optimized parameters for the subsequent S4 weight integration stage. The specific optimized process is as follows:

[0096] S3-1, Training Data Preprocessing. This step normalizes the low-resource ASR dataset to improve data quality, ensuring it meets model input requirements and providing a reliable data foundation for fine-tuning training.

[0097] Specifically, the implementation targets low-resource scenarios such as dialects and minority languages, and prioritizes four stages: audio processing, feature extraction, text processing, and dataset partitioning.

[0098] Audio processing primarily addresses the issues of inconsistent quality and mixed formats in the original audio signals. This involves standardizing the original audio files, including setting the sampling rate to, for example, 16kHz (to match the input specifications of most pre-trained speech models), converting to mono, applying noise reduction algorithms (such as spectral subtraction), and performing speech activity detection (VAD, such as energy threshold-based methods) to remove silent segments and retain valid speech signals.

[0099] Feature extraction aims to convert the preprocessed audio waveform into a numerical feature representation more suitable for model processing. This involves converting the preprocessed audio into the acoustic features required by the model, such as extracting 40-dimensional Mel-frequency cepstral coefficients (MFCC) or Fbank features (typical configuration: 25ms frame length, 10ms frame shift, to balance temporal resolution and computational efficiency).

[0100] Text processing aims to standardize and clean the annotated text corresponding to the speech, ensuring correct alignment between the text and audio features and eliminating irrelevant variations. Standardization of the annotated text includes Unicode standardization, full-width / half-width character encoding, traditional Chinese to simplified Chinese (for Chinese dialect scenarios), removal of punctuation marks, and unification of numbers to Arabic numerals to ensure consistent text formatting.

[0101] Data partitioning is a crucial step in preventing model overfitting and objectively evaluating its performance. The processed data is divided into training and validation sets in an 8:2 ratio to avoid overfitting. For example, for a dialect dataset with a total duration of 5 hours, preprocessing yields approximately 4 hours of training data and 1 hour of validation data. This data partitioning ratio is based on common machine learning practices and effectively ensures training efficiency and model generalization ability.

[0102] S3-2, PEFT Training Framework Setup. The purpose of this step is to construct the computational framework and configure the training rules required for Spectrum-Phase Residual Low-Rank Adaptive (SPaRLoRA) fine-tuning. This framework ensures that subsequent iterative training can be performed correctly and efficiently. Specific preferred implementation methods include model ensemble, optimizer configuration, and loss function selection.

[0103] First, in S3-2-1, model integration is performed, combining the residual-based weights W obtained in stage S2. res (Set to frozen) and LoRA's A and B matrices (set to trainable) are integrated into the linear layer of the original speech recognition model. The forward propagation computation path is defined, and the speech recognition model is built. The forward propagation calculation formula for the linear layer is defined as follows:

[0104]

[0105] Where X is the input training set speech data, and γ is the scaling factor.

[0106] This step ensures that the output of the model during forward propagation includes both the residual base weights and the LoRA increments.

[0107] Subsequently, when configuring the optimizer in S3-2-2, only the parameters of matrices A and B are included in the update list, while W is frozen. res And all other parameters of the model.

[0108] Finally, S3-2-3 considers the sequence characteristics of the ASR task when selecting the loss function, choosing either the Connectionist Temporal Classification (CTC) loss function or the Cross-Entropy loss function. The CTC loss function is suitable for handling problems where the input and output sequence lengths are inconsistent (such as dialect recognition), while the Cross-Entropy loss function is suitable for frame-level classification tasks. In low-resource dialect ASR tasks, the CTC loss function is preferred to better handle variable-length sequences and phoneme variations.

[0109] Preferably, training hyperparameters can also be set, such as the number of training epochs and the batch size.

[0110] S3-3, Training Iteration and Monitoring. Through multiple iterations on the training data, forward propagation, loss calculation, backpropagation, and parameter updates are performed cyclically to progressively optimize the values ​​of matrices A and B. The training process is monitored to ensure convergence and prevent overfitting. Specifically, the optimized implementation includes four sub-steps: forward propagation, backpropagation, loss calculation, and convergence monitoring.

[0111] S3-3-1, Forward Propagation: Input the training set speech data into S3-2 to build the integrated speech recognition model, perform forward propagation, calculate the output through the LoRA linear layer, and then obtain the prediction result through the decoder. The decoder can be a sequence-to-sequence decoder based on the attention mechanism.

[0112] S3-3-2, Backpropagation: The gradient is calculated based on the loss function. Gradient updates are performed only on the parameters of matrices A and B. Backpropagation is then used to adjust the values ​​of matrices A and B, generating weight increments. The formula is:

[0113]

[0114] The residual basic weights W res With weight increment By merging, we obtain the updated weight matrix W. new ,

[0115] W new =W res +

[0116] S3-3-3, calculate using the Connection Temporal Classification (CTC) loss function or the cross-entropy loss function. The loss L.

[0117] S3-3-4, Convergence Monitoring: Repeat steps S3-3-1 to S3-3-3 until the model converges. The preferred monitoring strategy for determining convergence is as follows: After each training round, calculate the gradient of the loss L with respect to A and B on the validation set. When the gradient signal... and When the model no longer shows a significant decrease (i.e., the change is less than a threshold) after several consecutive rounds (e.g., 3 rounds), it is determined that the model has converged, training is stopped, and overfitting is avoided. This threshold is set based on experience to ensure stable convergence of the model in low-resource scenarios.

[0118] At this point, the parameters of matrices A and B after the final iteration are determined, thus obtaining the trained matrices A and B, denoted as A. trained and B trained .

[0119] In summary, the S3 training execution phase achieves efficient optimization of the A and B matrices through data preprocessing, PEFT framework construction, and iterative monitoring, providing rapid convergence and robust performance improvement for low-resource ASR tasks.

[0120] S4, Weight Integration and Output Steps.

[0121] This stage is the final step in the SPaRLoRA method. Its main goal is to integrate the optimization results obtained from the S3 training execution stage with the model's base weights to generate a complete, independent, and high-performance final speech recognition model. Specifically, this stage receives the output from the S3 stage—the trained A and B matrices (denoted as A). trained and B trained and the residual base weights W obtained in stage S2 res Through precise mathematical calculations and rigorous verification processes, model weight files are generated that can be directly used for inference deployment. The entire process ensures that the model achieves low-resource task adaptability without introducing any additional inference overhead.

[0122] S4-1, weighted calculation.

[0123] After training, this step is crucial for technical implementation. Through mathematical operations, the trainable adapter parameters are combined with the frozen base weights to form a unified and efficient weight matrix. First, based on the A obtained from training in phase S3... trained B trained And the scaling factor γ (whose value is determined by the parameters α and r configured in stage S1, for example, γ=2 when α=64, r=32), calculate the final weight increment of LoRA. The calculation formula is as follows:

[0124]

[0125] Then, the calculated final weight increments will be... With residual basic weights W res The weights are summed to obtain the final weights W of the fine-tuned linear layer. final :

[0126] W final =W res +

[0127] Will With residual basic weights W res Merge to generate a single, finely adjusted weight W. final It includes the basic capabilities of the pre-trained model, incorporates the ability to adapt to low-resource ASR tasks, and has the same dimension as the linear layer weights of the original model, ensuring that there is no additional computational overhead during model inference and maintaining the same inference speed as the original model.

[0128] S4-2, after completing the weight merging, this step is responsible for rigorously validating the generated final model to ensure that its performance meets the standards before outputting it.

[0129] First, model files are generated and replaced based on the final weights W. final Update the weights of the corresponding linear layers in the model and generate a complete fine-tuned model weight file.

[0130] Next, the fine-tuned model is used to perform inference on the validation set to check its recognition performance and inference efficiency. The goal is to ensure that the CER metric meets the requirements of the low-resource ASR task while maintaining the same inference speed as the original model. The calculation formula is as follows:

[0131]

[0132] Where S is the number of replacements, D is the number of deletions, I is the number of insertions, and N is the total number of characters in the annotation text.

[0133] In a specific instance, on a low-resource Hakka dialect test set (1 hour in length), the fine-tuned model is required to reduce its CER value to below 15%, significantly outperforming the baseline model (CER > 25%) that does not use the SPaRLoRA method. Simultaneously, inference efficiency needs to be verified, ensuring that, under the same hardware environment, the average inference time per audio segment of the fine-tuned model does not increase by more than 5% compared to the original model.

[0134] Finally, a deployment compatibility check is performed to verify that the generated weight file can be loaded by standard inference code without any modification, ensuring that it can be used normally in various deployment environments (such as local servers and mobile devices), and finally outputting a complete model that meets the performance and efficiency requirements, thus completing the entire process of building a low-resource speech recognition model.

[0135] In summary, the S4 weight integration and output stage, through rigorous merging calculations and comprehensive verification tests, successfully transformed the training results of SPaRLoRA into a practical, high-performance speech recognition model, providing a reliable solution for applications in low-resource scenarios.

[0136] As can be seen, this invention creatively incorporates the physical characteristics (spectrum and phase) of speech signals as strong prior knowledge into the efficient fine-tuning process of parameters in large-scale pre-trained models. By designing a structured initialization strategy and residual learning mechanism, the model is guided to perform efficient and targeted optimization in data-scarce scenarios. This solution deeply integrates signal processing priors with deep learning fine-tuning techniques, providing a complete low-resource speech recognition model construction scheme. It effectively overcomes the shortcomings of existing technologies and provides a powerful technical tool for dialect preservation and vertical domain speech applications, possessing significant theoretical value and broad application prospects.

[0137] As attached Figure 2 As shown, the present invention also provides a construction apparatus for a low-resource speech recognition model based on low-rank adaptive fine-tuning, comprising:

[0138] The preprocessing preparation module is used to preprocess the model based on the pre-trained speech recognition model, complete the configuration of basic data and parameters, and establish the basis for low-rank adaptive fine-tuning of the model.

[0139] The initialization module is used to initialize the low-rank adapter structure based on the low-rank adaptive fine-tuning and to perform residual processing on the model weights.

[0140] The training module is used to update the low-rank adapter parameters through a parameter fine-tuning framework using a low-resource speech recognition dataset, and to monitor the training process to optimize the model.

[0141] The output module integrates the trained low-rank adapter parameters with the residual model weights to output the final speech recognition model.

[0142] Preferably, the preprocessing preparation module specifically includes: a weight extraction unit for loading a pre-trained speech recognition model; locating and selecting the target linear layer, and recording its input dimension d and output dimension; extracting and storing the basic weight matrix W of the target linear layer; a LORA parameter configuration unit for configuring key parameters of low-rank adaptive (LoRA); and a DFT basis calculation preparation unit for preparing resources for discrete Fourier transform basis vector (DFT basis) calculation.

[0143] Preferably, the initialization module specifically includes: an FFT transform unit for performing a Fast Fourier Transform (FFT); a DFT basis calculation unit for generating a DFT basis that conforms to the characteristics of the speech spectrum; an amplitude spectrum and phase spectrum splicing unit for splicing the amplitude spectrum and phase spectrum; an A / B matrix initialization unit for calculating the initial values ​​of the low-rank matrices A and B of the LoRA; and a residual basis weight calculation unit for calculating the residual basis weight matrix.

[0144] Preferably, the training module specifically includes: a training data preprocessing unit for obtaining preprocessed low-resource training set speech data; a training framework building unit for building the PEFT training framework; and a training iteration and monitoring unit for performing model training iterations and convergence monitoring.

[0145] Preferably, the training iteration and monitoring unit specifically includes the following sub-units:

[0146] The LORA forward propagation unit is used to input the training set speech data into the speech recognition model and calculate the predicted text sequence output through the forward propagation calculation formula.

[0147] The LORA backpropagation unit is used to calculate the gradient based on the loss function and update the parameter values ​​of matrices A and B.

[0148] The weight increment update unit is used to generate weight increments. The formula is:

[0149]

[0150] Obtain the updated weight matrix W new ,

[0151] W new =W res +

[0152] The loss calculation unit is used to calculate the loss value L between the model output and the true label based on the loss function.

[0153] The gradient signal calculation unit is used to calculate the gradient of the loss L with respect to B and A with respect to the validation set after each training round.

[0154]

[0155] Where L is the loss value, Output the gradient of the loss to the model.

[0156] The convergence monitoring unit determines the model's convergence status based on the gradient signal and stops training once the preset conditions are met.

[0157] Preferably, the output module specifically includes a weight merging unit and an output unit, which are used to calculate the final weights of the fine-tuned linear layer and output the final speech recognition model, respectively.

[0158] In addition, the present invention provides an electronic device, comprising: a processor and a memory; the memory storing instructions executable by the processor; the processor being configured to, when executing the instructions, cause the electronic device to perform the method as described in the present invention.

[0159] And a computer program product is provided, the computer program product including instructions, which, when executed on a computer, cause the computer to perform the method as described in this invention.

[0160] As can be seen, this invention creatively incorporates the physical characteristics (spectrum and phase) of speech signals as strong prior knowledge into the efficient fine-tuning process of parameters in large-scale pre-trained models. By designing a structured initialization strategy and residual learning mechanism, the model is guided to perform efficient and targeted optimization in scenarios with scarce data.

[0161] In summary, the beneficial effects of the present invention include:

[0162] First, this invention proposes an initial parameter construction method based on the magnitude and phase spectra of DFT basis vectors for LoRA fine-tuning training, which extracts the spectral features of speech signals and solves the limitation of existing ASR algorithms that are difficult to fully match the inherent features of speech signals due to the generalization of initialization strategies.

[0163] Secondly, this invention proposes a LoRA optimization method that removes the spectral prior interpretable components from the pre-trained weights. By using residual processing, the training is focused on mid- and low-frequency features, which solves the problem of many parameters and long training time in ASR models under low resource conditions.

[0164] Finally, the convergence monitoring method using gradient signals proposed in this invention can effectively focus on the impact of speech features on the loss, solving the problems of slow convergence and easy overfitting of ASR models under low resource conditions.

[0165] In summary, this invention, through the aforementioned spectrum-phase residual initialization and efficient parameter fine-tuning techniques, effectively solves the technical pain points of low model training efficiency, loss of phase information, and poor cross-stage adaptation in traditional low-resource scenarios without altering the original model architecture, and significantly improves the accuracy of dialect recognition, niche word recognition, and hot word adaptation.

[0166] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0167] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for constructing a low-resource speech recognition model, characterized in that, Includes the following steps: S1, Preprocessing preparation steps: Based on the pre-trained speech recognition model, perform model preprocessing, complete the configuration of basic data and parameters, and establish the basis for low-rank adaptive fine-tuning of the model. S2, Initialization construction steps: Based on the aforementioned low-rank adaptive fine-tuning foundation, perform low-rank adapter structure initialization and residual processing on model weights; The S2 initialization construction step specifically includes: S2-1, generating discrete Fourier transform basis vectors that conform to the characteristics of speech spectrum; S2-2, Calculate the initial values ​​of the low-rank matrices A and B of LoRA; S2-3, Calculate the residualized basic weight matrix; First, calculate the initial low-rank matrix. Using the matrices A and B constructed in S2-2 and the scaling factor γ determined in stage S1, calculate the initial low-rank matrix: The matrix The dimension is consistent with the basic weight matrix W. Then, residual processing is performed: the initial low-rank matrix is ​​subtracted from the basic weight matrix W. The residual basic weights W are obtained. res And freeze, the calculation formula is: IN res =W- W res The complex components in the basic weights that cannot be simply explained by spectral priors are retained; S3, Training execution steps: using a low-resource speech recognition dataset, the low-rank adapter parameters are updated through a parameter fine-tuning framework, and the training process is monitored to optimize the model; S4, Weight Integration Step: Integrate the low-rank adapter parameters obtained from step S3 with the model weights after residualization in step S2 to output the final speech recognition model. S3-2-1, Perform model integration, and integrate the residual basic weights W. res The A and B matrices of LoRA are integrated into the linear layer of the speech recognition model. The forward propagation calculation formula for the linear layer is defined as follows: Where Output is the model output, X is the input training set speech data, and γ is the scaling factor; S3-2-2, Configure the optimizer, add the parameters of matrices A and B to the update list, and freeze other model parameters; S3-2-3, Choose the loss function.

2. The method as described in claim 1, wherein, The S1 preprocessing preparation step specifically includes: S1-1, Model Weight Extraction: Load the pre-trained speech recognition model; locate and select the target linear layer; extract the basic weight matrix W of the target linear layer; S1-2, Configure key parameters for low-rank adaptive LoRA; S1-3, Resource preparation for calculating discrete Fourier transform basis vectors.

3. The method as described in claim 2, wherein, The S3 training execution steps specifically include: S3-1, Training set data preprocessing; the preprocessing includes at least audio processing, feature extraction, text processing, and data partitioning; S3-2, PEFT training framework setup; S3-3, Training Iteration and Monitoring.

4. The method of claim 3, wherein, The S3-3 training iteration and monitoring steps specifically include: S3-3-1, Forward Propagation: The training set data is input into the speech recognition model, and the predicted text sequence is output by calculating using the forward propagation calculation formula; S3-3-2, Backpropagation: Calculate the gradient based on the loss function, update the parameter values ​​of matrices A and B; generate weight increments. The formula is: Obtain the updated weight matrix W new , IN new =W res + S3-3-3, Loss Calculation: Calculate the loss value L between the model output and the true label based on the loss function; S3-3-4, Convergence Monitoring: Repeat training steps S3-3-1 to S3-3-3 until the model converges; the convergence strategy adopted is: after each round of training, calculate the gradient of the loss L with respect to B and A for the validation set, and then use the gradient signal... and Determine the model's convergence status and stop training once the preset conditions are met; The gradient calculation formula is as follows: Where L is the loss value, Output the gradient of the loss to the model.

5. The method of claim 4, wherein, The S4 weight integration step specifically includes: S4-1, After training, the final value A of matrices A and B based on LoRA is... trained B trained And the scaling factor γ, calculate the final weight increment ΔW of LoRA. trained : ； S4-2, Calculate the final weights W of the fine-tuned linear layer. final : IN final =W res + ; S4-3, based on the W final The weights of the linear layers of the model are updated to generate a complete fine-tuned model weight file, and the final speech recognition model is output.

6. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, it causes the electronic device to implement the method as described in any one of claims 1-5.

7. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-5.

Images