A Speech Recognition Fine-Tuning Method Based on Subspace Decomposition and Recombination

Abstract

This invention discloses a speech recognition fine-tuning method based on subspace decomposition and recombination. This invention effectively decomposes the low-rank space of the LoRA module into multiple subspaces, adds learnable weights to different subspaces, and recombines all subspaces. This allows the recombined subspaces to be transformed into an efficient computation mode during training and an inference mode during inference, significantly improving inference and storage efficiency. While ensuring efficient parameter fine-tuning, it also solves the problem of high inference costs under high concurrency requests.

Description

Technical Field

[0001] This invention relates to the field of intelligent speech processing, specifically a speech recognition fine-tuning method based on subspace decomposition and recombination. Background Technology

[0002] With the rapid development of deep learning technology, current speech recognition models can achieve human-level accuracy in general scenarios. However, in many speech recognition scenarios, due to limited data resources, accent changes, and post-illness sequelae, satisfactory recognition results cannot be obtained. Therefore, adapting pre-trained speech models to different speech recognition scenarios through fine-tuning techniques is the main approach to solving this problem. Traditional full-model fine-tuning methods are limited by low parameter efficiency and time-consuming training, failing to meet practical needs.

[0003] Low-rank adaptation (LoRA), as a parameter-efficient fine-tuning method, can adapt pre-trained speech models to different speech recognition scenarios by introducing learnable low-rank matrices, significantly reducing the computational and storage costs of the speech recognition model fine-tuning stage. However, in the multi-scenario inference stage, the LoRA method still needs to perform additional matrix operations, causing the computational load and latency to increase rapidly with the increase of input size and concurrent requests. Especially in large-scale real-time user interaction scenarios, inference latency becomes the main bottleneck.

[0004] Existing technologies have not explored sufficient ways to optimize LoRA inference efficiency, making it difficult to significantly reduce inference latency in high-concurrency scenarios while maintaining speech recognition accuracy. For example, the paper "W. Liu, Y. Qin, Z. Peng and T. Lee,"Sparsely Shared Lora on Whisper for Child Speech Recognition," ICASSP 2024 -2024 IEEE International Conference on Acoustics, Speech and Signal Processing, Seoul, Korea, Republic of, 2024, pp. 11751-11755, doi: 10.1109 / ICASSP48485.2024.10447004." improves parameter efficiency by globally sharing a low-rank matrix, but this method does not significantly reduce computational load and therefore cannot effectively reduce inference latency.

[0005] Therefore, there is an urgent need for an innovative method for fine-tuning speech recognition, which optimizes the inference process of low-rank matrices through subspace decomposition and recombination, thereby achieving a balance between computational efficiency and model performance and meeting the real-time requirements of high-concurrency speech recognition applications. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a speech recognition fine-tuning method based on subspace decomposition and recombination.

[0007] The technical solution of this invention to solve the aforementioned technical problem is to provide a speech recognition fine-tuning method based on subspace decomposition and recombination, which includes the following steps: Step 1: Select a corpus as the dataset and divide the dataset into a training set, a validation set, and a test set; Step 2: Select the original speech pre-trained model; the original speech pre-trained model includes multiple self-attention modules and multiple cross-attention modules, each of which contains multiple linear layers; then add a LoRA module in parallel to each linear layer, and set the rank r of the LoRA module; each LoRA module consists of two low-rank matrices, namely a dimension-reduced projection matrix A and an dimension-up projected matrix B; the two low-rank matrices form a low-rank space; Step 3: Divide the dimensionality-reduced projection matrix A obtained in Step 2 into blocks. The upgraded projection matrix B obtained in step 2 is divided into blocks. The low-rank space of the LoRA module is decomposed into n subspaces, each subspace consisting of... Composition, in which Furthermore, the rank r of the LoRA module is an integer multiple of the number of subspaces n; each subspace shares the input, and the sum of the outputs of all subspaces is the final output; Step 4: Use the subspace recombination method to change the combination of the subspaces obtained in Step 3 to obtain the recombined subspace; Step 5: Using the relevant properties of the block matrix, the recombined subspace obtained in Step 4 is equivalently transformed into a recombined subspace of an efficient computing mode; Step 6: Use the training set obtained in Step 1 to train the parameters of the recombinant subspace of the efficient computing mode obtained in Step 5, and obtain the trained recombinant subspace. Step 7: Based on the properties of matrix multiplication, transform the trained recombined subspace obtained in Step 6 into the recombined subspace of the inference mode; then load the parameters of the recombined subspace of the inference mode into the original speech pre-trained model in Step 2 to obtain the loaded speech pre-trained model; then use the validation set obtained in Step 1 to validate the loaded speech pre-trained model to obtain the final model. Step 8: Input the test set obtained in Step 1 into the final model obtained in Step 7, output the predicted text result, and complete the fine-tuning of speech recognition based on subspace decomposition and recombination.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention decomposes the low-rank space of the LoRA module into multiple subspaces, adds learnable weights to different subspaces, and reorganizes all subspaces. This allows the reorganized subspaces to be transformed into efficient computing modes during training and into inference modes during inference, significantly improving inference and storage efficiency. While ensuring efficient parameter fine-tuning, it solves the problem of high inference cost during high-concurrency requests.

[0009] (2) The present invention can be transformed into a training form during the training phase to maintain the same training efficiency as the LoRA method. Compared with the full model fine-tuning method, the present invention can not only significantly reduce the memory usage during training, but also effectively reduce the training time.

[0010] (3) After training, the present invention can be transformed into an inference form. This method significantly reduces the computational complexity and computational load of the speech recognition model during the inference stage, and reduces inference latency. At the same time, it maintains high recognition accuracy, solves the problem of high inference cost in high-concurrency scenarios, and provides an efficient solution for real-time speech recognition applications.

[0011] (4) This invention can be transferred to any speech pre-trained model, and has the advantages of good portability and high flexibility. Attached Figure Description

[0012] Figure 1 This is a schematic diagram illustrating the combination of the LoRA module and the linear layer in this invention; Figure 2 This is a schematic diagram of the subspace decomposition process in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the subspace recombination process in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the training mode in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the reasoning mode in Embodiment 1 of the present invention. Detailed Implementation

[0013] Specific embodiments of the present invention are given below. These specific embodiments are only used to further illustrate the present invention in detail and do not limit the scope of protection of the present invention.

[0014] This invention provides a speech recognition fine-tuning method based on subspace decomposition and recombination (hereinafter referred to as the method), which includes the following steps: Step 1: Select a corpus as the dataset and divide the dataset into a training set, a validation set, and a test set; Preferably, in step 1, the corpus is an English accent corpus; more preferably, the English accent corpus uses both AESRC and Librispeech100; AESRC is an English accent dataset containing English accent data from ten countries and corresponding text, with a total audio duration of 200 hours; Librispeech100 is a 100-hour normal English accent dataset.

[0015] Preferably, in step 1, the ratio of the training set, validation set, and test set is 8:1:1. In this embodiment, the ratio of the training set, validation set, and test set for AESRC is 8:1:1, and the ratio of the training set, validation set, and test set for Librispeech100 is also 8:1:1.

[0016] Step 2: Select the original speech pre-trained model; the original speech pre-trained model includes multiple self-attention modules and multiple cross-attention modules, each of which contains multiple linear layers; then add a LoRA module in parallel to each linear layer, and set the rank r of the LoRA module; each LoRA module consists of two low-rank matrices, namely a dimension-reduced projection matrix A and an dimension-up projected matrix B; the two low-rank matrices form a low-rank space; Preferably, in step 2, the original speech pre-training model adopts the Whisper model, whose structure includes an encoder and a decoder connected in sequence; the encoder consists of 10 to 20 encoding layers connected in sequence, each encoding layer including a self-attention module and a perceptron layer connected in sequence; the decoder consists of 10 to 20 decoding layers connected in sequence, each decoding layer including a self-attention module, a cross-attention module and a perceptron layer connected in sequence.

[0017] Preferably, in step 2, the LoRA module is combined with the linear layer in the following way: Figure 1 As shown; then, based on the assemblage method, the forward process containing the LoRA module is obtained as follows: (1) In equation (1), It is the input to the linear layer and the LoRA module. It is the sum of the outputs of the linear layer and the LoRA module; and Let A and B be two low-rank matrices in the LoRA module, where d and r are the number of neurons in the linear layer and the rank of the LoRA module, respectively; and A and B form a low-rank space. The pre-trained weights represent the linear layers of the self-attention and cross-attention modules in the original speech pre-trained model, and It does not participate in training; only the parameters of the LoRA module participate in training.

[0018] Step 3: Divide the dimensionality-reduced projection matrix A obtained in Step 2 into blocks. The upgraded projection matrix B obtained in step 2 is divided into blocks. That is, the low-rank space of the LoRA module is decomposed into n subspaces, each subspace being composed of... Composition, in which Furthermore, the rank r of the LoRA module is an integer multiple of the number of subspaces n; each subspace shares the input, and the sum of the outputs of all subspaces is the final output; Preferably, in step 3, the process of splitting the low-rank space of the LoRA module is as follows: (2) In equation (2), Y is the output of the LoRA module, and .

[0019] Step 4: Use the subspace recombination method to change the combination of the subspaces obtained in Step 3 to obtain the recombined subspace; Preferably, in step 4, the subspace recombination method is expressed as: (3) In equation (3), They are respectively Learnable weights They are respectively The learnable weights; the n subspaces in equation (2) are independent of each other, but after the subspaces are reorganized, there is interaction between different subspaces in equation (3), and they have learnable weights.

[0020] Step 5: Using the relevant properties of the block matrix, the recombined subspace obtained in Step 4 is equivalently transformed into a recombined subspace of an efficient computing mode; Preferably, in step 5, the recombinant subspace of the efficient computing mode is represented as: (4) In equation (4), An identity matrix that does not participate in parameter updates; It is a mixed matrix, a low-rank matrix, composed of... and We obtain the result by matrix multiplication; Equation (4) represents the output of the recombined subspace of the efficient computation mode; Equation (4) can make full use of the efficient matrix multiplication operator, greatly accelerate the training process, and maintain almost the same training efficiency as LoRA in the model training stage.

[0021] Step 6: Use the training set obtained in Step 1 to train the parameters of the recombinant subspace of the efficient computing mode obtained in Step 5, and obtain the trained recombinant subspace. Preferably, in step 6, the cross-entropy loss function is used as the objective function for training. After training, the parameters and weights of all subspaces are fixed.

[0022] Preferably, in step 6, the parameters of the recombinant subspace of the efficient computing mode are: , as well as .

[0023] Step 7: Based on the properties of matrix multiplication, transform the trained recombined subspace obtained in Step 6 into the recombined subspace of the inference mode; then load the parameters of the recombined subspace of the inference mode into the original speech pre-trained model in Step 2 to obtain the loaded speech pre-trained model; then use the validation set obtained in Step 1 to validate the loaded speech pre-trained model to obtain the final model. Preferably, in step 7, the reorganized subspace of the reasoning pattern is represented as: (5) In equation (5), and ; The output represents the reorganized subspace of the inference mode. The time and space complexity of this invention during inference are both 2dr / n, which is only 1 / n of that of the LoRA method. Therefore, it can significantly improve the inference efficiency of the LoRA method, effectively reduce the inference cost under high concurrency requests, and alleviate storage pressure.

[0024] Step 8: Input the test set obtained in Step 1 into the final model obtained in Step 7, output the predicted text result, and complete the fine-tuning of speech recognition based on subspace decomposition and recombination.

[0025] Example 1: Step 1: Select Librispeech100 and AESRC as datasets, and divide them into training, validation and test sets according to the 8:1:1 splitting method. Step 2: The original speech pre-training model adopts the Whisper model, which is representative in the speech domain; the rank of the LoRA module is set to r=4; In step 2, the forward process including the LoRA module is as follows:

[0026] in, It is the input to the linear layer and the LoRA module. It is the sum of the outputs of the linear layer and the LoRA module; and Let d be two low-rank matrices in the LoRA module, and d be the number of neurons in the linear layer. These are the pre-trained weights, representing the linear layers of the self-attention and cross-attention modules in the Whisper model.

[0027] Step 3: Divide the dimensionality-reduced projection matrix A obtained in Step 2 into blocks. The upgraded projection matrix B obtained in step 2 is divided into blocks. This means decomposing the low-rank space of the LoRA module into two subspaces; In step 3, the low-rank space partitioning process of the LoRA module is as follows: Figure 2 As shown:

[0028] Where Y is the output of the LoRA module, and .

[0029] Step 4: Use the subspace recombination method to change the combination of the subspaces obtained in Step 3 to obtain the recombined subspace; In step 4, the subspace recombination method is represented as follows: Figure 3 As shown:

[0030] in, They are respectively Learnable weights.

[0031] Step 5: Using the relevant properties of the block matrix, the recombined subspace obtained in Step 4 is equivalently transformed into a recombined subspace of an efficient computing mode; In step 5, the recombined subspace of the efficient computing model is represented as follows: Figure 4 As shown:

[0032] in, An identity matrix that does not participate in parameter updates; It is called a mixing matrix, by and The matrix is ​​multiplied to obtain the result.

[0033] Step 6: Use the training set obtained in Step 1 to train the parameters of the recombinant subspace of the efficient computing mode obtained in Step 5, and obtain the trained recombinant subspace. In step 6, the cross-entropy loss function is used as the objective function for training, and the training is stopped after 10 iterations when the loss no longer decreases.

[0034] In step 6, the parameters of the recombinant subspace of the efficient computation mode are: , as well as .

[0035] Step 7: Based on the properties of matrix multiplication (i.e., the associative law of matrix multiplication), transform the trained recombined subspace obtained in Step 6 into the recombined subspace of the inference mode; load the parameters of the recombined subspace of the inference mode into the original speech pre-trained model in Step 2 to obtain the loaded speech pre-trained model; then use the validation set obtained in Step 1 to validate the loaded speech pre-trained model to obtain the final model. In step 7, the reorganized subspace of the reasoning pattern is represented as follows: Figure 5 As shown:

[0036] in, and The present invention has a time complexity and space complexity of 2d during inference, which is only half that of the LoRA method. Therefore, it can significantly improve the inference efficiency of the LoRA method, effectively reduce the inference cost under high concurrency requests, and alleviate storage pressure.

[0037] Step 8: Input the test set obtained in Step 1 into the final model obtained in Step 7, output the predicted text result, and complete the fine-tuning of speech recognition based on subspace decomposition and recombination.

[0038] To verify the effectiveness of the method of the present invention, the following experiments were conducted: 1. Comparative Experiment Table 1 compares the performance and inference efficiency of E-LoRA (the method of this invention is denoted as E-LoRA) with LoRA and its mainstream variants (MOSLoRA and MELoRA). NOS represents the number of subspaces, ROS represents the rank of the subspace, and NSP represents the number of stored parameters. The number of training parameters is consistent for all methods.

[0039] Table 1. Comparison of performance and inference efficiency of different fine-tuning methods

[0040] As shown in Table 1, E-LoRA demonstrates a significant performance improvement over MOSLoRA and MELoRA on both datasets. Furthermore, after training, E-LoRA requires only half the storage space of the other three, thus greatly improving inference efficiency and effectively alleviating the problem of high inference costs under high concurrency requests.

[0041] 2. Ablation test Table 2 shows the ablation of two key designs in E-LoRA. In this experiment, the rank of LoRA was set to 8.

[0042] Table 2 Ablation Experiment Results of E-LoRA

[0043] As shown in Table 2, comparing experiments 1 and 2, it can be seen that subspace reorganization can reduce storage space requirements to 1 / 8. Comparing experiments 2 and 3, it can be seen that learnable weights can achieve performance improvement.

[0044] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A speech recognition fine-tuning method based on subspace decomposition and recombination, characterized in that, The method includes the following steps: Step 1: Select a corpus as the dataset and divide the dataset into a training set, a validation set, and a test set; Step 2: Select the original speech pre-trained model; the original speech pre-trained model includes multiple self-attention modules and multiple cross-attention modules, each of which contains multiple linear layers; then add a LoRA module in parallel to each linear layer, and set the rank r of the LoRA module; each LoRA module consists of two low-rank matrices, namely a dimension-reduced projection matrix A and an dimension-up projected matrix B; the two low-rank matrices form a low-rank space; Step 3: Divide the dimensionality-reduced projection matrix A obtained in Step 2 into blocks. The upgraded projection matrix B obtained in step 2 is divided into blocks. The low-rank space of the LoRA module is decomposed into n subspaces, each subspace consisting of... Composition, in which Furthermore, the rank r of the LoRA module is an integer multiple of the number of subspaces n; each subspace shares the input, and the sum of the outputs of all subspaces is the final output; Step 4: Use the subspace recombination method to change the combination of the subspaces obtained in Step 3 to obtain the recombined subspace; Step 5: Using the relevant properties of the block matrix, the recombined subspace obtained in Step 4 is equivalently transformed into a recombined subspace of an efficient computing mode; Step 6: Use the training set obtained in Step 1 to train the parameters of the recombinant subspace of the efficient computing mode obtained in Step 5, and obtain the trained recombinant subspace. Step 7: Based on the properties of matrix multiplication, transform the trained recombined subspace obtained in Step 6 into the recombined subspace of the inference mode; then load the parameters of the recombined subspace of the inference mode into the original speech pre-trained model in Step 2 to obtain the loaded speech pre-trained model; then use the validation set obtained in Step 1 to validate the loaded speech pre-trained model to obtain the final model. Step 8: Input the test set obtained in Step 1 into the final model obtained in Step 7, output the predicted text result, and complete the fine-tuning of speech recognition based on subspace decomposition and recombination.

2. The speech recognition fine-tuning method based on subspace decomposition and recombination according to claim 1, characterized in that, In step 1, the corpus used is the English accent corpus; the English accent corpus uses both AESRC and Librispeech100; AESRC is an English accent dataset containing English accent data from ten countries and corresponding text, with a total audio duration of 200 hours; Librispeech100 is a 100-hour normal English accent dataset. In step 1, the ratio of the training set, validation set, and test set is 8:1:

1.

3. The speech recognition fine-tuning method based on subspace decomposition and recombination according to claim 1, characterized in that, In step 2, the original speech pre-training model adopts the Whisper model, which consists of an encoder and a decoder connected in sequence. The encoder consists of 10 to 20 encoding layers connected in sequence, each encoding layer including a self-attention module and a perceptron layer connected in sequence. The decoder consists of 10 to 20 decoding layers connected in sequence, each decoding layer including a self-attention module, a cross-attention module and a perceptron layer connected in sequence.

4. The speech recognition fine-tuning method based on subspace decomposition and recombination according to claim 1, characterized in that, In step 2, based on the combination method of the LoRA module and the linear layer, the forward process containing the LoRA module is obtained as follows: （1） In equation (1), It is the input to the linear layer and the LoRA module. It is the sum of the outputs of the linear layer and the LoRA module; and Let d and r be two low-rank matrices in the LoRA module, where d and r are the number of neurons in the linear layer and the rank of the LoRA module, respectively. , where represents the pre-trained weights, and represents the linear layers of the self-attention module and cross-attention module in the original speech pre-trained model.

5. The speech recognition fine-tuning method based on subspace decomposition and recombination according to claim 4, characterized in that, In step 3, the process of splitting the low-rank space of the LoRA module is as follows: （2） In equation (2), Y is the output of the LoRA module, and .

6. The speech recognition fine-tuning method based on subspace decomposition and recombination according to claim 5, characterized in that, In step 4, the subspace recombination method is expressed as: （3） In equation (3), They are respectively Learnable weights They are respectively Learnable weights.

7. The speech recognition fine-tuning method based on subspace decomposition and recombination according to claim 6, characterized in that, In step 5, the recombinant subspace of the efficient computing model is represented as: （4） In equation (4), An identity matrix that does not participate in parameter updates; It is a mixed matrix, a low-rank matrix, composed of... and We obtain the result by matrix multiplication; This represents the output of the recombined subspace of the efficient computing mode.

8. The speech recognition fine-tuning method based on subspace decomposition and recombination according to claim 1, characterized in that, In step 6, the cross-entropy loss function is used as the objective function for training.

9. The speech recognition fine-tuning method based on subspace decomposition and recombination according to claim 7, characterized in that, In step 6, the parameters of the recombinant subspace of the efficient computation mode are: , as well as .

10. The speech recognition fine-tuning method based on subspace decomposition and recombination according to claim 9, characterized in that, In step 7, the reorganized subspace of the reasoning pattern is represented as: （5） In equation (5), and ; The output of the reorganized subspace representing the reasoning pattern.

Images