Large language model fine-tuning system and method for edge implementation
By employing tensor chain decomposition, two-stage quantization, and adaptive optimization, the storage and computational overhead issues of deploying large language models on edge devices are addressed, achieving efficient fine-tuning accuracy and stability, making it suitable for resource-constrained embedded devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2026-04-30
- Publication Date
- 2026-05-29
AI Technical Summary
The deployment of existing large language models on edge devices is limited by computational and storage overhead. Existing efficient parameter fine-tuning methods are not accurate enough and training is unstable under extremely low parameter budgets. There is a lack of systematic solutions for efficient connection from the training stage to the inference stage.
Employing tensor chain decomposition and two-stage quantization, amplitude and direction decoupling and adaptive optimization, and a knowledge distillation module, the low-rank update matrix is reshaped into a high-order tensor form, and tensor chain decomposition is applied. Combined with two-stage quantization and adaptive optimization strategies, a knowledge distillation loss function is constructed, and an adapter file is exported for deployment on the edge device.
It improves fine-tuning accuracy and training stability with extremely low parameters, reduces storage consumption and inference latency, supports fast switching in multi-task scenarios, and is suitable for edge devices with limited storage resources and computing power.
Smart Images
Figure CN122114076A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neural network technology in artificial intelligence and machine learning, specifically to a lightweight fine-tuning system and method for large language models implemented at the edge. More particularly, it relates to a lightweight fine-tuning framework for large language models based on tensor decomposition and knowledge distillation for edge implementation. Background Technology
[0002] With the groundbreaking advancements in the field of natural language processing, Large Language Models (LLMs) have demonstrated powerful versatility in various tasks such as question answering, translation, and summarization. Achieving this capability typically requires scaling the model's parameter count to billions or even hundreds of billions. However, this rapid increase in model size leads to enormous computational and storage overhead. This characteristic severely restricts the practical deployment and application of Large Language Models on edge devices such as mobile terminals, embedded systems, and IoT gateways.
[0003] To address the aforementioned issues, Parameter-Efficient Fine-Tuning (PEFT) technology emerged. Its core idea is to freeze the backbone parameters of the pre-trained model and adapt it to downstream tasks by introducing and updating only a small number of additional parameters. Currently, mainstream PEFT methods can generally be divided into three categories: the first is the adapter method, which achieves adaptation by inserting trainable small modules into the original network; the second is the prompt-tuning method, which fine-tunes by introducing trainable cue vectors into the input or hidden layers; and the third is the low-rank adaptation (LoRA) method, which approximates weight updates by introducing low-rank matrix branches alongside the pre-trained weights. Due to its advantage of merging the low-rank matrix back into the backbone weights during inference without introducing additional inference latency, it has become the most widely adopted PEFT scheme.
[0004] Existing research, while low-rank adaptation methods such as LoRA achieve good parameter efficiency by freezing backbone parameters and training only low-rank matrices, still result in a relatively large number of trainable parameters for edge devices. Hint-based fine-tuning methods, although adding fewer parameters, have limited adjustability of the backbone representation, leading to significantly lower accuracy compared to full-parameter fine-tuning in scenarios with few samples and complex tasks. In contrast, tensor decomposition-based methods can further compress trainable parameters by tens or even hundreds of times while maintaining considerable accuracy; however, existing work mostly focuses on single-structure compression and has not systematically addressed the training instability introduced by tensor decomposition. Furthermore, current research on efficient parameter fine-tuning is primarily conducted in GPU training environments, failing to fully consider the impact of strict storage, computing power, and power consumption constraints on the entire fine-tuning process, and lacking a systematic connection between parameter compression in the training phase and efficient low-bit-width deployment in the inference phase. Therefore, lightweight fine-tuning methods for large language models with high fine-tuning accuracy for edge devices remain a research direction that urgently needs further exploration.
[0005] Patent application CN118153715A discloses a method for fine-tuning a large language model based on low-rank matrix factorization. This method includes obtaining the pre-trained weight file of the large language model and the question-answer pairs required for fine-tuning; compressing the precision of the model parameters in the pre-trained weight file using a double quantization method; replacing the fully connected layers in the large language model structure with LoRA layers; performing two inverse quantizations on the precision-compressed model parameters in batches and calculating the output of the LoRA layer; calculating the loss based on the output of the LoRA layer in each batch, and dynamically updating the parameters of the LoRA layer based on backpropagation of the loss, until all batches are processed, outputting the fine-tuned large language model. However, this patent cannot completely solve the existing technical problems, nor can it meet the needs of this invention. Summary of the Invention
[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a lightweight fine-tuning system and method for large language models implemented at the edge.
[0007] The lightweight fine-tuning system for large language models with edge-oriented implementation provided by the present invention includes: The tensor chain decomposition and two-stage quantization module is used to reshape the low-rank update matrix of the linear layer in the large language model into a high-order tensor form, and apply tensor chain decomposition to obtain multiple tensor kernels. Then, two-stage quantization is applied to the multiple tensor kernels to obtain lightweight tensor kernel parameters. The amplitude and direction decoupling and adaptive optimization module is used to decompose the weight matrix corresponding to the lightweight tensor kernel parameters into amplitude components and direction components, and assign different learning rates to the upper and lower projection matrices. The knowledge distillation module is used to construct the knowledge distillation loss function by taking the output probability distribution of the teacher model as an additional supervision signal. The adapter deployment and inference module is used to export the lightweight tensor kernel parameters as an adapter file, load the pre-trained backbone model resident in the edge device, and read the corresponding tensor kernel from the adapter file when executing downstream tasks, reconstruct the complete update weight matrix online, and perform forward calculation after superimposing it with the backbone weight of the pre-trained backbone model.
[0008] Preferably, the pre-training weights are set as follows: The update process is represented as ,in and Let r be two low-rank update matrices, where r is the rank and satisfies ;Will and They are respectively reshaped into higher-order tensor forms, with the following shapes: and Then, apply TT decomposition to both, and represent each update matrix as follows: The chain product of TT kernels, denoted as and The expression is:
[0009] in, For the first One TT core The One side slice, For the first One TT core The One side slice, Let TT be the rank, where ; Represents the pre-trained weight matrix The number of rows and columns, This represents the model's output vector. This represents the input vector of the model. This represents the upprojection low-rank update matrix in low-rank fitting. This represents the downprojection low-rank update matrix in low-rank fitting. express The size of the d-th dimension after quantization express The size of the d-th dimension after quantization This represents the first rank of the TT decomposition. Denotes the terminal rank of the TT decomposition. and They are and The representation obtained after performing tensor chain decomposition.
[0010] Preferably, in the first stage of the two-stage quantization, 8-bit quantization is applied to each tensor kernel obtained from tensor chain decomposition, expressed as:
[0011] in, This represents the quantized tensor obtained after 8-bit quantization of the k-th TT core. Represents the quantization function. This represents the original tensor of the k-th TT kernel before quantization. Indicates the index of the TT core. This indicates the order of the TT decomposition.
[0012] Preferably, in the second stage of the two-stage quantization, during the process of gradually reconstructing the quantized tensor kernel into a complete linear layer, 8-bit quantization is applied again to the intermediate tensor obtained at each reconstruction step, as expressed in the following expression:
[0013] in, This represents the intermediate quantization tensor at step k. This represents the K-bit quantization function for the second stage. This represents the (i+1)th quantized TT core.
[0014] Preferably, the amplitude direction decoupling and adaptive optimization module decomposes the weight matrix into amplitude components. and directional components Two parts.
[0015] Preferably, the loss function constructed by the knowledge distillation module is defined as a weighted sum of cross-entropy loss and KL divergence loss.
[0016] Preferably, the adapter deployment and inference module exports the lightweight tensor kernel parameters and the corresponding amplitude components as independent lightweight adapter files; The adapter deployment and inference module distributes the adapter file over the network or pre-stores it locally in the non-volatile storage of the end device.
[0017] Preferably, the adapter deployment and inference module is located on an embedded device on the edge side, where the pre-trained backbone model is permanently loaded into the device's dynamic random access memory with bf16 precision.
[0018] Preferably, when the adapter deployment and inference module executes a specific downstream task, it reads the adapter file corresponding to the task from the non-volatile storage of the end device, loads the tensor kernel into the dynamic random access memory, and reconstructs the complete update weight matrix online from the tensor kernel for each fine-tuning layer, and performs forward calculation after superimposing it with the backbone weight.
[0019] The lightweight fine-tuning method for large language models with edge-oriented implementation provided by the present invention includes: Step 1: Reshape the low-rank update matrix of the linear layer in the large language model into a high-order tensor form, and apply tensor chain decomposition to obtain multiple tensor kernels; Step 2: Apply two-stage quantization to the multiple tensor kernels to obtain lightweight tensor kernel parameters; Step 3: Decompose the weight matrix corresponding to the lightweight tensor kernel parameters into amplitude components and direction components, and assign different learning rates to the upper and lower projection matrices; Step 4: Introduce a knowledge distillation mechanism, using the output probability distribution of the teacher model as an additional supervision signal to construct a knowledge distillation loss function; Step 5: Export the lightweight tensor kernel parameters as an adapter file, load the pre-trained backbone model resident in the edge device, and when executing downstream tasks, read the corresponding tensor kernel from the adapter file, reconstruct the complete update weight matrix online, and perform forward calculation after superimposing it with the backbone weight of the pre-trained backbone model.
[0020] Compared with the prior art, the present invention has the following beneficial effects: (1) This invention addresses the technical problem of insufficient accuracy of existing parameter-efficient fine-tuning methods under extremely low parameter budgets. By performing tensor chain decomposition on the low-rank update matrix, the number of trainable parameters is compressed to one-tenth or even one-hundredth of that of traditional low-rank fitting methods. Furthermore, a knowledge distillation mechanism is introduced, using the output probability distribution of the teacher model as an additional supervisory signal to compensate for the insufficient model representation capability under extremely low parameter quantities. Experimental data show that on the TinyLLaMA-1.1B model, this invention requires only 0.28M trainable parameters, achieving an average accuracy of 63.07% on six publicly available benchmark tasks. After introducing knowledge distillation, this accuracy is further improved to 66.56%, representing a 3.65% improvement over traditional low-rank fitting methods and an 11.42% improvement over existing tensor decomposition methods.
[0021] (2) This invention addresses the constraints of limited storage resources and computing power on edge devices by compressing tensor kernel parameters through a two-stage quantization technique. The first stage applies 8-bit quantization to each tensor kernel, reducing storage footprint; the second stage applies quantization again to intermediate tensors during reconstruction, reducing floating-point operation overhead. Experimental data shows that on the NVIDIA Jetson Orin Nano edge platform, the storage footprint of a single adapter is approximately 0.70 MB, and only 275,200 trainable parameters are required, demonstrating significant advantages in both storage and parameter efficiency compared to traditional low-rank adaptation methods and existing tensor decomposition methods.
[0022] (3) To address the technical problem of unstable training during fine-tuning in existing tensor decomposition methods, this invention introduces an amplitude-direction decoupling update mechanism. The weight matrix is decomposed into amplitude and direction components for separate optimization, and an adaptive optimization strategy is used to assign different learning rates to the upper and lower projection matrices, ensuring that the update steps of the two matrices match their respective dimensional features. These techniques effectively improve the training stability of the model after tensor decomposition and guarantee convergence of fine-tuning under extremely low parameter budgets.
[0023] (4) This invention addresses the technical problem of the lack of a systematic connection between parameter compression in the training phase and low-bit-width deployment in the inference phase in existing research on efficient parameter fine-tuning. It designs a complete edge deployment scheme. By exporting the quantized tensor kernel parameters as an independent lightweight adapter file, it supports network distribution or local pre-storage in the non-volatile storage of the edge device. During inference, the pre-trained backbone model is residently loaded with bf16 precision, and the adapter is loaded on demand and reconstructed and updated online, realizing fast switching of the adapter in multi-task scenarios. At the same time, it avoids repeated loading of the backbone model, which greatly reduces the memory occupation and inference latency of the edge device. Attached Figure Description
[0024] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a diagram illustrating the overall framework of a lightweight fine-tuning and edge deployment method for large language models based on dual-quantization tensor chain decomposition and decoupled update. Figure 2 Flowchart of a lightweight fine-tuning method for large language models implemented at the edge. Detailed Implementation
[0025] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0026] Example This invention addresses the problems of existing efficient parameter fine-tuning methods when deployed on edge devices, such as the still large number of trainable parameters, insufficient fine-tuning accuracy under extremely low parameter budgets, and training instability after tensor decomposition. It proposes a novel lightweight fine-tuning and edge deployment method for large language models. By improving the compression method and training mechanism of the low-rank update matrix, while maintaining or exceeding the fine-tuning accuracy of existing methods, it significantly reduces the number of trainable parameters and the adapter storage size. Furthermore, it enables efficient loading and rapid multi-task switching of the compressed adapter on edge embedded devices, thus providing a systematic lightweight fine-tuning and efficient deployment solution for large language models on edge hardware platforms with limited storage resources and computing power.
[0027] The overall framework of the proposed method for lightweight fine-tuning and edge deployment of large language models based on dual-quantization tensor chain (TT) decomposition and decoupled update is as follows: Figure 1 As shown, this method takes the linear layer in a large language model as the processing object. First, it performs tensor chain decomposition and two-stage quantization on the low-rank update matrix to obtain lightweight TT kernel parameters. Then, it introduces an amplitude-direction decoupling mechanism and an adaptive optimization strategy to perform stable and efficient fine-tuning training on the TT kernel parameters. Finally, the quantized TT kernel parameters are deployed as a lightweight adapter to an embedded device on the edge. Efficient inference is achieved through online weight reconstruction, and the adapter can be quickly switched in multi-task scenarios.
[0028] like Figure 2 The flowchart for a lightweight fine-tuning method for large language models implemented at the edge includes the following steps: Step 1: Reshape the low-rank update matrix of the linear layer in the large language model into a high-order tensor form, and apply tensor chain decomposition to obtain multiple tensor kernels; Step 2: Apply two-stage quantization to the multiple tensor kernels to obtain lightweight tensor kernel parameters; Step 3: Decompose the weight matrix corresponding to the lightweight tensor kernel parameters into amplitude components and direction components, and assign different learning rates to the upper and lower projection matrices; Step 4: Introduce a knowledge distillation mechanism, using the output probability distribution of the teacher model as an additional supervision signal to construct a knowledge distillation loss function; Step 5: Export the lightweight tensor kernel parameters as an adapter file, load the pre-trained backbone model resident in the edge device, and when executing downstream tasks, read the corresponding tensor kernel from the adapter file, reconstruct the complete update weight matrix online, and perform forward calculation after superimposing it with the backbone weight of the pre-trained backbone model.
[0029] The specific technical solution is as follows: 1) For linear layers in large language models that require fine-tuning, construct a low-rank update within the standard LoRA framework. Let the pre-trained weights be... The update process is represented as ,in and Let r be two low-rank update matrices, where r is the rank and satisfies Based on this, and They are respectively reshaped into higher-order tensor forms, with the following shapes: and Then, apply TT decomposition to both, and represent each update matrix as follows: The chain product of TT kernels, denoted as and As shown in formula (1):
[0030] in, For the first One TT core The One side slice, For the first One TT core The One side slice, Let TT be the rank, where ; Represents the pre-trained weight matrix The number of rows and columns, This represents the model's output vector. This represents the input vector of the model. This represents the low-rank update matrix of the up-projection in LoRA. This represents the low-rank update matrix of the downprojection in LoRA. express The size of the d-th dimension after quantization express The size of the d-th dimension after quantization This represents the first rank of the TT decomposition. This represents the terminal rank of the TT decomposition.
[0031] Secondly, a two-stage quantization is applied to the parameters after TT decomposition to further compress storage and reduce computational overhead. The quantization process is carried out in two stages. Taking the above projection matrix as an example: In the first stage, 8-bit quantization is applied to each TT core obtained from TT decomposition, as shown in formula (2):
[0032] in, This represents the quantized tensor obtained after 8-bit quantization of the k-th TT core. Represents the quantization function. This represents the original tensor of the k-th TT kernel before quantization. Indicates the index of the TT core. This indicates the order of the TT decomposition.
[0033] in The floating-point value is approximated to the nearest discrete value. In the second stage, during the process of gradually reconstructing the quantized TT kernel into a complete linear layer, the intermediate tensor obtained at each step of reconstruction is subjected to 8-bit quantization again, as shown in formula (3):
[0034] in, This represents the intermediate quantization tensor at step k. This represents the K-bit quantization function for the second stage. This represents the (i+1)th quantized TT core.
[0035] Similarly, the TT kernels of each projection matrix are quantized. During training, a pass-through estimator is used to maintain gradient propagation. The first stage compresses the storage footprint of the TT kernel, and the second stage reduces the floating-point operation overhead during reconstruction. The combination of these two approaches makes the model particularly suitable for edge devices with limited memory and computing power.
[0036] 2) Decompose the weight matrix into magnitude components. and directional components The system consists of two parts, and an adaptive optimization strategy is introduced, which assigns different learning rates to the upper and lower projection matrices to match the update steps of the two matrices with their respective dimensional features. Based on the above fine-tuning framework, a knowledge distillation mechanism is introduced. The output probability distribution of the teacher model is used as an additional supervision signal, and the knowledge distillation loss function is defined as a weighted sum of cross-entropy loss and KL divergence loss.
[0037] 3) Export the quantized TT core parameters and corresponding amplitude components obtained in the above steps as independent lightweight adapter files. These adapter files can be downloaded over the network or pre-stored locally in the non-volatile memory of the edge device. On the embedded edge device, the pre-trained backbone model is permanently loaded into the device's DRAM with bf16 precision, and the backbone model remains unchanged throughout its usage cycle. When executing a specific downstream task, the adapter file for the corresponding task is read from the device's non-volatile memory, and the TT core is loaded into the DRAM. For each fine-tuning layer, the complete updated weight matrix is reconstructed online from the TT core and compared with the backbone weights. After superposition, forward computation is performed.
[0038] Beneficial effects: 1) This invention demonstrates stable performance advantages on models with extremely low parameter budgets and very small scales. Taking the TinyLLaMA-1.1B model as an example, DuQTTA requires only 0.28M trainable parameters and achieves an average accuracy of 63.07% on six publicly available benchmark tasks. KD-DuQTTA, after incorporating knowledge distillation, further improves the average accuracy to 66.56% with the same number of parameters, representing a 3.65% improvement over LoRA and an 11.42% improvement over LoRETTArep. Table 1 lists a detailed performance comparison of different fine-tuning methods on TinyLLaMA-1.1B.
[0039] Table 1. Performance comparison of different fine-tuning methods on TinyLLaMA-1.1B
[0040] 2) The KD-DuQTTA method of this invention has significant resource advantages on the edge embedded platform. Using the NVIDIA Jetson Orin Nano development kit as the verification platform and TinyLLaMA-1.1B as the model, inference was performed with bf16 accuracy. This invention requires only 275,200 trainable parameters, with a single adapter storage footprint of approximately 0.70 MB. On six classification tasks, the average accuracy is improved by 3.54% compared to the LoRA method and by 11.18% compared to the LoRETTArep method. Tables 2 and 3 list a detailed performance and resource comparison of different methods on the edge platform.
[0041] Table 2. Performance and resource comparison of different fine-tuning methods on TinyLLaMA-1.1B on the edge platform.
[0042] Table 3. Performance and resource comparison of different fine-tuning methods on TinyLLaMA-1.1B on the edge platform.
[0043] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.
[0044] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A lightweight fine-tuning system for large language models oriented towards edge implementation, characterized in that, include: The tensor chain decomposition and two-stage quantization module is used to reshape the low-rank update matrix of the linear layer in the large language model into a high-order tensor form, and apply tensor chain decomposition to obtain multiple tensor kernels. Then, two-stage quantization is applied to the multiple tensor kernels to obtain lightweight tensor kernel parameters. The amplitude and direction decoupling and adaptive optimization module is used to decompose the weight matrix corresponding to the lightweight tensor kernel parameters into amplitude components and direction components, and assign different learning rates to the upper and lower projection matrices. The knowledge distillation module is used to construct the knowledge distillation loss function by taking the output probability distribution of the teacher model as an additional supervision signal. The adapter deployment and inference module is used to export the lightweight tensor kernel parameters as an adapter file, load the pre-trained backbone model resident in the edge device, and read the corresponding tensor kernel from the adapter file when executing downstream tasks, reconstruct the complete update weight matrix online, and perform forward calculation after superimposing it with the backbone weight of the pre-trained backbone model.
2. The lightweight fine-tuning system for large language models oriented to edge implementation according to claim 1, characterized in that, Let the pre-training weights be... The update process is represented as ,in and Let r be two low-rank update matrices, where r is the rank and satisfies ;Will and They are respectively reshaped into higher-order tensor forms, with the following shapes: and Then, apply TT decomposition to both, and represent each update matrix as follows: The chain product of TT kernels, denoted as and The expression is: in, For the first One TT core The One side slice, For the first One TT core The One side slice, Let TT be the rank, where ; Represents the pre-trained weight matrix The number of rows and columns, This represents the model's output vector. This represents the input vector of the model. This represents the upprojection low-rank update matrix in low-rank fitting. This represents the downprojection low-rank update matrix in low-rank fitting. express The size of the d-th dimension after quantization express The size of the d-th dimension after quantization This represents the first rank of the TT decomposition. Denotes the terminal rank of the TT decomposition. and They are and The representation obtained after performing tensor chain decomposition.
3. The lightweight fine-tuning system for large language models oriented to edge implementation according to claim 1, characterized in that, The first stage of the two-stage quantization involves applying 8-bit quantization to each tensor kernel obtained from tensor chain decomposition, expressed as follows: in, This represents the quantized tensor obtained after 8-bit quantization of the k-th TT kernel. Represents the quantization function. This represents the original tensor of the k-th TT kernel before quantization. Indicates the index of the TT core. This indicates the order of the TT decomposition.
4. The lightweight fine-tuning system for large language models oriented to edge implementation according to claim 3, characterized in that, In the second stage of the two-stage quantization, as the quantized tensor kernel is gradually reconstructed into a complete linear layer, 8-bit quantization is applied again to the intermediate tensor obtained at each reconstruction step, as expressed in the following expression: in, This represents the intermediate quantization tensor at step k. This represents the K-bit quantization function for the second stage. This represents the (i+1)th quantized TT core.
5. The lightweight fine-tuning system for large language models oriented to edge implementation according to claim 1, characterized in that, The amplitude direction decoupling and adaptive optimization module decomposes the weight matrix into amplitude components. and directional components Two parts.
6. The lightweight fine-tuning system for large language models oriented to edge implementation according to claim 1, characterized in that, The loss function constructed by the knowledge distillation module is defined as a weighted sum of cross-entropy loss and KL divergence loss.
7. The lightweight fine-tuning system for large language models oriented to edge implementation according to claim 1, characterized in that, The adapter deployment and inference module exports the lightweight tensor kernel parameters and the corresponding amplitude components as independent lightweight adapter files. The adapter deployment and inference module distributes the adapter file over the network or pre-stores it locally in the non-volatile storage of the end device.
8. The lightweight fine-tuning system for large language models oriented to edge implementation according to claim 1, characterized in that, The adapter deployment and inference module, located on the embedded device, loads the pre-trained backbone model at bf16 precision into the device's dynamic random access memory.
9. The lightweight fine-tuning system for large language models oriented to edge implementation according to claim 1, characterized in that, When executing a specific downstream task, the adapter deployment and inference module reads the corresponding adapter file from the non-volatile storage of the end device, loads the tensor kernel into the dynamic random access memory, and reconstructs the complete update weight matrix online from the tensor kernel for each fine-tuning layer. After superimposing it with the backbone weights, it performs forward computation.
10. A method for lightweight fine-tuning of a large language model with edge-oriented implementation, applied to any one of claims 1 to 9, characterized in that, include: Step 1: Reshape the low-rank update matrix of the linear layer in the large language model into a high-order tensor form, and apply tensor chain decomposition to obtain multiple tensor kernels; Step 2: Apply two-stage quantization to the multiple tensor kernels to obtain lightweight tensor kernel parameters; Step 3: Decompose the weight matrix corresponding to the lightweight tensor kernel parameters into amplitude components and direction components, and assign different learning rates to the upper and lower projection matrices; Step 4: Introduce a knowledge distillation mechanism, using the output probability distribution of the teacher model as an additional supervision signal to construct a knowledge distillation loss function; Step 5: Export the lightweight tensor kernel parameters as an adapter file, load the pre-trained backbone model resident in the edge device, and when executing downstream tasks, read the corresponding tensor kernel from the adapter file, reconstruct the complete update weight matrix online, and perform forward calculation after superimposing it with the backbone weight of the pre-trained backbone model.