Low-rank adaptation system based on differential polarization rank search

By using a low-rank adaptation system based on differentiable polarizable rank search, the problem of parameter waste caused by fixed rank allocation in low-rank adaptation methods is solved. Dynamic rank allocation and adaptive optimization are achieved, which improves model performance and reduces computational resource consumption. It is suitable for multi-task concurrency and resource-constrained device deployment.

CN121457560APending Publication Date: 2026-02-03ANHUI UNIV
0 Cites 1 Cited by

Patent Information

Application Number
CN202511457001.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-10-13
Publication Date
2026-02-03

AI Technical Summary

Technical Problem

Existing low-rank adaptation methods suffer from parameter waste due to fixed-rank allocation and lack dynamic perception and continuous rank allocation capabilities, making it difficult to achieve efficient parameter optimization. This is especially problematic in multi-task concurrency and resource-constrained device deployments.

Method used

A low-rank adaptive system based on differentiable polarization rank search is adopted. Dynamic rank allocation and adaptive optimization are achieved through modular architecture design, including an initialization module, a warm-up training module, a joint optimization module, a sparse pruning module, and a final fine-tuning module. The loss function is optimized by using L2 regularization and polarization regularization terms, and the rank allocation is dynamically adjusted by combining sparse pruning technology and neural architecture search.

Benefits of technology

It significantly reduces computing resource consumption and memory usage, improves model performance in downstream tasks, and is suitable for multi-task concurrency, resource-constrained device deployment, and large-scale model fine-tuning, achieving lightweight deployment.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention relates to the field of natural language processing, in particular to a low-rank adaptation system capable of differential polarization rank search, which comprises an initialization module, a warm-up training module, a joint optimization module, a sparse pruning module and a final fine tuning module. According to the method, dynamic rank allocation and continuous search are realized by introducing trainable rank architecture parameters, and model parameter waste and calculation overhead are remarkably reduced by combining polarization regularization and L2 regularization strategies. According to the method, the performance of the large model in the downstream task can be effectively improved, meanwhile, the method is suitable for scenes with multi-task concurrence, resource-constrained device deployment and high real-time requirements, and a technical solution is provided for efficient adaptation of the large-scale pre-training model.
Need to check novelty before this filing date? Find Prior Art