A method and system for model light fine-tuning and storage medium

Abstract

The application relates to a method for model lightweight fine-tuning, comprising the following steps: providing a parameter-frozen pre-trained base model; setting an adapter in a network layer of the base model; mapping features input into the network layer through a low-rank mapping unit contained in the adapter; wherein the mapping comprises the following steps: dynamically selecting a target rank, the target rank being a positive integer not greater than a preset maximum rank of the low-rank mapping unit; dynamically configuring an effective rank of the low-rank mapping unit according to the selected target rank; performing feature mapping by using the low-rank mapping unit with the configured effective rank to obtain adapted features; and fusing the adapted features with backbone path features of the network layer to obtain the output of the network layer. Through the above design, the application solves the balance problem between resource consumption and performance maintenance faced by large-scale pre-trained models in downstream task fine-tuning, and provides a feasible technical path for realizing efficient application of the model in a resource-limited environment.

Description

Technical Field

[0001] This invention relates to the field of computer vision and machine learning model optimization technology, and in particular to a method, system and storage medium for lightweight fine-tuning of models. Background Technology

[0002] Temporal Action Detection (TAD) aims to accurately locate the start and end times of actions and identify their categories in unedited long video sequences. In recent years, large-scale video foundation models based on the Transformer architecture (such as VideoMAE) have demonstrated outstanding performance in video understanding tasks. However, adapting these models to downstream TAD tasks typically requires fine-tuning.

[0003] Currently, the most direct fine-tuning method is full fine-tuning, which involves updating all parameters of the entire base model. While this method achieves good performance, it consumes enormous computational and storage resources. For example, for the VideoMAE-Small model (approximately 22M parameters), full fine-tuning requires over 24 GB of training memory, severely limiting its application in computationally constrained environments.

[0004] To address the resource challenges of full-scale fine-tuning, parameter-efficient fine-tuning techniques have emerged. These techniques aim to adapt pre-trained models by introducing a small number of trainable parameters. However, existing parameter-efficient fine-tuning methods still face challenges when applied to complex temporal action detection tasks. Some methods, while significantly reducing the number of parameters, may limit the model's ability to capture complex spatiotemporal features, leading to performance degradation; while other methods, although maintaining high performance, still have room for improvement in parameter efficiency, failing to sufficiently lower the deployment threshold. Therefore, how to overcome the severe computing power and storage bottlenecks faced by large-scale video base models when deployed in vertical scenarios, enabling them to achieve efficient and high-precision task adaptation on resource-constrained consumer hardware, has become an urgent industry challenge.

[0005] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The technical problem this application aims to solve is: how to fundamentally overcome the inherent contradiction between model performance and fine-tuning efficiency when fine-tuning large-scale pre-trained models for downstream tasks.

[0007] The technical solution adopted in this application to solve the above-mentioned technical problems is as follows.

[0008] This application proposes a method for lightweight fine-tuning of models, comprising the following steps: Provide a pre-trained base model and freeze its parameters; An adapter is set in at least one network layer of the base model; the features input to the network layer are mapped through the low-rank mapping units contained in the adapter. The mapping process includes: dynamically selecting a target rank, which is a positive integer not greater than a preset maximum rank of the low-rank mapping unit; dynamically configuring the effective rank of the low-rank mapping unit according to the selected target rank; and performing feature mapping using the low-rank mapping unit with the configured effective rank to obtain the adapted features. The adapted features are fused with the backbone path features of the network layer to obtain the output of the network layer.

[0009] In some embodiments, dynamically configuring the effective rank of the low-rank mapping unit includes: according to the target rank: r, extracting the first r rows of a predefined first low-rank matrix to obtain a first sub-matrix, and extracting the first r columns of a predefined second low-rank matrix to obtain a second sub-matrix; and using the first sub-matrix and the second sub-matrix to perform linear mapping on the features input to the network layer.

[0010] In some embodiments, after linearly mapping the features input to the network layer using the first submatrix and the second submatrix, the method further includes scaling the mapping result using a scaling factor.

[0011] In some embodiments, the adapter further includes a nonlinear activation branch; the low-rank mapping unit included in the adapter maps the features input to the network layer, specifically including: performing dimensionality reduction projection on the features input to the network layer to obtain bottleneck features; performing dynamic low-rank mapping on the bottleneck features through the low-rank mapping unit, and simultaneously performing nonlinear transformation on the bottleneck features through the nonlinear activation branch; fusing the result of the dynamic low-rank mapping with the result of the nonlinear transformation; and performing dimensionality increase projection on the fused features to obtain adapted features.

[0012] In some embodiments, the adapted features are fused with the backbone path features of the network layer by adding weighted residuals, where the weighting coefficients are learnable parameters.

[0013] In some embodiments, the base model is a video base model based on the Transformer architecture; the network layer is a Transformer module.

[0014] In some embodiments, a lightweight model fine-tuning system is also provided, comprising: The model provisioning and freezing module is configured to provide a pre-trained base model and freeze its parameters. An adapter setting module is configured to set up an adapter in at least one network layer of the base model, the adapter containing a low-rank mapping unit; The feature mapping module is configured to map the features input to the network layer through the low-rank mapping unit included in the adapter to obtain adapted features. The feature mapping module is configured to perform the following: dynamically select a target rank, which is a positive integer not greater than a preset maximum rank of the low-rank mapping unit; dynamically configure the effective rank of the low-rank mapping unit according to the selected target rank; and perform feature mapping using the low-rank mapping unit with the configured effective rank. The feature fusion module is configured to fuse the adapted features with the backbone path features of the network layer to obtain the output of the network layer.

[0015] In some embodiments, a lightweight fine-tuning method for a model for temporal action localization is also provided, comprising the following steps: Provide a pre-trained base model and freeze its parameters; An adapter is set in at least one network layer of the base model; the features input to the network layer are mapped through the low-rank mapping units contained in the adapter; wherein the mapping includes: dynamically selecting a target rank, the target rank being a positive integer not greater than a preset maximum rank of the low-rank mapping unit; dynamically configuring the effective rank of the low-rank mapping unit according to the selected target rank; and performing feature mapping using the low-rank mapping unit with the configured effective rank to obtain the adapted features. The adapted features are fused with the backbone path features of the network layer to obtain the output of the network layer, thereby obtaining multi-scale temporal features. Multi-scale temporal features are fed into a time-series action detection head without anchor boxes. Based on the multi-scale temporal features, the action category and the offset from the action start and end time boundaries are directly predicted by the time-series action detection head.

[0016] In some embodiments, a dynamic low-rank adapter is also provided, applied to the method for lightweight model fine-tuning of the present invention. The dynamic low-rank adapter includes a low-rank mapping unit configured to configure its effective rank according to a dynamically selected target rank to map features input to the network layer.

[0017] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the method of the present invention for lightweight model fine-tuning.

[0018] The present invention has the following beneficial effects: This invention, through the technical feature of "providing a pre-trained base model and freezing its parameters," fundamentally avoids updating massive amounts of model parameters during fine-tuning, thereby greatly reducing the consumption of computational and storage resources during training and laying the foundation for achieving extremely high fine-tuning efficiency. Through the technical features of "setting an adapter in at least one network layer of the base model" and "mapping the features input to the network layer through the low-rank mapping units contained in the adapter," a dedicated task adaptation interface with a very small number of parameters is introduced, isolating the fine-tuning process within a limited parameter space and significantly improving parameter efficiency. Furthermore, by "dynamically selecting a target rank" and "dynamically configuring low-rank mapping units according to the selected target rank," this invention further enhances efficiency. The technical feature of "effective rank of rank mapping units" uses a dynamic rank sampling mechanism to force low-rank mapping units to learn robust representations with nested structures during training. This enables low-rank mapping units to have strong feature capture and adaptation capabilities even with extremely low parameter budgets, thus ensuring that model performance does not decline due to a sharp reduction in the number of parameters. The technical feature of "fusing the adapted features with the backbone path features of the network layer" ensures that the task-specific knowledge learned by the adapter can be organically combined with the original general representation capabilities of the pre-trained base model. Without destroying the powerful capabilities of the original model, the information required by the downstream task is smoothly injected, further stabilizing the final model performance.

[0019] In summary, this invention, through the synergistic combination of the aforementioned technical features, constitutes an efficient parameter fine-tuning paradigm. This paradigm significantly reduces reliance on computational and storage resources by freezing the base model parameters, laying the foundation for efficient fine-tuning. Through the dynamic low-rank mapping unit in the adapter, flexible and robust task adaptation is achieved within an extremely limited parameter space, ensuring model performance on downstream tasks. Finally, feature fusion effectively integrates the specific knowledge learned by the adapter with the general capabilities of the base model, ensuring the integrity of model performance. These technical features support and synergize with each other, jointly addressing the challenge of balancing resource consumption and performance maintenance when fine-tuning large-scale pre-trained models for downstream tasks from different dimensions, providing a feasible technical path for the efficient application of models in resource-constrained environments.

[0020] Other beneficial effects of the present invention will be further described below. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is an architecture diagram of a timing action detection system based on a dynamic low-rank adapter (ViDA Adapter); Figure 2 A comparison diagram of the internal structure of the ViDA Adapter (Dynamic Low-Rank Adapter); Figure 3 This is a comparison chart of detection accuracy on the THUMOS-14 dataset; Figure 4 This is a comparison diagram of the memory of the present invention and the prior art.

[0022] Figure 5 The figure shows the ablation experiment results of this invention; Figure 6 This is a schematic diagram of the Dynamic LoRA architecture. Detailed Implementation

[0023] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0024] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0025] This invention recognizes that the fundamental problem of existing efficient parameter fine-tuning methods—the difficulty in balancing model performance and fine-tuning efficiency—lies in the fact that fixed adapter structures or fixed low-rank approximations cannot effectively capture multi-scale, dynamically evolving feature patterns in downstream tasks (especially complex tasks such as temporal action detection) within a limited parameter space. This limitation is the root cause of the inherent contradiction between model performance and fine-tuning efficiency. To address this issue, this invention utilizes a dynamic rank sampling and configuration mechanism to guide the low-rank parameter space to self-organize into a multi-level representation structure with nested characteristics during training. Accordingly, this invention proposes an adapter scheme that embeds a dynamically configurable low-rank mapping unit in the bypass of the frozen pre-trained model network layer. This scheme aims to overcome the expression bottleneck of fixed rank through dynamism, thereby significantly improving the adapter's ability to model complex features (high performance) while maintaining an extremely low number of trainable parameters (high efficiency), ultimately fundamentally reconciling the contradiction between model performance and fine-tuning efficiency.

[0026] Specifically, during the feature extraction stage, the pre-trained weights of the backbone network are completely frozen. VideoMAE-S, pre-trained on large-scale video data based on a masked autoencoder, has learned a general spatiotemporally aware manifold with high generalization ability. When faced with temporal action localization datasets containing a large amount of redundant background and complex motion interference, forcibly updating global parameters using the traditional full-scale fine-tuning paradigm can easily lead to severe catastrophic forgetting. Therefore, freezing the backbone network not only greatly reduces memory overhead but also preserves the robust representation ability of the underlying layers for general video spatiotemporal patterns.

[0027] In some embodiments, reference Figure 1 The entire architecture mainly consists of three parts: a frozen backbone network's basic feature extraction layer (network layer), a dynamically low-rank adapter (ViDA Adapter) injected via selective bypass, and a task-oriented temporal action detection head (used to process the network layer's output). The method for lightweight model fine-tuning in this invention introduces a dynamically low-rank adapter (ViDA Adapter) into the bypass of the frozen backbone network's basic layer. Essentially, this adapter learns a high-density task-adaptive manifold specifically for action localization tasks within a very small parameter space without rewriting the global basic representation. By introducing a dynamic rank sampling mechanism during training, the model can adaptively meet the differentiated requirements of different layers for effective rank, thus smoothly transitioning and projecting the general continuous video features extracted by the video base model onto the temporal detection head with extremely low parameter cost, ultimately achieving high-precision action boundary regression and classification.

[0028] In some embodiments, reference Figure 2 The Dynamic Low-Rank Adapter (ViDA Adapter) employs a bottleneck-driven dual-parallel architecture, inserted as a lightweight, fine-tunable component after each Transformer module in the pre-trained backbone network. Its overall feature transformation process mainly includes four stages: dimensionality reduction projection, dual-parallel mapping, dimensionality increase projection, and residual connections. Unlike traditional simple serial adapters, the dual-parallel design allows nonlinear feature transformation and low-rank task adaptation to be computed in parallel and complement each other within a very small bottleneck space. This design not only significantly reduces the number of parameters and training overhead while maintaining the model's expressive power, but also naturally supports adjusting the effective rank on demand during the inference phase based on hardware resources.

[0029] In some embodiments, for each Transformer module, let its hidden layer input features be... (Where the original embedding dimension d=384), the input features are projected through a dimension reduction matrix. Mapping to the preset bottleneck dimension b (this paper sets the compression rate to 0.25, i.e., b=96), as shown in the following formula: This bottleneck design effectively controls the amount of floating-point operations required for subsequent dual-path computation, and also forces the model to adapt to the task in a more compact representation space, thereby improving the generalization ability of the features.

[0030] In obtaining bottleneck characteristics The model then feeds the features into two parallel branches. The first branch is a standard nonlinear activation branch to extract basic local nonlinear features, as shown in the following formula:

[0031] The second path is the dynamic low-rank mapping branch, used to capture task-specific spatiotemporal context changes. The computation of this branch encompasses three closely linked steps: dynamic rank sampling, low-rank decomposition, and scaling. Its overall calculation is shown in the following formula:

[0032] First, during the dynamic rank sampling phase, the system will, in each forward propagation of training, sample data from the interval [1, r]. max A target rank r is uniformly and randomly sampled from the top (the maximum rank r is set in this experiment). max (16). This random sampling forms a nested structure in the parameter space, ensuring that even subnetworks with minimal rank can learn effective spatiotemporal representations, thus allowing submatrices to be truncated as needed during the inference phase. The specific mechanism and manifestation of this nested structure are as follows: because the sampling follows a uniform distribution, the subset of parameters with earlier indices (i.e., smaller rank) in the low-rank matrix has a significantly higher probability of participating in forward and backward propagation during training than parameters with later indices. This asymmetric optimization frequency forces the network to compress the most core and generalizable action spatiotemporal representations into a smaller-rank subspace, while parameters corresponding to larger rank tend to fit finer-grained residual features. The second step is the low-rank decomposition operation, as shown in the formula... and For low-rank matrices, the former uses Kaiming initialization, while the latter uses zero initialization to ensure that the main path is not disturbed in the early stages of training; during calculation, the model dynamically extracts the first r rows of A and the first r columns of B based on the sampled rank r (i.e., and This results in an effective transformation that significantly reduces the number of parameters while maintaining expressive power.

[0033] Finally, dynamic scaling is implemented. To avoid abnormal fluctuations in the output feature amplitude when sampling at larger ranks, a scaling factor is introduced in the calculation (a is a fixed scaling factor, which is set to a=32 in this paper). This design makes the effective feature scale roughly the same under different ranks, which greatly improves the stability of end-to-end joint optimization.

[0034] Subsequently, the features were processed through two parallel branches. and The element-by-element addition and fusion within the bottleneck space is calculated using the following formula: Features after fusion After dimensional projection matrix Map back to the original embedding dimension and apply residual connections, as shown in the following formula: Among them, residual scalar As a learnable parameter, the initial value is set to 1 to ensure stability in the early stages of training and avoid drastic perturbation of the pre-trained general features.

[0035] The following will further describe specific embodiments of the present invention. These embodiments are merely illustrative and do not mean that the present invention is limited to the following examples.

[0036] First, the unedited video is sampled and segmented into video segments containing 768 frames each. The pre-trained parameters of the 12 Transformer layers of the pre-trained backbone network are completely frozen. The ViDA adapter designed in this invention is inserted into all 12 layers of the pre-trained backbone network, such as... Figure 1 As shown.

[0037] The multi-scale temporal features output from the pre-trained backbone network are ultimately fed into an anchorless detection head to predict the start and end points of actions and their categories. The model uses a weighted sum of Focal Loss and Distance Intersection over Union (DIoU) Loss for backpropagation updates. The anchorless detection head does not require pre-setting a large number of temporal prior boxes (anchors) of different scales; instead, it directly classifies the feature points at each timestamp and regresses to predict the temporal distance between that feature point and the actual start and end points of the action. This prediction mechanism naturally aligns with DIoU Loss. The model ultimately uses a weighted sum of Focal Loss for the classification task and DIoU Loss for the boundary regression task to complete the joint update of backpropagation and network parameters.

[0038] like Figure 2 As shown, for any layer of the pre-trained backbone network, the forward computation process of the ViDA adapter is decomposed into the following steps: (1) The input feature X is mapped to the preset bottleneck dimension through a fully connected layer (in this embodiment, the compression rate is set to 0.25, that is, the bottleneck dimension is 96).

[0039] (2) The dimensionality-reduced features are simultaneously input into two parallel branches. The first branch is the GELU nonlinear activation branch; the second branch is the Dynamic LoRA branch (i.e., low-rank mapping unit). The Dynamic LoRA branch contains a first low-rank matrix A and a second low-rank matrix B, with the maximum rank set to 16. The outputs of the two branches are added element-wise.

[0040] (3) Dimensional Upgrading Fully Connected Projection and Residual Connection: The fused features are upgraded from 96 dimensions to the original 384 dimensions through a fully connected layer. After being multiplied by the learnable scalar gamma, they are added to the original input features X to form a residual, and the final features are output.

[0041] In this structure, a single adapter has only 77,281 parameters, and a total of approximately 927K trainable parameters across 12 layers.

[0042] To avoid overfitting the model to a single low-rank structure, this embodiment introduces a dynamic rank sampling mechanism during the training phase. In each forward iteration during training, the system obtains the target rank r of the current iteration by uniformly distributing random sampling from the interval [1, 16].

[0043] Subsequently, the system truncates the low-rank matrix A to the first r rows and the matrix B to the first r columns. It then uses the truncated submatrices to perform a linear mapping on the features and multiplies them by a dynamic scaling factor alpha (in this embodiment, the hyperparameter alpha is fixed at 32). This mechanism forces the first rows / columns of the matrix to learn the most crucial low-frequency temporal information, while the latter rows / columns learn high-frequency detail information, forming a high-order temporal representation with nested characteristics.

[0044] After 60 epochs of training, the inference phase begins. At this point, dynamic sampling stops, and forward propagation is performed directly using the preset maximum rank (i.e., the complete A and B matrices).

[0045] Regarding DyLoRA, the following needs to be explained: refer to Figure 6 The core idea of ​​DyLoRA is to dynamically truncate and sample low-rank matrices during training, forcing the model to learn a multi-rank nested representation space in a single training cycle. Specifically, when fine-tuning using DyLoRA, a maximum rank r needs to be preset first. max and initialize the corresponding full low-rank matrix. and In each forward pass during the training phase, the system no longer uses the complete A and B matrices, but instead starts from a pre-defined interval [1, r]. maxWithin a given range, a target rank r is dynamically sampled according to a uniform distribution, as shown in the following formula:

[0046] After determining the dynamic rank *r* of the current iteration, the model performs row / column truncation on the original low-rank matrix. Specifically, matrix A is dynamically truncated by removing the first *r* rows, and matrix B is dynamically truncated by removing the first *r* columns, denoted as ... and At this point, the calculation process of this forward propagation step is reconstructed as shown in the following formula:

[0047] During the backpropagation phase, the gradient is only propagated back and updates the parameters of the truncated r rows and r columns; the undisturbed parts remain unchanged in the current iteration. Because the front part of the parameters (i.e., the rows / columns with smaller indices) is frequently selected and updated in multiple random samplings, the model is forced to concentrate the most core and important low-frequency task knowledge in the head of the low-rank matrix; while the parameters at the tail (the rows / columns with larger indices) are used to supplement high-frequency detailed features.

[0048] refer to Figure 3 , 4 5. Experimental data shows that the overall Avg-mAP of the present invention reaches 67.97% on the THUMOS-14 dataset, proving that the "dynamic low-rank dual-channel adapter" of the present invention successfully solves the problem of lightweight fine-tuning of end-to-end temporal action detection in long videos under the extreme constraint of consuming only 3.9GB of video memory.

[0049] Specifically, Figure 3 This paper presents a comparison of the detection accuracy of the ViDA-TAD method with a fully frozen backbone network baseline on the THUMOS-14 dataset. Despite the limitations of the extremely small parameter space, ViDA-TAD still outperforms the baseline model across all IoU thresholds, achieving an overall Avg-mAP of 67.97%, a significant improvement of 9.39% compared to the fully frozen baseline model, demonstrating its strong task adaptability.

[0050] Figure 4 compares the model engineering overhead metrics. In long video end-to-end training tasks, full fine-tuning requires updating all pre-trained parameters of the backbone network (approximately 22M) and storing a large number of optimizer states and intermediate activation values, resulting in a single-card training memory requirement of up to 24.6 GB, placing high demands on computing hardware. In contrast, the ViDA-TAD method freezes the backbone network, controlling the number of trainable parameters to approximately 927K. Under this constraint, the model's training memory is effectively reduced to 3.9 GB.

[0051] Figure 5 illustrates the module-level ablation experiments. After introducing a dynamic low-rank branch separately on top of the baseline model, the average accuracy of the model improved from 58.58% to 64.15%. This demonstrates the feasibility and efficiency of applying low-rank approximation theory to the end-to-end temporal action localization architecture. When the complete dual-path adapter architecture ViDA Adapter, which includes nonlinear activation functions, was constructed, the model performance reached its optimal level of 67.97%. Compared to networks containing only linear low-rank branches, the additional nonlinear bypass contributed an additional 3.82% improvement in accuracy. This indicates that the severe background noise interference and drastic changes in action scale in temporal action localization tasks often require a highly nonlinear feature transformation space for high-dimensional fitting. The dual-path collaborative architecture of this invention successfully overcomes the theoretical limitations of pure linear low-rank matrices in expressing nonlinear spatiotemporal mapping relationships, enabling lightweight end-to-end models to achieve high-precision temporal action boundary localization and deep semantic classification with extremely low parameter tuning.

[0052] In summary, this application has the following advantages compared with the prior art: Focusing on the computational and storage bottlenecks of large-parameter video foundation models when deployed in vertical scenarios, this paper innovatively proposes a method for lightweight model fine-tuning. Addressing the industry pain points of high memory overhead and high loading latency in cloud deployments for full fine-tuning, this method, while completely freezing the VideoMAE-S spatiotemporal feature extraction base, uses a bypass injection mechanism to isolate task-specific knowledge of action localization within a very small parameter space for efficient learning.

[0053] In terms of core component design, a dual-path adapter (ViDAAdapter) with nonlinear activation and dynamic low-rank branching was constructed. The innovatively introduced dynamic rank sampling mechanism allows the model to learn high-order temporal representations with nested characteristics in a single joint optimization, giving the network the flexibility to truncate effective rank as needed during inference, effectively overcoming the theoretical limitations of traditional static low-rank fine-tuning when dealing with complex action boundaries. Specifically, the dynamic rank sampling mechanism randomly selects the activation sub-matrix during each forward propagation, which is mathematically equivalent to applying a structured dropout and parameter space regularization to the rank dimension of the features. Traditional static low-rank matrices are prone to co-adaptation of internal parameters during training; once forcibly truncated during inference, their representational power drops drastically. Dynamic sampling forcibly breaks this dependency, forcing each sub-part of the low-rank matrix (i.e., the sub-matrix corresponding to different target rank r) to... and Each of these must independently optimize the loss function, thus requiring independent and effective spatiotemporal representation capabilities. Furthermore, since the sampling mechanism operates within the interval [1, r...],... mar Uniform random sampling within the low-rank matrix results in parameters at different index positions experiencing asymmetric optimization frequencies. Parameters at the earlier indices (minimum rank space) are reused and updated in most forward computations, while parameters at later indices are only updated when a larger rank is sampled. This decreasing update frequency mechanism forces the network to perform an implicit principal component analysis (PCA): to minimize the overall expected loss, the model must compress the most core, fundamental information for action semantic classification into the first row / column of parameters, while distributing fine-grained information such as handling complex action boundaries and motion ambiguity to subsequent additional spaces. This decreasing information density arrangement naturally constructs a highly nested manifold structure within the parameter space, thus ensuring, in principle, the smoothness and robustness of truncating arbitrary ranks as needed during the inference phase.

[0054] Validation results on the THUMOS-14 dataset show that ViDA-TAD achieves a mean accuracy (Avg-mAP) of 67.97% even under stringent engineering constraints, including only 4.2% of trainable parameters (approximately 927K) and a 6.3-fold reduction in single-card training memory (down to 3.9 GB), representing a significant improvement of 9.39% over the frozen baseline. In summary, this method achieves an excellent balance between algorithmic localization accuracy and engineering deployment overhead.

[0055] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] The background section of this invention may include background information about the problems or environment in which the invention is being developed, and is not necessarily a description of prior art. Therefore, the content included in the background section does not constitute an admission of prior art by the applicant.

[0060] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A method for lightweight fine-tuning of a model, characterized in that, Includes the following steps: Provide a pre-trained base model and freeze its parameters; An adapter is set in at least one network layer of the base model; the features input to the network layer are mapped through the low-rank mapping units contained in the adapter; The mapping process includes: dynamically selecting a target rank, which is a positive integer not greater than a preset maximum rank of the low-rank mapping unit; dynamically configuring the effective rank of the low-rank mapping unit according to the selected target rank; and performing feature mapping using the low-rank mapping unit with the configured effective rank to obtain adapted features. The adapted features are fused with the backbone path features of the network layer to obtain the output of the network layer.

2. The method for lightweight fine-tuning of a model according to claim 1, characterized in that, The effective rank of the low-rank mapping unit is dynamically configured as follows: according to the target rank r, the first r rows of the predefined first low-rank matrix are truncated to obtain the first sub-matrix, and the first r columns of the predefined second low-rank matrix are truncated to obtain the second sub-matrix; the first sub-matrix and the second sub-matrix are used to perform linear mapping on the features input to the network layer.

3. The method for lightweight fine-tuning of a model according to claim 2, characterized in that, After linearly mapping the features input to the network layer using the first submatrix and the second submatrix, the method further includes scaling the mapping result using a scaling factor.

4. The method for lightweight fine-tuning of a model according to claim 1, characterized in that, The adapter also includes a non-linear activation branch; The adapter includes a low-rank mapping unit that maps features input to the network layer. Specifically, this includes: performing dimensionality reduction projection on the features input to the network layer to obtain bottleneck features; performing dynamic low-rank mapping on the bottleneck features through the low-rank mapping unit, while simultaneously performing nonlinear transformation on the bottleneck features through the nonlinear activation branch; fusing the result of the dynamic low-rank mapping with the result of the nonlinear transformation; and performing dimensionality increase projection on the fused features to obtain the adapted features.

5. The method for lightweight fine-tuning of a model according to claim 1, characterized in that, The adapted features are fused with the backbone path features of the network layer by adding weighted residuals, where the weighting coefficients are learnable parameters.

6. The method for lightweight fine-tuning of a model according to claim 1, characterized in that, The basic model is a video basic model based on the Transformer architecture; the network layer is a Transformer module.

7. A lightweight model fine-tuning system, characterized in that, include: The model provisioning and freezing module is configured to provide a pre-trained base model and freeze its parameters. An adapter setting module is configured to set an adapter in at least one network layer of the base model, the adapter comprising a low-rank mapping unit; The feature mapping module is configured to map the features input to the network layer through the low-rank mapping unit included in the adapter to obtain adapted features; wherein, the feature mapping module is configured to perform: dynamically selecting a target rank, the target rank being a positive integer not greater than a preset maximum rank of the low-rank mapping unit; dynamically configuring the effective rank of the low-rank mapping unit according to the selected target rank; and performing feature mapping using the low-rank mapping unit with the configured effective rank. The feature fusion module is configured to fuse the adapted features with the backbone path features of the network layer to obtain the output of the network layer.

8. A lightweight fine-tuning method for a model used in temporal action localization, characterized in that, Includes the following steps: Provide a pre-trained base model and freeze its parameters; An adapter is set in at least one network layer of the base model; features input to the network layer are mapped through low-rank mapping units included in the adapter; wherein the mapping includes: dynamically selecting a target rank, the target rank being a positive integer not greater than a preset maximum rank of the low-rank mapping unit; dynamically configuring the effective rank of the low-rank mapping unit according to the selected target rank; and performing feature mapping using the low-rank mapping unit with the configured effective rank to obtain adapted features. The adapted features are fused with the backbone path features of the network layer to obtain the output of the network layer, thereby obtaining multi-scale temporal features. The multi-scale temporal features are fed into a time-series action detection head without anchor boxes; the action detection head directly predicts the type of action and the offset from the start and end time boundaries of the action based on the multi-scale temporal features.

9. A dynamic low-rank adapter, characterized in that, The method for lightweight model fine-tuning according to any one of claims 1 to 10, wherein the dynamic low-rank adapter includes a low-rank mapping unit configured to configure its effective rank according to a dynamically selected target rank to map features input to the network layer.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method for lightweight model fine-tuning as described in any one of claims 1 to 10.

Images