Dynamic parameter efficient fine tuning method and device for visual model

Abstract

The invention relates to a dynamic parameter efficient fine tuning method and device for a visual model. The method comprises the following steps: constructing a double-layer dynamic parameter selection mechanism, dynamically allocating parameter update budget for a model according to gradient norms of network layers at a macroscopic level, and converting into selection probability according to parameter gradient at a microscopic level, and accurately identifying and activating a parameter subset which is most critical to a current task; meanwhile, a cross-cycle parameter distribution transfer engine is designed, historical and current key parameters are fused, and a momentum Top-K secondary screening strategy is utilized, so that disastrous forgetting is effectively inhibited and stable migration of knowledge is ensured while the exploratory property of the model for new intelligence is maintained; and finally, the model is efficiently updated through the generated sparse mask, so that the model can quickly master specific image target classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a dynamic parameter efficient fine-tuning method and device for a visual model. BACKGROUND

[0002] With the wide application of unmanned aerial vehicle (UAV) technology in fields such as civilian aerial photography, power inspection, agricultural plant protection, and emergency rescue, UAV image processing has become the core support for improving operation efficiency and decision-making accuracy. Visual-based intelligent perception technology plays an irreplaceable role in UAV target detection, image semantic segmentation, and scene risk assessment. The "pre-training-fine-tuning" paradigm provides an effective way to quickly adapt large visual base models trained based on massive general image data to specific UAV image processing tasks, and has become an important technical direction in the field of UAV intelligent application.

[0003] However, the particularity of the UAV operation environment poses severe challenges to model deployment. First, with the number of visual base model parameters reaching tens of billions or even higher, full fine-tuning on embedded computing modules, lightweight edge devices, and other hardware carried by UAVs brings unbearable computational resource consumption and storage pressure, severely restricting the real-time response capability during UAV operation. Second, the UAV operation scene is complex and variable, ranging from dense urban building groups to remote mountain forests, from high-altitude wide-range aerial photography to low-altitude close-range inspection, and the diversified scene requires the model to have rapid adaptation capability. Traditional full fine-tuning process is cumbersome and time-consuming, and it is difficult to meet the timeliness requirements of flexible UAV operation.

[0004] As a countermeasure, parameter efficient fine-tuning technology can exhibit performance comparable to or even better than full fine-tuning by optimizing only a small number of key parameters, significantly reducing hardware resource consumption, and providing a feasible path for real-time and lightweight deployment of UAV image processing. Among the many parameter efficient fine-tuning methods, selective fine-tuning is an important technical route. This method does not introduce any additional parameters, and its core idea is to identify and update only the parameter subset most relevant to a specific UAV image processing task, such as line feature extraction parameters for power inspection tasks or crop shape recognition parameters for agricultural plant protection tasks, thereby achieving efficient and accurate knowledge adaptation and meeting the strict requirements of low latency and high reliability for UAV operation.

[0005] The existing selective fine-tuning technology mainly follows two types of processing paradigms: one is a fixed module optimization method, which limits the optimization range to specific components predefined in the model, such as bias terms or specific network layers, but the selection granularity is coarse, which is difficult to capture the fine-grained image features related to unmanned aerial vehicle target detection, scene semantic segmentation and other tasks; the second is a static subset selection method, which relies on importance metrics such as gradient and Fisher information, and selects a fixed sparse parameter subset for optimization based on a small number of unmanned aerial vehicle samples before training. Although this method has achieved good performance and high computational efficiency in regular vision tasks, it has significant limitations in unmanned aerial vehicle image processing applications. First, the unmanned aerial vehicle operating environment presents a high degree of dynamicity and complexity, from high-altitude strong light direct radiation to low-altitude shadow obstruction, from clear imaging on sunny days to image interference in rainy, foggy and dusty weather. The distribution of visual data obtained by unmanned aerial vehicles has high heterogeneity and feature diversity. The existing static optimization strategy follows a fixed and unchanging parameter selection mechanism, ignores the dynamic evolution of the model's cognitive state during unmanned aerial vehicle image learning, and cannot adaptively adjust the optimization strategy according to different operating environments and image characteristics, which becomes a key bottleneck restricting the performance of unmanned aerial vehicle vision models. Second, in actual unmanned aerial vehicle operations, high-quality labeled data is often extremely scarce, and special scene data acquisition is difficult, and target sample distribution is uneven. In these complex operating scenarios with scarce and unevenly distributed data, the performance of static fine-tuning methods degrades significantly, which may lead to missed detection or misjudgment of key targets, and it is difficult to meet the strict requirements of high reliability and strong robustness in the fields of power inspection, agricultural production and emergency rescue. SUMMARY

[0006] Therefore, it is necessary to provide a dynamic parameter efficient fine-tuning method and device for a vision model capable of improving image processing accuracy in view of the above technical problems.

[0007] A dynamic parameter efficient fine-tuning method for a vision model, the method comprising: obtaining a downstream data set for a specific unmanned aerial vehicle image processing task, and replacing the original classification head of a pre-trained vision base model with a new classification head adapted to the target classes of the downstream data set, while constructing a loss function required for model optimization; initializing a binary update mask with the same dimension as the parameter set of the pre-trained vision base model, and setting a parameter update period, when the number of training iteration steps meets the periodic scheduling condition, triggering the subsequent parameter selection and update process; calculating the gradient of the parameters of the pre-trained vision base model at the previous iteration time based on the loss function, calculating the importance score of each layer according to the gradient characteristics of each network layer, determining the parameter update budget allocation ratio of each layer according to the importance score of each layer, and finally allocating the total update budget to each network layer according to the ratio to obtain the parameter update quota of each layer; For each layer parameter update quota, the gradient feature of each parameter is calculated as the parameter importance score in each network layer, the parameter importance score is converted into a parameter selection probability distribution, and the corresponding number of parameters is sampled without repetition in each layer according to the probability distribution, and all the sampled parameters constitute the preliminary candidate update set of the current period; The final update set of the last update period is obtained, the preliminary candidate update set is combined with the final update set of the last update period, and an extended candidate pool is constructed; the comprehensive importance score of each parameter in the extended candidate pool after the fusion of the historical momentum information and the current gradient feature is calculated, and the parameters with the highest comprehensive importance score and the number equal to the total update budget are selected from the extended candidate pool to constitute the final update set of the current period; According to the final update set, the initialized binary update mask is updated, and the parameters of the pre-trained visual basic model are sparsely updated according to the binary update mask and the gradient information, and the updated model is applied to the downstream unmanned aerial vehicle image processing task data, and the target class probability distribution is output.

[0008] A dynamic parameter efficient fine-tuning device for a visual model, the device comprising: A downstream task adaptation module is configured to obtain a downstream data set for a specific unmanned aerial vehicle image processing task, and replace the original classification head of the pre-trained visual basic model with a new classification head adapted to the target classes of the downstream data set, while constructing a loss function required for model optimization; An update strategy initialization module is configured to initialize a binary update mask with the same dimension as the parameter set of the pre-trained visual basic model, and set a parameter update period, and trigger the subsequent parameter selection and update process when the number of training iteration steps meets the periodic scheduling condition; A hierarchical update budget allocation module is configured to calculate the gradient of the parameters of the pre-trained visual basic model at the last iteration based on the loss function, calculate the importance score of each layer based on the gradient feature of each network layer, determine the parameter update budget allocation proportion of each layer according to the importance score of each layer, and finally allocate the total update budget to each network layer according to the proportion to obtain the parameter update quota of each layer; An intra-layer parameter preliminary selection module is configured to calculate the gradient feature of each parameter as the parameter importance score in each network layer for each layer parameter update quota, convert the parameter importance score into a parameter selection probability distribution, and sample the corresponding number of parameters without repetition in each layer according to the probability distribution, and all the sampled parameters constitute the preliminary candidate update set of the current period; The cross-cycle parameter selection module is used to obtain the final update set of the previous update cycle, merge the preliminary candidate update set with the final update set of the previous update cycle to construct an extended candidate pool; calculate the comprehensive importance score of each parameter in the extended candidate pool after fusing the historical momentum information with the current gradient features, and select the parameters with the highest comprehensive importance score and whose number is equal to the total update budget from the extended candidate pool to form the final update set of the current cycle. The sparse update and task inference module is used to update the initialized binary update mask based on the final update set. It then performs sparse updates on the parameters of the pre-trained visual base model based on the binary update mask and gradient information. The updated model is applied to downstream UAV image processing task data, outputting the target category probability distribution. The aforementioned method and apparatus for efficient fine-tuning of dynamic parameters for visual models first performs model reading and data processing, loading a pre-trained large visual model and preparing task data for a specific UAV image processing target. Subsequently, it precisely triggers the parameter reselection and update process according to a preset cycle. Based on this, it dynamically allocates update budgets for each network layer based on the total gradient norm, and then probabilistically selects parameters within each layer based on the gradient magnitude of individual parameters, forming a preliminary candidate parameter set. This candidate set is merged with the updated parameter set from the previous cycle, and through a momentum-based secondary screening mechanism, it determines the optimal parameter subset for the current iteration, balancing the exploratory nature of new UAV operating scenario information with the stability of historical knowledge. This achieves accurate knowledge adaptation of the pre-trained visual model to downstream UAV image processing tasks. Attached Figure Description

[0009] Figure 1 This is a flowchart illustrating an efficient method for fine-tuning dynamic parameters of a vision model in one embodiment. Figure 2 This is a framework diagram of a dynamic parameter fine-tuning device for a vision model in one embodiment. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0011] In one embodiment, such as Figure 1 As shown, an efficient method for fine-tuning dynamic parameters of visual models is provided, including the following steps: Step 102: Obtain the downstream dataset for a specific UAV image processing task, replace the original classification head of the pre-trained visual base model with a new classification head adapted to the target category of the downstream dataset, and construct the loss function required for model optimization.

[0012] This step is the initialization preparation for fine-tuning tasks. Taking the UAV image processing as an example, first, a visual basic model pre-trained on a large amount of general image data is loaded, and its complete parameter set is represented as , which contains network layers, represented as: ; Subsequently, the downstream dataset for image classification can be represented as: ; where is a satellite image slice containing a single image target, and is its corresponding one-hot encoded class label.

[0013] In order to enable the model to handle the classification task, the original output layer is replaced with a new classification head for image categories. For input image , the model's output prediction probability distribution is: ; The goal of the entire fine-tuning process is to minimize the cross-entropy loss function between the model's prediction and the true label: ; where represents the cross-entropy loss function, represents the parameter set of the pre-trained visual basic model, represents the total number of samples in the downstream dataset, represents the total number of target categories in the downstream dataset, represents the true label of the sample belonging to the category, represents the model's prediction probability that the sample belongs to the category.

[0014] By replacing the new classification head that adapts to the target categories of the downstream task, the problem of mismatch between the output dimension of the general model and the UAV task is solved, enabling the model to directly focus on target recognition in UAV scenarios, such as fault points in power inspection and crop types in agricultural plant protection; the specifically constructed loss function provides a clear direction for model optimization, accurately quantifying the difference between the prediction result and the true label, avoiding training bias caused by mismatched objective functions, and combining the UAV exclusive downstream dataset to enable the model to preferentially learn image features strongly related to actual work, such as target morphology under the UAV overhead perspective and target texture in complex environments, laying the foundation for subsequent efficient fine-tuning.

[0015] Step 104, initialize a binary update mask with the same dimension as the pre-trained visual base model parameter set, and set the parameter update period. When the training iteration step number meets the periodic scheduling condition, trigger the subsequent parameter selection and update process.

[0016] Before training begins, all parameters of the model are in a frozen state by default, and a binary update mask with the same dimension as the parameter set is defined , and the initial state is a zero matrix. In order to allow the model to periodically adjust its focus to adapt to the features of different batches of image samples under limited computing resources, this method uses a periodic scheduling mechanism.

[0017] First, set a global sparsity rate , which defines the proportion of parameters participating in the update to the total number of parameters. The total number of updatable parameters is:

[0018] At the same time, set an update period . Only when the training iteration step number meets the condition , the system will trigger a complete recalculation of the parameter selection and update mask . At other time steps, the model will continue to use the mask determined in the last period for sparse training.

[0019] By constructing a sparse training scheduling mechanism that adapts to the hardware and task characteristics of unmanned aerial vehicles, the all-zero initialized binary update mask provides a carrier for subsequent parameter sparse update, avoiding the huge computing and storage overheads brought by full-parameter fine-tuning from the source. Perfectly adapt to the limited hardware resources such as embedded computing modules, lightweight edge devices carried by unmanned aerial vehicles; the preset period and the periodic triggering condition-training iteration step number meets t mod T=0, balances the training stability and scene adaptability, the mask remains unchanged within the period, the model can be continuously optimized based on the stable parameter subset, reduces the convergence shock caused by frequent parameter switching, and the periodic parameter selection can respond to the dynamic changes of unmanned aerial vehicle operation scenes in time, solving the dual problems of traditional static fine-tuning unable to adapt to the dynamic nature of unmanned aerial vehicle scenes and full-tuning hardware carrying insufficient.

[0020] Step 106, calculate the gradient of the parameters of the pre-trained visual base model at the last iteration based on the loss function, calculate the importance score of each layer according to the gradient characteristics of each network layer, and then determine the parameter update budget allocation ratio of each layer according to the importance score of each layer. Finally, allocate the total update budget to each network layer according to the ratio to obtain the parameter update quota of each layer.

[0021] ​Hierarchical dynamic parameter budget allocation: in the iteration step number , first calculate the gradient of the loss function on the model parameters at the last time. Then, for each network layer, calculate the spectral norm of all parameter gradients inside it as the importance score of the layer for the current task. Based on this score, calculate the proportion of the parameter update budget that each layer should obtain in the total budget, and allocate the specific parameter budget according to the proportion. Through hierarchical dynamic parameter budget allocation, the precise tilt of computing resources is realized. Based on the last iteration parameter gradient calculated by the loss function, the contribution of each network layer to the current UAV task can be truly reflected. Taking the gradient L2 norm as the layer importance score can objectively distinguish key layers from non-key layers; then allocate the total update budget according to the score ratio and determine the update quota of each layer, which avoids the problem of insufficient resources for key layers and waste of resources for non-key layers caused by average allocation of resources. The limited update resources are preferentially used for network layers that are more critical to the performance improvement of the UAV task, maximizing resource utilization efficiency while adapting to the differentiated needs of different UAV tasks for network layers.

[0022] Step 108, for each layer parameter update quota, calculate the gradient feature of each parameter in each network layer as the parameter importance score, convert the parameter importance score into a parameter selection probability distribution, and sample the corresponding number of parameters in each layer without repetition according to the probability distribution. All sampled parameters constitute the preliminary candidate update set of the current period.

[0023] Intra-layer dynamic parameter selection: after obtaining the budget in each layer, fine selection of parameters in the layer is performed. For any parameter in the layer , use its gradient norm as its importance score, which reflects the marginal benefit of fine-tuning the parameter to reduce classification errors.

[0024] Then, through the Softmax function with temperature coefficient , these importance scores are converted into a probability distribution, which allows higher probability to be given to parameters with higher gradients when selecting, while retaining a certain degree of exploration. According to this probability distribution, sample parameters from each layer without repetition. All selected parameters constitute the preliminary candidate update set of the current period.

[0025] Using the absolute gradient of a single parameter within a layer as an importance score can capture fine-grained key parameters that traditional fixed-module fine-tuning cannot cover, adapting to the fine-grained target recognition needs in UAV scenarios. By using a Softmax function with a temperature coefficient to transform the score into a selection probability distribution, it ensures that key parameters with high gradients have a high selection probability, while also preserving the exploration space for potentially useful parameters, avoiding the model getting trapped in local optima, and improving feature capture capabilities, especially in UAV scenarios with small sample sizes. Non-re-sampling within layers ensures the diversity of the initial candidate update set, avoiding the uniformity of parameter updates, helping the model learn UAV task features from multiple perspectives, and enhancing its adaptability to complex scenarios.

[0026] Step 110: Obtain the final update set of the previous update cycle, merge the preliminary candidate update set with the final update set of the previous update cycle to construct an extended candidate pool; calculate the comprehensive importance score of each parameter in the extended candidate pool after fusing the historical momentum information with the current gradient features, and select the parameters with the highest comprehensive importance score and whose number is equal to the total update budget from the extended candidate pool to form the final update set of the current cycle.

[0027] Frequent switching of training parameters can lead to training instability. To ensure that the model does not forget the features it has already mastered that distinguish the target when learning rare image models or camouflage paint schemes that appear in new batches, a momentum mechanism is introduced to perform secondary filtering of parameters.

[0028] make This is the parameter set from the previous update cycle. First, an expanded candidate set is constructed. Integrate the selected parameters at the current moment Parameter selection based on historical information Subsequently, for each parameter in the set... Calculate the importance score of its momentum enhancement. :in, It is its score in the previous period. This is the momentum decay factor. Finally, the sample with the highest momentum fraction is selected from the extended pool. These parameters form the final update set.

[0029] The engine strategically merges the initial candidate parameter subset selected by the two-layer dynamic parameter selection module in the current cycle with the validated and actually updated parameter subset from the previous update cycle. Based on this, the engine initiates a momentum-based secondary selection decision mechanism, which not only considers the instantaneous gradient of the parameters but also deeply integrates their historical momentum information. This allows for a comprehensive evaluation of the immediate responsiveness and long-term value of each parameter in continuous UAV operation scenario information analysis. By establishing continuity and stability in parameter updates across different method adjustment cycles, the engine effectively suppresses potential severe oscillations or catastrophic forgetting that may occur when the model faces small-sample, highly adversarial UAV image processing data.

[0030] The cross-period parameter distribution transfer mechanism proposed in this application effectively maintains the exploratory nature and stability of the parameter update process by fusing selected parameters from historical periods with new parameters in the current period and utilizing a momentum-based secondary screening strategy. In scenarios with few-shot learning, image classification accuracy is significantly improved, and the model's robustness in complex environments is significantly enhanced.

[0031] Simultaneously, through a two-layer dynamic parameter selection mechanism, the computational budget can be dynamically allocated based on task relevance at the macro level, and the subset of key parameters that contributes most to the current image processing task can be accurately identified and activated at the micro level. This fundamentally solves the problems of static selection strategies and inability to adapt to dynamic changes in tasks in traditional efficient parameter fine-tuning methods, as well as the computational and storage overhead that full fine-tuning cannot withstand on edge computing platforms.

[0032] Step 112: Update the initialized binary update mask according to the final update set, sparsely update the parameters of the pre-trained visual base model according to the binary update mask and gradient information, apply the updated model to the downstream UAV image processing task data, and output the target category probability distribution.

[0033] based on Update binary mask Subsequently, the model parameters are updated according to sparsity rules and driven by the cross-entropy loss function.

[0034] Based on the binary mask updated from the final update set, the positions of the parameters that need to be updated in the current iteration are clearly defined. Combined with the sparse update rule, only the key parameters corresponding to "1" in the mask are updated, further reducing computational overhead and ensuring that the model can run in real time on UAV hardware. The updated model is applied to downstream UAV image processing tasks, outputting the probability distribution of target categories, which can directly provide quantitative basis for UAV operation decisions. Compared with traditional model output, the decision basis is clearer, which can effectively reduce the risk of misjudgment and omission, and meet the stringent requirements of UAVs for high reliability and real-time performance in fields such as power line inspection and emergency rescue. At the same time, compared with full fine-tuning, the computational and storage overhead is reduced and the model adaptation time is shortened, which fully meets the timeliness requirements of modern UAV operations.

[0035] The aforementioned method for efficient fine-tuning of dynamic parameters for visual models first involves model reading and data processing, loading a pre-trained large-scale visual model and preparing task data for a specific UAV image processing target. Then, the parameter reselection and update process is precisely triggered according to a preset cycle. Based on this, an update budget is dynamically allocated to each network layer based on the total gradient norm, and probabilistic selection is performed within each layer based on the gradient magnitude of individual parameters to form a preliminary candidate parameter set. This candidate set is merged with the updated parameter set from the previous cycle, and through a momentum-based secondary screening mechanism, the optimal parameter subset for the final iteration, balancing the exploratory nature of new UAV operating scenario information with the stability of historical knowledge, is determined, achieving precise knowledge adaptation of the pre-trained visual model to downstream UAV image processing tasks.

[0036] In one embodiment, the loss function required to construct the model optimization is: ; in, Represents the cross-entropy loss function. This represents the parameter set of the pre-trained visual base model. N This represents the total number of samples in the downstream dataset. This represents the total number of target categories in the downstream dataset. Indicates the first i The sample belongs to the first c The true label of the category, The model represents the first i The sample belongs to the first c Predicted probability of the category.

[0037] In one embodiment, the periodic scheduling condition is that the remainder of the training iteration number modulo the update period is zero.

[0038] In one embodiment, calculating the importance score for each layer includes: In iteration steps First, calculate the loss function. For the model parameters at the previous time step gradient For each network layer , , This represents the total number of network layers in the pre-trained visual base model; Calculate the spectral norm of the gradients of all parameters within the network layer, and use this as a score to measure the importance of the network layer to the current task: ; in, Represents the loss function In the model of the previous time step Layer parameters The gradient.

[0039] In one embodiment, the parameter update budget allocation ratio for each layer is determined based on the importance score of each layer. Finally, the total update budget is allocated to each network layer according to this ratio to obtain the parameter update quota for each layer, including: Calculate the proportion of the total budget allocated to updating the parameters that each layer should receive based on the importance scores of each layer. And allocate specific parameter budgets accordingly. : ; ; in, Represents network layer Rate the importance of the current task. Represents network layer Rate the importance of the current task. This indicates the total number of updated parameters.

[0040] In one embodiment, the parameter importance score is transformed into a parameter selection probability distribution, including: The parameter importance score is transformed into a parameter selection probability distribution as follows: ; in, For temperature coefficient, Presentation layer Any parameter within The gradient norm, j Indicates that within the same network layer, except for the first i The sequence numbers of other parameters besides the one parameter.

[0041] In one embodiment, the preliminary candidate update set is merged with the final update set from the previous update cycle to construct an expanded candidate pool, including: The initial candidate update set is merged with the final update set from the previous update cycle to construct an expanded candidate pool: ; in, This is the final update set from the previous update cycle. This is a preliminary candidate update set.

[0042] In one embodiment, the comprehensive importance score of each parameter in the expanded candidate pool, after fusing historical momentum information with the current gradient features, is calculated, including: The comprehensive importance score calculated by fusing the historical momentum information and current gradient features of each parameter in the expanded candidate pool is as follows: ; in, It is its score in the previous period. It is the momentum decay factor. Presentation layer Any parameter within The gradient norm.

[0043] In one embodiment, parameters with the highest overall importance score and whose number equals the total update budget are selected from the expanded candidate pool to form the final update set for the current period, including: The parameters with the highest overall importance score, whose number equals the total update budget, are selected from the expanded candidate pool to form the final update set for the current period: ; in, To determine the overall importance score, To expand the candidate pool, Indicates the total number of updated parameters. Represents network parameters, To select the highest score Parameter operation.

[0044] In one embodiment, the parameters of the pre-trained visual base model are sparsely updated based on the binary update mask and gradient information, including: Based on the binary update mask and gradient information, the parameters of the pre-trained visual base model are sparsely updated to obtain the updated model parameters: ; in, It's the learning rate. Represents element-wise multiplication. Update the initialized binary mask for the final update set. Represents the model parameters at the previous time step. The gradient.

[0045] In specific embodiments, experimental results show that the average recognition rate of targets in sea surface images reached 94.7% under complex environmental conditions such as detection distances of over 30km, high cloud cover (>60%), and targets using camouflage. Meanwhile, compared to full-scale fine-tuning, its computational and storage overhead was reduced by over 99%, and the rapid model adaptation time was shortened to less than 1 hour, fully meeting the timeliness requirements of modern UAV operations.

[0046] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0047] In one embodiment, such as Figure 2 As shown, a highly efficient fine-tuning device for dynamic parameters of visual models is designed, the device comprising: Downstream task adaptation module 202 is used to acquire downstream datasets for specific UAV image processing tasks, replace the original classification head of the pre-trained visual base model with a new classification head that adapts to the target category of the downstream dataset, and construct the loss function required for model optimization. The update strategy initialization module 204 is used to initialize a binary update mask with the same dimension as the parameter set of the pre-trained visual base model and set the parameter update period. When the number of training iterations meets the periodic scheduling condition, the subsequent parameter selection and update process is triggered. The hierarchical update budget allocation module 206 is used to calculate the gradient of the parameters of the pre-trained visual base model at the previous iteration time based on the loss function, calculate the importance score of each layer based on the gradient characteristics of each network layer, determine the parameter update budget allocation ratio of each layer according to the importance score of each layer, and finally allocate the total update budget to each network layer according to the ratio to obtain the parameter update quota of each layer. The in-layer parameter initial selection module 208 is used to update the parameters of each layer. It calculates the gradient feature of each parameter in each network layer as the parameter importance score, converts the parameter importance score into a parameter selection probability distribution, and samples the parameters corresponding to the number of update slots in each layer without repetition based on the probability distribution. All sampled parameters constitute the preliminary candidate update set for the current period. The cross-cycle parameter selection module 210 is used to obtain the final update set of the previous update cycle, merge the preliminary candidate update set with the final update set of the previous update cycle to construct an extended candidate pool; calculate the comprehensive importance score of each parameter in the extended candidate pool after fusing the historical momentum information with the current gradient features, and select the parameters with the highest comprehensive importance score and whose number is equal to the total update budget from the extended candidate pool to form the final update set of the current cycle. The sparse update and task inference module 212 is used to update the initialized binary update mask according to the final update set, sparsely update the parameters of the pre-trained visual base model according to the binary update mask and gradient information, apply the updated model to the downstream UAV image processing task data, and output the target category probability distribution.

[0048] For specific limitations regarding the efficient fine-tuning device for dynamic parameters of vision models, please refer to the limitations of the efficient fine-tuning method for dynamic parameters of vision models described above, which will not be repeated here. Each module in the aforementioned efficient fine-tuning device for dynamic parameters of vision models can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0049] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0050] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for efficient fine-tuning of dynamic parameters in visual models, characterized in that, The method includes: Obtain a downstream dataset for a specific UAV image processing task, replace the original classification head of the pre-trained visual base model with a new classification head adapted to the target category of the downstream dataset, and construct the loss function required for model optimization. Initialize a binary update mask with the same dimension as the parameter set of the pre-trained visual base model, and set the parameter update period. When the number of training iterations meets the periodic scheduling condition, trigger the subsequent parameter selection and update process. The gradient of the parameters of the pre-trained visual base model at the previous iteration time is calculated based on the loss function. The importance score of each layer is calculated based on the gradient characteristics of each network layer. Then, the parameter update budget allocation ratio of each layer is determined according to the importance score of each layer. Finally, the total update budget is allocated to each network layer according to the ratio to obtain the parameter update quota of each layer. For each layer of parameter update slots, the gradient feature of each parameter is calculated as a parameter importance score in each network layer. The parameter importance score is converted into a parameter selection probability distribution. Based on the probability distribution, parameters corresponding to the number of update slots are sampled in each layer without repetition. All sampled parameters constitute the preliminary candidate update set for the current period. Obtain the final update set of the previous update cycle, merge the preliminary candidate update set with the final update set of the previous update cycle to construct an extended candidate pool; calculate the comprehensive importance score of each parameter in the extended candidate pool after fusing the historical momentum information with the current gradient features, and select the parameters with the highest comprehensive importance score and whose number is equal to the total update budget from the extended candidate pool to form the final update set of the current cycle. The initial binary update mask is updated based on the final update set. The parameters of the pre-trained visual base model are sparsely updated based on the binary update mask and gradient information. The updated model is then applied to downstream UAV image processing task data to output the target category probability distribution.

2. The method according to claim 1, characterized in that, The periodic scheduling condition is that the remainder of the training iteration number divided by the update period is zero; the loss function required for model optimization is: ； in, Represents the cross-entropy loss function. This represents the parameter set of the pre-trained visual base model. N This represents the total number of samples in the downstream dataset. This represents the total number of target categories in the downstream dataset. Indicates the first i The sample belongs to the first c The true label of the category, The model represents the first i The sample belongs to the first c The predicted probability of the category.

3. The method according to claim 1, characterized in that, The calculation of importance scores for each layer includes: In iteration steps First, calculate the loss function. For the model parameters at the previous time step gradient For each network layer , , This represents the total number of network layers in the pre-trained visual base model; Calculate the spectral norm of the gradients of all parameters within the network layer, and use this as a score to measure the importance of the network layer to the current task: ； in, Represents the loss function In the model of the previous time step Layer parameters The gradient.

4. The method according to claim 1, characterized in that, The parameter update budget allocation ratio for each layer is determined based on its importance score. Finally, the total update budget is allocated to each network layer according to this ratio, resulting in parameter update slots for each layer, including: Calculate the proportion of the total budget allocated to updating the parameters that each layer should receive based on the importance scores of each layer. And allocate specific parameter budgets accordingly. : ； ； in, Represents network layer Rate the importance of the current task. Represents network layer Rate the importance of the current task. This indicates the total number of updated parameters.

5. The method according to claim 1, characterized in that, The parameter importance score is transformed into a parameter selection probability distribution, including: The parameter importance score is converted into a parameter selection probability distribution as follows: ； in, For temperature coefficient, Presentation layer Any parameter within The gradient norm, j Indicates that within the same network layer, except for the first i The sequence numbers of other parameters besides the one parameter.

6. The method according to claim 1, characterized in that, The preliminary candidate update set is merged with the final update set of the previous update cycle to construct an expanded candidate pool, including: The initial candidate update set is merged with the final update set of the previous update cycle to construct an expanded candidate pool: ； in, This is the final update set from the previous update cycle. This is a preliminary candidate update set.

7. The method according to claim 1, characterized in that, Calculate the comprehensive importance score of each parameter in the expanded candidate pool after fusing historical momentum information with the current gradient features, including: The comprehensive importance score calculated by fusing the historical momentum information and current gradient features of each parameter in the expanded candidate pool is as follows: ； in, It is its score in the previous period. It is the momentum decay factor. Presentation layer Any parameter within The gradient norm.

8. The method according to claim 1, characterized in that, The parameters with the highest overall importance score, whose number equals the total update budget, are selected from the expanded candidate pool to form the final update set for the current period, including: The parameters with the highest overall importance score, whose number equals the total update budget, are selected from the expanded candidate pool to form the final update set for the current period: ； in, To determine the overall importance score, To expand the candidate pool, Indicates the total number of updated parameters. Represents network parameters, To select the highest score Parameter operation.

9. The method according to claim 1, characterized in that, The parameters of the pre-trained visual base model are sparsely updated based on the binary update mask and gradient information, including: Based on the binary update mask and gradient information, the parameters of the pre-trained visual base model are sparsely updated to obtain the updated model parameters: ； in, It's the learning rate. Represents element-wise multiplication. Update the initialized binary mask for the final update set. Represents the model parameters at the previous time step. The gradient.

10. A highly efficient fine-tuning device for dynamic parameters of visual models, characterized in that, The device includes: The downstream task adaptation module is used to acquire downstream datasets for specific UAV image processing tasks, replace the original classification head of the pre-trained visual base model with a new classification head that adapts to the target category of the downstream dataset, and construct the loss function required for model optimization. The update strategy initialization module is used to initialize a binary update mask with the same dimension as the parameter set of the pre-trained visual base model and set the parameter update period. When the number of training iterations meets the periodic scheduling condition, the subsequent parameter selection and update process is triggered. The hierarchical update budget allocation module is used to calculate the gradient of the parameters of the pre-trained visual base model at the previous iteration time based on the loss function, calculate the importance score of each layer according to the gradient characteristics of each network layer, determine the parameter update budget allocation ratio of each layer according to the importance score of each layer, and finally allocate the total update budget to each network layer according to the ratio to obtain the parameter update quota of each layer. The in-layer parameter initial selection module is used to calculate the gradient feature of each parameter in each network layer as a parameter importance score for the parameter update quota of each layer. The parameter importance score is converted into a parameter selection probability distribution. Based on the probability distribution, the parameters corresponding to the number of update quotas are sampled in each layer without repetition. All sampled parameters constitute the preliminary candidate update set for the current period. The cross-cycle parameter selection module is used to obtain the final update set of the previous update cycle, merge the preliminary candidate update set with the final update set of the previous update cycle to construct an extended candidate pool; calculate the comprehensive importance score of each parameter in the extended candidate pool after fusing the historical momentum information with the current gradient features, and select the parameters with the highest comprehensive importance score and whose number is equal to the total update budget from the extended candidate pool to form the final update set of the current cycle. The sparse update and task inference module is used to update the initialized binary update mask according to the final update set, sparsely update the parameters of the pre-trained visual base model according to the binary update mask and gradient information, apply the updated model to the downstream UAV image processing task data, and output the target category probability distribution.

Images