Dynamic parameter efficient fine-tuning method and device for vision model

Abstract

The application relates to a dynamic parameter efficient fine-tuning method and device for a visual model. The method comprises the following steps: constructing a double-layer dynamic parameter selection mechanism, dynamically allocating a parameter update budget for the model according to the gradient norm of each network layer at a macro level, and accurately identifying and activating a parameter subset most critical to the current task according to the selection probability converted from the parameter gradient at a micro level; meanwhile, a cross-cycle parameter distribution transmission engine is designed, a momentum Top-K secondary screening strategy is used to fuse historical and current key parameters, the exploration of the model to new information is maintained, the catastrophic forgetting is effectively inhibited, and the stable migration of knowledge is ensured; finally, the model is efficiently updated through the generated sparse mask, so that the model can quickly master specific image target classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a dynamic parameter efficient fine-tuning method and device for a visual model. BACKGROUND

[0002] With the wide application of unmanned aerial vehicle (UAV) technology in fields such as civilian aerial photography, power inspection, agricultural plant protection, and emergency rescue, UAV image processing has become the core support for improving operation efficiency and decision-making accuracy. Visual-based intelligent perception technology plays an irreplaceable role in UAV target detection, image semantic segmentation, and scene risk assessment. The "pre-training-fine-tuning" paradigm provides an effective way to quickly adapt large visual base models trained based on massive general image data to specific UAV image processing tasks, and has become an important technical direction in the field of UAV intelligent application.

[0003] However, the particularity of the UAV operation environment poses severe challenges to model deployment. First, with the number of visual base model parameters reaching tens of billions or even higher, full fine-tuning on embedded computing modules, lightweight edge devices, and other hardware carried by UAVs brings unbearable computational resource consumption and storage pressure, severely restricting the real-time response capability during UAV operation. Second, the UAV operation scene is complex and variable, ranging from dense urban building groups to remote mountain forests, from high-altitude wide-range aerial photography to low-altitude close-range inspection, and the diversified scene requires the model to have rapid adaptation capability. Traditional full fine-tuning process is cumbersome and time-consuming, and it is difficult to meet the timeliness requirements of flexible UAV operation.

[0004] As a countermeasure, parameter efficient fine-tuning technology can exhibit performance comparable to or even better than full fine-tuning by optimizing only a small number of key parameters, significantly reducing hardware resource consumption, and providing a feasible path for real-time and lightweight deployment of UAV image processing. Among the many parameter efficient fine-tuning methods, selective fine-tuning is an important technical route. This method does not introduce any additional parameters, and its core idea is to identify and update only the parameter subset most relevant to a specific UAV image processing task, such as line feature extraction parameters for power inspection tasks or crop shape recognition parameters for agricultural plant protection tasks, thereby achieving efficient and accurate knowledge adaptation and meeting the strict requirements of low latency and high reliability for UAV operation.

[0005] Existing selective fine-tuning techniques mainly follow two processing paradigms: one is the fixed-module optimization method, which limits the optimization scope to specific predefined components in the model, such as bias terms or specific network layers. However, its selection granularity is coarse, making it difficult to capture fine-grained image features relevant to tasks such as UAV target detection and scene semantic segmentation. The other is the static subset selection method, which relies on importance metrics such as gradients and Fisher information to select a fixed sparse subset of parameters for optimization based on a small number of UAV samples before training. Although these methods have achieved good performance and high computational efficiency in conventional vision tasks, they have exposed significant limitations in UAV image processing applications. First, the UAV operating environment is highly dynamic and complex, ranging from direct sunlight at high altitudes to shadows at low altitudes, and from clear imaging in sunny weather to image interference in rainy, foggy, and dusty weather. The visual data acquired by UAVs has extremely high heterogeneity and feature diversity. Existing static optimization strategies follow a fixed parameter selection mechanism, ignoring the dynamic evolution of the model's cognitive state during UAV image learning. This makes it impossible to adaptively adjust the optimization strategy according to different operational environments and image characteristics, becoming a key bottleneck restricting the performance of UAV visual models. Secondly, in actual UAV operations, high-quality labeled data is often extremely scarce, and problems such as difficulty in acquiring data for special scenarios and uneven distribution of target samples are common. In complex operational scenarios with scarce and unevenly distributed data, the performance degradation of static fine-tuning methods is particularly pronounced, potentially leading to missed detections or misjudgments of key targets. This makes it difficult to meet the stringent requirements for high reliability and robustness in UAV applications such as power line inspection, agricultural production, and emergency rescue. Summary of the Invention

[0006] Therefore, it is necessary to provide a method and apparatus for efficient fine-tuning of dynamic parameters of vision models that can improve the accuracy of image processing, in order to address the above-mentioned technical problems.

[0007] An efficient method for fine-tuning dynamic parameters of visual models, the method comprising:

[0008] Obtain downstream datasets for specific UAV image processing tasks, replace the original classification head of the pre-trained visual base model with a new classification head adapted to the target category of the downstream dataset, and construct the loss function required for model optimization.

[0009] Initialize a binary update mask with the same dimension as the parameter set of the pre-trained visual base model, and set the parameter update period. When the number of training iterations meets the periodic scheduling condition, trigger the subsequent parameter selection and update process.

[0010] The gradient of the parameter at the last iteration of the pre-trained visual base model is calculated based on a loss function, the importance score of each layer is calculated according to the gradient features of each network layer, the parameter update budget allocation ratio of each layer is determined according to the importance score of each layer, and finally the total update budget is allocated to each network layer according to the ratio to obtain the parameter update quota of each layer;

[0011] For the parameter update quota of each layer, the gradient feature of each parameter in each network layer is calculated as the parameter importance score, the parameter importance score is converted into a parameter selection probability distribution, and the corresponding number of parameters is sampled without repetition in each layer according to the probability distribution, and all the sampled parameters constitute the preliminary candidate update set of the current period;

[0012] The final update set of the last update period is obtained, the preliminary candidate update set is combined with the final update set of the last update period, and an extended candidate pool is constructed; the comprehensive importance score of each parameter in the extended candidate pool is calculated by fusing the historical momentum information and the current gradient feature, and the parameters with the highest comprehensive importance score and equal to the total update budget are selected from the extended candidate pool to constitute the final update set of the current period;

[0013] The initialized binary update mask is updated according to the final update set, and the parameters of the pre-trained visual base model are sparsely updated according to the binary update mask and the gradient information, and the updated model is applied to the downstream unmanned aerial vehicle image processing task data to output the target class probability distribution.

[0014] A dynamic parameter efficient fine-tuning device for a visual model, the device comprising:

[0015] A downstream task adaptation module is configured to obtain a downstream data set for a specific unmanned aerial vehicle image processing task, replace the original classification head of the pre-trained visual base model with a new classification head adapted to the target classes of the downstream data set, and construct a loss function required for model optimization;

[0016] An update strategy initialization module is configured to initialize a binary update mask with the same dimension as the parameter set of the pre-trained visual base model, and set a parameter update period, and trigger the subsequent parameter selection and update process when the number of training iteration steps meets the periodic scheduling condition;

[0017] A hierarchical update budget allocation module is configured to calculate the gradient of the parameter at the last iteration of the pre-trained visual base model based on a loss function, calculate the importance score of each layer according to the gradient features of each network layer, determine the parameter update budget allocation ratio of each layer according to the importance score of each layer, and finally allocate the total update budget to each network layer according to the ratio to obtain the parameter update quota of each layer;

[0018] The intra-layer parameter preliminary selection module is configured to calculate gradient features of each parameter in each network layer as parameter importance scores, convert the parameter importance scores into parameter selection probability distributions, and sample parameters corresponding to the number of update quotas in each layer without repetition according to the probability distributions, so that all the sampled parameters form a preliminary candidate update set of the current period;

[0019] The cross-period parameter selection module is configured to obtain a final update set of a previous update period, combine the preliminary candidate update set with the final update set of the previous update period to construct an extended candidate pool, calculate comprehensive importance scores of each parameter in the extended candidate pool after fusing historical momentum information and current gradient features, and select parameters with the highest comprehensive importance scores from the extended candidate pool, so that the selected parameters form a final update set of the current period.

[0020] The sparse update and task reasoning module is configured to update an initialized binary update mask according to the final update set, perform sparse update on parameters of the pre-trained visual base model according to the binary update mask and gradient information, and apply the updated model to downstream unmanned aerial vehicle image processing task data to output a target class probability distribution. The above-mentioned dynamic parameter efficient fine-tuning method and device for a visual model are configured to first perform model reading and data processing, load a pre-trained visual large model, and prepare task data for a specific unmanned aerial vehicle image processing target; then accurately trigger the parameter reselection and update process according to a preset period; on this basis, the total gradient norm of each network layer is used to dynamically allocate an update budget to the network layer, and then the gradient size of each parameter is used to probabilistically select parameters in the network layer to form a preliminary candidate parameter set; the preliminary candidate parameter set is combined with updated parameters of a previous period, and a momentum-based secondary selection mechanism is used to determine an optimal parameter subset that takes into account the exploratory nature of new unmanned aerial vehicle operating scene information and the stability of historical knowledge, so as to realize precise knowledge adaptation of the pre-trained visual model to downstream unmanned aerial vehicle image processing tasks. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 FIG. 1 is a flowchart of a dynamic parameter efficient fine-tuning method for a visual model according to an embodiment of the present application;

[0022] Figure 2 FIG. 2 is a framework diagram of a dynamic parameter efficient fine-tuning device for a visual model according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0024] In one embodiment, as shown in Figure 1 A visual model-oriented dynamic parameter efficient fine-tuning method is provided, comprising the following steps:

[0025] Step 102, obtaining a downstream data set for a specific unmanned aerial vehicle image processing task, and replacing the original classification head of the pre-trained visual basic model with a new classification head adapted to the target categories of the downstream data set, while constructing the loss function required for model optimization.

[0026] This step is the initialization preparation for the fine-tuning task. Taking unmanned aerial vehicle image processing as an example, first, a visual basic model pre-trained on a large amount of general image data is loaded, and its complete parameter set is represented as , containing network layers, represented as:

[0027] ;

[0028] Subsequently, the downstream data set for image classification can be represented as:

[0029] ;

[0030] Wherein, is a satellite image slice containing a single image target, and is its corresponding one-hot encoded category label.

[0031] In order to enable the model to process the classification task, its original output layer is replaced with a new classification head for image categories. For input image , the prediction probability distribution of the model output is:

[0032] ;

[0033] The goal of the entire fine-tuning process is to minimize the cross-entropy loss function between the model prediction and the true label:

[0034] ;

[0035] Wherein, represents the cross-entropy loss function, represents the parameter set of the pre-trained visual basic model, represents the total number of samples in the downstream data set, represents the total number of target categories in the downstream data set, represents the true label of the sample belongs to the category, represents the prediction probability of the model for the The sample belongs to the first category.

[0036] By replacing the new classification head that adapts to the target class of the downstream task, the problem of mismatch between the output dimension of the general model and the UAV task is solved, so that the model can directly focus on target recognition in the UAV environment, such as fault points in power inspection and crop types in agricultural plant protection; The loss function constructed for the model optimization provides a clear direction, accurately quantifies the difference between the prediction result and the real label, avoids the training deviation caused by the mismatch of the objective function, and combines the UAV exclusive downstream data set to make the model learn the image features related to the actual work first, such as the target shape under the UAV overhead view and the target texture in the complex environment, and lay the foundation for subsequent efficient fine-tuning.

[0037] Step 104, initialize a binary update mask with the same dimension as the parameter set of the pre-trained visual base model, and set the parameter update period. When the training iteration step number meets the periodic scheduling condition, trigger the subsequent parameter selection and update process.

[0038] Before the training starts, all parameters of the model are in a frozen state, and a binary update mask with the same dimension as the parameter set of the pre-trained visual base model is defined, and the initial state is a zero matrix. In order to let the model periodically adjust its focus to adapt to the features of different batches of image samples under limited computing resources, this method adopts a periodic scheduling mechanism.

[0039] First, set a global sparsity rate , which defines the proportion of parameters participating in the update to the total parameter quantity. The total number of updateable parameters is:

[0040]

[0041] At the same time, set an update period . Only when the training iteration step number meets the condition , the system will trigger a complete parameter selection and update mask recalculation. At other time steps, the model will follow the mask determined in the last period for sparse training.

[0042] By constructing a sparse training scheduling mechanism that adapts to the hardware and task characteristics of the unmanned aerial vehicle, the binary update mask of full zero initialization provides a carrier for subsequent parameter sparse update, avoids the huge calculation and storage overhead caused by full parameter fine-tuning from the source, and perfectly adapts to the limited hardware resources such as embedded computing modules and lightweight edge devices carried by unmanned aerial vehicles; The preset period and the periodic triggering condition-training iteration step number satisfies t mod T=0, which balances the training stability and scene adaptability, the mask remains unchanged within the period, the model can be continuously optimized based on the stable parameter subset, and the convergence shock caused by frequent parameter switching is reduced, while the periodic parameter selection can timely respond to the dynamic changes of the unmanned aerial vehicle operation scene, solving the dual problems of traditional static fine-tuning that cannot adapt to the dynamic nature of the unmanned aerial vehicle scene and the hardware carrying capacity of full fine-tuning.

[0043] Step 106, based on the loss function, the gradient of the parameter of the pre-trained visual basic model at the last iteration time is calculated, the importance score of each layer is calculated according to the gradient characteristics of each network layer, and then the parameter update budget allocation ratio of each layer is determined according to the importance score of each layer, and finally the total update budget is allocated to each network layer according to the ratio, and the parameter update quota of each layer is obtained.

[0044] Hierarchical dynamic parameter budget allocation: at the iteration step , first, the gradient of the loss function on the model parameter at the last time is calculated. Then, for each network layer, the spectral norm of all parameter gradients in the layer is calculated as the importance score of the layer to the current task. Based on this score, the proportion of the parameter update budget that each layer should obtain in the total budget is calculated, and the specific parameter budget is allocated according to this proportion. Through hierarchical dynamic parameter budget allocation, the calculation resources are precisely tilted. The gradient of the parameter at the last iteration calculated based on the loss function can truly reflect the contribution of each network layer to the current unmanned aerial vehicle task, and the gradient L2 norm and other characteristics can be used as the layer importance score to objectively distinguish between key layers and non-key layers; Then, the total update budget is allocated according to the score ratio and the update quota of each layer is determined, which avoids the problem of insufficient resources for key layers and waste of resources for non-key layers caused by equal allocation of resources, and prioritizes the use of limited update resources for network layers that are more critical to the performance improvement of unmanned aerial vehicle tasks, maximizing resource utilization efficiency while adapting to the differentiated needs of different unmanned aerial vehicle tasks for network layers.

[0045] Step 108, for each layer parameter update quota, the gradient characteristics of each parameter in each network layer are calculated as the parameter importance score, the parameter importance score is converted into a parameter selection probability distribution, and according to the probability distribution, the corresponding number of parameters is sampled without repetition in each layer, and all sampled parameters constitute the preliminary candidate update set of the current period.

[0046] Intra-layer dynamic parameter selection: after obtaining the budget in each layer, fine selection of parameters in the layer is performed. any one of the parameters within a layer , using its gradient norm as its importance score, which reflects the marginal benefit of fine-tuning this parameter in reducing classification errors.

[0047] Then, these importance scores are transformed into a probability distribution by a Softmax function with temperature , which allows higher-probability selection of parameters with higher gradients while preserving some exploration. According to this probability distribution, a number of parameters are sampled without repetition from each layer. All the selected parameters form the preliminary candidate update set for the current epoch.

[0048] Using the absolute value of the gradient of a single parameter within a layer as its importance score can capture fine-grained key parameters that cannot be covered by traditional fixed module fine-tuning, adapting to the fine-grained target recognition requirements in the UAV scene. By transforming the score into a selection probability distribution using a Softmax function with temperature, we can ensure that key parameters with high gradients have high selection probabilities, while preserving the exploration space for potentially useful parameters, avoiding the model from falling into a local optimum, especially in the small sample scene of UAVs, which can improve the feature capture ability. Sampling within the layer without repetition ensures the diversity of the preliminary candidate update set, avoiding parameter update singularization, helping the model to learn UAV task features from multiple angles, and enhancing the adaptability to complex scenes.

[0049] Step 110, obtain the final update set of the last update cycle, combine the preliminary candidate update set with the final update set of the last update cycle to construct an extended candidate pool; calculate the comprehensive importance score of each parameter in the extended candidate pool after fusing the historical momentum information and the current gradient feature, select the parameters with the highest comprehensive importance score from the extended candidate pool, and the number of parameters is equal to the total update budget, to form the final update set of the current cycle.

[0050] Frequent switching of training parameters can cause instability in training. To ensure that the model does not forget the mastered features that distinguish targets when learning rare image types or camouflage coatings in new batches, a momentum mechanism is introduced to perform secondary screening of parameters.

[0051] Let be the parameter set of the last update cycle. First, construct an extended candidate set , which fuses the selected parameters at the current time and the selected parameters based on historical information; then, calculate the momentum-enhanced importance score of each parameter in the set: where is its score in the last cycle, is a momentum decay factor. Finally, the parameter with the highest momentum score is selected from the extended pool to form the final update set.

[0052] The preliminary candidate parameter subset selected by the double-layer dynamic parameter selection module in the current period is strategically merged with the parameter subset verified and actually updated in the last update period. On this basis, the engine starts the momentum secondary screening decision mechanism, which not only considers the instantaneous gradient of the parameter, but also deeply integrates the momentum information of its historical update, so as to comprehensively evaluate the real-time response ability and long-term value of each parameter in the continuous unmanned aerial vehicle scene information analysis. Between different method adjustment periods, the continuity and stability of parameter update are established to effectively suppress the possible violent oscillation or catastrophic forgetting of the model when facing small sample and high adversarial unmanned aerial vehicle image processing data.

[0053] The cross-period parameter distribution transfer mechanism proposed in the application effectively maintains the exploratory and stability in the parameter update process by fusing the selected parameters in the historical period and the new parameters in the current period, and using the momentum secondary screening strategy. In the face of small sample learning scenarios, the image classification accuracy is significantly improved, and the robust adaptive ability of the model in complex environments is significantly enhanced.

[0054] At the same time, through the double-layer dynamic parameter selection mechanism, the calculation budget can be dynamically allocated according to the task correlation at the macro level, and the key parameter subset that contributes most to the current image processing task can be accurately identified and activated at the micro level. This fundamentally solves the problem of static selection strategy in traditional parameter efficient fine-tuning method, which cannot adapt to the dynamic changes of the task, and the problem of high computational and storage overhead of full fine-tuning on edge computing platform.

[0055] Step 112, update the initialized binary update mask according to the final update set, and perform sparse update on the parameters of the pre-trained visual base model according to the binary update mask and gradient information. Apply the updated model to the downstream unmanned aerial vehicle image processing task data, and output the target class probability distribution.

[0056] Based on update the binary mask After that, the update of the model parameters follows the sparse rule and is driven by the cross-entropy loss function.

[0057] ​Based on the binary mask updated from the final update set, the positions of the parameters that need to be updated in the current iteration are clearly defined. Combined with the sparse update rule, only the key parameters corresponding to "1" in the mask are updated, further reducing computational overhead and ensuring that the model can run in real time on UAV hardware. The updated model is applied to downstream UAV image processing tasks, outputting the probability distribution of target categories, which can directly provide quantitative basis for UAV operation decisions. Compared with traditional model output, the decision basis is clearer, which can effectively reduce the risk of misjudgment and omission, and meet the stringent requirements of UAVs for high reliability and real-time performance in fields such as power line inspection and emergency rescue. At the same time, compared with full fine-tuning, the computational and storage overhead is reduced and the model adaptation time is shortened, which fully meets the timeliness requirements of modern UAV operations.

[0058] The aforementioned method for efficient fine-tuning of dynamic parameters for visual models first involves model reading and data processing, loading a pre-trained large-scale visual model and preparing task data for a specific UAV image processing target. Then, the parameter reselection and update process is precisely triggered according to a preset cycle. Based on this, an update budget is dynamically allocated to each network layer based on the total gradient norm, and probabilistic selection is performed within each layer based on the gradient magnitude of individual parameters to form a preliminary candidate parameter set. This candidate set is merged with the updated parameter set from the previous cycle, and through a momentum-based secondary screening mechanism, the optimal parameter subset for the final iteration, balancing the exploratory nature of new UAV operating scenario information with the stability of historical knowledge, is determined, achieving precise knowledge adaptation of the pre-trained visual model to downstream UAV image processing tasks.

[0059] In one embodiment, the loss function required to construct the model optimization is:

[0060] ;

[0061] in, Represents the cross-entropy loss function. This represents the parameter set of the pre-trained visual base model. N This represents the total number of samples in the downstream dataset. This represents the total number of target categories in the downstream dataset. Indicates the first i The sample belongs to the first c The true label of the category, The model represents the first i The sample belongs to the first c Predicted probability of the category.

[0062] In one embodiment, the periodic scheduling condition is that the remainder of the training iteration number modulo the update period is zero.

[0063] In one embodiment, calculating the importance score for each layer includes:

[0064] In the iteration step number , first calculate the loss function The gradient of the model parameters at the last moment , for each network layer , , The total number of network layers of the pre-trained visual base model is represented;

[0065] The spectral norm of all parameter gradients in the network layer is calculated as the importance score of the network layer to the current task:

[0066] ;

[0067] Wherein, The gradient of the loss function to the last moment Layer parameters in the model.

[0068] In one embodiment, the parameter update budget allocation ratio of each layer is determined according to the importance score of each layer, and finally the total update budget is allocated to each network layer according to the ratio to obtain the parameter update quota of each layer, including:

[0069] According to the importance score of each layer, calculate the proportion of the parameter update budget that each layer should obtain in the total budget , and allocate the specific parameter budget according to the proportion:

[0070] ;

[0071] ;

[0072] Wherein, The importance score of the network layer to the current task, The importance score of the network layer to the current task, The total number of updated parameters is represented.

[0073] In one embodiment, the parameter importance score is converted into a parameter selection probability distribution, including:

[0074] The parameter importance score is converted into a parameter selection probability distribution as follows:

[0075] ;

[0076] Wherein, The temperature coefficient is represented, The layer the gradient norm of any one parameter in the layer j represents the sequence number of other parameters in the same network layer except the first i

[0077] In one of the embodiments, the preliminary candidate update set is merged with the final update set of the last update cycle to construct an extended candidate pool, including:

[0078] The preliminary candidate update set is merged with the final update set of the last update cycle to construct an extended candidate pool, including:

[0079] ;

[0080] wherein, is the final update set of the last update cycle, is the preliminary candidate update set.

[0081] In one of the embodiments, the comprehensive importance score of each parameter in the extended candidate pool after the fusion of the historical momentum information and the current gradient feature is calculated, including:

[0082] The comprehensive importance score of each parameter in the extended candidate pool after the fusion of the historical momentum information and the current gradient feature is calculated, including:

[0083] ;

[0084] wherein, is the score thereof in the last cycle, is a momentum decay factor, represents the gradient norm of any one parameter in the layer

[0085] In one of the embodiments, the parameters with the highest comprehensive importance score and the number equal to the total update budget are selected from the extended candidate pool to constitute the final update set of the current cycle, including:

[0086] The parameters with the highest comprehensive importance score and the number equal to the total update budget are selected from the extended candidate pool to constitute the final update set of the current cycle, including:

[0087] ;

[0088] wherein, is the comprehensive importance score, is the extended candidate pool, represents the total number of update parameters, represents the network parameter, is the operation of selecting the parameters with the highest score .​​​​

[0089] In one embodiment, the parameters of the pre-trained visual base model are sparsely updated based on the binary update mask and gradient information, including:

[0090] Based on the binary update mask and gradient information, the parameters of the pre-trained visual base model are sparsely updated to obtain the updated model parameters:

[0091] ;

[0092] in, It's the learning rate. Represents element-wise multiplication. Update the initialized binary mask for the final update set. Represents the model parameters at the previous time step. The gradient.

[0093] In specific embodiments, experimental results show that the average recognition rate of targets in sea surface images reached 94.7% under complex environmental conditions such as detection distances of over 30km, high cloud cover (>60%), and targets using camouflage. Meanwhile, compared to full-scale fine-tuning, its computational and storage overhead was reduced by over 99%, and the rapid model adaptation time was shortened to less than 1 hour, fully meeting the timeliness requirements of modern UAV operations.

[0094] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0095] In one embodiment, such as Figure 2 As shown, a highly efficient fine-tuning device for dynamic parameters of visual models is designed, the device comprising:

[0096] Downstream task adaptation module 202 is used to acquire downstream datasets for specific UAV image processing tasks, replace the original classification head of the pre-trained visual base model with a new classification head that adapts to the target category of the downstream dataset, and construct the loss function required for model optimization.

[0097] The update strategy initialization module 204 is configured to initialize a binary update mask with the same dimension as the parameter set of the pre-trained visual base model, and set a parameter update period. When the number of training iteration steps meets the periodic scheduling condition, the subsequent parameter selection and update process is triggered.

[0098] The hierarchical update budget allocation module 206 is configured to calculate the gradient of the parameters of the pre-trained visual base model at the previous iteration time based on the loss function, calculate the importance score of each layer based on the gradient characteristics of each network layer, determine the parameter update budget allocation proportion of each layer according to the importance score of each layer, and finally allocate the total update budget to each network layer according to the proportion to obtain the parameter update quota of each layer.

[0099] The intra-layer parameter preliminary selection module 208 is configured to calculate the gradient characteristics of each parameter in each network layer as a parameter importance score for the parameter update quota of each layer, convert the parameter importance score into a parameter selection probability distribution, and sample the corresponding number of parameters in each layer according to the probability distribution without repetition, so that all the sampled parameters constitute the preliminary candidate update set of the current period.

[0100] The cross-period parameter selection module 210 is configured to obtain the final update set of the previous update period, combine the preliminary candidate update set with the final update set of the previous update period to construct an extended candidate pool, calculate the comprehensive importance score of each parameter in the extended candidate pool after fusing the historical momentum information and the current gradient characteristics, and select the parameters with the highest comprehensive importance score and equal to the total update budget from the extended candidate pool to constitute the final update set of the current period.

[0101] The sparse update and task reasoning module 212 is configured to update the initialized binary update mask according to the final update set, perform sparse update on the parameters of the pre-trained visual base model according to the binary update mask and the gradient information, and apply the updated model to the downstream unmanned aerial vehicle image processing task data to output the target class probability distribution.

[0102] The specific limitations of the dynamic parameter efficient fine-tuning device for a visual model can be referred to the limitations of the dynamic parameter efficient fine-tuning method for a visual model described above, which will not be repeated here. Each module in the dynamic parameter efficient fine-tuning device for a visual model can be realized by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0103] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, as long as the combinations of technical features do not contradict each other, they shall be considered within the scope of the present disclosure.

[0104] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it shall not be understood as a limitation on the scope of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these shall be within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A method for efficient fine-tuning of dynamic parameters in visual models, characterized in that, The method includes: Obtain a downstream dataset for a specific UAV image processing task, replace the original classification head of the pre-trained visual base model with a new classification head adapted to the target category of the downstream dataset, and construct the loss function required for model optimization. Initialize a binary update mask with the same dimension as the parameter set of the pre-trained visual base model, and set the parameter update period. When the number of training iterations meets the periodic scheduling condition, trigger the subsequent parameter selection and update process. The gradient of the parameters of the pre-trained visual base model at the previous iteration time is calculated based on the loss function. The importance score of each layer is calculated based on the gradient characteristics of each network layer. Then, the parameter update budget allocation ratio of each layer is determined according to the importance score of each layer. Finally, the total update budget is allocated to each network layer according to the ratio to obtain the parameter update quota of each layer. For each layer of parameter update slots, the gradient feature of each parameter is calculated as a parameter importance score in each network layer. The parameter importance score is converted into a parameter selection probability distribution. Based on the probability distribution, parameters corresponding to the number of update slots are sampled in each layer without repetition. All sampled parameters constitute the preliminary candidate update set for the current period. Obtain the final update set of the previous update cycle, merge the preliminary candidate update set with the final update set of the previous update cycle to construct an extended candidate pool; calculate the comprehensive importance score of each parameter in the extended candidate pool after fusing the historical momentum information with the current gradient features, and select the parameters with the highest comprehensive importance score and whose number is equal to the total update budget from the extended candidate pool to form the final update set of the current cycle. The initial binary update mask is updated based on the final update set. The parameters of the pre-trained visual base model are sparsely updated based on the binary update mask and gradient information. The updated model is then applied to downstream UAV image processing task data to output the target category probability distribution.

2. The method according to claim 1, characterized in that, The periodic scheduling condition is that the remainder of the training iteration number divided by the update period is zero; the loss function required for model optimization is: ； in, Represents the cross-entropy loss function. This represents the parameter set of the pre-trained visual base model. N This represents the total number of samples in the downstream dataset. This represents the total number of target categories in the downstream dataset. Indicates the first i The sample belongs to the first c The true label of the category, The model represents the first i The sample belongs to the first c The predicted probability of the category.

3. The method according to claim 1, characterized in that, The calculation of importance scores for each layer includes: In iteration steps First, calculate the loss function. For the model parameters at the previous time step gradient For each network layer , , This represents the total number of network layers in the pre-trained visual base model; Calculate the spectral norm of the gradients of all parameters within the network layer, and use this as a score to measure the importance of the network layer to the current task: ； in, Represents the loss function In the model of the previous time step Layer parameters The gradient.

4. The method according to claim 1, characterized in that, The parameter update budget allocation ratio for each layer is determined based on its importance score. Finally, the total update budget is allocated to each network layer according to this ratio, resulting in parameter update slots for each layer, including: Calculate the proportion of the total budget allocated to updating the parameters that each layer should receive based on the importance scores of each layer. And allocate specific parameter budgets accordingly. : ； ； in, Represents network layer Rate the importance of the current task. Represents network layer Rate the importance of the current task. This indicates the total number of updated parameters.

5. The method according to claim 1, characterized in that, The parameter importance score is transformed into a parameter selection probability distribution, including: The parameter importance score is converted into a parameter selection probability distribution as follows: ； in, For temperature coefficient, Presentation layer Any parameter within The gradient norm, j Indicates that within the same network layer, except for the first i The sequence numbers of other parameters besides the one parameter.

6. The method according to claim 1, characterized in that, The preliminary candidate update set is merged with the final update set of the previous update cycle to construct an expanded candidate pool, including: The initial candidate update set is merged with the final update set of the previous update cycle to construct an expanded candidate pool: ； in, This is the final update set from the previous update cycle. This is a preliminary candidate update set.

7. The method according to claim 1, characterized in that, Calculate the comprehensive importance score of each parameter in the expanded candidate pool after fusing historical momentum information with the current gradient features, including: The comprehensive importance score calculated by fusing the historical momentum information and current gradient features of each parameter in the expanded candidate pool is as follows: ； in, It is its score in the previous period. It is the momentum decay factor. Presentation layer Any parameter within The gradient norm.

8. The method according to claim 1, characterized in that, The parameters with the highest overall importance score, whose number equals the total update budget, are selected from the expanded candidate pool to form the final update set for the current period, including: The parameters with the highest overall importance score, whose number equals the total update budget, are selected from the expanded candidate pool to form the final update set for the current period: ； in, To determine the overall importance score, To expand the candidate pool, Indicates the total number of updated parameters. Represents network parameters, To select the highest score Parameter operation.

9. The method according to claim 1, characterized in that, The parameters of the pre-trained visual base model are sparsely updated based on the binary update mask and gradient information, including: Based on the binary update mask and gradient information, the parameters of the pre-trained visual base model are sparsely updated to obtain the updated model parameters: ； in, It's the learning rate. Represents element-wise multiplication. Update the initialized binary mask for the final update set. Represents the model parameters at the previous time step. The gradient.

10. A highly efficient fine-tuning device for dynamic parameters of visual models, characterized in that, The device includes: The downstream task adaptation module is used to acquire downstream datasets for specific UAV image processing tasks, replace the original classification head of the pre-trained visual base model with a new classification head that adapts to the target category of the downstream dataset, and construct the loss function required for model optimization. The update strategy initialization module is used to initialize a binary update mask with the same dimension as the parameter set of the pre-trained visual base model and set the parameter update period. When the number of training iterations meets the periodic scheduling condition, the subsequent parameter selection and update process is triggered. The hierarchical update budget allocation module is used to calculate the gradient of the parameters of the pre-trained visual base model at the previous iteration time based on the loss function, calculate the importance score of each layer according to the gradient characteristics of each network layer, determine the parameter update budget allocation ratio of each layer according to the importance score of each layer, and finally allocate the total update budget to each network layer according to the ratio to obtain the parameter update quota of each layer. The in-layer parameter initial selection module is used to calculate the gradient feature of each parameter in each network layer as a parameter importance score for the parameter update quota of each layer. The parameter importance score is converted into a parameter selection probability distribution. Based on the probability distribution, the parameters corresponding to the number of update quotas are sampled in each layer without repetition. All sampled parameters constitute the preliminary candidate update set for the current period. The cross-cycle parameter selection module is used to obtain the final update set of the previous update cycle, merge the preliminary candidate update set with the final update set of the previous update cycle to construct an extended candidate pool; calculate the comprehensive importance score of each parameter in the extended candidate pool after fusing the historical momentum information with the current gradient features, and select the parameters with the highest comprehensive importance score and whose number is equal to the total update budget from the extended candidate pool to form the final update set of the current cycle. The sparse update and task inference module is used to update the initialized binary update mask according to the final update set, sparsely update the parameters of the pre-trained visual base model according to the binary update mask and gradient information, apply the updated model to the downstream UAV image processing task data, and output the target category probability distribution.

Images