Federal fine tuning and acceleration method for visual language basic model

Abstract

The invention discloses a federal fine tuning and acceleration method for a visual language basic model, and belongs to the field of computer vision, and the fine tuning and acceleration method comprises the following specific steps: I, a server carries out the mixed feature knowledge distillation of a CLIP model and a lightweight model through a public data set, and generates a lightweight decoupling model; iI, the server deploys the lightweight decoupling model to each client, initializes and trains a residual block at each client, and executes local training of the client; iII, uploading the training result of each client to a server, and collecting client and model states by the server to perform federal aggregation; according to the method, terminal calculation and storage overhead is greatly reduced, personalization is realized without damaging global prior, privacy requirements are met, uplink data volume is reduced, bottleneck equipment can be identified, training arrangement is adjusted according to the bottleneck equipment, performance loss caused by lagging behind and delay jitter is reduced, and both global generalization capability and local personalization capability are considered.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a federated fine-tuning and acceleration method for a visual language basic model. Background Technology

[0002] Federated Fine-Tuning (FFT) has emerged as a privacy-preserving fine-tuning paradigm. By aggregating trainable parameters through federated learning (FL), FFT ensures the security of private data. FFT relies on diverse sample data for domain adaptation, thus driving the fine-tuning process to migrate to mobile devices that provide highly heterogeneous data. However, applying FFT in mobile edge computing (MEC) environments faces two key challenges: limited computational resources and system heterogeneity. Existing work mainly focuses on combining Parameter Efficient Fine-Tuning (PEFT) with FFT to reduce the number of trainable parameters. CLIP-Adapters deploy frozen CLIPs on mobile devices, fine-tuning only a small fraction of the model parameters. However, mobile devices inevitably require performing the full forward propagation of the CLIP, which still incurs significant computational overhead. Currently, model sizes on mobile devices are typically limited to under 100MB. Most existing CLIP variants exceed this limit. Therefore, existing work still requires deploying the complete base model on devices with limited computational resources. On the other hand, the heterogeneous computational capabilities of mobile devices and the fluctuations in wireless network states lead to system heterogeneity in MEC environments. This heterogeneity introduces uncertainty in FL synchronization time, significantly increasing the overall completion time of FFT. AAFL considers changing the number of local iterations to address the laggard effect. However, existing solutions are designed based on static resource assumptions and cannot guarantee the convergence of FL in wireless networks; therefore, it is particularly important to invent a federated fine-tuning and acceleration method for the visual language fundamental model. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies by proposing a federated fine-tuning and acceleration method for a visual language fundamental model.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: A federated fine-tuning and acceleration method for a visual language foundation model is presented, with the specific steps of which are as follows: Ⅰ. The server uses a public dataset to perform hybrid feature knowledge distillation on the CLIP model and the lightweight model to generate a lightweight decoupled model; II. The server deploys the lightweight decoupled model to each client, initializes and trains residual blocks on each client, and performs local training on the client. III. Upload the training results from each client to the server, and the server collects the client and model states for federated aggregation; IV. Solve for the optimal number of local iterations in each round using PSO, and control the next round of training until training is complete.

[0005] As a further aspect of the present invention, the specific steps of the server in step I performing hybrid feature knowledge distillation on the CLIP model and the lightweight model using a public dataset are as follows: S1.1: The server selects multiple sets of image-text pairs. The resulting image and text dataset serves as a public dataset for distillation. The image encoder (IE) in the CLIP model is used to compute the values ​​of each image in the public dataset. The image feature vector, denoted as The text encoder TE in the CLIP model is used to compute the text in the public dataset. The text feature vector, denoted as The lightweight model DM, based on the current parameters, calculates the values ​​of each image in the public dataset. The image feature vector, denoted as ,in, ; S1.2: Perform vector normalization on the acquired image and text feature vectors to obtain the corresponding unit vectors. Based on the unit vectors of each image and text, use the first... Using images as anchor points, calculate the image... With text The matching probability is then calculated. and The mean squared error (MSE) between the matching probabilities and the mean squared error is linearly combined with weights to generate the overall loss function of MFKD. S1.3: From the public dataset according to the predetermined batch size Sampling is performed during the sampling process, and then the image feature vectors of the IE are calculated sequentially for each batch. TE text feature vector and the image feature vector of DM Then, each feature vector is normalized, the MSE and matching probability of the batch are calculated, and the overall MFKD loss value of the batch is obtained. S1.4: Based on the overall loss value of MFKD, the Adam optimizer is used for backpropagation updates, and RKD is used to control the total number of training rounds. At the same time, after a preset training period, the feature approximation and semantic preservation capabilities of the lightweight model DM are evaluated using validation graphs. If the feature approximation and semantic preservation capabilities begin to decrease after multiple iterations, training is stopped. After training, the parameters of the lightweight model DM with the highest feature approximation and semantic preservation capabilities are saved for distribution to each client.

[0006] As a further aspect of the present invention, the specific formula for calculating the matching probability in S1.2 is as follows: In the formula, Representative image With text The matching probability; This represents the temperature coefficient used in CLIP model pre-training; The first one representing the output of the lightweight model DM The unit vectors of the image feature vectors after Euclidean norm normalization; The first character representing the output of the text encoder TE The unit vector of the group of character feature vectors after Euclidean norm normalization; The specific calculation formula for the overall loss function of MFKD described in S1.2 is as follows: In the formula, ; Represents public datasets The number of samples; Represents hyperparameters, and ; Represents Euclidean distance.

[0007] As a further aspect of the present invention, the specific steps of initializing and training residual blocks on each client and performing local training on the client in step II are as follows: S2.1: The server will send the parameter file of the lightweight model DM, which has been trained and validated on the public dataset, to each client currently participating in the training. After receiving it, each client will load the parameter file into its local model and set all parameters of the lightweight model DM to a frozen state, i.e., untrainable. At the same time, the client will set the residual block RB with the same output dimension as the current local model. S2.2: Each client, based on the frozen DM parameters, performs forward inference on the local input image through the local model to obtain the corresponding image feature vector. Then, the output image feature vector is input into the residual block RB, and the output residual vector of RB is calculated. After that, the image feature vector output by the local model is fused with the output residual vector of RB using weighted residuals to generate image features for downstream tasks. S2.3: Input the image features used for downstream tasks into the classification head to obtain the corresponding logits, and calculate the cross-entropy loss between the logits and the local label as the loss for the local training of the corresponding client. Set the number of local iterations in this round. Each client loops to the set number of iterations in batches locally, and performs forward propagation and loss calculation on each mini-batch. S2.4: After each batch completes forward propagation, calculate the gradient of the corresponding cross-entropy loss and update the residual block RB parameters based on the MGD rule. At the same time, in each iteration, calculate the cosine similarity between the image feature vector output by the local model and the output residual vector of RB. Then, through linear transformation, map the cosine similarity obtained in this round to the weight parameters of the residual flow. S2.5: After each client completes the preset number of iterations, it saves the residual block RB parameters and classification head parameters obtained from the final training locally, calculates and records the final local loss and local gradient of this round of training.

[0008] As a further aspect of the present invention, the specific calculation formula for the weighted residual in S2.2 is as follows: In the formula, Represents the final image feature embedding; The weight parameters represent the output of the residual block; The image feature vector representing the output of the local model; This represents the output residual vector of RB; The specific calculation formula for the fusion weights mentioned in S2.4 is as follows: In the formula, The Euclidean norm representing the eigenvectors; , as well as The meaning is consistent with the meaning of the unknown quantity in the above weighted residual calculation formula.

[0009] As a further aspect of the present invention, the specific steps of the server collecting client and model states for federated aggregation in step III are as follows: S3.1: At the end of each round of federated communication, each client summarizes the residual block RB parameters, local loss values, local gradients and gradient change rates obtained in this round of training, and packages the publicly shareable system and model running status information into personal parameters, and then uploads the client information to the server. S3.2: The server calculates the aggregation weight of each client based on the amount of local data. Then, based on the aggregation weight of each client, the server performs a weighted average of the received RB parameters to obtain new global RB parameters. At the same time, the server calculates the corresponding wireless transmission rate and the time cost of this round of transmission and processing based on the individual parameters of each client. S3.3: The server calculates the global gradient by weighted summation based on the local gradients received from each client. Then, it calculates the gradient deviation of each client's local gradient from the global gradient and sorts each client in descending order of gradient deviation, while recording the corresponding ranking of each client. S3.4: Organize the acquired data into a dictionary set according to the corresponding client, and then send the new global RB parameters to each client through the server. At the same time, send the wireless transmission rate of each client calculated by the server in this round, the time cost of transmission and processing in this round, and the gradient deviation ranking as control information to the corresponding client. After the transmission is completed, the server records the feedback of broadcast success or failure and records the broadcast time in the log for time budget calculation.

[0010] As a further aspect of the present invention, the specific calculation formula for the global RB parameter in S3.2 is as follows: In the formula, Representing the The weight of each client; Represents the updated local model parameters; Represents the total number of local iterations; The specific calculation formula for the wireless transmission rate mentioned in S3.3 is as follows: In the formula, Representing the Channel bandwidth for each client; Representing the The transmit power of each client; Representing the Channel gain for each client; Represents power spectral density; The specific formula for calculating the time cost of transmission and processing mentioned in S3.3 is as follows: In the formula, Represents the size of the transmitted data; Representing the Client CPU frequency; This represents the number of CPU cycles required to process one sample of data. Represents transmission time; This represents the processing time.

[0011] As a further aspect of the present invention, the specific steps for solving the optimal number of local iterations in each round using PSO in step IV are as follows: S4.1: After the server sends out the global RB parameters, based on the calculated wireless transmission rate, the time cost of transmission and processing in this round, and the gradient deviation, a corresponding objective function is established with the goal of minimizing the global loss function in each round, and the time constraint of the objective function is set at the same time. S4.2: Set the local iteration number to multiple positive integers, and use each candidate value as a set of particle positions in PSO. Then, randomly generate multiple sets of particles within the candidate integer range, and save the current position, historical best and group best of each particle. S4.3: Based on the current wireless transmission rate, the time cost of transmission and processing in this round, and the gradient deviation, calculate the objective function value of each particle and use it as its corresponding fitness. Then, according to the discretization speed update rule of PSO, update the position and velocity of each particle. When the fitness change value of each particle converges to the preset range after multiple iterations, stop the iteration. S4.4: After stopping the iteration, sort the particles according to their fitness from high to low, select the particle with the highest current ranking as the global optimal particle, and then use the integer corresponding to the particle as the local iteration number for this round of output. The local iteration number is then sent to each client as the iteration number for the next round of local training.

[0012] As a further aspect of the present invention, the specific calculation formula for the objective function in S4.1 is as follows: In the formula, Representing the The number of local iterations selected by the round server.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: The federated fine-tuning and acceleration method of this visual language foundation model distills the CLIP image encoder into a lightweight decoupled model DM through hybrid feature knowledge distillation (MFKD). The DM is frozen on the edge and a small residual block RB is connected to learn local domain features. The client uses local training with momentum to optimize the RB, while the server periodically aggregates the RB and collects device computing power and network status. Based on the convergence upper bound, the server uses discrete particle swarm optimization (PSO) to adaptively solve for the optimal number of local iterations in each round to meet the time budget and accelerate global convergence. This significantly reduces the terminal's computing and storage overhead, achieves personalization without destroying global priors, meets privacy requirements and reduces the amount of uplink data, can identify bottleneck devices and adjust the training schedule accordingly, reduces performance loss caused by lagging and latency jitter, and balances global generalization ability with local personalization ability. Attached Figure Description

[0014] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0015] Figure 1 This is a flowchart of a federated fine-tuning and acceleration method for a visual language basic model proposed in this invention. Figure 2 This is an overall architecture diagram of a federated fine-tuning and acceleration method for a visual language basic model proposed in this invention; Figure 3 This is a schematic diagram of the hybrid feature knowledge distillation (MFKD) method, which is a federated fine-tuning and acceleration method for a visual language basic model proposed in this invention. Figure 4 This diagram illustrates the comparison of total time consumption between the federated fine-tuning and acceleration methods for a visual language basic model proposed in this invention. Detailed Implementation

[0016] Reference Figure 1-4 A federated fine-tuning and acceleration method for a visual language foundation model is described, with the specific steps as follows: The server uses a public dataset to perform hybrid feature knowledge distillation on the CLIP model and the lightweight model, generating a lightweight decoupled model.

[0017] For details, please refer to Figure 1-3 It can be seen that the server selects multiple sets of image-text pairs. The resulting image and text dataset serves as a public dataset for distillation. The image encoder (IE) in the CLIP model is used to compute the values ​​of each image in the public dataset. The image feature vector, denoted as The text encoder TE in the CLIP model is used to compute the text in the public dataset. The text feature vector, denoted as The lightweight model DM, based on the current parameters, calculates the values ​​of each image in the public dataset. The image feature vector, denoted as ,in, The feature vectors of each image and text are normalized to obtain the corresponding unit vectors. Based on the unit vectors of each image and text, the first... Using images as anchor points, calculate the image... With text The matching probability is then calculated. and The mean squared error (MSE) between the matching probabilities and the MSE is linearly combined with the matching probabilities and the MSE according to weights to generate the overall loss function of MFKD. This loss function is then applied to a public dataset at a predetermined batch size. Sampling is performed during the sampling process, and then the image feature vectors of the IE are calculated sequentially for each batch. TE text feature vector and the image feature vector of DM Then, each feature vector is normalized, the MSE and matching probability of the batch are calculated, and the overall MFKD loss value of the batch is obtained. Based on the overall MFKD loss value, the Adam optimizer is used for backpropagation update, and RKD is used to control the total number of training rounds. At the same time, after a preset training period, the feature approximation and semantic preservation ability of the lightweight model DM are evaluated using the validation graph. If the feature approximation and semantic preservation ability begin to decrease after multiple iterations, training is stopped. After training, the parameters of the lightweight model DM with the highest feature approximation and semantic preservation ability are saved for distribution to each client.

[0018] In addition, it should be noted that the specific formula for calculating the matching probability is as follows: In the formula, Representative image With text The matching probability; This represents the temperature coefficient used in CLIP model pre-training; The first one representing the output of the lightweight model DM The unit vectors of the image feature vectors after Euclidean norm normalization; The first character representing the output of the text encoder TE The unit vector of the group of character feature vectors after Euclidean norm normalization; The specific formula for calculating the overall loss function of MFKD is as follows: In the formula, ; Represents public datasets The number of samples; Represents hyperparameters, and ; Represents Euclidean distance.

[0019] The server deploys the lightweight decoupled model to each client, initializes and trains residual blocks on each client, and performs local training on the client.

[0020] For details, please refer to Figure 1-2It is understood that the server will distribute the parameter file of the lightweight model DM, which has been trained and validated on a public dataset, to each client currently participating in the training. Upon receiving the file, each client loads the parameter file into its local model and sets all parameters of the lightweight model DM to a frozen state, i.e., untrainable. Simultaneously, each client sets a residual block RB with the same output dimension as its local model. Based on the frozen DM parameters, each client performs forward inference on the local input image using its local model to obtain the corresponding image feature vector. The output image feature vector is then input into the residual block RB, and the output residual vector of RB is calculated. Finally, a weighted residual is used to fuse the image feature vector output by the local model with the output residual vector of RB to generate image features for downstream tasks. These image features are then input into the classification head to obtain the corresponding lo values. The system calculates the cross-entropy loss between logits and local labels as the loss for local training on the corresponding client. It sets the number of local iterations for each round, and each client iterates locally in batches until the set number of iterations is reached. Forward propagation and loss calculation are performed on each mini-batch. After forward propagation in each batch, the gradient of the corresponding cross-entropy loss is calculated, and the residual block RB parameters are updated based on the MGD rule. Simultaneously, during each iteration, the cosine similarity between the image feature vector output by the local model and the output residual vector of RB is calculated. Then, a linear transformation is used to map the cosine similarity obtained in this round to the weight parameters of the residual flow. After each client completes the preset number of iterations, the final trained residual block RB parameters and classification head parameters are saved locally. The final local loss and local gradient of this training round are calculated and recorded.

[0021] It should be further explained that the specific formula for calculating the weighted residual is as follows: In the formula, Represents the final image feature embedding; The weight parameters represent the output of the residual block; The image feature vector representing the output of the local model; This represents the output residual vector of RB; The specific formula for calculating the fusion weight is as follows: In the formula, The Euclidean norm representing the eigenvectors; , as well as The meaning is consistent with the meaning of the unknown quantity in the above weighted residual calculation formula.

[0022] The training results from each client are uploaded to the server, and the server collects the client and model states for federated aggregation.

[0023] For details, please refer to Figure 1-2 It can be seen that at the end of each round of federated communication, each client summarizes the residual block RB parameters, local loss values, local gradients, and gradient change rates obtained in this round of training. Simultaneously, it packages publicly shareable system and model runtime information into personal parameters. Then, it uploads all client information to the server. The server calculates the aggregate weights based on the amount of local data from each client. Then, based on the aggregate weights of each client, the server performs a weighted average of the received RB parameters to obtain new global RB parameters. At the same time, the server calculates the corresponding wireless transmission rate and the time cost of transmission and processing in this round based on the personal parameters of each client. The server then calculates the weights based on the received data from each client. The local gradient of each client is calculated by weighted summation to obtain the global gradient. Then, the gradient deviation of each client's local gradient from the global gradient is calculated, and each client is sorted from high to low according to the gradient deviation. At the same time, the corresponding ranking of each client is recorded. The acquired data is organized into a dictionary set according to the corresponding client. Then, the new global RB parameters are sent to each client through the server. At the same time, the wireless transmission rate of each client calculated by the server in this round, the time cost of transmission and processing in this round, and the gradient deviation ranking are sent to the corresponding clients as control information. After the transmission is completed, the server records the feedback of broadcast success or failure and records the broadcast time in the log for time budget calculation.

[0024] In addition, it should be noted that the specific calculation formula for the global RB parameter is as follows: In the formula, Representing the The weight of each client; Represents the updated local model parameters; Represents the total number of local iterations; The specific formula for calculating wireless transmission rate is as follows: In the formula, Representing the Channel bandwidth for each client; Representing the The transmit power of each client; Representing the Channel gain for each client; Represents power spectral density; The specific formula for calculating the time cost of transmission and processing is as follows: In the formula, Represents the size of the transmitted data; Representing the Client CPU frequency; This represents the number of CPU cycles required to process one sample of data. Represents transmission time; This represents the processing time.

[0025] The optimal number of local iterations in each round is determined by PSO, and the next round of training is controlled until training is complete.

[0026] For details, please refer to Figure 1-2 It can be seen that after the server sends out the global RB parameters, based on the calculated wireless transmission rate, the time cost of transmission and processing in this round, and the gradient bias, a corresponding objective function is established with the goal of minimizing the global loss function in each round. At the same time, the time constraint of the objective function is set, and the value of the local iteration number is set to multiple sets of positive integers. Each candidate value is used as a set of particle positions in PSO. Then, multiple sets of particles are randomly generated within the candidate integer range, and the current position, historical best, and group best of each particle are saved. Based on the current wireless transmission rate, the time cost of transmission and processing in this round, and the gradient bias, the objective function value of each particle is calculated and used as its corresponding fitness. Then, according to the discretization speed update rule of PSO, the position and velocity of each particle are updated. When the fitness change value of each particle converges to the preset range after multiple iterations, the iteration stops. After stopping the iteration, each particle is sorted from high to low fitness, and the particle with the current first ranking is selected as the global optimal particle. Then, the integer corresponding to this particle is used as the local iteration number of the output in this round, and the local iteration number is sent to each client as the iteration number of the next round of local training.

[0027] It should be noted that the specific formula for calculating the objective function is as follows: In the formula, Representing the The number of local iterations selected by the round server.

[0028] It should be further explained that the reference Figure 1 , 4 As shown, the time consumption of different FL acceleration algorithms changed with the number of training epochs, displaying the time consumption of each epoch and the total time consumption in the first 20 training epochs. DESA-FFT showed the lowest time consumption on both datasets, reducing the time consumption by 20.7% compared to the best baseline AAFL.

[0029] Furthermore, due to its dynamic state-aware design, DESA-FFT exhibits more stable time consumption per round compared to the other two methods. The results show that DESA-FFT improves overall training efficiency by dynamically adjusting the number of local iterations based on a real-time network and model state awareness.

Claims

1. A federated fine-tuning and acceleration method for a visual language basic model, characterized in that, The specific steps of this fine-tuning and acceleration method are as follows: Ⅰ. The server uses a public dataset to perform hybrid feature knowledge distillation on the CLIP model and the lightweight model to generate a lightweight decoupled model; II. The server deploys the lightweight decoupled model to each client, initializes and trains residual blocks on each client, and performs local training on the client. III. Upload the training results from each client to the server, and the server collects the client and model states for federated aggregation; IV. Solve for the optimal number of local iterations in each round using PSO, and control the next round of training until training is complete.

2. The method for federated fine-tuning and acceleration of a visual language basic model according to claim 1, characterized in that, The specific steps of the server in step I for performing hybrid feature knowledge distillation on the CLIP model and the lightweight model using a public dataset are as follows: S1.1: The server selects multiple sets of image-text pairs. The resulting image and text dataset serves as a public dataset for distillation. The image encoder (IE) in the CLIP model is used to compute the values ​​of each image in the public dataset. The image feature vector, denoted as The text encoder TE in the CLIP model is used to compute the text in the public dataset. The text feature vector, denoted as The lightweight model DM, based on the current parameters, calculates the values ​​of each image in the public dataset. The image feature vector, denoted as ,in, ; S1.2: Perform vector normalization on the acquired image and text feature vectors to obtain the corresponding unit vectors. Based on the unit vectors of each image and text, use the first... Using images as anchor points, calculate the image... With text The matching probability is then calculated. and The mean squared error (MSE) between the matching probabilities and the mean squared error is linearly combined with weights to generate the overall loss function of MFKD. S1.3: From the public dataset according to the predetermined batch size Sampling is performed during the sampling process, and then the image feature vectors of the IE are calculated sequentially for each batch. TE text feature vector and the image feature vector of DM Then, each feature vector is normalized, the MSE and matching probability of the batch are calculated, and the overall MFKD loss value of the batch is obtained. S1.4: Based on the overall loss value of MFKD, the Adam optimizer is used for backpropagation updates, and RKD is used to control the total number of training rounds. At the same time, after a preset training period, the feature approximation and semantic preservation capabilities of the lightweight model DM are evaluated using validation graphs. If the feature approximation and semantic preservation capabilities begin to decrease after multiple iterations, training is stopped. After training, the parameters of the lightweight model DM with the highest feature approximation and semantic preservation capabilities are saved for distribution to each client.

3. The method for federated fine-tuning and acceleration of a visual language basic model according to claim 2, characterized in that, The specific formula for calculating the matching probability mentioned in S1.2 is as follows: In the formula, Representative image With text The matching probability; This represents the temperature coefficient used in CLIP model pre-training; The first one representing the output of the lightweight model DM The unit vectors of the image feature vectors after Euclidean norm normalization; The first character representing the output of the text encoder TE The unit vector of the group of character feature vectors after Euclidean norm normalization; The specific calculation formula for the overall loss function of MFKD described in S1.2 is as follows: In the formula, ; Represents public datasets The number of samples; Represents hyperparameters, and ; Represents Euclidean distance.

4. The method for federated fine-tuning and acceleration of a visual language basic model according to claim 2, characterized in that, The specific steps for initializing and training residual blocks on each client and performing local training on the client, as described in step II, are as follows: S2.1: The server will send the parameter file of the lightweight model DM, which has been trained and validated on the public dataset, to each client currently participating in the training. After receiving it, each client will load the parameter file into its local model and set all parameters of the lightweight model DM to a frozen state, i.e., untrainable. At the same time, the client will set the residual block RB with the same output dimension as the current local model. S2.2: Each client, based on the frozen DM parameters, performs forward inference on the local input image through the local model to obtain the corresponding image feature vector. Then, the output image feature vector is input into the residual block RB, and the output residual vector of RB is calculated. After that, the image feature vector output by the local model is fused with the output residual vector of RB using weighted residuals to generate image features for downstream tasks. S2.3: Input the image features used for downstream tasks into the classification head to obtain the corresponding logits, and calculate the cross-entropy loss between the logits and the local label as the loss for the local training of the corresponding client. Set the number of local iterations in this round. Each client loops to the set number of iterations in batches locally, and performs forward propagation and loss calculation on each mini-batch. S2.4: After each batch completes forward propagation, calculate the gradient of the corresponding cross-entropy loss and update the residual block RB parameters based on the MGD rule. At the same time, in each iteration, calculate the cosine similarity between the image feature vector output by the local model and the output residual vector of RB. Then, through linear transformation, map the cosine similarity obtained in this round to the weight parameters of the residual flow. S2.5: After each client completes the preset number of iterations, it saves the residual block RB parameters and classification head parameters obtained from the final training locally, calculates and records the final local loss and local gradient of this round of training.

5. The method for federated fine-tuning and acceleration of a visual language basic model according to claim 4, characterized in that, The specific steps for the server to collect client and model states and perform federated aggregation as described in step III are as follows: S3.1: At the end of each round of federated communication, each client summarizes the residual block RB parameters, local loss values, local gradients and gradient change rates obtained in this round of training, and packages the publicly shareable system and model running status information into personal parameters, and then uploads the client information to the server. S3.2: The server calculates the aggregation weight of each client based on the amount of local data. Then, based on the aggregation weight of each client, the server performs a weighted average of the received RB parameters to obtain new global RB parameters. At the same time, the server calculates the corresponding wireless transmission rate and the time cost of this round of transmission and processing based on the individual parameters of each client. S3.3: The server calculates the global gradient by weighted summation based on the local gradients received from each client. Then, it calculates the gradient deviation of each client's local gradient from the global gradient and sorts each client in descending order of gradient deviation, while recording the corresponding ranking of each client. S3.4: Organize the acquired data into a dictionary set according to the corresponding client, and then send the new global RB parameters to each client through the server. At the same time, send the wireless transmission rate of each client calculated by the server in this round, the time cost of transmission and processing in this round, and the gradient deviation ranking as control information to the corresponding client. After the transmission is completed, the server records the feedback of broadcast success or failure and records the broadcast time in the log for time budget calculation.

6. The method for federated fine-tuning and acceleration of a visual language basic model according to claim 5, characterized in that, The specific calculation formula for the global RB parameter mentioned in S3.2 is as follows: In the formula, Representing the The weight of each client; Represents the updated local model parameters; Represents the total number of local iterations; The specific calculation formula for the wireless transmission rate mentioned in S3.3 is as follows: In the formula, Representing the Channel bandwidth for each client; Representing the The transmit power of each client; Representing the Channel gain for each client; Represents power spectral density; The specific formula for calculating the time cost of transmission and processing mentioned in S3.3 is as follows: In the formula, Represents the size of the transmitted data; Representing the Client CPU frequency; This represents the number of CPU cycles required to process one sample of data. Represents transmission time; This represents the processing time.

7. The method for federated fine-tuning and acceleration of a visual language basic model according to claim 5, characterized in that, The specific steps for solving the optimal number of local iterations in each round using PSO, as described in step IV, are as follows: S4.1: After the server sends out the global RB parameters, based on the calculated wireless transmission rate, the time cost of transmission and processing in this round, and the gradient deviation, a corresponding objective function is established with the goal of minimizing the global loss function in each round, and the time constraint of the objective function is set at the same time. S4.2: Set the local iteration number to multiple positive integers, and use each candidate value as a set of particle positions in PSO. Then, randomly generate multiple sets of particles within the candidate integer range, and save the current position, historical best and group best of each particle. S4.3: Based on the current wireless transmission rate, the time cost of transmission and processing in this round, and the gradient deviation, calculate the objective function value of each particle and use it as its corresponding fitness. Then, according to the discretization speed update rule of PSO, update the position and velocity of each particle. When the fitness change value of each particle converges to the preset range after multiple iterations, stop the iteration. S4.4: After stopping the iteration, sort the particles according to their fitness from high to low, select the particle with the highest current ranking as the global optimal particle, and then use the integer corresponding to the particle as the local iteration number for this round of output. The local iteration number is then sent to each client as the iteration number for the next round of local training.

8. The method for federated fine-tuning and acceleration of a visual language basic model according to claim 7, characterized in that, The specific calculation formula for the objective function described in S4.1 is as follows: In the formula, Representing the The number of local iterations selected by the round server.

Images