A resource-aware based nested multi-modal adapter personalized federated learning method

Abstract

The application relates to the technical field of federated learning and multi-modal model training, in particular to a resource-aware nested multi-modal adapter personalized federated learning method, which comprises the following steps: S1: initialization: a server initializes a pre-trained visual language model backbone, a maximum shared projection space, a client set, client resource information, an adapter capacity level set and security aggregation related parameters. Through client computing, communication and storage resource differences, different multi-modal adapter structures with different capacities are allocated to different clients, so that low-resource clients can participate in federated training with small training overhead, and meanwhile, high-resource clients can fully utilize their local computing capacity, thereby improving the adaptability of model training in a resource heterogeneous environment.

Description

Technical Field

[0001] This invention relates to the field of federated learning and multimodal model training technology, specifically a personalized federated learning method based on resource-aware nested multimodal adapters. Background Technology

[0002] In federated learning, the server first initializes a global model and sends the model parameters to the clients participating in the training. After receiving the model, the client trains it locally using its local dataset, and then uploads the updated model parameters or gradients to the server. The server receives the parameters uploaded by the clients, aggregates them to obtain a new global model, and continues to distribute it to the next round of clients participating in the training. This process iterates through multiple rounds to complete model training. Federated learning enables collaborative modeling among multiple parties without directly collecting raw data from clients, and is therefore widely used in privacy-sensitive scenarios such as mobile terminals, IoT devices, healthcare, and edge intelligence.

[0003] Federated learning is essentially a distributed machine learning technique. Traditional centralized machine learning typically requires concentrating data from various devices or institutions onto a server for unified training, which can easily lead to privacy breaches, high data transfer costs, and insufficient data compliance. Federated learning mitigates the privacy risks associated with centralized training to some extent by keeping data locally on the client side and only uploading model update results. Furthermore, federated learning can utilize local computing resources on the client side, reducing the need for raw data transfer and improving the feasibility of model training in a distributed environment.

[0004] FedAvg is a fundamental aggregation algorithm commonly used in federated learning. In each training round, a subset of clients participate in local training. These clients use their local data to update the model using stochastic gradient descent, and then upload the updated model parameters to the server. The server then performs a weighted average of the uploaded parameters based on the number of client samples or preset weights to generate a new global model. While FedAvg is simple in structure and easy to implement, it typically requires that the model parameters uploaded by different clients have the same structure and dimensions. When there are significant differences in resources or model structure among clients, the ordinary federated averaging method can be limited.

[0005] Personalized Federated Learning (PFL) is a method developed based on traditional federated learning. Traditional federated learning typically aims to train a uniform global model. However, in practical applications, different clients often have varying data distributions, acquisition environments, task requirements, and device capabilities, making it difficult for a single global model to adapt to all clients simultaneously. Personalized Federated Learning addresses this by introducing client-local parameters, personalized model branches, local fine-tuning modules, or a combination of shared and private parameters. This allows the model to learn common knowledge across clients while also adapting to the local data characteristics of different clients.

[0006] Vision-language models are a class of multimodal models capable of processing both visual and textual information simultaneously. These models typically include an image encoder and a text encoder, and through joint representation learning of image features and text semantic features, they can perform tasks such as image classification, image-text retrieval, cross-modal matching, medical image analysis, and multimodal understanding. In recent years, pre-trained vision-language models have demonstrated strong transfer capabilities across various downstream tasks; however, these models are usually large in scale. Directly fine-tuning the complete model in a federated learning environment incurs high computational, storage, and communication overhead.

[0007] Parameter-efficient fine-tuning is a technique for reducing the cost of fine-tuning large-scale pre-trained models. This technique typically freezes the backbone parameters of the pre-trained model and trains only a small number of newly added parameters or lightweight modules, such as cue parameters, low-rank modules, and adapter modules. By using parameter-efficient fine-tuning, the number of training and communication parameters can be reduced while maintaining the core capabilities of the pre-trained model, thereby improving the deployment feasibility of large models in resource-constrained scenarios.

[0008] Low-rank training is a common approach to model compression and efficient training. The basic idea is to approximate the original high-dimensional parameter space using a lower-rank parameter matrix or a lower-dimensional subspace, thereby reducing the number of trainable parameters and computational complexity. In large model fine-tuning and federated learning, low-rank structures can reduce the burden of local training and communication overhead on the client side. However, when there are significant differences in client resources, fixing the low-rank structure may limit the training capabilities of high-resource clients, while a uniform high-rank structure may exceed the capacity of low-resource clients.

[0009] Secure aggregation is a crucial technique in federated learning for protecting the privacy of client-side model updates. This technique typically requires clients to mask their parameters or gradients before uploading their model updates, ensuring the server only receives the aggregated results of multiple client updates and cannot directly view the plaintext updates from individual clients. Secure aggregation mitigates the potential privacy risks during model update uploads and is a commonly used technique in privacy-preserving federated learning.

[0010] Therefore, the following problems still exist in the existing technology: 1. The unified adapter structure is difficult to adapt to clients with heterogeneous resources; Second, direct zero-fill aggregation of heterogeneous adapter parameters will dilute shared projection updates; Third, conventional security aggregation makes it difficult to verify the consistency between heterogeneous adapter structures and aggregation results; Fourth, multimodal federated training needs to simultaneously consider personalization, local resource constraints, communication efficiency, and aggregation integrity.

[0011] Based on this, the present invention provides a resource-aware nested multimodal adapter personalized federated learning method to solve the above-mentioned technical problems. Summary of the Invention

[0012] The purpose of this invention is to provide a personalized federated learning method based on resource-aware nested multimodal adapters. This invention allocates multimodal adapter structures of different capacities to different clients by taking advantage of differences in client computing, communication and storage resources. This enables low-resource clients to participate in federated training with less training overhead, while enabling high-resource clients to make full use of their local computing power, thereby improving the adaptability of model training in heterogeneous resource environments.

[0013] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a resource-aware, nested multimodal adapter-based personalized federated learning method, comprising the following steps: S1: Initialization: The server initializes the pre-trained visual language model backbone, maximum shared projection space, client set, client resource information, adapter capacity hierarchy set, and security aggregation related parameters; S2: Resource assessment and capacity allocation: The server allocates the corresponding adapter rank and adapter insertion layer to the client based on the client's computing power, communication bandwidth, and storage capacity; S3: Slice Distribution and Local Training: The server sends a shared projection slice corresponding to the capacity level to the client selected for training. The client trains the multimodal adapter locally using private data and retains the local personalized adapter parameters. S4: Shared Update Generation and Secure Upload: The client generates a shared projection update based on the local training results, constructs an activation mask, embeds the shared projection update into the maximum shared projection space, and then quantizes, masks, commits, and signs it before uploading it to the server. S5: Secure Aggregation and Structure Verification: The server sums the mask updates uploaded by the client to obtain the aggregated update result and generates a coordinate-level aggregate denominator. The client or verifier verifies the consistency of the aggregated result, commitment, signature, and round structure descriptor. S6: Model Update: When validation passes, the server updates the shared projection parameters according to the coordinate-level nested aggregation rules and uses the updated shared projection parameters for the next round of training; when validation fails, the aggregation result of this round is rejected, and the training is terminated or re-executed.

[0014] The initialization process in S1 includes the following steps: S1.1: The server initializes a pre-trained visual language model containing an image encoder and a text encoder, and freezes the backbone parameters of the image encoder and the text encoder during training; S1.2: The server determines the maximum adapter rank. and maximum number of adapter insertion layers Initialize the maximum shared projection space; S1.3: The server initializes the client set and each client's local private dataset, and records the client's computing power, communication power and storage power; S1.4: The server sets an ordered set of adapter capacity levels.

[0015] The resource assessment and capacity allocation process in S2 includes the following steps: S2.1: The server pre-measures the computational, communication, and storage overhead of each candidate capacity level and provides this information to the client. Construct a set of feasible capacity levels; S2.2: The server selects the highest capacity level that the client can support from the set of feasible capacity levels; when the client cannot meet the resource requirements of the lowest capacity level, the server does not select the client to participate in this round of training. S2.3: The server inserts layers based on the client's adapter. Determine the set of active layers.

[0016] The slice distribution and local training process in S3 includes the following steps: S3.1: In the first At the start of a training round, the server selects a set of clients from the client set to participate in this training round. ; S3.2: For clients participating in training The server is configured according to its capacity. The server extracts the corresponding shared projection slice from the maximum shared state; the server then sends the shared projection slice, capacity configuration, set of activation layers, current model version, and training epoch information to the client. S3.3: The client receives the shared projection slice and builds the multimodal adapter only in the activation layer on the fully frozen visual language model backbone; S3.4: The adapter branch connects in parallel with the frozen Transformer layer; S3.5: The client uses local private data to train a multimodal adapter and obtains updated shared projection slices and local personalized projection parameters.

[0017] In S3.3, the modality-specific lower projection matrix and upper projection matrix are retained locally on the client, and the shared projection slice is used for cross-client collaborative aggregation.

[0018] The specific steps of the shared update generation and secure upload process in S4 are as follows: S4.1: The client constructs an activation mask based on its adapter rank and set of activation layers; S4.2: The client calculates the difference in activity shared projection slices before and after local training, and the client aggregates the weights based on the public data. Generate weighted activity updates; S4.3: The client performs trimming and quantification of weighted activity updates; S4.4: For any client participating in training... Both parties generate paired random masks based on the shared seed; S4.5: The client generates an additive homomorphic commitment to the quantized update; S4.6: The server generates a round structure descriptor based on the set of participating clients, capacity configuration, aggregate weights, shared slice summary, aggregate denominator summary, model version, masking strategy, and quantization parameters for this round. The client generates a signature for the commitment and round structure descriptor; S4.7: The client uploads the mask update, commitment, signature, and corresponding structure verification information to the server.

[0019] In S4.4, the paired random masks cancel each other out when the server sums the vectors uploaded by all clients.

[0020] The specific steps of the secure aggregation and structural verification process in S5 are as follows: S5.1: The server receives the mask updates uploaded by participating clients and sums them up; S5.2: The server calculates the effective aggregation denominator at the coordinate level based on the aggregation weight and activation mask of each client; S5.3: The server sends the aggregate quantization update, the aggregate commitment blinding value, the aggregate denominator digest, and the commitments and signatures of each client to the client or the verifier.

[0021] The specific steps for the client or authenticator in S5.3 are as follows: S5.3.1: The client or verifier verifies whether the aggregated quantized update is consistent with the sum of updates committed by the client using the following formula; S5.3.2: The client or verifier verifies the signature of each client and checks whether the set of participating clients, capacity configuration, aggregate weight, activation mask, aggregate denominator, model version, shared slice digest, masking strategy and quantization parameters are consistent with the round structure descriptor; S5.3.3: If both the contemporary number verification and the structural consistency verification pass, accept the aggregation result of this round; otherwise, reject the aggregation result of this round and terminate or re-execute this round of training.

[0022] The model update process in S6 consists of the following steps: S6.1: The server performs signed decoding and dequantization on the verified aggregate quantization update to obtain the shared projection aggregate molecule. ; S6.2: For any coordinate after shared projection vectorization The server updates the maximum shared state according to the following rules: ; Among them, when When no client trains the coordinates in this round, the server keeps the coordinates unchanged; when At that time, the server only uses the client updates that were actually trained on that coordinate for weighted averaging; S6.3: The server will use the updated maximum shared state for the next round of training and send the latest shared projection slices corresponding to its capacity level to subsequent participating clients; the clients will continue to retain the local modality-specific adapter parameters and will not upload them to the server.

[0023] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention utilizes the differences in client computing, communication, and storage resources to allocate multimodal adapter structures of different capacities to different clients, enabling low-resource clients to participate in federated training with less training overhead, while allowing high-resource clients to fully utilize their local computing power, thereby improving the adaptability of model training in heterogeneous resource environments.

[0024] 2. This invention retains modality-specific adapter parameters locally on the client and only aggregates shared projection parameters across clients. It can achieve global knowledge sharing while preserving client-specific capabilities, making it suitable for personalized federated learning scenarios with large differences in data distribution.

[0025] 3. This invention adopts nested slice training and coordinate-level nested aggregation, so that each shared projected coordinate is averaged only by the client that actually trained that coordinate, avoiding the effective update dilution problem caused by ordinary zero-filling aggregation, and improving the accuracy of heterogeneous adapter shared parameter aggregation.

[0026] 4. This invention adopts a structure-bound secure aggregation method, which protects the privacy of individual client updates while binding the aggregation result with information such as the client set, adapter rank layer configuration, activation mask, aggregation denominator, model hash, and quantization parameters. This can detect abnormal situations such as aggregation replacement, client deletion, replay, and structural inconsistency, thereby improving the aggregation credibility of the federated training process. Attached Figure Description

[0027] Figure 1 This is an overall flowchart of a resource-aware nested multimodal adapter personalized federated learning method according to the present invention.

[0028] Figure 2 This is a system architecture diagram of a personalized federated learning method based on a resource-aware nested multimodal adapter according to the present invention.

[0029] Figure 3 This diagram illustrates the resource adaptive adapter training and secure aggregation in a resource-aware nested multimodal adapter personalized federated learning method of the present invention. Detailed Implementation

[0030] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0031] Example: like Figures 1-3 As shown, this embodiment provides a resource-aware, nested multimodal adapter-based personalized federated learning method, including the following steps: S1: Initialization: The server initializes the pre-trained visual language model backbone, maximum shared projection space, client set, client resource information, adapter capacity hierarchy set, and security aggregation related parameters; S2: Resource assessment and capacity allocation: The server allocates the corresponding adapter rank and adapter insertion layer to the client based on the client's computing power, communication bandwidth, and storage capacity; S3: Slice Distribution and Local Training: The server sends a shared projection slice corresponding to the capacity level to the client selected for training. The client trains the multimodal adapter locally using private data and retains the local personalized adapter parameters. S4: Shared Update Generation and Secure Upload: The client generates a shared projection update based on the local training results, constructs an activation mask, embeds the shared projection update into the maximum shared projection space, and then quantizes, masks, commits, and signs it before uploading it to the server. S5: Secure Aggregation and Structure Verification: The server sums the mask updates uploaded by the client to obtain the aggregated update result and generates a coordinate-level aggregate denominator. The client or verifier verifies the consistency of the aggregated result, commitment, signature, and round structure descriptor. S6: Model Update: When validation passes, the server updates the shared projection parameters according to the coordinate-level nested aggregation rules and uses the updated shared projection parameters for the next round of training; when validation fails, the aggregation result of this round is rejected, and the training is terminated or re-executed.

[0032] The initialization process in S1 consists of the following steps: S1.1: The server initializes a pre-trained visual language model containing an image encoder and a text encoder, and freezes the backbone parameters of the image encoder and the text encoder during training; S1.2: The server determines the maximum adapter rank. and maximum number of adapter insertion layers Initialize the maximum shared projection space; S1.3: The server initializes the client set and each client's local private dataset, and records the client's computing power, communication power and storage power; S1.4: The server sets an ordered set of adapter capacity levels.

[0033] The resource assessment and capacity allocation process in S2 involves the following steps: S2.1: The server pre-measures the computational, communication, and storage overhead of each candidate capacity level and provides this information to the client. Construct a set of feasible capacity levels; S2.2: The server selects the highest capacity level that the client can support from the set of feasible capacity levels; when the client cannot meet the resource requirements of the lowest capacity level, the server does not select the client to participate in this round of training. S2.3: The server inserts layers based on the client's adapter. Determine the set of active layers.

[0034] The slice distribution and local training process in S3 are detailed as follows: S3.1: In the first At the start of a training round, the server selects a set of clients from the client set to participate in this training round. ; S3.2: For clients participating in training The server is configured according to its capacity. The server extracts the corresponding shared projection slice from the maximum shared state; the server then sends the shared projection slice, capacity configuration, set of activation layers, current model version, and training epoch information to the client. S3.3: The client receives the shared projection slice and builds the multimodal adapter only in the activation layer on the fully frozen visual language model backbone; S3.4: The adapter branch connects in parallel with the frozen Transformer layer; S3.5: The client uses local private data to train a multimodal adapter and obtains updated shared projection slices and local personalized projection parameters.

[0035] In S3.3, the modality-specific downprojection matrix and upprojection matrix are retained locally on the client, and shared projection slices are used for cross-client collaborative aggregation.

[0036] The specific steps for generating and securely uploading shared updates in S4 are as follows: S4.1: The client constructs an activation mask based on its adapter rank and set of activation layers; S4.2: The client calculates the difference in activity shared projection slices before and after local training, and the client aggregates the weights based on the public data. Generate weighted activity updates; S4.3: The client performs trimming and quantification of weighted activity updates; S4.4: For any client participating in training... Both parties generate paired random masks based on the shared seed; S4.5: The client generates an additive homomorphic commitment to the quantized update; S4.6: The server generates a round structure descriptor based on the set of participating clients, capacity configuration, aggregate weights, shared slice summary, aggregate denominator summary, model version, masking strategy, and quantization parameters for this round. The client generates a signature for the commitment and round structure descriptor; S4.7: The client uploads the mask update, commitment, signature, and corresponding structure verification information to the server.

[0037] In S4.4, the paired random masks cancel each other out when the server sums the vectors uploaded by all clients.

[0038] The specific steps of the secure aggregation and structural verification process in S5 are as follows: S5.1: The server receives the mask updates uploaded by participating clients and sums them up; S5.2: The server calculates the effective aggregation denominator at the coordinate level based on the aggregation weight and activation mask of each client; S5.3: The server sends the aggregate quantization update, the aggregate commitment blinding value, the aggregate denominator digest, and the commitments and signatures of each client to the client or the verifier.

[0039] In S5.3, the specific steps for the client or authenticator are as follows: S5.3.1: The client or verifier verifies whether the aggregated quantized update is consistent with the sum of updates committed by the client using the following formula; S5.3.2: The client or verifier verifies the signature of each client and checks whether the set of participating clients, capacity configuration, aggregate weight, activation mask, aggregate denominator, model version, shared slice digest, masking strategy and quantization parameters are consistent with the round structure descriptor; S5.3.3: If both the contemporary number verification and the structural consistency verification pass, accept the aggregation result of this round; otherwise, reject the aggregation result of this round and terminate or re-execute this round of training.

[0040] The specific steps of the model update process in S6 are as follows: S6.1: The server performs signed decoding and dequantization on the verified aggregate quantization update to obtain the shared projection aggregate molecule. ; S6.2: For any coordinate after shared projection vectorization The server updates the maximum shared state according to the following rules: ; Among them, when When no client trains the coordinates in this round, the server keeps the coordinates unchanged; when At that time, the server only uses the client updates that were actually trained on that coordinate for weighted averaging; S6.3: The server will use the updated maximum shared state for the next round of training and send the latest shared projection slices corresponding to its capacity level to subsequent participating clients; the clients will continue to retain the local modality-specific adapter parameters and will not upload them to the server.

[0041] like Figures 1-3As shown, this embodiment provides a resource-aware nested multimodal adapter personalized federated learning method, the specific method is as follows: First, S1: Initialization: The server initializes the pre-trained visual language model backbone, maximum shared projection space, client set, client resource information, adapter capacity hierarchy set, and security aggregation related parameters; the specific steps are as follows: S1.1: The server initializes the pre-trained visual language model containing image encoder and text encoder, and freezes the backbone parameters of image encoder and text encoder during training; S1.2: The server, according to the maximum adapter rank... and maximum number of adapter insertion layers Initialize the maximum shared projection space, the first Round The shared projection matrix of the candidate layers is represented as: ; The shared projection matrix of all candidate layers constitutes the first... Maximum shared state in rounds: ; in, This represents the maximum shared state during the t-th round of training. For the tth round The shared projection matrix corresponding to each candidate layer; This is the set of candidate activation layers corresponding to the maximum capacity adapter; This represents the Transformer layer index in the visual language model.

[0042] S1.3: The server initializes the client set and each client's local private dataset, and records the client's computing power, communication power, and storage power. The resource status is represented as follows: ; in, , and Clients respectively Computational budget, communication budget, and storage budget; S1.4: Server settings include an ordered set of adapter capacity levels: ; in, and Let be the adapter rank and the adapter insertion level of the g-th capacity level, respectively. This represents the nested relationship between capacity levels.

[0043] S2: Resource Assessment and Capacity Allocation: The server allocates the corresponding adapter rank and adapter insertion level to the client based on the client's computing power, communication bandwidth, and storage capacity; the specific steps are as follows: S2.1: The server pre-measures the computational, communication, and storage overhead of each candidate capacity level and provides this information to the client. Construct a set of feasible capacity levels: ; S2.2: The server selects the maximum capacity tier that the client can support from the set of feasible capacity tiers. ; When a client cannot meet the resource requirements of the minimum capacity level, the server will not select that client to participate in this round of training; S2.3: The server inserts layers based on the client's adapter. Determine the set of activation layers: ; in, This represents the number of Transformer layers in the encoder of the visual language model. Indicates client The actual set of layers for which the adapter is enabled.

[0044] S3: Slice Distribution and Local Training: The server sends shared projected slices corresponding to the capacity level to the clients selected for training. The clients train the multimodal adapter locally using their private data and retain their local personalized adapter parameters. The specific steps are as follows: S3.1: In the first At the start of a training round, the server selects a set of clients from the client set to participate in this training round. ; S3.2: For clients participating in training The server is configured according to its capacity. Extract the corresponding shared projection slice from the maximum shared state: ; The server sends shared projection slices, capacity configuration, activation layer set, current model version, and training epoch information to the client. S3.3: The client receives shared projection slices and builds a multimodal adapter only in the activation layers on the fully frozen visual language model backbone; for modalities... Client In the The adapter branch of the layer is represented as: ; in, and These are the mode-specific lower projection matrix and upper projection matrix, respectively. To share projection slices, It is a non-linear activation function; the mode-specific lower projection matrix and upper projection matrix are kept locally on the client, and the shared projection slice is used for cross-client collaborative aggregation; S3.4: The adapter branch is connected in parallel with the frozen Transformer layer, and its output is represented as: ; in, For frozen Transformer layers, S3.5: The client uses local private data to train the multimodal adapter and obtains the updated shared projection slices and local personalized projection parameters.

[0045] S4: Shared Update Generation and Secure Upload: The client generates a shared projection update based on the local training results, constructs an activation mask, embeds the shared projection update into the maximum shared projection space, and then quantizes, masks, commits, and signs it before uploading it to the server; the specific steps are as follows: S4.1: The client constructs an activation mask based on its adapter rank and set of activation layers; for the first... Coordinates in the layer shared projection matrix The activation mask is defined as: ; in, For the client The shared projected coordinates are actually trained. S4.2: The client calculates the difference in activity shared projection slices before and after local training: ; in, The client completes the local training of the shared projection slices; the client aggregates the weights according to the public data. Generate weighted activity update: ; S4.3: Client-side trimming and quantification of weighted activity updates: ; in, For quantitative scale, To trim the boundaries, For integer modulo; S4.4: For any client participating in training... Both parties generate paired random masks based on a shared seed; the client... Generate masked upload vector: ; in, For the client Uploaded mask update vector; Update the vector for the quantized shared projection; Paired random masks generated between clients; For quantization modulus.

[0046] Pairwise random masks cancel each other out when the server sums the vectors uploaded by all clients; S4.5: Clients generate additive homomorphic commitments to quantization updates: ; in, To commit to the blinding factor; S4.6: The server generates a round structure descriptor based on the set of participating clients in this round, capacity configuration, aggregate weights, shared slice summary, aggregate denominator summary, model version, masking strategy, and quantization parameters. The client generates a signature for the commitment and round structure descriptor: ; S4.7: The client uploads the mask update, commitment, signature, and corresponding structure verification information to the server.

[0047] S5: Secure Aggregation and Structure Verification: The server sums the mask updates uploaded by the client to obtain the aggregated update result and generates a coordinate-level aggregate denominator. The client or verifier verifies the consistency of the aggregated result, commitment, signature, and round structure descriptor. The specific steps are as follows: S5.1: The server receives the mask updates uploaded by participating clients and sums them: ; Since the paired random masks cancel each other out, the aggregation result satisfies: ; in, The server receives the mask update summation result from all participating clients' uploads; Let be the set of clients participating in the training round t; S5.2: The server calculates the effective aggregation denominator at the coordinate level based on the aggregation weight and activation mask of each client: ; For any shared coordinate , This only includes the client-side aggregated weights used in actual training at that coordinate; S5.3: The server sends the aggregated quantization update, aggregated commitment blinding value, aggregated denominator summary, and each client's commitment and signature to the client or verifier. The specific steps for the client or verifier are as follows: S5.3.1: The client or verifier verifies whether the aggregated quantized update matches the sum of the updates committed by the client using the following formula: ; in, S5.3.2: The client or verifier verifies the signature of each client and checks whether the set of participating clients, capacity configuration, aggregation weight, activation mask, aggregation denominator, model version, shared slice summary, masking strategy, and quantization parameters are consistent with the round structure descriptor; S5.3.3: If both the generational verification and structural consistency verification pass, the aggregation result of this round is accepted; otherwise, the aggregation result of this round is rejected, and the training of this round is terminated or re-executed. S6: Model update: When the verification passes, the server updates the shared projection parameters according to the coordinate-level nested aggregation rules and uses the updated shared projection parameters for the next round of training; when the verification fails, the aggregation result of this round is rejected, and the training of this round is terminated or re-executed. The specific steps are as follows: S6.1: The server performs signed decoding and dequantization on the aggregate quantization update that has passed the verification to obtain the shared projection aggregation numerator. S6.2: For any coordinate after shared projection vectorization The server updates the maximum shared state according to the following rules: ; Among them, when When no client trains the coordinates in this round, the server keeps the coordinates unchanged; when At this time, the server only uses the client updates that were actually trained on that coordinate for weighted averaging; S6.3: The server uses the updated maximum shared state for the next round of training and sends the latest shared projection slice corresponding to its capacity level to subsequent participating clients; the client continues to retain the local modality-specific adapter parameters and does not upload them to the server.

[0048] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0049] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A personalized federated learning method based on resource-aware nested multimodal adapters, characterized in that, Includes the following steps: S1: Initialization: The server initializes the pre-trained visual language model backbone, maximum shared projection space, client set, client resource information, adapter capacity hierarchy set, and security aggregation related parameters; S2: Resource assessment and capacity allocation: The server allocates the corresponding adapter rank and adapter insertion layer to the client based on the client's computing power, communication bandwidth, and storage capacity; S3: Slice Distribution and Local Training: The server sends a shared projection slice corresponding to the capacity level to the client selected for training. The client trains the multimodal adapter locally using private data and retains the local personalized adapter parameters. S4: Shared Update Generation and Secure Upload: The client generates a shared projection update based on the local training results, constructs an activation mask, embeds the shared projection update into the maximum shared projection space, and then quantizes, masks, commits, and signs it before uploading it to the server. S5: Secure Aggregation and Structure Verification: The server sums the mask updates uploaded by the client to obtain the aggregated update result and generates a coordinate-level aggregate denominator. The client or verifier verifies the consistency of the aggregated result, commitment, signature, and round structure descriptor. S6: Model Update: When validation passes, the server updates the shared projection parameters according to the coordinate-level nested aggregation rules and uses the updated shared projection parameters for the next round of training; when validation fails, the aggregation result of this round is rejected, and the training is terminated or re-executed.

2. The personalized federated learning method based on resource-aware nested multimodal adapters according to claim 1, characterized in that, The initialization process in S1 includes the following steps: S1.1: The server initializes a pre-trained visual language model containing an image encoder and a text encoder, and freezes the backbone parameters of the image encoder and the text encoder during training; S1.2: The server determines the maximum adapter rank. and maximum number of adapter insertion layers Initialize the maximum shared projection space; S1.3: The server initializes the client set and each client's local private dataset, and records the client's computing power, communication power and storage power; S1.4: The server sets an ordered set of adapter capacity levels.

3. The personalized federated learning method based on resource-aware nested multimodal adapters according to claim 1, characterized in that, The resource assessment and capacity allocation process in S2 includes the following steps: S2.1: The server pre-measures the computational, communication, and storage overhead of each candidate capacity level and provides this information to the client. Construct a set of feasible capacity levels; S2.2: The server selects the highest capacity level that the client can support from the set of feasible capacity levels; when the client cannot meet the resource requirements of the lowest capacity level, the server does not select the client to participate in this round of training. S2.3: The server inserts layers based on the client's adapter. Determine the set of active layers.

4. The personalized federated learning method based on resource-aware nested multimodal adapters according to claim 1, characterized in that, The slice distribution and local training process in S3 includes the following steps: S3.1: In the first At the start of a training round, the server selects a set of clients from the client set to participate in this training round. ; S3.2: For clients participating in training The server is configured according to its capacity. Extract the corresponding shared projection slice from the maximum shared state; The server sends shared projection slices, capacity configuration, activation layer set, current model version, and training epoch information to the client. S3.3: The client receives the shared projection slice and builds the multimodal adapter only in the activation layer on the fully frozen visual language model backbone; S3.4: The adapter branch connects in parallel with the frozen Transformer layer; S3.5: The client uses local private data to train a multimodal adapter and obtains updated shared projection slices and local personalized projection parameters.

5. The personalized federated learning method based on resource-aware nested multimodal adapters according to claim 4, characterized in that, In S3.3, the modality-specific lower projection matrix and upper projection matrix are retained locally on the client, and the shared projection slice is used for cross-client collaborative aggregation.

6. The personalized federated learning method based on resource-aware nested multimodal adapters according to claim 1, characterized in that, The specific steps of the shared update generation and secure upload process in S4 are as follows: S4.1: The client constructs an activation mask based on its adapter rank and set of activation layers; S4.2: The client calculates the difference in activity shared projection slices before and after local training, and the client aggregates the weights based on the public data. Generate weighted activity updates; S4.3: The client performs trimming and quantification of weighted activity updates; S4.4: For any client participating in training... Both parties generate paired random masks based on the shared seed; S4.5: The client generates an additive homomorphic commitment to the quantized update; S4.6: The server generates a round structure descriptor based on the set of participating clients, capacity configuration, aggregate weights, shared slice summary, aggregate denominator summary, model version, masking strategy, and quantization parameters for this round. The client generates a signature for the commitment and round structure descriptor; S4.7: The client uploads the mask update, commitment, signature, and corresponding structure verification information to the server.

7. The personalized federated learning method based on resource-aware nested multimodal adapters according to claim 6, characterized in that, In S4.4, the paired random masks cancel each other out when the server sums the vectors uploaded by all clients.

8. The personalized federated learning method based on resource-aware nested multimodal adapters according to claim 1, characterized in that, The specific steps of the secure aggregation and structural verification process in S5 are as follows: S5.1: The server receives the mask updates uploaded by participating clients and sums them up; S5.2: The server calculates the effective aggregation denominator at the coordinate level based on the aggregation weight and activation mask of each client; S5.3: The server sends the aggregate quantization update, the aggregate commitment blinding value, the aggregate denominator digest, and the commitments and signatures of each client to the client or the verifier.

9. A personalized federated learning method based on resource-aware nested multimodal adapters according to claim 8, characterized in that, The specific steps for the client or authenticator in S5.3 are as follows: S5.3.1: The client or verifier verifies whether the aggregated quantized update is consistent with the sum of updates committed by the client using the following formula; S5.3.2: The client or verifier verifies the signature of each client and checks whether the set of participating clients, capacity configuration, aggregate weight, activation mask, aggregate denominator, model version, shared slice digest, masking strategy and quantization parameters are consistent with the round structure descriptor; S5.3.3: If both the contemporary number verification and the structural consistency verification pass, accept the aggregation result of this round; otherwise, reject the aggregation result of this round and terminate or re-execute this round of training.

10. A personalized federated learning method based on resource-aware nested multimodal adapters according to claim 1, characterized in that, The model update process in S6 consists of the following steps: S6.1: The server performs signed decoding and dequantization on the verified aggregate quantization update to obtain the shared projection aggregate molecule. ; S6.2: For any coordinate after shared projection vectorization The server updates the maximum shared state according to the following rules: ； Among them, when When no client trains the coordinates in this round, the server keeps the coordinates unchanged; when At that time, the server only uses the client updates that were actually trained on that coordinate for weighted averaging; S6.3: The server will use the updated maximum shared state for the next round of training and send the latest shared projection slices corresponding to its capacity level to subsequent participating clients; the clients will continue to retain the local modality-specific adapter parameters and will not upload them to the server.

Images