Personalized visual prompt coordination federated learning method based on dual-path contrastive learning

Abstract

The application relates to the technical field of federated learning, in particular to a personalized visual prompt coordination federated learning method based on a double-path contrast learning, which comprises the following steps: S1, a main server module issues an aggregation instruction; S2, each client module sends a scrambled local model parameter to the main server module after ending training processing; S3, the main server module performs interference processing and first restoration processing; S4, a slave server module performs second restoration processing, obtains a process result through calculation, and performs aggregation processing; S5, the main server module sends each scrambled final aggregation result to a corresponding client module; and S6, each client module receiving the scrambled final aggregation result updates a local model parameter, performs training processing, and jumps to the step S2. The application provides a lightweight, personalized and structure-maintained federated visual prompt learning method, which can balance communication, precision and security.

Description

Technical Field

[0001] This application relates to the field of federated learning technology, and more particularly to a personalized visual cue coordinated federated learning method based on dual-path contrastive learning. Background Technology

[0002] With the widespread application of multimodal large models in visual recognition tasks, more and more terminal devices hope to leverage the powerful generalization capabilities of large models to achieve visual recognition functions. In actual deployments, images collected by terminal devices often contain user privacy information and cannot be directly uploaded to the cloud for centralized training. At the same time, the cost of uploading large amounts of model parameters or image data to terminal devices is high.

[0003] Federated learning enables collaborative training among multiple terminal devices without sharing raw data. Cue learning has also demonstrated advantages such as fewer parameters and faster transfer learning in centralized scenarios. However, most research on federated cue learning remains focused on sharing cue information, neglecting the individuality of local data and rarely addressing the security issues of the entire learning system. To address these technical problems, this invention proposes a personalized visual cue coordinated federated learning method based on dual-path contrastive learning. Summary of the Invention

[0004] This application separates the storage and use of the client module's private encrypted values, and uses scrambling and interference processing to ensure that data is always transmitted in encrypted form. Furthermore, this application introduces a dual-path system of global and local branches in the local model, and incorporates dual contrastive loss in the feature space. This application aims to provide a lightweight, personalized, and structure-preserving federated visual cue learning method that achieves a balance between communication, accuracy, and security.

[0005] This application provides a personalized visual cue coordinated federated learning method based on dual-path contrastive learning, including the following steps:

[0006] S1. The main server module sends an aggregation instruction to several client modules among all the client modules. Each client module that receives the aggregation instruction performs training processing on its local model.

[0007] S2. After receiving the aggregation instruction, each client module uses its own common key value to scramble the local model parameters and sends the scrambled local model parameters to the main server module.

[0008] S3. For each scrambled local model parameter received, the master server module performs interference processing and first restoration processing, and sends the interference result and the first restoration result to the slave server module.

[0009] S4. The server module performs a second restoration process on each received interference result, calculates the process result by combining the received first restoration result, and performs aggregation processing on all process results, sending each scrambled preliminary aggregation result to the main server module.

[0010] S5. The main server module obtains the final aggregation result after scrambling from each preliminary aggregation result and sends the final aggregation result after scrambling to the corresponding client module.

[0011] S6. Upon receiving the corresponding scrambling final aggregation result, each client module updates its local model parameters and performs training processing on the local model when it receives the aggregation instruction from the main server module, then jumps to S2.

[0012] In a preferred embodiment, the client module performs training processing on the local model, including the following steps:

[0013] S11. Obtain the image category label set and the semantic description set corresponding to each image category label, and perform serialization encoding on each semantic description to obtain a fixed-length label sequence, and construct a text label tensor;

[0014] S12. The CLIP text encoder with frozen input parameters of the text tag tensor is forward propagated to obtain the text feature tensor, and the text feature tensor is copied and expanded along the batch dimension to obtain the batch-level text feature tensor.

[0015] S13. Feed the batch-level text feature tensors into the trainable global linear mapping layer and the local linear mapping layer respectively to obtain the batch-level global feature tensor and the batch-level personalized feature tensor respectively. Use the trainable fusion coefficients to perform a weighted sum of the two feature tensors to obtain the batch-level fused feature tensor.

[0016] S14. Use the conv1 layer of the parameter-frozen CLIP image encoder as a feature extractor to process the batch-level image data tensor to obtain the batch-level image feature tensor. Perform serialization and position information embedding processing on the batch-level image feature tensor, and concatenate the learnable category token vector with the batch-level image feature tensor.

[0017] S15. Remove the category token vector from the batch-level image feature tensor, use the batch-level image feature tensor as a query, use the batch-level fused feature tensor as a key and value, and perform cross-attention calculation to obtain the semantically enhanced batch-level image feature tensor.

[0018] S16. Adaptive pooling is performed on the semantically enhanced batch-level image feature tensor to generate a preset number of visual cue tokens. The semantically enhanced batch-level image feature tensor and the visual cue token sequence are concatenated, and the category token vector is further concatenated to obtain the final batch-level image feature tensor.

[0019] S17. Perform forward computation on the Vit Transformer encoder with frozen input parameters of the final batch-level image feature tensor, extract the visual semantic tensor from the output batch-level image feature tensor, and project the visual semantic tensor onto the target feature dimension through a linear layer.

[0020] S18. Calculate the initial matching score matrix based on the visual semantic tensor and the text tag tensor. In the initial matching score matrix, aggregate and calculate several initial matching scores corresponding to each category to obtain the category matching score matrix.

[0021] S19. Calculate the classification loss, visual consistency loss, feature-level contrast loss, and sample-level contrast loss, and calculate the total loss. Update the local model parameters through the backpropagation algorithm, and jump to S13 before the training termination condition is met.

[0022] Compared with the prior art, the beneficial effects of this application are at least as follows:

[0023] 1. The separate storage and use of private passwords for client modules ensures that even if any server module is illegally attacked, the client module's private data cannot be obtained, avoiding single point of leakage. Through scrambling and interference processing, all transmitted data is in encrypted form, so even if illegally intercepted, no useful data can be obtained. In short, the aggregation of all local model parameters can be completed securely without leaking the private data of any client module, thus improving the overall security of the federated learning system.

[0024] 2. A dual-path structure for the prompt generator has been added: a global branch and a local branch. The global branch processes text features through a global linear layer to generate a global context, used to capture semantic information shared across client modules; the local branch processes text features through personalized linear layers and a cross-attention mechanism to generate a personalized context, adapted to the specific data distribution of the client module. The outputs of the two branches are weighted and fused to form the final context representation. This dual-path structure innovatively achieves a balance between "global knowledge sharing" and "local data adaptability," effectively solving the data heterogeneity problem in federated learning, and significantly improving the model's performance on client modules without sacrificing global knowledge learning.

[0025] 3. A feature-level contrastive learning mechanism integrating global and local branch outputs effectively addresses the negative transfer problem caused by data distribution differences in federated learning by establishing cross-domain alignment relationships at the feature space level. This mechanism constructs contrastive learning constraints between common semantic features captured by the global branch and client-specific features extracted by the local branch, enabling the model to automatically identify and strengthen domain-invariant features in cross-domain scenarios while suppressing domain-specific noise. This feature-level alignment not only significantly improves the model's generalization ability in the target domain but also enhances fault diagnosis accuracy in small sample scenarios, avoiding the performance degradation caused by domain differences in traditional methods.

[0026] 4. The sample-level contrastive learning mechanism constructs positive and negative sample pairs within a batch and uses cosine similarity to measure the relationship between samples. This makes the feature representations of samples of the same category more tightly aggregated and the feature representations of samples of different categories more significantly separated. This mechanism requires no additional labeled data and relies solely on sample pairs generated by data augmentation techniques, effectively solving the problem of insufficient feature representation in small-sample scenarios. By incorporating sample-level contrastive loss into the total loss function, the model can automatically learn more discriminative feature representations, significantly improving the model's image classification accuracy and generalization ability in complex backgrounds. This provides a highly robust and adaptable solution for small-sample image classification in federated learning environments. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the personalized visual cue coordinated federated learning method based on dual-path contrastive learning in this application;

[0029] Figure 2 This is a schematic diagram of the local model in this application. Detailed Implementation

[0030] This application provides a personalized visual cue coordinated federated learning method based on dual-path contrastive learning. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 The personalized visual cue coordinated federated learning method based on dual-path contrastive learning in this application embodiment includes the following main steps:

[0032] S1. The main server module sends an aggregation instruction to several client modules among all the client modules. Each client module that receives the aggregation instruction performs training processing on its local model.

[0033] S2. After receiving the aggregation instruction, each client module uses its own common key value to scramble the local model parameters and sends the scrambled local model parameters to the main server module.

[0034] S3. For each scrambled local model parameter received, the master server module performs interference processing and first restoration processing, and sends the interference result and the first restoration result to the slave server module.

[0035] S4. The server module performs a second restoration process on each received interference result, calculates the process result by combining the received first restoration result, and performs aggregation processing on all process results, sending each scrambled preliminary aggregation result to the main server module.

[0036] S5. The main server module obtains the final aggregation result after scrambling from each preliminary aggregation result and sends the final aggregation result after scrambling to the corresponding client module.

[0037] S6. Upon receiving the corresponding scrambled final aggregation result, each client module updates its local model parameters and enters a waiting state.

[0038] Specifically, considering the technical problems of existing technologies: 1) Shared hints are forced to compromise when aggregating server modules, leading to a decrease in local accuracy; 2) There is a lack of explicit local-global knowledge alignment mechanisms, and global hints are prone to drift on local data; 3) The semantic structure between local samples is not fully explored, and the model does not make sufficient use of intra-class compactness and inter-class separation; 4) There is a lack of consideration for the security of the entire learning system. To solve these technical problems, steps S1 to S6 are mainly proposed.

[0039] In step S1, the main server module sends aggregation instructions to several client modules among all client modules. The main server module specifically uses an average aggregation method. Each client module receiving the aggregation instruction performs training processing on its local model, which will be described in detail below. In step S2, after completing the training processing, each client module receiving the aggregation instruction uses its own public key value to scramble the local model parameters and sends the scrambled local model parameters to the main server module. The local model parameters are specifically in vector form. It should be noted that each client module in the system possesses its own public and private key values, specifically obtained through the Distributed Key Generation (DKG) protocol based on the CKKS scheme supporting threshold decryption. The public key value is pre-broadcast to other modules, allowing them to scramble the data to be sent. Only the module possessing the private key value can perform the decryption process. The scrambling process specifically uses the CKKS (Cheon-Kim-Kim-Song) homomorphic encryption algorithm for encryption. In step S3, the master server module performs interference processing and a first restoration processing on each received scrambled local model parameter, and then sends the interference result and the first restoration result to the slave server module. The interference processing and the first restoration processing will be described in detail below. In step S4, for each received interference result, the slave server module performs a second restoration processing to obtain a second restoration result, calculates the process result by combining it with the received paired first restoration results, and then performs aggregation processing on all process results to obtain a preliminary aggregation result. It also generates preliminary aggregation results for each scrambled parameter and sends each preliminary aggregation result to the master server module. Step S4 will be described in detail below. In step S5, the master server module obtains the final aggregation results for each scrambled parameter from the preliminary aggregation results (the process of obtaining these results will be described in detail below), and sends each final aggregation result to the corresponding client module. The corresponding client module refers to the multiple client modules that sent the scrambled local model parameters to the master server module in step S2. In step S6, each client module that receives the corresponding scrambled final aggregation result first uses its own private key to restore the received scrambled final aggregation result to obtain the corresponding final aggregation result. Then, it updates its local model parameters according to the corresponding final aggregation result and enters a waiting state, that is, waiting to receive the aggregation instruction again. It should be noted that after step S6, execution generally jumps to step S1 to continue, unless the execution count threshold is reached. The execution count threshold is set according to the actual application scenario.

[0040] Furthermore, the client module performs training processing on the local model, including the following steps:

[0041] S11. Obtain the image category label set and the semantic description set corresponding to each image category label, and perform serialization encoding on each semantic description to obtain a fixed-length label sequence, and construct a text label tensor;

[0042] S12. The CLIP text encoder with frozen input parameters of the text tag tensor is forward propagated to obtain the text feature tensor, and the text feature tensor is copied and expanded along the batch dimension to obtain the batch-level text feature tensor.

[0043] S13. Feed the batch-level text feature tensors into the trainable global linear mapping layer and the local linear mapping layer respectively to obtain the batch-level global feature tensor and the batch-level personalized feature tensor respectively. Use the trainable fusion coefficients to perform a weighted sum of the two feature tensors to obtain the batch-level fused feature tensor.

[0044] S14. Use the conv1 layer of the parameter-frozen CLIP image encoder as a feature extractor to process the batch-level image data tensor to obtain the batch-level image feature tensor. Perform serialization and position information embedding processing on the batch-level image feature tensor, and concatenate the learnable category token vector with the batch-level image feature tensor.

[0045] S15. Remove the category token vector from the batch-level image feature tensor, use the batch-level image feature tensor as a query, and use the batch-level fused feature tensor obtained in step S13 as a key and value to perform cross-attention calculation to obtain the semantically enhanced batch-level image feature tensor.

[0046] S16. Adaptive pooling is performed on the semantically enhanced batch-level image feature tensor to generate a preset number of visual cue tokens. The semantically enhanced batch-level image feature tensor and the visual cue token sequence are concatenated, and the category token vector is further concatenated to obtain the final batch-level image feature tensor.

[0047] S17. Perform forward computation on the Vit Transformer encoder with frozen input parameters of the final batch-level image feature tensor, extract the visual semantic tensor from the output batch-level image feature tensor, and project the visual semantic tensor onto the target feature dimension through a linear layer.

[0048] S18. Calculate the initial matching score matrix based on the visual semantic tensor and the text tag tensor. In the initial matching score matrix, aggregate and calculate several initial matching scores corresponding to each category to obtain the category matching score matrix.

[0049] S19. Calculate the classification loss, visual consistency loss, feature-level contrast loss, and sample-level contrast loss, and calculate the total loss. Update the local model parameters through the backpropagation algorithm, and jump to S13 before the training termination condition is met.

[0050] Specifically, see, for example Figure 2 As shown, in step S11, an image category label set and a semantic description set corresponding to each image category label are obtained. For example, the image category label set has 20 image category labels, and the semantic description set of each image category label has 5 semantic descriptions. Serialization encoding is performed on each semantic description to obtain a fixed-length label sequence to construct a text label tensor. The shape of the text label tensor is, for example, [100, 77]. In step S12, the text tag tensor is input into the pre-trained CLIP text encoder with frozen parameters, i.e., the CLIP (ViT-L / 14) model text encoder, for forward propagation. Specifically, through a self-attention mechanism and a multi-layer feedforward network, each semantic description is mapped to a high-dimensional continuous vector space to obtain a text feature tensor. The shape of the text feature tensor is, for example, [100, 768], where 768 is the feature dimension, defined by the CLIP text encoder architecture. The text feature tensor is also copied and expanded along the batch dimension to obtain a batch-level text feature tensor with a shape, for example, [128, 100, 768], in order to match the number of image samples in the current batch, where 128 is the number of image samples in the same batch. In step S13, the batch-level text feature tensors are fed into a trainable global linear mapping layer and a local linear mapping layer for processing, respectively, to obtain a batch-level global feature tensor with a shape of, for example, [128, 100, 768], and a batch-level personalized feature tensor with a shape of, for example, [128, 100, 768]. This step provides a new dual-branch prompting learning mechanism, which includes two parallel learnable paths with the same structure. The global path aims to extract general semantic patterns shared across client modules, while the local path aims to capture specific semantic representations adapted to local data distribution. In order to dynamically balance global universality and local adaptability, a batch-level fused feature tensor is obtained by weighted summation of the two feature tensors through trainable fusion coefficients. The initial value of the fusion coefficients can be 0.5.

[0051] In step S14, the conv1 layer of the pre-trained CLIP image encoder (ViT-L / 14) with frozen parameters is used as a feature extractor to process batch-level image data tensors with shapes such as [128, 3, 224, 224], where 3 represents the number of color channels and the two 224 represent the image height and width, respectively. Specifically, the image is evenly divided into multiple regular image blocks, and a linear projection is performed on each image block to generate a corresponding feature vector, thereby obtaining a batch-level image feature tensor with shapes such as [128, 1024, 14, 14], where 1024 is the feature dimension and the two 14 represent the height and width of the image block, respectively. For the batch-level image feature tensor... Serialization and location information embedding are performed. Specifically, the spatial dimensions (height and width of the image patch) are first flattened, resulting in a batch-level image feature tensor with the shape [128, 1024, 196]. The feature dimension and sequence dimension are swapped, resulting in a batch-level image feature tensor with the shape [128, 196, 1024]. A learnable parameter vector, or learnable class token vector, with the shape [128, 1, 1024] is generated. Initially, it is randomly assigned a value, and during training, it is optimized and adjusted through gradient descent. The learnable class token vector is then concatenated with the batch-level image feature tensor, resulting in a batch-level image feature tensor with the shape [128, 197, 1024]. In step S15, the category token vector is first removed from the batch-level image feature tensor, and the shape of the obtained batch-level image feature tensor is restored to [128, 196, 1024] for example. Then, the batch-level image feature tensor is used as a query, and the batch-level fused feature tensor obtained in step S13 is used as a key and value to perform cross-attention calculation, so as to obtain a semantically enhanced batch-level image feature tensor with a shape of [128, 196, 1024] for example. In step S16, adaptive pooling is performed on the semantically enhanced batch-level image feature tensor to generate a preset number of visual cue tokens carrying key visual information, with a shape such as [128, 4, 1024]. The semantically enhanced batch-level image feature tensor and the visual cue token sequence are concatenated, and the shape of the concatenated batch-level image feature tensor is [128, 200, 1024]. The above-mentioned category token vector and the concatenated batch-level image feature tensor are then concatenated to obtain the final batch-level image feature tensor with a shape such as [128, 201, 1024].In step S17, the final batch-level image feature tensor is input into the pre-trained parameter-frozen Vit Transformer encoder for forward computation. The shape of the output batch-level image feature tensor is, for example, [128, 201, 1024]. A visual semantic tensor is extracted from the output batch-level image feature tensor. Specifically, the feature vector at the starting position is extracted as the visual semantic tensor, which is a highly generalized semantic representation of the entire image. Its shape is, for example, [128, 1024]. In order to align with the text feature space, the visual semantic tensor is projected onto the target feature dimension through a linear layer. The shape of the obtained semantic tensor is [128, 768]. The target feature dimension is the text feature dimension.

[0052] In step S18, an initial matching score matrix is ​​calculated based on the visual semantic tensor and the text tag tensor. Specifically, the cosine similarity between each row of the visual semantic tensor and each row of the text tag tensor is calculated, and each cosine similarity is multiplied by a learnable scaling factor scalar. The shape of the initial matching score matrix is, for example, [128, 100]. In the initial matching score matrix, several initial matching scores corresponding to each category are aggregated and calculated to obtain a category matching score matrix. The shape of the category matching score matrix is, for example, [128, 20], which represents the matching score of each image sample in the same batch with each image category. Specifically, the average value of multiple initial matching scores of the same image category is calculated in each row of the initial matching score matrix. In step S19, the classification loss, visual consistency loss, feature-level contrast loss, and sample-level contrast loss are calculated. The calculation of each loss will be described in detail below, and the total loss is calculated based on each loss. Specifically, corresponding weights are assigned to each loss: the weight of the classification loss is set to 1, the weight of the visual consistency loss to 0.5, the weight of the feature-level contrast loss to 0.1, and the weight of the sample-level contrast loss to 0.1. The local model parameters are updated using the backpropagation algorithm. It should be noted that the updated local model parameters include the global linear mapping layer parameters, the local linear mapping layer parameters, the cross-attention module parameters, the fusion coefficients, the class token vector, and the scaling factor scalar. However, the model parameters uploaded to the main service only include the global linear mapping layer parameters and the cross-attention module parameters. Before the training termination condition is met, execution jumps to step S13 to continue. The termination condition can be completing a preset number of epochs, which is set according to the actual application scenario.

[0053] Furthermore, calculating the classification loss includes: in the class matching score matrix, using the Softmax function to convert all matching scores in each row into a probability distribution, obtaining the true class label of the sample corresponding to each row, calculating the negative natural logarithm of the matching probability corresponding to the true class label to obtain the sample loss, and calculating the average of all sample losses as the classification loss.

[0054] Specifically, this section explains how to calculate the classification loss. First, the softmax function is used to convert all matching scores in each row of the class matching score matrix into a probability distribution. Second, the true class label of the image sample corresponding to each row is obtained, and the negative natural logarithm of the matching probability corresponding to the true class label is calculated to obtain the sample loss. Finally, the average of all sample losses is used as the classification loss. The classification loss is the core supervision signal, which is directly used to measure the difference between the classification prediction and the true class label.

[0055] Furthermore, the calculation of visual consistency loss includes: performing enhancement processing on images in the same batch, performing forward computation on the Transformer encoder of the batch-level image data tensor with frozen Vit input parameters, extracting the reference visual tensor from the output batch-level image feature tensor, projecting the reference visual tensor onto the target feature dimension through a linear layer, and performing L2 normalization processing on the reference visual tensor and visual semantic tensor, calculating the cosine similarity between each row in the visual semantic tensor and the corresponding row in the reference visual tensor, subtracting the cosine similarity from 1 as the sample visual loss, and calculating the average of all sample visual losses as the visual consistency loss.

[0056] Specifically, the calculation of visual consistency loss is explained. First, enhancement processing is performed on the images in the same batch, such as random cropping or rotation. Similar to step S17 above, the enhanced batch-level image data tensor is processed using a Transformer encoder with frozen parameters for Vit. A reference visual tensor is extracted based on the output batch-level image feature tensor. The reference visual tensor is projected onto the target feature dimension through a linear layer. Second, L2 normalization is performed on each row of the reference visual tensor and the visual semantic tensor. The cosine similarity between each row in the visual semantic tensor and the corresponding row in the reference visual tensor is calculated. For example, the cosine similarity between the first row in the visual semantic tensor and the first row in the reference visual tensor is calculated. They correspond to the same image sample. 1 minus the cosine similarity is taken as the sample visual loss. Finally, the average of all sample visual losses is taken as the visual consistency loss.

[0057] Furthermore, calculating the feature-level contrastive loss includes the following steps:

[0058] S191. Perform L2 normalization and reshaping on the batch-level global feature tensor and the batch-level personalized feature tensor to obtain the global feature matrix and the personalized feature matrix.

[0059] S192. Treat each row in the global feature matrix as an anchor point, the corresponding row in the personalized feature matrix as a positive sample, and all other rows in the personalized feature matrix as negative samples to calculate the InfoNCE loss value, and calculate the average of all InfoNCE loss values ​​as the first InfoNCE loss.

[0060] S193. Treat each row in the personalized feature matrix as an anchor point, the corresponding row in the global feature matrix as a positive sample, and all other rows in the global feature matrix as negative samples to calculate the InfoNCE loss value, and calculate the average of all InfoNCE loss values ​​as the second InfoNCE loss.

[0061] S194. Calculate the average of the first InfoNCE loss and the second InfoNCE loss as the feature-level contrast loss.

[0062] Specifically, this section explains how to calculate the feature-level contrast loss. First, L2 normalization is performed on each row of the batch-level global feature tensor and the batch-level personalized feature tensor. The normalized batch-level global feature tensor and batch-level personalized feature tensor are then reshaped into two-dimensional global feature matrices and personalized feature matrices, with shapes such as [1280, 768]. Second, each row in the global feature matrix is ​​used as an anchor point, the corresponding row in the personalized feature matrix is ​​used as a positive sample, and all other rows in the personalized feature matrix are used as negative samples to calculate the InfoNCE loss value. For example, the first row in the global feature matrix is ​​used as an anchor point, and the first row in the personalized feature matrix is ​​used as a positive sample, since the corresponding image samples and semantic descriptions are the same. All other rows in the personalized feature matrix are used as negative samples. Based on this, the average of all InfoNCE loss values ​​is used as the first InfoNCE loss. Similarly, we calculate the InfoNCE loss by treating each row in the personalized feature matrix as an anchor point, the corresponding row in the global feature matrix as a positive sample, and all other rows in the global feature matrix as negative samples. Based on this, we use the average of all InfoNCE loss values ​​as the second InfoNCE loss. Finally, we calculate the average of the first and second InfoNCE losses and use this as the feature-level contrastive loss.

[0063] Furthermore, the calculation of the sample-level contrast loss includes: performing L2 normalization on the visual semantic tensor, taking each row of the visual semantic tensor as anchor points, taking all other rows with the same corresponding true class label as positive samples, taking all other rows with different corresponding true class labels as negative samples, calculating the InfoNCE loss value, and calculating the average of all InfoNCE loss values ​​as the sample-level contrast loss.

[0064] Specifically, this section describes how to calculate the sample-level contrastive loss. First, L2 normalization is performed on each row of the visual semantic tensor. Each row is treated as an anchor point. All other rows whose corresponding ground truth labels are the same as their corresponding ground truth labels are considered positive samples, and all other rows whose corresponding ground truth labels are different from their corresponding ground truth labels are considered negative samples. The InfoNCE loss value is calculated accordingly. It's important to note that the InfoNCE loss value is not calculated when there are no positive samples. Second, the average of all InfoNCE loss values ​​is taken as the sample-level contrastive loss.

[0065] The above method is particularly suitable for applications with high precision requirements, such as in the field of flexible production of high-end fabrics. Multiple weaving factories within the same textile alliance often face the dual challenges of collaborative quality inspection and rapid pattern changeover. Although weaving the same series of fabrics, the loom models, yarn sources, and workshop temperature and humidity environments vary significantly between factories. Furthermore, the morphology and location of defects in different batches of fabric are highly random. To address this complex situation, the text-guided branch focuses on fine-grained alignment of professional descriptions such as "weft break," "skipped yarn," and "oil stains" with fabric images, ensuring a one-to-one correspondence between defect semantics and visual features. Meanwhile, the personalized prompt branch effectively captures the unique weave background and lighting textures of each factory through learnable context tokens, allowing the model to retain common knowledge while also considering local style differences. Crucially, a cross-factory knowledge-sharing channel is built by aggregating only global prompt parameters. When a factory encounters a rare defect like "cloud spots" for the first time, its local model can still quickly activate a sensitive response to cloud spots by leveraging the prompt weights uploaded by other factories, without needing to send back any original fabric images. This parameter-level knowledge fusion enables the system to maintain high recall and high consistency in quality inspection even in environments with frequent fabric color changes and unpredictable defect types, significantly reducing brand risks and rework costs caused by defective products leaking out. Furthermore, the entire backbone network (CLIP) model is frozen during training; only a small number of trainable parameters need to be trained and uploaded to achieve superior performance.

[0066] Furthermore, each client module pre-stores its own private password value, and the private password value of each client module is pre-divided into a first share of private password value and a second share of private password value. The first share of private password value is stored in the master server module, and the second share of private password value is stored in the slave server module.

[0067] Specifically, as mentioned above, each client module has its own public and private key values. It should also be noted that the private key value of each client module is pre-split into a first share and a second share. The public key value can be obtained by summing the first and second shares. This is based on the CKKS scheme (T-CKKS) that supports threshold decryption and is implemented through the Distributed Key Generation (DKG) protocol. The first share of the private key value is stored in the master server module, and the second share is stored in the slave server module. The advantage of this is that only by using the first and second shares of the private key values ​​to perform the first and second restoration processes, i.e., partial decryption, respectively, and then using the publicly available combination algorithm, such as a linear combination based on Lagrange interpolation, to combine the first and second restoration results, can the complete restoration result be obtained.

[0068] Furthermore, the main server module performs interference processing and first restoration processing on the scrambled local model parameters, including the following steps:

[0069] S41. Generate an arbitrary numerical sequence, use the common secret value of the client module corresponding to the scrambled local model parameters to scramble the arbitrary numerical sequence to obtain the scrambled arbitrary numerical sequence, and add the scrambled arbitrary numerical sequence to the scrambled local model parameters to obtain the interference result.

[0070] S42. Use the first share private value of the client module corresponding to the scrambled local model parameters to perform the first restoration process on the interference result to obtain the first restoration result.

[0071] Specifically, in step S41, the main server module generates an arbitrary numerical sequence, specifically representing it as a vector. This arbitrary numerical sequence is then scrambled using the common secret value of the client module sending the scrambled local model parameters. As mentioned above, the scrambling process refers to encryption using the CKKS homomorphic encryption algorithm. The CKKS homomorphic encryption algorithm can directly process data in vector form, obtaining a scrambled arbitrary numerical sequence in vector form. This scrambled arbitrary numerical sequence is then homomorphically added to the scrambled local model parameters to obtain the interference result. This can be understood as adding the values ​​at the same positions in two vectors. In step S42, the first share of the private secret value of the client module sending the scrambled local model parameters is used to perform a first restoration process (partial decryption) on the interference result to obtain the first restoration result.

[0072] Furthermore, step S5 above includes the following detailed steps:

[0073] S51. The server module uses the second share private password value of the client module corresponding to the interference result to perform a second restoration process on the interference result, and calculates the process result based on the second restoration result and the first restoration result corresponding to the interference result.

[0074] S52. Calculate the mean of all process results as the preliminary aggregation result, use the common secret value of several corresponding client modules to scramble the preliminary aggregation result respectively, and send each scrambled preliminary aggregation result to the main server module.

[0075] Specifically, in step S51, the server module uses the second share private key value of the client module corresponding to the received interference result to perform a second restoration process, i.e., partial decryption, on the received interference result to obtain a second restoration result. Then, using a publicly available combination algorithm, such as a linear combination based on Lagrange interpolation, the second restoration result is combined with the first restoration result corresponding to the received interference result to generate a process result. The process result is actually the result of homomorphically adding the local model parameters uploaded by the client module corresponding to the received interference result to the corresponding arbitrary numerical sequence. In step S52, the mean of all process results is calculated as the preliminary aggregation result. Since the process result is specifically in vector form, it can be understood as calculating the mean of each value at the same position in all vectors. The preliminary aggregation result is scrambled using the common key value of multiple client modules. Multiple client modules refer to multiple client modules that have sent scrambled local model parameters to the main server module. Subsequently, each scrambled preliminary aggregation result is sent to the main server module.

[0076] Furthermore, the main server module obtains the final aggregation result after scrambling from each preliminary aggregation result after scrambling, including: for each preliminary aggregation result after scrambling, calculating the mean of all arbitrary numerical sequences in this round to obtain the mean sequence, using the common secret value of the client module corresponding to the preliminary aggregation result after scrambling to scramble the mean sequence to obtain the scrambled mean sequence, and subtracting the scrambled mean sequence from the preliminary aggregation result after scrambling.

[0077] Specifically, this section explains how the main server module obtains the final aggregation results after each scrambling step. First, it determines all arbitrary numerical sequences generated by the main server module in this round and calculates the mean of all arbitrary numerical sequences to obtain the mean sequence. Second, for each received preliminary aggregation result after scrambling, it uses the common secret value of the client module corresponding to the received preliminary aggregation result after scrambling to scramble the mean sequence, thereby obtaining the scrambled mean sequence. Finally, it homomorphically subtracts the preliminary aggregation result after scrambling from the scrambled mean sequence to obtain the final aggregation result.

[0078] In summary, this application has at least the following beneficial effects:

[0079] 1. The separate storage and use of private passwords for client modules ensures that even if any server module is illegally attacked, the client module's private data cannot be obtained, avoiding single point of leakage. Through scrambling and interference processing, all transmitted data is in encrypted form, so even if illegally intercepted, no useful data can be obtained. In short, the aggregation of all local model parameters can be completed securely without leaking the private data of any client module, thus improving the overall security of the federated learning system.

[0080] 2. A dual-path structure for the prompt generator has been added: a global branch and a local branch. The global branch processes text features through a global linear layer to generate a global context, used to capture semantic information shared across client modules; the local branch processes text features through personalized linear layers and a cross-attention mechanism to generate a personalized context, adapted to the specific data distribution of the client module. The outputs of the two branches are weighted and fused to form the final context representation. This dual-path structure innovatively achieves a balance between "global knowledge sharing" and "local data adaptability," effectively solving the data heterogeneity problem in federated learning, and significantly improving the model's performance on client modules without sacrificing global knowledge learning.

[0081] 3. A feature-level contrastive learning mechanism integrating global and local branch outputs effectively addresses the negative transfer problem caused by data distribution differences in federated learning by establishing cross-domain alignment relationships at the feature space level. This mechanism constructs contrastive learning constraints between common semantic features captured by the global branch and client-specific features extracted by the local branch, enabling the model to automatically identify and strengthen domain-invariant features in cross-domain scenarios while suppressing domain-specific noise. This feature-level alignment not only significantly improves the model's generalization ability in the target domain but also enhances fault diagnosis accuracy in small sample scenarios, avoiding the performance degradation caused by domain differences in traditional methods.

[0082] 4. The sample-level contrastive learning mechanism constructs positive and negative sample pairs within a batch and uses cosine similarity to measure the relationship between samples. This makes the feature representations of samples of the same category more tightly aggregated and the feature representations of samples of different categories more significantly separated. This mechanism requires no additional labeled data and relies solely on sample pairs generated by data augmentation techniques, effectively solving the problem of insufficient feature representation in small-sample scenarios. By incorporating sample-level contrastive loss into the total loss function, the model can automatically learn more discriminative feature representations, significantly improving the model's image classification accuracy and generalization ability in complex backgrounds. This provides a highly robust and adaptable solution for small-sample image classification in federated learning environments.

[0083] Furthermore, to verify the performance of the proposed method, its performance on categories seen during training and its generalization ability on new categories were tested, i.e., base-to-new. Each dataset was divided into base and new categories. During training, only data from the base category was used, while the accuracy of the new category was evaluated during testing. Hit-means (HM) is a more robust metric; if the base and new accuracies differ significantly, HM will be much lower than the ordinary arithmetic mean, better reflecting the model's true generalization ability. The HM calculation method is as follows: .

[0084] Thirty client modules were tested on nine datasets: Caltech101, Flowers102, FGVCAircraft, UCF101, OxfordPets, Food101, DTD, StanfordCars, and SUN397. The experimental metrics are shown in the table below.

[0085] Table 1

[0086]

[0087] As shown in the table above, the method has higher accuracy than other methods on the base-to-new task. HM also indicates that the method in this application has stronger generalization ability on the base-to-new generalization task.

[0088] The performance of the proposed method on the Cross-Dataset Generalization task was also tested. The Cross-Dataset Generalization task involves training on a single dataset and testing on multiple datasets. Specifically, the method was trained on ImageNet and its generalization performance was tested on the Caltech101, Flowers102, FGVCAircraft, UCF101, OxfordPets, Food101, DTD, StanfordCars, SUN397, and EuroSAT datasets. Experimental metrics are shown in the table below.

[0089] Table 2

[0090]

[0091] As shown in the table above, Cross-Dataset Generalization tasks still show better performance than other methods, with varying degrees of performance improvement in both the source and target domains.

[0092] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0093] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0094] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0095] It should be noted that all information or data obtained and processed in this application are carried out in compliance with relevant national data protection laws and regulations and with authorization from the owner of the relevant device.

Claims

1. A personalized visual prompt coordination federated learning method based on double-path contrastive learning, characterized in that, The method includes the following steps: S1. The main server module sends an aggregation instruction to several client modules among all the client modules. Each client module that receives the aggregation instruction performs training processing on its local model. S2. After receiving the aggregation instruction, each client module uses its own common key value to scramble the local model parameters and sends the scrambled local model parameters to the main server module. S3. For each scrambled local model parameter received, the master server module performs interference processing and first restoration processing, and sends the interference result and the first restoration result to the slave server module. S4. The server module performs a second restoration process on each received interference result, calculates the process result by combining the received first restoration result, and performs aggregation processing on all process results, sending each scrambled preliminary aggregation result to the main server module. S5. The main server module obtains the final aggregation result after scrambling from each preliminary aggregation result and sends the final aggregation result after scrambling to the corresponding client module. S6. Upon receiving the corresponding scrambling final aggregation result, each client module updates its local model parameters and performs training processing on the local model when it receives the aggregation instruction from the main server module, then jumps to S2.

2. The method of claim 1, wherein, The client module performs training processing on the local model, including the following steps: S11. Obtain the image category label set and the semantic description set corresponding to each image category label, and perform serialization encoding on each semantic description to obtain a fixed-length label sequence, and construct a text label tensor; S12. The CLIP text encoder with frozen input parameters of the text tag tensor is forward propagated to obtain the text feature tensor, and the text feature tensor is copied and expanded along the batch dimension to obtain the batch-level text feature tensor. S13. Feed the batch-level text feature tensors into the trainable global linear mapping layer and the local linear mapping layer respectively to obtain the batch-level global feature tensor and the batch-level personalized feature tensor respectively. Use the trainable fusion coefficients to perform a weighted sum of the two feature tensors to obtain the batch-level fused feature tensor. S14. Use the conv1 layer of the parameter-frozen CLIP image encoder as a feature extractor to process the batch-level image data tensor to obtain the batch-level image feature tensor. Perform serialization and position information embedding processing on the batch-level image feature tensor, and concatenate the learnable category token vector with the batch-level image feature tensor. S15. Remove the category token vector from the batch-level image feature tensor, use the batch-level image feature tensor as a query, use the batch-level fused feature tensor as a key and value, and perform cross-attention calculation to obtain the semantically enhanced batch-level image feature tensor. S16. Adaptive pooling is performed on the semantically enhanced batch-level image feature tensor to generate a preset number of visual cue tokens. The semantically enhanced batch-level image feature tensor and the visual cue token sequence are concatenated, and the category token vector is further concatenated to obtain the final batch-level image feature tensor. S17. Perform forward computation on the Vit Transformer encoder with frozen input parameters of the final batch-level image feature tensor, extract the visual semantic tensor from the output batch-level image feature tensor, and project the visual semantic tensor onto the target feature dimension through a linear layer. S18. Calculate the initial matching score matrix based on the visual semantic tensor and the text tag tensor. In the initial matching score matrix, aggregate and calculate several initial matching scores corresponding to each category to obtain the category matching score matrix. S19. Calculate the classification loss, visual consistency loss, feature-level contrast loss, and sample-level contrast loss, and calculate the total loss. Update the local model parameters through the backpropagation algorithm, and jump to S13 before the training termination condition is met.

3. The method according to claim 2, characterized in that, Calculating the classification loss involves: using the Softmax function to convert all matching scores in each row of the class matching score matrix into a probability distribution, obtaining the true class label of the sample corresponding to each row, calculating the negative natural logarithm of the matching probability corresponding to the true class label to obtain the sample loss, and calculating the average of all sample losses as the classification loss.

4. The method according to claim 2, characterized in that, The calculation of visual consistency loss includes: performing enhancement processing on images in the same batch; performing forward computation on the Transformer encoder of the batch-level image data tensor with frozen Vit input parameters; extracting the reference visual tensor from the output batch-level image feature tensor; projecting the reference visual tensor onto the target feature dimension through a linear layer; performing L2 normalization processing on the reference visual tensor and visual semantic tensor; calculating the cosine similarity between each row in the visual semantic tensor and the corresponding row in the reference visual tensor; subtracting the cosine similarity from 1 as the sample visual loss; and calculating the average of all sample visual losses as the visual consistency loss.

5. The method according to claim 2, characterized in that, Calculating feature-level contrastive loss involves the following steps: S191. Perform L2 normalization and reshaping on the batch-level global feature tensor and the batch-level personalized feature tensor to obtain the global feature matrix and the personalized feature matrix. S192. Treat each row in the global feature matrix as an anchor point, the corresponding row in the personalized feature matrix as a positive sample, and all other rows in the personalized feature matrix as negative samples to calculate the InfoNCE loss value, and calculate the average of all InfoNCE loss values ​​as the first InfoNCE loss. S193. Treat each row in the personalized feature matrix as an anchor point, the corresponding row in the global feature matrix as a positive sample, and all other rows in the global feature matrix as negative samples to calculate the InfoNCE loss value, and calculate the average of all InfoNCE loss values ​​as the second InfoNCE loss. S194. Calculate the average of the first InfoNCE loss and the second InfoNCE loss as the feature-level contrast loss.

6. The method according to claim 2, characterized in that, The calculation of sample-level contrast loss includes: performing L2 normalization on the visual semantic tensor, taking each row of the visual semantic tensor as anchor points, taking all other rows with the same corresponding true class label as positive samples, taking all other rows with different corresponding true class labels as negative samples, calculating the InfoNCE loss value, and calculating the average of all InfoNCE loss values ​​as the sample-level contrast loss.

7. The method according to claim 1, characterized in that, Each client module stores its own private password in advance, and the private password of each client module is pre-divided into a first share of private password and a second share of private password. The first share of private password is stored in the master server module, and the second share of private password is stored in the slave server module.

8. The method according to claim 7, characterized in that, The main server module performs interference processing and first restoration processing on the scrambled local model parameters, including the following steps: S41. Generate an arbitrary numerical sequence, use the common secret value of the client module corresponding to the scrambled local model parameters to scramble the arbitrary numerical sequence to obtain the scrambled arbitrary numerical sequence, and add the scrambled arbitrary numerical sequence to the scrambled local model parameters to obtain the interference result. S42. Use the first share private value of the client module corresponding to the scrambled local model parameters to perform the first restoration process on the interference result to obtain the first restoration result.

9. The method according to claim 8, characterized in that, S4 includes the following steps: S51. The server module uses the second share private password value of the client module corresponding to the interference result to perform a second restoration process on the interference result, and calculates the process result based on the second restoration result and the first restoration result corresponding to the interference result. S52. Calculate the mean of all process results as the preliminary aggregation result, use the common secret value of several corresponding client modules to scramble the preliminary aggregation result respectively, and send each scrambled preliminary aggregation result to the main server module.

10. The method according to claim 9, characterized in that, The main server module obtains the final aggregation results after scrambling from each preliminary aggregation result after scrambling, including: for each preliminary aggregation result after scrambling, calculating the mean of all arbitrary numerical sequences in this round to obtain the mean sequence, using the common secret value of the client module corresponding to the preliminary aggregation result after scrambling to scramble the mean sequence to obtain the scrambled mean sequence, and subtracting the scrambled mean sequence from the preliminary aggregation result after scrambling.

Images