Transform-based small sample image recognition method in federated learning environment

Abstract

The invention discloses a small sample image recognition method in a federated learning environment based on Transform, and the method comprises the steps: carrying out the image recognition of an image sample through employing a visual converter model, and capturing the information of different scales in an image through employing a multi-scale window in the visual converter model; a self-adaptive multi-scale window attention mechanism is introduced into the visual converter model, and feature extraction under the sparse data condition is enhanced. The invention also provides a collaborative attention strategy to realize cross-client exchange of privacy protection semantic knowledge. Aiming at common class imbalance and overfitting problems in small sample setting, the invention designs an improved focus loss function based on class entropy. The invention also introduces a progressive knowledge distillation framework to maintain the model consistency among the training rounds.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a small sample image recognition method in a federated learning environment based on Transformer. Background Art

[0002] Federated learning (FL), a privacy-preserving collaborative machine learning model, has seen rapid adoption in recent years due to the exponential expansion of decentralized edge devices and growing concerns about data privacy. FL allows multiple clients to train a common global model without sharing their original data, thereby protecting data privacy and locality. Federated learning allows multiple devices or organizations to jointly train a machine learning model while protecting the privacy of their respective data. In this framework, data does not need to be stored centrally but is instead distributed across various participants. Model training is conducted via encrypted communication protocols, ensuring data security and privacy.

[0003] Federated learning (FL) is particularly well-suited for distributed mobile computing environments, where user data cannot be centrally aggregated due to privacy constraints. References (McMahan, E. Moore, D. Ramage, S. Hampson, and B.A. Arcas, "Communication-Efficient Learning of Deep Networks from Decentralized Data," in Artificial Intelligence and Statistics (2017), pp. 1273–1282) propose a classic global optimization algorithm, FedAvg, which moves most model update computations to local user nodes, significantly reducing server load and communication overhead. In FedAvg, each client trains for k iterations and then transmits its model parameters to a central server. The server then aggregates the parameters using simple averaging and distributes the updated global model to all clients. Reference (H. Lee, S. Sharma, M. Zaheer, and T. Li, Efficient Adaptive Federated Optimization, arXiv preprint arXiv:2410.18117 (2024)) proposes FedOpt as an alternative to FedAvg that considers learning rate adjustments during the aggregation process. FedOpt improves performance by including a more efficient optimizer in the global model aggregation process.

[0004] The Vision Transformer (ViT) takes inspiration from Natural Language Processing (NLP) and approaches image understanding as a sequence prediction task. While traditional convolutional neural networks (CNNs) process visual data using translation-invariant filters and local receptive fields, ViT partitions the input image into a series of fixed-sized patches and applies global self-attention to each patch. ViT's global modeling capabilities enable it to capture long-term dependencies and complex interactions between geographic regions in an image.

[0005] A key advantage of ViT is its ability to model the relationship between any pair of patches, regardless of their spatial distance. This contrasts with CNNs, which require multiple stacked layers to achieve a large receptive field. However, ViT relies on large datasets for training, is computationally expensive, and lacks a built-in inductive bias towards locality, which poses significant challenges, particularly in data-limited settings such as few-shot learning. These limitations are exacerbated in federated learning settings, where data is fragmented, often scarce, and non-IID (non-independent and identically distributed) across clients. Therefore, while ViT provides a strong foundation for vision tasks, it requires significant adaptation to function effectively in the federated few-shot setting. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a more accurate small sample image recognition method in a federated learning environment based on Transformer.

[0007] The technical solution adopted by the present invention to solve the above technical problems is: a small sample image recognition method in a Transformer-based federated learning environment, which uses a visual transformer model to perform image recognition on image samples. The visual transformer model uses multi-scale windows to capture information of different scales in the image, and the setting method of the multi-scale window is as follows: the input of the visual transformer model is set as a feature tensor, and the window size is defined to correspond to different granularities; for each window size, the feature map is divided into multiple windows using an expansion operation; each local window is regarded as an independent token sequence to which multi-head self-attention is applied.

[0008] Preferably, a visual transformer model is used to perform image recognition on image samples. The visual transformer model uses a multi-scale window to capture different scale information in the image. The multi-scale window is set as follows: Let the input of the visual transformer model be the feature tensor , defining a set of window sizes Corresponding to different granularities; for each window size , use the expansion operation to divide the feature map into multiple windows: ;in represents the local window at k; Each local window is treated as an independent token sequence with multi-head self-attention applied , It is expressed as follows: ;in, , , ,in , and is a learnable projection matrix, is the key dimension of each head.

[0009] Preferably, in the visual transformer model, the captured scale information is fused, and the scale fusion result X′ is as follows: ;in, are learnable weights normalized using a softmax function to reflect the importance of each scale.

[0010] Preferably, in the visual transformer model, the scale fusion result Connect with its global mixed representation to get the representation result : .

[0011] Preferably, in the visual transformer model, a cross-client collaborative attention mechanism is adopted to compute the attention summaries of all attention heads at a given transformer layer; Note the head Note Summary Defined as: Where, For the i-th client, represents the attention weight of the j-th token under the attention head h, and N is the total number of tokens.

[0012] Preferably, the attention summary is encrypted before being transmitted to the central server, and the encryption method is as follows: the server performs a secure aggregation on all participating clients, and the aggregation result It is expressed as follows: ;in Indicates a secure encryption mechanism; After aggregation, the server processes the aggregation results. Decrypt and average attention template Broadcast to all clients; ;in is the decryption function; Each client then uses this average attention template Guide its local attention map to update and achieve semantic alignment, semantic alignment results : ;in, is a hyperparameter.

[0013] Preferably, in the visual transformer model, an entropy-guided improved focal loss function is used to dynamically adjust the weight of each class during training; the entropy of the classification prediction distribution is calculated during training, ;here, represents the entropy of class c, is the predicted probability of the i-th sample belonging to class c, is the number of local samples marked as class c; Use these entropy values ​​to derive adaptive class weights: ; where C is the number of classes; For items 1 to C Sum, is the feature importance score; The entropy-guided improved focal loss function is defined as: ;in is the training sample set of class c.

[0014] Preferably, in the visual transformer model, for a sample x, the knowledge distillation loss is calculated using the KL divergence between the softened output of the global model and the output of the local model : ;in, is the temperature scaling function, ;in is the raw output of the visual transformer model, and T is the temperature hyperparameter; represents the global model parameters of the communication round t, represents the local model parameters of client i.

[0015] Preferably, in the visual transformer model, the final objective function of each client is for: ; ;in, is the total number of communication rounds, It is a standard mission loss. Introducing regularization from the global model.

[0016] Preferably, a computer-readable storage medium stores a computer program / instruction thereon, which, when executed by a processor, implements the steps of a small sample image recognition method in a Transformer-based federated learning environment.

[0017] Preferably, a computer system comprises: Memory, used to store computer programs / instructions; A processor is used to execute the computer program / instructions to implement the steps of a small sample image recognition method in a Transformer-based federated learning environment.

[0018] The beneficial effects of the present invention are: the present invention proposes a new small-sample image recognition method in a federated learning environment based on Transformer, which can achieve effective visual recognition on distributed clients with minimal labeled data. The visual transformer model of the present invention introduces an adaptive multi-scale window attention mechanism to enhance feature extraction under sparse data conditions. In addition, the present invention also proposes a collaborative attention strategy to achieve cross-client exchange of privacy-preserving semantic knowledge. In response to the common class imbalance and overfitting problems in small-sample settings, the present invention designs an improved class entropy-based focal loss function. In addition, the present invention also introduces a progressive knowledge distillation framework to maintain model consistency between training rounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the principle of the adaptive multi-scale window attention mechanism in the method of the present invention; Figure 2 Schematic diagram of the principle of the cross-client collaborative attention mechanism in the method of the present invention; Figure 3 is a training example graph of entropy-guided focal loss in the method of the present invention; Figure 4 This is the evolution of the test accuracy of the three models in the first-level experiment after 300 communication rounds; Figure 5 This is a graph of the accuracy of different loss functions for rare and frequent classes in the third-level experiment. DETAILED DESCRIPTION

[0020] The present invention will now be described in further detail with reference to the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention. Example 1

[0021] The Vision Transformer (ViT) approaches image understanding as a sequence prediction task. The resulting sequence passes through a multi-layer Transformer module, each of which includes a multi-head self-attention (MSA) mechanism. The MSA module splits the input into multiple heads, each of which learns a different attention pattern and aggregates their outputs.

[0022] Optimization of the first aspect of the present invention. In the small sample image recognition method of the present invention, an adaptive multi-scale window attention mechanism is proposed. The core idea is to divide the feature map into multiple sets with different levels of spatial resolution, apply local self-attention in each window, and then use learnable weights to aggregate information. Learnable weights aggregate information: when merging or summarizing this information, one or more weights are used. These weights can be learned through the training process, that is, they are not fixed, but can be automatically adjusted according to the data to optimize the model performance.

[0023] The present invention provides a small sample image recognition method in a federated learning environment based on Transformer. The method uses a visual transformer model to perform image recognition on image samples. The visual transformer model uses a multi-scale window to capture information of different scales in the image. The multi-scale window is set as follows: Let the input of the visual transformer model be the feature tensor , defining a set of window sizes Corresponding to different granularities (e.g., w1=4, w2=8, w3=16). Granularity refers to the level of detail at which data or information is divided or organized.

[0024] For each window size , use the expansion operation to divide the feature map into multiple windows: ;in represents the local window at k.

[0025] Each local window is treated as an independent token sequence to which Multi-head Self-Attention (MSA) is applied , It is expressed as follows: ;here, , , ,in , and is a learnable projection matrix, is the key dimension of each head.

[0026] This local attention calculation limits the receptive field to a small neighborhood, reducing complexity and increasing spatial sensitivity. Projection matrices are commonly used to map data from high-dimensional space to low-dimensional space, or for feature extraction and dimensionality reduction. "Learnable" means that the parameters (i.e., the elements in the matrix) of these projection matrices are not fixed but can be adjusted and optimized during training. This allows the model to automatically learn the most appropriate projection matrix based on the training data, thereby improving model performance and accuracy. Multi-Head Self-Attention (MSA) is an attention mechanism that allows the model to simultaneously focus on different parts of the input data when processing it. This mechanism works by splitting the input data into multiple "heads," independently calculating attention weights for each head, and then combining the outputs of these heads. This approach improves the model's ability to capture complex relationships in the input data.

[0027] In order to combine the outputs of different window scales, we adopt a weighted fusion strategy to adaptively balance the attention of different resolutions. In the visual transformer model, the captured scale information is fused, and the scale fusion result X′ is as follows: ;in, are learnable weights normalized using a softmax function to reflect the importance of each scale. This design enables the model to emphasize fine details in some areas while maintaining coarse context in others. Fine details refer to very specific or subtle features or information; coarse context refers to relatively broad or less specific background information or context.

[0028] To achieve interaction across different spatial resolutions, we introduce a cross-scale fusion mechanism in which the scale fusion result X′ is connected with its global hybrid representation in the visual transformer model to obtain the representation result : ; Characterization results The combination of local spatial attention and global semantic summarization enhances representational completeness while maintaining computational efficiency. Local spatial attention typically refers to the ability of a model to focus on specific local regions or features within an image in image processing or vision tasks. Global semantic summarization refers to the process of integrating or summarizing the global semantic information of an image or text to capture overall meaning. Representational completeness refers to the ability of a model or result to comprehensively and accurately represent or reflect the characteristics and information of the input data. Computational efficiency refers to the speed and resource utilization of a model when performing a task. Efficient algorithms can complete tasks in a shorter time using fewer resources.

[0029] This adaptive multi-scale window attention of the present invention not only reduces the overhead of traditional global attention, but also improves the adaptability of the model to different input structures, which is especially important in federated learning scenarios because each client may encounter significantly different data types, image resolutions, or task-specific features. In addition, the local window design contains an inductive bias that favors spatial continuity, which complements the global modeling capabilities of the model. An adaptive multi-scale window attention mechanism of the present invention combines coarse-to-fine feature modeling and hierarchical attention fusion to achieve efficient computing and robust representation learning across heterogeneous data distribution clients. Robust representation learning means that the model can learn stable and effective feature representations from the data, which have a certain resistance to changes or noise in the data.

[0030] A fundamental challenge in federated few-shot learning scenarios lies in the inherent data heterogeneity across clients. Each client collects data independently, often from different sources or under different environmental conditions, resulting in non-IID (non-independent and identically distributed) data partitions. Traditional federated averaging techniques (such as FedAvg) simply average model weights across clients without considering semantic differences in local data representations, which can lead to unstable convergence, model collapse, and poor generalization to unseen tasks.

[0031] The second aspect of the present invention is optimized for the small sample image recognition method. A cross-client collaborative attention mechanism is proposed. In the visual transformer model, this cross-client collaborative attention mechanism is used to compute an attention summary for all attention heads at a given transformer layer. An attention summary is a specified aggregation or summary of the attention graphs of all attention heads to obtain a comprehensive representation. Each attention head can independently focus on different parts of the input data.

[0032] Note the head Note Summary Defined as: Where, For the i-th client, represents the attention weight of the jth token under the attention head h, and N is the total number of tokens. These attention summaries capture the average structural focus of the model on the input space.

[0033] To maintain privacy, these summaries are encrypted before being transmitted to the central server. The server performs a secure aggregation of all participating clients. It is expressed as follows: ;in, Represents a secure cryptographic function (e.g., homomorphic encryption or differential privacy mechanisms).

[0034] After aggregation, the server processes the aggregation results. Decrypt and average attention template Broadcast to all clients. The average attention template refers to a template or model used to represent or calculate the average attention when processing information or data.

[0035] ;in is the decryption function.

[0036] Each client then uses this average attention template Guide its local attention map to update and achieve semantic alignment, semantic alignment results : ;in, is a hyperparameter.

[0037] This co-adjustment process ensures that while clients are still learning from their own data, their attention mechanisms are pushed towards a jointly learned shared semantic structure. The advantages of this mechanism are twofold. First, it improves generalization by providing a consistent representation space across heterogeneous clients. Second, it does so without exchanging raw data, nor sharing raw data or gradients, making it inherently privacy-preserving. The mechanism enables semantic-level interaction across clients while maintaining privacy. This approach allows the global model to learn meaningful cross-client relationships and update itself in a direction that better captures generalizable patterns.

[0038] A key obstacle in federated few-shot learning is class imbalance, exacerbated by the limited number of labeled examples and the dispersed, non-IID nature of client data. Certain classes may be overrepresented in certain clients, while other classes may be underrepresented or completely missing. Traditional loss functions, such as cross-entropy, assume a balanced data distribution and fail to address overfitting and undergeneralization issues for rare classes.

[0039] To alleviate this problem, the third aspect of the present invention optimizes and enhances the focal loss in the small-sample image recognition method. In the visual transformer model of the present invention, an entropy-guided improved focal loss is used to dynamically adjust the weight of each class during training, causing the model to focus more on ambiguous or unrepresentative classes and less on overconfident predictions.

[0040] Standard focal loss is defined as: ;in, is the predicted probability of target category t, is the weight factor for a specific category, is a focusing parameter used to reduce the relative loss of correctly classified samples.

[0041] However, in the small sample federation setting, Setting it to a fixed value is suboptimal. Instead, we recommend computing the entropy of the categorical predictive distribution during training: ;here, represents the entropy of class c, is the predicted probability of the i-th sample belonging to class c, is the number of local samples labeled as class c.

[0042] We use these entropy values ​​to derive adaptive class weights: ; where C is the number of classes. This weighting scheme increases the influence of classes that are difficult to classify due to scarce or overlapping features, while reducing the influence of classes that are easy to distinguish.

[0043] The final entropy-guided improved focal loss is defined as: ;in, is the training sample set of class c.

[0044] This formulation introduces two fundamental improvements. First, the entropy-based reweighting is dynamically updated during training rounds, ensuring that the model focus shifts based on the evolving classification difficulty. Second, the approach maintains computational efficiency and can be easily incorporated into local training on edge clients.

[0045] This approach is more effective in federated learning scenarios, where each client may have a highly skewed class distribution. Instead of simply amplifying the gradient of the minority class, which can lead to instability, the entropy-guided loss exploits semantic uncertainty to softly guide the training focus, which leads to smoother convergence, improved generalization of rare classes, and better coordination when aggregating local models.

[0046] In traditional federated learning, the global model is periodically aggregated and redistributed to participating clients. However, due to non-IID data and limited samples on each client, local model updates often deviate significantly from the global optimum, leading to unstable convergence and performance degradation. This problem is particularly prominent in few-shot learning scenarios, where local updates are highly sensitive to noise and class imbalance.

[0047] In order to alleviate this problem, in the optimization of the fourth aspect of the present invention, in the small sample image recognition method of the present invention, we propose a progressive knowledge distillation framework.

[0048] In the Vision Transformer model, for a sample x, the knowledge distillation loss is calculated using the KL (Kullback-Leibler) divergence between the softened output of the global model and the output of the local model: ;in, represents the global model parameters of the communication round t, represents the local model parameters of client i.

[0049] Furthermore, to soften the predictions, we apply a temperature scaling function that increases anisotropy and reveals inter-class relationships: ;in where is the raw output (logits) of the model, and T is the temperature hyperparameter. Higher values ​​of T result in a smoother distribution, which helps the student model better mimic the teacher's behavior. The temperature scaling function is used to smooth or "soften" the predicted probability distribution. "Temperature" is an adjustable parameter that changes the entropy of the predicted distribution.

[0050] Furthermore, to control the impact of knowledge distillation on training rounds, we introduce a stepwise weighting term : ;in, is the total number of communication rounds.

[0051] In the visual transformer model, the final objective function of each client is for: Here, is a standard task loss (e.g., cross entropy or focal loss), Regularization is introduced from the global model. Initially, Small, allowing the client to learn independently. As training progresses, the weights of the distilled terms gradually increase, and the client gradually aligns with the global semantic structure.

[0052] This gradual mechanism avoids abrupt convergence or premature homogenization, both of which can harm generalization. Furthermore, since only soft predictions (not raw data or gradients) are shared, the approach remains privacy-preserving and communication-efficient.

[0053] The framework integrates global knowledge that changes over time through regularized learning. Each client computes soft labels on local samples using the received global model and updates its own model using hard labels and soft targets. This hybrid objective improves representation alignment while maintaining task-specific learning.

[0054] The above optimization was verified by data set and the results were analyzed as follows: Dataset Description: We conduct experiments on two popular datasets, miniImageNet and tieredImageNet, to evaluate the effectiveness of our few-shot image recognition method in a federated setting. Both datasets have separate class splits, making them suitable for few-shot classification problems. The miniImageNet dataset has 600 images per class for each of its 100 classes, each measuring 84×84 pixels. In the federated setting, each client receives a subset of training classes; depending on the configuration, each class has either one (1-shot) or five (5-shot) labeled examples to account for the sparsity of edge data.

[0055] Experiments were conducted using PyTorch 2.0 in a federated simulation environment implemented using the Flower framework. Simulations assumed 10 clients per round, with each client having non-overlapping data partitions reflecting non-IID characteristics. Model training was performed on a distributed cluster equipped with dual Intel Xeon Gold 6226R CPUs, four NVIDIA A100 TensorCore GPUs, 512GB of DDR4 ECC RAM, and Ubuntu 22.04 LTS as the operating system. The software stack included CUDA version 12.1 and Python version 3.10. Communication delays were simulated using Flower's built-in delay injection tool to emulate realistic network conditions. Each federated round consisted of three local training epochs per client using the SGD optimizer, with an initial learning rate of 0.01 and a batch size of 8. The knowledge distillation temperature was fixed at 4 throughout the experiments.

[0056] The first-level experiment evaluates the adaptive multi-scale window attention mechanism optimized by the first aspect of the present invention. To evaluate the effectiveness of the adaptive multi-scale window attention module proposed in this invention, a controlled ablation experiment was conducted on the miniImageNet dataset using a 5-class 5-sample joint setting. In small-sample learning, 5 classes and 5 samples refer to randomly selecting 5 classes from the dataset during each training or testing phase, with 5 samples from each class.

[0057] Three model variants are compared: (1) a baseline model with global self-attention (no window partitioning), (2) a fixed-scale attention model with uniformly sized windows, and (3) our proposed Visual Transformer model with adaptive multi-scale attention.

[0058] The goal of this comparison is to evaluate the impact of dynamically adjusting the window granularity on model performance, convergence behavior, and spatial feature extraction. Fixed-scale models cannot adapt to data complexity across clients, while global attention has high computational cost and weak local feature encoding. In contrast, our proposed module allows each client to adaptively perform attention on both coarse-grained and fine-grained regions, thereby better capturing semantic and structural information.

[0059] Figure 4 The figure shows the evolution of the test accuracy of the three models in the first-level experiment over 300 communication rounds. A communication round refers to the number of times each node (such as a client or server) exchanges information or updates the model. Test accuracy refers to the accuracy of the model's prediction results on the test dataset. Compared with the two baseline configurations (1) and (2), the model using the adaptive multi-scale attention of our invention not only converges faster but also has higher final accuracy.

[0060] The global attention baseline model (1) achieves 66.4% accuracy after 300 epochs, while the fixed-scale model (2) achieves 68.9%. Our adaptive multi-scale model (3) outperforms both, achieving 72.1%. This improvement is attributed to its ability to flexibly extract discriminative features at different spatial scales, which is particularly beneficial in few-shot scenarios where generalization from limited examples is crucial.

[0061] The second level of experiments evaluates the collaborative cross-client attention mechanism optimized by the second aspect of the present invention. To study the effectiveness of the collaborative cross-client attention mechanism, a set of experiments is designed to isolate and evaluate its impact on generalization and inter-client consistency. The core idea behind this module is to achieve semantic alignment across heterogeneous clients without sharing raw data or model weights. By exchanging encrypted attention summaries, the present method facilitates indirect knowledge transfer and helps mitigate the impact of non-IID data distributions.

[0062] We compare the following three variants: (a) Local only: Each client trains independently without any aggregation or communication (no collaboration).

[0063] (b) FedAvg (full model sharing): standard federated averaging where the client and server share model parameters.

[0064] (c) Our invention (collaborative attention): Clients exchange encrypted attention vectors to align their attention heads towards a global semantic structure.

[0065] The model is evaluated on unseen test classes in a 5-class 1-sample setting on the miniImageNet dataset and the tieredImageNet dataset. Table 1 shows the average classification accuracy over all clients.

[0066]

[0067] As shown in Table 1, method (c) achieves significant improvements over both the local-only method (a) and the traditional FedAvg strategy (b). These results demonstrate that the semantic alignment strategy of this embodiment successfully bridges the gap in representation between customers, leading to better generalization to unseen categories.

[0068] In addition to accuracy, we also evaluate the consistency of the model in terms of internal attention behavior. Table 2 reports the average attention consistency variance.

[0069]

[0070] Table 2 shows that the collaborative attention mechanism (c) of the present invention significantly reduces the attention variance between clients by more than 40% on both datasets. This result provides empirical evidence that clients trained using the method of this embodiment develop more consistent internal representations.

[0071] In the third level of experiments, in order to evaluate the impact of the proposed entropy-guided improved focal loss, experiments are conducted to evaluate the performance of two classes, especially in the underrepresented class and the overall training dynamics. Class imbalance is a common problem in few-shot federated learning, where clients often encounter highly skewed local distributions. Traditional loss functions, such as cross-entropy or fixed-weighted focal loss, tend to either overfit the dominant class or undertrain the rare class. Our method introduces a dynamic weighting scheme based on predicted entropy, which enables the model to adaptively emphasize uncertain or minority samples during training.

[0072] We first analyzed the model's ability to improve performance on rare classes. To this end, we divided the test classes into two groups: the top 25% most frequent classes (the frequent group) and the bottom 25% least frequent classes (the rare group). We then compared the average classification accuracy across groups using three loss variants: (A) cross-entropy loss, (B) focal loss with static weights, and (C) our entropy-guided focal loss.

[0073] Figure 5The results show that in the third level experiment, Figure 5 Shown are the accuracies for rare and frequent classes using different loss functions on the miniImageNet dataset (1-shot) (1 class, 1 example). Method (C) significantly improves performance on rare classes while maintaining strong accuracy on frequent classes. Specifically, for the miniImageNet dataset, method (C) achieves 47.8% accuracy on rare classes, compared to 42.3% and 44.7% under cross entropy (A) and standard focal loss (B), respectively. On frequent classes, method (C) achieves comparable or slightly better performance, demonstrating that it does not sacrifice learning for rare classes.

[0074] The results show that the entropy-guided focal loss of the present invention effectively improves the robustness of the model in unbalanced small sample tasks. It enhances the learning of rare categories, stabilizes local training, and is more suitable for federated learning environments.

[0075] Fourth-level experiments,To evaluate the effectiveness of the progressive knowledge distillation framework of our invention, we conducted ablation experiments to measure its impact on model consistency and generalization, especially in the case of few-shot learning in a non-IID federated setting.,This module aims to mitigate client drift and stabilize training by gradually introducing soft guidance from the global model.

[0076] We consider the following three configurations for comparison: (10) No Distillation: The client only uses hard labels and standard loss function for local training. No global soft supervision is used.

[0077] (20) Global-only Distillation: Each client is guided by the soft label of the global model in each round of communication, and the distillation weights remain unchanged.

[0078] (30) Progressive Distillation (Ours): The client can obtain soft guidance from a global model with time-varying distillation weights that increase with the number of communication rounds.

[0079] We first evaluate the degree of divergence between the models across rounds. Specifically, we measure the Euclidean distance between the parameters of each client's local model and the global model at each communication round. The average divergence is shown in Table 3.

[0080]

[0081] As shown in Table 3, our progressive approach (30) significantly reduces model divergence compared to the two schemes (10) and (20). The gradual integration of soft supervision helps align local models with the global objective over time without causing abrupt homogenization that could reduce task-specific adaptation capabilities.

[0082] Next, we evaluate the generalization of each method by testing it on a set of unseen clients and novel classes. These clients were not part of the training and thus serve as a proxy for the true adaptability of the learned global model. Table 4 summarizes the average precision in the 5-class 5-sample setting on the miniImageNet dataset and the tieredImageNet dataset.

[0083]

[0084] Table 4 shows that our method (30) consistently outperforms the other two baselines (10) and (20) in terms of generalization. The improvement in accuracy is more significant on the tieredImageNet dataset, which has higher intra-class diversity, indicating that the progressive mechanism is more suitable for adapting to complex semantic structures. We observe that the dynamic scheduling of distillation weights helps prevent over-regularization in the early stages, when local models are still forming class boundaries. As training progresses, the influence of the global model continues to increase, helping to strengthen semantic consistency and enhance consistency across clients.

[0085] Due to data heterogeneity, limited local samples and strict privacy restrictions, small-shot image recognition in a federated learning environment faces unique challenges. To address these issues, the present invention proposes a novel Transformer-based small-shot image recognition method in a federated learning environment, which can achieve effective visual recognition on distributed clients with minimal labeled data. The visual transformer model of the present invention introduces an adaptive multi-scale window attention mechanism to enhance feature extraction under sparse data conditions. In addition, the present invention also proposes a collaborative attention strategy to achieve cross-client exchange of privacy-preserving semantic knowledge. To address the common class imbalance and overfitting problems in the small-shot setting, the present invention designs an improved class entropy-based focal loss function. In addition, the present invention also introduces a progressive knowledge distillation framework to maintain model consistency between training rounds. Experiments conducted on multiple small-shot image recognition in a federated environment show that the present method outperforms other existing baseline methods in terms of accuracy, communication efficiency and generalization. Example 2

[0086] A computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the steps of a small sample image recognition method in a Transformer-based federated learning environment in Example 1. Example 3

[0087] A computer system comprising: Memory, used to store computer programs / instructions; A processor is used to execute the computer program / instructions to implement the steps of a small sample image recognition method in a Transformer-based federated learning environment in Example 1.

[0088] The above description only describes specific embodiments of the present invention. Various examples do not limit the essential content of the present invention. After reading the description, ordinary technicians in the relevant technical field can modify or deform the specific embodiments described above without departing from the essence and scope of the invention.

Claims

1. A small-sample image recognition method in a Transformer-based federated learning environment uses a visual transformer model to perform image recognition on image samples. The visual transformer model uses a multi-scale window to capture information of different scales in the image. The following features are present: The setting method of the multi-scale window is as follows: The input of the visual transformer model is assumed to be a feature tensor, and window sizes are defined corresponding to different granularities. For each window size, an expansion operation is used to divide the feature map into multiple windows. Each local window is regarded as an independent token sequence to which multi-head self-attention is applied.

2. The small sample image recognition method in a Transformer-based federated learning environment according to claim 1, characterized in that: Let the input of the visual transformer model be the feature tensor , defining a set of window sizes Corresponding to different particle sizes; For each window size , use the expansion operation to divide the feature map into multiple windows: ； in, represents the local window at k; Each local window is treated as an independent token sequence with multi-head self-attention applied , It is expressed as follows: ； in, , , ,in , and is a learnable projection matrix, is the key dimension of each head.

3. The small sample image recognition method in a Transformer-based federated learning environment according to claim 2, characterized in that: In the visual transformer model, the captured scale information is fused, and the scale fusion result X′ is as follows: ； in, are learnable weights normalized using a softmax function to reflect the importance of each scale.

4. The small sample image recognition method in a federated learning environment based on Transformer according to claim 1, characterized in that: In the visual transformer model, the scale fusion result Connect with its global mixed representation to get the representation result : 。 5. The small sample image recognition method in a federated learning environment based on Transformer according to claim 1, characterized in that: In the visual transformer model, a cross-client collaborative attention mechanism is adopted to compute the attention summary of all attention heads at a given transformer layer; Note the head Note Summary Defined as: ； Where, For the i-th client, represents the attention weight of the j-th token under the attention head h, and N is the total number of tokens.

6. The method for small-sample image recognition in a federated learning environment based on Transformer according to claim 4, characterized in that: The attention summary is encrypted before being transmitted to the central server. The encryption method is as follows: the server performs a secure aggregation on all participating clients. The aggregation result It is expressed as follows: ； in, Indicates a secure encryption mechanism; After aggregation, the server processes the aggregation results. Decrypt and average attention template Broadcast to all clients; ； in, is the decryption function; Each client then uses this average attention template Guide its local attention map to update and achieve semantic alignment, semantic alignment results : ； in, is a hyperparameter.

7. The method for small-sample image recognition in a federated learning environment based on Transformer according to claim 1, characterized in that: In the visual transformer model, an entropy-guided improved focal loss function is used to dynamically adjust the weight of each class during training; the entropy of the classification prediction distribution is calculated during training. ； in, represents the entropy of class c, is the predicted probability of the i-th sample belonging to class c, is the number of local samples marked as class c; Use these entropy values ​​to derive adaptive class weights: ； Where C is the number of classes; For items 1 to C Sum, is the feature importance score; The entropy-guided improved focal loss function is defined as: ； in is the training sample set of class c.

8. The small sample image recognition method in a federated learning environment based on Transformer according to claim 1, characterized in that: In the visual transformer model, for a sample x, the knowledge distillation loss is calculated using the KL divergence between the softened output of the global model and the output of the local model : ； in, is the temperature scaling function, ； in is the raw output of the visual transformer model, and T is the temperature hyperparameter; represents the global model parameters of the communication round t, represents the local model parameters of client i.

9. The small sample image recognition method in a federated learning environment based on Transformer according to claim 8, characterized in that: In the visual transformer model, the final objective function of each client is for: ； ； in, is the total number of communication rounds, It is a standard mission loss. Introducing regularization from the global model.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of a small sample image recognition method in a Transformer-based federated learning environment described in any one of claims 1-8 are implemented.

Images