Chinese painting classification training method based on federal learning

Abstract

The invention relates to a Chinese painting classification training method based on transverse federated learning, and the training method comprises the following steps: 1, constructing a federated learning platform which takes a federated learning framework as a basis and is added with participants; step 2, adopting a federated learning framework of a client / server architecture as a training framework to realize multi-party joint training, and jointly establishing a global model; step 3, the server sends current global model parameters to each client participating in federated learning framework training; and step 4, under the client / server architecture, training the received global model on the local data set by each local device, and the like. Compared with the prior art, the method has the beneficial effects that a federated learning framework containing three-party participation is provided on the basis of the Federated Scope, a global model can be jointly established without data transmission through the federated learning framework, and an effective and privacy-safe solution is provided for cooperative research among multiple platforms.

Description

Technical Field

[0001] The present invention relates to a Chinese painting classification method, and more particularly to a Chinese painting classification training method based on federated learning. Background Art

[0002] Traditional Chinese paintings are a unique art form that have played an important role in the development of Chinese aesthetics. They cover a wide range of styles and rich expressive brushstrokes. With the development of digital technology, it provides a good way to protect and inherit traditional Chinese paintings. Among them, the digital classification and application of artistic paintings is an important part of the digital protection of cultural heritage, which lays the foundation for the digital storage, restoration, promotion and dissemination of cultural heritage and cultural works.

[0003] In recent years, a large number of scholars have conducted in-depth research on the classification of Chinese paintings. Among them, the authors of document 1, Jiang W et al., combined discrete cosine transform (DCT) and convolutional neural network (CNN) to classify the two painting techniques of Gongbi and Xieyi, achieving a good effect of 95% (Jiang W, Wang Z, Jin JS, et al. DCT-CNN-based classification method for the Gongbi and Xieyi techniques of Chinese ink-wash paintings [J]. Neurocomputing, 2019, 330 (FEB.22): 280-286. DOI: 10.1016 / j.neucom.2018.11.003.); the authors of document 2, Liu et al., proposed an algorithm that calculates the local patch similarity through Euclidean distance, then combines the similarity coefficients between images into a similarity matrix, and uses the AdaBoost classifier to classify the style of Chinese paintings (Liu X. Art painting image classification based on neural network [J]. Computational Intelligence and Neuroscience, 2022, 2022.); the author of document 3, Sun M et al. proposed a sparse hybrid CNN under a deep learning framework to extract brushstroke features, and classified 120 paintings by 6 famous artists, achieving an accuracy of 81% (Sun M, Zhang D, Wang Z, et al. Monte Carlo convex hull model for classification of traditional Chinese paintings [J]. Neurocomputing, 2016, 171: 788-797.); Reference 4 author Jiang W extracted low-level, medium-level, and high-level features through CNN for fusion, and gave different weights to the fused features, thereby optimizing the model and achieving an accuracy of 89.2% (Jiang W, Meng X, Xi J. Multilevel Attention and Multiscale Feature Fusion Network for Author Classification of Chinese Ink-Wash Paintings [J]. Discrete Dynamics in Nature and Society, 2022, 2022.); The authors of Reference 5, Yuzhi L et al., extracted the features of traditional Chinese paintings by fine-tuning the pre-trained VGG-F model, and introduced the mutual information theory on the basis of the DEFC algorithm, and proposed an embedded classification algorithm based on mutual information (MIDEC), and classified 100 paintings of 10 painters respectively (Yuzhi L, Jiachuan S, Bin H. Improved embedded learning for classification of Chinese paintings [J]. Journal of Computer-Aided Design & Computer Graphics, 2018, 30 (5): 893-900.). .

[0004] Chinese paintings are precious cultural heritage and are usually protected by copyright law. However, in the digital age, the use of Chinese paintings and their honor infringe the copyright of the painters or their heirs. Therefore, in the process of dissemination, research and development, it is necessary to consider both copyright and privacy issues, as well as how to efficiently collaborate while protecting privacy.

[0005] Federated learning is a distributed training technology that trains models on local devices and then sends the updated model parameters to a central server for aggregation, thereby achieving global model training while protecting user privacy. Combined with federated learning, model training can be performed on local devices without directly sharing original data, thereby protecting data copyright and privacy.

[0006] Federated learning is widely used in the field of computer vision. In addition, it has been studied in depth for different types of tasks. For example, in the object detection task, Yu P et al. in Reference 6 proposed a KL divergence method to reduce the model accuracy loss caused by Non-IID (non-independent distribution) (Yu P, Liu Y. Federated Object Detection: Optimizing Object Detection Model with Federated Learning [C] / / ICVISP 2019: 3rd International Conference on Vision, Image and Signal Processing. 2019. DOI: 10.1145 / 3387168.3387181.);

[0007] There are also frameworks that include different types of tasks. In document 7, Liu Y et al. proposed FedVision, which supports the use of models such as FastRCNN and YOLOv3 to facilitate target detection in smart city scenarios. However, FedVision only supports the FedAvg algorithm and single GPU training (Liu Y, Huang A, Luo Y, et al. Fedvision: An online visual object detection platform powered by federated learning [C] / / Proceedings of the AAAI conference on artificial intelligence. 2020, 34 (08): 13172-13179.); the author of document 8, He C et al., pointed out that FedCV is an extended computer vision federated learning platform that supports multi-GPU training while also providing implementations of different visual tasks and various FL algorithms (He C, Dilipbhai Shah A, Tang Z, et al. FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks [J]. arXiv e-prints, 2021. DOI: 10.48550 / arXiv.2111.11066.).

[0008] The introduction of federated learning in the task of traditional Chinese painting classification enables multi-platform assistance and information sharing while ensuring privacy and copyright security. For example, a classification algorithm can be jointly trained by combining data from multiple parties such as museums, galleries, research institutes, and private exhibition halls.

[0009] The key components of federated learning include data sources, federated learning systems, and users. In a federated learning system, each data source is responsible for data preprocessing, collaboratively building a shared learning model, and feeding back the final output results to the user.

[0010] From the perspective of the data features and sample overlap of the participants, federated learning can be specifically divided into horizontal federated learning and vertical federated learning. Horizontal federated learning is a form of federated learning in which multiple participants (such as devices, institutions, or entities) have the same feature set, but their data distribution is different. Vertical federated learning is a form of federated learning in which multiple participants (such as devices, institutions, or entities) have different feature sets. In vertical federated learning, the differences in feature sets between participants are more obvious.

[0011] In today's digital era, the above-mentioned existing technologies have not been able to effectively solve the problem of how to effectively carry out collaborative training while protecting privacy, which requires considering copyright and privacy issues during the dissemination, research and development of Chinese paintings. Summary of the invention

[0012] The purpose of the present invention is to overcome the defects in the prior art, use distributed training technology, utilize deep learning networks, and combine horizontal federated learning to provide a Chinese painting classification training method based on federated learning. At the same time, the influence of different backbone networks and different aggregation algorithms on the experiment is also studied, effectively avoiding the data copyright and privacy issues that are easy to arise.

[0013] The Chinese painting classification training method based on horizontal federated learning includes the following steps:

[0014] Step 1. Build a federated learning platform with participants based on the federated learning framework (FederatedScope);

[0015] Step 2. Use the client / server federated learning framework as the training framework to achieve multi-party joint training. Through federated learning (FL), a global model is jointly established without transmitting data.

[0016] Step 3. The server sends the current global model parameters to each client participating in the federated learning framework training;

[0017] Step 4. Under the client / server architecture, each local device trains the received global model on the local dataset;

[0018] Step 5. The client transmits the local model parameter updates to the central server using stochastic gradient descent, and then the central server uses the aggregation method to update the global model parameters;

[0019] Step 6. The client repeats the process of updating the global model parameters in each round, gradually improving the performance of the global model to adapt to the local data distribution of each participant;

[0020] Step 7. When all clients need to jointly train the global model, the objective function is as follows:

[0021]

[0022]

[0023] Where: W is the weight of the model. This method selects three models for experiments, MobileNet-V2, EfficientNet and MobileViT, f (k) (W) is the local objective function of the kth client, which is used to measure the local empirical risk on the heterogeneous dataset, and l is the loss function of the global model, which is set to the cross entropy loss.

[0024] Furthermore, in step 1, participants in the federated learning platform include museums, galleries, and research institutes that own local data and participate in model training, and a horizontal federated learning approach is used to classify and train the data.

[0025] Furthermore, in step 2, the client is a device or entity that owns local data and participates in model training, and the server is a central server or federated learning coordinator, which is the central node responsible for coordinating and managing global model training.

[0026] Furthermore, in step 2, the following steps will be performed before the server training starts:

[0027] Step 2.1 The server initializes a global model by integrating the local model training results of multiple decentralized institutions;

[0028] Step 2.2 The server sends the initialized global model to each local device participating in federated learning.

[0029] Furthermore, in step 3, the global model includes a neural network model of an input layer, multiple hidden layers and an output layer, each layer contains a number of neurons, and the parameters of the global model include weights and biases of all layers. The bias (Bias) is a parameter of the neuron, which is a constant used to adjust the difference between the model prediction result and the actual result.

[0030] Further, in step 4, under the client / server architecture, each local device trains the received global model on the local dataset: wherein the main steps of performing a round of training include:

[0031] Step 4.1 Broadcast mode: The server broadcasts the latest global model to the clients involved;

[0032] Step 4.2 Local training: Once the global model is received, the client performs local training using its own trainer based on its private data;

[0033] Step 4.3 Model parameter update: After local training, the client sends the trained local model parameter updates back to the central server;

[0034] Step 4.4 Federated Aggregation: With the assistance of the aggregator, the central server performs federated aggregation on the received model parameter updates and optimizes the global model.

[0035] Furthermore, in step 5, the aggregation method used is the FedAvg and FedProx federation algorithms.

[0036] Furthermore, in step 6, the process is repeated in each round, and after the last round of training is completed, multiple rounds of iterations need to be repeated until the global model reaches the expected performance or reaches the preset number of training rounds.

[0037] Compared with the prior art in this technical field, the present invention has the following beneficial effects:

[0038] 1. The Chinese painting classification training method based on federated learning described in the present invention provides a federated learning framework involving three parties on the basis of FederatedScope. Through the federated learning framework, a global model can be jointly established without transmitting data, providing an effective and privacy-safe solution for collaborative research among multiple platforms.

[0039] 2. The Chinese painting classification training method based on federated learning described in the present invention and the established federated learning global model platform not only take into account the special needs of the Chinese painting field, but also provide a more private and secure learning method, so that the model platform can complete the federated learning task without sharing the original data, which is helpful to promote the research progress in the field of Chinese painting and promote knowledge sharing and communication among institutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of the basic structure of the server and client.

[0041] Figure 2 Schematic diagram of the weight update process of the global model for Chinese painting classification in horizontal federated learning. DETAILED DESCRIPTION

[0042] Now combined with the instruction manual Figure 1-2, a specific embodiment of the Chinese painting classification training method based on federated learning described in the present invention is introduced in detail.

[0043] Example:

[0044] The Chinese painting classification training method based on federated learning described in the present invention is as follows: Figure 1-2 As shown, the following steps are included:

[0045] Step 1. Build a federated learning platform with participants based on the federated learning framework (FederatedScope), such as Figure 1 As shown, participants in the federated learning platform include museums, galleries, and research institutes that own local data and participate in model training, and use horizontal federated learning to classify and train data;

[0046] Step 2. Use the client / server federated learning framework as the training framework to achieve multi-party joint training. Through federated learning (FL), a global model is jointly established without transmitting data. The server is a central server or a federated learning coordinator. The server serves as the central node responsible for coordinating and managing global model training. Before the server training begins, it will be initialized by integrating the local model training results of multiple decentralized institutions into a global model, and then sending the global model to each local device participating in the federated learning.

[0047] Step 3. The server sends the current global model parameters to each client participating in the federated learning framework training. The global model includes a neural network model of an input layer, multiple hidden layers, and an output layer. Each layer contains a number of neurons. The parameters of the global model include the weights and biases of all layers. The bias is a parameter of the neuron, which controls the activation state of the neuron. The value of the bias can adjust the threshold of the neuron, thereby affecting the output of the neuron.

[0048] Step 4. Under the client / server architecture, each local device trains the received global model on the local dataset:

[0049] Step 4.1 Broadcast mode: The server broadcasts the latest global model to the clients involved;

[0050] Step 4.2 Local training: Once the global model is received, the client performs local training using its own trainer based on its private data;

[0051] Step 4.3 Model parameter update: After local training, the client sends the trained local model parameter updates back to the central server;

[0052] Step 4.4 Federated Aggregation: With the assistance of the aggregator, the central server performs federated aggregation on the received model parameter updates and optimizes the global model;

[0053] Step 5. The client transmits the local model parameter updates to the central server using stochastic gradient descent, and then the central server uses FedAvg and FedProx federation algorithms to update the global model parameters;

[0054] Step 6. The client repeats the process of updating the global model parameters in each round, gradually improving the performance of the global model to adapt to the local data distribution of each participant. In each round, after the previous round of training, it needs to repeat multiple rounds of iteration until the global model reaches the expected performance or reaches the preset number of training rounds; Figure 2 The arrangement and interaction of the steps are encapsulated in Figure 1. W represents the model weight, and D represents the Chinese painting data. In a round of training, the client (Client C; from left to right, they are museums, galleries, and research institutes) sends the server ( Figure 1 The weights distributed by Server S are decrypted, and then the data and weights are combined into the model for training to obtain local weights, which are then encrypted. The encrypted local weights are then securely aggregated using FedAvg or FedProx to optimize and update the global model.

[0055] Step 7. When all clients need to jointly train the global model, the objective function is as follows:

[0056]

[0057]

[0058] Where: W is the weight of the model. This method selects three models for experiments, MobileNet-V2, EfficientNet and MobileViT, f (k) (W) is the local objective function of the kth client, which is used to measure the local empirical risk on the heterogeneous dataset, and l is the loss function of the global model, which is set to the cross entropy loss.

[0059] Verification Example

[0060] In order to further illustrate the superior technical effects of the present invention in actual use, experiments were conducted using PyTorch on an NVIDIA 2080Ti GPU. In order to verify the technical performance of the present invention, two lightweight CNN models, EfficientNet and MobileNet-V2, and a lightweight network MobileViT based on an attention mechanism were selected for experiments.

[0061] In this verification example, two classic federated learning algorithms, FedAvg and FedProx, were selected for comparative experiments. This helps to gain a deeper understanding of the performance of different algorithms under the framework and provide a reference for subsequent improvements and optimizations.

[0062] Regarding the selection of models, EfficientNet and MobileNet-V2 are lightweight CNN models suitable for mobile devices and resource-constrained environments, while MobileViT represents a new generation of image classification models based on the attention mechanism, which is expected to show better performance in federated learning tasks. This multi-model selection design enables the system to comprehensively consider the characteristics of different models, thereby more comprehensively evaluating the versatility and applicability of the federated learning framework.

[0063] The Chinese painting classification training method based on federated learning described in the present invention conducts detailed experimental verification on two classic algorithms, FedAvg and FedProx, to understand their classification accuracy of the federated learning framework under different models. This comprehensive experimental design helps to better understand the advantages and limitations of the federated learning algorithm and provides a basis for future research. Table 1 shows the classification accuracy of the federated learning framework under different models.

[0064] Table 1

[0065]

[0066] In Table 1, round_num refers to the training round. For MobileNetV2 and ResNet50 experiments, 128*128 is used as the data size for centralized training, while MobileViT sets the data size to 224*224. Among these models, only MobileNetV2 and ResNet50 contain pre-trained parameters.

[0067] In these experiments, the process of federated learning in each round (or training round) involves initializing the global model, sending the current global model parameters to each client, the client training the local model on local data, the local model parameter updates transmitted to the central server, and then the central server uses some aggregation algorithm to update the global model parameters. This process is repeated in each round, gradually improving the performance of the global model to adapt to the local data distribution of each participant.

[0068] It should be noted that the MobileNetV2 and ResNet50 models contain pre-trained parameters, which may help the initial global model to better learn common features, while MobileViT has the best experimental results. The main reason is that MobileViT uses MobileViT blocks to replace local convolution processing and performs global processing through Transformer, which reduces the number of network parameters and computational complexity, and achieves better classification results on fewer round_num.

[0069] Round_num is an important hyperparameter in the experiment and needs to be adjusted according to the actual problem and the nature of the data to achieve better training results.

[0070] The second set of experiments in this verification focuses on different model aggregation algorithms. Table 2 shows the classification efficiency of classification accuracy under different aggregation algorithms. The specific experimental results are shown in Table 2 below.

[0071] Table 2

[0072]

[0073] As mentioned above, FedAvg and FedProx, as classic algorithms for federated learning, have achieved good technical results in the experiment. FedAvg is most suitable when the data differences between devices are small, while FedProx is suitable when there are some outliers or abnormal data between devices. In this experiment, the classification efficiency of the FedProx algorithm is slightly better than that of FedAvg. It is worth noting that in this experiment, it was observed that the overall classification efficiency of the FedProx algorithm was slightly higher than that of FedAvg. This advantage mainly comes from the introduction of regularization terms in FedProx, which enhances the robustness of the algorithm to abnormal data and reduces the impact of these data on the model.

[0074] The present invention is not limited to the above-mentioned embodiments. Without departing from the essential content of the present invention, any deformation, improvement, and substitution that can be thought of by those skilled in the art shall fall within the protection scope of the claims of the present invention.

Claims

1. A Chinese painting classification training method based on horizontal federated learning, characterized in that: The following steps are involved: Step 1. Build a federated learning platform with participants based on the federated learning framework; Step 2. Use the client / server federated learning framework as the training framework to achieve multi-party joint training. Through federated learning, a global model is jointly established without transmitting data. Step 3. The server sends the current global model parameters to each client participating in the federated learning framework training; Step 4. Under the client / server architecture, each local device trains the received global model on the local dataset; Step 5. The client uses stochastic gradient descent to transmit the local model parameter updates to the central server, and then the central server uses the aggregation method to update the global model parameters; Step 6. The client repeats the process of updating the global model parameters in each round, gradually improving the performance of the global model to adapt to the local data distribution of each participant; Step 7. When all clients need to jointly train the global model, the objective function is as follows: Where: W is the weight of the model. This method selects three models for experiments, MobileNet-V2, EfficientNet and MobileViT, f (k) (W) is the local objective function of the kth client, which is used to measure the local empirical risk on the heterogeneous dataset, and l is the loss function of the global model, which is set to the cross entropy loss.

2. The Chinese painting classification training method based on horizontal federated learning according to claim 1 is characterized in that: In step 1, participants in the federated learning platform include museums, galleries, and research institutes that own local data and participate in model training, and use horizontal federated learning to classify and train the data.

3. The Chinese painting classification training method based on horizontal federated learning according to claim 1 is characterized in that: In step 2, the client is a device or entity that owns local data and participates in model training, and the server is a central server or federated learning coordinator, which is the central node responsible for coordinating and managing global model training.

4. The Chinese painting classification training method based on horizontal federated learning according to claim 1 is characterized in that: In step 2, the following steps will be performed before server training begins: Step 2.1 The server initializes a global model by integrating the local model training results of multiple decentralized institutions; Step 2.2 The server sends the initialized global model to each local device participating in federated learning.

5. The Chinese painting classification training method based on horizontal federated learning according to claim 1 is characterized in that: In step 3, the global model includes a neural network model of an input layer, multiple hidden layers and an output layer. Each layer contains a number of neurons. The parameters of the global model include weights and biases of all layers. Here, the bias is a parameter of the neuron, which is a constant used to adjust the difference between the model prediction result and the actual result.

6. The Chinese painting classification training method based on horizontal federated learning according to claim 1 is characterized in that: In step 4, under the client / server architecture, each local device trains the received global model on the local dataset: the main steps of performing a round of training include: Step 4.1 Broadcast mode: The server broadcasts the latest global model to the clients involved; Step 4.2 Local training: Once the global model is received, the client performs local training using its own trainer based on its private data; Step 4.3 Model parameter update: After local training, the client sends the trained local model parameter updates back to the central server; Step 4.4 Federated Aggregation: With the assistance of the aggregator, the central server performs federated aggregation on the received model parameter updates and optimizes the global model.

7. The Chinese painting classification training method based on horizontal federated learning according to claim 1 is characterized in that: In step 5, the aggregation methods used are FedAvg and FedProx federation algorithms.

8. The Chinese painting classification training method based on horizontal federated learning according to claim 1 is characterized in that: In step 6, it is repeated in each round. After the previous round of training is completed, multiple rounds of iterations need to be repeated until the global model reaches the expected performance or reaches the preset number of training rounds.

Images