Adaptive communication federal learning method suitable for dynamic network environment

Abstract

The invention discloses a self-adaptive communication federated learning method applicable to a dynamic network environment, which is applied to the technical field of edge intelligent computing and aims at solving the problems of low network resource utilization rate, high communication overhead and the like caused by inherent network resource limitation, network dynamic property, isomerism and the like of edge computing in the prior art. According to the invention, based on a decentralized federated learning training model, each client only needs to communicate with one neighbor client, so that network resources are more fully utilized, then a parameter screening mechanism is provided, and the clients only need to exchange screened parameters, so that the communication overhead is reduced; according to the method, the communication topology can be reconstructed in a dynamic network environment, and meanwhile, the model can be efficiently trained to reach expected precision.

Description

Technical Field

[0001] The present invention belongs to edge intelligent computing technology, and in particular to an adaptive communication federated learning technology. Background Art

[0002] With the rapid development of 5G and the Internet of Things (IoT), smartphones, wearable devices, mobile networks, etc. have become widely popular, and edge devices have become ubiquitous in modern society. Edge devices continue to generate a large amount of private data, such as photos, voice, etc., which are of great value for the development of intelligent applications. At the same time, the introduction of edge computing (EC) stores data locally and pushes more computing power to the edge of the network for data analysis. In EC, federated learning (FL) is proposed as an emerging paradigm. The federated learning framework can make full use of the rich private data in edge devices without compromising privacy, and train machine learning (ML) models while keeping the data decentralized. FL is a distributed machine learning framework combined with secure encryption technology. This type of framework is envisioned as a promising solution for achieving ubiquitous intelligence and is very suitable for applications in fields such as data security, healthcare, and medical analysis, so it has attracted much attention.

[0003] There are two main types of FL architectures: centralized federated learning (CFL) using parameter servers and decentralized federated learning (DFL) using gossip communication. CFL centralizes the model parameters trained by all clients using local data to a central server, and obtains a global model by aggregating the parameters, which is then distributed to each client. This model means that CFL is bound to be limited by single point failure risks, trust dependence, and communication bottlenecks. In order to solve the above problems, the DFL model chooses to abandon the parameter server and only exchanges parameters between adjacent clients. In this way, DFL improves fault tolerance because clients continuously update the knowledge of available nodes, enhances the robustness of the network, and reduces the risk of single point failure. In addition, DFL improves the trust problem by distributing trust among clients, which determines the trust of the entire federation, thereby greatly reducing single point attacks. And DFL allows a more even distribution of communication load among clients, thereby reducing the communication bottleneck problem to a certain extent.

[0004] However, there are still three major challenges for DFL to achieve efficient model training: 1) Limited and heterogeneous resources. In actual large-scale situations, training clients are usually distributed in different geographical locations, and there may be considerable differences in bandwidth, local data volume, and computing power. How to achieve optimal resource allocation between various clients in DFL training remains an open problem. 2) Dynamic network conditions. In practical applications, clients are usually connected to the network through wireless links. Due to the mobility of clients and the instability of links, the state of wireless links is usually time-varying. It is difficult to cope with the dynamically changing network status while maintaining a good convergence rate, which also introduces new challenges to DFL. 3) Large communication overhead. At the same time, modern models often involve millions of parameters, and high communication costs will be incurred during iterative communication, which poses a huge obstacle to the feasibility of DFL. Reference A. Koloskova, T. Lin, S. U. Stich, and M. Jaggi, “Decentralized deep learning with arbitrary communication compression,” in Proc. Int. Conf. Learn. Representations, 2020, pp. 1–22. A client selection scheme based on heterogeneous client resources is designed, but each client is still required to communicate with multiple connected peers and a fixed communication topology is used to ensure the convergence of the model, lacking comprehensive consideration of network dynamics. Reference A. M. Abdelmoniem and M. Canini, “Dc2: Delay-aware compression control for distributed machine learning,” in IEEE INFOCOM 2021-IEEE Conference on Computer Communications. IEEE, 2021, pp. 1–10. Quantization technology is used to reduce the communication load, but a fixed or identical compression ratio is assigned to all parameters, without considering the importance of specific parameters in the model, which seriously reduces the training efficiency. Therefore, it is of great practical significance to study an adaptive communication federated learning method suitable for dynamic network environments. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a federated learning method for adaptively selecting peers for communication connections and performing parameter screening by considering the importance of parameters for heterogeneous clients in a dynamic network environment.

[0006] The technical solution adopted by the present invention to solve the above technical problems is an adaptive communication federated learning method suitable for a dynamic network environment, comprising the steps of:

[0007] Steps to establish a decentralized federated learning framework: adopt a network topology without a central server, each client in the network hosts a copy of the global model parameters and uses local data for training. At the same time, only one neighbor client is selected for model parameter exchange in each round of training;

[0008] Coordinator configuration steps: The coordinator makes a comprehensive assessment of the network resources of each client. Before the client starts a round of model parameter iteration, it makes a comprehensive assessment based on the network resources of each client, selects a neighbor client as the communication object for the next round of model parameter iteration for each client, and broadcasts the configured communication topology to each client;

[0009] Model training steps: Each client performs each round of model parameter iteration according to the consensus optimization algorithm;

[0010] Model parameter exchange steps based on dynamic thresholds: After each client completes a round of model parameter iteration, it determines whether the change in the model parameters of this round reaches the set parameter change threshold. If so, the parameters that meet the change threshold are sent to the selected neighboring clients. If not, the parameters that do not meet the change threshold are not sent, and the client starts the next round of model parameter iteration.

[0011] The present invention integrates client selection and parameter screening to accelerate DFL training, makes a comprehensive evaluation of network resources, and thereby completes the adaptive peer selection of the client. At the same time, it simplifies the communication topology construction in the iteration process to the greatest extent, taking into account the parameter screening during model exchange, thereby reducing the communication overhead of training as much as possible.

[0012] The beneficial effects of the present invention are as follows: it has the advantages of low communication overhead, fast convergence speed and improved network resource utilization, and can be directly applied to Internet of Things technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Schematic diagram of the process of the embodiment method.

[0014] Figure 2 It is a comparison of the convergence performance of the method of the present invention and the baseline on the IID data set, including the comparison of the method of the present invention, the FedAvg algorithm, the DPSGD algorithm, the Choco algorithm, and the GossipFL algorithm;

[0015] in, Figure 2 (a) is the training accuracy change curve of the MNIST dataset on CNN. Figure 2(b) is the training accuracy change curve of the CIFAR-10 dataset on CNN. Figure 2 (c) is the training accuracy change curve of the CIFAR-10 dataset on ResNet-20.

[0016] Figure 3 The communication overhead comparison of the method in the present invention with the baseline on the IID data set includes the comparison of the method in the present invention, the FedAvg algorithm, the DPSGD algorithm, the Choco algorithm, and the GossipFL algorithm.

[0017] in, Figure 3 (a) is the communication overhead curve of the MNIST dataset on CNN. Figure 3 (b) is the communication overhead curve of the CIFAR-10 dataset on CNN. Figure 3 (c) is the communication overhead curve of the CIFAR-10 dataset on ResNet-20.

[0018] Figure 4 The communication overhead comparison of the proposed method to achieve the target accuracy on the IID data set is compared with the baseline, including the comparison of the proposed algorithm, FedAvg algorithm, DPSGD algorithm, Choco algorithm, and GossipFL algorithm;

[0019] in, Figure 4 (a) is a histogram of the communication overhead required to achieve target accuracy on the MNIST dataset on CNN. Figure 4 (b) is a histogram of the communication overhead required to achieve target accuracy on the CIFAR-10 dataset on CNN. Figure 4 (c) A histogram of the communication cost required to achieve target accuracy on the CIFAR-10 dataset on ResNet-20.

[0020] Figure 5 It is a comparison of the convergence performance of the method of the present invention on the non-IID data set with the baseline, including the comparison of the algorithm of the present invention, the FedAvg algorithm, the DPSGD algorithm, the Choco algorithm, and the GossipFL algorithm;

[0021] in, Figure 5 (a) is the training accuracy change curve of the MNIST dataset on CNN. Figure 5 (b) is the training accuracy change curve of the CIFAR-10 dataset on CNN.

[0022] Figure 6 The communication overhead comparison of the method of the present invention and the baseline on the non-IID data set includes the comparison of the algorithm of the present invention, the FedAvg algorithm, the DPSGD algorithm, the Choco algorithm, and the GossipFL algorithm.

[0023] in, Figure 6 (a) is the communication overhead curve of the MNIST dataset on CNN. Figure 6 (b) is the communication overhead curve of the CIFAR-10 dataset on CNN.

[0024] Figure 7 The communication overhead comparison of the proposed method to achieve the target accuracy on the non-IID data set is compared with the baseline, including the comparison of the proposed algorithm, FedAvg algorithm, DPSGD algorithm, Choco algorithm, and GossipFL algorithm;

[0025] in, Figure 7 (a) is a histogram of the communication overhead required to achieve target accuracy on the MNIST dataset on CNN. Figure 7 (b) A histogram of the communication cost required to achieve target accuracy on the CIFAR-10 dataset on CNN. DETAILED DESCRIPTION

[0026] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0027] Example method description diagram Figure 1 As shown, the specific implementation steps include:

[0028] Step 1: Data Sampling

[0029] The MNIST and CIFAR10 datasets are selected. MNIST contains 70,000 grayscale handwritten digit images of 28×28 pixels, divided into 10 classes, of which 60,000 samples are used for training and 10,000 samples are used for testing. CIFAR10 contains 60,000 color images of 32×32 pixels, also divided into 10 classes, of which 50,000 are training images and 10,000 are testing images. Except for the case of model performance evaluation on non-IID data, all datasets are uniformly distributed among clients by default.

[0030] Step 2: Build a decentralized federated learning framework

[0031] The federated learning framework can train a large number of distributed data sets collected on the client. Decentralization means that each client uses local data to train a local copy of the model parameters. Decentralized federated learning includes a set of distributed clients. Each client Each has its own local dataset D iIn the absence of a parameter server, clients use local datasets to collaboratively optimize ML models. In the present invention, each client only needs to exchange models with one neighboring client and does not need to share its original data. Formally, the loss function f of client i is i (x) can be expressed as:

[0032]

[0033] where x is the local copy of the model parameters, D i is the dataset of client i, and f(x;ξ) is the loss function calculated by a specific sample ξ. In the decentralized federated learning framework, each client trains the local model copy x using the local dataset, and then exchanges model parameters with a selected neighbor client to ensure that the two clients jointly train the same model.

[0034] Step 3: Calculate and transmit information

[0035] The coordinator is set in the algorithm, which is mainly used to calculate and transmit global information required in the training process, such as the gossip matrix. At the beginning of training, the coordinator needs to initialize the connection with the client. After the client starts training, the coordinator enters a loop iteration. Each round of iteration includes the following three steps:

[0036] 1) The coordinator aggregates some lightweight information from the client, such as bandwidth, data volume, computing power, etc.

[0037] 2) The coordinator generates the gossip matrix G k , used to determine the communication topology in the next iteration and save the latest timestamp matrix T k ;

[0038] 3) The coordinator broadcasts G to all clients k , and notifies the client to proceed to the next iteration. Compared with the model parameters with tens or hundreds of millions of bytes, the information aggregated by the coordinator is extremely concise, with less than a kilobyte of information. Therefore, the communication between the client and the coordinator does not cause any communication pressure and can be easily expanded.

[0039] Step 4: Select Client

[0040] In order to reduce communication overhead, the present invention allows each client to communicate with only one neighbor client. The coordinator comprehensively considers the network resources of different clients, including bandwidth, local data volume and computing power, when selecting the client solution. These time-varying information is processed by the coordinator in each iteration and passed to each client, guiding the client to dynamically select neighbors for communication.

[0041] Since the various resource indicators vary greatly, the entropy weight method is used in the coordinator to process the information, which can objectively determine the weights of different indicators and finally obtain the comprehensive evaluation index γ. Specifically, the bandwidth B of the neighbor client j of client i is ij Take for example, where the number of neighbor clients of client i is L. First, Calculate the sample weight, where represents the bandwidth sample ratio of client i’s neighbor client j, reflecting the relative importance of the indicator. Calculate the information entropy of the bandwidth index of client i. The larger the information entropy, the greater the difference of the index and the greater the contribution to the comprehensive evaluation. The coefficient μ is usually taken as The information entropy of the local data volume V and computing power P of client i is also calculated in the same way as bandwidth B, that is, *={B,V,P}. The next step is to determine the weight of each indicator The greater the difference, the higher the weight of the indicator. 3 (1-entropy * ) represents the (1-entropy * ), * = {B, V, P} for accumulation. Finally, the comprehensive evaluation γ of client i is calculated as follows:

[0042] γ=γ B ω B +γ V ω V +γ P ω P

[0043] where γ * Represents the sample weight, ω * Indicates the indicator weight. Combining the entropy weight method can most objectively evaluate the client based on the three indicators of bandwidth, local data volume and computing power, maximize the utilization of network resources, and further improve the convergence speed. Finally, the neighbor client with the largest comprehensive evaluation γ is used as the client for communication in this round.

[0044] Step 5: Train the model

[0045] Decentralized federated learning consists of a set of distributed clients Each client They all share the same consensus algorithm, and the training goal of the model is to minimize the loss function. Formally, the loss function of client i can be expressed as: where x is the local copy of the model parameters, D iis the dataset of client i, and f(x;ξ) is the loss function calculated by a specific sample ξ. FL seeks to minimize the global loss function F(x), which is transformed into an optimization problem Where S is the number of clients participating in model training, D i and D represent the local dataset and global dataset of client i, respectively.

[0046] Specifically, in each iteration, client i completes a round of training locally to obtain a local model Client i receives and uses the model parameters of neighbor client j in this round (kth iteration) Complete the local model update to get Client i first calculates the gossip matrix G k Select neighbor client j for this round of communication (kth iteration) and receive the neighbor model after parameter screening Based on G k Take a weighted average:

[0047]

[0048] Among them G k,ij Represents the gossip matrix G k The element values ​​corresponding to client i and client j in G are as follows: when client j is selected as the neighbor client in this round, k,ij is 1, otherwise, G k,ij is 0. Since only one neighbor client is selected for communication in each iteration, only client i is in the gossip matrix G k The corresponding i-th row has only one element value of 1, and the rest are 0.

[0049] After that, client i updates the local parameters for the next round of training (k+1) to obtain

[0050]

[0051] Where λ is the learning rate, For local models The corresponding loss function In the next round of iteration k+1, the model The parameters are transmitted.

[0052] Step 6: Set dynamic thresholds

[0053] During the model training process shown in step 5, many parameters do not change or change only slightly in a certain iteration. When the parameter change is too small, the impact on model convergence is minimal, and these parameters do not need to be sent to neighbors. As the number of algorithm iterations increases, the number of parameters that need to be sent to neighbor clients decreases, which can significantly save communication costs. The key to properly controlling the number of parameters sent between clients is to set a parameter change threshold, and only parameters with a change above this threshold are sent, and those below this threshold are not sent. If client i and client j have some updated parameters in a certain iteration, but the change does not reach the threshold, the values ​​of these parameters can be updated locally and used for the next iteration.

[0054] First determine the cumulative parameter error APE, using R k It means that all missing neighbor update parameters are limited to no higher than APE. The corresponding iteration parameter error in each iteration is called IPE, which is expressed as r k Assume that in the kth iteration, client i sends Right now Rather than but The APE of client i in the kth iteration is:

[0055]

[0056] The model iteration process of client i can be expressed as:

[0057]

[0058] Thus, we can find for:

[0059]

[0060] when When is small enough, the above formula can be approximated in derivative form:

[0061]

[0062] Assume that the second-order gradient of the loss function corresponding to the local model parameters of client i is bounded by U, that is, Then we have:

[0063]

[0064] Then the maximum value of APE can be obtained as:

[0065]

[0066] The threshold of APE after k iterations is limited to R k, the change during training is less than The parameter does not need to be sent, and this expression is the dynamic threshold.

[0067] Step 7: Swap model parameters

[0068] According to the dynamic threshold limit set in step 6, each client screens the parameters allowed to be transmitted, and then passes the parameters that meet the threshold limit to the neighboring clients selected in step 4 to ensure that the same model can be trained in the end.

[0069] The effects of the present invention are further illustrated by the following simulation and experimental verification:

[0070] Experimental results:

[0071] The present invention uses convolutional neural networks (CNNs) with different configurations to train on different data sets, namely, a CNN model applied to the MNIST data set, a CNN model for the CIFAR10 data set, and a ResNet-20 model for CIFAR10. The batch sizes of MNIST-CNN, CIFAR10-CNN, and ResNet-20 are set to 32, 50, and 16, respectively, and the number of epochs is set to 100, 300, and 150, respectively, to ensure the convergence of the model. In addition, the initial learning rates of MNIST-CNN, CIFAR10-CNN, and ResNet-20 are 0.05, 0.04, and 0.1, respectively, and all decay at a rate of 0.992.

[0072] In the experiment of the present invention, four classic algorithms were selected as performance comparison baselines, namely FedAvg, D-PSGD, ChocoPSGD and GossipFL. FedAvg is a popular centralized algorithm. For decentralized algorithms, D-PSGD and ChocoPSGD (compression ratio c = 100) were selected, as well as SNAP which also adopts the parameter selection scheme, and GossipFL, the most advanced algorithm at present, was selected.

[0073] In the present invention, the following three indicators are used to evaluate the performance of the algorithm: 1) test accuracy; 2) convergence speed; 3) traffic consumption. Specifically, in each round of training, the client calculates the accuracy of the local model on the test data set and reports it to the coordinator for averaging, and finally obtains the average test accuracy of all models as the overall test accuracy. In order to more fully demonstrate the performance comparison of the algorithms, the present invention compares the epochs and traffic consumption required by different algorithms when the test accuracy reaches the same target.

[0074] First, the training top-1 accuracy of the proposed algorithm and the baseline on the IID and non-IID datasets is compared. The three evaluation models trained on the IID dataset by the proposed algorithm all achieve the same level of accuracy as the current state-of-the-art algorithms, such as Figure 2 As shown in the figure, the algorithm of the present invention achieved 99.33%, 82.57% and 90.61% top-1 accuracy in the training of the three evaluation models, which verifies the convergence of the algorithm of the present invention. In addition, the algorithm of the present invention is also verified on non-IID data sets. In reality, most training data is distributed in non-IID data, which usually makes training more difficult. Figure 5 As shown in the figure, in the non-IID dataset, the MNIST dataset has a lower training difficulty, and most algorithms have only a slight loss of accuracy, among which the algorithm of the present invention has the least loss and almost achieves the same accuracy. Almost all algorithms show obvious loss of accuracy on the CIFAR10 dataset with non-IID characteristics, but the algorithm of the present invention still has higher accuracy in this scenario. It can be seen from the comprehensive experimental verification results that the algorithm of the present invention can achieve the same level of convergence performance on IID data and better convergence performance on non-IID data.

[0075] We select the maximum accuracy that all algorithms can achieve as the target accuracy. We also compare the epochs required to achieve the same accuracy. The fewer epochs required, the faster the convergence speed. In the IID dataset, we set the target accuracy of MNIST-CNN to 98%, and the target accuracy of CIFAR10-CNN and ResNet-20 to 80%. The performance comparison of the algorithms is shown in Figure 2. Figure 2 As shown, it can be seen that the algorithm of the present invention reaches the target accuracy faster than other algorithms. In the training of the MNIST-CNN evaluation model, all algorithms except FedAvg reach the target accuracy very quickly, and the algorithm of the present invention reaches it almost at the same time as the fastest Choco-PSGD. In CIFAR10-CNN and ResNet-20, which are more difficult to train, the algorithm of the present invention achieves the same accuracy as other algorithms while increasing the training speed by 11.7% and 29.6% respectively. The algorithm performance comparison of non-IID datasets is shown in Figure 2. Figure 6 As shown in the figure, the algorithm of the present invention achieves the fastest convergence speed in the training of all evaluation models. The target accuracy rates of MNIST-CNN and CIFAR10-CNN are 95% and 40% respectively. Choco-PSGD cannot converge, DPSGD also faces great convergence difficulties, and FedAvg can converge but performs poorly. Compared with GossipFL, which performs best in the baseline, the convergence speed of the algorithm of the present invention is improved by up to 18.1%. The experimental results prove the advantages of the algorithm of the present invention in accelerating model training.

[0076] We compared the communication overheads of different algorithms to demonstrate the communication efficiency of the algorithm of the present invention. Figure 3 and Figure 6 The accuracy of each algorithm relative to the cumulative communication overhead is shown. It can be seen that the algorithm of the present invention achieves the same accuracy on both IID and non-IID datasets while reducing traffic consumption. Figure 4 and Figure 7 The comparison of communication overhead required to achieve the same accuracy is more intuitively demonstrated, and the communication overhead of the algorithm of the present invention on each data set shows advantages. The algorithm of the present invention is improved over the most advanced algorithm GossipFL, especially in the training of ResNet-20, the communication overhead is saved by 12.2%. Compared with Choco-PSGD, which performs well in compression models, the algorithm of the present invention saves up to 56.9% of communication traffic. Compared with FedAvg and DPSGD, two algorithms that are susceptible to high communication traffic, the algorithm of the present invention is greatly improved, and the communication overhead differs by 2 orders of magnitude. The performance of the algorithm of the present invention on non-IID data sets is also excellent. When the target accuracy of MNIST-CNN and CIFAR10-CNN is set to 95% and 30% respectively, the algorithm of the present invention saves 15.5% of the traffic and achieves the same accuracy as GossipFL. FedAvg needs to spend about 10 times the communication overhead of the algorithm of the present invention, and the two algorithms DPSGD and Choco-PSGD are even difficult to achieve the target accuracy. In general, the algorithm of the present invention requires less traffic to achieve the same level of accuracy as other algorithms, and has significant advantages in communication efficiency.

[0077] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. An adaptive communication federated learning method suitable for a dynamic network environment, characterized in that: Includes steps: Steps to establish a decentralized federated learning framework: adopt a network topology without a central server, each client in the network hosts a copy of the global model parameters and uses local data for training. At the same time, only one neighbor client is selected for model parameter exchange in each round of training; Coordinator configuration steps: The coordinator makes a comprehensive assessment of the network resources of each client. Before the client starts a round of model parameter iteration, it makes a comprehensive assessment based on the network resources of each client, selects a neighbor client as the communication object for the next round of model parameter iteration for each client, and broadcasts the configured communication topology to each client; Model training steps: Each client performs each round of model parameter iteration according to the consensus optimization algorithm; Model parameter exchange steps based on dynamic thresholds: After each client completes a round of model parameter iteration, it determines whether the change in the model parameters of this round reaches the set parameter change threshold. If so, the parameters that meet the change threshold are sent to the selected neighboring clients. If not, the parameters that do not meet the change threshold are not sent, and the client starts the next round of model parameter iteration.

2. The method according to claim 1, characterized in that: The coordinator collects the bandwidth, local data volume and computing power of each client to make a comprehensive evaluation of the network resources of each client, and selects the neighbor client with the largest comprehensive evaluation value as the neighbor client for the next round of model parameter exchange.

3. The method according to claim 2, characterized in that: The network resources of each client are comprehensively evaluated and calculated as follows: c = c B oh B +g V oh V +g P oh P ; Among them, the indicator code *={B,V,P}, B represents bandwidth, V represents local data volume, P represents computing power, γ * Indicates the sample weight of indicator*, ω * Indicates the indicator weight of the indicator*.

4. The method according to claim 3, characterized in that: The sample weight of the indicator * of client i is γ * Expressed as The indicator weight of client i Among them, the information entropy of the indicator * of client i ∑ 3 (1-entropy * ) represents the (1-entropy * ) to accumulate; * ij It represents the index of client i’s neighbor client j, the number of neighbor clients of client i is L, and μ is the coefficient.

5. The method according to claim 4, characterized in that: The coefficient μ is taken 6. The method according to claim 1, characterized in that: The parameter change threshold is a dynamic threshold.

7. The method according to claim 1, characterized in that: The parameter change threshold is Among them, k is the current iteration number, λ k is the learning rate of the kth iteration, R k To limit the threshold of the cumulative parameter error after k iterations, U is the boundary of the second-order gradient of the loss function corresponding to the local model parameters of the client.

Images