A gradient upload method for federated learning in air computing under MIMO channels

Abstract

The present invention proposes an uplink communication design scheme for an over-the-air computing federated learning (OA-FL) system based on MIMO technology. The scheme includes a novel sparse coded multiplexing (SCoM) scheme. The scheme integrates sparse compression coding and MIMO multiplexing technology, aiming to solve the problem of high communication overhead and learning performance loss in over-the-air computing federated learning.

Description

Technical Field

[0001] The present invention belongs to the field of information and communication technology, and relates to a gradient uploading method for air computing federated learning under a MIMO channel. Background Art

[0002] The sixth generation (6G) of wireless communications will support a connection density of millions of wireless devices per square kilometer. This will provide a solid foundation for realizing the vision of ubiquitous intelligence. Developing powerful intelligent models requires leveraging the diversity of data on a large number of edge devices. A simple paradigm is to require edge devices to upload local data to a central parameter server (PS) for centralized model training. However, uploading raw data incurs huge communication overhead and may threaten user privacy. To avoid these shortcomings, federated learning (FL) is a promising alternative that enables edge devices to jointly train machine learning (ML) models while maintaining user local data. Instead of uploading raw data, in federated learning training, each edge device sends its gradient updates to the central server, which aggregates local gradients, updates the global model, and sends the global model back to the edge devices.

[0003] Since limited communication resources (such as time, bandwidth, and space) are difficult to support the communication needs of large-scale edge devices, gradient upload has become a key bottleneck for FL deployment on wireless networks. In recent years, the number of AI model parameters has been growing. For example, Resnet152 has 60 million parameters, while GPT-3 has 175 billion parameters. However, due to bandwidth and latency limitations, the available wireless communication bandwidth is usually small. For example, 1 LTE frame with 5MHz bandwidth and 10ms coherence time can only carry 50,000 symbols. Fortunately, in federated learning, compared with the local gradient of each device, the server is more concerned with the gradient after local gradient aggregation. The aggregated gradient is usually the average of all local gradients. Based on this feature of federated learning, air computing federated learning (OA-FL) is proposed, in which edge devices transmit local gradients by sharing wireless resources. Using the simulated superposition of electromagnetic waves, local gradients are aggregated in wireless transmission. Compared with the traditional orthogonal multiple access (OMA) method, the communication resources required for air computing federated learning do not increase with the number of devices, which greatly alleviates the bottleneck effect of communication on federated learning.

[0004] Due to the broad prospects of air computing federated learning, many research works have been devoted to designing more efficient air computing federated learning systems. The existing technology proposes that local gradients can be sparse, compressed and quantized before uploading to reduce communication overhead without causing significant loss of learning accuracy. Using partially orthogonal compression matrices and Turbs-CS, a low-complexity gradient sparse compression coding scheme can be implemented. The air computing federated learning system using the above scheme has lower communication overhead and faster convergence rate.

[0005] However, all existing gradient compression coding schemes are based on single-input single-output (SISO) systems. Multiple-input multiple-output (MIMO) with array signal processing has been widely recognized as a powerful technology to enhance system capacity. MIMO multiplexing transmits multiple data streams in parallel through an antenna array, which can significantly reduce the number of channels used. However, MIMO multiplexing causes interference between data streams, which destroys the aggregated gradient and test accuracy of OA-FL. The influence of inter-stream interference can be suppressed by reasonably designing the precoding matrix on the device side and the post-processing matrix on the server side. The prior art uses the pseudo-inverse matrix of the channel matrix as the precoding matrix, and uses differential geometry optimization techniques to derive a closed post-processing matrix, or selects part of the data stream according to the receiving antenna. However, the above methods are all based on channel matrix inversion, which may significantly amplify the noise and thus aggravate the aggregation error of the gradient. Especially when some devices are in deep fading, the above schemes will produce huge performance losses. Summary of the invention

[0006] The present invention proposes an uplink communication design scheme for an air computing federated learning (OA-FL) system based on MIMO technology. The scheme includes a novel sparse coded multiplexing (SCoM) scheme. The scheme integrates sparse compression coding and MIMO multiplexing technology, aiming to solve the above-mentioned problems of high communication overhead and learning performance loss in air computing federated learning.

[0007] The present invention considers an OA-FL system consisting of a parameter server (PS) and M edge devices, where there are N R antennas, each edge device has N T antennas. Although the use of MIMO multiplexing technology to transmit multiple data streams in parallel on the antenna array reduces the uplink communication overhead, it also causes the gradient to be interfered by the data streams during uploading, which in turn leads to gradient aggregation errors and affects the learning accuracy. The system uses alternating optimization (AO) and alternating direction multiplier method (ADMM) to design the MIMO transmit and receive precoding matrix to reduce the gradient aggregation error.

[0008] The technical solution adopted by the present invention comprises the following steps:

[0009] S1, such as Figure 1 As shown, the OA-FL system consists of 1 PS and M edge devices, and the global loss function is defined as

[0010]

[0011] In the formula, The number of samples in the dataset in the entire system, Q m is the number of samples in the local dataset of device m. is the network model parameter, and D is the total number of model parameters. m (·) is the local loss function of device m, which is specifically expressed as

[0012]

[0013] Where, f(θ;ξ m，n ) is about the sample ξ m，n The point-wise loss function is . Assume that the maximum communication round of FL task training is T.

[0014] S2, server (PS) generates compression matrix and flip vector Sent to all edge devices, where C represents the length of the compressed gradient. The compression matrix A = SΞ, where To select the matrix, it is obtained by randomly selecting C rows of the D / 2×D / 2-dimensional identity matrix and rearranging them; is the discrete Fourier transform (DFT) matrix, whose (d, d′)th element is expressed as is the imaginary unit. Each element of the flip vector s is uniformly sampled from a binary distribution of {-1, 1}. A and s are later used to calculate the gradient of the transmission.

[0015] In communication round t, the following steps are performed:

[0016] S3, the server and the device estimate the channel information (CSI), assuming that the channel remains unchanged when the gradient is uploaded in each communication round, and the server has a global channel information matrix

[0017] S4. The server designs the transmitting end precoding matrix by an alternating optimization method and the receiving end post-processing matrix Specifically, the server obtains the optimal

[0018]

[0019]

[0020] Where, m and m′ respectively indicate device m and device m′, q m′ =Q m′ / Q is the normalized data set size, σ noise is the variance of Gaussian white noise in the channel, is the gradient matrix G of device m′ m′ The gradient matrix G of device m m The correlation coefficient between them, the gradient matrix G m The definition of is given in SX. P0 is the transmitting power limit. [M] = {m|1≤m≤M}. tr(·) is the matrix trace operation. ||·|| F Operate to find the Frobenius norm of a matrix.

[0021] S5, receiving end post-processing matrix F (t) The optimal value of is given by the following formula:

[0022]

[0023] Among them, I is the unit matrix.

[0024] S6, each transmitting end precoding matrix Given by the following update formula

[0025] P m ←(B m +γI) -1 (C m +γ(Z m +V m ));

[0026]

[0027]

[0028] V m ←V m +Z m -P m .

[0029] in, is the intermediate variable matrix, γ is the penalty factor, Z m Auxiliary variables introduced by ADMM algorithm, ζ m 、V m is the dual variable.

[0030] S7, through iterating S5 and S6, until the objective function value of the optimization problem in S4 converges. At this time, the optimal The server will precode the matrix and the global model θ (t) Send to each edge device.

[0031] S8. Each device performs gradient descent locally and calculates the local gradient

[0032] S9, such as Figure 2 As shown in Figure 1, it is a signal flow diagram of the local gradient uploaded by SCoM to PS in one communication round. The device uploads the local gradient using the optimized precoding matrix. In round t, the edge device maps the gradient to a complex version, as given by the following formula:

[0033]

[0034] In the formula, To take the real part operation, To obtain complex number operations. The device calculates the cumulative gradient according to the following formula:

[0035]

[0036] In the formula, Represents the sparse error accumulation vector of device m in round t. Based on the accumulated gradient Device computes sparse gradients As shown in the following formula:

[0037]

[0038] Where λ∈[0,1] represents the sparsity. sp(·) preserves the accumulated gradient The λD / 2 gradient element with the largest absolute value in . The device updates the sparse error accumulation vector according to the following formula:

[0039]

[0040] The device then uses sparse gradient Calculate the normalized gradient

[0041]

[0042] Among them, ⊙ is the element-by-element product, is the sparse gradient The variance of for The dth element of . The device is based on the normalized gradient Using the matrix A mentioned in S2 compression:

[0043]

[0044] Where C is the length of the compressed gradient, is the compression ratio.

[0045] S10, to transmit multi-stream data, the device will compress the gradient Split into multiple streams, forming the following matrix

[0046]

[0047] Among them, N s is the number of data streams, For the matrix The nth row of is the number of columns of the matrix. Finally, the device is based on the optimized transmitting end precoding matrix and the gradient matrix Calculate the transmission signal matrix

[0048]

[0049] The device will Send to the MIMO channel.

[0050] S11, at the PS end, using the post-processing matrix F (t) Get the processed signal as follows

[0051]

[0052] PS for the processed matrix After vectorization, we get the following compressed sensing problem

[0053]

[0054] Among them, vec(·) is a vectorized operation, The processed matrix With error-free aggregation matrix The error matrix, and w (t) =vec(W (t)T ).

[0055] S12, such as Figure 3 As shown, the server uses the TurboCS algorithm to solve the compressed sensing problem in S11. The TurboCS algorithm iteratively solves the aggregated gradient First, module A gives the estimation results of the linear minimum mean square error (LMMSE) estimator

[0056]

[0057]

[0058] in, is the prior mean of module A, is the prior variance of module A, is the noise vector w (t) The variance of is the posterior mean of module A, is the posterior variance of module A. According to the LMMSE output, the external information given by module A is calculated.

[0059]

[0060]

[0061] in, is the external information mean of module A, is the external information variance of module A. Both are used as the prior information of module B, that is,

[0062] Then, in module B, based on the prior information The minimum mean square error (MMSE) estimator gives the posterior estimate

[0063]

[0064]

[0065] in, Calculate the conditional variance, is the posterior mean of module B, is the posterior variance of module B. The external information output by module B is

[0066]

[0067]

[0068] in, is the external information mean of module B, is the external information variance of module B. Both are used as the prior information of module B, that is, Modules AB iterate each other until Converges, and the estimated gradient is

[0069] S13 and PS are estimated according to the following formula:

[0070]

[0071] in, PS updates the model according to the following formula:

[0072]

[0073] Where η is the learning rate.

[0074] S14. The server transmits the updated global model back to each device via an error-free broadcast channel.

[0075] S15: If the communication round t>T is reached, the process ends; otherwise, the process goes to S3.

[0076] The improvements of the present invention can be summarized as follows: First, the present invention proposes a novel local gradient upload scheme SCoM for over-the-air computing federated learning, which is composed of MIMO multiplexing technology and compression coding technology. The proposed scheme significantly reduces the communication overhead of uploading gradients while achieving the same learning accuracy. Secondly, the present invention develops a low-complexity algorithm based on alternating optimization (AO) and alternating direction multiplier method (ADMM) to optimize the precoding and post-processing matrices, thereby avoiding the problem of significant increase in gradient aggregation error caused by channel inversion in existing schemes. Finally, the present invention gives the optimal number of multiplexed data streams that minimizes the gradient aggregation error, that is, the minimum number of transmitting and receiving antennas. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 :System model

[0078] Figure 2 :Schematic diagram of Turbo-CS algorithm in SCoM scheme

[0079] Figure 3 :Flowchart of device uploading gradient in SCoM solution

[0080] Figure 4 :Simulation equipment distribution diagram

[0081] Figure 5 : The curve of learning accuracy using SCoM scheme with different numbers of multiplexed data streams

[0082] Figure 6 :Curve of channel usage times when using different transmission schemes to achieve the same learning accuracy DETAILED DESCRIPTION

[0083] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.

[0084] The parameter settings of the specific method are as follows:

[0085] Consider a FL system consisting of 20 devices and a central server. The devices are evenly distributed in a circle with a radius of 100m centered on the base station, such as Figure 4 As shown. The base station height is 10 meters. The size of each task dataset is 60,000 samples, and there are 3,000 samples on each device. The experiment of the present invention trained two FL tasks, based on the MNIST and FMNIST datasets respectively. Each dataset has two data distributions, namely 1) independent and identically distributed (iid), in which all data are shuffled and then evenly distributed to 20 devices; 2) non-independent and identically distributed (non-iid), in which each device randomly selects 4 categories and then randomly extracts 750 samples from each selected category. The model of the FL task consists of a 2-layer convolutional layer (each layer contains 5x5 convolution kernels, 2x2 maximum pooling, ReLU activation function and batchnorm layer), 1 fully connected layer and 1 softmax output layer. The learning rate is set to 0.001. The training communication round is set to T=500.

[0086] According to the above parameter settings, the specific steps of the simulation are as follows:

[0087] S1, such as Figure 1 As shown, the OA-FL system consists of 1 PS and M edge devices, and the global loss function is defined as

[0088]

[0089] In the formula, The number of samples in the dataset in the entire system, Q m is the number of samples in the local dataset of device m. is the network model parameter, and D is the total number of model parameters. m (·) is the local loss function of device m, which is specifically expressed as

[0090]

[0091] Where, f(θ;ξ m，n ) is about the sample ξ m，n The point-wise loss function is . Assume that the maximum communication round of FL task training is T.

[0092] S2, server (PS) generates compression matrix and flip vector Sent to all edge devices, where C represents the length of the compressed gradient. The compression matrix A = SΞ, where To select the matrix, it is obtained by randomly selecting C rows of the D / 2×D / 2-dimensional identity matrix and rearranging them; is the discrete Fourier transform (DFT) matrix, whose (d, d′)th element is expressed as is an imaginary unit. Each element of the flip vector s is uniformly sampled from a binary distribution of {-1, 1}. A and s are later used to calculate the gradient of the transmission. In communication round t, the following steps are performed:

[0093] S3, the server and the device estimate the channel information (CSI), assuming that the channel remains unchanged when the gradient is uploaded in each communication round, and the server has a global channel information matrix

[0094] S4. The server designs the transmitting end precoding matrix by an alternating optimization method and the receiving end post-processing matrix Specifically, the server obtains the optimal

[0095]

[0096]

[0097] Where, m and m′ respectively indicate device m and device m′, q m′ =Q m′ / Q is the normalized data set size, σ noise is the variance of Gaussian white noise in the channel, is the gradient matrix G of device m′ m′ The gradient matrix G of device m m The correlation coefficient between them, the gradient matrix G m The definition of is given in SX. P0 is the transmitting power limit. [M] = {m|1≤m≤M}. tr(·) is the matrix trace operation. ||·|| F Operate to find the Frobenius norm of a matrix.

[0098] S5, receiving end post-processing matrix F (t) The optimal value of is given by the following formula:

[0099]

[0100] Among them, I is the unit matrix.

[0101] S6, each transmitting end precoding matrix Given by the following update formula

[0102] P m ←(B m +γI)-1 (C m +γ(Z m +V m ));

[0103]

[0104]

[0105] V m ←V m +Z m -P m .

[0106] in, is the intermediate variable matrix, γ is the penalty factor, Z m Auxiliary variables introduced by ADMM algorithm, ζ m 、V m is the dual variable.

[0107] S7, through iterating S5 and S6, until the objective function value of the optimization problem in S4 converges. At this time, the optimal The server will precode the matrix and the global model θ (t) Send to each edge device.

[0108] S8. Each device performs gradient descent locally and calculates the local gradient

[0109] S9, such as Figure 2 As shown in Figure 1, the signal flow diagram of the local gradient uploaded to the PS by SCoM in one communication round. The device uploads the local gradient using the optimized precoding matrix. In round t, the edge device maps the gradient to a complex version, as given by the following formula:

[0110]

[0111] In the formula, To take the real part operation, To obtain complex number operation. The device calculates the cumulative gradient according to the following formula:

[0112]

[0113] In the formula, Represents the sparse error accumulation vector of device m in round t. Based on the accumulated gradient Device computes sparse gradients As shown in the following formula:

[0114]

[0115] Where γ∈[0,1] represents the sparsity. sp(·) preserves the accumulated gradient The γD / 2 gradient element with the largest absolute value in . The device updates the sparse error accumulation vector according to the following formula:

[0116]

[0117] The device then uses sparse gradient Calculate the normalized gradient

[0118]

[0119] Among them, ⊙ is the element-by-element product, is the sparse gradient The variance of for The dth element of . The device is based on the normalized gradient Using the matrix A mentioned in S2 compression:

[0120]

[0121] Where C is the length of the compressed gradient, is the compression ratio.

[0122] S10, to transmit multi-stream data, the device will compress the gradient Split into multiple streams, forming the following matrix

[0123]

[0124] Among them, N s is the number of data streams, For the matrix The nth row of is the number of columns of the matrix. Finally, the device is based on the optimized transmitting end precoding matrix and the gradient matrix Calculate the transmission signal matrix

[0125]

[0126] The device will Send to the MIMO channel.

[0127] S11, at the PS end, using the post-processing matrix F (t) Get the processed signal as follows

[0128]

[0129] PS for the processed matrix After vectorization, we get the following compressed sensing problem

[0130]

[0131] Among them, vec(·) is a vectorized operation, The processed matrix With error-free aggregation matrix The error matrix, and w (t) =vec(W (t)T ).

[0132] S12, such as Figure 3 As shown, the server uses the TurboCS algorithm to solve the compressed sensing problem in S11. The TurboCS algorithm iteratively solves the aggregated gradient First, module A gives the estimation results of the linear minimum mean square error (LMMSE) estimator

[0133]

[0134]

[0135] in, is the prior mean of module A, is the prior variance of module A, is the noise vector w (t) The variance of is the posterior mean of module A, is the posterior variance of module A. According to the LMMSE output, the external information given by module A is calculated.

[0136]

[0137]

[0138] in, is the external information mean of module A, is the external information variance of module A. Both are used as the prior information of module B, that is,

[0139] Then, in module B, based on the prior information The minimum mean square error (MMSE) estimator gives the posterior estimate

[0140]

[0141]

[0142] in, Calculate the conditional variance, is the posterior mean of module B, is the posterior variance of module B. The external information output by module B is

[0143]

[0144]

[0145] in, is the external information mean of module B, is the external information variance of module B. Both are used as the prior information of module B, that is, Modules AB iterate each other until Converges, and the estimated gradient is

[0146] S13 and PS are estimated according to the following formula:

[0147]

[0148] in, PS updates the model according to the following formula:

[0149]

[0150] Where η is the learning rate.

[0151] S14. The server transmits the updated global model back to each device via an error-free broadcast channel.

[0152] S15: If the communication round t>T is reached, the process ends; otherwise, the process goes to S3.

[0153] exist Figure 5 In the paper, the number of multiplexed data streams N is studied. s Impact on the learning performance of the SCoM method, where the number of transmitting antennas is set to N T =4, the number of receiving antennas is set to N R =8, the sparsity is set to λ=0.05, and the number of channel usages is set to K=1584. The experimental results are the average of 10 Monte Carlo tests. Figure 5 The test accuracy is shown as a function of the number of multiplexed data streams N. S It can be seen that under the four data distributions, the peak of the model learning accuracy occurs at N S =N T This is because for a fixed number of channel usages K, a smaller N SA smaller compression ratio κ is required, which leads to more information loss in sparse coding; while a larger N S Larger compressed gradient aggregation mean square error due to the bottleneck of the number of antennas

[0154] exist Figure 6 In the present invention, the relationship between the total number of channel usages required for various transmission schemes and the relative learning accuracy is shown. Figure 6 As shown, in the four data distributions, the algorithm proposed in the present invention consumes the least communication overhead while achieving the same learning accuracy and is significantly better than all baselines, which clearly demonstrates the superiority of the scheme proposed in the present invention.

Claims

1. A gradient uploading method for over-the-air computing federated learning in MIMO channels, characterized in that: The following steps are involved: S1. Definition: The federated learning (FL) system consists of 1 central parameter server PS and M edge devices. The global loss function is defined as: In the formula, is the number of samples in the data set in the entire system, Q m is the number of samples in the local data set of device m, is the network model parameter, D is the total number of model parameters, F m (·) is the local loss function of device m: Where, f(θ;ξ m，n ) is about the sample ξ m，n The point-by-point loss function sets the maximum communication round of FL task training to T; S2, let PS generate compression matrix and flip vector Sent to all edge devices, where C represents the compressed gradient length, and the compression matrix L = SΞ, where To select the matrix, it is obtained by randomly selecting C rows of the D / 2×D / 2-dimensional identity matrix and rearranging them; is the discrete Fourier transform matrix, and its (d, d′)th element is expressed as is an imaginary unit, and each element of the flip vector s is uniformly sampled from a binary distribution of {-1, 1}; L and s are used to calculate the gradient of the transmission; S3, the server and the device estimate the channel information, assuming that the channel remains unchanged when the gradient is uploaded in each communication round, and the server has a global channel information matrix Where t refers to the communication round t; S4. The server designs the transmitting end precoding matrix by an alternating optimization method and the receiving end post-processing matrix Specifically, the server obtains the optimal Where m and m′ refer to device m and device m′ respectively, q m′ =Q m′ / Q is the normalized data set size, σ noise is the variance of Gaussian white noise in the channel, is the gradient matrix G of device m′ m′ The gradient matrix G of device m m The correlation coefficient between them, P0 is the transmitting end power limit, [M] = {m|1≤m≤M}, tr(·) is the matrix trace operation, ||·|| F To find the matrix Frobenius norm operation; S5, receiving end post-processing matrix F (t) The optimal value of is given by the following formula: Among them, I is the unit matrix; S6, each transmitting end precoding matrix Given by the following update formula P m ←(B m +γI) -1 (C m +γ(Z m +V m )); In m ←V m +Z m -P m . in, is the intermediate variable matrix, γ is the penalty factor, Z m is the auxiliary variable introduced by the alternating direction method of multipliers (ADMM) algorithm, ζ m 、V m is the dual variable; S7, through iterating S5 and S6, until the objective function value of the optimization problem in S4 converges, the optimal The server will precode the matrix and the global model θ (t) Send to each edge device; S8. Each device performs gradient descent locally and calculates the local gradient S9. The device uploads the local gradient using the optimized precoding matrix. Within round t, the edge device maps the local gradient to a complex number: In the formula, To take the real part operation, To obtain complex operations, based on The device calculates the cumulative gradient according to the following formula: In the formula, Represents the sparse error accumulation vector of device m in round t, based on the accumulated gradient Device computes sparse gradients As shown in the following formula: Among them, λ∈[0,1] represents the sparsity, and sp(·) retains the accumulated gradient The λD / 2 gradient element with the largest absolute value in The device updates the sparse error accumulation vector according to the following formula: The device then uses sparse gradient Calculate the normalized gradient Among them, ⊙ is the element-by-element product, is the sparse gradient The variance of for The dth element of the device is based on the normalized gradient Use the matrix L mentioned in S2 to compression: Where C is the length of the compressed gradient, is the compression ratio; S10, to transmit multi-stream data, the device will compress the gradient Split into multiple streams, forming the following matrix Among them, N s is the number of data streams, For the matrix The nth row of is the number of columns of the matrix. Finally, the device is based on the optimized transmitting end precoding matrix and the gradient matrix Calculate the transmission signal matrix The device will Send to MIMO channel; S11, at the PS end, using the post-processing matrix F (t) Get the processed signal as follows PS for the processed matrix After vectorization, we get the following compressed sensing problem: Among them, vec(·) is a vectorized operation, The processed matrix With error-free aggregation matrix The error matrix, and w (t) =vec(W (t)T ); S12, the server uses the TurboCS algorithm to solve the compressed sensing problem in S11, and the TurboCS algorithm iteratively solves the aggregated gradient First, module A gives the estimation results of the linear minimum mean square error (LMMSE) estimator in, is the prior mean of module A, is the prior variance of module A, is the noise vector w (t) The variance of is the posterior mean of module A, is the posterior variance of module A; according to the LMMSE output result, calculate the external information given by module A: in, is the external information mean of module A, is the external information variance of module A, and both are used as the prior information of module B, that is, Then, in module B, based on the prior information The minimum mean square error (MMSE) estimator gives the posterior estimate in, Calculate the conditional variance, is the posterior mean of module B, is the posterior variance of module B; the external information output by module B is in, is the external information mean of module B, is the external information variance of module B, and both are used as the prior information of module A, that is, Modules AB iterate each other until Converges, and the estimated gradient is S13, PS obtain the estimated aggregation gradient according to the following formula: in, PS performs the global model θ according to the following formula (t) renew: Where η is the learning rate; S14, the server transmits the updated global model back to each device via an error-free broadcast channel; S15: If the communication round t>T is reached, the process ends; otherwise, the process goes to S3.

Images