Asynchronous federated learning method and system based on dynamic contribution perception and old adaptation

Abstract

The invention discloses an asynchronous federal learning method and system based on dynamic contribution perception and old adaptation, and the method comprises the steps: receiving a corresponding contribution degree after a server broadcasts a latest global model and a contribution degree vector; judging whether the received contribution degree meets a training condition, if so, downloading a latest global model for training, and if not, continuing to use an old version global model for training; fusing the loss function, the model deviation penalty and the old suppression item, and constructing a local optimization objective function; calculating a dynamic learning rate; and executing gradient descent to update the local model, and sending the local model to the server, so that the server performs weighted aggregation on the local model by using the dynamic aggregation weight after receiving the client local model greater than or equal to a preset number, updates the client contribution degree, and performs normalization to generate a contribution degree vector. The method gives consideration to the communication efficiency and the model precision, is suitable for a large-scale distributed machine learning scene, and has the advantages of high robustness and good adaptability.

Description

Technical Field

[0001] The present invention relates to an asynchronous federated learning method and system based on dynamic contribution perception and staleness adaptation, belonging to the technical field of federated learning. Background Art

[0002] Due to its distributed computing and privacy-preserving properties, federated learning has been widely used in fields such as smart devices, the Internet of Things, and healthcare. Specifically, federated learning allows multiple clients to train models locally, with updates from each client aggregated through a server to optimize the global model. However, traditional synchronous federated learning methods have some drawbacks. In traditional synchronous federated learning, the central server must wait for all clients to complete local training and upload model updates. Consequently, offline clients or those with poor computing power can slow down global model aggregation.

[0003] Building on this foundation, asynchronous federated learning methods have emerged. Asynchronous federated learning allows the server to aggregate model updates from some clients immediately after receiving them, rather than waiting for all clients to complete training. This approach can significantly improve training efficiency, especially when there are a large number of clients or high network latency. However, existing asynchronous federated learning methods still have some challenges. First, in an asynchronous update system, local models from different clients are trained, updated, and uploaded asynchronously. Therefore, they may contain completely different fresh information and have varying degrees of staleness. Furthermore, data samples from different clients are likely not independent and identically distributed, resulting in data heterogeneity between clients. Second, due to computing power heterogeneity, clients with smaller data volumes and higher computing power appear to be relatively active, participating more frequently in global aggregation and potentially having a greater impact on model aggregation. Clients with larger data volumes and lower computing power may be neglected, resulting in reduced global model performance.

[0004] However, currently, most existing asynchronous federated learning methods adopt static weight aggregation strategies, which fail to fully consider the dynamic contribution differences and staleness effects between clients. Summary of the Invention

[0005] In view of this, the present invention provides an asynchronous federated learning method, device, system and storage medium based on dynamic contribution perception and staleness adaptation, which can significantly improve the performance of asynchronous federated learning in complex network environments and data distribution while ensuring model accuracy.

[0006] The first object of the present invention is to provide an asynchronous federated learning method based on dynamic contribution perception and staleness adaptation.

[0007] The second object of the present invention is to provide an asynchronous federated learning device based on dynamic contribution perception and staleness adaptation.

[0008] The third object of the present invention is to provide an asynchronous federated learning system based on dynamic contribution perception and staleness adaptation.

[0009] A fourth object of the present invention is to provide a computer-readable storage medium.

[0010] The first object of the present invention can be achieved by adopting the following technical solutions:

[0011] An asynchronous federated learning method based on dynamic contribution perception and staleness adaptation, applied to a client, includes:

[0012] After the server broadcasts the latest global model and contribution vector, it receives the corresponding contribution;

[0013] Determine whether the received contribution meets the training conditions. If so, download the latest global model to train the local model. If not, continue to use the old version of the global model to train the local model.

[0014] The loss function, model deviation penalty and staleness suppression term are integrated to construct the local optimization objective function;

[0015] Adjust the step size based on the client's historical average latency and calculate the dynamic learning rate;

[0016] Perform gradient descent to update the local model based on the local optimization objective function and dynamic learning rate;

[0017] The updated local model is sent to the server so that after receiving a number of client local models greater than or equal to the preset number, the server uses the dynamic aggregation weight to weighted aggregate the local models, updates the global model, and updates the client contribution according to the client participation status, and normalizes the updated contribution to generate a contribution vector.

[0018] Furthermore, the determination of whether the received contribution satisfies the training condition specifically includes:

[0019] The received contribution is compared with the local data ratio. If the received contribution is less than the local data ratio, the training condition is met; otherwise, the training condition is not met.

[0020] Furthermore, the fusion loss function, model deviation penalty and staleness suppression term are used to construct a local optimization objective function, as shown in the following formula:

[0021]

[0022] Among them, fk (w k ) is the loss function of client k, w k is the local model of client k, is the regularization term, λ is the regularization coefficient, γ t-τ ||w k || 2 is the staleness suppression term, τ is the round when client k last participated in the aggregated global model, γ is the decay coefficient, The latest global model.

[0023] Furthermore, the step size is adjusted according to the historical average delay of the client and the dynamic learning rate is calculated, specifically including:

[0024] Adjust the step size based on the historical average latency of the client, as follows:

[0025]

[0026] in, is the dynamic step size factor of client k, is the historical average delay of client k, is the delay of client k in the τth round;

[0027] Using the dynamic step factor, the dynamic learning rate is calculated as follows:

[0028]

[0029] in, is the dynamic learning rate of client k, is the global learning rate.

[0030] Furthermore, the local model is updated by performing gradient descent based on the local optimization objective function and the dynamic learning rate, as shown in the following formula:

[0031]

[0032] in, is the updated local model of client k, f k (w k ) is the loss function of client k, is the current local model of client k, is the dynamic learning rate of client k, τ is the round when client k last participated in the aggregated global model, The latest global model.

[0033] Furthermore, the method of weighted aggregation of the local model using the dynamic aggregation weight to update the global model specifically includes:

[0034] Trigger the aggregation operation and generate dynamic aggregation weights based on the client's local data ratio and the client's current normalized contribution.

[0035] Utilize dynamic aggregation weights to weight the current local model of the client and update the global model.

[0036] Furthermore, the client contribution is updated according to the client participation status, as shown in the following formula:

[0037]

[0038] in, is the current contribution of client k, is the contribution of the update, β k is the proportion of local data of client k, ρ is the global forgetting factor, τ is the round when client k last participated in the aggregated global model, and γ is the decay coefficient.

[0039] The second object of the present invention can be achieved by adopting the following technical solutions:

[0040] An asynchronous federated learning device based on dynamic contribution perception and staleness adaptation, the device comprising:

[0041] A receiving module is used to receive the corresponding contribution after the server broadcasts the latest global model and contribution vector;

[0042] The judgment module determines whether the received contribution meets the training conditions. If so, it downloads the latest global model to train the local model. If not, it continues to use the old version of the global model to train the local model.

[0043] A construction module is used to integrate the loss function, model deviation penalty and staleness suppression term to construct a local optimization objective function;

[0044] The calculation module is used to adjust the step size based on the historical average delay of the client and calculate the dynamic learning rate;

[0045] A first update module is used to perform gradient descent to update the local model according to the local optimization objective function and the dynamic learning rate;

[0046] The second update module is used to send the updated local model to the server, so that after receiving a number of client local models greater than or equal to a preset number, the server uses dynamic aggregation weights to weighted aggregate the local models, updates the global model, and updates the client contribution according to the client participation status, and normalizes the updated contribution to generate a contribution vector.

[0047] The third object of the present invention can be achieved by adopting the following technical solutions:

[0048] An asynchronous federated learning system based on dynamic contribution perception and staleness adaptation, the system comprising N clients and a server, wherein the server is connected to the N clients respectively;

[0049] The client is used to execute the above-mentioned asynchronous federated learning method;

[0050] The server is used to initialize the global model and contribution vector and broadcast them. After receiving a number of client local models greater than or equal to a preset number, it uses dynamic aggregation weights to weighted aggregate the local models, updates the global model, updates the client contributions, and broadcasts the latest global model and contribution vector.

[0051] The fourth object of the present invention can be achieved by adopting the following technical solutions:

[0052] A computer-readable storage medium stores a program, which, when executed by a processor, implements the above-mentioned asynchronous federated learning method.

[0053] The present invention has the following beneficial effects compared to the prior art:

[0054] 1. This invention significantly improves the communication efficiency and model accuracy of asynchronous federated learning through dynamic contribution perception and staleness adaptation mechanisms.

[0055] 2. The present invention fully considers the dynamic contribution differences and staleness effects among clients through the calculation of dynamic aggregation weights, thereby improving the performance of the global model.

[0056] 3. The present invention enhances the robustness of the model to non-independent and identically distributed data through regularization terms and dynamic step size adjustment, making it more adaptable.

[0057] 4. The present invention avoids the delay caused by waiting for all clients and improves training efficiency by setting a dynamic aggregation threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0059] Figure 1 This is a schematic diagram of the asynchronous federated learning system architecture based on dynamic contribution perception and staleness adaptation according to Example 1 of the present invention.

[0060] Figure 2This is a simplified flowchart of the asynchronous federated learning method based on dynamic contribution perception and staleness adaptation of Example 1 of the present invention.

[0061] Figure 3 This is a detailed flowchart of the asynchronous federated learning method based on dynamic contribution perception and staleness adaptation of Example 1 of the present invention.

[0062] Figure 4 This is a structural block diagram of an asynchronous federated learning device based on dynamic contribution perception and staleness adaptation according to embodiment 2 of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0064] Example 1:

[0065] like Figure 1 As shown, this embodiment provides a personalized federated learning system based on adaptive prototype aggregation, which includes N clients and a server S. The server S is a central server, which is connected to N clients respectively. Each client has local data D; each client also has a neural network model, which serves as a local model.

[0066] like Figure 2 and Figure 3 As shown, this embodiment also provides an asynchronous federated learning method based on dynamic contribution perception and staleness adaptation. The method is mainly implemented through the above-mentioned client and specifically includes the following steps:

[0067] S301: After the server broadcasts the latest global model and contribution vector, it receives the corresponding contribution.

[0068] In this embodiment, the server first performs initialization and parameter setting, specifically: initializing the global model Server Initialization Contribution Vector K represents the number of all clients, the initial value Set to the client's local data ratio Set the contribution attenuation factor γ, global forgetting factor ρ, and dynamic aggregation threshold N th , where N th That is the preset number. When the server receives N thGlobal aggregation begins when the local model of each client is updated.

[0069] The server broadcasts the latest global model and contribution vector S t , it is easy to understand that the latest global model and contribution vector of the first round of server broadcast are and S 0 , the latest global model and contribution vector of the t-th round broadcast are and S t .

[0070] S302: Determine whether the received contribution satisfies the training conditions.

[0071] In this embodiment, the contribution received by client k in the tth round of broadcast is The contribution and local data ratio β k For comparison, if the contribution Less than the local data ratio β k , then the training conditions are met, and the latest global model is downloaded. Perform local training, otherwise, the contribution Greater than or equal to the local data ratio β k , does not meet the training conditions, continue to use the old version of the global model for training to avoid repeated calculations and reduce communication overhead.

[0072] The following steps S303 to S305 are the process of updating the local model:

[0073] S303: Fusion loss function, model deviation penalty, and staleness suppression term to construct a local optimization objective function.

[0074] In this embodiment, client k is based on local data D k , integrating the loss function, model deviation penalty and staleness suppression term, constructing the local optimization objective function, as follows:

[0075]

[0076] Among them, f k (w k ) is the loss function of client k, such as the cross entropy loss function, w k is the local model of client k, is the regularization term, λ is the regularization coefficient, for example, λ = 0.1, which is used to limit the local model from deviating from the global model, γ t-τ ||w k || 2 is the staleness suppression term, τ is the round when client k last participated in the aggregated global model, and γ is the decay coefficient.

[0077] S304: Adjust the step size according to the historical average delay of the client and calculate the dynamic learning rate.

[0078] In this embodiment, the client k first performs dynamic step size adjustment, as shown in the following formula:

[0079]

[0080] in, is the dynamic step factor of client k, that is, the step factor of the t-th round broadcast, is the historical average delay of client k, is the delay of client k in the τth round. The higher the delay, the larger the learning rate amplification factor of the client.

[0081] Then client k uses the dynamic step factor to calculate the dynamic learning rate as follows:

[0082]

[0083] in, is the dynamic learning rate of client k, that is, the learning rate of the t-th round of broadcast, is the global learning rate.

[0084] S305: Perform gradient descent to update the local model according to the local optimization objective function and the dynamic learning rate.

[0085] In this embodiment, client k performs gradient descent to update the local model based on the local optimization objective function and dynamic learning rate, as shown in the following formula:

[0086]

[0087] in, is the updated local model of client k, f k (w k ) is the loss function of client k, is the current local model of client k.

[0088] S306: Send the updated local model to the server.

[0089] In this embodiment, the client k sends the updated local model to the server. When the server receives the local models of the client that is greater than or equal to the preset number, that is, receives m t ≥N th After the local model is generated, the dynamic aggregation weight is used to perform weighted aggregation on the local model. Specifically, the aggregation operation is triggered, combined with the client's local data ratio β k , the client's current normalized contribution Generate dynamic aggregation weights Using dynamic aggregation weights, the client's current local model is weighted and aggregated to update the global model; the server updates the client's contribution based on the client's participation status, as shown in the following formula:

[0090]

[0091] in, is the current contribution of client k, is the contribution of the update, and ρ is the global forgetting factor, which is used to suppress the weight monopoly of long-term active clients.

[0092] The server normalizes the updated contribution to generate a contribution vector, broadcasts the updated global model and contribution vector, and repeats steps S301 to S306 until the convergence index or training round is reached.

[0093] In summary, the method of the present invention achieves efficient model training in a non-independent and identically distributed data environment, adopts a dynamic aggregation strategy, performs real-time evaluation and adjustment on the model updates uploaded by the client, and introduces the client contribution vector to effectively alleviate the model obsolescence problem caused by communication delays; at the same time, the dynamic learning rate adjustment mechanism and obsolescence suppression strategy based on historical delay feedback are introduced to further improve the model convergence speed and overall performance; it takes into account both communication efficiency and model accuracy, is suitable for large-scale distributed machine learning scenarios, and has the advantages of strong robustness and good adaptability. Its comparison with the existing technology is shown in Table 1 below.

[0094] Table 1 Comparison between the method of the present invention and the prior art

[0095]

[0096]

[0097] It should be noted that although the method operations of the above embodiments are described in a particular order, this does not require or imply that the operations must be performed in that particular order, or that all of the illustrated operations must be performed to achieve the desired results. Rather, the depicted steps may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into a single step, and / or a single step may be broken down into multiple steps.

[0098] Example 2:

[0099] like Figure 4 As shown, this embodiment provides an asynchronous federated learning device based on dynamic contribution perception and staleness adaptation. The device includes a receiving module 401, a judgment module 402, a construction module 403, a calculation module 404, a first update module 405, and a second update module 406. The specific functions of each module are as follows:

[0100] Receiving module 401, configured to receive the corresponding contribution after the server broadcasts the latest global model and contribution vector;

[0101] The judgment module 402 judges whether the received contribution meets the training conditions. If so, the latest global model is downloaded to train the local model. If not, the old version of the global model is used to train the local model.

[0102] A construction module 403 is used to integrate the loss function, the model deviation penalty, and the staleness suppression term to construct a local optimization objective function;

[0103] A calculation module 404 is configured to adjust the step size based on the historical average delay of the client and calculate a dynamic learning rate;

[0104] A first updating module 405 is configured to perform gradient descent to update the local model according to the local optimization objective function and the dynamic learning rate;

[0105] The second updating module 406 is used to send the updated local model to the server, so that after receiving a number of client local models greater than or equal to a preset number, the server uses the dynamic aggregation weight to perform weighted aggregation on the local model, updates the global model, and updates the client contribution.

[0106] It should be noted that the device provided in this embodiment is only illustrated by the division of the above-mentioned functional modules. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above.

[0107] Example 3:

[0108] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the asynchronous federated learning method of the above-mentioned embodiment 1 is implemented as follows:

[0109] After the server broadcasts the latest global model and contribution vector, it receives the corresponding contribution; determines whether the received contribution meets the training conditions. If so, it downloads the latest global model to train the local model; if not, it continues to use the old version of the global model to train the local model; integrates the loss function, model deviation penalty and staleness suppression term to construct a local optimization objective function; adjusts the step size according to the historical average delay of the client and calculates the dynamic learning rate; performs gradient descent to update the local model according to the local optimization objective function and the dynamic learning rate; sends the updated local model to the server so that after receiving a number of client local models greater than or equal to the preset number, the server uses the dynamic aggregation weight to perform weighted aggregation on the local model, updates the global model, and updates the client contribution.

[0110] It should be noted that the computer-readable storage medium of the present embodiment may be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0111] In this embodiment, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or device. Furthermore, in this embodiment, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution device, apparatus, or device. The computer program contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0112] The computer readable storage medium can be written in one or more programming languages ​​or a combination thereof to execute the computer program for performing the present embodiment, including object-oriented programming languages ​​such as Java, Python, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect via the Internet).

[0113] The foregoing description is merely a preferred embodiment of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive. The scope of the present invention is defined by the appended claims, not the foregoing description, and all variations that come within the meaning and range of equivalents of the claims are intended to be included within the present invention.

Claims

1. An asynchronous federated learning method based on dynamic contribution perception and staleness adaptation, applied to the client, characterized by: The method comprises: After the server broadcasts the latest global model and contribution vector, it receives the corresponding contribution; Determine whether the received contribution meets the training conditions. If so, download the latest global model to train the local model. If not, continue to use the old version of the global model to train the local model. The loss function, model deviation penalty and staleness suppression term are integrated to construct the local optimization objective function; Adjust the step size based on the client's historical average latency and calculate the dynamic learning rate; Perform gradient descent to update the local model based on the local optimization objective function and dynamic learning rate; The updated local model is sent to the server so that after receiving a number of client local models greater than or equal to the preset number, the server uses the dynamic aggregation weight to weighted aggregate the local models, updates the global model, and updates the client contribution according to the client participation status, and normalizes the updated contribution to generate a contribution vector.

2. The asynchronous federated learning method according to claim 1, characterized in that: The determining whether the received contribution satisfies the training condition specifically includes: The received contribution is compared with the local data ratio. If the received contribution is less than the local data ratio, the training condition is met; otherwise, the training condition is not met.

3. The asynchronous federated learning method according to claim 1, characterized in that: The fusion loss function, model deviation penalty and staleness suppression term are used to construct the local optimization objective function, as shown in the following formula: Among them, f k (w k ) is the loss function of client k, w k is the local model of client k, is the regularization term, λ is the regularization coefficient, γ t-τ ||w k || 2 is the staleness suppression term, τ is the round when client k last participated in the aggregated global model, γ is the decay coefficient, It is the latest global model.

4. The asynchronous federated learning method according to claim 1, characterized in that: The step size is adjusted according to the historical average delay of the client and the dynamic learning rate is calculated, specifically including: Adjust the step size based on the historical average latency of the client, as follows: in, is the dynamic step size factor of client k, is the historical average delay of client k, is the delay of client k in the τth round; Using the dynamic step factor, the dynamic learning rate is calculated as follows: in, is the dynamic learning rate of client k, is the global learning rate.

5. The asynchronous federated learning method according to claim 1, characterized in that: According to the local optimization objective function and dynamic learning rate, gradient descent is performed to update the local model, as shown in the following formula: in, is the updated local model of client k, f k (w k ) is the loss function of client k, is the current local model of client k, is the dynamic learning rate of client k, τ is the round when client k last participated in the aggregated global model, It is the latest global model.

6. The asynchronous federated learning method according to claim 1, characterized in that: The method of weighted aggregation of the local model using the dynamic aggregation weight to update the global model specifically includes: Trigger the aggregation operation and generate dynamic aggregation weights based on the client's local data ratio and the client's current normalized contribution. Utilize dynamic aggregation weights to weight the current local model of the client and update the global model.

7. The asynchronous federated learning method according to claim 1, characterized in that: The client contribution is updated according to the client participation status as follows: in, is the current contribution of client k, is the contribution of the update, β k is the proportion of local data of client k, ρ is the global forgetting factor, τ is the round when client k last participated in the aggregated global model, and γ is the decay coefficient.

8. An asynchronous federated learning device based on dynamic contribution perception and staleness adaptation, characterized in that: The device comprises: A receiving module is used to receive the corresponding contribution after the server broadcasts the latest global model and contribution vector; The judgment module determines whether the received contribution meets the training conditions. If so, it downloads the latest global model to train the local model. If not, it continues to use the old version of the global model to train the local model. A construction module is used to integrate the loss function, model deviation penalty and staleness suppression term to construct a local optimization objective function; The calculation module is used to adjust the step size based on the historical average delay of the client and calculate the dynamic learning rate; A first update module is used to perform gradient descent to update the local model according to the local optimization objective function and the dynamic learning rate; The second update module is used to send the updated local model to the server, so that after receiving a number of client local models greater than or equal to a preset number, the server uses dynamic aggregation weights to weighted aggregate the local models, updates the global model, and updates the client contribution according to the client participation status, and normalizes the updated contribution to generate a contribution vector.

9. An asynchronous federated learning system based on dynamic contribution perception and staleness adaptation, characterized by: The system includes N clients and a server, wherein the servers are connected to the N clients respectively; The client is used to execute the asynchronous federated learning method according to any one of claims 1 to 7; The server is used to initialize the global model and contribution vector and broadcast them. After receiving a number of client local models greater than or equal to a preset number, it uses dynamic aggregation weights to weighted aggregate the local models, updates the global model, updates the client contributions, and broadcasts the latest global model and contribution vector.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the asynchronous federated learning method according to any one of claims 1 to 7 is implemented.

Images