Expired Weight Vector Decomposition-Based Asynchronous Associative Learning System and Method

Abstract

This prevents the problem in asynchronous federative learning environments where expired components included in the client's regional model parameters are duplicated and reflected in the update of the global model. [Solution] An asynchronous federated learning system 200 that removes expired model parameters in an asynchronous federated learning environment and improves the convergence of the global model includes a global model receiving unit that receives the global model, a regional model receiving unit that receives regional model parameters learned by the client, a difference vector calculation unit that calculates a difference vector between the global model and the regional model parameters, a projection vector calculation unit that extracts a projection vector of the regional model parameters, an orthogonal vector extraction unit that extracts an orthogonal vector orthogonal to the projection vector, and a global model update unit that applies the orthogonal vector to the global model and generates a modified global model.

Claims

1. In an asynchronous federative learning system, which removes expired (weight stale) model parameters in an asynchronous federative learning environment and improves the convergence of the global model, The receiving unit receives the full-range model; A regional model receiver that receives regional model parameters learned by the client based on the received global model; A difference vector calculation unit that calculates the difference vector between the overall model and the regional model parameters; A projection vector calculation unit that extracts projection vectors of the regional model parameters based on the difference vector; An orthogonal vector extraction unit that extracts orthogonal vectors which are vectors in a direction perpendicular to the projection vector; and A whole-span model update unit that applies the orthogonal vector to the whole-span model and generates a modified whole-span model; An expired weight vector decomposition-based asynchronous associative learning system, including...

2. The aforementioned whole-area model update unit is The cosine similarity between the projection vector and the update direction of the global model is calculated. The expired weight vector decomposition-based asynchronous associative learning system according to claim 1, wherein when the cosine similarity is negative, the projection vector and the orthogonal vector are not reflected in the update of the global model.

3. The aforementioned whole-area model update unit is The expired weight vector decomposition-based asynchronous associative learning system according to claim 1, wherein if the projection vector is larger than the update vector of the global model, the projection vector is reflected in the global model by adjusting its magnitude without removing it.

4. The aforementioned whole-area model update unit is The expired weight vector decomposition-based asynchronous associative learning system according to claim 1, wherein only the excess magnitude of the projection vector is reflected in the global model, or the magnitudes of the projection vector and the orthogonal vector are adjusted by a predefined scaling factor and then reflected in the global model.

5. The aforementioned regional model receiving unit is The system receives global model parameters at the start of training and regional model parameters upon completion of training from each client. The asynchronous federated learning system based on expired weight vector decomposition according to claim 1, wherein the difference vector calculation unit calculates a learned regional update vector based on the difference between the two received model parameters.

6. The projection vector calculation unit, The expired weight vector decomposition-based asynchronous federative learning system according to claim 1, characterized in that a projection vector is calculated by performing an inner product operation on the regional model parameters with respect to the direction of the difference vector between the regional model parameters and the global model, and then multiplying the result by the unit vector again.

7. The orthogonal vector extraction unit is, The expired weight vector decomposition-based asynchronous associative learning system according to claim 1, characterized by subtracting the projection vector from the regional model parameters and calculating an orthogonal vector containing only components orthogonal to the difference vector.

8. The aforementioned whole-area model update unit is The expired weight vector decomposition-based asynchronous associative learning system according to claim 1, characterized in that it generates a modified whole-space model by summing the whole-space model and the orthogonal vector, multiplying the orthogonal vector by a predetermined whole-space weight coefficient (η), and then adding the result to the whole-space model.

9. The aforementioned regional model receiving unit is The expired weight vector decomposition-based asynchronous federative learning system according to claim 1, characterized in that it is configured to receive both the global model at the time each client starts local learning and the regional model parameters learned based on that global model, and to grasp the differences between each global model that are out of sync over time.

10. The difference vector calculation unit, The expired weight vector decomposition-based asynchronous federative learning system according to claim 1, characterized in that it calculates the parameter difference between the whole-space model at the time when the client starts local learning and the whole-space model at the current time and generates a difference vector.

11. The expired weight vector decomposition-based asynchronous federative learning system according to claim 1, further comprising a whole-field model transfer unit that transfers the modified whole-field model generated by the whole-field model update unit to a client, enabling each client to perform subsequent local learning based on the modified whole-field model.

12. Receiving the full-range model; Based on the received global model, the client receives the regional model parameters learned by the client; A step of calculating the difference vector between the overall model and the regional model parameters; A step of calculating the projection vector of the regional model parameters based on the difference vector; A step of extracting orthogonal vectors that are vectors in a direction perpendicular to the aforementioned projection vector; The steps include: applying the orthogonal vector to the entire model to generate a modified entire model; and An expired weight vector decomposition-based asynchronous federative learning method, including the step of transferring the modified full-range model to the client.

13. The step of calculating the aforementioned projection vector is: A step of calculating a projection vector by performing an inner product operation on the regional model parameters with respect to a unit vector normalized in the direction of the difference vector, and multiplying the result by the unit vector; The expired weight vector decomposition-based asynchronous associative learning method according to claim 12, characterized by including the following:

14. The step of extracting the orthogonal vector is as follows: A step of subtracting the projection vector from the regional model parameters and calculating an orthogonal vector that includes only components orthogonal to the difference vector; The expired weight vector decomposition-based asynchronous associative learning method according to claim 12, characterized by including the following:

15. The step of generating the modified global model is: A step in which a predetermined global weighting coefficient is multiplied by the orthogonal vector, the result is added to the global model, and a modified global model is generated; The expired weight vector decomposition-based asynchronous associative learning method according to claim 12, characterized by including the following:

16. In order to determine whether the aforementioned regional model parameter is an expired parameter, The step of calculating the difference vector and extracting the orthogonal vector only when the regional model parameters exceed a threshold set based on the number of updates between the global model at the time of generation and the global model at the current time, or the Euclidean distance between each vector; The expired weight vector decomposition-based asynchronous associative learning method according to claim 12, further comprising the above.

Images