Apparatus and method for tackling data heterogeneity in federated learning using intermediate layer representation regularization

Abstract

A federated learning method performed by a server and a terminal is disclosed. The federated learning method may include the step of the server transmitting a global model to the terminal; and the step of receiving local parameters from the terminal. The local parameters may be determined by the terminal acquiring at least one of a previous model and a global model, acquiring a representation of the acquired model, and updating a current model using the representation.

Description

Technology Field

[0001] This specification discloses a neural network model training method. Background Technology

[0002] Real-world data essential for enhancing intelligent services is distributed across numerous edge devices (e.g., IoT devices, personal smartphones, or data storage spaces of different organizations). While deep learning can benefit from large datasets generated by mass collection, rising security concerns and privacy regulations may prohibit servers from acquiring data from edge devices. Consequently, this may impose some limitations on the centralized training of deep neural network models.

[0003] Federated Learning (FL), which enables edge devices to collaboratively train models without sharing data with a central server, has emerged as a viable option to meet these requirements. In particular, the Federated Averaging (FedAvg) algorithm has emerged as an approach for training models in distributed environments where data privacy is a concern. FedAvg operates by having each edge device train a local model with its own data before sending trained parameters to a server. The server aggregates the received parameters into a single global model that inherits the trained features of the local models.

[0004] In practice, significant data heterogeneity can occur in FL systems because the local data of each device varies depending on the device's characteristics and behavior. Heterogeneous data poses a major problem in federated learning, leading to slow convergence and suboptimal model performance. Prior art literature

[0005] Korean Patent Publication No. 10-2021-0150293 (December 10, 2021) The problem to be solved

[0006] To solve the above-mentioned problem, an embodiment of the present invention presents a method to prevent model performance degradation caused by heterogeneous data in federated learning. means of solving the problem

[0007] To solve the above-mentioned problem, the present specification discloses a model learning method performed by a terminal according to one embodiment. The model learning method according to one embodiment may include the step of acquiring at least one model among a previous model and a global model; the step of acquiring a representation of the acquired model; and the step of updating a current model using the representation. The terminal may be configured to include a memory and a processor.

[0008] According to one embodiment, the representation can be obtained for each intermediate layer constituting the acquired model.

[0009] According to one embodiment, a previous model, a global model, and a current model are determined, and a representation is obtained for the previous model, the global model, and the current model, and the current model can be updated based on the representation obtained for the previous model, the global model, and the current model.

[0010] According to one embodiment, the current model is updated based on a representation loss, and the representation loss may be determined based on at least one of the similarity between the representation obtained from the previous model and the representation obtained from the current model, and the similarity between the representation obtained from the current model and the representation obtained from the global model.

[0011] According to one embodiment, the representation loss can be determined for each layer constituting the current model.

[0012] According to one embodiment, the current model is updated by applying weights to the representation loss, and the weights can be determined for each layer constituting the current model.

[0013] According to one embodiment, the representation of the acquired model can be determined by performing an operation on the intermediate layer result value of the acquired model with a predetermined value.

[0014] According to one embodiment, the representation loss may be determined as a value that lowers the similarity between the representation obtained from the previous model and the representation obtained from the current model, and increases the similarity between the representation obtained from the current model and the representation obtained from the global model.

[0015] According to one embodiment, the weight may be determined based on the similarity between the representation obtained from the current model and the representation obtained from the global model.

[0016] In addition, to solve the above-mentioned problem, the present specification discloses a terminal comprising a memory and a processor according to one embodiment. The terminal can obtain at least one model among a previous model and a global model, obtain a representation of the obtained model, and update a current model using the representation.

[0017] In addition, to solve the above-mentioned problem, the present specification discloses a model learning method performed by a server according to one embodiment. The model learning method performed by the server may include the step of transmitting a global model to a terminal; and the step of receiving local parameters from the terminal. The local parameters may be determined by the terminal acquiring at least one of a previous model and a global model, acquiring a representation of the acquired model, and updating a current model using the representation. The server may be configured to include a communication unit and a processor.

[0018] The method performed by the aforementioned terminal and / or server may be provided in the form of a computer program for performing the method, recorded on a computer-readable recording medium. Effects of the invention

[0019] This specification discloses a method for introducing a normalization term into the local training process of federated learning as a simple and effective way to prevent model performance degradation caused by heterogeneous data in federated learning. The normalization term can be calculated based on representations extracted from the intermediate layers of the distributed model. For example, by utilizing the representations of all intermediate layers and assigning appropriate weights to each contribution, more granular normalization can be provided to the training process.

[0020] The FedIntR disclosed herein can be implemented by integrating normalization into the local training process. Self-supervised learning with additional loss across intermediate layers can improve the performance of the model in downstream tasks. By integrating intermediate representations into the FL process, more effective normalization for data heterogeneity issues can be implemented.

[0021] In addition, since FedIntR automatically determines the contribution of other intermediate layers to normalization based on the similarity between local and global representations, it can be considered a general approach that eliminates the need to manually select which layers to include in normalization.

[0022] Thus, distributed edge devices (e.g., IoT devices, smartphones, data storage spaces, etc.) can jointly train models capable of providing intelligent services using federated learning mechanisms that have greater tolerance to data heterogeneity problems. Brief explanation of the drawing

[0023] Figure 1 is a diagram showing an example of an integrated learning method. Figure 2 is a diagram illustrating a local loss calculation method. Figure 3 is a diagram showing the algorithm of FedIntR. FIG. 4 is a diagram showing a server-terminal structure according to one embodiment. FIG. 5 is a diagram showing the configuration of a server and a terminal according to one embodiment. FIG. 6 is a diagram illustrating a method for updating a terminal model according to one embodiment. FIG. 7 is a diagram illustrating a method for updating a server model according to one embodiment. FIG. 8 is a diagram showing data exchange between a server and a terminal according to one embodiment. Figure 9 is a block diagram illustrating a computing environment. Specific details for implementing the invention

[0024] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, this is merely illustrative and the present invention is not limited thereto.

[0025] In describing the embodiments of the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such detailed descriptions may unnecessarily obscure the essence of the present invention. Furthermore, the terms described below are defined in consideration of their functions within the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments of the present invention and should not be limiting in any way. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described.

[0026] In the following description, terms such as "transmission," "communication," "transmission," "reception," and other terms of similar meaning regarding signals or information include not only the direct transfer of signals or information from one component to another but also the transfer through other components. In particular, "transmission" or "transmitting" a signal or information to one component refers to the final destination of the signal or information and does not mean a direct destination. The same applies to the "reception" of signals or information. Furthermore, in this specification, two or more data or information are "related" means that if one data (or information) is obtained, at least a portion of another data (or information) can be obtained based thereon.

[0027] Additionally, terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0028] By utilizing representations from at least one intermediate layer and assigning appropriate weights to each contribution, more granular regularization can be provided to the training process. Below, the idea of ​​using representations extracted from intermediate layers to overcome performance degradation caused by non-IID (non-independent and identically distributed) data in FL is described. This specification presents a method for enhancing the similarity between intermediate layer representations of local and global models, which is referred to as the FedIntR (Federated learning Intermediate Representations) algorithm (hereinafter "FedIntR"). In one embodiment, FedIntR can calculate a regularization term based on contrast loss using local and global intermediate representations. Additionally, FedIntR can automatically calculate layer-specific weights to determine the extent to which each intermediate layer contributes to the regularization term. Furthermore, FedIntR may assign greater contribution weights to layers with higher similarity between global and local representations.

[0029] Figure 1 is a diagram illustrating an example of federated learning using FedIntR. FedIntR solves the following problems to form a global model Trains

[0030] [Mathematical Formula 1]

[0031]

[0032] Here is the local loss of the i-th client, is the dataset of the i-th client, N represents the total number of clients, and is. In addition to cross-entropy loss, FedIntR is a global model and local models from the previous round Add the normalization term calculated with the help of the intermediate expression.

[0033] FedIntR according to one embodiment can be implemented by integrating normalization into the local training step of FedAvg. For example, FedIntR can be implemented by integrating normalization into the local training step 2 of FedAvg. For example, FedIntR can be implemented by modifying the local training process of Vanilla FedAvg. Since Vanilla FedAvg is a known algorithm, a description is omitted.

[0034] Referring to Fig. 1, four main steps can be performed for each training round t. First, the central server [provides] the global model to client i. It can distribute (S110). Next, client i personal data Local model for It can be trained (S120). Here, the local model is a global model and previous local model It may be a model normalized by an intermediate representation. Next, the trained local model can be returned to the server (S130). Next, the server returns the global model Local parameters received from each client can be aggregated to update (S140).

[0035] Figure 2 is a diagram illustrating the local loss calculation method of FedIntR. As shown in Figure 2, the local model (210), global model (220) and previous local model Using the intermediate representation z of (230), FedIntR can calculate the normalization term (242). More specifically, in FIG. 2, the local model (210) The intermediate expression z is z 1, ..., z K It is built as a global model (220) The intermediate expression z is z- g1, ..., z g K It is described as, and the previous local model (230) The intermediate expression z is z- p 1, ..., z p K It is illustrated as such. Weights (250) and representation losses (260) can be calculated for each layer using the intermediate representation z. And a normalization term (242) can be calculated using the weights and representation losses. This normalization term (242) and the cross-entropy loss The local training loss (240) can be calculated by adding (241). An example of FedIntR can be specified in pseudocode as shown in FIG. 3.

[0036] The FedIntR algorithm of Fig. 3 is explained in more detail. The local training process (310) is for the i-th client to receive the global model from the central server. It starts by receiving. The client is the previous local model the current model After synchronizing with (the model we intend to train), cast Synchronizes with. The total of the unfolded model structure from which expressions can be extracted Assume it contains layers (e.g., convolutional or fully connected layers). Given input x, FedIntR is the current local model , previous local model , global model of Each expressed in the nth layer , and Extracts the expression. The extraction of the expression is performed by the projection head as shown in FIG. 2. ~ This can be done by passing the layer output through to obtain a representation. Here, the projection head q can be composed of at least one (dense) layer, and the obtained representation z can be obtained in the form of a numeric vector.

[0037] FedIntR can calculate the representation loss for the k-th layer using the following mathematical formula.

[0038] [Mathematical Formula 2]

[0039]

[0040] Here is the temperature parameter, and sim() is the similarity function. The following cosine similarity function can be used as the similarity function.

[0041] [Mathematical Formula 3]

[0042]

[0043] Mathematical Equation 2 is explained in more detail. Layer-specific representation loss is a local expression go Global expressions while moving further from It is recommended to make it closer to . Through this, the normalization term is a local expression and previous local expression Maximize the distance between and local representation and global expressions The distance between them can be minimized.

[0044] Layer-by-layer representation loss It is calculated for the k-th layer of the model, and each layer can have different importance. Therefore, layer-specific representation loss Different weights for each can be assigned. For the normalization term Layered weights representing the contribution is using the softmax function and It is determined based on the similarity of. For example, The nth layer is compared to other layers and If the similarity between them is higher, the representation loss of that layer A higher weight can be assigned to it. Specifically, It can be calculated as follows.

[0045] [Mathematical Formula 4]

[0046]

[0047] Here It is 1.

[0048] FedIntR is all layers About and Calculate. Then and It can be incorporated into the local training loss as a normalization term. Local loss is defined as (5) and can be calculated using the following mathematical formula.

[0049] [Mathematical Formula 5]

[0050]

[0051] Here represents the cross-entropy loss, and the second element is the balancing parameter. It is a normalization term that has

[0052] FIG. 4 is a diagram showing a server-terminal structure according to one embodiment. A federated learning method according to one embodiment may be performed by adopting the aforementioned FedIntR. A system for federated learning may be configured to include a server (410) and at least one terminal (421, ..., 423) as shown in FIG. 4. The terminal may be configured to include one or N terminals.

[0053] FIG. 5 is a diagram showing the configuration of a server and a terminal according to one embodiment. FIG. 5 is a diagram showing the minimum configuration that a server may have. As shown in FIG. 5, the server may be configured to include a processor (510), a communication unit (520), and a memory (500). The terminal may also be configured to include a processor (510), a communication unit (520), and a memory (500) as shown in FIG. 5. Through the communication unit included in the server and the terminal, the server and the terminal can perform data communication. In addition, using the processor and memory included in the server and the terminal, the server and the terminal can perform federated learning.

[0054] For example, the processor of the terminal may operate to perform the following model learning method. FIG. 6 is a diagram illustrating a model update method of a terminal according to one embodiment. The model learning method performed by the terminal according to one embodiment may include the step of obtaining at least one model among a previous model and a global model (S610), the step of obtaining a representation of the obtained model (S620), and the step of updating a current model using the representation (S630). Additionally, the terminal may receive a global model from a server using a communication unit and transmit local parameters constituting the updated current model to the server.

[0055] More specifically, the above expression can be obtained for each intermediate layer constituting the above-mentioned model. For example, the above expression can be determined by performing an operation on a predetermined value on the result value of an intermediate layer of the above-mentioned model.

[0056] In addition, a previous model, a global model, and a current model are determined, and representations are obtained for the said previous model, global model, and current model, and the said current model can be updated based on the representations obtained for the said previous model, global model, and current model.

[0057] Here, the current model is updated based on a representation loss, and the representation loss may be determined based on at least one of the similarity between the representation obtained from the previous model and the representation obtained from the current model, and the similarity between the representation obtained from the current model and the representation obtained from the global model. Here, the representation loss may be determined for each layer constituting the current model.

[0058] The above representation loss can be determined as a value that lowers the similarity between the representation obtained from the previous model and the representation obtained from the current model, and increases the similarity between the representation obtained from the current model and the representation obtained from the global model.

[0059] In addition, the current model can be updated by applying weights to the representation loss. The weights can be determined for each layer constituting the current model. The weights can be determined based on the similarity between the representation obtained from the current model and the representation obtained from the global model.

[0060] Additionally, a server according to one embodiment includes a communication unit and a processor, and each component of the server may operate to perform the following model learning method. FIG. 7 is a diagram illustrating a model update method of a server according to one embodiment. For example, the model learning method performed by the server may include the step of transmitting a global model to a terminal (S710); and the step of receiving local parameters from the terminal (S720). Here, the local parameters may be determined by the terminal acquiring at least one of a previous model and a global model, acquiring a representation of the acquired model, and updating the current model using the representation.

[0061] FIG. 8 is a diagram illustrating data exchange between a server and a terminal according to one embodiment. The server may transmit a global model to the terminal (S810). Next, the terminal may determine local parameters using the received global model (S820). This may be performed as described with reference to FIG. 6. For example, as described above, the terminal may determine local parameters by updating the model using the representations of the previous model, the global model, and the current model (S820). Next, the terminal may transmit local parameters to the server (S830).

[0062] As mentioned above, the federated learning method using FedIntR can be implemented by performing model training on the server and terminal that incorporates normalization into the local training process. Self-supervised learning with additional loss across intermediate layers can improve model performance in downstream tasks. By integrating intermediate representations into the FL process, more effective normalization for data heterogeneity issues can be implemented.

[0063] As mentioned above, representations of intermediate layers of the overall model structure can be used to normalize the local training process in federated learning. This eliminates the need for manual determination of which intermediate layers to include in the normalization process. Furthermore, more information can be integrated into the normalization process to effectively guide the local training process.

[0064] As previously mentioned, weights may be assigned to different intermediate layers of the model to determine the contribution to the normalization term. As previously mentioned, the contribution weight for each intermediate layer may be calculated as a different value for each layer. This avoids the performance degradation of the global model that would occur if the same contribution weight were assigned (i.e., taking the average of all intermediate representation losses as the normalization).

[0065] In addition, since the contribution of each intermediate layer can be calculated using the similarity between the local intermediate representation and the global intermediate representation, the contribution of each layer to the normalization term can be not determined manually.

[0066] In this specification, the term "module" may refer to a functional and structural combination of hardware for carrying out the technical concept of the present invention and software for driving said hardware. For example, the "module" may refer to a logical unit of a specific code and a hardware resource for executing said code, and does not necessarily refer to physically connected code or a single type of hardware.

[0067] FIG. 9 is a block diagram illustrating a computing environment (10) including a computing device suitable for use in exemplary embodiments. In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those described below.

[0068] The illustrated computing environment (10) includes a computing device (12). In one embodiment, the computing device (12) may be the aforementioned server or terminal. The computing device (12) includes at least one processor (14), a computer-readable storage medium (16), and a communication bus (18). The processor (14) may enable the computing device (12) to operate according to the exemplary embodiment mentioned above. For example, the processor (14) may execute one or more programs stored in the computer-readable storage medium (16). The one or more programs may include one or more computer-executable instructions, and the computer-executable instructions may be configured to enable the computing device (12) to perform operations according to the exemplary embodiment when executed by the processor (14).

[0069] A computer-readable storage medium (16) is configured to store computer-executable instructions or program code, program data and / or other suitable forms of information. A program (20) stored in the computer-readable storage medium (16) includes a set of instructions executable by a processor (14). In one embodiment, the computer-readable storage medium (16) may be memory (volatile memory such as random access memory, non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other forms of storage media that are accessed by a computing device (12) and capable of storing desired information, or a suitable combination thereof.

[0070] The communication bus (18) interconnects various other components of the computing device (12), including the processor (14) and the computer-readable storage medium (16).

[0071] The computing device (12) may also include one or more input / output interfaces (22) and one or more network communication interfaces (26) that provide interfaces for one or more input / output devices (24). The input / output interfaces (22) and the network communication interfaces (26) are connected to a communication bus (18). The input / output devices (24) may be connected to other components of the computing device (12) through the input / output interfaces (22). An exemplary input / output device (24) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (24) may be included inside the computing device (12) as a component constituting the computing device (12), or it may be connected to the computing device (12) as a separate device distinct from the computing device (12).

[0072] Although representative embodiments of the present invention have been described in detail above, those skilled in the art will understand that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined by the claims set forth below as well as equivalents thereof.

Claims

Claim 1 A method for training a model performed by a terminal comprises: a step of receiving and storing a global model from a server (wherein the global model is a model generated by the server aggregating the latest local models received from multiple terminals); a step of determining a current model to be trained based on personal data and a previous model stored in the terminal; a step of obtaining a layer-specific representation of each model by performing an operation on an intermediate layer result value corresponding to a plurality of layers of each model for each of the previous model, the global model, and the current model; a step of determining a contrast-based representation loss for each layer of the current model, comprising a term that decreases the similarity between the representation obtained from the previous model and the representation obtained from the current model, and a term that increases the similarity between the representation obtained from the current model and the representation obtained from the global model; a step of calculating weights for each layer of the current model based on the similarity between the representation obtained from the current model and the representation obtained from the global model; and a step of generating the latest local model by updating the current model based on a learning loss that includes a supervised loss and a weighted sum of the layer-specific representation loss reflecting the layer-specific weights. A model training method comprising the step of transmitting the latest local model to the server. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 delete Claim 9 delete Claim 10 A terminal comprising memory and a processor, wherein the processor receives a global model from a server and stores it in the memory (wherein the global model is a model generated by the server aggregating the latest local models received from a plurality of terminals), determines a previous model stored in the memory and a current model to be trained based on personal data, and for each of the previous model, the global model, and the current model, calculates a predetermined value on an intermediate layer result value corresponding to a plurality of layers of each model to obtain a layer-by-layer representation of each model, and for each layer of the current model, determines a contrast-based representation loss including a term that decreases the similarity between the representation obtained from the previous model and the representation obtained from the current model, and a term that increases the similarity between the representation obtained from the current model and the representation obtained from the global model, and for each layer of the current model, calculates a weight based on the similarity between the representation obtained from the current model and the representation obtained from the global model, and, for each layer of the current model, updates the current model based on a learning loss including a supervised loss and a weighted sum of the layer-by-layer representation loss reflecting the layer-by-layer weights to generate the latest local model. Claim 11 In a model training method performed by a server, a step of transmitting a global model to a terminal (wherein the global model is a model generated by the server aggregating the latest local models received from multiple terminals); A model learning method comprising the step of receiving local parameters from the terminal, wherein the local parameters are determined by the terminal receiving and storing the global model from the server, determining a current model to be trained based on personal data and a previous model stored in the terminal, and for each of the previous model, the global model, and the current model, calculating a predetermined value on an intermediate layer result value corresponding to a plurality of layers of each model to obtain a layer-specific representation of each model, and for each layer of the current model, determining a contrast-based representation loss including a term that decreases the similarity between the representation obtained from the previous model and the representation obtained from the current model, and a term that increases the similarity between the representation obtained from the current model and the representation obtained from the global model, and for each layer of the current model, calculating weights based on the similarity between the representation obtained from the current model and the representation obtained from the global model, and updating the current model based on a learning loss including a supervised loss and a weighted sum of layer-specific representation losses reflecting the layer-specific weights. Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 In a server comprising a communication unit and a processor, the processor controls the communication unit to transmit a global model to a terminal (wherein the global model is a model generated by the server aggregating the latest local models received from a plurality of terminals), and the processor controls the communication unit to receive local parameters from the terminal, wherein the local parameters are determined by the terminal receiving and storing the global model from the server, determining a previous model stored in the terminal and a current model to be trained based on personal data, and for each of the previous model, the global model, and the current model, calculating a predetermined value on an intermediate layer result value corresponding to a plurality of layers of each model to obtain a layer-specific representation of each model, and for each layer of the current model, determining a contrast-based representation loss including a term that decreases the similarity between the representation obtained from the previous model and the representation obtained from the current model, and a term that increases the similarity between the representation obtained from the current model and the representation obtained from the global model, and for each layer of the current model, calculating a weight based on the similarity between the representation obtained from the current model and the representation obtained from the global model, and the map loss and the layer-specific A server determined by updating the current model based on a learning loss including a weighted sum of layer-by-layer representation losses with weights. Claim 19 A computer-readable recording medium having a computer program recorded thereon for performing the method of either paragraph 1 or paragraph 11. Claim 20 A computer program recorded on a computer-readable recording medium, wherein the computer program is for performing the method of either claim 1 or claim 11.

Images