Federated learning method and system using local model

The federated learning method trains local models with varying depths on edge devices, allowing terminals with limited resources to contribute to a global model, enhancing data utilization and preventing parameter mismatch.

KR103045925B1Active Publication Date: 2026-11-11SEOUL NATIONAL UNIVERSITY R&DB FOUNDATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
KR1020230057622
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-05-03
Publication Date
2026-11-11
Estimated Expiration
2043-05-03

AI Technical Summary

Technical Problem

Existing federated learning methods face inefficiencies due to terminals with insufficient hardware resources being excluded and performance degradation when training different local models with reduced channel widths, limiting their participation in model training.

Method used

A federated learning method and system that trains local models with different depths of learning channels on edge devices and uses a central server to aggregate these models, enabling terminals with limited resources to contribute to training a global model.

Benefits of technology

Enables terminals with insufficient hardware resources to participate in federated learning, enhances data utilization, and prevents parameter mismatch by training sub-models simultaneously, improving the global model's learning capability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112023049396167-PAT00001_ABST
    Figure 112023049396167-PAT00001_ABST
Patent Text Reader

Abstract

One embodiment of the present disclosure provides a federated learning method. The method comprises the steps of: each edge device training its own local model based on training data acquired by each edge device; the central server receiving training parameters from the edge devices as training results of the local models; and the central server training a global model using the training parameters, wherein the local model includes a predetermined number of consecutive sub-models among the sub-models included in the training model structure of the global model.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a federated learning method and system using local models, and more specifically, to a federated learning method and system for training a global model based on the learning results of local models having different depths of learning channels. Background Technology

[0002] Terminals participating in federated learning require both computing power to train a model using their own data and a communication network to exchange the model with a server.

[0003] Conventionally, for federated learning, various types of terminals trained and exchanged a single type of model. However, having various types of terminals train on a single type is highly inefficient and presents the problem of potentially limiting the participation of terminals with insufficient hardware resources.

[0004] Accordingly, methodologies have been studied in which, instead of all terminals learning a single type of model, they learn models of different sizes and aggregate the learned local models on a server.

[0005] For example, methods to reduce the channel width of the global model have been studied to create models of different sizes. However, there is a problem in that the performance of the global model degrades during the process of aggregating different local models created by reducing the channel width of the global model.

[0006] In addition, there is a problem in that terminals with insufficient hardware resources are excluded to aid in model training.

[0007] Accordingly, a method is needed to enable terminals with limited hardware resources to participate in federated learning and contribute to model training. The problem to be solved

[0008] The present disclosure aims to solve the problems of the aforementioned prior art by providing a federated learning method and system that trains local models with different depths of learning channels on edge devices and trains a global model based on the learning results of the local models.

[0009] The technical problems that the present invention aims to solve are not limited to the technical problems described above, and other technical problems of the present invention may be derived from the following description. means of solving the problem

[0010] As a technical means for solving the aforementioned technical problem, an embodiment according to the first aspect of the present disclosure provides a federated learning method using a plurality of edge devices and a central server connected thereto. The method comprises the steps of: each of the edge devices training their respective local models based on training data acquired by each; the central server receiving training parameters from the edge devices as training results of the local models; and the central server training a global model using the training parameters, wherein the local model includes a predetermined number of consecutive sub-models among the sub-models included in the training model structure of the global model.

[0011] Additionally, an embodiment according to a second aspect of the present disclosure provides a federated learning system comprising a plurality of edge devices and a central server connected via communication thereto. The system comprises a communication module, at least one processor, and a memory electrically connected to the processor and storing at least one code executed by the processor, wherein the memory stores code that, when executed through the processor, causes the processor to train each local model based on learning data acquired by each of the edge devices, the central server to receive learning parameters as learning results of the local models from the edge devices, and the central server to train a global model using the learning parameters, wherein the local model includes a predetermined number of consecutive sub-models among the sub-models included in the learning model structure of the global model. Effects of the invention

[0012] According to the present invention, local models with different numbers (depths) of learning channels can be trained on edge devices, and a global model can be trained based on the training results of the local models.

[0013] According to the present invention, different local models are trained on multiple terminals based on channel depth rather than channel width, thereby enabling terminals with insufficient hardware resources to participate in federated learning, and accordingly, the global model can learn more data.

[0014] In addition, according to the present invention, parameter mismatch phenomena can be prevented by training sub-models simultaneously.

[0015] In addition, according to the present invention, a knowledge distillation method can be utilized during the learning process to enable the local model having the most layer blocks to assist in the learning of the global model.

[0016] The effects of the present invention are not limited to the effects described above, but include all effects understood from the following description. Brief explanation of the drawing

[0017] FIG. 1 is a drawing illustrating a plurality of edge devices connected to a central server in communication according to an embodiment of the present invention. Figure 2 is a diagram illustrating the detailed configuration of the central server shown in Figure 1. Figure 3 is a diagram illustrating the detailed configuration of the edge device shown in Figure 1. Figure 4 is a diagram illustrating an example of a central server performing federated learning when there are 4 sub-models. FIG. 5 is a flowchart illustrating the sequence of a federated learning method using a federated learning system according to another embodiment of the present invention. Specific details for implementing the invention

[0018] The present disclosure will be described in detail below with reference to the attached drawings. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. Furthermore, the attached drawings are intended only to facilitate understanding of the embodiments disclosed herein, and the technical concept disclosed herein is not limited by the attached drawings. All terms used herein, including technical and scientific terms, should be interpreted in the sense generally understood by those skilled in the art to which the present disclosure pertains. Terms defined in advance should be interpreted as having additional meanings consistent with relevant technical literature and the present disclosure, and should not be interpreted in a highly ideal or restrictive sense unless otherwise defined.

[0019] In order to clearly explain the present disclosure in the drawings, parts unrelated to the explanation have been omitted, and the size, form, and shape of each component shown in the drawings may be varied. Throughout the specification, identical or similar parts are denoted by identical or similar reference numerals.

[0020] Suffixes such as "module" and "part" for components used in the following description are assigned or used interchangeably solely for the sake of ease of drafting the specification, and do not inherently possess distinct meanings or roles. Furthermore, in describing the embodiments disclosed in this specification, detailed descriptions of related prior art have been omitted where it is determined that such detailed descriptions could obscure the essence of the embodiments disclosed in this specification.

[0021] Throughout the specification, when it is stated that a part is "connected (connected, contacted, or coupled)" to another part, this includes not only cases where they are "directly connected (connected, contacted, or coupled)," but also cases where they are "indirectly connected (connected, contacted, or coupled)" with other members interposed therebetween. Furthermore, when it is stated that a part "includes (provides, or provides)" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but rather allows for additional "included (provided, or provided)" of other components.

[0022] Terms indicating ordinal numbers, such as first, second, etc., used herein are used solely for the purpose of distinguishing one component from another and do not limit the order or relationship of the components. For example, the first component of the present disclosure may be named the second component, and similarly, the second component may be named the first component. Singular forms used herein should be interpreted to include plural forms unless explicitly to the contrary.

[0023] FIG. 1 is a drawing illustrating a plurality of edge devices connected to a central server in communication according to an embodiment of the present invention.

[0024] Referring to FIG. 1, the federated learning system may include a central server (100) and edge devices (200). Although only one edge device (200) is shown in the drawing, a plurality of edge devices (200) may be implemented in a form connected to the central server (100) by a wireless network.

[0025] The central server (100) receives learning parameters as learning results of local models from a plurality of edge devices (200). For example, the learning parameters may include the weights of the local models of each of the plurality of edge devices (200).

[0026] The central server (100) trains a global model using learning parameters. For example, the central server (100) can train a global model based on the weights of a received local model. Here, the local model may include a predetermined number of consecutive sub-models among the sub-models included in the learning model structure of the global model.

[0027] The edge device (200) trains its local model based on the acquired training data. For example, if there are multiple edge devices (200), each of the multiple edge devices (200) can train its own local model based on the training data acquired by each.

[0028] For example, an edge device (200) may receive an initial global model or a predetermined number of consecutive sub-models included in the initial global model from a server, and perform learning based on the training data held by each edge device (200). The edge device (200) may hold a local model trained based on the training data and may transmit the training parameters of the local model to the central server (100). The central server (100) may update the global model based on the training parameters of the local model.

[0029] FIG. 2 is a drawing showing the detailed configuration of the central server (100) shown in FIG. 1, and FIG. 3 is a drawing showing the detailed configuration of the edge device (200) shown in FIG. 1.

[0030] Referring to FIGS. 2 and FIGS. 3, the central server and edge device are each configured to include a communication module, a processor, and memory.

[0031] The communication module (110) of the central server can transmit and receive data with the edge device (200). For example, the communication module (110) of the central server can transmit a trained global model to the edge device (200), and the communication module (110) of the central server can receive training parameters as the training results of the local model from the edge device (200).

[0032] The communication module (210) of the edge device can transmit and receive data with the central server (100). For example, the communication module (210) of the edge device can receive a learned global model from the central server (100), and the communication module (210) of the edge device can transmit learning parameters as the learning results of the local model to the central server (100).

[0033] The communication module (110, 210) may be a device including hardware and software necessary to transmit and receive signals, such as control signals or data signals, through a wired or wireless connection with another network device.

[0034] The processor (120, 220) may include various types of devices that control and process data. The processor (120, 220) may refer to a data processing device embedded in hardware having a physically structured circuit to perform a function expressed by code or instructions included in a program.

[0035] In one example, the processor (120, 220) may be implemented in the form of a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc., but the scope of the invention is not limited thereto.

[0036] The processor (120) of the central server (100) performs operations according to the code stored in memory (130).

[0037] The memory (130) of the central server (100) is electrically connected to the processor (120) and stores at least one code executed by the processor (120). The memory (130) stores code that causes the processor (120) to perform the following functions and procedures when executed through the processor (120).

[0038] In the memory (130) of the central server (100), code is stored that causes learning parameters to be received from edge devices as learning results of the local model.

[0039] For example, a local model may include some or all of the sub-models included in the learning model structure of a global model. If the local model includes one sub-model, it may be in the form of including the first sub-model included in the learning model structure of a global model. If the local model includes two or more sub-models, it may be in the form of continuously including the first sub-model and the next sub-model included in the learning model structure of a global model for a specified number of reasons. Each of the sub-models includes a layer block, a bottleneck layer connected to the layer block, and a logit, wherein the layer block is connected to the layer block of the next sub-model of the sub-model it includes, the bottleneck layer allows the sub-model it includes to have a normal output even without the next sub-model to which it should be connected, and the logit corresponds to the output of the sub-model.

[0040] In the memory (130) of the central server (100), code is stored that causes a global model to be trained using learning parameters. For example, the memory (130) may store code that causes a global model to be trained by integrating the learning parameters of multiple local models.

[0041] The memory (130) of the central server (100) may store code that causes the learned global model to be transmitted to edge devices.

[0042] The processor (220) of the edge device (200) performs operations according to the code stored in the memory (230).

[0043] The memory (230) of the edge device (200) may store code that causes a local model to be trained based on acquired training data.

[0044] In the memory (230) of the edge device (200), code that causes the local model to be trained using a knowledge distillation technique may be stored. For example, in the memory (230) of the edge device (200), code that causes the local model to be trained based on a cross-entropy loss function and a self-distillation loss function for the local model may be stored.

[0045] In the memory (230) of the edge device (200), code can be stored that causes the influence of the learning results between the sub-models included in the local model to affect each other when training a local model.

[0046] For example, in the memory (230) of the edge device (200), code may be stored that causes the learning result of the first sub-model, the learning result of the second sub-model, and the learning result of the third sub-model to be shared among the first to third sub-models, respectively, when training a third local model including the first to third sub-models.

[0047] Figure 4 is a diagram illustrating an example of a federated learning system performing federated learning when there are 4 sub-models.

[0048] Referring to FIG. 4, the global model includes a plurality of local models, and the local models may include at least one sub-model. In FIG. 4, for convenience of explanation, a global model having four sub-models will be assumed and described.

[0049] The global model may include first to fourth sub-models (310, 320, 330, 340). Additionally, the first sub-model (310) may constitute a first local model, the first and second sub-models (310, 320) may constitute a second local model, the first to third sub-models (310, 320, 320) may constitute a third local model, and the first to fourth sub-models (310, 320, 330, and 340) may constitute a fourth local model. A plurality of edge devices may each train a model based on one of the first to fourth local models. Here, when the entire learning structure of the global model is expressed as channels, the first local model may be described as having a shallow learning channel depth, and the fourth local model may be described as having a deep learning channel depth. Therefore, all edge devices train the first sub-model (310), and edge devices capable of training the fourth sub-model (340) must have the fourth local model. Consequently, the first local model has the shallowest depth of the training channel but a large amount of data that can be trained, and in the order of the second local model, third local model, and fourth local model, the depth of the training channel becomes deeper or larger, but the amount of data that can be trained decreases.

[0050] In other words, a local model may include at least one sub-model (310, 320, 330, 340). A local model may be in the form of including at least one sub-model, starting with a starting sub-model and then a subsequent sub-model, in a specified number of consecutive sub-models. For example, a first local model may include a first sub-model (310), a second local model may include first and second sub-models (310, 320), a third local model may include first to third sub-models (310, 320, 330, 340), and a fourth local model may include first to fourth sub-models (310, 320, 330, 340).

[0051] Each submodel can include layer blocks and bottleneck layers (bottlenecks), and has logits as output.

[0052] A layer block may be connected to one of the bottleneck layers of a sub-model including the next layer block and the layer block. For example, the first layer block (311) may be connected to the second layer block (321) of the second sub-model (320) and the first bottleneck layer (312) of the first sub-model (310).

[0053] A bottleneck layer may include learning content from the next layer block to the last layer block. For example, the first bottleneck layer (312) may include learning content from the second layer block to the fourth layer block (312, 313, 314), the second bottleneck layer (322) may include learning content from the third layer block and the fourth layer block (313, 314), and the third bottleneck layer (332) may include learning content of the fourth layer block (314).

[0054] More specifically, the first bottleneck layer (312) simply stores the learning content of the second to fourth layer blocks (321, 331, 341) so that an edge device capable of learning only the first local model can learn the first local model and immediately output the learning result.

[0055] The logit is converted into a probability value between 0 and 1 corresponding to the output of the model for the input and processed. For example, the first logit (313) of the first sub-model (310) is converted into a probability value between 0 and 1 corresponding to the output of the first sub-model (310) for the input and output, the second logit (323) is converted into a probability value of the output of the second sub-model (320) and output, the third logit (333) is converted into a probability value of the output of the third sub-model (330) and output, and the fourth logit (343) is converted into a probability value of the output of the fourth sub-model (340) and output.

[0056] To address the problem where edge devices learn less data as they include more layer blocks, edge devices can train their local models using knowledge distillation techniques. For example, an edge device can train its local model using a cross-entropy loss function (L) for the sub-models included in its local model. ce ) and self-distillation loss function(L kl A local model can be trained based on ).

[0057] When an edge device trains a local model, it basically uses a cross-entropy loss function for the sub-models existing within the local model. For example, when an edge device trains a local model containing multiple layer blocks, it trains the local model in a direction that minimizes the cross-entropy loss function for the logit, which is the output of each sub-model.

[0058] In addition, when the edge device trains a local model, it utilizes a self-distillation loss function for the sub-models existing within the local model. Specifically, when the edge device trains a local model containing multiple layer blocks, it trains the local model in a direction such that the logit, which is the output of each sub-model, becomes similar to the logits of other sub-models.

[0059] FIG. 5 is a flowchart illustrating the sequence of a federated learning method using a federated learning system according to another embodiment of the present invention.

[0060] The vehicle sensor calibration method described below can be performed by the federated learning system described above with reference to FIGS. 1 to 4. Accordingly, the description of the embodiment of the present invention described above with reference to FIGS. 1 to 4 can be applied in the same way to the embodiment described below, and any content that overlaps with the description above will be omitted. The steps described below do not necessarily have to be performed in order, the order of the steps can be set in various ways, and the steps may be performed almost simultaneously.

[0061] Referring to FIG. 5, the federated learning method includes a local model training step (S100) in which edge devices receive training parameters (S200) in which a central server receives training parameters (S300) in which a central server receives a global model training step (S300).

[0062] The edge device local model training step (S100) is a step in which each of the multiple edge devices trains its own local model based on the training data acquired by each.

[0063] For example, when edge devices train their local models using a knowledge distillation technique during the local model training step (S100), they can train the sub-models included in their local models to influence each other based on the training results. Here, each of the sub-models may be a different type of model containing a predetermined number of training channels among the training channels included in the training channel structure of the local model.

[0064] The central server receiving learning parameters step (S200) is a step in which the central server receives learning parameters from edge devices as learning results of the local model. Here, the local model includes a preset number of learning channels among the learning channels included in the learning channel structure of the global model.

[0065] The global model training step (S300) in which the central server trains the global model using training parameters is a step in which the central server trains the global model using training parameters. For example, in the global model training step (S300), the central server can upgrade the learning capability of the global model by combining the training parameters.

[0066] Although not illustrated in the drawing, the federated learning method may further include a step in which a central server transmits a trained global model to edge devices.

[0067] A person skilled in the art to which this disclosure pertains will understand that, based on the foregoing description, other specific forms can be easily modified without altering the technical spirit or essential features of this disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of this disclosure is defined by the claims set forth below, and all modifications or variations derived from the meaning and scope of the claims and equivalents thereof should be interpreted as being included within the scope of this disclosure. The scope of this application is defined by the claims set forth below rather than by the foregoing detailed description, and all modifications or variations derived from the meaning and scope of the claims and equivalents thereof should be interpreted as being included within the scope of this application.

Claims

Claim 1 A federated learning method using one or more edge devices and a central server connected thereto, comprising: a) a step in which each of the one or more edge devices trains their respective local models based on training data acquired by each; b) a step in which the central server receives training parameters as training results of the local models from the edge devices; and c) a step in which the central server trains a global model using the training parameters, wherein the local model is configured to include a predetermined number of consecutive sub-models based on the depth of the training channel rather than the channel width among the sub-models included in the training model structure of the global model, and each of the sub-models includes a layer block, a bottleneck layer, and a logit, enabling independent output without the next sub-model, and wherein the edge device performs knowledge distillation among a plurality of sub-models within the local model, wherein the training is performed such that the training result of a shallow depth sub-model capable of learning a relatively larger amount of data influences the training result of a deep depth sub-model with a smaller amount of data learning. Claim 2 A federated learning method according to claim 1, wherein the local model includes some or all of the sub-models included in the learning model structure of the global model, wherein if one sub-model is included, it is in the form of including the first sub-model included in the learning model structure of the global model, and if two or more sub-models are included, it is in the form of continuously including the first sub-model and the next sub-model included in the learning model structure of the global model as many times as specified. Claim 3 delete Claim 4 A federated learning method according to claim 1, wherein the layer block is connected to the layer block of the next sub-model of the sub-model to which it is included, the bottleneck layer allows the sub-model to have a normal output even without the next sub-model to which it is included to be connected, and the logit corresponds to the output of the sub-model. Claim 5 A federated learning system comprising a central server connected via communication with one or more edge devices that train their respective local models based on training data acquired by each device, wherein the central server comprises: a communication module; and at least one processor. A federated learning system comprising a memory electrically connected to the processor and storing at least one code executed by the processor, wherein the memory stores code that, when executed through the processor, causes the processor to receive learning parameters as learning results of the local model from the edge device and to train a global model using the learning parameters, wherein the local model is configured to include a predetermined number of consecutive sub-models based on the depth of the learning channel rather than the channel width among the sub-models included in the learning model structure of the global model, and each of the sub-models includes a layer block, a bottleneck layer, and a logit, enabling independent output without the next sub-model, and wherein the edge device performs knowledge distillation among a plurality of sub-models within the local model, wherein the learning results of shallow depth sub-models capable of learning a relatively larger amount of data influence the learning results of deep depth sub-models with a smaller amount of data learning. Claim 6 A federated learning system according to claim 5, wherein the local model includes some or all of the sub-models included in the learning model structure of the global model, wherein if one sub-model is included, it is in the form of including the first sub-model included in the learning model structure of the global model, and if two or more sub-models are included, it is in the form of continuously including the first sub-model and the next sub-model included in the learning model structure of the global model a specified number. Claim 7 delete Claim 8 A federated learning system according to claim 5, wherein the layer block is connected to the layer block of the next sub-model of the sub-model to which it is included, the bottleneck layer allows the sub-model to have a normal output even without the next sub-model to which it is included to be connected, and the logit corresponds to the output of the sub-model.