Device, method, and storage medium for managing data for model in federated learning

Abstract

This electronic device may comprise: a memory that stores instructions; a communication circuit; and at least one processor. The instructions, when executed by the processor, instruct the electronic device to: obtain a reference model from a server for federated learning; provide a service by using input data obtained from a user on the basis of the reference model, while storing the input data; in response to receiving a signal requesting learning of the reference model from the server, determine whether a size of data of each of a plurality of labels in a data set stored for the service and including the input data is within a reference range; generate a model from the reference model by using the data set in which the data of the label is adjusted to have the size within the reference range on the basis of determining that the size of the data of a label among the plurality of labels is outside the reference range; and transmit the generated model to the server.

Description

Device, method, and storage medium for managing data for models in federated learning

[0001] The following descriptions relate to devices, methods, and storage media for managing data for models in federated learning.

[0002] An electronic device can collect data based on user input. The electronic device can then use the data based on the user input to create a model. For example, the model may include an artificial intelligence (AI) model. For example, the electronic device can create (or learn) the model based on federated learning. For example, federated learning may refer to a technique in which an external electronic device connected to the electronic device and the electronic device collaborate to create the model.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] An electronic device may include a memory that stores instructions. The electronic device may include a communication circuit. The electronic device may include a processor. The instructions, when executed by the processor, may cause the electronic device to obtain a reference model from a server for federated learning via the communication circuit. The instructions, when executed by the processor, may cause the electronic device to store input data obtained from a user based on the reference model in the memory while providing a service using the input data. The instructions, when executed by the processor, may cause the electronic device to determine, in response to receiving a signal requesting learning of the reference model from the server, whether the size of each of a plurality of labels in a data set that is stored for the service in the memory and includes the input data is within a reference range. The instructions, when executed by the processor, may cause the electronic device to generate a model from the reference model using the data set adjusted so that the data of the label has a size within the reference range, based on determining that the size of the data of the label among the plurality of labels is outside the reference range. The instructions, when executed by the processor, may cause the electronic device to transmit the generated model to the server via the communication circuit.

[0005] In a method performed by an electronic device, the method may include an operation of obtaining a reference model from a server for federated learning. The method may include an operation of storing input data obtained from a user while providing a service using input data based on the reference model. The method may include an operation of determining, in response to receiving a signal requesting learning of the reference model from the server, whether the size of data of each of a plurality of labels in a data set stored for the service and including the input data is within a reference range. The method may include an operation of generating a model from the reference model using the data set in which the data of the labels is adjusted to have the size within the reference range based on a determination that the size of the data of a label among the plurality of labels is outside the reference range. The method may include an operation of transmitting the generated model to the server.

[0006] A non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by a processor of an electronic device including a communication circuit, cause the electronic device to obtain a reference model from a server for federated learning via the communication circuit. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the processor, cause the electronic device to store input data acquired from a user based on the reference model in the memory while providing a service using the input data. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the processor, cause the electronic device to determine, in response to receiving a signal requesting learning of the reference model from the server, whether a size of data of each of a plurality of labels in a data set stored for the service and including the input data is within a reference range. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the processor, cause the processor to generate a model from the reference model using the data set adjusted so that the data of the label has the size within the reference range based on determining that the size of the data of the label among the plurality of labels is outside the reference range. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the processor, cause the processor to transmit the generated model to the server via the communication circuit.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0008] FIG. 2 illustrates an example of how an electronic device adjusts data for training a model in federated learning, according to one embodiment.

[0009] FIG. 3 illustrates an example of a network for coordinating data for model learning in federated learning, according to one embodiment.

[0010] FIG. 4 illustrates an example of a signal flow for a method of adjusting data for training a model in federated learning, according to one embodiment.

[0011] FIG. 5a illustrates an example of a model for generating augmentation data using initial data, according to one embodiment.

[0012] FIG. 5b illustrates an example of a method for generating extended data using initial data based on a vector representing movement distance, according to one embodiment.

[0013] FIG. 6 illustrates an example of a method for generating an aggregate model based on federated learning, according to one embodiment.

[0014] FIG. 7 illustrates an example of an operational flow for a method of adjusting data for model training in federated learning, according to one embodiment.

[0015] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0016] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0017] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled. However, this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." Conditions described as "more than" may be replaced with "more than," conditions described as "less than," and conditions described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of the elements from A (including A) to B (including B).

[0018] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0019] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0020] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0021] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0022] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0023] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0024] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0025] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0026] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0027] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0028] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0029] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0030] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0031] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0032] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0033] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0034] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0035] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0036] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0037] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0038] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0039] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0040] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0041] An artificial intelligence model (or deep learning model) may collect user data to improve accuracy and usability. For example, the artificial intelligence model may learn (or adaptively learn or adapt, train, adaptively train, create, or update) based on the user's data. In this case, if the user's data relates to the user's personal information, the collection of the data for training the artificial intelligence model may be restricted. Hereinafter, the artificial intelligence model may be referred to as a model.

[0042] The learning of the artificial intelligence model may include federated learning (FL). For example, in the FL, a server for the FL may generate (or learn) a model. The model initially generated by the server may be referred to as a base model or a reference model. The generated reference model may be transmitted from the server to electronic devices connected to the server. Each of the electronic devices may use a service based on the received reference model. For example, the service may be referred to as an AI service that utilizes the reference model. For example, the service may be provided based on the execution of a specific software application. Each of the electronic devices may collect data from the user according to the use of the service and perform learning (or updating) of the model based on the collected data. The model learned based on the data may be a personalized model with respect to the user. After the above learning is completed, each of the electronic devices can transmit parameters for the updated model to the server. For example, the parameters may include vector values ​​that constitute (or define) the updated model. For example, the parameters may represent information that constitutes the model. The server can generate a new model (or a reference model) based on the models received from each of the electronic devices. As described above, by utilizing the FL, the model can be trained using the user's actual usage information while protecting the user's personal information. Accordingly, the model can be advanced.

[0043] In the above example, an example is described in which a new reference model is generated in the server based on models of the electronic devices transmitted to the server, and one of the electronic devices other than the server may update the model of the electronic device based on a model received from at least one electronic device different from the electronic device among the electronic devices. The FL performed based on communication between the electronic devices as described above may be referred to as a P2P (peer to peer) FL.

[0044] In the above FL, the electronic device can learn (or, create, update, train) the reference model acquired from the server. At this time, the electronic device can learn the reference model based on data acquired while providing a service using the reference model. At this time, the data may be input by a user using the electronic device. The data may be referred to as input data, user data, or user input data. For example, the performance of a model learned from the reference model may be determined based on the quantity and quality of the data. For example, a data size appropriate for the size of the reference model may be required. If the size of the data is significantly different from the size of the reference model, data imbalance (e.g., underfitting or overfitting) may occur. For example, if the number of parameters of the reference model is 100,000 and the number of data is 10, overfitting may occur, making learning impossible. In addition, an appropriate ratio may be required according to the label of the data. For example, when using a model (e.g., MNIST (modified national institute of standards and technology database)) that recognizes images representing 0 to 9 and converts the recognized information into numbers corresponding to the numbers, the labels may include 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9, respectively. For example, the labels may indicate data (or output data) output from the input data based on the model or a unit that distinguishes the output data. For example, the labels may be referenced as categories, classes, or types. For example, the labels may be determined according to the provided services. In the above example, the labels may be labels for number recognition services.For example, when training the model using 100 images representing 0 and 5 images representing each of 1 to 9, the model is likely to recognize 0 regardless of the input data. This is because the training data of the model is biased toward 0.

[0045] As described above, the data imbalance problem can be referred to as non-iid (non-independent identically distributed). To address non-iid, methods can be used to share or delete at least a portion of the data causing the imbalance. However, using a data sharing method may eliminate the advantage of FL, which provides high security for users' personal information. Furthermore, using a data deletion method may limit model learning due to the reduced amount of collected data sets. In other words, FL requires time for learning because it collects real-world user data, but using a data deletion method may require even more time.

[0046] As described above, when learning the reference model (or models), the electronic device needs to manage the data set acquired (or collected) from the electronic device. Hereinafter, the device, method, and storage medium according to embodiments of the present disclosure can manage the collected data set and adjust the ratio of labels within the data set according to the size of the reference model (or models) received from the server. Accordingly, the device, method, and storage medium according to embodiments of the present disclosure can efficiently operate the FL by using data that is personalized for the user and adjusted (or refined) according to the model. In addition, the device, method, and storage medium according to embodiments of the present disclosure can generate a model with relatively high accuracy according to the FL by using the adjusted data.

[0047] FIG. 2 illustrates an example of how an electronic device adjusts data for training a model in federated learning, according to one embodiment.

[0048] The electronic device (101) of FIG. 2 may represent an example of the electronic device (101) of FIG. 1. For example, FIG. 2 illustrates an example of an electronic device (101) that is a smartphone, but the present disclosure is not limited thereto. For example, the electronic device (101) may be implemented by software or hardware and software, such as a server. The server (250) of FIG. 2 may represent a server for the FL. For example, the server (250) may be connected to a plurality of electronic devices. For example, the plurality of electronic devices may include the electronic device (101) and represent devices for the FL.

[0049] FIG. 2 illustrates examples (201, 202, 203, 204) of a method for performing the FL by discriminating (or determining) whether an adjustment is required for a data set by an electronic device (101) and adjusting at least a portion of the data set.

[0050] In example (201), according to one embodiment, the server (250) can obtain an input (210). For example, the server (250) can recognize an input (210) performed by a user. FIG. 2 illustrates examples (201, 204) of a server (250) implemented as a web page (or web), but embodiments of the present disclosure are not limited thereto.

[0051] Referring to example (201), the server (250) can set values ​​to be used for the FL. For example, the values ​​may be values ​​for setting data for learning a model to be received from each of the plurality of electronic devices for the FL. For example, the values ​​may include whether to correct (or adjust) the data, the degree of correction (or adjustment) for the data, or a reference range for the size of the data. In example (201), the server (250) can display visual objects for setting the values. In addition, for example, the server (250) can display a visual object requesting the execution of the FL. For example, based on obtaining an input (210) for the visual object requesting the execution of the FL, the server (250) can transmit a signal requesting the execution of the FL to the plurality of electronic devices.

[0052] Referring to example (202), the electronic device (101) can perform a determination on a data set stored in the electronic device (101) (or memory). For example, the electronic device (101) can perform the determination in response to receiving the signal requesting the execution of the FL. Referring to example (202), the electronic device (101) can recognize data of the data set stored in the electronic device (101). For example, the electronic device (101) can recognize first data (221), second data (222), and third data (223). For example, the first data (221) can represent data of a label A. For example, the second data (222) can represent data of a label B. For example, the third data (223) can represent data of a label C. The above A label, B label, and C label may be labels set for specific services.

[0053] According to one embodiment, the electronic device (101) can determine whether the size of each data of the labels of the data set is within a reference range. For example, the reference range can be set with respect to a reference value (225). For example, if the reference value (225) is 100, the reference range can include 90 to 110. For example, the electronic device (101) can recognize that the first data (221) of the A label is within the reference range. For example, the electronic device (101) can recognize that the second data (222) of the B label is outside the reference range. For example, the electronic device (101) can recognize that the third data (223) of the C label is outside the reference range. Referring to the above, the electronic device (101) can determine that adjustment is required for the second data (222) of the B label. Additionally, the electronic device (101) can determine that adjustment is required for the third data (223) of the C label. For example, the electronic device (101) can recognize that augmentation is required for the second data (222) of the B label, and can recognize that reduction is required for the third data (223) of the C label.

[0054] Referring to example (203), the electronic device (101) can perform data adjustment. For example, the electronic device (101) can perform the adjustment based on the result of the determination for each of the labels within the data set. For example, the electronic device (101) can perform expansion on the second data (222) of the B label. By performing the expansion, the electronic device (101) can generate expanded second data (232) from the second data (222). For example, the electronic device (101) can perform reduction on the third data (223) of the C label. By performing the reduction, the electronic device (101) can generate reduced third data (233) from the third data (223). Each of the expanded second data (232) and the reduced third data (233) can have a size within the reference range.

[0055] According to one embodiment, the electronic device (101) may utilize sampling in performing the reduction. For example, the electronic device (101) may generate reduced third data (233) by performing the sampling on the third data (223) of the C label. For example, the sampling may include random sampling.

[0056] According to one embodiment, the electronic device (101), when performing the expansion, may generate initial data and generate expanded data from the initial data using the user's input data. For example, the initial data may represent arbitrary (or virtual) data generated for at least one label requiring adjustment. For example, the initial data may be generated based on a data generation model. For example, the data generation model may include a variational auto encoder (VAE). However, the embodiments of the present disclosure are not limited thereto. For example, the user's input data may include data collected by the electronic device (101) based on input obtained from the user. For example, the input data may be collected (or acquired) and stored while providing a service using a reference model acquired from the server (250). For example, the expanded data may represent data for the at least one label generated from the arbitrary generated initial data using the input data.

[0057] For example, the electronic device (101) may use a movement distance tracking or style transferring technique to generate the extended data from the initial data. For example, the electronic device (101) may also generate the extended data based on the movement distance tracking or style transferring technique by using video information (e.g., an image acquired through a camera or an image based on a digital signal acquired through a digitizer) together with the initial data. Specific details regarding the movement distance tracking and the style transferring are described below with reference to FIGS. 4, 5A, and 5B.

[0058] Although not illustrated in FIG. 2, according to one embodiment, the electronic device (101) may perform learning (or updating, training) of the reference model using a data set on which adjustments have been performed (e.g., first data (231), expanded second data (232), and reduced third data (233)). At this time, at least a portion of the adjusted data set may include unadjusted data. For example, the first data (231) may represent data that has not been adjusted (or has not changed, is maintained) from the first data (221). In other words, the first data (231) may be the same data as the first data (221). For example, the electronic device (101) may generate a model learned from the reference model using the adjusted data set. Furthermore, according to one embodiment, the electronic device (101) may transmit the generated model to the server (250).

[0059] In the example of FIG. 2, an example is shown in which an electronic device (101) performs the determination, but the embodiments of the present disclosure are not limited thereto. For example, each of the plurality of electronic devices that received the signal from the server (250) can perform the determination.

[0060] Referring to example (204), the server (250) may generate an aggregate model based on the FL. For example, the server (250) may generate the aggregate model using models received from the plurality of electronic devices including the electronic device (101). For example, the models may include models received from each of the plurality of electronic devices and learned from each of the plurality of electronic devices. For example, the models may include models learned using the adjusted data set in the electronic device (101). For example, the server (250) may display a visual object (240) indicating that the aggregate model is generated.

[0061] In FIG. 2, examples are shown in which the operations of examples (202) and (203) are performed based on the user's input (210) to the server (250) implemented as a web page, but the embodiments of the present disclosure are not limited thereto. For example, the server (250) may recognize a specified schedule (e.g., periodic or aperiodic) as a trigger for performing the operations of examples (202) and (203) without obtaining the user's input. The server (250) may, in response to the trigger, transmit a signal requesting the execution of the FL.

[0062] Referring to the above, the electronic device, method, and storage medium according to the embodiments of the present disclosure can determine (or decide) whether adjustments are required for data acquired (or collected) from a user when using the FL, and learn (or create, update) a model for the service by performing adjustments to the data. An example of a network including an electronic device (101) and a server (250) that determine whether adjustments are required for the data and perform the adjustments is described below in FIG. 3 .

[0063] FIG. 3 illustrates an example of a network for coordinating data for model learning in federated learning, according to one embodiment.

[0064] Referring to FIG. 3, an example of a network for generating a personalized model using models acquired from external electronic devices (310, 320, 330) is illustrated. For example, FIG. 3 illustrates an example of the network including one electronic device (101), one server (250), and three external electronic devices (310, 320, 330), but embodiments of the present disclosure are not limited thereto. For example, the number of components constituting the network may vary.

[0065] Referring to FIG. 3, an exemplary situation is illustrated in which an electronic device (101), external electronic devices (310, 320, 330), and a server (250) are connected to each other based on a wired network and / or a wireless network. In FIG. 3, an example is illustrated in which the electronic device (101) and the external electronic device (310) (or the external electronic device (320), the external electronic device (330)) are not connected to each other, but this is merely for convenience of explanation, and the embodiments of the present disclosure are not limited thereto. In FIG. 3, the electronic device (101) and the external electronic device (310) (or the external electronic device (320), the external electronic device (330)) may be connected to each other. For example, the wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), or a combination thereof. For example, the wireless network may include a network such as long term evolution (LTE), 5g new radio (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low-energy (BLE), or a combination thereof. Although the electronic device (101), external electronic devices (310, 320, 330), and server (250) are illustrated as being directly connected, the electronic device (101), external electronic devices (310, 320, 330), and server (250) may be indirectly connected via one or more routers and / or access points (APs).

[0066] Referring to FIG. 3, according to one embodiment, an electronic device (101) may include at least one of a processor (301), a communication circuit (303), and a memory (305). The processor (301), the communication circuit (303), and the memory (305) may be electrically and / or operably coupled with each other by a communication bus. Hereinafter, the hardware components being operably coupled may mean that a direct connection or an indirect connection is established, wired or wireless, between the hardware components such that a second hardware component is controlled by a first hardware component among the hardware components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware components illustrated in FIG. 3 (e.g., at least a portion of the processor (301), the communication circuit (303), and the memory (305)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware components included in the electronic device (101) are not limited to those illustrated in FIG. 3. For example, the electronic device (101) may include only some of the hardware components illustrated in FIG. 3.

[0067] According to one embodiment, the processor (301) of the electronic device (101) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP), and / or a neural processing unit (NPU). The number of processors (301) may be one or more. For example, the processor (301) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core. The processor (301) of FIG. 3 may include the processor (120) of FIG. 1.

[0068] For example, the processor (301) of the electronic device (101) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to individually and / or collectively perform the various functions described below in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor is capable of executing program instructions to accomplish or perform various functions.

[0069] According to one embodiment, the communication circuit (303) of the electronic device (101) may include hardware for supporting transmission and / or reception of electrical signals between the electronic device (101), the server (250), and external electronic devices (310, 320, 330). The communication circuit (303) may include, for example, at least one of a modem (modulator and demodulator (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (303) may support transmission and / or reception of electrical signals based on various types of communication means, such as Ethernet, Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). The communication circuit (303) of FIG. 3 may include the communication module (190) and / or the antenna module (197) of FIG. 1.

[0070] According to one embodiment, the memory (305) of the electronic device (101) may include a hardware component for storing data and / or instructions input to and / or output from the processor (301). The memory (305) may include, for example, a volatile memory such as a random-access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multimedia card (eMMC). The memory (305) of FIG. 3 may include the memory (130) of FIG. 1.

[0071] Although not shown in FIG. 3, according to one embodiment, the electronic device (101) may include a display. For example, the display of the electronic device (101) may output visualized information to a user. The number of displays included in the electronic device (101) may be one or more. For example, the display may be controlled by a processor (301) and / or a graphic processing unit (GPU) (not shown) to output visualized information to a user. The display may include a flat panel display (FPD) and / or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), a digital mirror device (DMD), one or more light emitting diodes (LEDs), and / or a micro LED. The LED may include an organic LED (OLED). The display may include the display module (160) of FIG. 1.

[0072] Although not shown in FIG. 3, according to one embodiment, the electronic device (101) may include a camera. The camera may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. A plurality of optical sensors included in the camera may be arranged in the form of a two-dimensional array. The camera may acquire electrical signals from each of the plurality of optical sensors substantially simultaneously, and generate an image corresponding to light reaching the optical sensors of the two-dimensional array, the image including a plurality of pixels arranged two-dimensionally.

[0073] In addition, although not shown in FIG. 3, the electronic device (101) may include a sensor. For example, the sensor may include at least one sensor. For example, the sensor may include at least a portion of the sensor module (176) of FIG. 1. For example, the sensor may include an IMU (or IMU sensor). For example, the sensor may include a gyro sensor, a gravity sensor, and / or an acceleration sensor. For example, the sensor may include a health sensor, a touch sensor, or a digitizer (or a digitizer sensor, a digitizer sensor board). For example, the electronic device (101) may obtain input data (or data used for model learning) according to the use of a service by a user of the electronic device (101) based on the camera or the sensor. For example, the input data may include information acquired while the user uses the electronic device (101) (e.g., data acquired through a motion sensor (e.g., a gyro sensor, a gravity sensor, an acceleration sensor), data acquired based on the health sensor, data acquired through the touch sensor, or data acquired through the digitizer). Using the input data, the electronic device (101) may perform learning (or adaptive training) of a model based on the FL.

[0074] Additionally, although not illustrated in FIG. 3, the electronic device (101) according to one embodiment may include an output means for outputting information in a form other than a visualized form. For example, the electronic device (101) may include a speaker for outputting an acoustic signal. For example, the electronic device (101) may include a motor for providing haptic feedback based on vibration.

[0075] Referring to FIG. 3, according to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (301) of the electronic device (101) may be stored in the memory (305) of the electronic device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, when an application is installed in the electronic device (e.g., the electronic device (101)), it may mean that one or more instructions provided in the form of an application are stored in the memory (305), and that the one or more applications are stored in a format executable by the processor of the electronic device (e.g., a file having an extension designated by the operating system of the electronic device (101)). According to one embodiment, the electronic device (101) may execute one or more instructions stored in the memory (305) to perform the operation of FIG. 4 or FIG. 7.

[0076] Referring to FIG. 3, programs installed in the electronic device (101) may be classified into one of different layers, including an application layer, a framework layer, and / or a hardware abstraction layer (HAL), based on a target. For example, programs (e.g., drivers) designed to target the hardware of the electronic device (101) (e.g., the communication circuit (303), the display, the sensor) may be classified within the hardware abstraction layer. For example, programs (e.g., a data determination module (305-1), a data adjustment module (305-3), and / or a model training module (305-5)) designed to target at least one of the hardware abstraction layer and / or the application layer may be classified within the framework layer. Programs classified into the framework layer may provide an executable API (Application Programming Interface) based on other programs.

[0077] For example, within the application layer, programs designed to target users controlling the electronic device (101) may be classified. For example, programs classified within the application layer may include applications that provide services to the user. However, embodiments of the present disclosure are not limited thereto. For example, programs classified within the application layer may call APIs to cause the execution of functions supported by programs classified within the framework layer.

[0078] According to one embodiment, the electronic device (101) may store user input data in the memory (305). For example, the user input data may include at least one of data acquired through a motion sensor (e.g., a gyro sensor, a gravity sensor, an acceleration sensor), data acquired based on the health sensor, data acquired through the touch sensor, data acquired through the digitizer, data acquired through the camera, data acquired through an input device (e.g., a speaker), or data processed from the data. For example, the electronic device (101) may acquire (or collect) the input data based on the user's input. For example, the electronic device (101) may acquire (or collect) the input data while providing a service based on a reference model acquired from the server (250). For example, the memory (305) may store the input data for the service. For example, multiple labels may be defined for the service, and the input data may be stored in memory (305) for a corresponding label among the multiple labels. The input data stored in the multiple labels may be referred to as a data set. For example, the memory (305) may store input data for multiple services.

[0079] Referring to FIG. 3, an electronic device (101) according to one embodiment can determine whether to adjust the data set stored in the memory (305) based on the execution of the data determination module (305-1). For example, the electronic device (101) can determine whether to adjust data of each of the plurality of labels in the data set. For example, assume that the plurality of labels include a first label, a second label, and a third label. However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) can determine whether to adjust first data of the first label. For example, the electronic device (101) can determine whether to adjust second data of the second label. For example, the electronic device (101) can determine whether to adjust third data of the third label. For example, the electronic device (101) can determine whether to adjust the size of the first data is within a reference range. For example, the electronic device (101) can determine whether the size of the second data is within the reference range. For example, the electronic device (101) can determine whether the size of the third data is within the reference range.

[0080] According to one embodiment, the data determination module (305-1) may be included in a function (or operation) based on a software application stored in the memory (305). For example, when the function of the software application is executed while the electronic device (101) is running the software application, the data determination module (305-1) may be executed. Accordingly, the data determination module (305-1) may determine whether to classify and adjust a label for input data acquired based on the user's input (e.g., intentional or unintentional operation). For example, when the software application is a note application, the electronic device (101) may acquire the user's input (e.g., handwriting) and execute a function of the software application that converts the handwriting into text. While executing the above function, the electronic device (101) can determine a label according to the text based on the execution of the data determination module (305-1) and determine whether data for the label is adjusted.

[0081] According to one embodiment, the reference range may be set for a reference value. For example, the reference value may be determined based on the size of the reference model acquired from the server (250). For example, the size of the reference model may be determined based on the number of parameters constituting the reference model. For example, when the number of parameters of the reference model is 10,000 to 50,000, the reference value may be 100, and the reference range may be 90 to 110. For example, when the number of parameters of the reference model is 50,000 to 100,000, the reference value may be 500, and the reference range may be 470 to 530. For example, when the number of parameters of the reference model is 100,000 to 1,000,000, the reference value may be 1000, and the reference range may be 900 to 1,100. However, the above examples are merely for convenience of explanation, and the embodiments of the present disclosure are not limited thereto. Alternatively, for example, the reference value and the reference range may have designated values. For example, the designated values ​​may be determined based on input to the server (250) that generates the reference model. For example, the designated values ​​may be referenced as design values ​​determined when generating the reference model.

[0082] According to one embodiment, the electronic device (101) may determine that adjustment (or expansion) of the data is required when the size of data of each of the plurality of labels of the data set is less than the reference range. For example, the electronic device (101) may determine that expansion of the data is required when data of a label among the plurality of labels of the data set is outside the reference range and the data of the label is smaller than the reference range. For example, when the size of the first data is less than the minimum value of the reference range, the electronic device (101) may determine that expansion is required. Furthermore, according to one embodiment, the electronic device (101) may determine that adjustment (or reduction) of the data is required when the size of the data exceeds the reference range. For example, the electronic device (101) may determine that reduction of the data is required when data of a label among the plurality of labels of the data set is outside the reference range and the data of the label is larger than the reference range. For example, if the size of the second data exceeds the maximum value of the reference range, the electronic device (101) may determine that the reduction is necessary. Furthermore, according to one embodiment, if the size of the data is within the reference range, the electronic device (101) may determine that no adjustment of the data is necessary. For example, if the size of the third data is greater than or equal to the minimum value of the reference range and less than or equal to the maximum value, the electronic device (101) may determine that no adjustment of the third data is necessary.

[0083] Referring to FIG. 3, an electronic device (101) according to one embodiment may perform adjustments on data requiring adjustment based on the execution of a data adjustment module (305-3). In the example, the electronic device (101) may perform adjustments on the first data and adjustments on the second data. For example, the adjustments may include reductions or expansions of the data.

[0084] According to one embodiment, the electronic device (101) may perform reduction on data. For example, if the size of the second data exceeds the maximum value of the reference range, the electronic device (101) may determine that reduction is necessary and perform reduction on the second data. For example, the electronic device (101) may perform sampling on the second data. For example, the electronic device (101) may perform the sampling so that the size of the second data has a size within the reference range. For example, the sampling may include random sampling. In the above example, an example of performing the reduction (or the sampling) on ​​the second data of the second label is described, but the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may perform sampling on data of a label requiring reduction among the plurality of labels.

[0085] According to one embodiment, the electronic device (101) may perform expansion on data. For example, if the size of the first data is less than the minimum value of the reference range, the electronic device (101) may determine that expansion is necessary and perform expansion on the first data. For example, the expansion may refer to generating personalized expansion data for the user. For example, the expansion data may be generated using the user's input data from initial data generated according to a label. For example, the input data may include data acquired (collected) while the reference model's service is provided. For example, the expansion may include distance traveled tracking or style conversion. For specific details on distance traveled tracking, refer to FIG. 5B below. For specific details on style conversion, refer to FIG. 5A below.

[0086] According to one embodiment, the electronic device (101) may adjust data within the data set based on the reduction or expansion. For example, the electronic device (101) may recognize the adjusted data set including data expanded from the first data, data reduced from the second data, and the third data. In the above example, an example in which the first data and the second data are adjusted is described, but the embodiments of the present disclosure are not limited thereto. For example, if data of each of the plurality of labels within the data set is within the reference range, the electronic device (101) may not perform adjustment of the data set.

[0087] Referring to FIG. 3, an electronic device (101) according to an embodiment may perform training (or learning, adaptive learning, adaptation, adaptive training, update, generation) of the reference model based on the adjusted data set based on the execution of the model training module (305-5). For example, the electronic device (101) may perform adjustment on the first data and adjustment on the second data, and then generate a model learned (or adaptively learned) from the reference model based on the adjusted data set. In the above example, an example of training (or adaptively training) the reference model based on the adjusted data set is described, but the embodiments of the present disclosure are not limited thereto. For example, when data of each of the plurality of labels in the data set is within the reference range, the reference model may be learned based on the unadjusted data set.

[0088] As described above, the electronic device (101) may collect input data while providing a specific service using the reference model for the specific service, and may adjust the data using the input data. An example in which the electronic device (101) trains (or updates, learns, or generates) the reference model using a data set including the adjusted data is described, but the embodiments of the present disclosure are not limited thereto. For example, after the electronic device (101) transmits a model trained from the reference model to the server (250), the server (250) may generate an aggregated model (hereinafter, referred to as an aggregate model) from a plurality of models including the trained model. The plurality of models may include models acquired from the electronic device (101) and each of the external electronic devices (310, 320, and 330). For example, the plurality of models may include models acquired from the electronic device (101) and external electronic devices (310, 320, 330) within a specified time interval in response to a request transmitted from the server (250). At this time, the integrated model may represent a model generated according to FL. Thereafter, the server (250) may transmit the integrated model to the electronic device (101) and external electronic devices (310, 320, 330). Thereafter, the electronic device (101) may recognize the integrated model as a new reference model and perform training. For specific details on a method for generating the integrated model, reference may be made to the example of FIG. 6 below.

[0089] In the example of FIG. 3, each of the external electronic devices (310, 320, and 330) may include substantially the same components of the electronic device (101) (e.g., the processor (301), the communication circuit (303), and the memory (305)). For example, the external electronic device (310) may include a processor, a communication circuit, and a memory. For example, the external electronic device (320) may include a processor, a communication circuit, and a memory. For example, the external electronic device (330) may include a processor, a communication circuit, and a memory. The description of the processor, the communication circuit, and the memory may be substantially the same as the description of the processor (301), the communication circuit (303), and the memory (305).

[0090] Referring to FIG. 3, the server (250) may include a processor (351), a communication circuit (353), and a memory (355). The description of each of the processor (351), the communication circuit (353), and the memory (355) may be substantially identical to the description of the processor (301), the communication circuit (303), and the memory (305). For example, the server (250) may be associated with a specific service. For example, the specific service may include a domain.

[0091] According to one embodiment, the memory (355) of the server (250) may include a reference model generation module (355-1) and a model integration module (355-3). For example, the server (250) may generate (or train) and store a reference model for FL based on the reference model generation module (355-1). For example, the server (250) may include an application programming interface (API) to be used when the electronic device (101) or external electronic devices (310, 320, 330) downloads the reference model. For example, the API may include a rest API. For example, the server (250) may integrate a plurality of models acquired from the electronic device (101) and external electronic devices (310, 320, 330) based on the model integration module (355-3). For example, the server (250) can generate (or train, update, or learn) an integrated model from the plurality of models. For example, the method for generating the integrated model may include a method of utilizing representative values ​​(e.g., average values) for the parameters of each of the plurality of models. Specific details related thereto are described below in FIG. 6.

[0092] FIG. 4 illustrates an example of a signal flow for a method of adjusting data for training a model in federated learning, according to one embodiment.

[0093] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0094] The electronic device (101) of FIG. 4 may be an example of the electronic device (101) of FIG. 3. The external electronic device (310) of FIG. 4 may be an example of the external electronic device (310) of FIG. 3. The server (350) of FIG. 4 may be an example of the server (250) of FIG. 3. In the example of FIG. 4, an example of a server (250) connected to one external electronic device (310) is illustrated, but the embodiments of the present disclosure are not limited thereto. For example, the embodiments of the present disclosure may also be applied when the server (250) is connected to a plurality of external electronic devices.

[0095] Referring to FIG. 4, in operation (400), the server (250) may generate a reference model. For example, the server (250) may generate the reference model for FL. For example, the server (250) may store the generated reference model. For example, the reference model may be a model for a service provided by the electronic device (101) and the external electronic device (310).

[0096] In operation (405), the server (250) may transmit the reference model. For example, the server (250) may establish connections with multiple electronic devices for the FL. For example, the server (250) may establish a connection with the electronic device (101). For example, the server (250) may establish a connection with an external electronic device (310). For example, the server (250) may transmit the reference model to each of the electronic device (101) and the external electronic device (310) with which the connection has been established.

[0097] In operation (410), the electronic device (101) may provide a service and store input data. For example, the electronic device (101) may provide the service based on the reference model. For example, the electronic device (101) may obtain (or collect) input data while providing the service. For example, the electronic device (101) may obtain the user's input data for the service. For example, the input data may include an input performed by the user on the electronic device (101). For example, the input data may include the user's speech, the user's touch input, or the user's usage history, the user's movement information, and the user's biometric information. For example, the speech, the touch input, and the usage history may be referred to as intentional inputs for inducing a specific function of the user. For example, the movement information and the biometric information may be referred to as unintentional inputs collected according to the user's usage of the electronic device (101). However, the embodiments of the present disclosure are not limited thereto.

[0098] According to one embodiment, the input data may be acquired while the user uses the service. For example, the electronic device (101) may provide the service using the reference model. For example, the electronic device (101) may generate output data from the input data using the reference model. The output data may be used to provide the service. While the service is provided using the reference model, the electronic device (101) may acquire the input data. For example, the electronic device (101) may store the input data in the memory (305) for the service. For example, the input data may be stored in a portion of the memory (305) corresponding to a label. In other words, the input data may be stored for the label among a plurality of labels. For example, data of the plurality of labels may be referred to as a data set, and the data set may include the input data.

[0099] In operation (415), the server (250) may transmit a signal requesting learning. For example, the server (250) may transmit the signal to the electronic device (101) and the external electronic device (310) connected to the server (250). For example, the server (250) may transmit the signal to the electronic device (101) and the external electronic device (310) in response to receiving an input instructing an update of the model for the FL. The signal may be transmitted simultaneously to the electronic devices in response to the input for synchronization between the electronic devices for the FL. Alternatively, for example, the server (250) may transmit the signal to the electronic device (101) and the external electronic device (310) based on a specified time point. For example, the specified time point may be set periodically or aperiodically.

[0100] In operation (420), the electronic device (101) may determine whether adjustments are required for the data set. For example, the electronic device (101) may determine whether data for each of the plurality of labels within the data set requires adjustment. For example, the electronic device (101) may determine whether the size of data for a label included in the plurality of labels is within a reference range.

[0101] In one embodiment, the reference range may be set for a reference value. For example, the reference value may be determined based on the size of the reference model obtained from the server (250). For example, the size of the reference model may be determined based on the number of parameters constituting the reference model.

[0102] According to one embodiment, the electronic device (101) may determine whether the size of the data of the label is smaller than the reference range when the size of the data of the label among the plurality of labels of the data set is outside the reference range. For example, the electronic device (101) may determine that the data of the label requires expansion when the size of the data of the label is smaller than the reference range. In addition, for example, the electronic device (101) may determine that the data of the label requires reduction when the size of the data of the label is larger than the reference range.

[0103] According to one embodiment, the electronic device (101) may determine that adjustment (or expansion) of the data is required when the size of the data of each of the plurality of labels of the data set is less than the reference range. For example, the electronic device (101) may determine that expansion of the data of the label is required when the size of the data of the label is less than the reference range. In addition, for example, the electronic device (101) may determine that reduction of the data of the label is required when the size of the data of the label exceeds the reference range. In addition, according to one embodiment, the electronic device (101) may determine that adjustment of the data is not required when the size of the data of the label is within the reference range.

[0104] In operation (425), the electronic device (101) may perform data adjustment. For example, the electronic device (101) may recognize at least one label requiring adjustment among the plurality of labels within the data set. For example, the electronic device (101) may perform adjustment on data of each of the at least one label. For example, the adjustment may include reduction or expansion. For convenience of explanation, it is assumed below that the at least one label requiring adjustment includes a first label and a second label. For example, the first label may include first data, and the second label may include second data. The first data and the second data may be included in the data set.

[0105] According to one embodiment, the electronic device (101) may perform reduction on data. For example, if the size of the second data exceeds the maximum value of the reference range, the electronic device (101) may determine that reduction is necessary and perform reduction on the second data. For example, the electronic device (101) may perform sampling on the second data. For example, the electronic device (101) may perform the sampling so that the size of the second data has a size within the reference range. For example, the sampling may include random sampling.

[0106] According to one embodiment, the electronic device (101) may perform expansion on data. For example, if the size of the first data is less than the minimum value of the reference range, the electronic device (101) may determine that expansion is necessary and perform expansion on the first data. For example, the expansion may indicate generating personalized expansion data for the user. For example, the expansion data may be generated using the input data from initial data generated according to a label.

[0107] According to one embodiment, the electronic device (101) may adjust data within the data set based on the reduction or expansion. For example, the electronic device (101) may recognize the adjusted data set including data expanded from the first data and data reduced from the second data. In the above example, an example in which the first data and the second data are adjusted is described, but the embodiments of the present disclosure are not limited thereto.

[0108] In the above example, an example of determining whether adjustment of the data is required based on the reference range is described, but the embodiments of the present disclosure are not limited thereto. According to one embodiment, the electronic device (101) can recognize the ratio of data between the plurality of labels of the data set. For example, the electronic device (101) can recognize the amount of data of each of the plurality of labels. The electronic device (101) can sample data of other labels among the plurality of labels according to a label having less data, or expand data of other labels among the plurality of labels according to a label having more data. For convenience of explanation, it is assumed that the data set is a MINST data set. For example, the data set can include a "0" label, a "1" label, a "2" label, and a "3" label. The above data set may include 129 data for the "0" label, 112 data for the "1" label, 309 data for the "2" label, and 318 data for the "3" label. When using the above sampling, the electronic device (101) may reduce the size of the data for the "2" label to 115 and reduce the size of the data for the "3" label to 120. Alternatively, when using the above expansion, the electronic device (101) may expand the size of the data for the "0" label to 298 and expand the size of the data for the "1" label to 325. According to one embodiment, the electronic device (101) may simultaneously perform an operation of recognizing a ratio of data between the plurality of labels in operation (420), and perform an adjustment accordingly in operation (425).

[0109] In operation (430), the electronic device (101) may generate a model learned from the reference model. For example, the electronic device (101) may generate the model learned from the reference model based on the adjusted data set.

[0110] In FIG. 4, an example is shown in which an electronic device (101) performs operations (410), (420), (425), and (430), but the embodiments of the present disclosure are not limited thereto. For example, an external electronic device (310) may also obtain another input data of another user for the service, determine whether to adjust another data set including the other input data, and perform adjustment. Thereafter, the external electronic device (310) may generate another model from the reference model based on the adjusted another data set.

[0111] In operation (435), the electronic device (101) and the external electronic device (310) can transmit the learned model. For example, the electronic device (101) can transmit the model to the server (250). Additionally, for example, the external electronic device (310) can transmit another model to the server (250).

[0112] According to one embodiment, the electronic device (101) may transmit the results of learning for the model to the server (250) together with the model. For example, the results of learning may include a loss value, a cost value, and a confidence level. Furthermore, according to one embodiment, the electronic device (101) may transmit information about the adjusted data set to the server (250) together with the model. For example, the information about the adjusted data set may include whether the adjusted data set is expanded (or reduced) by label (or category) and the expansion (or reduction) ratio by label.

[0113] In operation (440), the server (250) may generate an integrated model. For example, the server (250) may generate the integrated model from the model and the other model. For example, the integrated model may be configured (or formed, generated) based on representative values ​​(e.g., average values) of the parameters of the model and the parameters of the other model. For specific details related thereto, reference may be made to FIG. 6 below.

[0114] According to one embodiment, when generating the integrated model, the server (250) may utilize the learning results corresponding to the model and the learning results corresponding to the other model. For example, the server (250) may compare the first loss value of the model with the second loss value of the other model. For example, if the first loss value is smaller than the second loss value, the server (250) may apply a relatively high weight to the parameters of the model. By applying the weight, the integrated model may be a model that reflects the model more than the other model. Alternatively, for example, the server (250) may compare each of the first loss value and the second loss value with a reference loss value. For example, the server (250) may identify the second loss value that is larger than the reference loss value. Accordingly, the integrated model is generated based on the parameters of the model, and the other model may be excluded from the model for generating the integrated model.

[0115] In one embodiment, the server (250) may utilize the information regarding the adjusted data set when generating the integrated model. For example, when calculating the representative value for generating the integrated model, the server (250) may apply relatively low weights to the parameters of the model using the extended data. This may be because the extended data is arbitrary data generated within the electronic device, rather than being acquired based on the user's actual use.

[0116] Although not illustrated in FIG. 4, according to one embodiment, the server (250) may transmit the integrated model to the electronic device (101) and the external electronic device (310). For example, the integrated model may be used as a new reference model. For example, the electronic device (101) may perform operations (410) to (430) on the integrated model, generate a new model, and then transmit the generated new model to the server (250). In other words, the electronic device (101) may repeatedly perform at least some of the operations of FIG. 4 using the integrated model.

[0117] FIG. 5a illustrates an example of a model for generating augmentation data using initial data, according to one embodiment.

[0118] FIG. 5A illustrates an example of a model (hereinafter, referred to as a data expansion model (500)) used for data manipulation. The data expansion model (500) of FIG. 5A is merely exemplary, and the embodiments of the present disclosure are not limited thereto. For example, the data expansion model (500) may include other models other than those illustrated in FIG. 5A, or may be implemented as a single model that performs the functions of multiple models. For example, the data expansion model (500) may be defined for each service.

[0119] Referring to FIG. 5A, the electronic device (101) may include a data expansion model (500). For example, the electronic device (101) may expand data of a label requiring adjustment using the data expansion model (500). For example, the data expansion model (500) may include at least one of a data generation model (510), a style conversion model (520), or a data mapping model (530).

[0120] According to one embodiment, the electronic device (101) may generate initial data (515) for the label requiring adjustment using the data generation model (510). For example, the initial data (515) may generate arbitrary (or virtual) data of the label generated by the data generation model (510). For example, the type of the initial data (515) may include an image, voice, or text. For example, the initial data (515) may be passed to the style conversion model (520) and the data mapping model (530).

[0121] According to one embodiment, the electronic device (101) may generate extended data (540) from the initial data (515) using the style conversion model (520). For example, the style conversion model (520) may represent a model in which the user's style is learned. For example, the user's style may represent a type of the user's input data acquired while the user uses the specific service. For example, assume that the specific service is an OCR (optical character recognition) service, the user's style is an A font (calligraphical style or font), and an "English" label among a plurality of labels related to the voice recognition service. For example, the electronic device (101) may generate initial data (515) such as "ABC" formed (or written) in an X font, "DEF." formed in a Y font, and "XYZ." formed in a Z font. The style conversion model (520) may be a model learned based on the A font. At this time, exemplary data formed (or written) in the A font may be included in the learning data (535) of the style conversion model (520). The electronic device (101) may use the style conversion model (520) to generate extended data (540) such as "ABC" formed in the A font, "DEF" formed in the A font, and "XYZ" written in the A font from each of the initial data (515).

[0122] As described above, the style conversion model (520) may be trained based on training data (535). According to one embodiment, the style conversion model (520) may be trained using a pair of information including the initial data (515) and the user's input data when the initial data (515) and the user's input data are mapped to each other. Alternatively, the style conversion model (520) may be trained using training data (535) generated by the data mapping model (530) when the initial data (515) and the user's input data are not mapped to each other. For example, the training data (535) may include a pair of information including decoded data generated by the data mapping model (530) from the input data and the user's input data.

[0123] In one embodiment, the mapping of the initial data (515) and the user's input data may be understood as having a parallel data pair. For example, if the initial data (515) is "Hello" formed (or written) in font A, and the user's input data is "Hello" formed in font B, the initial data (515) and the user's input data may be a parallel data pair (or may be mapped) since they have information about the same label (e.g., English). Conversely, if the initial data (515) is "Hello.", but the user's input data is "Bonjour." in French formed in font B, the initial data (515) and the user's input data may not be a parallel data pair (or may not be mapped) since they do not have the same label (e.g., English-French). Referring to the above, the style conversion model (520) can learn about the user's style by using the initial data (515) generated by the data generation model (510) and the user's input data when the initial data (515) and the user's input data are mapped.

[0124] According to one embodiment, the style transformation model (520) may obtain training data (535) from the data mapping model (530) when the initial data (515) and the user's input data are not mapped. For example, the data mapping model (530) may include an auto encoder. According to one embodiment, the electronic device (101) may utilize the data mapping model (530) when it does not recognize or does not have the user's input data mapped to the initial data (515). The data mapping model (530) may generate decoded data corresponding to the user's input data from the user's input data. For example, when the user's input data is "Bonjour," the data mapping model (530) may generate decoded data of "Hello" with the B font. Generating the decoded data may be referred to as data recovery. The decoded data generated by the data mapping model (530) and the user's input data may be included in the training data (535). After this, the style conversion model (520) can be trained on the user's style using learning data (535).

[0125] In one embodiment, the style conversion model (520) may include a data mapping model (530). For example, the style conversion model (520) may include the data mapping model (530) or at least a portion (e.g., an encoder portion) of the data mapping model (530). In other words, although the example of FIG. 5 illustrates a data extension model (500) in which the data mapping model (530) is implemented as a separate model from the style conversion model (520), the embodiments of the present disclosure are not limited thereto. For example, the data extension model (500) may include a style conversion model (520) that includes at least a portion of the data mapping model (530). For example, the style conversion model (520) including the data mapping model (530) may be trained (or generated) using initial data (515) and user input data. In this case, the initial data (515) and the user input data may be data pairs for the same label.

[0126] Referring to the above, FIG. 5A illustrates an example in which an electronic device (101) includes a data mapping model (530), but the embodiments of the present disclosure are not limited thereto. For example, the data mapping model (530) may be omitted.

[0127] Referring to the above, the electronic device (101) can generate extended data (540) using a data extension model (500) based on determining that data needs to be expanded. In FIG. 5A, a data extension model (500) including multiple models is illustrated, but this is merely for convenience of explanation, and the electronic device (101) can generate extended data (540) using a single data extension model (500) including functions of multiple models.

[0128] FIG. 5b illustrates an example of a method for generating extended data using initial data based on a vector representing movement distance, according to one embodiment.

[0129] FIG. 5b illustrates an example (550) of a method for generating extended data based on the vector representing the movement distance.

[0130] Referring to an example (550) of FIG. 5b, the electronic device (101) can generate extended data (580) from initial data (560) generated based on a data generation model (510) using user input data (570). For example, the initial data (560) of FIG. 5b can be understood as being substantially the same as the initial data (515) generated by the data generation model (510) of FIG. 5a. In the example (550) of FIG. 5b, image-format data (560, 570, 580) for the “2” label among a plurality of labels (e.g., 0, 1, 2, 3, ..., 9) for a number recognition service are illustrated, but the embodiments of the present disclosure are not limited thereto. For example, embodiments of the present disclosure may also be applied to data in a format other than an image (e.g., voice or text) or to another label among the plurality of labels (e.g., label “3”).

[0131] According to one embodiment, the electronic device (101) can recognize feature values ​​(561, 562, 563) of the initial data (560). For example, the feature values ​​(561, 562, 563) can be recognized based on a corner detection algorithm (e.g., Harris corner detection or Shi&Tomasi corner detection) for the initial data (560). In addition, the electronic device (101) can recognize feature values ​​(571, 572, 573) of the input data (570). For example, the feature values ​​(571, 572, 573) can be recognized based on the corner detection algorithm for the input data (570). For example, the electronic device (101) can determine a movement vector (585) representing a movement distance or a movement direction between feature values ​​(561, 562, 563) and feature values ​​(571, 572, 573). For example, the feature values ​​of feature values ​​(561, 562, 563) and feature values ​​(571, 572, 573) can be referenced as coordinates or feature points. The movement vector (585) can be referenced as a movement coordinate or a movement distance coordinate.

[0132] According to one embodiment, a motion vector (585) may be recognized for each of the feature values ​​(561, 562, 563) and each of the corresponding feature values ​​(571, 572, 573). For example, the motion vector (585) may include a first vector between the feature value (561) and the feature value (571), a second vector between the feature value (562) and the feature value (572), and a third vector between the feature value (563) and the feature value (573). Referring to the above, the motion vector (585) may be calculated for feature point pairs of the feature values ​​(561, 562, 563) and the feature values ​​(571, 572, 573).

[0133] According to one embodiment, the electronic device (101) can generate a warping mesh based on a movement vector (585). The electronic device (101) can generate (or calculate) extended data (580) by performing warping projection on the initial data (560) using the generated warping mesh. For example, the warping performed based on the movement vector (585) can be understood as being substantially the same as the function (or operation) of the style conversion model (520) of FIG. 5A. For example, the movement vector (585) can be referred to as the style conversion model (520).

[0134] Referring to FIGS. 5A and 5B , the electronic device (101) can generate extended data based on initial data generated using a data generation model and user input data. Accordingly, the electronic device (101) can extend data for a specific label requiring adjustment of a specific service.

[0135] According to one embodiment, the electronic device (101) can learn a reference model received from a server (e.g., server (250) of FIG. 2) using a data set (or adjusted data set) including adjusted data. For example, the adjusted data may include data reduced based on sampling, data expanded based on distance traveled tracking, or data expanded based on style transformation. For example, the electronic device (101) can generate a model learned from the reference model using the adjusted data set. However, the embodiments of the present disclosure are not limited thereto. For example, when no adjustment of the data set is required, the electronic device (101) can also generate a model learned from the reference model using the data set.

[0136] FIG. 6 illustrates an example of a method for generating an aggregate model based on federated learning, according to one embodiment.

[0137] FIG. 6 illustrates an example (600) of a method for a server (250) to generate an integrated model using models acquired from electronic devices (e.g., electronic device (101) and external electronic devices (310, 320, 330)). In the example (600) of FIG. 6, the server (250) acquires three models from the electronic device (101), the external electronic device (310), and the external electronic device (320) and generates an integrated model based thereon. However, the embodiments of the present disclosure are not limited thereto. For example, the server (250) may acquire two models or four or more models and generate an integrated model based thereon.

[0138] Referring to example (600), the server (250) can obtain a first model (610) from the electronic device (101), a second model (620) from the external electronic device (310), and a third model (630) from the external electronic device (320). For example, the first model (610) can include nine parameters (e.g., 1, 1, 4, 3, 2, 5, 6, 8, 1). For example, the second model (620) can include nine parameters (e.g., 3, 5, 9, 6, 1, 8, 3, 2, 3). For example, the third model (630) can include nine parameters (e.g., 7, 9, 5, 3, 4, 5, 2, 1, 1). Although an example of a model including nine parameters is illustrated in Example (600), this is merely for convenience of explanation, and the embodiments of the present disclosure are not limited thereto. For example, each of the first model (610), the second model (620), and the third model (630) may be a model generated (or learned, updated, or trained) from a reference model provided from the server (250).

[0139] In one embodiment, the server (250) may generate an integrated model (650) using representative values ​​(e.g., average values) for the parameters of the models (610, 620, 630). For example, the server (250) may calculate an average of the parameters for each location of the parameters constituting the model. For example, the server (250) can generate an integrated model (650) including nine parameters (e.g., 3.7 (=(1+3+7) / 3), 5 (=(1+5+9) / 3), 6 (=(4+9+5) / 3), 4 (=(3+6+3) / 3), 2.3 (=(2+1+4) / 3), 6 (=(5+8+5) / 3), 3.7 (=(6+3+2) / 3), 3.7 (=(8+2+1) / 3), and 1.7 (=(1+3+1) / 3). According to one embodiment, the server (250) can transmit the integrated model (650) to each of a plurality of electronic devices for the FL (e.g., the electronic device (101), the external electronic device (310), and the external electronic device (320)).

[0140] The above example illustrates an example of a method for generating an integrated model (650) using an average value, but the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may additionally utilize weights when calculating the average value. For example, the weights may be applied to a model (e.g., a first model (610)) of a specific electronic device (e.g., an electronic device (101)). In this case, the server (250) may generate the integrated model (650) by using the values ​​of the model of the specific electronic device with a higher weight (or a lower weight) than the models (e.g., a second model (620) and a third model (630)) of other electronic devices (e.g., an external electronic device (310) and an external electronic device (320)). Alternatively, for example, the electronic device (101) may generate the integrated model (650) using a different algorithm rather than generating the integrated model (650) using the representative value.

[0141] FIG. 7 illustrates an example of an operational flow for a method of adjusting data for model training in federated learning, according to one embodiment.

[0142] At least some of the methods of FIG. 7 may be performed by the electronic device (101) of FIG. 3. For example, at least some of the methods may be controlled by the processor (301) of the electronic device (101). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0143] According to one embodiment, in operation (710), the electronic device (101) may obtain a reference model from a server for federated learning (e.g., server (250) of FIG. 2). For example, the electronic device (101) may establish a connection with the server (250) for the FL. For example, the electronic device (101) may receive the reference model from the server (250) based on the connection. According to one embodiment, the reference model may be a model generated by the server (250). For example, the reference model may be a model for a service provided by the electronic device (101).

[0144] According to one embodiment, in operation (720), the electronic device (101) may store input data obtained from a user based on the reference model while providing the service. For example, the electronic device (101) may provide the service based on the reference model. For example, the electronic device (101) may obtain (or collect) the input data while providing the service. For example, the input data may include an input performed by the user to the electronic device (101). For example, the input data may include the user's speech, the user's touch input, or the user's usage history. However, the embodiments of the present disclosure are not limited thereto.

[0145] According to one embodiment, the input data may be acquired while the user uses the service. For example, the electronic device (101) may provide the service using the reference model. For example, the electronic device (101) may generate output data from the input data using the reference model. The output data may be used to provide the service. While the service is provided using the reference model, the electronic device (101) may acquire the input data. For example, the electronic device (101) may store the input data in the memory (305) for the service. For example, the input data may be stored in a portion of the memory (305) corresponding to a label. In other words, the input data may be stored for the label among a plurality of labels. For example, data of the plurality of labels may be referred to as a data set, and the data set may include the input data.

[0146] According to one embodiment, in operation (730), the electronic device (101) may determine whether the size of data of each of the plurality of labels in the data set stored for the service and including the input data is within a reference range. For example, in response to receiving a signal requesting learning of the reference model from the server (250), the electronic device (101) may determine whether the size of data of each of the plurality of labels in the data set is within the reference range.

[0147] In one embodiment, the electronic device (101) may receive the signal from the server (250). For example, the server (250) may transmit the signal to the electronic device (101) in response to receiving an input instructing an update of a model for the FL. The signal may be transmitted simultaneously to the electronic devices in response to the input for synchronization between the electronic devices for the FL. Alternatively, for example, the signal may be transmitted from the server (250) based on a specified period.

[0148] According to one embodiment, the electronic device (101) can determine whether adjustments are required for the data set. For example, the electronic device (101) can determine whether adjustments are required for data of each of the plurality of labels within the data set. For example, the electronic device (101) can determine whether the size of data of a label included in the plurality of labels is within the reference range.

[0149] In one embodiment, the reference range may be set for a reference value. For example, the reference value may be determined based on the size of the reference model obtained from the server (250). For example, the size of the reference model may be determined based on the number of parameters constituting the reference model.

[0150] According to one embodiment, the electronic device (101) may determine that adjustment (or expansion) of the data is required when the size of the data of each of the plurality of labels of the data set is less than the reference range. For example, the electronic device (101) may determine that expansion of the data of the label is required when the size of the data of the label is less than the reference range. In addition, for example, the electronic device (101) may determine that reduction of the data of the label is required when the size of the data of the label exceeds the reference range. In addition, according to one embodiment, the electronic device (101) may determine that adjustment of the data is not required when the size of the data of the label is within the reference range.

[0151] According to one embodiment, in operation (740), the electronic device (101) may generate a model from the reference model using a data set adjusted so that the data of the label has a size within the reference range. For example, the electronic device (101) may generate the model from the reference model using the adjusted data set based on determining that the size of the data of the label among the plurality of labels is outside the reference range. In this case, the model may represent a model learned from the reference model using the adjusted data set.

[0152] According to one embodiment, the electronic device (101) may perform data adjustment. For example, the electronic device (101) may recognize at least one label requiring adjustment among the plurality of labels within the data set. The label may be included in the at least one label. For example, the electronic device (101) may perform adjustment on data of each of the at least one label. For example, the adjustment may include reduction or expansion. For example, the electronic device (101) may perform reduction on the data. For example, the reduction may include sampling (e.g., random sampling). For example, the electronic device (101) may perform expansion on the data. For example, specific details regarding the expansion may be referred to FIGS. 5A and 5B described above.

[0153] According to one embodiment, the electronic device (101) can recognize the ratio of data among the plurality of labels of the data set. For example, the electronic device (101) can recognize the amount of data of each of the plurality of labels. The electronic device (101) can sample data of other labels among the plurality of labels to match a label having less data, or expand data of other labels among the plurality of labels to match a label having more data.

[0154] According to one embodiment, the electronic device (101) can generate a model learned from the reference model. For example, the electronic device (101) can generate the model learned from the reference model based on the adjusted data set.

[0155] According to one embodiment, in operation (750), the electronic device (101) may transmit the generated model to the server (250). For example, the electronic device (101) may transmit the model learned from the reference model to the server (250) for the FL. Thereafter, the model may be used to generate an integrated model within the server (250). For example, the integrated model may be configured (or formed, generated) based on representative values ​​(e.g., average values) of parameters of the model and parameters of the other models. For specific details related thereto, reference may be made to FIG. 6 described above.

[0156] As described above, the device, method, and storage medium according to embodiments of the present disclosure can manage a collected data set and adjust the ratio of labels within the data set according to the size of the reference model (or models) received from the server (250). Accordingly, the device, method, and storage medium according to embodiments of the present disclosure can efficiently operate the FL by using data that is personalized for the user and adjusted (or refined) according to the model. In addition, the device, method, and storage medium according to embodiments of the present disclosure can generate a model with relatively high accuracy according to the FL by using the adjusted data.

[0157] As described above, the electronic device (101) may include a memory (305) that stores instructions and includes one or more storage media. The electronic device (101) may include a communication circuit (303). The electronic device (101) may include at least one processor (301) that includes a processing circuit. The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to obtain a reference model from a server (250) for federated learning via the communication circuit (303). The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to store input data obtained from a user based on the reference model in the memory (305) while providing a service using the input data. The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to, in response to receiving a signal requesting learning of the reference model from the server (250), determine whether a size of data of each of a plurality of labels in a data set that is stored for the service in the memory (305) and includes the input data is within a reference range.The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to generate a model from the reference model using the data set adjusted so that the data of the label has the size within the reference range, based on determining that the size of the data of the label among the plurality of labels is outside the reference range. The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to transmit the generated model to the server (250) via the communication circuit (303).

[0158] According to one embodiment, the reference model may be generated by the server (250) for the service. The model may be aggregated with at least one model obtained by the server (250) from at least one external electronic device (101) connected to the server (250) for the service.

[0159] In one embodiment, the input data may include user input collected while the service is being provided. The input data may include at least one of speech, image, or touch input.

[0160] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to recognize at least one label among the plurality of labels, the size of the data of which is outside the reference range. The at least one label may include the label. The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to determine whether the size of the data of each of the at least one label is smaller than the reference range.

[0161] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to perform sampling of at least a portion of the data of the label, when the size of the data of the label is greater than the reference range, such that the data of the label has a size within the reference range. The sampling may include random sampling.

[0162] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to perform augmentation on the data of the label based on a data augmentation model, such that the data of the label has a size within the reference range when the size of the data of the label is smaller than the reference range. The augmentation may include generating augmentation data from initial data generated by the data augmentation model. The augmentation data may include personalized data for the user using the input data from the initial data.

[0163] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to generate the initial data using a data generation model. The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to recognize first feature values ​​of the initial data and second feature values ​​of the input data. The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to generate the extended data from the initial data using a vector for changing from the first feature values ​​to the second feature values. The data generation model may be included in the data extension model and may include a variational autoencoder (VAE).

[0164] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to generate the initial data using a data generation model. The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to generate the extended data from the initial data using a style transform model learned for the user. The data generation model and the style transform model may be included in the data extension model.

[0165] According to one embodiment, the style transformation model may be trained using the initial data and the input data when the initial data generated by the data generation model and the input data match with respect to the label, or may be trained using the decoded data generated by the data mapping model from the initial data and the input data when the initial data generated by the data generation model and the input data do not match with respect to the label. The data mapping model may be included in the data expansion model.

[0166] According to one embodiment, the style transformation model may include an encoder portion of the data mapping model.

[0167] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to determine whether a first size of first data of a first label among the plurality of labels is within the reference range. The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to determine whether a second size of second data of a second label different from the first label among the plurality of labels is within the reference range. The instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to perform expansion on the first data when the first size of the first data is smaller than the reference range. The above instructions, when individually or collectively executed by the at least one processor (301), may cause the electronic device (101) to perform sampling on the second data when the second size of the second data is greater than the reference range.

[0168] In one embodiment, the reference range may be set based on specified values ​​or determined based on the number of parameters constituting the reference model.

[0169] In one embodiment, the server (250) may include a web. The signal may be transmitted in response to an input to the web, or may be transmitted based on a specified period.

[0170] In a method performed by an electronic device (101) as described above, the method may include an operation of obtaining a reference model from a server (250) for federated learning. The method may include an operation of storing input data obtained from a user while providing a service using the input data based on the reference model. The method may include an operation of determining, in response to receiving a signal requesting learning of the reference model from the server (250), whether the size of data of each of a plurality of labels in a data set stored for the service and including the input data is within a reference range. The method may include an operation of generating a model from the reference model using the data set in which the data of the labels is adjusted to have the size within the reference range based on a determination that the size of the data of a label among the plurality of labels is outside the reference range. The method may include an operation of transmitting the generated model to the server (250).

[0171] According to one embodiment, the method may include an operation of recognizing at least one label among the plurality of labels whose data size falls outside the reference range. The at least one label may include the label. The method may include an operation of determining whether the size of the data of each of the at least one label is smaller than the reference range.

[0172] According to one embodiment, the method may include an operation of performing sampling on at least a portion of the data of the label so that the data of the label has a size within the reference range when the size of the data of the label is greater than the reference range. The sampling may include random sampling.

[0173] According to one embodiment, the method may include an operation of performing augmentation on the data of the label based on a data augmentation model, when the size of the data of the label is smaller than the reference range, so that the data of the label has a size within the reference range. The augmentation may include generating augmentation data from initial data generated by the data augmentation model. The augmentation data may include personalized data for the user using the input data from the initial data.

[0174] In one embodiment, the method may include an operation of generating the initial data using a data generation model. The method may include an operation of recognizing first feature values ​​of the initial data and second feature values ​​of the input data. The method may include an operation of generating the extended data from the initial data using a vector for changing from the first feature values ​​to the second feature values. The data generation model may be included in the data extension model and may include a variational autoencoder (VAE).

[0175] In one embodiment, the method may include generating the initial data using a data generation model. The method may include generating the extended data from the initial data using a style transformation model learned for the user including the input data. The data generation model and the style transformation model may be included in the data extension model.

[0176] The non-transitory computer-readable storage medium as described above may store one or more programs including instructions that cause, when individually or collectively executed by at least one processor (301) of an electronic device (101) including a communication circuit (303), to obtain a reference model from a server (250) for federated learning via the communication circuit (303). The non-transitory computer-readable storage medium may store one or more programs including instructions that cause, when individually or collectively executed by the at least one processor (301), to store input data obtained from a user based on the reference model in the memory (305) while providing a service using the input data. The non-transitory computer-readable storage medium may store one or more programs including instructions that cause the at least one processor (301) to, when individually or collectively executed, determine whether the size of data of each of a plurality of labels in a data set including the input data stored for the service is within a reference range in response to receiving a signal requesting learning of the reference model from the server (250). The non-transitory computer-readable storage medium may store one or more programs including instructions that cause the at least one processor (301) to, when individually or collectively executed, generate a model from the reference model using the data set adjusted so that the data of the label has the size within the reference range based on determining that the size of data of a label among the plurality of labels is outside the reference range.The non-transitory computer-readable storage medium may store one or more programs including instructions that, when individually or collectively executed by the at least one processor (301), cause the generated model to be transmitted to the server (250) via the communication circuit (303).

[0177] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0178] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0179] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0180] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0181] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0182] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), A memory (305) storing instructions and including one or more storage media; communication circuit (303); and At least one processor (301) comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor (301), cause the electronic device (101) to: Through the above communication circuit (303), a reference model is obtained from a server (250) for federated learning; While providing a service using input data obtained from a user based on the above reference model, the input data is stored in the memory (305); In response to receiving a signal requesting learning of the reference model from the server (250), determine whether the size of data of each of a plurality of labels in a data set stored for the service in the memory (305) and including the input data is within a reference range; Based on determining that the size of the data of the label among the plurality of labels is outside the reference range, a model is generated from the reference model using the data set adjusted so that the data of the label has the size within the reference range; and Causing the generated model to be transmitted to the server (250) through the communication circuit (303). Electronic devices (101).

2. In claim 1, The above reference model is generated by the server (250) for the above service, and The above model is aggregated with at least one model obtained from at least one external electronic device (101) connected to the server (250) by the server (250) for the above service. Electronic devices (101).

3. In claim 1, The above input data includes the user's input collected while the above service is provided, and The above input data includes at least one of speech, image, or touch input. Electronic devices (101).

4. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (301), cause the electronic device (101) to: Recognizing at least one label among the plurality of labels whose data size is outside the reference range, and the at least one label including the label; and causing a determination of whether the size of the data of each of the at least one label is smaller than the reference range, Electronic devices (101).

5. In claim 4, The above instructions, when individually or collectively executed by the at least one processor (301), cause the electronic device (101) to: If the size of the data of the label is greater than the reference range, sampling is performed on at least a portion of the data of the label so that the data of the label has a size within the reference range, The above sampling includes random sampling, Electronic devices (101).

6. In claim 4, The above instructions, when individually or collectively executed by the at least one processor (301), cause the electronic device (101) to: If the size of the data of the label is smaller than the reference range, augmentation of the data of the label is performed based on a data augmentation model so that the data of the label has a size within the reference range, The above extension includes generating augmentation data from initial data generated by the data expansion model, and The above extended data includes personalized data for the user using the input data from the initial data. Electronic devices (101).

7. In claim 6, The above instructions, when individually or collectively executed by the at least one processor (301), cause the electronic device (101) to: Generate the above initial data using a data generation model; Recognizing the first feature values ​​of the initial data and the second feature values ​​of the input data; and To cause the extended data to be generated from the initial data by using a vector for changing from the first feature values ​​to the second feature values, The above data generation model is included in the above data expansion model, and includes a variational autoencoder (VAE). Electronic devices (101).

8. In claim 6, The above instructions, when individually or collectively executed by the at least one processor (301), cause the electronic device (101) to: Generating the initial data using the data generation model; and Generating the extended data from the initial data using the style transform model learned for the above user, The above data generation model and the above style transformation model are included in the above data expansion model. Electronic devices (101).

9. In claim 8, The above style conversion model is: If the initial data and the input data generated by the data generation model match the label, learning is performed using the initial data and the input data, If the initial data generated by the data generation model and the input data do not match the label, learning is performed using the decoded data generated by the data mapping model from the initial data and the input data, and The above data mapping model is included in the above data extension model. Electronic device (101).

10. In claim 9, The above style transformation model includes an encoder part of the data mapping model. Electronic device (101).

11. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (301), cause the electronic device (101) to: Determine whether the first size of the first data of the first label among the plurality of labels is within the reference range; Determine whether the second size of the second data of the second label, which is different from the first label among the plurality of labels, is within the reference range; If the first size of the first data is smaller than the reference range, expansion is performed on the first data; and If the second size of the second data is greater than the reference range, causing sampling to be performed on the second data, Electronic device (101).

12. In claim 1, The above criteria range is: Set based on specified values, or Determined based on the number of parameters that constitute the above reference model, Electronic device (101).

13. In claim 1, The above server (250) includes a web, and The above signal is transmitted in response to an input to the web or is transmitted based on a specified period. Electronic device (101).

14. A method performed by an electronic device (101), wherein the method comprises: An operation of obtaining a reference model from a server (250) for federated learning; An action of storing input data while providing a service using input data obtained from a user based on the above reference model; In response to receiving a signal requesting learning of the reference model from the server (250), an operation of determining whether the size of data of each of a plurality of labels in a data set stored for the service and including the input data is within a reference range; An operation of generating a model from the reference model using the data set adjusted so that the data of the label has the size within the reference range based on determining that the size of the data of the label among the plurality of labels is outside the reference range; and Including an action of transmitting the generated model to the server (250). method.

15. A non-transitory computer-readable storage medium, when individually or collectively executed by at least one processor (301) of an electronic device (101) including a communication circuit (303): Through the above communication circuit (303), a reference model is obtained from a server (250) for federated learning; While providing a service using input data obtained from a user based on the above reference model, the input data is stored in the memory (305); In response to receiving a signal requesting learning of the reference model from the server (250), determine whether the size of data of each of a plurality of labels in a data set stored for the service and including the input data is within a reference range; Based on determining that the size of the data of the label among the plurality of labels is outside the reference range, a model is generated from the reference model using the data set adjusted so that the data of the label has the size within the reference range; and storing one or more programs including instructions causing the generated model to be transmitted to the server (250) via the communication circuit (303); A non-transitory computer-readable storage medium.

Images