Electronic device, and federated learning method for artificial intelligence model of electronic device

Abstract

This electronic device may comprise: a memory, which stores instructions and includes one or more storage media; and at least one processor including processing circuitry. The electronic device can: convert first input data into a plurality of pieces of first feature information by using a feature extraction model; generate a first classification model on the basis of the first input data and class information corresponding to each piece of input data; generate class-specific representative information on the basis of the first feature information belonging to the same class; transmit, to a plurality of external devices, the feature extraction model, the first classification model and the class-specific representative information; and receive a retrained feature extraction model by using first data and second data classified on the basis of the class-specific representative information.

Description

Federated learning method for electronic devices and artificial intelligence models of electronic devices

[0001] This document relates to the field of Federated Semi-supervised Learning technology using data collected from devices. In particular, it relates to an artificial intelligence model training method capable of achieving high performance even in situations where the data distribution does not follow an independent and identical distribution (non-iid) in a realistic open-set environment. The artificial intelligence model training method can be performed using electronic devices and non-transient storage media.

[0002] Existing Federated Semi-supervised Learning (FSSL) research assumes an environment where the class range of local data collected from the user's device (client) matches the data collected on the server that creates the artificial intelligence model. Since it is practically impossible to label local data, scenarios have been proposed where some local data is labeled (labels-at-client) or a small amount of data collected on the server is labeled (labels-at-server).

[0003] Existing research, such as FedoSSL (Zhang et al, 2023), was based on a labels-at-client scenario and aimed to mitigate biased learning between locally unknown classes and globally unknown classes that individual clients did not see. To this end, an uncertainty-aware suppressed loss was designed, and a calibration module was introduced to prevent knowledge transfer conflicts caused by inconsistencies in data distribution between clients.

[0004] Meanwhile, Semi-FL (Diao et al, 2022) conducted research in a labels-at-server scenario, but this has the problem of not considering the OpenSet environment. In semi-federated learning, pseudo-labeling is performed on a client without label data using a global model received from a server, followed by semi-supervised learning. However, when this method is applied to an OpenSet environment, pseudo-labeling is not performed properly, which can lead to a problem where the model training does not converge.

[0005] These existing studies have attempted to address the problem of label data scarcity and issues arising from non-iid distributions of local data, but research on server-side labeling scenarios, particularly in open sets environments, is still lacking. FedoSSL has demonstrated the ability to stabilize learning performance by reducing learning confusion on new data through uncertainty perception suppression loss and reducing knowledge conflicts between clients through a correction module, but this remains limited to client-side labeling environments.

[0006] Conventional technology assumes an unrealistic closed-set environment. In reality, since users cannot control the range of data collected or perform labeling, there is a problem in that it is impossible to select and use only high-quality data. Furthermore, in the Labels-at-clients scenario, each client must possess at least a small amount of label data; this is time-consuming and costly, and presents limitations in that it is difficult to guarantee consistency in label quality. Additionally, most federated learning research assumes that the data distribution among clients is identical, but in reality, since each client has a different data distribution, proper training is difficult. The federated learning method for electronic devices and artificial intelligence models of electronic devices according to this document was designed to solve the aforementioned problems.

[0007] The electronic device may include a memory that stores instructions and includes one or more storage media, and at least one processor that includes processing circuitry.

[0008] The electronic device converts first input data into a plurality of first feature information using a feature extraction model, generates a first classification model based on the first input data and class information corresponding to each input data, generates class-specific representative information based on the first feature information belonging to the same class, transmits the feature extraction model, the first classification model, and the class-specific representative information to the plurality of external devices, and receives a feature extraction model retrained using the first data and second data classified based on the class-specific representative information.

[0009] An artificial intelligence model learning method can be performed by an electronic device and a non-transient storage medium. In a computer-readable non-transient storage medium, a first input data is converted into a plurality of first feature information using a feature extraction model, a first classification model is generated based on the first input data and class information corresponding to each input data, representative information for each class is generated based on the first feature information belonging to the same class, the feature extraction model, the first classification model, and the representative information for each class are transmitted to the plurality of external devices, and control can be exercised to receive a feature extraction model retrained using the first data and second data classified based on the representative information for each class.

[0010] The federated learning method for an electronic device and an artificial intelligence model of the electronic device according to this document can provide an effective federated learning framework in a situation where a small amount of inlier dataset exists on the electronic device (e.g., server) and only unlabeled data containing both inliers and outliers exists on an external electronic device (e.g., client).

[0011] The federated learning method for an electronic device and an artificial intelligence model of the electronic device according to this document can provide a method to reliably perform feature training and classifier training of the model by utilizing both unlabeled data and client-local data with non-iid characteristics in an open set environment.

[0012] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0013] Figure 2a is a diagram of an example of a neural network (NN) structure.

[0014] FIG. 2b is a diagram illustrating the configuration of an artificial intelligence model according to one embodiment.

[0015] FIG. 3 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0016] FIG. 4 illustrates a federated learning method for an artificial intelligence model of an electronic device according to one embodiment.

[0017] FIG. 5 is a flowchart illustrating the process of training a model using open set data between an electronic device and an external electronic device according to one embodiment.

[0018] FIG. 6 is a flowchart illustrating the model learning operation of an electronic device according to one embodiment.

[0019] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0020] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0021] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0022] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0023] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0024] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0025] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0026] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0027] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0028] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0029] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0030] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0031] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0032] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0033] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0034] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0035] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0036] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0037] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0038] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0039] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0040] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0041] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0042] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0043] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0044] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0045] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0046] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0047] Figure 2a is a diagram of an example of a neural network (NN) structure.

[0048] Referring to FIG. 2a, an artificial neural network model may include a plurality of layers (L1 to Ln). Each of the plurality of layers (L1 to Ln) may be a linear layer or a non-linear layer, and according to an embodiment, at least one linear layer and at least one non-linear layer may be combined and referred to as a single layer. For example, a linear layer may include a convolution layer and a fully connected layer, and a non-linear layer may include a sampling layer, a pooling layer, and an activation layer.

[0049] According to one embodiment, computations based on an artificial intelligence model (e.g., inference, learning) can be performed in a central processing unit (CPU) and / or a neural processing unit (NPU).

[0050] In the description related to Fig. 2a, the operation of the model-based operation or layer can be understood as the operation of a central processing unit (CPU) and / or a neural processing unit (NPU).

[0051] For example, the first layer (L1) may be a convolution layer and the second layer (L2) may be a sampling layer. The artificial neural network model may further include an activation layer and may further include layers that perform other types of operations.

[0052] According to one embodiment, each of the plurality of layers may receive input image data or a feature or feature map generated from a previous layer as an input feature map. The plurality of layers may generate an output feature map by performing operations on the input feature map. A feature map may refer to comprehensive multidimensional information including a one-dimensional vector. A feature map may refer to data in which various characteristics of the input data are expressed. The feature maps (FM1, FM2, FM3) may take the form of, for example, a one-dimensional vector, a two-dimensional matrix, or a three-dimensional matrix. The feature maps (FM1 to FM3) have a width (W) (or column), a height (H) (or row), and a depth (D), which may correspond to the x-axis, y-axis, and z-axis, respectively, on a coordinate system. In this case, the depth (D) may be referred to as the number of channels. The number and types of the mentioned feature maps are merely examples and are not limited thereto, and may vary depending on the settings.

[0053] According to one embodiment, a first layer (L1) may generate a second feature map (FM2) by convolving a first feature map (FM1) with a weight map (WM). The weight map (WM) may represent an operation that filters the first feature map (FM1) and may be referred to as a filter or a kernel. For example, the depth of the weight map (WM), i.e., the number of channels, is the same as the depth of the first feature map (FM1), e.g., the number of channels, and the same channels of the weight map (WM) and the first feature map (FM1) may be convolved. The weight map (WM) may be shifted in a manner that traverses the first feature map (FM1) using it as a sliding window. The amount of shifting may be referred to as the "stride length" or "stride." During each shift, each weight included in the weight map (WM) may be multiplied and added to all feature values ​​in the region overlapping with the first feature map (FM1). As the first feature map (FM1) and the weight map (WM) are convolved, one channel of the second feature map (FM2) can be generated. Although one weight map (WM) is shown in FIG. 2a, in practice, multiple weight maps can be convolved with the first feature map (FM1) to generate multiple channels of the second feature map (FM2). In other words, the number of channels of the second feature map (FM2) can correspond to the number of weight maps.

[0054] According to one embodiment, the second layer (L2) can generate a third feature map (FM3) by changing the spatial size of the second feature map (FM2). For example, the second layer (L2) may be a sampling layer. The second layer (L2) may perform up-sampling or down-sampling, and the second layer (L2) may select some of the data included in the second feature map (FM2). For example, a 2-dimensional window (WD) may be shifted on the second feature map (FM2) in units of the window (WD) size (e.g., a 4 * 4 matrix), and a value at a specific location (e.g., row 1, column 1) in an area overlapping with the window (WD) may be selected. The second layer (L2) may output the selected data as data for the third feature map (FM3). As another example, the second layer (L2) may be a pooling layer. In this case, the second layer (L2) may select the maximum value (or average value of feature values) of the feature values ​​in the area overlapping with the window (WD) in the second feature map (FM2). The second layer (L2) may output the selected data as data for the third feature map (FM3).

[0055] According to one embodiment, a third feature map (FM3) with a changed spatial size can be generated from a second feature map (FM2). The number of channels in the third feature map (FM3) and the number of channels in the second feature map (FM2) may be the same. Meanwhile, according to an exemplary embodiment of the present disclosure, the computation speed of the sampling layer may be faster than that of the pooling layer, and the sampling layer may improve the quality of the output image (e.g., in terms of PSNR (Peak Signal to Noise Ratio)). For example, the computation time by the pooling layer may be longer than that by the sampling layer because it must calculate a maximum value or an average value.

[0056] According to the embodiment, the second layer (L2) may not be limited to a sampling layer or a pooling layer. For example, the second layer (L2) may be a convolution layer similar to the first layer (L1). The second layer (L2) may generate a third feature map (FM3) by convolving the second feature map (FM2) with a weight map. In this case, the weight map that underwent the convolution operation in the second layer (L2) may be different from the weight map (WM) that underwent the convolution operation in the first layer (L1).

[0057] According to one embodiment, an Nth feature map can be generated in an Nth layer through a plurality of layers including a first layer (L1) and a second layer (L2). The Nth feature map can be input to a classification layer located at the back end of an artificial neural network model where output data is output. The classification layer can generate an output image based on the Nth feature map. Additionally, the classification layer can receive a plurality of feature maps including not only the Nth feature map but also a first feature map (FM1) and a second feature map (FM2). The classification layer can generate an accurate classification result using the plurality of feature maps. According to one embodiment, the classification layer may be a convolution layer or a de-convolution layer.

[0058] FIG. 2b is a diagram illustrating the configuration of an artificial intelligence model according to one embodiment.

[0059] An artificial intelligence model according to one embodiment may include an input layer, a hidden layer, and an output layer.

[0060] According to one embodiment, the input layer is a layer associated with input values ​​input to an artificial intelligence model. In the hidden layer, a feature map can be output by performing a MAC (multiply-accumulate) operation and an activation operation on the input values.

[0061] According to one embodiment, the MAC operation may be an operation that multiplies each input value by a corresponding weight and sums the multiplied values. For example, the MAC operation may be an operation such as x_1*w_1+x_2*w_2+x_3*w_3+... +x_n*w_n.

[0062] According to one embodiment, the activation operation may be an operation that inputs the result of a MAC operation into an activation function and outputs a result value. The activation function may be of various types. For example, the activation function may include a sigmoid function, a tangent function, a ReLU function, a Leaky ReLU function, a Max Out function, and / or an ELU function, but there are no limitations on the types thereof.

[0063] According to one embodiment, the hidden layer may be composed of at least one layer. For example, if the hidden layer is composed of a first hidden layer and a second hidden layer, the first hidden layer performs MAC operations and activation operations based on input values ​​of an input system to output a feature map, and the feature map, which is the result value of the first hidden layer, may become the input value of the second hidden layer. The second hidden layer may perform MAC operations and activation operations based on the feature map, which is the result value of the first hidden layer.

[0064] According to one embodiment, the output layer may be a layer associated with the result value of an operation performed in the hidden layer.

[0065] FIG. 3 is a block diagram showing the configuration of an electronic device according to one embodiment.

[0066] According to one embodiment, the electronic device (101) is a server including a GPU having label data and can implement an artificial intelligence model including a feature extractor (302), an inlier classifier (310), an open set classifier (315), and an outlier classifier (320).

[0067] According to one embodiment, the feature extractor (302) can generate a feature map (304) by extracting features from input data. For example, when motion data collected from an accelerometer is input, features such as vibration patterns over time or frequency characteristics can be extracted and converted into a feature map. The feature map (304) thus generated can be input into the inlier classifier (310), open set classifier (315), and outlier classifier (320) to perform classification tasks suitable for each purpose. According to one embodiment, the feature extractor (302) can be implemented in the form of an artificial neural network. The feature extractor (302) can perform learning using the input data.

[0068] According to one embodiment, the inlier classifier (310) may be a primary classifier that classifies K inlier classes based on a feature map (304) calculated through a feature extractor (302). For example, the inlier classifier (310) may distinguish target classes that the model intends to recognize, such as walking, running, and climbing stairs. The electronic device (101) may generate representative information (prototypes) for each class and determine the class of new data based on this. The prototypes may be generated by calculating the average of the feature maps of data belonging to each class. The prototypes may be used as information representing the typical characteristics of the corresponding class.

[0069] According to one embodiment, the open set classifier (315) can classify into K+1 classes by considering outliers as an additional class. The open set classifier (315) can designate data that is not classified as an inlier among unlabeled data as an 'outlier' class and use it for training. The open set classifier (315) can improve the feature extraction ability of the model. For example, the open set classifier (315) can use undefined movements such as lying down, jumping in place, and riding a bicycle for training in motion classification.

[0070] According to one embodiment, an outlier classifier (320) can perform binary classification to distinguish whether input data is an inlier or an outlier. The outlier classifier (320) can perform binary classification by calculating either the Euclidean distance or the cosine similarity between the feature map of the new data and the prototype of each class. The outlier classifier (320) can classify the data as an outlier if the calculated distance exceeds a set threshold.

[0071] According to one embodiment, an electronic device (101) can perform federated learning by connecting with a plurality of clients (331, 332, ..., 33n). Each client is an edge device that collects unlabeled data and may possess a dataset of various distributions containing a mixture of inliers and outliers. The electronic device (101) can transmit a feature extractor model to the clients and receive back models trained on local data from each client to aggregate them. The electronic device (101) can aggregate models by applying weights proportional to the data size of each client.

[0072] According to one embodiment, an electronic device (101) can receive ratio information of inlier and outlier data from clients. The electronic device (101) can selectively include clients whose ratio of inlier data exceeds a set threshold value in learning. By selectively including clients in learning, the electronic device (101) can conduct efficient and stable model learning. The electronic device (101) can also efficiently utilize the computational resources of the clients.

[0073] According to one embodiment, the electronic device (101) can transmit the integrated model back to the clients. The electronic device (101) can continuously improve the model by calculating the weighted average of the results retrained from each client. The electronic device (101) can effectively utilize unlabeled data through an iterative learning process. The electronic device (101) can reduce learning instability caused by differences in data distribution between clients.

[0074] FIG. 4 illustrates a federated learning method for an artificial intelligence model of an electronic device according to one embodiment.

[0075] According to one embodiment, an electronic device (101) (e.g., a server) may obtain a small amount of label data (401) from a database to perform initial model training. The label data may include input data, such as accelerometer sensor data, and class information to which the data belongs (e.g., walking, running, climbing stairs, etc.).

[0076] According to one embodiment, the electronic device (101) can extract meaningful features from input data through a feature extractor (402) and convert them into a feature map (404). The electronic device (101) can use the generated feature map for training an inlier classifier (410), an open set classifier (415), and an outlier classifier (420). The inlier classifier (410) can be trained to classify K target classes. The outlier classifier (420) can determine whether each class is applicable by configuring K binary classifiers in a one-versus-all manner.

[0077] According to one embodiment, an electronic device (101) may transmit a learned model to an external electronic device (102) (e.g., a client). The transmitted model may include weights of a randomly initialized open set classifier (415) along with weights of a learned feature extractor (402), an inlier classifier (410), and an outlier classifier (420). The external electronic device (102) may receive the transmitted model and perform unsupervised learning with local data it possesses.

[0078] According to one embodiment, the electronic device (101) can generate a prototype (405) by averaging the feature map extracted from label data by class during the learning process. The prototype is information representing the representative features of each class and can be used as a criterion for determining whether the data held by the external electronic device (102) is an inlier or an outlier. The external electronic device (102) can calculate the feature similarity between the prototype and the local data and transmit the result to the electronic device (101).

[0079] According to one embodiment, an external electronic device (102) generates a feature map (434) of local data through a feature extractor (432) and can use it as input to an inlier classifier (440), an open set classifier (445), and an outlier classifier (450). Data is classified using an outlier discrimination expression (out_score > 0.5), and pseudo-labeling is performed on data identified as inliers to be used for training. At this time, if the amount of labeled data is less than a specified level, the external electronic device (102) can apply strong data augmentation. The external electronic device (102) can obtain training data using strong data augmentation and train while maintaining the consistency of the model.

[0080] According to one embodiment, an external electronic device (102) can train an inlier classifier (440) by using the input data as input to the inlier classifier (440) when the ratio of data identified as inliers among the input data exceeds a specified level. When the ratio of data identified as inliers among the input data is less than a specified level, the external electronic device (102) can train an open set classifier (445) and an outlier classifier (450) first.

[0081] According to one embodiment, when the electronic device (101) integrates models received from external electronic devices, it may apply weights proportional to the size of the data used for training for each model. Additionally, the electronic device (101) may prioritize the participation of devices with high inlier data ratios in training by considering the inlier data ratio of each external electronic device. The electronic device (101) may repeatedly perform the training process until the global model sufficiently converges. The global model may include a feature extractor (402), an inlier classifier (410), and an outlier classifier (420). The global model may include an open set classifier (415). The electronic device (101) may gradually improve the performance of the model through the repetition of training.

[0082] According to one embodiment, the electronic device (101) can increase the learning efficiency of the model through an open set classifier (415). The open set classifier (415) can proceed with learning by labeling data as 'outliers' even in situations where there is no data classified as an inlier among the unlabeled data. The open set classifier (415) can learn rich representations by utilizing as much data as possible in an open set environment. The model can effectively train a 'feature extractor (402)' that is indirectly shared with the 'inlier classifier (410)' and the 'outlier classifier (420)'.

[0083] According to one embodiment, the open set classifier (415) is characterized by generating a learning target based on pseudo-labeling, so the confidence level for a specific class may be relatively low. The classification accuracy of the open set classifier (415) may be lower than that of the 'inlier classifier (410)' and the 'outlier classifier (420)'. Due to the characteristic that the classification accuracy of the open set classifier (415) is relatively low, the electronic device (101) may use only the results of the 'inlier classifier (410)' and the 'outlier classifier (420)' when providing actual services. The use of classifier results is merely an example and is not limited to this, and may vary depending on the settings.

[0084] In one embodiment, the electronic device (101) may apply a stepwise learning strategy during fine-tuning. The electronic device (101) may train only the inlier classifier (410) and the outlier classifier (420) while keeping the feature extractor (402) frozen during the initial stage of training. Subsequently, the electronic device (101) may unfreeze the feature extractor (402) to enable training, while applying a small learning rate to prevent the model from changing abruptly. Through this stepwise learning strategy, the electronic device (101) can simultaneously secure the stability and performance of the model.

[0085] In one embodiment, the electronic device (101) can selectively train each layer of the model during fine-tuning. Here, the 'freeze' operation may mean a operation that fixes the weights of a specific layer so that they are not updated during the training process. Conversely, the 'unfreeze' operation may mean a operation that converts the weights of the fixed layer back into a state where they can be trained.

[0086] According to one embodiment, the electronic device (101) can freeze the feature extractor (402) while maintaining the weights of the corresponding layer in their current state so that no changes occur during the backpropagation process. This allows the focus to be on learning the classifiers in the upper layer while preserving the feature extraction capabilities of the already learned feature extractor. Subsequently, the electronic device (101) can gradually improve the performance of the entire model by unlocking the feature extractor (402) to enable fine-tuning.

[0087] In one embodiment, the electronic device (101) controls the classifiers to learn sufficiently based on the current feature extraction results while the feature extractor (402) is frozen, and then optimizes the entire model through an unfreeze process.

[0088] FIG. 5 is a flowchart illustrating the process of training a model using open set data between an electronic device and an external electronic device according to one embodiment.

[0089] According to one embodiment, the electronic device (101) is a flowchart illustrating the process of training a model using an external electronic device (102) and open set data. Open set data may refer to a data set containing data (outliers) that does not belong to the target class (inliers) that the model intends to classify.

[0090] In operation 502, the electronic device (101) can perform initial training of the feature extractor using a small amount of label data. In this process, the feature extractor, inlier classifier, and outlier classifier can be trained in a supervised learning manner. The inlier classifier is trained with the actual class of the input data as the correct answer, and the outlier classifier can be trained by being composed of K binary classifiers in a one-to-remainder (e.g., one-vs-all) manner.

[0091] According to one embodiment, the electronic device (101) can perform initial training of a feature extractor (e.g., the feature extractor (402) of FIG. 4) using a small amount of label data. The feature extractor (402) may mean a neural network that extracts meaningful features from input data.

[0092] According to one embodiment, an inlier classifier (e.g., the inlier classifier (410) of FIG. 4) is trained with the actual class of the input data as the correct answer and can accurately classify K target classes. An outlier classifier (e.g., the outlier classifier (420) of FIG. 4) can be trained by configuring K binary classifiers in a one-versus-all manner. Each binary classifier can perform the role of distinguishing between data belonging to a specific class and data that does not.

[0093] In operation 504, the electronic device (101) can transmit the learned feature extractor model to external electronic devices (102, 104). The model transmitted at this time may include the weights of a randomly initialized open set classifier (e.g., the open set classifier (415) of FIG. 4) along with the previously learned feature extractor (402), inlier classifier (410), and outlier classifier (420). The open set classifier (415) can perform the role of classifying into K+1 classes by treating outliers as a separate class.

[0094] In operation 506, external electronic devices (e.g., external electronic devices (102, 104) of FIG. 1) can learn a feature extractor (e.g., feature extractor (432) of FIG. 4) in an unsupervised learning manner using their respective local data based on the received model. Unsupervised learning may be a method of learning the inherent patterns and features of data by utilizing unlabeled data. In operation 508, the external electronic devices can transmit the learned feature extractor model back to the electronic device (101).

[0095] In operation 510, the electronic device (101) can merge feature extractor models received from external electronic devices. Merging can be performed by averaging the weights of each model equally. This may be done to integrate the knowledge learned from each external device to create a more robust model. The merging method may be changed differently depending on the performance of each model or the characteristics of the training data.

[0096] In operation 512, the electronic device (101) can transmit the global model and prototype to external electronic devices. The prototype can be generated by averaging the feature map extracted from the label data by class. This can be used as a reference point representing the representative features of each class.

[0097] In operation 514, external electronic devices (102) can calculate the ratio of inliers and outliers by comparing the feature similarity between the received prototype and the data they possess. Feature similarity may be a measure indicating how similar the data is to the prototype of each class. In operation 516, this ratio information can be transmitted to the electronic device (101).

[0098] In operation 520, the electronic device (101) can select clients to participate in learning based on received ratio information. The selection criteria may include a method of preferentially selecting external electronic devices whose ratio of inlier data exceeds a set threshold. This may be intended to increase the efficiency and accuracy of model learning.

[0099] In operations 522 through 526, selected external electronic devices (102) can perform model training using local data and transmit the results to the electronic device (101). At this time, the performance of the model can be improved by utilizing pseudo-labeling and strong data augmentation. Pseudo-labeling may be a technique that uses results predicted by the model with high confidence as temporary labels. Strong data augmentation may refer to a method of obtaining additional training data by applying various transformations to the input data and improving the generalization performance of the model using the obtained training data.

[0100] In operation 524, the electronic device (101) can merge the received models and perform fine-tuning. Fine-tuning may be an additional learning process to further improve the performance of the merged model. This entire process can be repeated until the model converges sufficiently to achieve optimal performance.

[0101] According to one embodiment, an electronic device (101) can perform federated semi-supervised learning. Here, the electronic device (101) may refer to a server equipped with a high-performance GPU. External devices (102) refer to edge devices that collect data. The electronic device (101) can transmit a first model, which is a feature extraction model (e.g., the feature extractor (432) of FIG. 4), to a plurality of external devices. At this time, the feature extraction model can extract meaningful features from input data. The electronic device (101) can receive a plurality of second models that have been unsupervised learned from data collected from each external device and generate a third model by merging them.

[0102] In one embodiment, the first input data can be converted into a plurality of first feature information (feature maps) through a third model. Here, a feature map refers to information that concisely represents the main features of the input data. The electronic device (101) can generate a first classification model (inlier classifier) ​​based on the first input data and class information corresponding to each data (e.g., action labels such as walking, running, etc.).

[0103] According to one embodiment, the electronic device (101) can generate class-specific representative information (prototype) by calculating the average value of first feature information belonging to the same class. This prototype is used as information representing the representative features of each class. The generated third model, first classification model, and class-specific representative information can be transmitted to external devices and used for additional learning.

[0104] In one embodiment, the first data refers to data belonging to a class that the classification model can recognize (inlier), and the second data refers to data belonging to a class that the classification model cannot recognize (outlier). The electronic device (101) can distinguish the received data into the first data and the second data based on the similarity with representative information for each class.

[0105] According to one embodiment, an electronic device (101) can calculate the similarity between first feature information and class-specific representative information to distinguish input data into inliers (first data) and outliers (second data). The second data identified as outliers can be trained through a second classification model (openset classifier) ​​that classifies them into separate classes. Additionally, the first data identified as inliers can be utilized for training the first classification model. The third model trained in this way can be continuously updated by receiving retraining results from external devices.

[0106] According to one embodiment, the electronic device (101) can obtain training data from a labeled database and perform initial training of a first model and a first classification model. At this time, the training data is a small amount of labeled data and can be used to train the basic feature extraction and classification capabilities of the model.

[0107] In one embodiment, when the electronic device (101) integrates a plurality of second models received from external devices, it may apply a weight proportional to the amount of data used for training by each model. This can have the effect of improving the performance of the integrated model by increasing the influence of the model trained with more data.

[0108] FIG. 6 is a flowchart illustrating the model learning operation of an electronic device according to one embodiment.

[0109] The operations described through FIG. 6 may be implemented based on instructions that can be stored in a computer recording medium or memory (e.g., memory (130) of FIG. 1). The illustrated method (600) may be executed by an electronic device (e.g., electronic device (101) of FIG. 1) described above through FIG. 1 to 5, and the technical features described above will be omitted below. The order of each operation in FIG. 6 may be changed, some operations may be omitted, and some operations may be performed simultaneously.

[0110] In operation 610, the electronic device (101) can transmit the feature extraction model to multiple external electronic devices (clients) and receive the retrained models. The feature extraction model can be implemented as a neural network that extracts meaningful features from input data. The feature extraction model can be pre-trained through supervised learning using a small amount of labeled data. The electronic device (101) can use the pre-training to control the model so that it effectively extracts key features of the data.

[0111] In operation 620, the electronic device (101) can create a new feature extraction model by merging models collected from external electronic devices (e.g., external electronic devices (102, 104) of FIG. 1). The electronic device (101) can integrate the models by applying weights proportional to the size of the data held by each external electronic device during the merging process. The electronic device (101) can assign higher weights to models trained with relatively more data.

[0112] In operation 630, the electronic device (101) can convert input data into feature information using a merged feature extraction model and, based on this, generate a classification model and representative information (prototype) for each class. The classification model can classify input data into one of predefined classes based on the extracted feature information. The prototype can be implemented as a representative value generated by averaging the features of data belonging to each class, and can be used as a reference point to determine how close new data is to each class.

[0113] In operation 640, the electronic device (101) can transmit the integrated feature extraction model, classification model, and class-specific prototypes to external electronic devices. The external electronic devices can use this information to determine whether the data they possess belongs to the class (inlier) targeted by the model or is an outlier. This determination can be made by calculating similarity with the prototype.

[0114] In operation 650, the electronic device (101) can receive and merge retrained models from external electronic devices. In this process, inlier data and outlier data classified based on class-specific prototypes can be utilized in different ways. Data classified as inliers can be directly used to improve the performance of the classification model, while outlier data can be used to enable the feature extraction model to extract more diverse features.

[0115] According to one embodiment, the electronic device (101) can receive ratio information of inlier and outlier data from each external device. The electronic device (101) can enable efficient model learning by selecting external devices whose ratio of inlier data exceeds a set threshold value (e.g., 50%) and transmitting a third model only to them.

[0116] In one embodiment, the electronic device (101) can find representative information of the class most similar to the second feature information by calculating the Euclidean distance. If the calculated distance exceeds a set threshold, the input data can be classified as an outlier. In this way, inliers and outliers can be effectively distinguished.

[0117] According to one embodiment, the electronic device (101) can generate a second classification model by defining second data classified as outliers into a new class. In this case, the second classification model shares the same third model (feature extractor) as the first classification model, thereby allowing outlier data to also contribute to improved feature extraction performance.

[0118] In one embodiment, motion data collected through an accelerometer may be used as input data. In this case, class information may include the user's movement state, such as walking, running, sitting, or climbing stairs, and a model that recognizes the user's activity can be trained through this.

[0119] In one embodiment, the electronic device (101) can improve overall feature extraction performance by training the first classification model using only inlier data and training the third model using both inlier and outlier data. Additionally, the performance of the model can be gradually improved by continuously updating the model parameters by calculating the weighted average of the retraining results of the third model received from external devices.

[0120] The electronic device (101) can repeatedly perform the operations described in FIG. 6 until the performance of the model converges sufficiently. In each iteration, the model learns more data, and more sophisticated feature extraction and classification may be possible. In each iteration, the model's performance may gradually improve.

[0121] According to one embodiment, an electronic device (101) can transmit a first model to a plurality of external devices, receive a plurality of second models learned using data collected from the plurality of external devices, and integrate the plurality of second models to generate a third model. The electronic device (101) can convert the first input data into a plurality of first feature information using the third model, and generate a first classification model based on the first input data and class information corresponding to each input data. The electronic device (101) can generate class-specific representative information based on the first feature information belonging to the same class, and transmit the third model, the first classification model, and class-specific representative information to the plurality of external devices. The electronic device (101) can receive the third model that has been retrained using the first data and second data classified based on the class-specific representative information.

[0122] According to one embodiment, an electronic device (101) may utilize an artificial intelligence model comprising a feature extractor that extracts meaningful features from input data to perform federated semi-supervised learning, and classifiers that classify data based on the extracted features. In this case, the feature extractor may perform the role of generating a feature map that concisely represents the main features of the data.

[0123] According to one embodiment, the electronic device (101) is a server equipped with a high-performance GPU, and the external devices may refer to edge devices that collect data. The electronic device (101) can transmit a first model, which is a feature extraction model, to a plurality of external devices. At this time, the feature extraction model performs the role of extracting meaningful features from input data. The electronic device (101) can receive a plurality of second models that have been unsupervised learned from data collected from each external device and generate a third model by merging them.

[0124] In the process of generating a feature map, the electronic device (101) can automatically learn and extract important features from the input data. For example, in the case of accelerometer data, features such as vibration patterns over time or frequency characteristics can be extracted. This feature extraction can be implemented through deep learning models such as a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN).

[0125] According to one embodiment, the classifier may be largely composed of an inlier classifier (first classification model) and an open set classifier (second classification model). The inlier classifier can perform the role of classifying classes that the model can recognize (e.g., walking, running, climbing stairs, etc.). On the other hand, the open set classifier can perform the role of separately classifying data of a new class (outlier) that the inlier classifier cannot recognize.

[0126] According to one embodiment, the first model is a feature extraction model that extracts a feature map from input data, the second model is a feature extraction model trained using an unsupervised learning method with data collected from a plurality of external devices, and the third model may be a feature extraction model generated by merging a plurality of second models. Additionally, the first data may be data classifiable by the classification model, and the second data may be data that cannot be classified by the classification model.

[0127] According to one embodiment, the electronic device (101) can calculate the similarity between the first feature information and the representative information for each class to separate the first input data into the first data and the second data, and can generate and train a second classification model that classifies the second data into a separate class. Additionally, the first classification model can be trained using the first data, and the third model can be updated using the retraining results of the third model received from a plurality of external devices.

[0128] According to one embodiment, when generating representative information (prototype) for each class, the electronic device (101) may calculate and utilize the average of the feature maps of the data belonging to the corresponding class. This information represents the typical characteristics of each class and can be used as a criterion for determining which class new data belongs to.

[0129] According to one embodiment, the electronic device (101) obtains labeled training data from a database for training a first model and a first classification model, and can initially train the first model and the first classification model using the labeled training data.

[0130] According to one embodiment, the model learning process of the electronic device (101) can be broadly divided into a pre-learning stage and an actual learning stage. In the pre-learning stage, basic performance of feature extractors and classifiers can be secured by utilizing a small amount of labeled data. At this time, the labeled data may consist of data that is representative of each class, and learning may be carried out in a supervised learning manner.

[0131] In the actual learning phase, the performance of the model can be continuously improved by utilizing various data collected from external devices. In this process, the electronic device (101) can apply a differentiated learning strategy by considering the characteristics of the data collected from each external device. For example, in an external device with a high proportion of inlier data, learning focused on improving the performance of the classifier can be performed, and in an external device with a large amount of outlier data, learning focused on improving the feature extraction capability of the feature extractor can be performed.

[0132] During the model integration process, the weights of models trained on each external device can be reflected in proportion to the data size. This is based on the assumption that a model trained with more data can contribute more significantly to the overall model's performance improvement. The integrated model can be transmitted back to the external devices for further training, and as this process is repeated, the model's performance can be gradually improved.

[0133] According to one embodiment, when integrating a plurality of second models, the electronic device (101) can generate a third model by applying a weight proportional to the size of the training data of each second model.

[0134] According to one embodiment, the electronic device (101) receives ratio information of first data and second data collected from each external device from a plurality of external devices, and can select external devices where the ratio of the first data is higher than a preset reference value and transmit a third model.

[0135] The electronic device (101) may apply threshold-based judgment criteria for outlier detection. If the distance between the feature map of new data and the prototype of the closest class exceeds a set threshold, the data may be classified as an outlier. In this case, the threshold may be adaptively set considering the characteristics and distribution of the data.

[0136] OpenSet classifiers can proceed with training by defining data classified as outliers as a new class. This enables the effective processing of data with new patterns that inlier classifiers cannot recognize. Furthermore, by sharing the same feature extractor between the OpenSet classifier and the inlier classifier, outlier data can also contribute to improving the model's feature extraction capabilities.

[0137] According to one embodiment, the electronic device (101) calculates the Euclidean distance between representative information of the class most similar to the second feature information, and if the calculated distance exceeds a preset threshold, the second input data can be classified as second data.

[0138] When calculating similarity, various distance measurement methods such as cosine similarity or Euclidean distance can be utilized. For example, by calculating the Euclidean distance between the feature map of new data and the prototype of each class, the closest class can be identified as the class of the data.

[0139] According to one embodiment, the electronic device (101) can define the second data as an additional class to create a second classification model, and allow the second classification model and the first classification model to learn by sharing the same third model.

[0140] According to one embodiment, the first input data is motion data collected through an acceleration sensor, and the class information may include the user's motion state corresponding to the motion data.

[0141] According to one embodiment, the electronic device (101) can be controlled to perform learning for a first classification model using only the first data, and to improve feature extraction performance for a third model using both the first data and the second data.

[0142] According to one embodiment, the electronic device (101) can calculate a weighted average of the retraining results of the third model received from a plurality of external devices and update the parameters of the third model based on the weighted average.

[0143] According to one embodiment, the electronic device (101) may utilize a weighted average of retraining results received from external devices when updating model parameters. In this case, the weights may be applied differentially considering the data characteristics and learning performance of each external device, thereby improving the performance of the model.

[0144] The aforementioned learning method may be suitable for effectively processing data that exhibits various patterns for each user, particularly accelerometer data collected from wearable devices. For example, the electronic device (101) can perform accurate classification by defining basic movements such as walking or running as inlier classes, and improve the model's feature extraction ability by treating newly observed unusual movement patterns as outliers.

[0145] The embodiments of this document disclosed in this specification and drawings are merely specific examples presented to facilitate the explanation of the technical content according to the embodiments of this document and to aid in understanding the embodiments of this document, and are not intended to limit the scope of the embodiments of this document. Accordingly, the scope of the embodiments of this document should be interpreted to include all modifications or variations derived based on the technical concept of the embodiments of this document, in addition to the embodiments disclosed herein.

Claims

1. In an electronic device, Memory that stores instructions and includes one or more storage media; It includes at least one processor comprising processing circuitry, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device The first input data is converted into a plurality of first feature information using a feature extraction model, and A first classification model is generated based on the above first input data and class information corresponding to each input data, and Representative information for each class is generated based on the first feature information belonging to the same class, and The above feature extraction model, the above first classification model, and the above class-specific representative information are transmitted to a plurality of external devices, and An electronic device that controls receiving a feature extraction model retrained using first data and second data classified based on representative information for each class.

2. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Transmit the first model to multiple external devices, and Receive a plurality of second models trained using data collected from the plurality of external devices, and Controls the creation of a third model by integrating the above plurality of second models, and The above first model is a feature extraction model that extracts a feature map from input data, and The second model above is a feature extraction model trained in an unsupervised learning manner using data collected from the plurality of external devices, and An electronic device characterized in that the above-mentioned third model is a feature extraction model generated by merging a plurality of second models.

3. In Paragraph 1, The above first data is data classifiable in a classification model, and An electronic device characterized in that the above second data is data that cannot be classified in the above classification model.

4. In Paragraph 1, When the above instructions are executed by the at least one processor, the electronic device, Calculate the similarity between the first feature information and the representative information for each class to separate the first input data into first data and second data, and A second classification model is created and trained to classify the above second data into a separate class, and The first classification model is trained using the first data above, and An electronic device that controls the updating of a feature extraction model using the retraining results of the feature extraction model received from the plurality of external devices.

5. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Labeled training data for training a feature extraction model and the first classification model is obtained from a database, and An electronic device that controls the feature extraction model and the first classification model to be initially trained using the labeled training data.

6. In Paragraph 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device An electronic device that controls the generation of the third model by applying weights proportional to the training data size of each second model when integrating the plurality of second models.

7. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Receiving ratio information of first data and second data collected from each external device from the plurality of external devices, and An electronic device that controls the transmission of a feature extraction model by selecting external devices whose ratio of the first data is higher than a preset reference value.

8. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Calculate the Euclidean distances between the feature information extracted from the input data using a feature extraction model and the representative information for each class, respectively, and An electronic device that controls the input data to be classified as the second data, which is unclassifiable in the first classification model, when the smallest distance among the above-calculated Euclidean distances exceeds a preset threshold.

9. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device A second classification model is created by defining the above second data as an additional class, and An electronic device that controls the second classification model and the first classification model to learn by sharing the same feature extraction model.

10. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device The first input data above is motion data collected through an acceleration sensor, and The above class information is an electronic device that includes a user's operation state corresponding to the above movement data.

11. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device The above first classification model performs learning using only the above first data, and The above feature extraction model is an electronic device that controls the feature extraction performance to improve by utilizing both the first data and the second data.

12. In Paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device Calculate the weighted average of the retraining results of the feature extraction model received from multiple external devices, and An electronic device that controls the updating of parameters of the feature extraction model based on the above weighted average.

13. A computer-readable non-transient storage medium storing one or more programs comprising instructions executable by a processor of an electronic device, The first input data is converted into a plurality of first feature information using a feature extraction model, and A first classification model is generated based on the above first input data and class information corresponding to each input data, and Generate representative information for each class based on the above-mentioned first feature information belonging to the same class, and The above feature extraction model, the above first classification model, and the above class-specific representative information are transmitted to a plurality of external devices, and A computer-readable non-transient storage medium that controls receiving a feature extraction model retrained using first data and second data classified based on representative information for each class.

14. In Paragraph 13, When the above instructions are executed individually or collectively by at least one processor, the electronic device Transmit the first model to multiple external devices, and Receive a plurality of second models trained using data collected from the plurality of external devices, and Controls the creation of a third model by integrating the above plurality of second models, and The above first model is a feature extraction model that extracts a feature map from input data, and The second model above is a feature extraction model trained in an unsupervised learning manner using data collected from the plurality of external devices, and A computer-readable non-transient storage medium characterized in that the third model is a feature extraction model generated by merging the plurality of second models.

15. In Paragraph 13, The above first data is data classifiable in a classification model, and A computer-readable non-transient storage medium characterized in that the second data is data that cannot be classified in the classification model.

Images