Server for generating images by applying reallocated weights based on federated learning, method and system for the same

The image generation server uses reassigned weights based on federated learning to enhance global model training efficiency and quality by aggregating client contributions, addressing inefficiencies in conventional federated learning systems.

KR103046452B1Active Publication Date: 2026-11-10PROMEDIUS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
KR1020240098007
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2026-11-10
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

Conventional federated learning-based image generation technologies face inefficiencies in global model training and the quality of generated images, particularly when aggregating data from multiple institutions.

Method used

An image generation server applies reassigned weights based on federated learning, using a processor to receive parameters from multiple clients, calculate Frechet Inception Distance (FID) scores, normalize and aggregate these weights into a global model, and repeat operations until performance converges, allowing for high-quality image generation and improved generalization.

Benefits of technology

This approach enhances the efficiency of global model training and generates high-quality images by leveraging the contributions of multiple clients, improving the generalization ability of the generator model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112024080737432-PAT00061_ABST
    Figure 112024080737432-PAT00061_ABST
Patent Text Reader

Abstract

The present disclosure includes: a communication unit that communicates with a plurality of clients; a memory in which at least one process related to the generation of an image with reassigned weights applied based on federated learning is stored; and a processor that performs operations according to the process. The processor may include the operation of receiving parameters of a local model learned based on an artificial intelligence generation model from a plurality of clients through the communication unit; the operation of generating a synthetic fake image based on parameters of a generator model among the parameters of the respective local models received; the operation of calculating a Frechet Inception Distance (FID) score between a plurality of clients based on the respective generated synthetic fake images; the operation of calculating a sum of FID scores for each client by summing the calculated FID scores; the operation of calculating a reassigned weight for each client by transforming the calculated sum of FID scores to have a negative correlation; the operation of normalizing the calculated reassigned weight for each client; the operation of aggregating the normalized reassigned weight for each client for each parameter of the generator model into a global model; and the operation of repeating the above operations until the performance of the global model converges.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present disclosure relates to an image generation server, method, and system that apply reassigned weights based on federated learning. Background Technology

[0002] With the recent expansion of the scope of artificial intelligence applications, technologies that increase data diversity to generate realistic images by utilizing AI generative models such as Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), Autoregressive Models, Flow-based Models, and Diffusion Models as local models are becoming increasingly important.

[0003] Such data generation technology through local models has been able to solve the problem of data scarcity in various fields.

[0004] However, the successful training of local models relied heavily on large amounts of high-quality data.

[0005] In other words, when local models combine and use data held by various institutions or organizations, they can learn based on richer and more diverse data, which can improve the quality of the generated data results.

[0006] However, conventional local models faced problems regarding data sharing and export when attempting to aggregate and train data held by multiple institutions or organizations simultaneously.

[0007] Accordingly, Federated Learning (FL) methods are being introduced recently, which allow multiple institutions to train a global model by centrally aggregating model weights trained individually, without directly sharing their respective data.

[0008] However, conventional federated learning-based image generation technology had limitations in improving the efficiency of global model training and in generating high-quality images.

[0009] Therefore, improved technology is required to increase the efficiency of global model training and generate high-quality images. Prior art literature

[0010] Republic of Korea Registered Patent Publication 10-2660745 (2024.04.22.) The problem to be solved

[0011] The purpose of the embodiments disclosed in this disclosure is to provide a method for generating high-quality images and increasing the efficiency of global model training by applying federated learning-based reassigned weights to a global model based on the contributions of multiple clients to generate images.

[0012] In addition, the embodiments disclosed in this disclosure aim to provide a method for generating high-quality images while improving the generalization ability of a generator model, by enabling the generation of images by applying reallocated weights differently to a global model according to the performance of a generator model trained based on an artificial intelligence generator model.

[0013] The problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0014] An image generation server applying reassigned weights based on federated learning according to one aspect of the present disclosure for achieving the aforementioned technical problem comprises: a communication unit that performs communication with a plurality of clients; and a memory in which at least one process related to the generation of an image applying reassigned weights based on federated learning is stored. The system includes a processor that performs operations according to the above process, wherein the processor may include the operation of receiving parameters of a local model learned based on an artificial intelligence generation model from a plurality of clients through the communication unit, the operation of generating a synthetic fake image based on parameters of a generator model among the parameters of the locally received local models, the operation of calculating a Frechet Inception Distance (FID) score between the plurality of clients based on the generated synthetic fake images, the operation of calculating a sum of FID scores for each client by summing the calculated FID scores, the operation of calculating a reassignment weight for each client by converting the calculated sum of FID scores to have a negative correlation, the operation of normalizing the calculated reassignment weight for each client, the operation of aggregating the normalized reassignment weights for each client into a global model for each parameter of the generator model, and the operation of repeating the above operations until the performance of the global model converges.

[0015] In addition, the above image may be characterized as being a medical image.

[0016] In addition, the medical image may be characterized as being at least one of an X-ray (radiography) image, a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, an ultrasound image, an endoscopic image, a thermography image, a nuclear medicine image, and an optical image.

[0017] Additionally, the processor may further include an operation to determine whether to perform weight initialization in the next training cycle for the global model to which the reassigned weights are applied.

[0018] Additionally, the processor may further include an operation of evaluating the performance of the generator model as high when the FID score is lower than a preset reference score, and evaluating the performance of the generator model as low when the FID score is higher than a preset reference score.

[0019] Additionally, the processor may further include the operation of aggregating a high-level first weight, reallocated to each parameter of the generator model, into the global model when the FID score is lower than a preset reference score, and aggregating a low-level second weight, reallocated to each parameter of the generator model, into the global model when the FID score is higher than the reference score.

[0020] A method for generating images with reassigned weights based on federated learning, performed by a server according to another aspect of the present disclosure, may include: receiving parameters of a local model learned based on an artificial intelligence generation model from a plurality of clients through a communication unit of the server; generating synthetic fake images based on parameters of a generator model among the parameters of the respectively received local models by a processor of the server; calculating a Frechet Inception Distance (FID) score between the plurality of clients based on the respectively generated synthetic fake images by the processor; calculating a sum of FID scores for each client by summing the calculated FID scores by the processor; calculating a reassigned weight for each client by transforming the calculated sum of FID scores to have a negative correlation by the processor; normalizing the calculated reassigned weight for each client by the processor; aggregating the normalized reassigned weight for each client for each parameter of the generator model into a global model by the processor; and repeating the steps until the performance of the global model converges.

[0021] Additionally, the method may further include a step of determining whether to perform weight initialization in the next training cycle on the global model to which the reassigned weights are applied by the processor.

[0022] Additionally, the method may further include the step of evaluating the performance of the generator model as high when the FID score is lower than a preset reference score by the processor, and evaluating the performance of the generator model as low when the FID score is higher than a preset reference score.

[0023] Additionally, the aggregation step may be characterized by the processor further aggregating a high-level first weight, reallocated to each parameter of the generator model, into the global model when the FID score is lower than a preset reference score, and further aggregating a low-level second weight, reallocated to each parameter of the generator model, into the global model when the FID score is higher than the reference score.

[0024] A federated learning-based image generation system with reassigned weights according to another aspect of the present disclosure comprises: a plurality of clients that train local models based on an artificial intelligence generation model; and a server that communicates with the plurality of clients and controls operations related to the generation of images with reassigned weights based on federated learning. The server may include the operation of receiving parameters of the local model from each of the plurality of clients, the operation of generating synthetic fake images based on parameters of the generator model among the parameters of the local model received by each, the operation of calculating FID (Frechet Inception Distance) scores between the plurality of clients based on the generated synthetic fake images, the operation of calculating a sum of FID scores for each client by summing the calculated FID scores, the operation of calculating a reassignment weight for each client by converting the sum of the calculated FID scores to have a negative correlation, the operation of normalizing the calculated reassignment weight for each client, the operation of aggregating the normalized reassignment weights for each client into a global model for each parameter of the generator model, and the operation of repeating the operation until the performance of the global model converges.

[0025] In addition, a computer program stored on a computer-readable recording medium for executing a method for implementing the present disclosure may be further provided.

[0026] In addition, a computer-readable recording medium for recording a computer program for executing a method for implementing the present disclosure may be further provided. Effects of the invention

[0027] According to the aforementioned means for solving the problem of the present disclosure, images can be generated by applying federated learning-based reallocated weights to a global model based on the contributions of multiple clients, thereby increasing the efficiency of global model learning and providing the effect of generating high-quality images.

[0028] In addition, according to the aforementioned means for solving the problem of the present disclosure, images can be generated by applying reallocated weights differently to a global model based on the performance of a generator model trained on an artificial intelligence generation model, thereby providing the effect of generating high-quality images while improving the generalization ability of the generator model.

[0029] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0030] FIG. 1 is a diagram showing an image generation system with reassigned weights based on federated learning of the present disclosure. Figure 2 is a diagram showing the configuration of the server of Figure 1. FIG. 3 is a diagram illustrating an example of an image generation method based on federated learning and applying reassigned weights according to the present disclosure. FIG. 4 is a diagram illustrating, as an example, the process of calculating the FID score for each client, the sum of the FID scores for each client, and the reallocation weight for each client by the processor of the present disclosure. FIG. 5 is a diagram illustrating, as an example, an image generation process using federated learning-based reassigned weights with a generative adversarial neural network model of the present disclosure. FIGS. 6 and FIGS. 7 are drawings illustrating another example of an image generation method based on federated learning with reassigned weights according to the present disclosure. Specific details for implementing the invention

[0031] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and general content in the art to which this disclosure pertains or content that overlaps between embodiments is omitted. The terms 'part, module, component, block' as used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, components, blocks' may be implemented as a single component, or a single 'part, module, component, block' may include a plurality of components.

[0032] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0033] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0034] Throughout the specification, when it is stated that a component is located "on" another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.

[0035] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0036] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0037] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0038] The operating principles and embodiments of the present disclosure will be described below with reference to the attached drawings.

[0039] In this specification, the federated learning-based image generation system applying reassigned weights according to the present disclosure includes all of various devices capable of performing computational processing and providing results to a user. For example, the federated learning image generation system through dynamic re-weighting application according to the present disclosure may include a computer, a server, and a portable terminal, or may be in the form of any one of them.

[0040] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0041] A server processes information by communicating with external devices and may include application servers, computing servers, database servers, file servers, mail servers, proxy servers, and web servers.

[0042] A portable terminal is a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smartphones, etc., as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0043] In this specification, the image generation system applying reassigned weights based on federated learning according to the present disclosure may include: receiving parameters of a local model learned based on an artificial intelligence generation model from a plurality of clients, respectively; generating synthetic fake images based on parameters of a generator model among the parameters of the respective received local models; calculating FID (Frechet Inception Distance) scores between the plurality of clients based on the respective generated synthetic fake images; summing the calculated FID scores to calculate a sum of FID scores for each client; converting the calculated sum of FID scores to have a negative correlation to calculate a reassigned weight for each client, respectively; normalizing the calculated reassigned weight for each client, the normalized reassigned weight for each client for each parameter of the generator model, the aggregating the normalized reassigned weight for each client into a global model, and repeating the above operations until the performance of the global model converges.

[0044] Here, the parameters of the generator model may refer to the parameters of the network possessing generative capability in an artificial intelligence generative model. For example, in Generative Adversarial Networks, they may be the parameters of the generator model; in VAEs models, the parameters of the decoder model; in Autoregressive models, the parameters of the Autoregressive generator model or the total parameters of the network; in Flow-based models, the inverse transform parameters or the total parameters of the network; and in Diffusion models, the reverse diffusion parameters or the total parameters of the network.

[0045] At this time, the artificial intelligence generative model may be at least one of Generative Adversarial Networks, Variational Autoencoders (VAEs), Autoregressive Models, Flow-based Models, and Diffusion Models.

[0046] Preferably, the image generation system applying reassigned weights based on federated learning according to the present disclosure may include: receiving parameters of a local model trained based on a generative adversarial network from a plurality of clients, respectively; generating synthetic fake images based on parameters of a generator model among the parameters of the respective local models received, respectively; calculating Frechet Inception Distance (FID) scores between the plurality of clients based on the respective generated synthetic fake images, calculating a sum of FID scores for each client by summing the calculated FID scores, calculating reassigned weights for each client by transforming the calculated sum of FID scores to have a negative correlation, normalizing the calculated reassigned weights for each client, aggregating the normalized reassigned weights for each client for each parameter of the generator model into a global model, and repeating the above operations until the performance of the global model converges.

[0047] This federated learning-based image generation system applying reallocated weights can generate images by applying federated learning-based reallocated weights to a global model based on the contributions of multiple clients, thereby increasing the training efficiency of the global model and generating high-quality images. Furthermore, since the federated learning-based image generation system applying reallocated weights can generate images by applying reallocated weights differently to the global model depending on the performance of the generator model trained on an AI generative model, it can generate high-quality images while improving the generalization ability of the generator model.

[0048] Below, we will examine in detail an image generation system based on federated learning with reassigned weights.

[0049] FIG. 1 is a diagram showing an image generation system applying reallocated weights based on federated learning of the present disclosure. FIG. 2 is a diagram showing the configuration of the server of FIG. 1.

[0050] Referring to FIGS. 1 and 2, an image generation system (1000) that applies reassigned weights based on federated learning may include a plurality of clients (100) and servers (200).

[0051] Multiple clients (100) are users who use the service received from the server (200), and may be client devices or client programs. In this case, the client devices may be desktops, laptops, smartphones, tablets, etc. Additionally, the client programs may be web browsers, etc., that enable users to request the service via the web.

[0052] Here, multiple clients (100) can independently train local models based on an artificial intelligence generative model, and when a certain training cycle ends, each can transmit the parameters of the trained local model to the server (200). For example, the artificial intelligence generative model may be at least one of Generative Adversarial Networks, Variational Autoencoders (VAEs), Autoregressive Models, Flow-based Models, and Diffusion Models.

[0053] In this case, Generative Adversarial Networks (GANs) can output the parameters of local models learned as a result by having two or more neural network models face each other and train as if they were fighting against each other. Such Generative Adversarial Networks (GANs) can generalize even in situations where data is limited and can achieve good performance even with small datasets.

[0054] A server (200) may be configured to provide services to multiple clients (100) via a network. Here, the server (200) may perform operations related to federated learning-based image generation through the application of reallocated weights based on the contributions of multiple clients (100). At this time, the server (200) may also perform operations related to federated learning-based image generation through the application of platform-based reallocated weights. Here, the image may be a medical image. At this time, the medical image may be at least one of an X-ray (radiography) image, a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, an ultrasound image, an endoscopy image, a thermography image, a nuclear medicine image, and an optical image.

[0055] For example, the X-ray image may be at least one of a simple X-ray image, a digital radiography image, a fluoroscopy image, a mammography image, and a dual-energy X-ray absorptiometry image.

[0056] As another example, the computed tomography (CT) image may be at least one of a standard CT image, a helical CT image, a multi-detector CT image, and a low-dose CT image.

[0057] As another example, the magnetic resonance imaging (MRI) image may be at least one of a T1-weighted image, a T2-weighted image, a diffusion-weighted image, an fMRI image, and a magnetic resonance angiography (MRA) image.

[0058] As another example, the ultrasound image may be at least one of a 2D ultrasound image, a 3D ultrasound image, a 4D ultrasound image, and a Doppler ultrasound image.

[0059] As another example, the nuclear medicine imaging image may be at least one of a positron emission tomography (PET) image, a single photon emission tomography (SPECT) image, a radioiodine scan image, and a bone scan image.

[0060] As another example, the optical image may be at least one of a fluorescence image, an optical tomography (OCT) image, and an optical biological microscope image.

[0061] The server (200) may include a communication unit (210), memory (220), and a processor (230).

[0062] The communication unit (210) can communicate with multiple clients (100). Here, the communication unit (210) can receive data of a local model learned based on an artificial intelligence generation model from each of the multiple clients (100).

[0063] Here, the communication unit (210) may include a wireless communication module that supports various wireless communication methods, such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a Wi-Fi module and a WiBro (Wireless broadband) module.

[0064] The memory (220) can store data for an algorithm or a program that reproduces the algorithm for controlling the operation of components within the device, and can be implemented as at least one processor (230) that performs the aforementioned operation using the data stored in the memory (220). Here, the memory (220) and the processor (230) can each be implemented as separate chips. Additionally, the memory (220) and the processor (230) can be implemented as a single chip.

[0065] The memory (220) can store data supporting various functions of the device and programs for the operation of the processor (230), can store input / output data, and can store a number of application programs (or applications) running on the device, data for the operation of the device, and instructions. At least some of these application programs can be downloaded from an external server via wireless communication.

[0066] Such memory (220) may include at least one type of storage medium among flash memory type, hard disk type, SSD type (Solid State Disk type), SSD type (Silicon Disk Drive type), multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. Additionally, the memory (220) may be a database that is separated from the device but connected via wired or wireless connection.

[0067] The memory (220) can store data related to the generation of an image with federated learning-based reallocated weights applied, based on the contributions of multiple clients (100). The processor (230) can control operations related to the generation of an image with federated learning-based reallocated weights applied, based on the contributions of multiple clients (100).

[0068] The processor (230) can receive parameters of a local model trained based on an artificial intelligence generation model from a plurality of clients (100) through the communication unit (210). At this time, the processor (230) can generate a synthetic fake image based on the parameters of the generator model among the parameters of the local models received, calculate the FID (Frechet Inception Distance) score between the plurality of clients based on the generated synthetic fake images, and calculate the sum of the FID scores for each client by summing the calculated FID scores. Additionally, the processor (230) can calculate a reassignment weight for each client by converting the calculated sum of FID scores to have a negative correlation, and can normalize the calculated reassignment weight for each client. Here, FID may be an indicator used to evaluate the quality of the generated image. At this time, the processor (230) can compare the generated image with the set of images to be actually generated and calculate the distribution distance between the generated image and the images to be actually generated.

[0069] The processor (230) can aggregate the reassigned weights for each client, normalized for each parameter of the generator model, into a global model. Additionally, the processor (230) may further determine whether to perform weight initialization on the global model with the reassigned weights applied in the next training cycle.

[0070] Meanwhile, in the present disclosure, if the FID score is lower than a preset reference score, the processor (230) determines that the distribution distance between the generated image and the actual image to be generated is close and evaluates that the performance of the generator model is high. At this time, the performance of the generator model may be the generation capability of the generator model.

[0071] Here, the processor (230) can aggregate the first dynamic weights of a higher level, which are reallocated to each parameter of the generator model, into the global model when the FID score is lower than the preset reference score.

[0072] At this time, if the processor (230) evaluates that the performance of the generator model is high, it can generate a real-like image with a higher quality than the fake image of the generator model by using a global model with a high level of first dynamic weights applied.

[0073] In the present disclosure, if the FID score is higher than a preset reference score, the processor (230) determines that the distribution distance between the generated image and the actual image to be generated is far apart and evaluates that the performance of the generator model is low.

[0074] Here, the processor (230) can aggregate a lower level second dynamic weight, reassigned to each parameter of the generator model, into the global model when the FID score is higher than the reference score.

[0075] At this time, if the performance of the generator model is evaluated as low, the processor (230) can generate a real-like image with a higher quality than the fake image of the generator model by using a global model with a low level of second dynamic weights applied.

[0076] FIG. 3 is a diagram illustrating an example of an image generation method based on federated learning and applying reassigned weights according to the present disclosure. FIG. 4 is a diagram illustrating an example of a process of calculating an FID score for each client, a sum of FID scores for each client, and a reassigned weight for each client by a processor according to the present disclosure.

[0077] At this time, the image generation method applying reassigned weights based on federated learning may include a receiving step (S300), a first generation step (S310), a first calculation step (S320), a second calculation step (S330), a third calculation step (S340), a normalization step (S350), an aggregation step (S360), an iteration step (S370), and a second generation step (S380).

[0078] Here, the first calculation step (S320) can calculate the FID (Frechet Inception Distance) score between multiple clients based on the synthetic fake images each generated by the processor (230) in the first generation step (S310). At this time, the processor (230) can compare the generated images with the set of images to be actually generated and calculate the distribution distance between the generated images and the images to be actually generated.

[0079] In other words, in the first calculation step (S320), the processor (230) can calculate the FID score between each of the generated synthetic fake images. Here, the FID score represents the difference in distribution between a fake image generated from one client and a fake image generated from another client. In this case, a lower FID score indicates a more similar distribution.

[0080] Here, the FID score can be expressed by the following [Equation 1].

[0081] [Mathematical Formula 1]

[0082]

[0083] At this time, can be the FID of the first population, and can be the FID of the second population, n can be the pivot, i can be the number of the first client to compare the FID with, j can be the number of the second client to compare the FID with, and N can be the total number of clients.

[0084] In addition, the FID score can be expressed as the following [Equation 2] by expanding [Equation 1].

[0085] [Mathematical Formula 2]

[0086]

[0087] At this time, and can be the mean vector and covariance matrix for the feature distribution of the actual image, and and can be the mean vector and covariance matrix for the feature distribution of the generated image.

[0088] also, can be the Euclidean distance between the mean vectors of two distributions to measure the difference in the centers of the distributions, and The value is a measure of the difference between the covariance matrices of two distributions, where the covariance matrix represents the shape and size of the data distribution, and the diagonal sum may reflect the degree of spread of the distribution.

[0089] Additionally, in the present disclosure, the second calculation step (S330) calculates the sum of FID scores for each client by summing the FID scores calculated by the processor (230) in the first calculation step (S320), and the third calculation step (S340) can calculate the reassignment weight for each client by converting the sum of FID scores calculated by the processor (230) in the second calculation step (S330) to have a negative correlation.

[0090] For example, as illustrated in FIG. 4, the processor (230) has an FID score between the first client (A) and the second client (B). FID score between ) and the first client (A) and the third client (C) Summing ) to sum the FID scores for the first client (A) Calculate ) and the sum of the calculated FID scores ( The reassignment weight for the first client (A) can be calculated by converting it to have a negative correlation with respect to ).

[0091] In another example, the processor (230) has an FID score between the second client (B) and the first client (A). FID scores between ) and the second client (B) and the third client (C) Summing ) to sum the FID scores for the second client (B) Calculate ) and the sum of the calculated FID scores ( The reassignment weight for the second client (B) can be calculated by converting it to have a negative correlation with respect to ).

[0092] In another example, the processor (230) has an FID score between the third client (C) and the first client (A). FID scores between ) and the third client (C) and the second client (B) Summing ) to sum the FID scores for the third client (C) Calculate ) and the sum of the calculated FID scores ( The reassignment weight for the third client (C) can be calculated by converting it to have a negative correlation with respect to ).

[0093] At this time, the processor (230) can convert the calculated FID score sum into 1 / FID, which has a negative correlation. Additionally, the processor (230) can convert the calculated FID score sum into a function that has a negative correlation. For example, the function that has a negative correlation may be at least one of a linear function, a non-linear function, an exponential function, and a logarithmic function.

[0094] For example, a linear function can be represented by [Equation 3] below.

[0095] [Mathematical Formula 3]

[0096]

[0097] Such a linear function has a negative slope, and since f(x) decreases as x increases, it has a negative correlation.

[0098] As another example, a nonlinear function can be represented by [Equation 4] below.

[0099] [Mathematical Formula 4]

[0100]

[0101] Such non-linear functions are in the form of x squared multiplied by a negative number, and since the value of f(x) always decreases as x increases or decreases, they have a negative correlation.

[0102] As another example, the exponential function can be represented by [Equation 5] below.

[0103] [Mathematical Formula 5]

[0104]

[0105] Since the value of f(x) decreases exponentially as x increases, such an exponential function has a negative correlation.

[0106] As another example, the logarithmic function can be represented by [Equation 6] below.

[0107] [Mathematical Formula 6]

[0108]

[0109] This logarithmic function is in the form of multiplying the logarithm by a negative number; since the value of f(x) decreases as x increases, it has a negative correlation.

[0110] Meanwhile, in the present disclosure, functions having a negative correlation are not limited thereto.

[0111] Here, the processor (230) can calculate a reassignment weight αn for each client as shown in [Equation 7] below. At this time, the processor (230) can assign a higher reassignment weight as the FID score is lower.

[0112] [Mathematical Formula 7]

[0113]

[0114] In this case, αn can be a reallocation weight.

[0115] Additionally, in the normalization step (S350), the processor (230) can normalize the sum of the reassignment weights for each client calculated based on [Equation 8] below so that it becomes 1.

[0116] [Mathematical Formula 8]

[0117]

[0118] Additionally, in the aggregation step (S360), the processor (230) can aggregate the normalized, each client-specific reassigned dynamic weight for each parameter of the generator model into the global model (GM).

[0119] At this time, the processor (230) can generate a new global model as shown in [Equation 9] below by weighting the parameters of the generator model based on [Equation 8].

[0120] [Mathematical Formula 9]

[0121]

[0122] Here, It could be a new global model.

[0123] At this time, the processor (230) can transmit the parameters of the new global model to each client (100) through the communication unit (210), and each client (100) can update its local model based on the received parameters of the new global model to perform the next training round for learning.

[0124] FIG. 5 is a diagram illustrating, as an example, an image generation process using federated learning-based reassigned weights with a generative adversarial neural network model of the present disclosure.

[0125] Referring to FIGS. 4 and 5, the processor (230) can receive parameters of local models (101a, 101b, …101N) learned based on a generative adversarial neural network model from a plurality of clients (100) through a communication unit (210) (S300). Here, the global model (GM) may be set to Θ as an initial value.

[0126] At this time, the Generative Adversarial Network (GAN) can output parameters of the local models (101a, 101b, …101N) that are learned as a result by having two or more neural network models face each other and train them as if they are fighting against each other.

[0127] Here, the local models (101a, 101b, …101N) may include a generator model (101a1, 101b1, …101N1) and a discriminator model (101a2, 101b2, …101N2). At this time, the local models (101a, 101b, …101N) may receive Θ from the global model (GM) for each training round (r) for training.

[0128] Afterwards, the local models (101a, 101b, …101N) can train the generator models (101a1, 101b1, …101N1) and the discriminator models (101a2, 101b2, …101N2) using their local data Di. Through this training process, the local models (101a, 101b, …101N) can update the parameter θ.

[0129] The processor (230) can generate synthetic fake images (M1, M2, …M3) based on the parameters of the generator model (101a1, 101b1, …101N1) among the parameters of the local models (101a, 101b, …101N) received respectively (S310). Here, the processor (230) can calculate the FID (Frechet Inception Distance) score between multiple clients based on the generated synthetic fake images (M1, M2, …M3) respectively (S320).

[0130] At this time, the processor (230) can compare the generated image with the set of images to be actually generated and calculate the distribution distance between the generated image and the images to be actually generated.

[0131] In other words, the processor (230) can generate a set of fake images for each client based on the parameters of the generator model (101a1, 101b1, …101N1). At this time, the processor (230) can calculate the FID score between the generated sets of synthetic fake images (M1, M2, …M3). Here, the FID score represents the difference in distribution between a fake image generated from one client and a fake image generated from another client. At this time, a lower FID score indicates a more similar distribution.

[0132] Here, the FID score can be expressed by the following [Equation 1].

[0133] [Mathematical Formula 1]

[0134]

[0135] At this time, can be the FID of the first population, and can be the FID of the second population, n can be the pivot, i can be the number of the first client to compare the FID with, j can be the number of the second client to compare the FID with, and N can be the total number of clients.

[0136] In addition, the FID score can be expressed as the following [Equation 2] by expanding [Equation 1].

[0137] [Mathematical Formula 2]

[0138]

[0139] At this time, and can be the mean vector and covariance matrix for the feature distribution of the actual image, and and can be the mean vector and covariance matrix for the feature distribution of the generated image.

[0140] also, can be the Euclidean distance between the mean vectors of two distributions to measure the difference in the centers of the distributions, and The value is a measure of the difference between the covariance matrices of two distributions, where the covariance matrix represents the shape and size of the data distribution, and the diagonal sum may reflect the degree of spread of the distribution.

[0141] The processor (230) can calculate the sum of FID scores for each client by summing the calculated FID scores (S330), and calculate the reassignment weight for each client by converting the sum of the calculated FID scores to have a negative correlation (S340).

[0142] For example, as illustrated in FIG. 4, the processor (230) has an FID score between the first client (A) and the second client (B). FID score between ) and the first client (A) and the third client (C) Summing ) to sum the FID scores for the first client (A) Calculate ) and the sum of the calculated FID scores ( The reassignment weight for the first client (A) can be calculated by converting it to have a negative correlation with respect to ).

[0143] In another example, the processor (230) has an FID score between the second client (B) and the first client (A). FID scores between ) and the second client (B) and the third client (C) Summing ) to sum the FID scores for the second client (B) Calculate ) and the sum of the calculated FID scores ( The reassignment weight for the second client (B) can be calculated by converting it to have a negative correlation with respect to ).

[0144] In another example, the processor (230) has an FID score between the third client (C) and the first client (A). FID scores between ) and the third client (C) and the second client (B) Summing ) to sum the FID scores for the third client (C) Calculate ) and the sum of the calculated FID scores ( The reassignment weight for the third client (C) can be calculated by converting it to have a negative correlation with respect to ).

[0145] At this time, the processor (230) can convert the calculated FID score sum into 1 / FID, which has a negative correlation. Additionally, the processor (230) can convert the calculated FID score sum into a function that has a negative correlation. For example, the function that has a negative correlation may be at least one of a linear function, a non-linear function, an exponential function, and a logarithmic function.

[0146] For example, a linear function can be represented by [Equation 3] below.

[0147] [Mathematical Formula 3]

[0148]

[0149] Such a linear function has a negative slope, and since f(x) decreases as x increases, it has a negative correlation.

[0150] As another example, a nonlinear function can be represented by [Equation 4] below.

[0151] [Mathematical Formula 4]

[0152]

[0153] Such non-linear functions are in the form of x squared multiplied by a negative number, and since the value of f(x) always decreases as x increases or decreases, they have a negative correlation.

[0154] As another example, the exponential function can be represented by [Equation 5] below.

[0155] [Mathematical Formula 5]

[0156]

[0157] Since the value of f(x) decreases exponentially as x increases, such an exponential function has a negative correlation.

[0158] As another example, the logarithmic function can be represented by [Equation 6] below.

[0159] [Mathematical Formula 6]

[0160]

[0161] This logarithmic function is in the form of multiplying the logarithm by a negative number; since the value of f(x) decreases as x increases, it has a negative correlation.

[0162] Meanwhile, in the present disclosure, functions having a negative correlation are not limited thereto.

[0163] Here, the processor (230) can calculate a reassignment weight αn for each client as shown in [Equation 7] below. At this time, the processor (230) can assign a higher reassignment weight as the FID score is lower.

[0164] [Mathematical Formula 7]

[0165]

[0166] In this case, αn can be a reallocation weight.

[0167] The processor (230) can normalize the sum of the reassignment weights for each client calculated based on [Equation 8] below so that it becomes 1 (S350).

[0168] [Mathematical Formula 8]

[0169]

[0170] The processor (230) can aggregate the normalized, each client-specific reassigned dynamic weights for each parameter of the generator model (101a1, 101b1, …101N1) into a global model (GM) (S360).

[0171] At this time, the processor (230) can generate a new global model as shown in [Equation 9] below by weighting the parameters of the generator model (101a1, 101b1, …101N1) based on [Equation 8].

[0172] [Mathematical Formula 9]

[0173]

[0174] Here, It could be a new global model.

[0175] At this time, the processor (230) can transmit the parameters of the new global model to each client (100) through the communication unit (210), and each client (100) can update its local model (101a, 101b, …101N) based on the received parameters of the new global model to perform the next training round for learning.

[0176] The processor (230) may repeat the receiving step (S300), the first generation step (S310), the first calculation step (S320), the second calculation step (S330), the third calculation step (S340), the normalization step (S350), and the aggregation step (S360) for each training round for federated learning, and may repeat the process until the performance of the global model (GM) converges (S370).

[0177] FIGS. 6 and FIGS. 7 are drawings illustrating another example of an image generation method based on federated learning with reassigned weights according to the present disclosure.

[0178] Referring to FIGS. 6 and 7, the image generation method applying reassigned weights based on federated learning may include a receiving step (S600), a first generation step (S610), a calculation step (S621), a judgment step (S622), an evaluation step (S623, S624), an aggregation step (S661, S662), a repetition step (S671, S672), and a second generation step (S681, S682).

[0179] The processor (230) can receive parameters of a local model learned based on an artificial intelligence model from a plurality of clients (100) through the communication unit (210) (S600). Here, the global model (GM) may have Θ set as an initial value. For example, the artificial intelligence generative model may be at least one of Generative Adversarial Networks, Variational Autoencoders (VAEs), Autoregressive Models, Flow-based Models, and Diffusion Models.

[0180] At this time, the Generative Adversarial Network (GAN) can output the parameters of the local models learned as a result by having two or more neural network models face each other and train as if they were fighting against each other.

[0181] Here, the local model may include a generator model. In this case, for training, the local model may receive Θ from the global model (GM) for each training round (r).

[0182] Afterwards, the local model can train the generator model using its local data Di. Through this training process, the local model can update the parameter θ.

[0183] The processor (230) can generate synthetic fake images based on the parameters of the generator model among the parameters of the local models received (S610). Here, the processor (230) can calculate the FID (Frechet Inception Distance) score between multiple clients based on the generated synthetic fake images (S621).

[0184] At this time, if the processor (230) determines that the distribution distance between the generated image and the actual image to be generated is close when the FID score is lower than the preset reference score (e.g., S622), the processor may evaluate the performance of the generator model as high (S623). Here, the performance of the generator model may be the generation capability of the generator model.

[0185] After this, the processor (230) can aggregate the high-level first dynamic weights reassigned to each parameter of the generator model evaluated as having high performance into the global model (GM) (S661).

[0186] After this, the processor (230) may repeat the receiving step (S600), the first generation step (S610), the calculation step (S621), the judgment step (S622), the evaluation step (S623), and the aggregation step (S661) for each training round for federated learning, and may repeat the process until the performance of the global model (GM) converges (S671).

[0187] After this, the processor (230) can generate a real-like image with a higher quality than the fake image of the generator model by using a global model (GM) with a real-time high-level first dynamic weight applied (S681).

[0188] On the other hand, if the processor (230) determines that the distribution distance between the generated image and the actual image to be generated is far, and evaluates the performance of the generator model as low (S624), the FID score is higher than the preset reference score (No in S622).

[0189] After this, the processor (230) can aggregate the lower level second dynamic weights, which are reassigned to each parameter of the generator model evaluated as having low performance, into the global model (GM) (S662).

[0190] After this, the processor (230) may repeat the receiving step (S600), the first generation step (S610), the calculation step (S621), the judgment step (S622), the evaluation step (S624), and the aggregation step (S662) for each training round for federated learning, and may repeat the process until the performance of the global model (GM) converges (S672).

[0191] After this, the processor (230) can generate a real-like image with a higher quality than the fake image of the generator model by using a global model (GM) with a real-time lower level second dynamic weight applied (S682).

[0192] The processor (230) can aggregate the reassigned high-level first dynamic weights or the reassigned low-level second dynamic weights differently into the global model (GM), thereby increasing the efficiency of learning the global model (GM) and improving performance.

[0193] The present disclosure enables the generation of images by applying federated learning-based reassigned dynamic weights to a global model based on the contributions of multiple clients, thereby increasing the efficiency of global model training and generating high-quality images.

[0194] The present disclosure enables the generation of images by applying dynamically reassigned weights differently to a global model according to the performance of a generator model trained based on an artificial intelligence generator model, thereby improving the generalization ability of the generator model and enabling the generation of high-quality images.

[0195] The present disclosure can improve the generalization ability of a generator model by utilizing data from various sources while maintaining data ownership and privacy.

[0196] The present disclosure has higher learning efficiency and superior performance compared to a centralized learning method that collects all data in one place.

[0197] At least one component may be added or removed in response to the performance of the components illustrated in FIGS. 1, 2 and FIGS. 4, and 5. Additionally, it will be readily understood by those skilled in the art that the relative positions of the components may be changed in response to the performance or structure of the system.

[0198] Although FIGS. 3, 6, and 7 describe the execution of multiple steps sequentially, this is merely an illustrative explanation of the technical concept of the present embodiment. A person skilled in the art to which the present embodiment belongs could modify and adapt the order described in FIGS. 3, 6, and 7, or execute one or more of the multiple steps in parallel, without departing from the essential characteristics of the present embodiment. Therefore, FIGS. 3, 6, and 7 are not limited to a chronological order.

[0199] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. The instructions may be stored in the form of program code and, when executed by a processor, may generate a program module to perform the operation of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0200] Computer-readable recording media include all types of recording media that store instructions that can be decoded by a computer. Examples include ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, optical data storage devices, etc.

[0201] As described above, the disclosed embodiments have been explained with reference to the attached drawings. Those skilled in the art will understand that the present disclosure may be practiced in forms different from the disclosed embodiments without changing the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be interpreted restrictively. Explanation of the symbols

[0202] 1000: System 100: Client 200: Server 210: Communications Department 220: Memory 230: Processor

Claims

Claim 1 A communication unit that performs communication with multiple clients; a memory storing at least one process that applies client-specific reassigned weights, calculated by converting and normalizing the sum of each client's FID (Frechet Inception Distance) scores to have a negative correlation, to a global model as federated learning-based reassigned weights; The image generation server applying federated learning-based reassigned weights includes a processor that performs operations according to the above process, wherein the processor includes: receiving parameters of a local model learned based on an artificial intelligence generation model from a plurality of clients through the communication unit; generating synthetic fake images based on parameters of a generator model among the parameters of the locally received local models; calculating FID (Frechet Inception Distance) scores between the plurality of clients based on the synthetic fake images generated; summing the calculated FID scores to calculate a sum of FID scores for each client; converting the calculated sum of FID scores for each client to have a negative correlation to calculate a reassigned weight for each client; normalizing the calculated reassigned weight for each client; aggregating the normalized reassigned weight for each client into the global model as a federated learning-based reassigned weight for each parameter of the generator model; and repeating the above operations until the performance of the global model converges. Claim 2 An image generation server applying reassigned weights based on federated learning, characterized in that, in claim 1, the image is a medical image. Claim 3 An image generation server applying reassigned weights based on federated learning, characterized in that, in paragraph 2, the medical image is at least one of X-ray (radiography) image, computed tomography (CT) image, magnetic resonance imaging (MRI) image, ultrasound image, endoscopic image, thermography image, nuclear medicine image, and optical image. Claim 4 In claim 1, the processor further includes an operation to determine whether to perform weight initialization in the next learning cycle on the global model to which the reassigned weights are applied, a federated learning-based image generation server with reassigned weights applied. Claim 5 An image generation server applying reassigned weights based on federated learning, wherein, in claim 1, the processor further includes the operation of evaluating the performance of the generator model as high when the FID score is lower than a preset reference score, and evaluating the performance of the generator model as low when the FID score is higher than a preset reference score. Claim 6 An image generation server applying reassigned weights based on federated learning, wherein, in claim 1, the processor further comprises the operation of aggregating a high-level first weight reassigned to each parameter of the generator model into the global model when the FID score is lower than a preset reference score, and aggregating a low-level second weight reassigned to each parameter of the generator model into the global model when the FID score is higher than the reference score. Claim 7 A method for generating images with applied federated learning-based reallocated weights performed by a server, comprising: receiving parameters of a local model learned based on an artificial intelligence generation model from a plurality of clients through a communication unit of the server; generating synthetic fake images based on parameters of a generator model among the parameters of the respectively received local models by a processor of the server; calculating FID (Frechet Inception Distance) scores between the plurality of clients based on the respectively generated synthetic fake images by the processor; calculating a sum of FID scores for each client by summing the calculated FID scores by the processor; calculating a reallocated weight for each client by transforming the calculated sum of FID scores for each client to have a negative correlation by the processor; normalizing each of the calculated reallocated weights for each client by the processor; and aggregating the normalized reallocated weights for each client for each parameter of the generator model into a global model as the federated learning-based reallocated weights by the processor. A method comprising the step of repeating the above steps until the performance of the above global model converges. Claim 8 A method according to claim 7, further comprising the step of determining, by the processor, whether to perform weight initialization in the next training cycle on the global model to which the reassigned weights are applied. Claim 9 A method according to claim 7, further comprising the step of evaluating the performance of the generator model as high by the processor when the FID score is lower than a preset reference score, and evaluating the performance of the generator model as low when the FID score is higher than a preset reference score. Claim 10 A method according to claim 7, wherein the aggregation step is characterized by the processor further aggregating a high-level first weight, reassigned to each parameter of the generator model, into the global model when the FID score is lower than a preset reference score, and further aggregating a low-level second weight, reassigned to each parameter of the generator model, into the global model when the FID score is higher than the reference score. Claim 11 A computer-readable recording medium combined with a computer, storing a program for executing the method of any one of claims 7 through 10. Claim 12 Multiple clients training local models based on artificial intelligence generative models; A system for generating images with reassigned weights based on federated learning, comprising: a server that communicates with the plurality of clients and controls operations related to the generation of images with reassigned weights based on federated learning; wherein the server comprises the operations of receiving parameters of the local model from each of the plurality of clients, generating synthetic fake images based on parameters of the generator model among the parameters of the local model received by each, calculating FID (Frechet Inception Distance) scores between the plurality of clients based on the synthetic fake images generated by each; summing the calculated FID scores to calculate a sum of FID scores for each client, converting the calculated sum of FID scores for each client to have a negative correlation to calculate a reassigned weight for each client, normalizing the calculated reassigned weight for each client, aggregating the normalized reassigned weight for each client for each parameter of the generator model into a global model as a reassigned weight based on federated learning, and repeating the operations until the performance of the global model converges.