Model learning system and model learning method

Abstract

To maintain the specificity of client's data, while protecting privacy without reducing performance.SOLUTION: A model learning system according to an embodiment is a model learning system comprising a server and two or more clients communicably connected with the server. The two or more clients each include: a data acquisition unit that acquires data; an inherent model processing unit that processes the data acquired by the data acquisition unit based on an inherent model inherent to the client; and a common model processing unit that, based on the data processed by the inherent model processing unit, trains a common model provided by the server between the two or more clients and acquired from the server. The server updates the common model based on a difference in the common model before and after the training in the two or more clients.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The embodiments disclosed in this specification and the drawings relate to a model learning system and a model learning method. [Background technology]

[0002] Currently, machine learning algorithms (e.g., deep convolutional networks) are widely used in the medical field, and machine learning algorithms typically require a large number of different training data sets. Due to privacy concerns regarding medical data, it is generally not possible to collect and share patient data in a centralized data pool.

[0003] A multicenter research trial is a clinical trial conducted at more than one medical center or clinic. Most large-scale clinical trials are conducted at several clinical research centers. Multi-site learning (also known as federated learning) can effectively protect the privacy of medical data and is widely used in machine learning, such as deep learning, and can be an effective method for multicenter research trials.

[0004] Figure 6 shows an example of federated learning. As shown in Figure 6, the client side (i.e., each site) first reads the global model parameters from the server to initialize a local model, then updates it by performing multiple training iterations, and then sends the model difference to the server. At the end of each round on the client side, the server receives model differences from all clients through synchronous updates, updates the global model by weighting the received model differences from each client, and shares the new global model with the clients so that they can start processing the next round.

[0005] In the case of federated learning shown in Figure 6, the model configuration is the same for the server and each client. Considering privacy protection for medical data, a method is known in which a privacy protection module is installed in the client. In this method, noise is superimposed on the model difference on the client side before the model difference is sent to the server. While this method can effectively protect the privacy of medical data, the introduction of noise causes a decrease in learning performance compared to when a privacy protection module is not installed.

[0006] In reality, however, the images used as data by each client are often scanned using different protocols across clients from different manufacturers. This results in data idiosyncrasies between clients—that is, the data from one client may be typical and unique, but the amount of data is much smaller than that of other clients. Thus, the server obtains a new model by weighting and adding the model differences from each client. This unique data is assigned a small weight during weighting, making its contribution to the new model small. Through multiple rounds of processing by the new model, the uniqueness of the data gradually disappears. The new model is unsuitable for clients with unique data. Meanwhile, due to the unique nature of medical data, rare cases, for example, have high research value, and data loss and uniqueness diminish their value. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Chinese Patent Application Publication No. 110443063 Summary of the Invention [Problem to be solved by the invention]

[0008] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to maintain the uniqueness of client data while protecting privacy without degrading performance. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0009] A model learning system according to an embodiment includes a server and two or more clients each communicatively connected to the server. Each of the two or more clients includes a data acquisition unit that acquires data, a specific model processing unit that processes the data acquired by the data acquisition unit based on a specific model specific to that client, and a common model processing unit that trains a common model provided by the server between the two or more clients and acquired from the server based on the data processed by the specific model processing unit. The server updates the common model based on differences between the common models before and after training for each of the two or more clients. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an example of a model learning system according to this embodiment. [Figure 2] FIG. 2 is a diagram showing the overall configuration of another example of the model learning system according to this embodiment. [Figure 3] FIG. 3 is a schematic diagram showing the operation of the model learning system according to this embodiment. [Figure 4] FIG. 4 is a schematic diagram showing an example of the operation of the model learning system according to this embodiment. [Figure 5] FIG. 5 is a schematic diagram showing a modified example of the operation of the model learning system according to this embodiment. [Figure 6]FIG. 6 is a schematic diagram showing the operation of a specific example of the prior art associative learning. DETAILED DESCRIPTION OF THE INVENTION

[0011] The model learning system of this embodiment comprises a server and two or more clients each communicatively connected to the server, and each of the two or more clients comprises a data acquisition unit that acquires data, a specific model processing unit that processes the data acquired by the data acquisition unit based on a specific model specific to that client, and a common model processing unit that trains a common model shared by the server between the two or more clients and acquired from the server based on the data processed by the specific model processing unit, and the server updates the common model based on the differences between the common models of the two or more clients.

[0012] Furthermore, in the model learning system according to this embodiment, the server weights and adds two or more trained common models from the two or more clients based on the differences between the respective common models of the two or more clients, updates the common model, and generates a new common model.

[0013] Furthermore, in the model learning system of this embodiment, the specific model processing unit is characterized by including a first specific model processing unit that processes data acquired by the data acquisition unit based on a first specific model specific to the client to generate specific data, and a second specific model processing unit that processes the common model trained by the common model processing unit based on a second specific model specific to the client to generate the common model after training.

[0014] In addition, in the model learning system according to the present embodiment, the first specific model and the second specific model are iteratively updated on the client side.

[0015] Furthermore, in the model learning system according to this embodiment, the first unique model is similar between the two or more clients and is unknown to the server and other clients other than the client itself.

[0016] Furthermore, the model learning system of this embodiment is characterized in that the first specific model and / or the second specific model differ between the two or more clients and are unknown to the server side and other clients other than the client itself.

[0017] Furthermore, in the model learning system of this embodiment, the common model is a deep learning network or part of a deep learning network, and the first specific model and the second specific model are, respectively, a deep learning network, part of a deep learning network, or a machine learning model.

[0018] Furthermore, the model learning method in the model learning system according to this embodiment is a model learning method in a model learning system including a server and two or more clients each connected to the server for communication, wherein in the two or more clients, a data acquisition unit acquires data, a specific model processing unit processes the data acquired by the data acquisition unit based on a specific model specific to the client, a common model processing unit trains a common model acquired from the server based on the data processed by the specific model processing unit, the common model is shared between the two or more clients by the server, and the server updates the common model based on the difference between the common models of the two or more clients.

[0019] Furthermore, in the model learning method in the model learning system according to this embodiment, the server weights and adds two or more trained common models from the two or more clients based on the differences between the respective common models of the two or more clients, updates the common models, and generates a new common model.

[0020] Furthermore, in the model learning method in the model learning system of this embodiment, the specific model processing unit is characterized by including a first specific model processing unit that processes data acquired by the data acquisition unit based on a first specific model specific to the client to generate specific data, and a second specific model processing unit that processes the common model trained by the common model processing unit based on a second specific model specific to the client to generate the common model after training.

[0021] In addition, in the model learning method in the model learning system according to the present embodiment, the first specific model and the second specific model are updated iteratively on the client side.

[0022] Furthermore, in the model learning method in the model learning system of this embodiment, the first unique model is similar between the two or more clients and is unknown to the server side and other clients other than the client itself.

[0023] Furthermore, in the model learning method in the model learning system of this embodiment, the first specific model and / or the second specific model differ between the two or more clients and are unknown to the server side and other clients other than the client itself.

[0024] Furthermore, in the model learning method in the model learning system of this embodiment, the common model is a deep learning network or part of a deep learning network, and the first specific model and the second specific model are, respectively, a deep learning network, part of a deep learning network, or a machine learning model.

[0025] According to the model learning system and model learning method of this embodiment, it is possible to maintain the uniqueness of the client's data while protecting privacy without degrading performance.

[0026] The model learning system and model learning method according to the present embodiment will be described with reference to the drawings. Note that in this specification and drawings, only components related to the technical concept of the present embodiment will be described and shown, and other components will be omitted.

[0027] FIG. 1 is a diagram showing the overall configuration of an example of a model learning system 1000 according to this embodiment.

[0028] As shown in FIG. 1, the model learning system 1000 according to this embodiment includes a server 10 and two or more clients 100a, 100b, ... 100n (n is an integer greater than 2), each of which is connected to the server 10 so as to be able to communicate with the server 10.

[0029] Client 100a includes a data acquisition unit 101, a specific model processing unit 102, and a common model processing unit 103. Clients 100b to 100n have the same configuration as client 100a, and therefore will not be specifically illustrated here. Hereinafter, when there is no need to distinguish between clients 100a, 100b, ... 100n, they may be referred to as clients 100. In other words, model learning system 1000 includes a server 10 and two or more clients 100 each connected to server 10 so as to be able to communicate with it.

[0030] The data acquisition unit 101 acquires data. This data is, for example, medical image data such as CT (Computed Tomography) images, ultrasound images, MR (Magnetic Resonance Imaging) images, etc. Here, the data acquisition unit 101 is configured, for example, with a CPU (Central Processing Unit) or MCU (Micro Controller Unit), has a communication function, and can acquire data from a CT imaging device, an ultrasound imaging device, an MR imaging device, or a storage device in which at least one of CT images, ultrasound images, and MR images is stored.

[0031] The specific model processing unit 102 processes the data acquired by the data acquisition unit 101 based on a specific model specific to the client. Here, the specific model specific to the client may be, for example, a deep learning network, a part of a deep learning network, or a machine learning model, such as Random Forests (RF) or Principal Component Analysis (PCA). The processing performed on the data (image) acquired by the data acquisition unit 101 includes, for example, image encoding, feature extraction, position extraction, and the like. The specific model specific to the client may differ between clients. The specific model is updated by iteration (iterative learning) on ​​each client side. The term "client" refers to each client 100 having the specific model processing unit 102. For example, when the specific model processing unit 102 of client 100a is referred to as "client," the term "client" refers to client 100a.

[0032] The common model processing unit 103 trains a common model acquired from the server 10 based on the data processed by the specific model processing unit 102. The common model is shared (provided) by the server between two or more clients. Here, the common model is, for example, a deep learning network or a part of a deep learning network. Note that the common model is the same between each client 100 and the server 10.

[0033] The server 10 updates the common model based on the difference between the common model before and after training for each of the two or more clients 100. For example, the server 10 may weight and add two or more post-training common models from the two or more clients 100 based on the difference between the common models for each of the two or more clients 100, update the common model, and generate a new common model.

[0034] As described above, in this example, in the model learning system 1000, on each client 100 side, the data acquisition unit 101 acquires data, the specific model processing unit 102 processes the data acquired by the data acquisition unit 101 based on the specific model specific to that client, and the common model processing unit 103 trains the common model acquired from the server 10 based on the data processed by the specific model processing unit 102. On the server 10 side, the common model is updated based on the difference between the common models of two or more clients. This updated common model is applied by the common model processing unit of each client 100 at the next training, and iterations are performed on each client 100 side and the server 10 side.

[0035] Next, the overall configuration of another example of model learning system 1000 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing the overall configuration of another example of model learning system 1000 according to this embodiment.

[0036] In FIG. 2, the same processing units as those in the model learning system 1000 in FIG. 1 are denoted by the same reference numerals, and the description thereof will be omitted.

[0037] The example according to FIG. 2 differs from the example according to FIG. 1 in that the specific model processing unit 102 includes a first specific model processing unit 1021 and a second specific model processing unit 1022.

[0038] The first unique model processing unit 1021 processes the data acquired by the data acquisition unit 101 based on a first unique model unique to the client to generate unique data. Here, the first unique model may be, for example, an encoder module. This first unique model is similar among the clients 100, but is unknown to the server 10, and therefore the first unique model is also called a client common module. Note that "unknown to the server 10" refers to a state in which the server 10 does not store the first unique model and does not understand the algorithm of the first unique model.

[0039] In other words, the first unique model is a model common to the multiple clients 100. However, the first unique model does not necessarily have to be common to all of the multiple clients 100. In other words, the first unique model only needs to be a model common to at least one client 100 of the multiple clients 100.

[0040] The second specific model processing unit 1022 processes the common model trained by the common model processing unit 103 based on a second specific model specific to its own client, generating a trained common model. Here, the second specific model may be, for example, a deep learning (DL) model, a machine learning (ML) model, a decoder module, or a fully connected layer (FC) layer. The second specific model may be different between the server side and the client side, for example, and therefore the second specific model is also a model specific to each client. In addition, the second specific model fine-tunes the common model trained by the common model processing unit 103.

[0041] Furthermore, the first specific model and the second specific model may be, for example, a deep learning network, a part of a deep learning network, or a machine learning model, such as a random forest (RF) or a principal component analysis (PCA).

[0042] The first unique model and / or the second unique model may differ between two or more clients 100, and may be unknown to the server 10 and to clients 100 other than the client 100 itself (own device).

[0043] In other words, the first unique model and / or the second unique model are models that differ from each other between at least two clients 100 among the multiple clients 100. Furthermore, the first unique model and / or the second unique model may be unknown to the server 10, or may be unknown to other clients 100 different from the client 100 itself (own device).

[0044] The first and second specific models are updated by iteration (iterative learning) on ​​each client side, similar to the specific model described above. For example, the first specific model is the same in the first round in the client period, and is fine-tuned and updated in the training period.

[0045] (Example) The model learning system 1000 according to this embodiment will be described below with reference to Figures 2 to 4. Figure 3 is a schematic diagram showing the operation of the model learning system according to this embodiment. Figure 4 is a schematic diagram showing an example of the operation of the model learning system according to this embodiment.

[0046] As shown in Fig. 3, the model learning system 1000 includes a server and two clients, client-1 and client-2. In Fig. 3, the dashed line frame indicates the first specific model, the dashed line frame indicates the common model, and the thin solid line frame indicates the second specific model.

[0047] As described above, in the model learning system 1000 shown in FIG. 3, first, at each client 100 (client-1, client-2), the data acquisition unit 101 acquires data in round t, and here, for example, a graph CT image shown in FIG. 4 is acquired.

[0048] Next, the first specific model processing unit 1021 processes the data acquired by the data acquisition unit 101 based on the first specific model of its own client, and here, for example, since the first specific model is a feature extraction module, after being processed by the first specific model processing unit 1021, a feature map such as that shown in Figure 4 is obtained.

[0049] Next, the common model processing unit 103 trains the common model acquired from the server 10 based on the data processed by the specific model processing unit 102, here as the feature map, to obtain a segmentation map (raw segmentation) as shown in Figure 4.

[0050] Next, the second specific model processing unit 1022 processes the common model trained by the common model processing unit 103 based on the second specific model specific to its own client, and generates a trained common model, thereby obtaining, for example, the image shown in the final result of Figure 4.

[0051] Thereafter, the client transmits to the server 10 the model difference of the common module, which is the difference between the generated common model after training and the common model acquired from the server 10 at the beginning of the round.

[0052] In this round t, the first and second specific models are also updated. The first specific model is fine-tuned and updated during the training period, and the second specific model is updated as is.

[0053] 3, the server 10 updates the common model based on the differences between the common models of two or more clients, client-1 and client-2. For example, the server 10 updates the common model by weighting and adding the differences between the common models of the two or more clients to obtain a new common model.

[0054] The weighting may be performed so that the greater the difference between the common models, the greater the weight, or so that the smaller the difference between the common models, the greater the weight. Furthermore, the weighting may be performed based on a weight that is preset for each client. In other words, the weighting may not be performed according to the number of data. Furthermore, the weighting may not necessarily be performed.

[0055] This updated new common model is used in the next round t+1, and in the next round t+1, the above-mentioned processing by the first specific model processing unit 1021, the common model processing unit 103, the second specific model processing unit 1022, and the server 10 is repeated. Note that the round corresponds to the number of times the common model is updated.

[0056] According to the model learning system 1000 of this embodiment, compared to the conventional technology, rather than a client simply having the same model, it is preferable that the client includes a common model and a specific model, and the specific model includes the first specific model and the second specific model described above.

[0057] This allows data to be preprocessed with a specific model, preferably a first specific model, to protect the privacy of the real data without adding noise, i.e., without degrading performance. Furthermore, since this specific model is unknown to the server and other clients, data privacy can be protected without adding noise, i.e., without degrading performance.

[0058] Meanwhile, a second specific model that differs between clients can ensure data specificity.

[0059] The second specific model fine-tunes the common model trained by the common model processing unit 103. This further ensures data specificity.

[0060] (Variation) Next, a modified example of model learning system 1000 according to this embodiment will be described with reference to Figures 2, 3 and 5. Figure 5 is a schematic diagram showing a modified example of the operation of the model learning system according to this embodiment.

[0061] The modified example differs from the embodiment in that the specific configurations of the first specific model, the second specific model, and the common model are different, but the operation and functionality are the same, so detailed description will be omitted here.

[0062] That is, the common model may be a deep learning network or a part of a deep learning network, or a deep learning network or a part of a deep learning network cascaded and used. The first specific model and the second specific model may be a deep learning network, a part of a deep learning network, or a machine learning model, respectively, or a deep learning network, a part of a deep learning network, or a machine learning model cascaded and used. Furthermore, even when a cascaded application is applied, the technical effects of the embodiments can be achieved.

[0063] Although the embodiments have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. The novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as set forth in the claims. [Explanation of symbols]

[0064] 10 Servers 100 clients 101 Data Acquisition Unit 102 Eigenmodel processing unit 103 Common Model Processing Unit

Claims

1. A model learning system comprising a server and two or more clients each communicably connected to the server, Each of the two or more clients: a data acquisition unit for acquiring data; a specific model processing unit that processes the data acquired by the data acquisition unit based on a first specific model that is specific to the client; a common model processing unit that trains a common model provided by the server between the two or more clients and obtained from the server based on the data processed by the specific model processing unit; Equipped with the specific model processing unit processes the common model trained by the common model processing unit based on a second specific model specific to the client to generate a trained common model; the server updates the common model based on a difference between the common model before and after training for each of the two or more clients. Model learning system.

2. the server weights and adds two or more post-training common models from the two or more clients based on differences between the common models before and after training for each of the two or more clients, and updates the common model to generate a new common model. The model learning system of claim 1 .

3. The specific model processing unit a first specific model processing unit that processes the data acquired by the data acquisition unit based on the first specific model to generate specific data; a second specific model processing unit that processes the common model trained by the common model processing unit based on the second specific model to generate the trained common model; Including, 3. The model learning system according to claim 1 or 2.

4. the first specific model and the second specific model are iteratively updated on the client side; The model learning system of claim 1 .

5. The first specific model is the first specific models of the two or more clients are a common model; is not stored on the server, The model learning system of claim 1 .

6. The second specific model is a model that differs from the first specific models of the two or more clients. The model learning system of claim 1 .

7. the common model is a deep learning network or part of a deep learning network; the first specific model and the second specific model are each a deep learning network, a part of a deep learning network, or a machine learning model; The model learning system of claim 1 .

8. The data is medical image data. The model learning system of claim 1 .

9. A model learning method using a model learning system including a server and two or more clients each connected to the server so as to be able to communicate with the server, At each of the two or more clients, The data acquisition unit acquires the data, a specific model processing unit processes the acquired data based on a first specific model specific to the client; a common model processing unit that trains a common model shared by the server among the two or more clients and obtained from the server based on the processed data; the specific model processing unit processes the common model trained by the common model processing unit based on a second specific model specific to the client to generate a trained common model; the server updates the common model based on a difference between the common model before and after training in each of the two or more clients. Model learning methods.

Images