Model learning system and model learning method in model learning system

Abstract

The present invention provides a model learning system and a model learning method in a model learning system. The model learning system includes: a server; and two or more clients, each of which is connected to the server in a communicative manner, and the two or more clients respectively have: a data acquisition unit for acquiring data; an inherent model processing unit for processing the data acquired by the data acquisition unit based on the inherent model inherent to the client; and a general model processing unit for training the general model acquired from the server based on the data processed by the inherent model processing unit. The general model is shared between the two or more clients through the server, and the server updates the above-mentioned general model based on the difference between the respective general models of the two or more clients.

Description

Technical Field

[0001] Embodiments of the present invention relate to a model learning system and a model learning method in a model learning system, and in particular, to a model learning system for medical data and a model learning method in a model learning system. Background Art

[0002] Currently, machine learning algorithms (such as deep convolutional networks) are widely used in the medical field. However, machine learning algorithms typically require a large amount of diverse training data. Medical data involves privacy, so collecting and sharing patient data in a centralized data pool is usually not feasible.

[0003] Multicenter research trials are clinical trials conducted at more than one medical center or clinic. Most large clinical trials are conducted across several clinical research centers. Multisite learning (also known as federated learning) effectively protects medical data privacy and is widely used in machine learning, including deep learning, making it an effective approach for multicenter research trials.

[0004] Figure 6 An example of federated learning is shown in Figure 6 As shown, first, on the client side (i.e., each site), the local model is initialized by reading the global model parameters from the server, then updated by running multiple training iterations, and then the model differences are sent to the server. At the end of each round on the client side, the server performs a synchronous update from all clients to receive model differences, then updates the global model by weighting the model differences received from each client, and shares the new global model with the client so that the client can start the next round of processing.

[0005] exist Figure 6 In the federated learning scenario shown, the server and each client share the same model structure. To protect the privacy of medical data, a method exists where a privacy protection module is installed on the client. In this method, each client adds noise to the model differences and then sends these differences to the server. While this method can effectively protect the privacy of medical data, the introduction of noise can lead to reduced learning performance compared to not installing a privacy protection module.

[0006] On the other hand, in practice, the images used as data in each client often come from different manufacturers and are scanned based on different protocols between clients. Therefore, data specificity is prevalent between different clients, that is, the data of a certain client may be typical and specific, but the amount of data is far less than that of other clients. In this way, since the new model is obtained on the server side by weighting and adding the model differences from each client, the specific data will be given a smaller weight when weighting, so that the contribution of the specific data to the new model is small. After multiple rounds of processing based on the new model, the specificity of the data is gradually lost. For clients with specific data, the new model gradually becomes unsuitable for this client. On the other hand, due to the particularity of medical data, such as rare cases, which have high research value, the loss of data specificity will lead to a reduction in the research value. Summary of the Invention

[0007] In view of the above situation, the present invention aims to provide a model learning system and a model learning method in the model learning system that can achieve privacy protection without reducing performance and maintain client data specificity.

[0008] The model learning system involved in the present invention is characterized in that it includes: a server; and two or more clients, each of which is connected to the server in a communicative manner, and the two or more clients respectively have: a data acquisition unit for acquiring data; an inherent model processing unit for processing the data acquired by the data acquisition unit based on the inherent model inherent to the client; and a general model processing unit for training the general model acquired from the server based on the data processed by the inherent model processing unit, the general model being shared between the two or more clients through the server, and the server updating the above-mentioned general model based on the difference between the respective general models of the two or more clients.

[0009] In addition, the model learning system involved in the present invention is characterized in that the server weights and adds two or more trained general models from the two or more clients based on the difference of the respective general models of the two or more clients to update the above-mentioned general model and generate a new general model.

[0010] In addition, the model learning system involved in the present invention is characterized in that the inherent model processing unit includes: a first inherent model processing unit, which processes the data acquired by the data acquisition unit based on the first inherent model inherent to the client to generate inherent data; and a second inherent model processing unit, which processes the general model trained by the general model processing unit based on the second inherent model inherent to the client to generate the above-mentioned trained general model.

[0011] Furthermore, the model learning system according to the present invention is characterized in that the first intrinsic model and the second intrinsic model are updated by iteration on the client side.

[0012] In addition, the model learning system according to the present invention is characterized in that the first inherent model is similar between the two or more clients and is unknown on the server side and on the client side other than the client itself.

[0013] In addition, the model learning system according to the present invention is characterized in that the first inherent model and / or the second inherent model are different between the two or more clients and are unknown on the server side and other client sides other than the client.

[0014] In addition, the model learning system involved in the present invention is characterized in that the general model is a deep learning network or a part of a deep learning network; the first inherent model and the second inherent model are respectively a deep learning network, a part of a deep learning network or a machine learning model.

[0015] In addition, the present invention relates to a model learning method in a model learning system, which includes: a server; and two or more clients, each of which is connected to the server in a communicative manner. The model learning method in the model learning system is characterized in that, in the two or more clients, a data acquisition unit acquires data; an inherent model processing unit processes the data acquired by the data acquisition unit based on an inherent model inherent to the client; a general model processing unit trains the general model acquired from the server based on the data processed by the inherent model processing unit, and the general model is shared between the two or more clients through the server, and the server updates the above-mentioned general model based on the difference between the respective general models of the two or more clients.

[0016] In addition, the model learning method in the model learning system involved in the present invention is characterized in that the server weights and adds the two or more trained general models from the two or more clients based on the difference of the respective general models of the two or more clients to update the above-mentioned general model and generate a new general model.

[0017] In addition, the model learning method in the model learning system involved in the present invention is characterized in that the inherent model processing unit processes the data acquired by the data acquisition unit based on the first inherent model inherent to the client through the first inherent model processing unit included therein to generate inherent data; and processes the general model trained by the general model processing unit based on the second inherent model inherent to the client through the second inherent model processing unit included therein to generate the above-mentioned trained general model.

[0018] Furthermore, the model learning method in the model learning system according to the present invention is characterized in that the first intrinsic model and the second intrinsic model are updated by iteration on the client side.

[0019] In addition, the model learning method in the model learning system of the present invention is characterized in that the first inherent model is similar between the two or more clients and is unknown on the server side and other client sides other than the client.

[0020] In addition, the model learning method in the model learning system involved in the present invention is characterized in that the first inherent model and / or the second inherent model are different between the two or more clients, and are unknown on the server side and other client sides other than the client.

[0021] In addition, the model learning method in the model learning system involved in the present invention is characterized in that the general model is a deep learning network or a part of a deep learning network; the first inherent model and the second inherent model are respectively a deep learning network, a part of a deep learning network or a machine learning model.

[0022] Effects of the Invention

[0023] According to the model learning system and the model learning method in the model learning system involved in the present invention, privacy protection can be achieved without reducing performance, and the data specificity of the client can be maintained. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a diagram showing the overall configuration of an example of a model learning system according to the present invention.

[0025] Figure 2 This is a diagram showing the overall structure of another example of the model learning system according to the present invention.

[0026] Figure 3 This is a schematic diagram showing the operation of the model learning system according to the present invention.

[0027] Figure 4This is a schematic diagram showing an embodiment of the operation of the model learning system according to the present invention.

[0028] Figure 5 It is a schematic diagram showing a modified example of the operation of the model learning system according to the present invention.

[0029] Figure 6 This is a schematic diagram showing the operation of a specific example of federated learning in the related art. DETAILED DESCRIPTION

[0030] Embodiments of a model learning system and a model learning method in the model learning system according to the present invention will be described with reference to the accompanying drawings.

[0031] In addition, in this specification and the drawings, only components related to the technical concept of the present invention are described and shown, and other components are omitted.

[0032] Figure 1 10 is a diagram showing the overall configuration of an example of a model learning system 1000 according to the present invention.

[0033] like Figure 1 As shown, the model learning system 1000 involved in the present invention includes: a server 10; and two or more clients 100a, 100b, ... 100n (n is an integer greater than 2), and the clients 100a, 100b, ... 100n are respectively connected to the server 10 in a communicative manner.

[0034] Client 100a includes a data acquisition unit 101, a unique model processing unit 102, and a general model processing unit 103. Clients 100b to 100n have the same structure as client 100a, and their detailed illustration is omitted. Hereinafter, when there is no distinction between clients 100a, 100b, ..., 100n, they are sometimes referred to as client 100.

[0035] Data acquisition unit 101 is used to acquire data, such as CT images, ultrasound images, and MR images. Data acquisition unit 101 is comprised of, for example, a CPU or MCU, and has communication capabilities. It can obtain data from a CT imaging device, an ultrasound imaging device, an MR imaging device, or a storage device storing at least one of these images.

[0036] The intrinsic model processing unit 102 processes the data acquired by the data acquisition unit 101 based on the intrinsic model inherent to the client. Here, the intrinsic model inherent to the client is, for example, a deep learning network, a portion of a deep learning network, or a machine learning model, such as a random forest (RF) or principal component analysis (PCA). Furthermore, the processing performed on the data (image) acquired by the data acquisition unit 101 includes, for example, image encoding, feature extraction, position extraction, and the like. Furthermore, the intrinsic model inherent to the client can differ between clients. Furthermore, the intrinsic model is updated by each client through iteration.

[0037] The general model processing unit 103 trains a general model received from the server 10 based on the data processed by the specific model processing unit 102. The general model is shared between two or more clients via the server. Here, the general model is, for example, a deep learning network or a portion of a deep learning network. Furthermore, the general model is the same between each client 100 and the server 10.

[0038] The server 10 updates the general model based on the difference between the general models of the two or more clients. For example, the server 10 may update the general model by weighting and adding the two or more trained general models from the two or more clients 100 based on the difference between the general models of the two or more clients 100 to generate a new general model.

[0039] Thus, in this example, in the model learning system 1000, on each client 100, the data acquisition unit 101 acquires data; the unique model processing unit 102 processes the data acquired by the data acquisition unit 101 based on the unique model of that client; and the general model processing unit 103 trains the general model received from the server 10 based on the data processed by the unique model processing unit 102. On the server 10, the general model is updated based on the differences between the general models of two or more clients. This updated general model is then used by the general model processing unit of each client 100 during the next training cycle. Thus, iterations are performed separately on each client 100 and the server 10.

[0040] Below, refer to Figure 2 The overall structure of another example of the model learning system 1000 according to the present invention will be described. Figure 2 This is a diagram showing the overall configuration of another example of the model learning system 1000 according to the present invention.

[0041] exist Figure 2 In, with Figure 1 The same components of the model learning system 1000 are marked with the same reference numerals and their descriptions are omitted.

[0042] Figure 2 Examples involved with Figure 1 The difference from the above example is that in this example, the eigenmodel processing unit 102 includes a first eigenmodel processing unit 1021 and a second eigenmodel processing unit 1022 .

[0043] The first intrinsic model processing unit 1021 processes the data acquired by the data acquisition unit 101 based on the first intrinsic model inherent to the client to generate intrinsic data. Here, the first intrinsic model can be, for example, an encoder module. This first intrinsic model is common across clients 100 but unknown to the server 10. Therefore, it is also referred to as a client-wide module.

[0044] The second intrinsic model processing unit 1022 processes the universal model trained by the universal model processing unit 103 based on the second intrinsic model inherent to the client, generating a trained universal model. Here, the second intrinsic model may be, for example, a DL model, an ML model, a decoder module, or an FC layer (Fully Connected Layer). The second intrinsic model may differ between the server and client sides, and therefore, is also referred to as a client-specific model. Furthermore, the second intrinsic model fine-tunes the universal model trained by the universal model processing unit 103.

[0045] Alternatively, the first inherent model and the second inherent model may be, for example, a deep learning network, a portion of a deep learning network, or a machine learning model, such as random forest (RF) or principal component analysis (PCA).

[0046] In addition, the first inherent model and / or the second inherent model may be different between two or more clients 100 and may be unknown on the server 10 side and on the sides of other clients 100 other than the client 100 itself.

[0047] The first and second intrinsic models are updated by iteration on each client side, similar to the above-mentioned intrinsic models. For example, the first eigenmodel remains unchanged in the first round of the client period and is fine-tuned and updated during the training period.

[0048] (Example)

[0049] Below, refer to Figures 2 to 4 , an embodiment of the model learning system 1000 of the present invention is described. Figure 3 This is a schematic diagram showing the operation of the model learning system according to the present invention. Figure 4 This is a schematic diagram showing an embodiment of the operation of the model learning system according to the present invention.

[0050] like Figure 3 As shown in , the model learning system consists of a server and two clients a and b. Figure 3 In FIG, the dotted-line frame represents the first intrinsic model, the thick solid-line frame represents the universal model, and the thin solid-line frame represents the second intrinsic model.

[0051] In this way, Figure 3 In the model learning system 1000 shown, first, at each client 100 (client-1, client-2), at round t, the data acquisition unit 101 acquires data. For example, Figure 4 CT images are shown in the figure.

[0052] Next, the first inherent model processing unit 1021 processes the data acquired by the data acquisition unit 101 based on the first inherent model of the client. Here, for example, the first inherent model is a feature extraction module, so that after being processed by the first inherent model processing unit 1021, Figure 4 The feature map shown.

[0053] Next, the general model processing unit 103 trains the general model obtained from the server 10 based on the data processed by the inherent model processing unit 102, which is the feature map mentioned above, to obtain Figure 4 The original segmentation map is shown.

[0054] Next, the second inherent model processing unit 1022 processes the general model trained by the general model processing unit 103 based on the second inherent model inherent to the client to generate a trained general model. Here, for example, Figure 4 The final result is shown in the image.

[0055] Afterwards, the client sends the difference between the generated trained general model and the general model obtained from the server 10 at the beginning of the round, that is, the model difference of the general module to the server 10.

[0056] In addition, in this round t, the first intrinsic model and the second intrinsic model are also updated. The first intrinsic model is fine-tuned and updated during the training period, and the second intrinsic model is directly updated.

[0057] like Figure 3 As shown, the server 10 updates the general model based on the difference between the general models of two or more clients, namely, client-1 and client-2. For example, the server 10 adds weights to the differences between the general models of the two or more clients to update the general model and obtain a new general model.

[0058] The updated new general model is used in the next round t+1. In the next round t+1, the processing of the first inherent model processing unit 1021, the general model processing unit 103, the second inherent model processing unit 1022, and the server 10 is repeated iteratively.

[0059] Compared with the prior art, the model learning system 1000 according to the present invention does not simply have the same model on the client side, but includes a general model and a unique model. Preferably, the unique model includes the first unique model and the second unique model.

[0060] Thus, by preprocessing the data using an intrinsic model, preferably the first intrinsic model, data privacy protection can be achieved without adding noise, that is, privacy protection can be achieved without compromising performance. Furthermore, the intrinsic model is unknown to the server and other clients, further ensuring that data privacy protection can be achieved without adding noise, that is, privacy protection can be achieved without compromising performance.

[0061] On the other hand, the second inherent model that is different between clients can ensure the specificity of the data.

[0062] In addition, the second inherent model fine-tunes the general model trained by the general model processing unit 103. This can further ensure data specificity.

[0063] (Variation)

[0064] Below, refer to Figure 2 、 Figure 3 and Figure 5 , a modified example of the model learning system 1000 of the present invention is described. Figure 5 It is a schematic diagram showing a modified example of the operation of the model learning system according to the present invention.

[0065] The difference between the modified example and the embodiment is that, in the embodiment, the specific structures of the first inherent model, the second inherent model and the general model are different, but the actions and functions are the same, so detailed description is omitted here.

[0066] That is, the general model can be a deep learning network or a portion of a deep learning network; or a deep learning network or a portion of a deep learning network can be cascaded and utilized. The first inherent model and the second inherent model can be a deep learning network, a portion of a deep learning network, or a machine learning model, respectively; or a deep learning network, a portion of a deep learning network, or a machine learning model can be cascaded and utilized. Furthermore, the technical effects of the embodiments can also be achieved when a cascaded application is used.

[0067] While the embodiments of the present invention have been described above, these embodiments are provided as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope and spirit of the invention and are included within the scope of the invention set forth in the claims and their equivalents.

Claims

1. A model learning system, characterized in that: include: server; and two or more clients, each of which is connected to the server in a communicative manner, and each of the two or more clients comprises: a data acquisition unit for acquiring medical image data; an inherent model processing unit for processing the medical image data acquired by the data acquisition unit based on a first inherent model inherent to the client; and a general model processing unit for training the general model acquired from the server based on the medical image data processed by the inherent model processing unit, the general model being shared between the two or more clients through the server, the inherent model processing unit for processing the general model trained by the general model processing unit based on a second inherent model inherent to the client, thereby generating the trained general model, and the server for updating the general model based on the difference between the trained general models of the two or more clients.

2. The model learning system according to claim 1, wherein: The server weights and adds two or more trained general models from the two or more clients based on the difference between the respective general models of the two or more clients, thereby updating the general model and generating a new general model.

3. The model learning system according to claim 1 or 2, wherein: The inherent model processing unit includes: a first inherent model processing unit, which processes the medical image data acquired by the data acquisition unit based on the first inherent model inherent to the client to generate inherent data; and a second inherent model processing unit, which processes the general model trained by the general model processing unit based on the second inherent model inherent to the client to generate the trained general model.

4. The model learning system according to claim 3, wherein: The first intrinsic model and the second intrinsic model are updated through iteration on the client side.

5. The model learning system according to claim 3, wherein: The first inherent model is similar between the two or more clients and is unknown on the server side and other clients except the client.

6. The model learning system according to claim 3, wherein: The first intrinsic model and / or the second intrinsic model are different between the two or more clients and are unknown to the server and other clients except the client.

7. The model learning system according to claim 3, wherein: The general model is a deep learning network or a part of a deep learning network; the first inherent model and the second inherent model are respectively a deep learning network, a part of a deep learning network or a machine learning model.

8. A model learning method in a model learning system, the model learning system comprising: server; and two or more clients, each of which is connected to the server in a communicative manner. The model learning method in the model learning system is characterized in that, in the two or more clients, a data acquisition unit acquires medical image data; an inherent model processing unit processes the medical image data acquired by the data acquisition unit based on a first inherent model inherent to the client; The general model obtained from the server is trained by the general model processing unit based on the medical image data processed by the inherent model processing unit. The general model is shared between the two or more clients through the server. The inherent model processing unit processes the general model trained by the general model processing unit based on the second inherent model inherent to the client, thereby generating the trained general model. The server updates the general model based on the difference between the trained general models of the two or more clients.

9. The model learning method in the model learning system according to claim 8, characterized in that: The server weights and adds the two or more trained general models from the two or more clients based on the difference between the respective general models of the two or more clients, thereby updating the general model and generating a new general model.

10. The model learning method in the model learning system according to claim 8 or 9, characterized in that: The inherent model processing unit processes the medical image data acquired by the data acquisition unit based on the first inherent model inherent to the client through the first inherent model processing unit included therein to generate inherent data; and processes the general model trained by the general model processing unit based on the second inherent model inherent to the client through the second inherent model processing unit included therein to generate the trained general model.

11. The model learning method in the model learning system according to claim 10, characterized in that: The first intrinsic model and the second intrinsic model are updated through iteration on the client side.

12. The model learning method in the model learning system according to claim 10, characterized in that: The first inherent model is similar between the two or more clients and is unknown on the server side and other clients except the client.

13. The model learning method in the model learning system according to claim 10, characterized in that: The first intrinsic model and / or the second intrinsic model are different between the two or more clients and are unknown to the server and other clients except the client.

14. The model learning method in the model learning system according to claim 10, characterized in that: The general model is a deep learning network or a part of a deep learning network; the first inherent model and the second inherent model are respectively a deep learning network, a part of a deep learning network or a machine learning model.

Images