Machine learning system and machine learning method

The system enhances federated learning for personal authentication by using a server-client model with encrypted updates to maintain privacy and accuracy, addressing the limitations of existing techniques by considering class similarities and preventing data leakage.

JP7595036B2Active Publication Date: 2024-12-05HITACHI LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2022006933
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-01-20
Publication Date
2024-12-05
Estimated Expiration
2042-01-20

AI Technical Summary

Technical Problem

Existing federated learning techniques for personal authentication face challenges in maintaining accuracy while protecting privacy, as they either require sharing representative vectors that can lead to identity theft or compromise privacy, or they fail to consider class similarities, resulting in low model accuracy.

Method used

A machine learning system where a server holds a common model and clients hold confidential data and individual models, with the server transmitting the common model to clients for updating, and clients share encrypted learning results to maintain privacy and improve accuracy by considering class similarities.

Benefits of technology

The system achieves a highly accurate common model while protecting personal data and individual models, reducing the risk of impersonation and data leakage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595036000001
    Figure 0007595036000001
  • Figure 0007595036000002
    Figure 0007595036000002
  • Figure 0007595036000003
    Figure 0007595036000003
Patent Text Reader

Abstract

To acquire a high-accuracy common model while protecting concealing target data and an individual model.SOLUTION: A machine learning system has a server for holding a common model and a plurality of clients each for holding concealing target data and an individual model. The server transmits the common model to the plurality of clients. Each of the plurality of clients generates a learning result obtained by updating the common model based on its own concealing target data and its individual model, and transmits the generated learning result to the server. The server updates the common model based on the received learning result.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a machine learning system and a machine learning method. [Background technology]

[0002] Personal authentication is a process to confirm that a user is the person registered in advance, and is carried out in various IT (Information Technology) systems. Biometric authentication, which verifies a person's identity based on an individual's physical or behavioral characteristics, is attracting attention as a highly convenient and reliable means of identity verification, as it does not require the user to memorize a password or carry an IC (Integrated Circuit) card or other personal item.

[0003] The features used in biometric authentication include a user's fingerprint, face, iris, and veins. In biometric authentication, feature values ​​are extracted from these physical or behavioral features, and the extracted feature values ​​are compared with the feature values ​​at the time of registration to determine whether the person is the real person or not. In the past, image features designed by experts were often used as feature values, but in recent years, it has become common to extract features using machine learning, such as deep learning.

[0004] In personal authentication based on machine learning, features with high discriminatory power are learned based on training data, so a huge amount of training data is required to achieve highly accurate authentication. However, in recent years, laws regarding the privacy of personal data, such as the General Data Protection Regulation (GDPR), have become stricter, and clear consent must be obtained and reliable data management is required for the collection of personal data, making it difficult to collect huge amounts of biometric information, such as fingerprints, faces, irises, and veins, as training data.

[0005] A technology called federated learning has been proposed as one of the technologies to solve this problem. Federated learning is a technology that does not collect personal data, but instead aggregates and learns only the weights of models trained using personal data, making it possible to train a machine learning model while protecting the privacy of the personal data. Personal authentication to which federated learning is applied is expected to train a highly accurate model without collecting a huge amount of personal data.

[0006] However, a drawback of federated learning is that accuracy decreases when there is bias in the data for each client. In personal authentication, it is assumed that a client is basically an individual, and the personal data collected by the client is linked to one individual. In such a situation where a client contains only one class of data, data bias is most pronounced, and since the client cannot compare the data with other classes during learning, this leads to a significant decrease in accuracy. For this reason, it is difficult to directly apply federated learning proposed for general classification problems to personal authentication.

[0007] The technology described in Non-Patent Document 1 learns a highly accurate model under the assumption that one client stores personal data for only one person. In the technology described in Non-Patent Document 1, a representative vector associated with an individual is defined for each client, and each client performs learning so that a feature vector extracted from personal data by machine learning approaches the representative vector.

[0008] Furthermore, the technology described in Non-Patent Document 2, like the technology described in Non-Patent Document 1, performs learning while protecting privacy under the premise that personal data of only one person is stored in one client. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] “Federated Learning with Only Positive Labels. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:10946-10956”, Yu, F., Rawat, A.S., Menon, A., Kumar, S., 2020, [Retrieved January 4, 2022], Internet<https: / / proceedings.mlr.press / v119 / yu20f.html> [Non-Patent Document 2] “Federated Learning of User Verification Models Without Sharing Embeddings. Proceedings of the 38th International Conference on Machine Learning, in Proceedings of Machine Learning Research 139:4328-4336“, Hosseini, H., Park, H., Yun, S., Louizos, C., Soriaga, J., Welling, M.., 2021, [Retrieved January 4, 2022], Internet<https: / / proceedings.mlr.press / v139 / hosseini21a.html> Summary of the Invention [Problem to be solved by the invention]

[0010] In the technology described in Non-Patent Document 1, multiple clients transmit representative vectors to a server, and the server performs learning so that the distance between each representative vector becomes sufficiently large. If the server does not perform learning of the representative vector, the distance between the feature vectors of different classes does not become large, so that the feature vectors for all classes converge to the same vector, and a model that cannot distinguish between classes is learned. In contrast, in the technology described in Non-Patent Document 1, the server performs learning of the representative vector, so that the distance between the feature vectors of different classes becomes large, and a highly accurate model that can distinguish between classes can be learned.

[0011] In other words, the technology described in Non-Patent Document 1 requires that the representative vector of each class be shared between the client and the server. Since the representative vector is a representative value of the feature vector used during authentication, a third party who obtains the representative vector can spoof the individual associated with the representative vector.

[0012] Furthermore, a third party who obtains the representative vector and the trained model can infer the original personal data by a method called Model Inversion Attack. Therefore, in the method of sharing the representative vector with the server as in the technology described in Non-Patent Document 1, there is a risk of spoofing or personal data leakage due to eavesdropping on communications or cyber attacks on the server, and it can be said that privacy protection is insufficient.

[0013] On the other hand, in the technology described in Non-Patent Document 2, each client randomly assigns a representative vector that is at least a certain distance away, and learning is performed so that feature vectors extracted from personal data by machine learning approach the representative vector. In the technology described in Non-Patent Document 2, the client manages the representative vector, and the representative vector is not transmitted to the server. Therefore, in the technology described in Non-Patent Document 2, privacy regarding the representative vector is protected, that is, the risk of impersonation of an individual and leakage of personal data can be avoided.

[0014] However, in the technology described in Non-Patent Document 2, representative vectors are randomly assigned and similarity between classes is not taken into consideration. For example, very similar individuals should have similar feature vectors, and completely dissimilar individuals should have feature vectors that are far apart. However, in the technology described in Non-Patent Document 2, such similarity is not taken into consideration, and there is a risk of a situation occurring in which representative vectors of similar individuals are far apart. If a representative vector that does not take into consideration similarity between classes is used, as in the technology described in Non-Patent Document 2, the accuracy of the model obtained by learning will not be sufficiently high.

[0015] In other words, by protecting the privacy of the representative vector (an example of an individual model) while assigning a representative vector that takes into account the similarity between classes and performing training, it is possible to obtain a highly accurate trained model (an example of a common model) while reducing the risk of impersonation and personal data leakage.

[0016] Therefore, one aspect of the present invention is to obtain a highly accurate common model while protecting the data to be concealed and the individual models. [Means for solving the problem]

[0017] In order to solve the above problem, one aspect of the present invention employs the following configuration: A machine learning system includes a server that holds a common model and a plurality of clients, each of which holds data to be concealed and an individual model, wherein the server transmits the common model to the plurality of clients, and each of the plurality of clients generates a learning result in which the common model is updated based on the data to be concealed and the individual model that it holds, and transmits the generated learning result to the server, and the server updates the common model that it holds based on the learning results received from each of the plurality of clients. Effect of the Invention

[0018] According to one aspect of the present invention, it is possible to obtain a highly accurate common model while protecting personal data and individual models.

[0019] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a block diagram showing an example configuration of a machine learning system according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer that realizes each of a client, a learning server, and a parameter server according to the first embodiment. [Diagram 3] FIG. 11 is a sequence diagram showing an example of a federated learning process for personal authentication according to the first embodiment. [Figure 4] 10 is a flowchart illustrating an example of a personal data collection process according to the first embodiment. [Diagram 5] 11 is a flowchart illustrating an example of a local learning process according to the first embodiment. [Figure 6] 11 is a flowchart illustrating an example of a model update process according to the first embodiment. [Figure 7] 13 is a flowchart illustrating an example of an individual model storage process according to the first embodiment. [Figure 8] FIG. 11 is a sequence diagram showing an example of personal authentication processing using a common model according to the first embodiment. [Figure 9] FIG. 11 is a sequence diagram showing an example of a federated learning process for personal authentication according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, the same components are generally designated by the same reference numerals, and repeated explanations are omitted. Note that this embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention.

[0022] First Embodiment In the first embodiment, a machine learning system is described in which a client performs a confidentiality transformation on a representative vector (an example of an individual model) generated from personal data and then shares it with a learning server, the learning server optimizes the representative vector and then shares it with the client, and the client performs an inverse transformation of the confidentiality transformation and then stores the representative vector.

[0023] Fig. 1 is a block diagram showing an example of the configuration of a machine learning system. The machine learning system includes, for example, a client 1000, a learning server 1100, and a parameter server 1200, which are connected to one another via a network such as the Internet. Note that, in the example of Fig. 1, the machine learning system includes one client 1000, but may include multiple clients 1000.

[0024] The client 1000 includes functional units, namely, a personal data acquisition unit 1010, a local learning unit 1020, a model conversion unit 1030, a data encryption unit 1040, a data decryption unit 1050, and an authentication result output unit 1060. The client 1000 also includes a personal data storage unit 1090, a common model storage unit 1091, and a template storage unit 1092, each of which is a memory area for storing data.

[0025] The personal data acquisition unit 1010 acquires data associated with an individual (e.g., a user of the client 1000) from the individual. The local learning unit 1020 learns a common model and an individual model based on the personal data acquired by the personal data acquisition unit 1010. The model conversion unit 1030 performs confidentiality conversion on the representative vector generated from the personal data and the common model.

[0026] The data encryption unit 1040 encrypts the personal data and the representative vector to prevent leakage of personal information. The data decryption unit 1050 decrypts the personal data and the representative vector encrypted by the data encryption unit 1040 to extract the original data. The authentication result output unit 1060 outputs the result of personal authentication using the personal data.

[0027] The personal data storage unit 1090 stores personal data and representative vectors. The common model storage unit 1091 stores a trained common model. The trained common model is used for personal authentication. The template storage unit 1092 stores templates used for personal authentication processing using the trained common model.

[0028] The learning server 1100 includes, for example, functional units, a client selection unit 1110 and a model update unit 1120. The learning server 1100 also includes a common model storage unit 1190, which is an area for storing data.

[0029] The client selection unit 1110 selects clients 1000 that will participate in the current learning round from among the clients 1000 that can participate in learning. The model update unit 1120 updates the weights of the common model and the individual model using personal data. The model update unit 1120, for example, defines a loss function for the model and minimizes the loss function to update the weights. The common model storage unit 1190 holds the learned common model.

[0030] The parameter server 1200 includes, for example, a parameter generating unit 1210 which is a functional unit, and a parameter storage unit 1290 which is a storage area for storing data. The parameter server 1200 is an independent device from the client 1000 and the learning server 1100, and is assumed to have a sufficiently low risk of information leakage. However, if sufficient safety is ensured, the parameter server 1200 may exist in the same device, such as on a different virtual environment in the same physical server as the learning server 1100. The client 1000 and the parameter server 1200 may be integrated, that is, the client 1000 may generate and manage parameters.

[0031] The parameter generating unit 1210 generates parameters used for the confidentiality transformation of the individual model. The confidentiality transformation is a process of encrypting the individual model to make it difficult for a third party who does not know the parameters to restore or guess the individual model. For example, random projection or random padding is applied as a method for implementing encryption of the individual model. In particular, when an orthogonal normal matrix is ​​used as the transformation base, it is possible to make the distance between the individual models before the confidentiality transformation equal to the distance between the individual models after the confidentiality transformation. The parameter storage unit 1290 holds the parameters generated by the parameter generating unit 1210.

[0032] 2 is a block diagram showing an example of the hardware configuration of a computer that realizes each of the client 1000, the learning server 1100, and the parameter server 1200. The computer includes, for example, a CPU (Central Processing Unit) 2010, a memory 2020, an auxiliary storage device 2030, an input device 2040, an output device 2050, and a communication device 2060, which are connected to each other by an internal communication line such as a bus.

[0033] The CPU 2010 includes a processor and executes programs stored in the memory 2020. The memory 2020 includes a ROM (Read Only Memory), which is a non-volatile storage element, and a RAM (Random Access Memory), which is a volatile storage element. The ROM stores immutable programs (e.g., a Basic Input / Output System (BIOS)). The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the CPU 2010 and data used when the programs are executed.

[0034] The auxiliary storage device 2030 is a large-capacity non-volatile storage device such as a magnetic storage device (HDD (Hard Disk Drive)) or a flash memory (SSD (Solid State Drive)), and stores programs executed by the CPU 2010 and data used when the programs are executed. In other words, the programs are read from the auxiliary storage device 2030, loaded into the memory 2020, and executed by the CPU 2010.

[0035] The input device 2040 is a device that receives input from an operator, such as a keyboard, a touch panel, a smart device, or a mouse. The input device 2040 also includes a device for acquiring biometric information (an example of personal data), such as a biometric sensor, a scanner, or a camera. The output device 2050 is a device that outputs the execution result of a program in a format that can be viewed by the operator, such as a display device or a printer.

[0036] The communication device 2060 is a network interface device that controls communication with other devices in accordance with a predetermined protocol. The communication device 2060 may also include a serial interface such as a Universal Serial Bus (USB).

[0037] Some or all of the programs executed by the CPU 2010 may be provided to the computer from a removable medium (such as a CD-ROM or flash memory) that is a non-transitory storage medium, or from an external computer equipped with a non-transitory storage device via a network, and stored in a non-volatile auxiliary storage device 2030 that is a non-transitory storage medium. For this reason, the computer should preferably have an interface for reading data from removable media.

[0038] Each of the client 1000, the learning server 1100, and the parameter server 1200 is a computer system configured on a single physical computer or on multiple logically or physically configured computers, and may operate in separate threads on the same computer, or may operate on a virtual computer constructed on multiple physical computer resources.

[0039] The CPU 2010 of the computer constituting the client 1000 includes a personal data acquisition unit 1010, a local learning unit 1020, a model conversion unit 1030, a data encryption unit 1040, and a data decryption unit 1050. The CPU 2010 of the computer constituting the learning server 1100 includes a client selection unit 1110 and a model update unit 1120. The CPU 2010 of the computer constituting the parameter server 1200 includes a parameter generation unit 1210.

[0040] For example, the CPU 2010 of a computer constituting a client 1000 functions as a personal data acquisition unit 1010 by operating in accordance with a personal data acquisition program loaded into the memory 2020 of the computer, and functions as a local learning unit 1020 by operating in accordance with a local learning program loaded into the memory 2020 of the computer.

[0041] The same relationship exists between the programs and the functional units of the other functional units included in the CPU 2010 of the computer constituting the client 1000. The same relationship exists between the programs and the functional units of the functional units included in the CPU 2010 of the computer constituting the learning server 1100 and the functional units included in the CPU 2010 of the computer constituting the parameter server 1200.

[0042] Furthermore, the auxiliary storage device 2030 of the computer constituting the client 1000 provides storage areas for realizing the personal data storage unit 1090, the common model storage unit 1091, and the template storage unit 1092. The auxiliary storage device 2030 of the computer constituting the learning server 1100 provides a storage area for realizing the common model storage unit 1190. The auxiliary storage device 2030 of the computer constituting the parameter server 1200 provides a storage area for realizing the parameter storage unit 1290.

[0043] Fig. 3 is a sequence diagram showing an example of federated learning processing for personal authentication. Three parties, a client 1000, a learning server 1100, and a parameter server 1200, perform federated learning of a model used for personal authentication. In the example of Fig. 3, multiple (two) clients 1000 are shown, but since each of the clients 1000 executes similar processing, the processing of the client 1000 will be explained without duplication.

[0044] The personal data acquisition unit 1010 of the client 1000 acquires personal data from the user 210 (e.g., the user of the client 1000) (S3010). The personal data is data used for personal authentication, and includes, for example, physical information such as fingerprints, faces, irises, or veins, and information indicating behavioral characteristics such as an individual's acceleration information, movement history, browsing history, and purchase history. Any data to be kept secret may be used instead of the personal data.

[0045] The personal data acquisition process in step S3010 may be performed only for the purpose of collecting data, or the acquired personal data may be stored for the purpose of authentication. By repeatedly performing the process in step S3010, the personal data of the user 210 is accumulated in the client 1000.

[0046] When a client 1000 is occupied by one user 210, personal data for that specific user 210 is stored. On the other hand, for example, a client 1000 shared by multiple users 210 may store personal data of the multiple users, or personal data stored by other multiple clients 1000 may be aggregated in one client 1000 and used for federated learning. Details of step S3010 will be described later with reference to FIG. 4.

[0047] The client selection unit 1110 of the learning server 1100 selects participating clients 1000 that will participate in the learning in the current round (S3110). Specifically, for example, the client selection unit 1110 selects the participating clients 1000 by a method of selecting all clients 1000, a method of inquiring the clients 1000 as to whether they can participate in the learning and selecting the clients 1000 that have responded that they can participate, a method of randomly selecting clients 1000 according to a predetermined ratio, a method of selecting clients 1000 according to a predetermined upper limit, a method of calculating an evaluation value for the clients 1000 (for example, the more personal data collected, the higher the evaluation value, and the older the date and time of last participation in learning, the higher the evaluation value) and selecting a predetermined number of clients 1000 with high or low calculated evaluation values, a method of randomly selecting a predetermined number of clients 1000 after weighting according to the calculated evaluation value, or the like.

[0048] The client selection in step S3110 may be performed by the parameter server 1200 instead of the learning server 1100. In this case, the parameter server 1200 manages the clients 1000 that can participate, and selects a client from the clients 1000 that can participate by the selection method described above.

[0049] The model update unit 1120 of the learning server 1100 makes a parameter generation request to the parameter server 1200 (S3120), and the parameter generation unit 1210 of the parameter server 1200 generates parameters (S3210).

[0050] The model update unit 1120 of the learning server 1100 may include in the parameter generation request an IP address or a terminal number as an identifier that uniquely identifies each participating client 1000 selected in step S3110. However, if the parameter server 1200 executes the client selection process in step S3110, the model update unit 1120 of the learning server 1100 does not need to include data for identifying the client 1000 in the parameter generation request.

[0051] In federated learning, the model is updated through communication with the participating clients 1000 in a fixed period called a round, and the parameters are used commonly for all participating clients 1000 in the current round. These parameters are used in the individual model conversion process in step S3050, which will be described later, and are defined, for example, in the form of a randomly generated vector or matrix.

[0052] The model update unit 1120 of the learning server 1100 transmits the common model stored in the common model storage unit 1190 to each participating client 1000 (S3130). The common model is a machine learning model that inputs personal data and outputs a feature vector, and is, for example, a general machine learning model such as a neural network.

[0053] For example, when image data such as fingerprints, faces, irises, or veins are used as personal data, a deep learning model such as CNN (Convolutional Neural Networks) or Transformer can be applied as the common model. Note that the common model may be initialized in advance with random weights at the start of learning, or may be optimized in advance with weights pre-trained using personal data other than that stored in the client 1000.

[0054] The participating client 1000 receives the common model from the learning server 1100 (S3020), and the local learning unit 1020 of the participating client 1000 uses the personal data to execute a local learning process for updating the common model and the individual model (S3030).

[0055] As explained in step S3130, the common model is a machine learning model, while the individual model, unlike the common model, is a model that is not shared in its original form with the learning server 1100 (it is shared in an encrypted form) and is a different model for each client 1000.

[0056] For example, in distance learning in which feature vectors are extracted from personal data and the distance between the feature vectors reflects the similarity between the personal data, a representative vector of the feature vectors can be used as an individual model. This representative vector is sensitive information linked to the user 210 of the client 1000, and if this data is obtained by a third party, there is a risk of impersonation or personal data leakage. Details of the local learning process in step S3030 will be described later with reference to FIG. 5.

[0057] The model conversion unit 1030 of the participating client 1000 makes a parameter request to the parameter server 1200 (S3040). Note that in the example of Fig. 3, the parameter server 1200 distributes parameters to the participating clients 1000, thereby synchronizing the participating clients 1000, but the method by which the participating clients 1000 obtain the parameters is not limited to this.

[0058] For example, each participating client 1000 may generate parameters by performing conversion using a hash function on data that is commonly obtained by each participating client 1000, such as date and time. Also, each participating client 1000 may manage a hardware token that stores secret information and generate common parameters on the token. In this way, when each participating client 1000 generates parameters independent of the parameter server 1200, the machine learning system does not need to include the parameter server 1200.

[0059] The parameter generating unit 1210 of the parameter server 1200 verifies the parameter request from the participating client 1000 (S3220). Since the parameters are data equivalent to the encryption key used in the individual model conversion process in step S3050 described later, a third party who obtains the confidentially converted data and the parameters at the same time can decrypt the original data. Therefore, it is desirable that the parameters be shared only with the correct participating client 1000.

[0060] Therefore, in verifying the parameter request in step S3220, the parameter generation unit 1210 determines whether the participating client 1000 that sent the parameter request is a participating client 1000 for the current round, and sends the parameters to that participating client 1000 only if it determines that the participating client 1000 is a participating client 1000 (S3230).

[0061] When the parameter generation unit 1210 determines that it has received a parameter request from a client 1000 that is not a participating client 1000, it conceals the parameters by taking measures such as not sending parameters to the client 1000 or sending parameters different from the original parameters to the client 1000.

[0062] The model transformation unit 1030 of the participating client 1000 performs confidential transformation on the individual model based on the parameters received from the parameter server 1200, thereby generating a confidential individual model (S3050).

[0063] The individual model is a model associated with an individual defined by a vector or a matrix. For example, when distance learning is performed on a feature vector for personal data, a representative vector of the feature vector can be used as the individual model. The individual model includes information on the feature vector for personal data, and can be used to spoof or infer personal data. Therefore, when sharing an individual model with the learning server 1100, the parameter server 1200, and other clients 1000, the participating client 1000 performs a confidential transformation on the individual model to make the original individual model unavailable to these devices. The model conversion unit 1030 of the participating client 1000 transmits the common model and the private individual model as the learning results to the learning server 1100 (S3060).

[0064] In addition, when the participating client 1000 transmits the learning result directly to the learning server 1100, the learning server 1100 can recognize which participating client 1000 transmitted the transmitted learning result. In this case, if some personal information is leaked from the common model or the secret individual model, it is possible to identify which participating client 1000 the information belongs to. In order to prevent this, when transmitting the learning result from the participating client 1000 to the learning server 1100, a shuffle server may be interposed between the participating client 1000 and the learning server 1100, and the shuffle server may rearrange the order of the learning results received from the client 1000 or assign a different identifier to the learning results. In this way, it becomes impossible to grasp the correspondence between the data received from the learning server 1100 and the client 1000, thereby achieving higher security.

[0065] The learning server 1100 receives the common model and the secret individual models from the multiple participating clients 1000, and the model update unit 1120 of the learning server 1100 executes a model update process including aggregation of the common model and optimization of the individual models (S3140). Details of the model update process in step S3140 will be described later with reference to FIG. 6.

[0066] The model update unit 1120 of the learning server 1100 transmits the secret individual model updated in step S3140 to each participating client (S3150). Since a secret individual model exists for each participating client 1000, the model update unit 1120 transmits each secret individual model only to the corresponding participating client 1000. If the model update unit 1120 mistakenly transmits a secret individual model to another participating client 1000, the participating client 1000 will be able to decrypt the individual models of the other participating clients 1000 because it knows the parameters, creating a risk of impersonation or personal data inference.

[0067] The model transformation unit 1030 of the participating client 1000 acquires an individual model by performing an inverse transformation of the confidential transformation on the confidential individual model received from the learning server 1100, and stores the acquired individual model in the personal data storage unit 1090 of the participating client 1000 (S3070). Details of the individual model storage process in step S3070 will be described later with reference to FIG.

[0068] Through the above processing, it becomes possible to perform federated learning of a common model in the learning server 1100 while protecting the privacy of personal data and individual models in the client 1000.

[0069] 4 is a flowchart showing an example of the personal data collection process in step S3010. The personal data acquisition unit 1010 of the client 1000 acquires personal data from the user 210 (S4010). As described in step S3010, the personal data includes, for example, physical information of the user 210, such as a fingerprint, face, iris, and veins, and information indicating behavioral characteristics of the user 210, such as acceleration information, movement history, browsing history, and purchase history.

[0070] The personal data acquiring unit 1010 selects personal data to be used in the federated learning from the personal data acquired in step S4010 (S4020). Note that the personal data selection process in step S4020 may be omitted and all of the personal data acquired by the personal data acquiring unit 1010 in step S4010 may be used for learning, but the personal data acquiring unit 1010 can reduce the personal data by executing the data selection process in step S4020, and thus the amount of calculation for learning and the storage capacity of the personal data storage unit 1090 can be reduced.

[0071] In addition, in the personal data selection process of step S4020, the personal data acquisition unit 1010 can eliminate personal data that contributes little to learning or that has a negative effect on learning, thereby improving the accuracy of the final common model.

[0072] In addition, in the personal data selection process of step S4020, the personal data acquisition unit 1010 specifically calculates an evaluation value for each personal data, and selects personal data whose evaluation value is equal to or greater than a predetermined threshold, personal data whose evaluation value is less than a predetermined threshold, a predetermined number of personal data whose evaluation value is at the top, a predetermined number of personal data whose evaluation value is at the bottom, or personal data whose evaluation value is within a predetermined range.

[0073] In addition, the personal data acquisition unit 1010 may select personal data without calculating an evaluation value for each personal data, for example, by calculating an evaluation value for the distance between the acquired personal data or an evaluation value for a set of acquired personal data, and repeatedly excluding personal data based on the evaluation value.

[0074] For example, when face images are used as personal data, face images appearing in a video are very similar in every frame, and learning a large number of similar face images is inefficient and difficult to improve accuracy.Face images that are facing straight to the side, that are covered by hands, or that are outside the imaging range have too large an error and are therefore unsuitable as learning data.

[0075] In order to exclude such personal data, the personal data acquisition unit 1010 performs feature extraction for each face image to generate a feature vector, and if the distance between the feature vectors is sufficiently small, that is, if the face images are sufficiently similar, one of the face images can be excluded from the learning data, thereby excluding similar personal data. In addition, the personal data acquisition unit 1010 calculates a quality value including the face direction, the presence or absence of occlusion, and the presence or absence of protrusion from the face image, and can exclude inappropriate personal data by excluding face images whose quality value is equal to or greater than a predetermined value, or excluding face images whose quality value is less than the predetermined value.

[0076] Next, the data encryption unit 1040 of the client 1000 encrypts the personal data selected in step S4020 (S4030). The personal data encryption process in step S4030 is performed to prevent personal data leakage due to unauthorized access by a third party to the client 1000 and malware infection.

[0077] For example, if the client 1000 is trustworthy or if the security of the client 1000 is guaranteed by means other than data encryption, the personal data acquisition unit 1010 does not need to perform the personal data encryption process in step S4030.

[0078] In order to encrypt personal data, a private key is generally required. This private key is stored in the auxiliary storage device 2030 in the client 1000, in a secure area in the client 1000 such as a TEE (Trusted Execution Environment), or in an external medium such as a hardware token.

[0079] The data encryption unit 1040 may dynamically generate a secret key from personal data such as biometric information using a biometric encryption technique such as Fuzzy Extractor. In this case, the data encryption unit 1040 can dynamically generate a secret key from the personal data to be learned and execute the personal data encryption process, and there is no need to prepare a secret key in advance for encryption.

[0080] Next, data encryption unit 1040 stores the personal data encrypted in step S4030 in personal data storage unit 1090 (S4040). As described above, client 1000 can acquire personal data from user 210, encrypt the acquired personal data in a secure form, and store the encrypted data in personal data storage unit 1090.

[0081] 5 is a flowchart showing an example of the local learning process in step S3030. The data decryption unit 1050 of the participating client 1000 decrypts the personal data stored in the personal data storage unit 1090 (S5010). Note that to decrypt the personal data, the private key used when the personal data was encrypted is required. The data decryption unit 1050 acquires or generates this private key in the same procedure as in step S4030.

[0082] Next, the local learning unit 1020 of the participating client 1000 selects learning data to be used for local learning from the personal data decrypted in step S5010 (S5020). Specifically, for example, the local learning unit 1020 randomly selects a predetermined number of personal data from the decrypted personal data.

[0083] In addition, the local learning unit 1020 may calculate an evaluation value for each piece of personal data, and select as learning data personal data whose evaluation value is equal to or greater than a predetermined threshold, personal data whose evaluation value is less than a predetermined threshold, a predetermined number of personal data whose evaluation value is in the top ranks, a predetermined number of personal data whose evaluation value is in the bottom ranks, or personal data whose evaluation value is within a predetermined range.

[0084] In addition, the local learning unit 1020 may not calculate an evaluation value for each personal data, but may instead calculate an evaluation value for the distance between the acquired personal data or an evaluation value for a set of acquired personal data, and repeatedly exclude personal data based on the evaluation value to select personal data as learning data.

[0085] Furthermore, from the viewpoint of learning, if the distance between the feature vector calculated from personal data and the representative vector is too close, the contribution to learning is small and the effectiveness is low, and if it is too far, it is difficult to reduce the distance through learning. For this reason, the local learning unit 1020 calculates the distance between the feature vector and the representative vector, and leaves only personal data whose distance is within a predetermined range as learning data, thereby enabling efficient and highly accurate learning.

[0086] Next, local learning unit 1020 determines whether or not an individual model is stored in personal data storage unit 1090 (S5030). If local learning unit 1020 determines that an individual model is stored in personal data storage unit 1090 (S5030: YES), data decryption unit 1050 reads the encrypted individual model from personal data storage unit 1090 and decrypts it (S5041).

[0087] When it is determined that the individual model is not stored in the personal data storage unit 1090 (S5030: None), the local learning unit 1020 generates an individual model (S5040). Specifically, for example, the local learning unit 1020 generates a feature vector by inputting the personal data into a common model, and performs a predetermined statistical calculation such as an average value or a median value on the obtained feature vector to generate an individual model. The local learning unit 1020 may store a pre-learning model at the start of the federated learning as a common model and generate an individual model using the stored common model, or may generate an individual model using the latest common model received by the participating client 1000 in step S3020. The local learning unit 1020 may generate an individual model by initializing the individual model with random weights without using personal data.

[0088] The local learning unit 1020 learns the common model and the individual models using the learning data selected in step S5020, the individual models decoded in step S5040 or generated in step S5041, and the common model received in step S3020 (S5050). Specifically, for example, the local learning unit 1020 updates these models by defining a loss function for the models and searching for weights that minimize the loss function.

[0089] The loss function is a function that models the degree to which the personal data and its corresponding common model and individual model have an ideal relationship, and defines it as the loss for the ideal relationship. For example, when a representative vector of feature vectors generated from personal data is used as an individual model, the ideal relationship is when the feature vector generated by inputting all personal data into the common model is equal to the representative vector.

[0090] Therefore, when the sum of the squares of the distances between the feature vectors and the representative vectors is used as the loss function, the local learning unit 1020 minimizes the loss function, so that the feature vectors approach the representative vectors and an ideal relationship is formed.

[0091] In addition, when a deep learning model such as CNN (Convolutional Neural Networks) or Transformer is used as the common model, the local learning unit 1020 applies an optimization method such as stochastic gradient descent to learn these models by searching for weights of the common model and individual models that minimize the loss function.

[0092] Note that local learning unit 1020 may add noise to the personal data when learning these models. It is known that the personal data used in learning can be inferred from the learning results of participating clients 1000, but adding noise makes it difficult to infer the personal data.

[0093] 6 is a flowchart showing an example of the model update process in step S3140. The model update unit 1120 of the learning server 1100 determines whether each participating client 1000 that has transmitted the learning result to the learning server 1100 is the correct participating client 1000 selected in step S3110 (S6110).

[0094] The client selection unit 1110 records the terminal number, IP address, etc. as information for identifying the participating client 1000 selected in step S3110, and the model update unit 1120 makes the judgment in step S6110 by comparing the participating client 1000 that sent the learning result to the learning server 1100 in step S3140 with the above-mentioned identification information.

[0095] The model update unit 1120 executes the process from step S6120 onwards for the correct participating client 1000, and excludes the incorrect client 1000 from the target of the model update process. There is a known attack called a Model Poisoning Attack, in which a participating client 1000 transmits an incorrect learning result to reduce the accuracy of the model, and the model update unit 1120 executes the process of step S6110 to prevent this attack.

[0096] Next, the model update unit 1120 aggregates the common models received from the multiple participating clients 1000 (S6120). Specifically, for example, the model update unit 1120 calculates the weight of a single common model from the weights of the common models learned by each participating client 1000.

[0097] For example, the model update unit 1120 uses Federated Averaging, which calculates the average value of the weights of the common model learned by each participating client 1000 as the aggregated weight.

[0098] Since it is known that personal data can be inferred from the weights of the common model, it is preferable that the participating clients 1000 do not transmit the common model in its original form to the learning server 1100, and that the model update unit 1120 aggregates the weights of the common model while keeping the weights secret. A technique called Secure Aggregation has been proposed as a method for aggregating the weights of the common model while keeping them secret. By using this technique in combination with the present embodiment, it is possible to calculate the average weights of the common model while keeping the weights of the individual common models secret from the learning server 1100.

[0099] Next, the model update unit 1120 optimizes the individual models received from each participating client 1000 (S6130). For example, when a representative vector for a feature vector generated from personal data is used as an individual model, the representative vectors of the users 210 of each participating client 1000 that have been concealed are aggregated in the learning server 1100. Since this representative vector is a vector located at the center of the feature vectors output by inputting the personal data of each user 210 into a common model, if the distance between the representative vectors of two users 210 is small, it becomes difficult to distinguish the two users 210 by their feature vectors, and the discrimination ability decreases. Therefore, it is desirable that the distance between all representative vectors is equal to or greater than a predetermined value.

[0100] The model update unit 1120 can optimize the representative vector by finding a representative vector that reduces the loss function expressing this property (for example, by finding the gradient of the representative vector with respect to the loss function and updating it in the gradient direction). Specifically, for example, a margin value between the representative vectors is defined, and when the distance between the representative vectors is smaller than the margin, the margin and the square of the distance between the representative vectors are calculated, and a loss function based on the sum of the calculated squared distances can be defined. When the value of this loss function becomes 0, an ideal state is reached in which the distance between all representative vectors is equal to or greater than the margin. Note that the loss function is not limited to one that uses the margin and the distance between the representative vectors, and any function that reduces the output when the distance between the representative vectors becomes longer, such as a Softmax Cross Entropy function, can be applied.

[0101] Although the individual models are transmitted from the participating clients 1000 in a concealed state using parameters, it is possible to perform optimization equivalent to that when the models are not concealed, depending on the concealment method. For example, when an orthogonal normal matrix is ​​used as a parameter and concealment is performed using projective transformation, the distance between representative vectors before concealment and the distance between representative vectors after concealment become equal, so that equivalent optimization can be performed.

[0102] Next, the model update unit 1120 stores the common model aggregated in step S6120 in the common model storage unit 1190 (S6140).

[0103] 7 is a flowchart showing an example of the individual model storage process in step S3070. The model transformation unit 1030 of the participating client 1000 performs an inverse transformation of the confidential transformation on the confidential individual model transmitted by the learning server 1100 in step S3150 to generate an individual model (S7010).

[0104] This inverse transformation is a transformation from the secret individual model optimized by the learning server 1100 to an optimized individual model, and an individual model that can be used for model learning in step S5050 is acquired by this transformation. For example, when a projective transformation using orthogonal matrix parameters is used as the secret transformation, the model transformation unit 1030 can acquire an optimized individual model by performing a projective transformation using an inverse matrix of the parameters.

[0105] Next, the data encryption unit 1040 of the participating client 1000 encrypts the individual model obtained by the inverse conversion in step S7010 (S7020). Specifically, for example, the data encryption unit 1040 executes encryption using a secret key by any of the methods described in the encryption process of the personal data in step S4030. The model conversion unit 1030 stores the individual model encrypted in step S7020 in the personal data storage unit 1090 (S7030).

[0106] Fig. 8 is a sequence diagram showing an example of personal authentication processing using the common model learned in the procedure of Fig. 3. In addition to the personal authentication processing, Fig. 8 describes a model setting processing and a personal registration processing for performing the personal authentication processing.

[0107] First, steps S8010 and S8110 are executed as model setting processing. The model update unit 1120 of the learning server 1100 acquires the common model from the common model storage unit 1190 and transmits it to the client 1000 (S8110). The local learning unit 1020 of the client 1000 receives the common model and stores it in the common model storage unit 1091 (S8010). Note that the model setting processing does not need to be performed every time personal authentication is performed, and is performed by a developer or administrator of the authentication system when operation of a new authentication system is started, when the authentication system is upgraded, etc.

[0108] Next, steps S8020, S8030, and S8040 are executed as a personal registration process. The personal data acquisition unit 1010 of the client 1000 acquires (registration) personal data from the user 210 (S8020), and inputs the acquired personal data into the common model stored in the common model storage unit 1091 to generate a feature vector (S8030).

[0109] The personal data acquisition unit 1010 generates a registration template from the generated feature vector and stores it in the template storage unit 1092 (S8040). A template is information for registration generated from personal data. The personal data acquisition unit 1010 may treat the feature vector itself as a template. Note that the personal data acquisition unit 1010 may take measures to prevent the original personal data from leaking from the template by applying a predetermined template protection technology when generating the template.

[0110] Finally, steps S8050, S8060, S8070, and S8080 are executed as personal authentication processing. The personal data acquisition unit 1010 acquires personal data (for authentication) from the user 210 (S8050), and inputs the acquired personal data into the common model stored in the common model storage unit 1091 to generate a feature vector (S8060).

[0111] The authentication result output unit 1060 compares the feature vector generated in step S8060 with the template stored in the template storage unit 1092, and calculates the degree of similarity or dissimilarity as the comparison result (S8070).

[0112] For example, when performing authentication based on the distance between feature vectors, the authentication result output unit 1060 calculates the Hamming distance or Euclidean distance between the feature vectors and treats the calculated distance as dissimilarity. The authentication result output unit 1060 performs threshold processing or the like on the matching result, determines the authentication result as success or failure, and outputs the result to, for example, the output device 2050 of the computer constituting the client 1000 (S8080).

[0113] The matching result is not limited to a continuous value such as similarity or dissimilarity, and when a template is generated using a template protection technique, the matching result may be a binary value of authentication success or failure. In this case, the authentication result output unit 1060 does not need to execute the threshold processing described above, and may simply output the matching result itself as the authentication result.

[0114] As described above, the machine learning system in the first embodiment can learn a highly accurate common model while protecting the privacy of individual models.

[0115] Second Embodiment The machine learning system according to the second embodiment does not perform the concealment conversion process for the individual model that was performed in the first embodiment. In addition, the machine learning system according to the second embodiment uses the individual model that was initially assigned based on personal data in a fixed manner, that is, does not perform individual model optimization. Therefore, the machine learning system according to the second embodiment does not need to transmit and receive even the individual model that has been concealed, and can achieve higher privacy protection than the first embodiment.

[0116] However, since the machine learning system according to the second embodiment does not update individual models, there is a possibility that the individual models of two different clients 1000 may be similar in their initial assignments. In such a case, a common model that outputs similar feature vectors for the users of the two different clients 1000 is learned, resulting in reduced accuracy.

[0117] For this reason, in order for the machine learning system of the second embodiment to learn a highly accurate common model, it is necessary to initialize the common model with weights that have a relatively high discrimination ability. In this case, since the individual model does not change, discrimination ability between individuals cannot be learned, and accuracy is improved by reducing intra-individual variation so that all personal data is close to the individual model.

[0118] Fig. 9 is a sequence diagram showing an example of federated learning processing for personal authentication. In the following, of the processing in Fig. 9, differences from Fig. 3 will be described, and a description of processing similar to the processing in Fig. 3 will be omitted as appropriate.

[0119] 9 does not use parameters used in the concealment conversion, the machine learning system of the second embodiment does not need to include the parameter server 1200. Furthermore, since transmission and reception of parameters and conversion using parameters are not required, the processes of steps S3120, S3040, S3050, S3150, and S3070 are not executed. Furthermore, in the model learning process of step S5050 included in the local learning process of step S3030, an update process of the individual model is not performed.

[0120] In the first embodiment, if a third party other than the user of the client 1000 obtains both the secret individual model and the parameters through a sophisticated attack, the third party can obtain the individual model by inverse conversion.

[0121] Since a third party who obtains an individual model can spoof or estimate personal data, it is necessary to prevent leakage of the individual model. Therefore, in the first embodiment, security measures must be taken to minimize the risk of information leakage due to simultaneous attacks on the learning server 1100 and the parameter server 1200.

[0122] On the other hand, in the second embodiment, since no information related to the individual model is released outside the client 1000, even if the learning server 1100 is attacked, the individual model and personal data will not be immediately leaked. As a result, the second embodiment can realize more secure federated learning than the first embodiment.

[0123] The present invention is not limited to the above-mentioned embodiment, and various modifications are included. For example, the above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0124] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0125] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0126] 1000 client, 1010 personal data acquisition unit, 1020 local learning unit, 1030 model conversion unit, 1040 data encryption unit, 1050 data decryption unit, 1090 personal data storage unit, 1091 common model storage unit, 1100 learning server, 1110 client selection unit, 1120 model update unit, 1190 common model storage unit, 1200 parameter server, 1210 parameter generation unit, 1290 parameter storage unit

Claims

1. 1. A machine learning system, comprising: A server that holds a common model; A plurality of clients each holding data to be concealed and an individual model; The server transmits the common model to the plurality of clients; Each of the plurality of clients comprises: Generate a learning result by updating the common model based on the confidential data and the individual model held by the device itself; Transmitting the generated learning result to the server; The server updates the common model held by the server based on the learning results received from each of the plurality of clients; Each of the plurality of clients comprises: Generate a private individual model by performing a private transformation on the individual model held by the user, Transmitting the generated secret individual model to the server in association with identification information for identifying the client; The server, updating each of the secret individual models based on the secret individual models transmitted by the plurality of clients; Send each updated private individual model to a client identified by the corresponding identification information; Each of the plurality of clients comprises: Restoring an updated individual model from the updated secret individual model received from the server; A machine learning system that updates its own individual models.

2. The machine learning system according to claim 1, The machine learning system, wherein the concealment transformation includes a projective transformation with an orthogonal matrix as a parameter.

3. The machine learning system of claim 1, A parameter server for storing parameters; The parameter server transmits the parameters to the clients; Each of the multiple clients performs the confidentiality conversion using parameters received from the parameter server.

4. The machine learning system of claim 1, The data to be concealed is personal data, A first client included in the plurality of clients includes: The common model; A template based on a feature vector for enrollment obtained by inputting personal data for enrollment into the common model; Hold personal data for authentication, inputting the authentication personal data into the common model to obtain an authentication feature vector; A machine learning system that performs personal authentication based on the template and the acquired authentication feature vector.

5. The machine learning system of claim 1, The individual model held by each of the plurality of clients is It is a fixed model that is proprietary to the client, A machine learning system generated based on the common model and personal data held by the client.

6. A machine learning method using a machine learning system, comprising: The machine learning system includes: A server that holds a common model; A plurality of clients each holding data to be concealed and an individual model; The machine learning method includes: the server transmits the common model to the plurality of clients; Each of the plurality of clients generates a learning result by updating the common model based on the anonymity target data and the individual model held by the client; Each of the plurality of clients transmits the generated learning result to the server; The server updates the common model held by the server based on the learning results received from each of the plurality of clients; Each of the plurality of clients performs a private transformation on an individual model held by the client to generate a private individual model; each of the plurality of clients transmits the generated secret individual model to the server in association with identification information for identifying the client; The server updates each of the secret individual models based on the secret individual models transmitted by the multiple clients; The server transmits each of the updated secret individual models to a client indicated by the corresponding identification information; Each of the plurality of clients restores an updated individual model from the updated secret individual model received from the server; A machine learning method, wherein each of the multiple clients updates its own individual model.

7. The machine learning method according to claim 6, The machine learning method, wherein the concealment transformation includes a projective transformation with an orthogonal matrix as a parameter.

8. The machine learning method according to claim 6, The machine learning system includes a parameter server that stores parameters; The machine learning method includes: The parameter server transmits the parameters to the clients; The machine learning method, wherein each of the plurality of clients performs the confidentiality conversion using parameters received from the parameter server.

9. The machine learning method according to claim 6, The data to be concealed is personal data, A first client included in the plurality of clients includes: The common model; A template based on a feature vector for enrollment obtained by inputting personal data for enrollment into the common model; Hold personal data for authentication, The machine learning method includes: the first client inputs the authentication personal data into the common model to obtain an authentication feature vector; The first client performs personal authentication based on the template and the acquired authentication feature vector.

10. The machine learning method of claim 6, comprising: The individual model held by each of the plurality of clients is It is a fixed model that is proprietary to the client, A machine learning method generated based on the common model and personal data held by the client.

Citation Information

Patent Citations

  • Training and prediction methods and devices of federal transfer learning model

    CN110399742A

  • Federal learning-based model training method and electronic equipment

    CN113705823A