Learning system and learning method

Abstract

To provide a learning system and learning method that reduce a difference in authentication accuracy between individuals having different attributes by associative learning.SOLUTION: In a learning system, a training server manages a common model. Client terminals and a training data management server manage individual data, generate an individual model different for each individual from the common model and the individual data, share the common model and the individual model with the training server, receive the common model from the training server, update the common model and the individual model on the basis of the individual data, and transmit the updated common model and individual model to the training server. The training server classifies the common model and individual model transmitted from the plurality of client terminals on the basis of the individual model transmitted from the training data management server and updates the common model and the individual model in accordance with a classification result.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a learning system and a learning method. [Background technology]

[0002] Biometric authentication technology that authenticates individuals based on images of faces, fingerprints, irises, etc. is becoming increasingly common. In particular, the accuracy of facial recognition has been greatly improved through the use of deep learning. However, training a facial recognition model requires a large number of facial images.

[0003] In recent years, the development of privacy protection systems has made it difficult to collect biometric information, including facial images. To solve this problem, a technology called federated learning has attracted attention. This technology performs learning within client devices and updates models without collecting training data by sharing only the parameters of the trained model.

[0004] Furthermore, with regard to biometric authentication technology, differences in authentication accuracy between individuals with different attributes such as gender and skin color have attracted attention. A 2019 study by the US National Institute of Standards and Technology (NIST) reported that the rate of false recognition varies significantly between individuals depending on gender and skin color, and there have been cases where several companies have stopped offering facial recognition technology.

[0005] On the other hand, no federated learning method for personal authentication has been proposed that takes into account the difference in authentication accuracy (fairness) between individuals with different attributes such as gender and skin color.

[0006] Non-Patent Document 1 proposes a federated learning method for binary classification models that reduces the difference in classification accuracy between individuals with different attributes. Non-Patent Document 1 proposes a learning method that reduces the difference in classification accuracy between users with different attributes by adjusting weights based on an index related to fairness within the client terminal in addition to the amount of data available on the client terminal. Non-Patent Document 2 proposes a learning method for updating model parameters by weighting according to statistical risks between attributes during associative learning of a class classification model. Non-Patent Document 3 proposes a method for performing federated learning of a face recognition model. Non-Patent Document 4 proposes a method to reduce the difference in recognition accuracy between individuals with different attributes in deep learning of a face recognition model that is performed in a general environment, not in federated learning. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Yahya H. Ezzeldin, Shen Yan, Chaoyang He, Emilio Ferrara, Salman Avestimehr, “FairFed: Enabling Group Fairness in Federated Learning,” 35th Conference on Neural Information Processing Systems Workshop, 2021.<https: / / arxiv.org / abs / 2110.00857> [Non-patent document 2] Afroditi Papadaki, Natalia Martinez, Martin Bertran, “Federating for Learning Group Fair Models”, 35th Conference on Neural Information Processing Systems Workshop, 2021.<https: / / arxiv.org / abs / 2110.01999> [Non-patent document 3] Divyansh Aggarwal, Jiayu Zhou, Anil K. Jain, “FedFace: Collaborative Learning of Face Recognition Model,” International Joint Conference on Biometrics 2021.<https: / / arxiv.org / pdf / 2104.03008.pdf> [Non-patent document 4] Sixue Gong Xiaoming Liu Anil K. Jain, “Mitigating Face Recognition Bias via Group Adaptive Classifier,” Conference on Computer Vision and Pattern Recognition, 2021. <https: / / openaccess.thecvf.com / content / CVPR2021 / papers / Gong_Mitigating_Face_Recognition_Bias_via_Group_Adaptive_Classifier_CVPR_2021_paper.pdf> Summary of the Invention [Problem to be solved by the invention]

[0008] In Non-Patent Document 1, it is not possible to calculate the difference in authentication accuracy between users with different attributes within a client terminal, which results in differences in authentication accuracy between individuals with different attributes. In Non-Patent Document 2, it is not possible to calculate the similarity between different people, and it is not possible to calculate the false recognition rate, which corresponds to the statistical risk. As a result, there is a difference in authentication accuracy between individuals with different attributes. Non-Patent Document 3 does not take into consideration the attributes of users. Therefore, if there is a bias in the attributes in the data during learning, there will be a difference in authentication accuracy between individuals with different attributes. Non-Patent Document 4 does not take into consideration the specific circumstances of federated learning for personal authentication, which results in differences in authentication accuracy between individuals with different attributes.

[0009] An object of the present invention is to reduce the difference in authentication accuracy between individuals with different attributes through federated learning in a learning system. [Means for solving the problem]

[0010] A learning system according to one aspect of the present invention has a plurality of client terminals, a learning data management server, and a learning server, and updates models based on learning data. The learning server manages a common model, and the client terminals and the learning data management server manage personal data. The system generates individual models that are different for each individual from the common model and the personal data, shares the common model and the individual models with the learning server, receives the common model from the learning server, updates the common model and the individual models based on the personal data, and transmits the updated common model and the individual models to the learning server. The learning server classifies the common model and the individual models transmitted from the plurality of client terminals based on the individual models transmitted from the learning data management server, and updates the common model and the individual models according to the classification results. [Effects of the Invention]

[0011] According to one aspect of the present invention, in a learning system, it is possible to reduce the difference in authentication accuracy between individuals with different attributes through federated learning. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a learning system according to a first embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of a procedure for federated learning processing for personal authentication. [Figure 3] FIG. 10 is a diagram illustrating an example of a procedure for personal data collection processing. [Figure 4] FIG. 10 is a diagram illustrating an example of a procedure for an in-server learning process. [Figure 5] FIG. 10 is a diagram illustrating an example of a procedure for learning processing within a client terminal. [Figure 6] FIG. 10 illustrates an example of a procedure for a model update process. [Figure 7] FIG. 10 is a diagram illustrating an example of a procedure for an individual model storage process in a learning data management server. [Figure 8]FIG. 10 is a diagram illustrating an example of a procedure for an individual model storage process in a client terminal. [Figure 9] FIG. 10 is a diagram illustrating an example of a procedure for personal authentication processing. [Figure 10] FIG. 2 is a diagram illustrating a hardware configuration of a learning system. [Figure 11] FIG. 10 is a diagram illustrating a configuration of a learning system according to a second embodiment. [Figure 12] FIG. 10 is a diagram illustrating the configuration of a learning system according to a third embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of a method for classifying clients. [Figure 14] FIG. 10 is a diagram illustrating an example of aggregation of a common model. [Figure 15] FIG. 10 illustrates an example of optimization of an individual model. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment will be described with reference to the drawings. [Example]

[0014] Example 1 is a machine learning system characterized by sharing an individual model generated from personal data at a client terminal with a learning server, sharing multiple individual models generated at a learning data management server with the learning server, classifying the individual models received from each client terminal using the individual models received from the learning data management server, optimizing the common model and individual models based on the classification results, and then sharing the individual models with the client terminal and the learning data management server.

[0015] The procedure will be described below with reference to the drawings.

[0016] FIG. 1 is a diagram including an example configuration of a client terminal, a learning server, and a learning data management server. 1, reference numeral 1000 denotes a client terminal, which includes a personal data acquisition unit 1010, an in-client-terminal learning unit 1020, a data encryption unit 1030, a data decryption unit 1040, an authentication result output unit 1050, a personal data storage unit 1090, a common model storage unit 1091, and a template storage unit 1092. Each unit will be described below.

[0017] The personal data acquisition unit 1010 acquires data linked to an individual from an individual. The client terminal learning unit 1020 learns a common model and an individual model based on the personal data acquired by the personal data acquisition unit 1010. The data encryption unit 1030 encrypts the personal data and the individual model to prevent personal information from being leaked.

[0018] The data decryption unit 1040 extracts the original data by decrypting the personal data and individual model encrypted by the data encryption unit 1030. The authentication result output unit 1050 outputs the result of personal authentication using the personal data.

[0019] The personal data storage unit 1090 stores personal data and individual models. The common model storage unit 1091 stores the learned common model and utilizes it for personal authentication. The template storage unit 1092 creates registration templates to be used during personal authentication. In the first embodiment, one or more client terminals are included in the processing.

[0020] Reference numeral 1100 denotes a learning server, which includes a client terminal selection unit 1110, a client terminal classification unit 1130, a model update unit 1140, and a common model storage unit 1191. Each unit will be explained below.

[0021] The client terminal selection unit 1110 selects client terminals that will participate in the current learning round from among the client terminals that can participate in learning.

[0022] The client terminal classification unit 1130 classifies each client terminal based on the individual model received from the client terminal, the individual model received from the learning data management server, and attribute information such as gender and skin color.

[0023] The model update unit 1140 updates the weights of the common model and the individual model. The weights are updated by defining a loss function for the model and minimizing the loss function.

[0024] The common model that has been learned is stored in the common model storage unit 1191. The role of the learning server can also be played by the same terminal as the learning data management server.

[0025] Reference numeral 1200 denotes a learning data management server, which includes an intra-server learning unit 1210, a data encryption unit 1230, a data decryption unit 1240, an intra-server learning data storage unit 1290, and a common model storage unit 1291. Each unit will be described below.

[0026] The in-server learning unit 1210 updates the common model and the multiple individual models based on the learning data stored in the in-server learning data storage unit 1290 .

[0027] The data encryption unit 1230 encrypts the training data and the individual model to prevent leakage of personal information. The data decryption unit 1240 decrypts the training data and the individual model encrypted by the data encryption unit 1240 and extracts the original data.

[0028] The server learning data storage unit 1290 stores data used during server learning and the individual models corresponding to the data. The common model storage unit 1291 stores the learned common model.

[0029] The role of the learning data management server can also be performed by the same terminal as the learning server.

[0030] Next, the hardware configurations of the client terminal 1000, learning server 1100, learning data management server 1200, and shuffle server 1300 will be described with reference to FIG. In FIG. 10, 8010 is a CPU (Central Processing Unit), 8020 is a main storage device, 8030 is an auxiliary storage device, 8040 is an input device, 8050 is an output device, and 8060 is a communication device.

[0031] The CPU 8010 executes programs corresponding to the client terminal learning unit 1020 , the data encryption unit 1030 , the data decryption unit 1040 , the client terminal selection unit 1110 , the model update unit 1140 , and the parameter generation unit 1210 .

[0032] The main memory device 6020 is a device equivalent to the memory of a computer, and stores programs corresponding to the intra-client terminal learning unit 1020, data encryption unit 1030, data decryption unit 1040, client terminal selection unit 1110, model update unit 1140, and parameter generation unit 1210. Each process is realized by executing these programs on the CPU 8010.

[0033] Some or all of the programs and data may be stored in advance in the main memory device 6020, or may be introduced from a non-transitory storage medium or from an information processing device equipped with an external non-transitory storage device via a network.

[0034] The auxiliary storage device 8030 is a storage device represented by an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and corresponds to the personal data storage unit 1090, the common model storage unit 1091, the template storage unit 1092, the common model storage unit 1190, and the parameter storage unit 1290. The data stored in each unit is accumulated as data on the auxiliary storage device 8030.

[0035] The input device 8040 is used to read information in the personal data acquisition unit 1010. The input device 8040 can be configured to include devices such as a keyboard, a biometric sensor, a touch panel, a smart device, a scanner, a camera, and the like.

[0036] The output device 8050 is used to output information from the authentication result output unit 1050 through a device such as a display. The communication device 8060 is used for communication between the client terminal 1000, the learning server 1100, and the learning data management server 1200.

[0037] As described above, in the first embodiment, it is possible to build a highly accurate common model while reducing the difference in authentication accuracy between attributes.

[0038] The processing procedure of the present embodiment 1 will be described with reference to FIGS.

[0039] 2 is a diagram showing an example of the steps of federated learning processing for personal authentication that takes fairness into consideration. Each step will be explained below. Federated learning of a model used for personal authentication is performed by three parties: a client terminal 1000, a learning server 1100, and a learning data management server. First, the personal data acquisition unit 1010 in the client terminal 1000 collects personal data from the user 210 (S2010).

[0040] Personal data is data used for personal authentication, which is the subject of this embodiment, and includes, for example, physical information such as fingerprints, face, iris, and veins, and behavioral characteristics such as acceleration information, movement history, browsing history, and purchase history.

[0041] The acquisition of personal data may be performed solely for the purpose of data collection, or the acquired personal data may be stored for the purpose of authentication.

[0042] By repeatedly performing S2010, personal data of the user 210 is accumulated in the client terminal. When a client terminal is occupied by one user, personal data of one specific user is accumulated, but this is not limiting.

[0043] For example, personal data of multiple people may be stored in a client terminal shared by multiple people, or personal data stored in multiple other client terminals may be aggregated in the client terminal and used for federated learning. Details of S2010 will be explained later with reference to FIG. 3.

[0044] The learning server 1100 uses the client terminal selection unit 1110 to select client terminals that will participate in learning in each round (S2110).

[0045] Methods for selecting client terminals include selecting all client terminals, randomly selecting a statically or dynamically determined number of client terminals from among those that can participate in learning, setting some kind of evaluation value for client terminals and selecting client terminals with the highest or lowest evaluation value, selecting client terminals with an evaluation value above or below a certain threshold, and selecting client terminals after weighting them according to the evaluation value.

[0046] The learning server 1100 transmits the common model stored in the common model storage unit 1191 to each participating client terminal (S2120).

[0047] The common model is a machine learning model that inputs personal data and outputs a feature vector, and general machine learning models such as a linear regression model, a decision tree, and a neural network can be applied.

[0048] For example, when image data such as fingerprints, faces, irises, and veins are used as personal data, deep learning models such as CNNs (Convolutional Neural Networks) and Transformers can be applied.

[0049] The common model may be initialized with random weights at the start of learning, or may be pre-trained using personal data other than that stored in the client terminal and optimized with those weights.

[0050] The learning data management server 1200 receives the common model from the learning server 1100 (S2210), and updates the common model and the individual model in the server in the learning unit 1210 using the data stored in the learning data storage unit in the server (S2220).

[0051] The common model is a machine learning model as explained in S2120. The individual model in the server is a model that differs for each individual or attribute. Here, attributes refer to characteristics that individuals have, such as gender and skin color. For example, in distance learning, which extracts feature vectors from personal data and determines the distance between the feature vectors to reflect the similarity between the personal data, a representative vector of the feature vectors can be used as an individual model within the server.

[0052] Furthermore, when setting individual models for each attribute, it is also possible to use representative vectors obtained by grouping these representative vectors by attribute. Details of model updating by in-server learning will be explained later with reference to Figure 4.

[0053] The learning data management server 1200 transmits the common model and the individual model as learning results to the learning server 1100 (S2230). The client terminal 1000 receives the common model from the learning server 1100 (S2020), and updates the common model and the individual model using personal data in the client terminal learning unit 1020 (S2030).

[0054] As explained in S2120, the common model is a machine learning model. The individual model is a model that differs for each individual. If a client terminal stores personal data for only one person, that client terminal has one individual model.

[0055] For example, in distance learning, which extracts feature vectors from personal data and determines the distance between the feature vectors to reflect the similarity between the personal data, a representative vector of the feature vectors can be used as an individual model.

[0056] The representative vector is sensitive information linked to the user of the client terminal, and if this data were to be obtained by a third party, there would be a risk of identity theft or personal data leakage. Details of model updating through in-client terminal learning will be explained later with reference to Figure 5.

[0057] The client terminal 1000 transmits the common model and the individual models as the learning results to the learning server 1100 (S2040). When the common model and the individual models are transmitted directly from the client terminal 1000 to the learning server 1100, the learning server 1100 knows from which client terminal the common model and the individual model were transmitted.

[0058] In this case, if any personal information is leaked from the common model or the individual model, the client terminal corresponding to the leaked personal information can be identified.

[0059] To prevent this, when sending learning results from the client terminal 1000 to the learning server 1100, a shuffle server can be placed between them, and the shuffle server can rearrange the order of the data sent from the client terminal 1000 or assign a different identifier to it.

[0060] By doing so, it becomes impossible to grasp the correspondence between the data received from the learning server 1100 and the client terminal 1000, thereby achieving a high level of security.

[0061] The learning server 1100 receives the common model and the individual models from the multiple client terminals 1000 and the learning data management server 1200, and aggregates the common model and optimizes the individual models in the model update unit 1140 (S2140). Details of S2140 will be explained later with reference to FIG. 6.

[0062] The learning server 1100 transmits the individual model updated in S2140 to each client terminal and learning data management server (S2150). At this time, an individual model exists for each client terminal and learning data management server, and is transmitted only to the corresponding client terminal and learning data management server. If the individual model is transmitted to a different client terminal or learning data management server, there is a risk that the individual model could be used for impersonation or personal data inference.

[0063] The learning data management server 1200 stores the individual model received from the learning server 1100 in the server learning data storage unit 1290 (S2250).

[0064] Individual model storage will be described later with reference to FIG.

[0065] The client terminal 1000 stores the individual model received from the learning server 1100 (S2050) in the personal data storage unit 1090 (S2060). Individual model storage will be described later with reference to FIG.

[0066] As a result, it is possible to reduce the difference in authentication accuracy between client terminals while enabling federated learning of a common model on the learning server.

[0067] Next, the processing procedure of personal data collection S2010 will be described with reference to FIG. First, personal data is acquired from the user 210 (S3010). As explained in S, personal data includes physical information such as fingerprints, face, iris, and veins, as well as behavioral characteristics such as acceleration information, movement history, browsing history, and purchase history.

[0068] Next, the personal data acquired in S3010 is selected (S3020). In S3020, data to be used in federated learning is selected from the personal data acquired in S3010.

[0069] Although it is possible to use all acquired personal data for learning without performing S3020, by performing data selection and reducing the amount of personal data, it is possible to reduce the amount of calculation required for learning and the storage capacity of the personal data storage unit 1090.

[0070] In addition, excluding personal data that contributes little to learning or that has a negative impact is expected to lead to improved accuracy of the final common model.

[0071] Personal data is selected by calculating an evaluation value for each piece of personal data, and selecting personal data whose evaluation value is above or below a certain threshold, or personal data whose evaluation value is high or low, or personal data whose evaluation value is within a range.

[0072] It should be noted that instead of calculating an evaluation value for each piece of personal data, it is also possible to calculate the distance between pieces of personal data or an evaluation value for a set of personal data, and repeatedly exclude personal data based on the evaluation value to select personal data.

[0073] For example, when facial images are used as personal data, the facial images that appear in a video are very similar in every frame, and learning a large number of similar facial images is inefficient and does not improve accuracy.

[0074] Furthermore, facial images in which the subject is facing sideways, the face is obscured by a hand, or the face is outside the image have too large an error to be suitable for use as training data.

[0075] To exclude such personal data, for example, feature extraction is performed for each facial image to generate a feature vector, and if the distance between the feature vectors is sufficiently small, i.e., if the facial images are sufficiently similar, one of the feature vectors can be excluded from the learning data, thereby excluding similar personal data.

[0076] In addition, by calculating a quality value from the facial image, including the orientation of the face, whether it is occluded, whether it is protruding, etc., and excluding facial images with a quality value above or below a certain level, it is possible to exclude inappropriate personal data.

[0077] The client terminal 1000 encrypts the personal data selected in S3020 in the data encryption unit 1030 (S3030).

[0078] Encryption is performed to prevent personal data leakage due to unauthorized access by third parties to client terminals, malware infection, etc.

[0079] It should be noted that if the client terminal is trustworthy or if the security of the client terminal is guaranteed by means other than data encryption, data encryption may not be required.

[0080] Encryption generally requires a private key, which can be managed in a storage device within the client device, a secure area within the client device such as a Trusted Execution Environment (TEE), or an external medium such as a hardware token.

[0081] In addition, biometric encryption technology such as Fuzzy Extractor can be used to dynamically generate private keys from personal data such as biometric information.

[0082] In this case, if a private key can be dynamically generated from the personal data to be learned in this embodiment, there is no need to prepare a private key for encryption, and it is possible to dynamically generate a private key from the personal data obtained for personal authentication or personal data acquisition and use it to perform encryption.

[0083] The personal data encrypted in S3030 is stored in the personal data storage unit 1090 (S3040).

[0084] As described above, personal data can be acquired from the user 210, encrypted in a secure format, and then stored in the personal data storage unit 1090.

[0085] Next, the processing procedure of the in-server learning S2220 will be described with reference to FIG. First, the learning data stored in the learning data storage unit 1290 of the learning data management server 1200 is decrypted (S4010). Decrypting the learning data requires the private key used when encrypting it.

[0086] This private key can be managed in a storage device within the learning server, in a secure area within the learning server such as a TEE (Trusted Execution Environment), or in an external medium such as a hardware token.

[0087] The learning data is data used for personal authentication, which is the subject of this embodiment, and includes, for example, physical information such as fingerprints, face, iris, and veins, and behavioral characteristics such as acceleration information, movement history, browsing history, and purchase history.

[0088] It should be noted that these data do not necessarily have to be collected from individuals, but may be artificially generated data or data from multiple people synthesized through statistical calculations.

[0089] Next, learning data to be used for in-server learning is selected from the data stored in the in-server learning data storage unit 1290 (S4020).

[0090] The learning data can be selected by randomly extracting a predetermined number of learning data from the learning data stored in the server.

[0091] Alternatively, this can be done by calculating an evaluation value for each piece of training data and selecting training data whose evaluation value is above or below a certain threshold, or whose evaluation value is higher or lower than the threshold, or whose evaluation value is within a range.

[0092] Alternatively, the distance between learning data or an evaluation value for a set of learning data may be calculated, and the learning data may be repeatedly excluded based on the evaluation value to perform selection.

[0093] If the training data is labeled, it is possible to select the training data so that the difference in the proportion of each label is below a certain threshold, or to select the training data in different proportions for each label.

[0094] Furthermore, from the perspective of learning, if the distance between the feature vector calculated from personal data and the representative vector is too close, it will have little contribution to learning and will be less effective, and if it is too far, it will be difficult to shorten the distance through learning.

[0095] Therefore, by calculating the distance between the feature vector and the representative vector and leaving only the personal data where the distance is within a certain range, it is expected that efficient and highly accurate learning can be performed.

[0096] The process branches depending on whether or not the intra-server individual model corresponding to the training data selected in S4020 is stored in the intra-server training data storage unit 1290 (S4030).

[0097] If there is an intra-server individual model, the encrypted intra-server individual model is read out from the intra-server learning data storage unit 1290 and decrypted (S4050). If there is no intra-server individual model, an intra-server individual model is generated (S4040).

[0098] The generation of an individual model within a server can be performed by inputting training data into a common model to generate a feature vector, and then performing statistical calculations such as the mean and median on the resulting feature vector.

[0099] The common model used in this case may be a pre-trained model stored at the start of federated learning, or the latest common model received by the learning data management server 1200 in S2210. Note that the individual models in the server may be initialized with random weights without using data.

[0100] The common model and the individual model in the server are updated using the training data selected in S4020, the individual model in the server acquired in S4040 or S4050, and the common model acquired in S2210 (S2220).

[0101] Training of the common model and the individual models within the server is performed by defining a loss function for the model and searching for the weights that minimize that function.

[0102] The loss function models the degree of ideal relationship between the common model and the individual model in the server, and is defined as the loss relative to the ideal relationship.

[0103] For example, when using a representative vector of feature vectors generated from data as an individual model within a server, the ideal relationship would be for the feature vector generated by inputting all data into a common model to be equal to the representative vector. For this reason, the loss function used may be the sum of the squares of the distances between the feature vector and the representative vector, or the ratio of a function based on the dot product of the feature vector and the representative vector.

[0104] By minimizing such a loss function, the feature vector approaches the representative vector, forming an ideal relationship.

[0105] Furthermore, when deep learning models such as CNN (Convolutional Neural Networks) and Transformer are used as common models, optimization methods such as stochastic gradient descent are applied to minimize the loss function.

[0106] The weights of the common model and the individual model are searched for. Note that when training the model, noise may be added to the personal data before training.

[0107] It is known that the personal data used for learning can be inferred from the learning results at the client terminal 1000, but adding noise makes it difficult to infer the personal data. In this way, the common model and the individual model in the server are learned.

[0108] Next, the processing procedure of the client terminal learning S2030 will be described with reference to FIG. First, the client terminal 1000 decrypts the personal data stored in the personal data storage unit 1090 using the data decryption unit 1040 (S5010). To decrypt personal data, the private key used when encrypting it is required. This private key is obtained or generated using the same procedure as in S5010, and then decryption is performed.

[0109] Next, learning data to be used for learning within the client terminal is selected (S5020) from the personal data stored in the personal data storage unit 1090. The learning data can be selected by randomly extracting a predetermined number of pieces of personal data from the personal data storage unit 1090.

[0110] In addition, this can also be done by calculating an evaluation value for each personal data and selecting personal data whose evaluation value is above or below a certain threshold, or personal data whose evaluation value is above or below that threshold, or personal data whose evaluation value is within a range determined by the threshold.

[0111] Furthermore, it is also possible to select personal data by calculating the distance between personal data or an evaluation value for a set of personal data, and repeatedly excluding personal data based on the evaluation value.

[0112] Furthermore, from the perspective of learning, if the distance between the feature vector calculated from personal data and the representative vector is too close, it will have little contribution to learning and will be less effective, and if it is too far, it will be difficult to shorten the distance through learning.

[0113] Therefore, by calculating the distance between the feature vector and the representative vector and leaving only the personal data where the distance is within a certain range, it is expected that efficient and highly accurate learning can be performed.

[0114] The process branches (S5030) depending on whether or not an individual model is stored in the personal data storage unit 1090. If an individual model is stored, the encrypted individual model is read from the personal data storage unit 1090 and decrypted (S5050).

[0115] If there is no individual model, an individual model is generated (S5040). The individual model can be generated by inputting personal data into a common model to generate a feature vector, and then performing statistical calculations such as the mean and median on the obtained feature vector.

[0116] The common model used at this time may be a pre-trained model stored at the start of federated learning and used, or the latest common model received by the client terminal 1000 in S2020. Note that the individual models may be initialized with random weights without using personal data.

[0117] The common model and the individual models are updated using the training data selected in S5010, the individual models obtained in S5040 or S5050, and the common model received in S2020 (S5060).

[0118] The model is updated by defining a loss function for the model and searching for the weights that minimize that function. The loss function models the degree of ideal relationship between personal data and its corresponding common model and individual model, and is defined as the loss relative to the ideal relationship.

[0119] For example, when a representative vector of feature vectors generated from personal data is used as an individual model, the ideal relationship is for the feature vector generated by inputting all personal data into a common model to be equal to the representative vector.

[0120] For this reason, the sum of the squares of the distances between the feature vectors and the representative vectors is used as the loss function, and by minimizing this, the feature vectors approach the representative vectors, forming an ideal relationship.

[0121] Furthermore, when a deep learning model such as a CNN (Convolutional Neural Network) or a Transformer is used as the common model, an optimization method such as stochastic gradient descent is applied to search for weights for the common model and individual models that minimize the loss function.

[0122] Note that, during model training, noise may be added to the personal data before training. It is known that the personal data used for training can be inferred from the training results of the client terminal 1000, but adding noise in this way makes it difficult to infer the personal data.

[0123] In this way, the common model and the individual model are learned.

[0124] Next, the processing procedure of model update S2140 will be described with reference to FIG. First, it is verified whether the client terminal 1000 that has transmitted the learning result is the correct client terminal selected in S2110 (S6010).

[0125] S6010 records the terminal number, IP address, etc. as information for identifying the client terminal selected in S2110, and compares this with the client terminal that transmitted the learning results in S2130.

[0126] If the client terminal is correct, the process proceeds to S6010 and subsequent steps, and if the client terminal is incorrect, the client terminal is excluded from the model update.

[0127] This is done to prevent what is known as a Model Poisoning Attack, in which a client device sends fraudulent learning results, reducing the accuracy of the model.

[0128] Next, the client terminals are classified into classes using the individual model received from the client terminal 1000 and the individual model and attribute information transmitted from the learning data management server 1200 (S6020). Here, the classes are determined by the attributes of the individual model transmitted from the learning data management server. For example, there are two possible gender classes: male and female.

[0129] It is also possible to determine classes based on multiple attributes. For example, based on gender and eye color, four classes can be considered: male with black eyes, male with eyes other than black eyes, female with black eyes, and female with eyes other than black eyes.

[0130] Classification is performed based on evaluation values ​​that represent the relationship between the individual models sent from the learning data management server or the individual models for each attribute (attribute-specific models) obtained by averaging these individual models and the individual models of the client terminal.

[0131] As an evaluation value, the Euclidean distance, Mahalanobis distance, or cosine similarity between two individual models can be used.

[0132] Classification methods include classifying into classes of in-server individual models or attribute-specific models with high or low evaluation values, or classifying according to the proportion of in-server individual models or attribute-specific models with evaluation values ​​above or below a certain threshold.

[0133] It is also possible to use individual models sent from the learning data management server to learn classification models such as decision trees and neural networks, and then use these classification models to classify individual models sent from client terminals.

[0134] It should be noted that the individual models do not necessarily have to be classified into a single class, but can also be classified probabilistically.

[0135] Next, the common models received from the multiple client terminals 1000 and the learning data management server are aggregated (S6030). In this process, the weights of a single common model are calculated from the weights of multiple common models that have been separately trained by each client terminal and the learning data management server.

[0136] The calculation method can be, for example, to simply take the average of the weights, or to take the average after weighting based on the classification results performed by S6030.

[0137] Weighting methods based on the classification results include first calculating the average within each classified class and then calculating the average across all classes, or weighting based on the proportion of the class that an individual model occupies or its inverse.

[0138] The individual models received from the multiple client terminals 1000 are optimized (S6040). For example, if a representative vector for feature vectors generated from personal data is used as the individual model, the representative vectors of each client terminal and the learning data management server are aggregated in the learning server.

[0139] This representative vector is the vector located at the center when the personal data of each individual is input into a common model to generate a feature vector. Therefore, if the distance between the representative vectors of two individuals is small, it will be difficult to distinguish between the two individuals using their feature vectors, resulting in low discrimination ability.

[0140] Therefore, it is ideal for the distance between all representative vectors to be at least a certain value. This property can be expressed in the form of a loss function, and the representative vectors can be optimized by finding the representative vectors that minimize the loss function.

[0141] For example, a loss function can be defined by defining a margin value between representative vectors, and when the distance between the representative vectors is smaller than the margin, calculating the squares of the margin and the distance between the representative vectors and taking the sum of these.

[0142] When this loss function becomes 0, it becomes an ideal situation where the distance between all representative vectors is greater than or equal to the margin.

[0143] On the other hand, if the majority of client terminals participating in a round for all attributes have the same attribute, only the representative vector for that attribute will spread, and there is a possibility that there will be a difference in authentication accuracy between that attribute and other attributes.

[0144] To prevent this, when optimizing the representative vector, a loss function based on the classes classified by S6030 is added.

[0145] Examples of loss functions that can be added include functions of the difference between attributes of the maximum, average, and minimum distances between representative vectors, functions that represent the difference in distribution of representative vectors between attributes, and functions of the difference in variance of representative vectors between attributes.

[0146] Finally, the common model aggregated in S6030 is stored in the common model storage unit 1190 (S6050). As described above, the common model and individual models are updated, and the learned common model is stored in the common model storage unit 1191.

[0147] Next, the processing procedure of the individual model storage S2250 in the learning data management server 1200 will be described with reference to FIG.

[0148] First, the individual model sent by the learning server in S2150 is encrypted (S7010). The encryption is performed using a secret key prepared by some means.

[0149] The individual model encrypted in S7010 is stored in the server training data storage unit 1290 (S7020). As a result, the individual model sent from the training server can be encrypted and safely stored in the server training data storage unit 1290.

[0150] Next, the processing procedure of storing the individual model S2060 in the client terminal 1000 will be described with reference to FIG. First, the individual model sent by the server in S2150 is encrypted (S8010). The encryption is performed using a secret key prepared by some means, similar to the encryption of personal data in S3010.

[0151] The individual model encrypted in S8010 is stored in the personal data storage unit 1090 (S8020). As described above, the individual model sent from the server can be encrypted and then safely stored in the personal data storage unit 1090.

[0152] Next, a processing procedure for applying the common model learned in the procedure of FIG. 2 to personal authentication will be described with reference to FIG. The processing procedure consists of three steps: model setting, personal registration, and personal authentication. First, in model setting, the learning server 1100 obtains a common model from the common model storage unit 1191 and transmits it to the client terminal (S9110).

[0153] The client terminal 1000 receives the common model and stores it in the common model storage unit 1091 (S9010). S9010 is not performed every time personal authentication is performed, but is performed by a developer or an administrator when starting operation of a new authentication system or when upgrading the authentication system.

[0154] Next, in personal registration, personal data is acquired from the user 210 (S9020), and the personal data is input into the common model stored in the common model storage unit 1091 to generate a feature vector (S9030).

[0155] A registration template is generated from the obtained feature vector and stored in the template storage unit 1092 (S9040).

[0156] A template is information for registration generated from personal data, and a feature vector may be used as a template as is.

[0157] It should be noted that some template protection technology may be applied when generating the template to prevent the original personal data from being leaked from the template.

[0158] Finally, in personal authentication, personal data is acquired from the user 210 (S9050), and this is input into the common model stored in the common model storage unit 1091 to generate a feature vector (S9060).

[0159] The obtained feature vector is compared with the template stored in the template storage unit 1092, and similarity, dissimilarity, etc. are calculated as the comparison result (S9070).

[0160] For example, when authentication is performed based on the distance between feature vectors, the Hamming distance or Euclidean distance between the feature vectors is calculated and used as the dissimilarity.

[0161] Threshold processing or the like is applied to such matching results to determine whether authentication is successful or unsuccessful, and this result is output (S9080). Note that the matching results are not limited to continuous values ​​such as similarity and dissimilarity, and if a template is generated using template protection technology, the matching results may be a binary value of either authentication success or authentication failure. In this case, threshold processing is not necessary, and the matching results are directly output as the authentication result. As described above, personal authentication can be performed using the common model trained based on the processing in Figure 2.

[0162] The classification of clients performed in S6020 of FIG. 6 will be described with reference to FIG. Figure 13 shows an example of a method for classifying clients, assuming that one point in a two-dimensional feature space represents an individual model. In the example shown in FIG. 13, the client is classified into the attributes of the individual model 100 of the learning data management server that is closest to the individual model 110 of the client.

[0163] In this case, the client corresponding to the individual model of the client connected by the shortest distance r1 to the individual model of the learning data management server of attribute A1 is classified into attribute A1.

[0164] Similarly, a client corresponding to an individual model connected to the individual model of the learning data management server with attribute A2 by the shortest distance r2 is classified into attribute A2. As described above, clients can be classified using the individual model of the learning data management server and the individual model of the client.

[0165] The aggregation of common models (S6030) according to the presence or absence of client classification, which is performed in S6020 of FIG. 6, will be described with reference to FIG. FIG. 14A shows an example of aggregation of a common model when clients are not classified, and FIG. 14B shows an example of aggregation of a common model when clients are classified. In FIG. 14, the aggregation of common models of a total of four clients, three clients with attribute A1 and one client with attribute A2, is performed by averaging.

[0166] If the clients are not classified, the averaged common model is heavily influenced by the common model trained on the clients with attribute A1. In the example of Figure 14, the aggregated common model is biased, with the contribution rate of the common model trained on the data of attribute A1 being 75% and the contribution rate of the common model trained on the data of attribute A2 being 25%.

[0167] On the other hand, when classifying clients, it is possible to perform weighted averaging that takes into account the attributes of the training data of each common model. In the example of Figure 14, by assigning a weight of 50 / 3% to the common model trained with data for attribute A1 and a weight of 50% to the common model trained with data for attribute A2 and averaging them, the contribution rate of each attribute A1 and A2 to the aggregated common model can be made 50%.

[0168] Even if clients are not classified, it is possible to change the weight of each client for the common model to any value, but if the attributes are unknown, it is not possible to assign weights that take into account the contribution rate to each common model after aggregation.

[0169] As described above, by classifying clients, it becomes possible to aggregate common models taking into account their attributes.

[0170] The optimization of the individual model performed in S6040 of FIG. 6 will be described with reference to FIG. FIG. 15 shows an example of optimization of an individual model, assuming that one point on a unit circle in a two-dimensional feature space represents the individual model. In the example of Fig. 15, optimization can be performed by moving the points of the individual models so that the angles b1, b2, ... between the two individual models become larger. In this way, optimization of the individual models can be performed. [Example]

[0171] In the second embodiment, the processing performed by the learning server and the processing performed by the learning data management server in the first embodiment are performed by a single server.

[0172] In Example 1, by separating the learning server and the learning data management server, machine learning using this machine learning method could be realized even when the federated learning operation organization and the data providing organization for fairness consideration were different. As a result, the privacy of the learning data management server could be protected from the management entity of the learning server.

[0173] In the second embodiment, by integrating the learning server and the learning data management server, the overhead for communication between the two servers can be reduced and the server's computational resources can be utilized more effectively, although the advantages of integrating the learning server and the learning data management server are lost.

[0174] Here, only the differences from FIG. 2 will be described with reference to FIG. In Figure 11, a single learning server also serves as the learning data management server. As a result, there is no need to send and receive models between the learning server and the learning data management server, so the processes corresponding to S2210, S2230, and S2240 are not performed. The other processes are the same as those in Figure 2.

[0175] In Figure 11, after client terminal selection in S2110 and common model transmission in S2120, in-server learning in S2220 is performed, but as long as the weight of the common model used in in-server learning is equal to the weight of the common model transmitted to each client terminal, in-server learning S2220 may be performed at any time before or after client terminal selection S2110 and common model transmission S2120.

[0176] It is also possible to perform S2220 in parallel with S2110 and S2120. By performing parallel processing, it is possible to reduce the time required for one round (the series of processes from client terminal selection S2110 to individual model transmission S2150) compared to sequential processing. [Example]

[0177] In Example 3, the transmission and reception of the model in Example 1 is performed via a shuffle server. By using the shuffle server, it becomes difficult for the server to link the client terminal with the individual model and the shared model, thereby increasing the security of federated learning.

[0178] Here, only the differences from FIG. 2 will be described with reference to FIG. 1 and FIG.

[0179] Fig. 1 is a diagram including the configuration of a shuffle server. In Fig. 1, 1300 is a shuffle server, which is configured to include a parameter generation unit 1310, a shuffle processing unit 1320, and a parameter storage unit 1390. Each unit will be explained below.

[0180] The parameter generating unit 1310 generates parameters used when rearranging the order of the received individual models and common models and assigning identifiers to them (S2310).

[0181] The parameters generated in S2310 can be a correspondence table between the identifier of the client terminal when the learning result is received from the client terminal and the identifier when the learning result is sent to the learning server.

[0182] The shuffle processing unit 1320 performs shuffle processing (S2330, S2360) which will be described later. The parameter storage unit 1390 stores the parameters generated by the parameter generation unit 1320.

[0183] In the third embodiment, first, the shuffle server 1300 receives the common model and the individual model transmitted from the client terminal and the learning data management server in S2040 and S2230 (S2320).

[0184] At this time, the client terminal or the learning data management server may encrypt the transmitted data in a way that makes it difficult for the shuffle server to decrypt it. For example, by encrypting the data using the public key of the learning server, it is possible to transmit the data to the learning server while keeping the common model and individual models secret from the shuffle server.

[0185] The shuffle server 1300 acquires the parameters from the parameter storage unit 1390 and performs shuffle processing on the received common model and individual model (S2330).

[0186] Here, the shuffling process is a process for concealing the correspondence between the client terminal and the common model and individual model, and can involve randomly shuffling the identifiers that each client terminal has in advance, or assigning new identifiers to each client terminal, etc. After the shuffling process, the common model and individual models are sent to the training server (S2340).

[0187] Next, the shuffle server 1300 receives the updated individual model sent from the learning server 1100 (S2350). At this time, the learning server may encrypt the sent data in a form that makes it difficult for the shuffle server to decrypt it.

[0188] For example, encryption can be performed using the public key of each client terminal, or an encrypted common key can be received at the same time as receiving the common model and individual model, and encryption can be performed using that common key.

[0189] The received individual model is subjected to the reverse process of S2330, and the correspondence between the individual model and the client terminal is restored to the same state as when it was received in S2320 (S2360).The individual model is then sent to each corresponding client terminal and the learning data management server (S2370).

[0190] As described above, in the third embodiment, it is possible to build a highly accurate common model while increasing the security of federated learning via the shuffle server and reducing the difference in authentication accuracy between attributes.

[0191] In the above embodiment, the learning server classifies client terminals using individual models sent from the learning data management server and individual models sent from the client terminals, and aggregates common models and optimizes individual models according to the classification results.

[0192] According to the above embodiment, in federated learning for personal authentication, it is possible to reduce the difference in authentication accuracy that occurs between individuals with different attributes.

[0193] The above embodiment is based on the assumption that the user of the client terminal is an individual, but it is also applicable to cases where the user of the client terminal is an organization such as a company. [Explanation of symbols]

[0194] 1000 client terminals 1010 Personal Data Acquisition Department 1020 Client terminal internal learning unit 1030 Data Encryption Unit 1040 Data Decoding Unit 1050 Authentication result output section 1090 Personal Data Storage Unit 1091 Common Model Storage 1092 Template storage section 1100 Learning Server 1110 Client terminal selection unit 1130 Client terminal classification unit 1140 Model Update Department 1191 Common Model Storage 1200 Learning Data Management Server 1210 Server Learning Unit 1230 Data Encryption Unit 1240 Data Decoding Unit 1290 Server learning data storage unit 1291 Common Model Storage 1300 on the shuffle server 1310 Parameter Generation Unit 1320 Shuffle Processing Unit 1390 Parameter Storage

Claims

1. A learning system having a plurality of client terminals, a learning data management server, and a learning server, and updating a model based on learning data, The learning server Manage the common model, The client terminal and the learning data management server Manage your personal data, generating an individual model that differs for each individual from the common model and the personal data; sharing the common model and the individual model with the learning server; receiving the common model from the learning server, updating the common model and the individual models based on the personal data, and transmitting the updated common model and the individual models to the learning server; The learning server A learning system characterized by classifying the common model and the individual models transmitted from a plurality of the client terminals based on the individual models transmitted from the learning data management server, and updating the common model and the individual models according to the classification results.

2. The individual model is assigned attributes, The learning server The learning system described in claim 1, characterized in that the common model and the individual models transmitted from the multiple client terminals are classified by the attributes based on the attributes assigned to the individual models transmitted from the learning data management server.

3. The learning server classifying the common models transmitted from the plurality of client terminals based on the individual models transmitted from the learning data management server; The learning system according to claim 1, wherein the common model is updated by taking a weighted average for each cluster of the classification results.

4. The learning server classifying the individual models transmitted from the plurality of client terminals based on the individual models transmitted from the learning data management server; 2. The learning system according to claim 1, wherein the individual models are updated using gradients obtained by differentiating a function calculated for each cluster of the classification results with a parameter of the model. Stem.

5. The learning server updating the common model and the individual models by aggregating the common model according to the classification results and optimizing the individual models; 2. The learning system according to claim 1, further comprising: transmitting the optimized individual model to the client terminal.

6. further comprising a shuffle server; The client terminal and the learning data management server receiving the common model from the learning server; updating the common model and the individual model based on the personal data; Sending the updated common model and the individual models to the shuffle server; The shuffle server randomly rearranging the order of or assigning identifiers to the received common model and individual models, and transmitting the common model and the individual models to the learning server; The learning server The learning system described in claim 1, characterized in that the common model and the individual models transmitted from the plurality of client terminals are classified based on the common model and the individual models transmitted from the shuffle server, and the common model and the individual models are updated according to the classification results.

7. The client terminal and the learning data management server randomly generating a one-time common key, encrypting the common model and the individual model with the public key of the learning server, and transmitting the encrypted common model and the individual model to the shuffle server; The shuffle server performing the random reordering or the assignment of the identifiers to the received encrypted common model and the individual models, and transmitting the common model and the individual models to the learning server; The learning server The learning system described in claim 6, characterized in that the received common model and the individual models are decrypted using a private key of the learning server, the common model and the individual models transmitted from the plurality of client terminals are classified based on the individual models transmitted from the learning data management server, and the common model and the individual models are updated according to the classification results.

8. A learning system having a plurality of client terminals and a learning server, and updating a model based on learning data, The learning server Manage common models and personal data, The client terminal Manage said personal data; The client terminal and the learning server generating an individual model that differs for each individual from the common model and the personal data; The common model and the individual model are shared between the learning server and the client terminal; The client terminal receiving the common model from the learning server, updating the common model and the individual models based on the personal data, and transmitting the updated common model and the individual models to the learning server; The learning server creating the individual model from the personal data and the common model, and updating the common model and the individual model based on the personal data; A learning system characterized by classifying the common model and the individual models sent from multiple client terminals based on the individual model corresponding to the personal data, and updating the common model and the individual models according to the classification results.

9. The individual model is assigned attributes, The learning server The learning system described in claim 8, characterized in that the common model and the individual models transmitted from the multiple client terminals are classified by the attributes based on the attributes assigned to the individual models.

10. The learning server classifying the common model transmitted from the plurality of client terminals based on the individual model; The learning system according to claim 8, wherein the common model is updated by taking a weighted average for each cluster of the classification results.

11. The learning server classifying the individual models transmitted from the plurality of client terminals based on the individual models; The learning system according to claim 8, wherein the individual models are updated using gradients obtained by differentiating a function calculated for each cluster of the classification results with respect to model parameters.

12. The learning server updating the common model and the individual models by aggregating the common model according to the classification results and optimizing the individual models; 9. The learning system according to claim 8, wherein the optimized individual model is transmitted to the client terminal.

13. A learning method for updating a model based on learning data using a learning system having a plurality of client terminals and a learning server, comprising: The learning server manages the common model and personal data; Manage the personal data at the client terminal; The client terminal and the learning server generate an individual model that differs for each individual from the common model and the personal data; The common model and the individual model are shared between the learning server and the client terminal; receiving, at the client terminal, the common model from the learning server, updating the common model and the individual model based on the personal data, and transmitting the updated common model and the individual model to the learning server; A learning method characterized in that the learning server creates the individual model from the personal data and the common model, updates the common model and the individual model based on the personal data, classifies the common model and the individual models sent from multiple client terminals based on the individual model corresponding to the personal data, and updates the common model and the individual models according to the classification results.

14. The individual model is assigned attributes, The learning method according to claim 13, characterized in that the learning server classifies the common model and the individual models transmitted from the plurality of client terminals by the attributes based on the attributes assigned to the individual models.

15. The attributes assigned to the individual models are:

15. The method of claim 14, wherein the characteristic refers to a characteristic possessed by the individual, including gender or skin color.

Images