Learning method and learning system

Abstract

To achieve learning taking personal data on other persons into consideration in associative learning while maintaining safety in an environment in which only personal data on one person is usable on each client terminal.SOLUTION: A learning system S collects individual data for each individual, inputs the individual data for each individual into a common model, and generates individual models for each individual using the representative value of the output features for each individual. Then, the learning system S optimizes the individual models for each individual to minimize a loss function whose value decreases as the distance between the individual models for each individual increases, and generates virtual individual models for each individual based upon the relation between the optimized individual models for each individual and the individual models for each individual before the optimization. The learning system S further updates the common model and the individual models for each individual based upon the relation among the common model, the individual models for each individual, and the virtual individual models for each individual.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a machine learning system and a machine learning method. [Background technology]

[0002] Biometric authentication technology that performs personal authentication based on images of faces, fingerprints, irises, etc. has become widespread. In addition, with the recent advances in machine learning technology, personal authentication has come to be performed using machine learning models.

[0003] Training a machine learning model for personal authentication requires a large amount of personal data. However, in recent years, with the development of privacy protection systems, it has become more difficult to collect personal data, including biometric information. To solve this problem, a technology called federated learning has been attracting attention. This technology performs training on the client's own device and shares only the machine learning model after training, allowing the model to be updated without collecting training data.

[0004] However, in federated learning, when the client terminal is a personal terminal that stores data for only one person, the performance of the machine learning model for personal authentication after training is lower than that of general machine learning that is performed after aggregating the training data. This is because the training on the client terminal cannot take into account the output obtained from the data of others. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Divyansh Aggarwal, Jiayu Zhou, Anil K. Jain, “FedFace: Collaborative Learning of Face Recognition Model,” [online], International Joint Conference on Biometrics 2021., [Retrieved June 16, 2023], Internet<https: / / arxiv.org / pdf / 2104.03008.pdf> [Non-Patent Document 2] Qiang Meng, Feng Zhou, Hainan Ren, Tianshu Feng, Guochao Liu, Yuanqing Lin, “IMPROVING FEDERATED LEARNING FACE RECOGNITION VIAPRIVACY-AGNOSTIC CLUSTERS,” [online], 10thInternational Conference on Learning Representations 2022., [Retrieved June 16, 2023], Internet<https: / / openreview.net / pdf?id=7l1IjZVddDW> [Non-Patent Document 3] Binghui Chen, Weihong Deng, Haifeng Shen, “Virtual Class Enhanced Discriminative Embedding Learning,” 32ndConference on Neural Information Processing Systems 2018. [Retrieved June 16, 2023], Internet<https: / / papers.nips.cc / paper / 2018 / file / d79aac075930c83c2f1e369a511148fe-Paper.pdf> Summary of the Invention [Problem to be solved by the invention]

[0006] Non-Patent Document 1 proposes a method of federated learning of a face recognition model. Since the target situation is a situation where only one user's data is available on each client terminal, learning in the client terminal is performed in a form that minimizes a loss function based on the distance between a feature vector obtained from its own data and a feature vector that represents itself. Therefore, the learning in the client terminal does not take into account the feature vectors and data of other users, and the performance of the model after learning is lower than when general machine learning is performed by aggregating learning data.

[0007] Non-Patent Document 2 proposes a method of averaging the feature vectors of multiple individuals and sharing the feature vector with added noise between client terminals in order to take into account the feature vectors of others in learning at the client terminal. As a result, a loss function using the feature vectors of others can be used in learning at the client terminal. However, when applied to learning at a personal terminal, it is necessary to protect the average feature vector of one individual, and if sufficient noise is added for concealment, accuracy is significantly reduced. Therefore, it can be said that it is a method in which it is difficult to balance safety and accuracy.

[0008] Non-Patent Document 3 is a paper on a general machine learning method that is not federated learning. Non-Patent Document 3 proposes a method for improving the identifiability of individuals by creating a feature vector corresponding to a class that does not correspond to any individual and treating it as a class for other individuals. When this method is applied to federated learning, it is possible to make the distance between the feature vectors of individuals closer. However, this feature vector is generated from the personal data of one person, and does not take into account the feature vectors of other people who actually exist when applying federated learning.

[0009] Given the above issues, there is a demand for technology that can take into account the personal data of others during federated learning while maintaining security in an environment where each client terminal can only use the personal data of one person. [Means for solving the problem]

[0010] As one aspect for solving the above problem, there is provided a learning method executed by a learning system that learns a common model that inputs personal data related to an individual and outputs features, the learning system managing the common model, an individual model for each of the individuals, and a virtual individual model for each of the individuals, the learning system collecting the personal data for each of the individuals, inputting the personal data for each of the individuals into the common model and generating a representative value of the features for each of the individuals output as the individual model for each of the individuals, optimizing the individual model for each of the individuals so as to minimize a value of a loss function that becomes smaller as the distance between the individual models for each of the individuals increases, generating the virtual individual model for each of the individuals based on the relationship between the optimized individual model for each of the individuals and the individual model for each of the individuals before optimization, and updating the common model and the individual model for each of the individuals based on the relationship between the common model, the individual model for each of the individuals, and the virtual individual model for each of the individuals. Effect of the Invention

[0011] According to one embodiment of the present invention, it is possible to improve the performance of a machine learning model for personal authentication without sharing individual models between clients. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a learning system including a client terminal and a learning server according to an embodiment. [Diagram 2] 11 is a flowchart showing a federated learning process for personal authentication according to the embodiment. [Diagram 3] 10 is a flowchart showing a personal data collection process according to the embodiment. [Figure 4] 11 is a flowchart illustrating an internal learning process of a client terminal according to the embodiment. [Diagram 5] 11 is a flowchart illustrating a model update process according to the embodiment. [Figure 6]13 is a flowchart showing an individual model storage process in the client terminal according to the embodiment. [Figure 7] 13 is a flowchart showing a virtual individual model generation process in a client terminal according to the embodiment. [Figure 8] 13 is a flowchart showing a virtual individual model storage process in a client terminal according to the embodiment. [Figure 9] 4 is a flowchart showing a personal authentication process according to the embodiment. [Figure 10] FIG. 2 is a diagram illustrating a hardware configuration of a computer that realizes a client terminal and a learning server according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. This embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention. Not all of the elements and combinations thereof described in this embodiment are necessarily essential to the solution of the invention.

[0014] In the following description, the same components are generally designated by the same reference numerals, and repeated description will be omitted. The number of each component in the following description is not limited unless otherwise noted.

[0015] In the following explanation, the processing performed by a program may be described. A computer performs processing defined by a program using a processor (e.g., a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit)) while using memory, storage, communication ports, etc. Therefore, the processor may be the entity that performs processing by executing a program. The processor executes a program to realize a functional unit that performs processing.

[0016] In the machine learning system according to the following embodiment, an individual model generated from personal data in a client terminal is shared by a learning server, and the common model and the individual model are optimized, and then the individual model is shared by the client terminal. In the machine learning system, a virtual individual model is created in the client terminal based on the individual models before and after optimization.

[0017] (Configuration of machine learning system S) FIG. 1 is a diagram showing an example of the configuration of a machine learning system S including a client terminal 1000 and a learning server 1100 according to an embodiment.

[0018] As shown in FIG. 1, the machine learning system S includes an individual client terminal 1000 and a learning server 1100.

[0019] The client terminal 1000 includes a personal data acquisition unit 1010, an in-client terminal learning unit 1020, a data encryption unit 1030, a data decryption unit 1040, a personal authentication / result output unit 1050, and a virtual individual model generation unit 1060. The client terminal 1000 also includes a personal data storage unit 1090, a common model storage unit 1091, and a template storage unit 1092.

[0020] The personal data acquisition unit 1010 acquires personal data linked to an individual from the individual who uses the client terminal 1000 in question.

[0021] The client terminal learning unit 1020 learns the personal data acquired by the personal data acquisition unit 1010, and learns the common model and the individual model. The common model and the individual model will be described later.

[0022] The data encryption unit 1030 encrypts the personal data and the individual model to prevent leakage of personal information. The data decryption unit 1040 decrypts the personal data and the individual model encrypted by the data encryption unit 1030, and extracts the original personal data and the individual model.

[0023] The personal authentication / result output unit 1050 authenticates the individual using the personal data, and outputs the result of the personal authentication to an output device (not shown) or the like.

[0024] The virtual individual model generating unit 1060 generates a virtual individual model based on the individual model. The virtual individual model is an individual model of another user that is estimated from the relationship between the individual models before and after the update (e.g., the positional relationship in the individual model space) when the individual model of a certain user generated by a certain client terminal 1000 is updated by the learning server 1100. For example, in the learning server 1100, all individual models are optimized so as to be positioned as far apart as possible in order to improve the identifiability of individuals. For this reason, the positional relationship between the individual models before and after the update and the virtual individual model is adjusted by adjusting the position of the individual model after the update so that the individual model before the update and the virtual individual model are as far apart as possible. The virtual individual model is an individual model of another user that is estimated based on this viewpoint.

[0025] The personal data storage unit 1090 stores personal data and individual models. The common model storage unit 1091 stores the learned common model and utilizes it for personal authentication. The template storage unit 1092 creates registration templates to be used during personal authentication.

[0026] In this embodiment, one or more client terminals 1000 are involved in the processing.

[0027] The learning server 1100 includes a client terminal selection unit 1110 , a model update unit 1120 , and a common model storage unit 1191 .

[0028] The client terminal selection unit 1110 selects, from among the client terminals 1000 that can participate in the federated learning executed by the multiple client terminals 1000 included in the machine learning system S, a client terminal 1000 that will participate in the current learning round.

[0029] The model update unit 1120 updates the weights of the common model and the individual models. The weights are updated by defining a loss function for the model and minimizing the loss function.

[0030] The common model storage unit 1191 stores the learned common model.

[0031] Next, each process procedure in this embodiment will be described with reference to FIGS.

[0032] (Federated learning processing for personal authentication) 2 is a flowchart showing a federated learning process for personal authentication according to an embodiment. In FIG. 2, personal data of users 210a and 210b who operate two client terminals 1000a and 1000b are learned by the client terminals 1000a and 1000b, respectively, to generate a common model and an individual model for each of the users 210a and 210b.

[0033] In FIG. 2, federated learning of a machine learning model for personal authentication is performed between a client terminal 1000 and a learning server 1100.

[0034] Similarly, steps S2010 to S2080 are executed in the client terminals 1000a and 1000b.

[0035] First, in step S2010, the personal data acquisition unit 1010 of the client terminal 1000 executes a personal data collection process to collect personal data from the user 210. Details of the personal data collection process will be described later with reference to FIG.

[0036] The personal data is data used in personal authentication, which is the subject of this embodiment, and includes, for example, physical information such as fingerprints, faces, irises, veins, etc., and behavioral characteristics such as acceleration information, movement history, browsing history, and purchase history. Personal data may be acquired only for the purpose of data collection, or personal data acquired for the purpose of authentication may be accumulated. By repeatedly performing step S2010, personal data of the user 210 is accumulated in the client terminal 1000.

[0037] When the client terminal 1000 is occupied by one user, personal data of one specific user is stored, but this is not limited to this. For example, personal data of multiple users may be stored in a client terminal 1000 shared by multiple users, or personal data stored in multiple other client terminals 1000 may be aggregated in a client terminal 1000 and used for federated learning.

[0038] In step S2110, the client terminal selection unit 1110 of the learning server 1100 selects the client terminals 1000 that will participate in each learning round.

[0039] The method of selecting the client terminals 1000 may be a method of selecting all client terminals 1000 that can participate in the learning round. Alternatively, the method may be a method of randomly selecting a statically or dynamically determined number of client terminals 1000 from among the client terminals 1000 that can participate in the learning round. Alternatively, the method may be a method of determining some evaluation value for the client terminals 1000 and selecting the client terminals 1000 with the highest or lowest evaluation value. Alternatively, the method may be a method of selecting the client terminals 1000 with an evaluation value above or below a certain threshold value. Alternatively, the method may be a method of selecting the client terminals 1000 by weighting them according to their evaluation values.

[0040] In step S2120, the model update unit 1120 of the learning server 1100 transmits the common model stored in the common model storage unit 1191 to each of the client terminals 1000 participating in the learning round.

[0041] The common model is a machine learning model that inputs personal data and outputs a feature vector. The common model is used in common by multiple users 210 of multiple client terminals 1000. General machine learning models such as a linear regression model, a decision tree, and a neural network can be applied. When image data such as a fingerprint, face, iris, and vein are used as personal data, a deep learning model such as CNN (Convolutional Neural Networks) and Transformer can be applied to the common model. Note that the common model may be initialized with random weights at the start of learning, or may be pre-learned using personal data other than that stored in the client terminal 1000 and optimized with the weights.

[0042] In step S2020, the client terminal 1000 receives the common model transmitted in step S2120 from the learning server 1100. Next, in step S2030, the in-client terminal learning unit 1020 of the learning server 1100 executes in-client learning processing to update the common model and the individual model based on the personal data or the virtual individual model. Details of the in-client learning processing will be described later with reference to FIG. 4.

[0043] The common model is a machine learning model as described in step S2120. The individual model is a model that differs for each individual. When the client terminal 1000 stores personal data for only one person, the client terminal 1000 has one individual model.

[0044] For example, in distance learning, which extracts feature vectors from personal data and makes the distance between the feature vectors reflect the similarity between the personal data, a representative vector of the feature vectors can be used as an individual model. The representative vector is sensitive information linked to the user of the client terminal, and if it is obtained by a third party, there is a risk of impersonation or personal data leakage.

[0045] In step S2040, the client terminal 1000 transmits the common model and the individual model as the learning results of step S2030 to the learning server 1100. At this time, the client terminal 1000 may also transmit a virtual individual model to the learning server 1100 in addition.

[0046] In step S2130, the learning server 1100 receives the common model, the individual model, and the virtual individual model from the multiple client terminals 1000, 1000a, and 1000b. Then, in step S2140, the model update unit 1120 of the learning server 1100 executes a model update process that aggregates the common model and optimizes the individual models. Details of the model update process will be described later with reference to FIG. 5.

[0047] In step S2150, the learning server 1100 transmits the individual model updated in step S2140 to each client terminal. If the learning server 1100 also received a virtual individual model from the client terminal 1000 in step S2040, the learning server 1100 also transmits the virtual individual model.

[0048] At this time, since the individual model and the virtual individual model exist for each client terminal 1000, the corresponding individual model and virtual individual model are transmitted to each client terminal 1000. If the individual model were transmitted to a client terminal 1000 other than the client terminal 1000 linked to the individual model, there would be a risk of impersonation or personal data inference using the individual model.

[0049] In step S2050, the client terminal 1000 receives the individual model from the learning server 1100. Next, in step S2060, an individual model storage process is executed to store the individual model received from the learning server 1100 in the personal data storage unit 1090. Details of the individual model storage process will be described later with reference to FIG.

[0050] In step S2070, the virtual individual model generating unit 1060 of the client terminal 1000 executes a virtual individual model generating process for generating a virtual individual model based on the individual model stored in the personal data storage unit 1090 in step S2060.

[0051] In step S2080, the virtual individual model generating unit 1060 executes a virtual individual model storing process for storing the virtual individual model generated in step S2070 in the virtual individual model storing unit 1093. The virtual individual model storing process will be described in detail later with reference to FIG.

[0052] As a result, it is possible to improve the performance of a machine learning model for personal authentication without sharing individual models between client terminals 1000.

[0053] (Collection and processing of personal data) Fig. 3 is a flowchart showing the personal data collection process according to the embodiment. The personal data collection process shown in Fig. 3 is a detailed flowchart of step S2010 executed by the client terminal 1000a and the client terminal 1000b in Fig. 2.

[0054] First, in step S 3010 , the personal data acquisition unit 1010 of the client terminal 1000 acquires personal data from the user 210 .

[0055] Next, in step S3020, personal data acquisition unit 1010 selects personal data to be used in federated learning from the personal data acquired in step S3010. Although it is possible to use all acquired personal data for federated learning without performing step S3020, it is possible to reduce the amount of learning calculations and the storage capacity of personal data storage unit 1090 by performing data selection to reduce the personal data. In addition, it is expected that removing personal data that contributes little to the federated learning or personal data that has a negative impact will lead to improved accuracy of the common model that is ultimately obtained.

[0056] When selecting personal data, personal data whose evaluation value calculated for each personal data is above or below a threshold value, or a predetermined number of personal data from the top or bottom of the evaluation value, or personal data within a range of evaluation values ​​is selected.

[0057] Note that the selection of personal data may be performed by repeatedly excluding personal data based on the distance between personal data or an evaluation value calculated for a set of personal data, rather than calculating an evaluation value for each personal data.

[0058] For example, when using face images as personal data, face images appearing in a video are very similar in every frame, so learning a large number of similar face images is inefficient and does not improve accuracy.Face images that are facing straight to the side, that are covered by hands, or that are outside the image have too large an error and are therefore unsuitable for learning data.

[0059] In order to exclude such personal data, for example, feature extraction is performed for each face image to generate a feature vector, and if the distance between the feature vectors is sufficiently small, that is, if the face images are sufficiently similar, one of the face images is excluded from the learning data. This makes it possible to exclude similar personal data.

[0060] In addition, a quality value including the direction of the face, whether it is occluded, whether it protrudes from the image, etc. is calculated from the facial image, and inappropriate personal data can be excluded by excluding facial images with a quality value above or below a certain level.

[0061] Next, in step S1030, the data encryption unit 1030 of the client terminal 1000 encrypts the personal data selected in step S3020. Encryption is performed to prevent personal data leakage due to unauthorized access by a third party to the client terminal 1000, malware infection, and the like.

[0062] It should be noted that if the client terminal 1000 is trustworthy or if the security of the client terminal 1000 is guaranteed by means other than data encryption, data encryption may not be required.

[0063] Encryption generally requires a private key, which can be managed in a storage device within the client terminal, in a secure area within the client terminal such as a Trusted Execution Environment (TEE), or in an external medium such as a hardware token.

[0064] It is also possible to dynamically generate a private key from personal data such as biometric information using a biometric encryption technique such as Fuzzy Extractor. In this case, if a private key can be dynamically generated from the personal data to be learned in this embodiment, there is no need to prepare a private key for encryption. It is then possible to dynamically generate a private key from personal data obtained for personal authentication or personal data acquisition, and use the private key to perform encryption.

[0065] In step S3040, the data encryption unit 1030 stores the private data encrypted in step S3030 in the private data storage unit 1090.

[0066] As described above, personal data can be acquired from the user 210, encrypted in a secure form, and then stored in the personal data storage unit 1090.

[0067] (Learning process within the client terminal) FIG. 4 is a flowchart showing learning processing in a client terminal according to the embodiment.

[0068] First, in step S5010, the data decryption unit 1040 of the client terminal 1000 decrypts the personal data stored in the personal data storage unit 1090. To decrypt the personal data, the private key used when the data was encrypted is required. The private key is obtained or generated in the same manner as in step S3030.

[0069] Next, in step S5020, the intra-client-terminal learning unit 1020 selects learning data to be used for intra-client-terminal learning from the personal data stored in the personal data storage unit 1090. The selection of learning data can be performed by randomly extracting a predetermined number of personal data from the personal data storage unit 1090.

[0070] The selection of personal data may be performed by calculating an evaluation value for each piece of personal data, and selecting personal data whose evaluation value is above or below a certain threshold, or a predetermined number of personal data whose evaluation value is at the top or bottom of the evaluation value, or personal data whose evaluation value is within a range defined by a threshold.In addition, personal data may be selected by calculating the distance between personal data or an evaluation value for a set of personal data, and repeatedly excluding personal data based on the evaluation value.

[0071] Furthermore, from the perspective of learning, if the distance between the feature vector and the representative vector calculated from personal data is too close, the contribution to learning is small and the effectiveness is low, and if it is too far, it becomes difficult to reduce the distance through learning. Therefore, by calculating the distance between the feature vector and the representative vector and leaving only personal data whose distance is within a certain range, it is expected that efficient and highly accurate learning can be performed.

[0072] In step S5030, the client terminal 1000 branches the process depending on whether or not an individual model is stored in the personal data storage unit 1090. If an individual model is stored in the personal data storage unit 1090 (YES in step S5030), the data decryption unit 1040 reads the encrypted individual model from the personal data storage unit 1090 and decrypts it in step S5050.

[0073] If an individual model is not stored in the personal data storage unit 1090 (NO in step S5030), the intra-client terminal learning unit 1020 generates an individual model in step S5040.

[0074] The generation of the individual model in step S5040 is performed by inputting personal data into the common model to generate a feature vector, and performing statistical operations such as calculating a representative vector such as the average or median for the obtained feature vector. The feature vector is an example of a feature. The representative vector is an example of a representative value. The common model used at this time may be a pre-learned model stored at the start of the federated learning and used, or the latest common model received by the client terminal 1000 in step S2020 may be used. Note that the individual model may be initialized with random weights without using personal data.

[0075] Next, in step S5060, the client terminal 1000 branches the process depending on whether or not a virtual individual model is stored in the personal data storage unit 1090.

[0076] If the virtual individual model is not stored in the personal data storage unit 1090 (NO in step S5060), in step S5070 the client terminal learning unit 1020 updates the common model and the individual model. When updating the common model and the individual model, the personal data selected in step S5010, the individual model acquired in step S5040 or step S5050, and the common model received in step S2020 are used. The model is updated by defining a loss function for the model and searching for weights that minimize the function, regardless of the presence or absence of a virtual individual model.

[0077] In the absence of a virtual individual model, the loss function is defined as a loss for an ideal relationship by formulating the relationship between the personal data and the corresponding common model and individual model. For example, when using a representative vector of feature vectors generated from personal data as an individual model, the ideal relationship is one in which the feature vector generated by inputting all personal data into the common model is equal to the representative vector. In this case, a policy can be adopted in which the sum of the squares of the distances between the feature vectors and the representative vector is adopted as the loss function and the model is updated by minimizing this.

[0078] If the virtual individual model is present (YES in step S5060), in step S5080, the data decryption unit 1040 reads and decrypts the encrypted virtual individual model from the personal data storage unit 1090. The private key required for decryption is acquired or generated in the same procedure as in step S3030.

[0079] Next, in step S5090, a virtual individual model to be used for learning within the client terminal is selected from the virtual individual models stored in the personal data storage unit 1090. The selection of the virtual individual model can be performed by randomly extracting a predetermined number of virtual individual models from the personal data storage unit 1090.

[0080] This can also be done by calculating an evaluation value for each virtual individual model and selecting personal data whose evaluation value is equal to or greater than a threshold or less than the threshold, or a predetermined number of personal data whose evaluation value is higher or lower, or personal data whose evaluation value is within a range determined by the threshold. Also, a virtual individual model can be selected by calculating the distance between virtual individual models or an evaluation value for a set of virtual individual models and repeatedly excluding personal data based on the evaluation value.

[0081] In step S5100, the intra-client terminal learning unit 1020 updates the common model and the individual model. When updating the common model and the individual model, the personal data selected in step S5010, the individual model acquired in step S5040 or S5050, the virtual individual model selected in step S5090, and the common model received in step S2020 are used.

[0082] In step S5100, when there is a virtual individual model, the loss function is defined as the loss for an ideal relationship between the personal data and the common model, the individual model, and the virtual individual model therefor.

[0083] When the individual model and the virtual individual model are treated as feature vectors, a policy of updating the model can be adopted by minimizing a loss function. This loss function is defined according to the ratio or difference between the distance between the feature vector obtained from the personal data and the common model and the individual model, and the distance between the feature vector obtained from the personal data and the common model and the virtual individual model.

[0084] In addition, when a deep learning model such as CNN (Convolutional Neural Networks) or Transformer is used as the common model, an optimization method such as stochastic gradient descent is applied to search for weights of the common model and individual models that minimize the loss function.

[0085] Note that when learning the common model, noise may be added to the personal data, the individual model, the virtual individual model, and the common model before learning. It is known that the personal data used in learning can be inferred from the learning results in the client terminal 1000, but adding noise in this way makes it difficult to infer the personal data.

[0086] In this manner, the common model and the individual model are learned.

[0087] (Model update process) FIG. 5 is a flowchart showing a model update process according to the embodiment.

[0088] First, in step S6010, the model update unit 1120 of the learning server 1100 verifies whether the client terminal 1000 that has transmitted the learning result is the correct client terminal 1000 selected in step S2110. Specifically, the model update unit 1120 records the terminal number, IP address, etc. as information for identifying the client terminal 1000 selected in step S2110. Then, in step S2130, the model update unit 1120 verifies by comparing the terminal number, IP address, etc. of the client terminal 1000 that transmitted the learning result.

[0089] If the client terminal 1000 that transmitted the learning result is a valid client terminal 1000, the model update unit 1120 proceeds to the processing of step S6020 and subsequent steps. On the other hand, if the client terminal 1000 is an invalid client terminal 1000, the model update unit 1120 excludes the client terminal 1000 from the model update. This is done for the purpose of preventing a so-called Model Poisoning Attack, in which a client terminal 1000 transmits an invalid learning result to reduce the accuracy of the model.

[0090] Next, in step S6020, the model update unit 1120 aggregates the common models received from the multiple client terminals 1000. In this process, the weight of a single common model is calculated from the weights of the multiple common models learned by each client terminal 1000. As a method of calculating the weight, for example, a method of simply averaging the weights can be adopted.

[0091] Next, in step S6030, the model update unit 1120 optimizes the individual models received from the multiple client terminals 1000. For example, when a representative vector for feature vectors generated from personal data is used as the individual model, the representative vectors of each client terminal 1000 are aggregated in the learning server 1100.

[0092] This representative vector is the vector that is located at the center when each individual's personal data is input into a common model to generate a feature vector. Therefore, if the distance between the representative vectors of two individuals is small, it becomes difficult to distinguish the two individuals using their feature vectors.

[0093] Therefore, it is ideal for the distance between all representative vectors to be greater than a certain value. By expressing this property in the form of a loss function and finding a representative vector that reduces the loss function, the representative vector can be optimized.

[0094] For example, a loss function can be defined by defining a margin value between representative vectors, and when the distance between the representative vectors is smaller than the margin, calculating the square of the margin and the distance between the representative vectors and taking the sum of these values. When this loss function becomes 0, it is an ideal situation in which the distance between all representative vectors is equal to or greater than the margin.

[0095] In addition, at this time, if a virtual individual model is also received in addition to the individual model, processing may be performed to make the virtual individual model closer to an individual model of a client other than the client that sent the virtual individual model. In this case, the virtual individual model can more closely approximate the actual individual model, making it easier for each client terminal to take into account the individual models of others.

[0096] A method of aggregating multiple virtual individual models into a single virtual individual model by clustering the virtual individual models can also be used. In this case, multiple client terminals can treat the same virtual individual model as someone else's individual model, improving convergence.

[0097] On the other hand, these processes also lead to providing each client terminal with additional information about the individual models of other people. Therefore, when performing these processes, it is better to use measures to protect privacy, such as adding noise or randomly selecting virtual individual models to be processed.

[0098] Finally, in step S6040, the model update unit 1120 stores the common model aggregated in step S6020 in the common model storage unit 1191.

[0099] As a result of the above, the common model and the individual models are updated, and the aggregated common model is stored in the common model storage unit 1191.

[0100] (Individual model storage processing) FIG. 6 is a flowchart showing the individual model storage process in the client terminal 1000 according to the embodiment.

[0101] First, in step S8010, the data encryption unit 1030 of the client terminal 1000 encrypts the individual model transmitted by the learning server 1100 in step S2150. The encryption is performed by using a secret key prepared by some means, similar to the personal data encryption in step S3010.

[0102] Next, in step S8020, the data encryption unit 1030 stores the individual model encrypted in step S8010 in the personal data storage unit 1090.

[0103] As a result, the individual model sent from the server can be encrypted and then safely stored in the personal data storage unit 1090.

[0104] (Virtual individual model generation process) FIG. 7 is a flowchart showing a virtual individual model generation process in the client terminal 1000 according to the embodiment.

[0105] First, in step S8110, the data decryption unit 1040 decrypts the individual model stored in the personal data storage unit 1090 to obtain an individual model to be used for generating a virtual individual model.

[0106] Next, in step S8120, the virtual individual model generating unit 1060 selects an individual model to be used for generating the virtual individual model. The selection of the individual models may be performed by randomly extracting a predetermined number of individual models from the personal data storage unit 1090, or by extracting the newest individual models in chronological order.

[0107] Next, in step S8130, the virtual individual model generating unit 1060 generates a virtual individual model based on the selected individual model. The virtual individual model is generated based on the individual model.

[0108] For example, when a representative vector for a feature vector generated from personal data is used as an individual model, the feature vector is generated as a virtual individual model. One method for generating a virtual individual model is to use the latest individual model (individual model 1 after update by the learning server 1100) and the next latest individual model (individual model 2 before update by the learning server 1100). In this case, a virtual individual model can be generated by moving individual model 1 a predetermined distance in the direction of individual model 2 according to the amount of mutation from individual model 1 to individual model 2.

[0109] Here, the predetermined distance may be determined in advance for generating the virtual individual model, or may be determined based on a loss function used in the current learning within the client terminal and in optimizing the individual model within the learning server 1100. Furthermore, when there are multiple individual models, a method can be adopted in which a distribution of the individual models is obtained, and an individual model located away from the center of the distribution is treated as a virtual individual model as it is.

[0110] (Virtual individual model storage processing) FIG. 8 is a flowchart showing the virtual individual model storage process in the client terminal 1000 according to the embodiment.

[0111] First, in step S8210, the data encryption unit 1030 encrypts the virtual individual model generated in step S2070. The encryption is performed by using a secret key prepared by some means, similar to the personal data encryption in step S3010.

[0112] Next, in step S8220, the data encryption unit 1030 stores the virtual individual model encrypted in step S8210 in the individual data storage unit 1090.

[0113] At this time, the virtual individual models may be selected and reduced in number to reduce the storage capacity of the personal data storage unit 1090. As with the personal data, removing virtual individual models that contribute little to learning or virtual individual models that were generated relatively recently is expected to lead to improved accuracy of the common model that is finally obtained.

[0114] As a result, the individual model sent from the server can be encrypted and then safely stored in the personal data storage unit 1090.

[0115] (Personal authentication processing) 9 is a flowchart showing the personal authentication process according to the embodiment. The personal authentication process is composed of three steps: model setting, personal registration, and personal authentication.

[0116] First, steps S9010 to S9110 for setting the model will be described.

[0117] First, in step S9110, the learning server 1100 reads out the common model from the common model storage unit 1191 and transmits it to the client terminal 1000. Next, in step S9010, the client terminal 1000 receives the common model and stores it in the common model storage unit 1091.

[0118] Step S9010 is not performed every time personal authentication is performed, but is performed by a developer or an administrator when starting operation of a new authentication system or when upgrading the authentication system.

[0119] Next, steps S9020 to S9040 of individual registration will be described.

[0120] In step S9020, the personal data acquisition unit 1010 acquires personal data from the user 210. Next, in step S9030, the client terminal 1000 acquires a common model from the common model storage unit 1091, and inputs the personal data into the common model to generate a feature vector. Next, in step S9040, the client terminal 1000 generates a registration template from the feature vector obtained in step S9030, and stores it in the template storage unit 1092.

[0121] A template is information for registration generated from personal data, and a feature vector may be used as a template as is. Note that some template protection technology may be applied when generating the template to prevent the original personal data from being leaked from the template.

[0122] Finally, steps S9050 to S9080 of personal authentication will be described.

[0123] Finally, in step S9050, the personal authentication / result output unit 1050 acquires personal data from the user 210. Next, in step S9060, the personal authentication / result output unit 1050 inputs the personal data acquired in step S9050 to the common model stored in the common model storage unit 1091 to generate a feature vector.

[0124] Next, in step S9070, the personal authentication result output unit 1050 compares the feature vector obtained in step S9060 with the template stored in the template storage unit 1092, and calculates similarity, dissimilarity, etc. as the comparison result. For example, when authentication is performed based on the distance between feature vectors, the Hamming distance, Euclidean distance, etc. between the feature vectors is calculated and used as the dissimilarity.

[0125] Finally, in step S9080, the personal authentication result output unit 1050 performs threshold processing or the like on the comparison result of step S9070 to determine whether the authentication was successful or unsuccessful, and outputs the result.

[0126] The matching result is not limited to continuous values ​​such as similarity and dissimilarity, and if the template is generated using template protection technology, the matching result may be a binary value, either successful or unsuccessful. In that case, threshold processing is not required, and the matching result is directly output as the authentication result.

[0127] As a result, personal authentication can be performed by utilizing the common model learned based on the processing in FIG.

[0128] The above-described embodiment has been written assuming that the user of the client terminal 1000 is an individual, but it is also applicable to cases where the user of the client terminal 1000 is an organization or institution such as a company.

[0129] In the above embodiment, the processing executed by the client terminal 1000 may be executed by the learning server 1100, and vice versa. However, this is premised on the fact that personal data, individual models, and virtual individual models managed by a client terminal 1000 are not leaked to other client terminals 1000.

[0130] (Hardware configuration of Computer 8000) FIG. 10 is a diagram showing the hardware configuration of a computer 8000 that realizes the client terminal 1000 and the learning server 1100 according to the embodiment.

[0131] The computer 8000 includes a CPU (Central Processing Unit) 8010 , a main memory device 8020 , an auxiliary memory device 8030 , an input device 8040 , an output device 8050 , and a communication device 8060 .

[0132] The CPU 8010 executes programs corresponding to the client terminal learning unit 1020 , the data encryption unit 1030 , the data decryption unit 1040 , the virtual individual model generation unit 1060 , the client terminal selection unit 1110 , the model update unit 1120 , and the parameter generation unit 1210 .

[0133] The main memory device 8020 is a device equivalent to the memory of the computer 8000. The main memory device 8020 stores programs corresponding to the client terminal intra-learning unit 1020, the data encryption unit 1030, the data decryption unit 1040, the virtual individual model generation unit 1060, the client terminal selection unit 1110, and the model update unit 1120. By executing these programs in the CPU 8010, each process is realized.

[0134] A part or all of the programs and data may be stored in the main memory device 6020 in advance, or may be introduced from a non-transitory storage medium or from an information processing device equipped with an external non-transitory storage device via a network.

[0135] The auxiliary storage device 8030 is a storage device represented by a hard disk drive (HDD) or a solid state drive (SSD), and corresponds to the personal data storage unit 1090, the common model storage unit 1091, the template storage unit 1092, the virtual individual model storage unit 1093, and the common model storage unit 1191. Data stored in each unit is accumulated as data on the auxiliary storage device 8030.

[0136] The input device 8040 is used to read information in the personal data acquisition unit 1010. The input device 8040 can be configured to include devices such as a keyboard, a biosensor, a touch panel, a smart device, a scanner, and a camera.

[0137] The output device 8050 is used to output information from the authentication result output unit 1050 via a device such as a display. The communication device 8060 is used for communication between the client terminal 1000 and the learning server 1100.

[0138] As a result, in the above-described embodiment, learning in a client terminal can be performed taking into account the individual models of others, and a common model with higher accuracy can be constructed than in the case of general associative learning.

[0139] In other words, in federated learning for personal authentication, when the client terminal 1000 can only use one person's personal data for learning, the degradation in performance of the trained model that occurs when the individual models of other individuals cannot be taken into account is mitigated.

[0140] The present invention is not limited to the above-described embodiment, and can be implemented using any components without departing from the scope of the present invention. The above-described embodiments and modifications are merely examples, and the present invention is not limited to these contents as long as the features of the invention are not impaired. In addition, although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of ​​the present invention are also included in the scope of the present invention.

[0141] For example, some of the functions of each device in the above-described embodiments may be provided in other devices or other functional units, or functions provided in other devices or other functional units may be provided in the same device or the same functional unit. In other words, the distribution and integration of the processing functions of each function can be appropriately changed based on the processing load and execution efficiency. [Explanation of symbols]

[0142] S: machine learning system, 1000, 1000a, 1000b: client terminal, 1010: personal data acquisition unit, 1020: learning unit within client terminal, 1030: data encryption unit, 1040: data decryption unit, 1050: authentication and result output unit, 1060: virtual individual model generation unit, 1090: personal data storage unit, 1091, 1191: common model storage unit, 1092: template storage unit, 1093: virtual model storage unit, 1100: learning server, 1110: client terminal selection unit, 1120: model update unit, 8000: computer, 8010: CPU, 8020: main memory device.

Claims

1. A learning method executed by a learning system that learns a common model by inputting personal data relating to an individual and outputting features, comprising: The learning system includes: managing the common model used in common by a plurality of the individuals, an individual model generated from the personal data of each of the individuals, and virtual individual models which are the individual models of other individuals different from each of the individuals and which are estimated based on the individual models for each of the individuals; a processor of the learning system, collecting the personal data for each of the individuals; inputting the personal data for each of the individuals into the common model and generating a representative value of the feature amount for each of the individuals as the individual model for each of the individuals; optimizing the individual models for the individuals so as to minimize a value of a loss function whose value decreases as the distance between the individual models for the individuals increases; generating the virtual individual model for each of the individuals based on a relationship between the optimized individual model for each of the individuals and the individual model for each of the individuals before optimization; updating the common model and the individual model for each individual based on a relationship between the common model and the individual model for each individual, or a relationship between the common model, the individual model for each individual, and the virtual individual model for each individual; A learning method characterized by including each process.

2. 2. The learning method according to claim 1, The learning system includes a learning server and a terminal; the learning server manages the common model; the terminal manages the individual model for each of the individuals and the virtual individual model for each of the individuals; a processor of the terminal, collecting the personal data for each of the individuals; receiving the common model from the learning server; generating the individual model for each of the individuals; updating the common model and the individual model for each individual based on a relationship between the common model and the individual model for each individual; A learning method characterized by including each process.

3. 3. The learning method according to claim 2, a processor of the terminal, The common model and the individual model for each of the individuals are updated so as to minimize a value of a loss function based on a relationship between the common model and the individual model for each of the individuals. A learning method characterized by including processing.

4. 4. The learning method according to claim 3, a processor of the learning server, aggregating the common model updated by each of the terminals to update the common model managed by the learning server; optimizing the individual model for each of the individuals so as to minimize a value of a loss function whose value decreases as the distance between the individual models for each of the individuals updated by each of the terminals increases; a processor of the terminal, generating the virtual individual model for each of the individuals based on a relationship between the individual model for each of the individuals optimized by the learning server and the individual model for each of the individuals before being optimized by the learning server; The common model and the individual model for each of the individuals are updated so as to minimize a value of a loss function based on a relationship between the common model, the individual model for each of the individuals, and the virtual individual model for each of the individuals. A learning method characterized by including each process.

5. 5. The learning method according to claim 4, a processor of the terminal, The common model and the individual model for each individual are updated so as to minimize a loss function based on a distance between the feature amount for each individual generated based on the common model and the personal data for each individual and the individual model for each individual, and a distance between the feature amount and the virtual individual model for each individual. A learning method characterized by including processing.

6. 5. The learning method according to claim 4, a processor of the terminal, The individual model for each individual optimized by the learning server is moved in the space of the individual models toward the individual model for each individual before being optimized by the learning server according to the amount of variation of the individual model for each individual due to optimization, thereby generating the virtual individual model for each individual. A learning method characterized by including processing.

7. 5. The learning method according to claim 4, a processor of the terminal, The virtual individual model for each individual is generated based on the feature amount for each individual generated based on the common model and the personal data for each individual, and based on the distribution of the individual model for each individual. A learning method characterized by including processing.

8. 5. The learning method according to claim 4, a processor of the terminal, transmitting the updated individual model for each of the individuals and the generated virtual individual model for each of the individuals to the learning server; a processor of the learning server, modifying the virtual individual model for each of the individuals based on the individual model for each of the individuals received from the terminal; Transmitting the modified virtual individual model for each of the individuals to the terminal. A learning method characterized by including each process.

9. A learning system that learns a common model by inputting personal data relating to an individual and outputting features, managing the common model used in common by a plurality of the individuals, an individual model generated from the personal data of each of the individuals, and virtual individual models which are the individual models of other individuals different from each of the individuals and which are estimated based on the individual models for each of the individuals; The processor of the learning system collecting the personal data for each of the individuals; inputting the personal data for each of the individuals into the common model and generating a representative value of the feature amount for each of the individuals as the individual model for each of the individuals; optimizing the individual models for the individuals so as to minimize a value of a loss function whose value decreases as the distance between the individual models for the individuals increases; generating the virtual individual model for each of the individuals based on a relationship between the optimized individual model for each of the individuals and the individual model for each of the individuals before optimization; updating the common model and the individual model for each individual based on a relationship between the common model and the individual model for each individual, or a relationship between the common model, the individual model for each individual, and the virtual individual model for each individual; A learning system characterized by:

10. 10. The learning system of claim 9, The learning system includes a learning server and a terminal; the learning server manages the common model; the terminal manages the individual model for each of the individuals and the virtual individual model for each of the individuals; The processor of the terminal collecting the personal data for each of the individuals; receiving the common model from the learning server; generating the individual model for each of the individuals; updating the common model and the individual model for each individual based on a relationship between the common model and the individual model for each individual; A learning system characterized by:

11. The learning system according to claim 10, The processor of the terminal The common model and the individual model for each of the individuals are updated so as to minimize a value of a loss function based on a relationship between the common model and the individual model for each of the individuals. A learning system characterized by:

12. 12. The learning system of claim 11, The processor of the learning server aggregating the common model updated by each of the terminals to update the common model managed by the learning server; optimizing the individual model for each of the individuals so as to minimize a value of a loss function whose value decreases as the distance between the individual models for each of the individuals updated by each of the terminals increases; The processor of the terminal generating the virtual individual model for each of the individuals based on a relationship between the individual model for each of the individuals optimized by the learning server and the individual model for each of the individuals before being optimized by the learning server; The common model and the individual model for each of the individuals are updated so as to minimize a value of a loss function based on a relationship between the common model, the individual model for each of the individuals, and the virtual individual model for each of the individuals. A learning system characterized by:

13. 12. The learning system of claim 11, The processor of the terminal The common model and the individual model for each individual are updated so as to minimize a loss function based on a distance between the feature amount for each individual generated based on the common model and the personal data for each individual and the individual model for each individual, and a distance between the feature amount and the virtual individual model for each individual. A learning system characterized by:

14. 12. The learning system of claim 11, The processor of the terminal The individual model for each individual optimized by the learning server is moved in the space of the individual models toward the individual model for each individual before being optimized by the learning server according to the amount of variation of the individual model for each individual due to optimization, thereby generating the virtual individual model for each individual. A learning system characterized by:

15. 12. The learning system of claim 11, The processor of the terminal The virtual individual model for each individual is generated based on the feature amount for each individual generated based on the common model and the personal data for each individual, and based on the distribution of the individual model for each individual. A learning system characterized by:

16. 12. The learning system of claim 11, The processor of the terminal transmitting the updated individual model for each of the individuals and the generated virtual individual model for each of the individuals to the learning server; The processor of the learning server modifying the virtual individual model for each of the individuals based on the individual model for each of the individuals received from the terminal; Transmitting the modified virtual individual model for each of the individuals to the terminal. A learning system characterized by:

Images