MACHINE LEARNING DEVICE, MACHINE LEARNING METHOD AND MACHINE LEARNING PROGRAM

Abstract

A first learning unit (22) generates a first learning model by using each of n pieces of learning data as a subject, and performing training using the subject learning data. A model integration unit (23) generates an integrated model by integrating m pieces of first learning models selected from m pieces of first learning models. A data generation unit (24) generates new learning data by rewriting a label assigned to subject data with a soft label, which is a result obtained by inputting, to the integrated model, subject data that is different learning data than learning data used for training in generating the m pieces of first learning models that are the basis of the integrated models, as input. A second learning unit (25) generates a second learning model by performing training using the new learning data.

Description

Technical FieldThe present disclosure relates to a method of training a machine learning model.Background of the Prior ArtThe presence of a membership inference attack has been identified as a privacy problem in machine learning. The membership inference attack is an attack such as the following. Normal input data is input to an intervening machine learning model (hereinafter referred to as a target model), and an inference result to which the target model responds is observed. Thereby, it is specified whether or not the input data is included in the learning data of the target model (=observance of whether or not the input data is a membership).Patent Literature 1 and Non-Patent Literatures 1 and 2 describe defense measures against a membership inference attack.In Patent Literature 1, the learning data is divided into data including data protection information and data not including data protection information. The learning data containing data protection information is used only for training an input layer of a machine learning model. The learning data that does not include privacy information is used for retraining all layers of the machine learning model. The retraining machine learning model has resistance to membership inference attack.In Non-Patent Literature 1, learning data that is not labeled and does not include privacy information is assigned a label (hereinafter, referred to as a soft label) using a machine learning model trained on the learning data including privacy information. The learning data that does not include privacy information is, for example, publicly available data. Another machine learning model is trained on the learning data to which the soft label is attached. This trained machine learning model has resistance to membership inference attack.In Non-Patent Literature 2, the initial learning data is divided into a constant number n of sets of pieces. For each of the n sets of pieces, n-1 sets of pieces excluding the set are set as learning data. That is, n pieces of learning data including n-1 sets of pieces are set. Training is performed using each of the n pieces of learning data, and a machine learning model corresponding to each of the n pieces of learning data is generated. For each of n pieces of machine learning models, a set not included in the learning data used for training a subject machine learning model is input to the subject machine learning model as an input to obtain a soft label. The label of the sentence input as the input is rewritten with the soft label and regarded as new learning data. The machine learning model is subsequently trained on the basis of the new learning data. This trained machine learning model has resistance to membership inference attack.Reference listPatent Literature[Patent Literature 1] JP 2021-193533 ANon-Patent Literature[Non-Patent Literature 1] Virat Shejwalkar et. al, "Membership Privacy for Machine Learning Models Through Knowledge Transfer", AAAI2021[Non-Patent Literature 2] Rishav Chourasia et al, "Knowledge Cross-Distillation for Membership Privacy", PETS2022Summary of the InventionTechnical ProblemThe defenses described in Patent Literature 1 and Non-Patent Literature 1 require learning data that does not contain data protection information. In fields such as medicine and financial, since sensitive data can be used for machine learning, it is difficult to prepare the learning data that does not contain data protection information.The defense measure described in Non-Patent Literature 2 does not require learning data that does not contain data protection information. However, the defense measure described in Non-Patent Literature 2 requires additionally training n×(n-1) pieces of learning data depending on the number of divisions n of the learning data. Therefore, the calculation amount of the defense measure is larger than that of other conventional defense measures.The present disclosure aims to reduce the computational cost while eliminating the need for learning data that does not contain privacy information and provide resistance to a membership inference attack.Solution of the ProblemA machine learning device according to the present disclosure includes:a first learning unit to generate n pieces of first learning models by using each of n pieces of first learning data to which a label is assigned, where an integer n is equal to or greater than 3, as a subject, performing training using subject learning data, and generating the first learning model according to the subject learning data;a model integration unit for generating an integrated model by integrating m pieces of first learning models selected from the n pieces of first learning models generated by the first learning unit, wherein an integer m is less than n;a data generation unit for generating new learning data by rewriting a label assigned to subject data with a soft label that is a result obtained by inputting the subject data as input to the integrated model generated by the model integration unit, the subject data being learning data other than learning data used for training in generating the m pieces of the first learning models that are the basis of the integrated model; anda second learning unit to generate a second learning model by performing training using the new learning data generated by the data generation unit.Advantageous Effects of the InventionIn the present disclosure, a first learning model generated by performing training using subject learning data is integrated and an integrated model is generated, and new learning data is generated by the integrated model. Thereby, it is possible to reduce the computational cost and at the same time to eliminate the need for learning data that does not contain data protection information and provide resistance to membership inference attack.Brief Description of the DrawingsFIG. 1 is a configuration diagram of a machine learning device 10 according to Embodiment 1. FIG. 2 is a flowchart illustrating a processing flow of the machine learning device 10 according to Embodiment 1. FIG. 3 is an explanatory diagram of a concrete example of the operation of the machine learning device 10 according to Embodiment 1. FIG. 4 is a configuration diagram of the machine learning device 10 according to Modification 2. FIG. 5 is a configuration diagram of the machine learning device 10 according to Embodiment 2. FIG. 6 is a flowchart illustrating a processing flow of the machine learning device 10 according to Embodiment 2. FIG. 7 is an explanatory diagram of a concrete example of the operation of the machine learning device 10 according to Embodiment 2.DESCRIPTION OF EMBODIMENTSEmbodiment 1.*** Description of configuration ***A configuration of a machine learning device 10 according to Embodiment 1 will be described with reference to FIG. 1.The machine learning device 10 is a computer.The machine learning device 10 includes hardware components that are a processor 11, a memory 12, and a mass storage 13. The processor 11 is connected to other hardware components via signal lines and controls the other hardware components.The processor 11 is an IC that performs processing. IC is an abbreviation for Integrated Circuit (dt). Specific examples of the processor 11 include a CPU, a DSP, and a GPU. CPU is an abbreviation for Central Processing Unit (dt. central processing unit). DSP is an abbreviation for Digital Signal Processor (dt. Digital Signal Processor). GPU is an abbreviation for Graphics Processing Unit (dt. Graphics Processing Unit).The working memory 12 is a storage device that temporarily stores data. Specific examples of the random access memory 12 include an SRAM and a DRAM. SRAM is an abbreviation for Static Random Access Memory (dt. static random access memory). DRAM is an abbreviation for Dynamic Random Access Memory (dt. Dynamic Random Access Memory).The mass storage 13 is a storage device that retains data. A specific example of the mass storage 13 is an HDD. HDD is an abbreviation for Hard Disk Drive (Hard Disk Drive). Alternatively, the mass storage 13 may be a portable recording medium, for example, an SD (registered trademark) memory card, CompactFlash (registered trademark), a NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. SD is an abbreviation for secure digital (dt. secured digital). DVD is an abbreviation for Digital Versatile Disk (dt. digital versatile disk).The machine learning device 10 includes, as functional components, a data division unit 21, a first learning unit 22, a model integration unit 23, a data generation unit 24, and a second learning unit 25.The mass storage 13 stores programs that implement the functions of the individual functional components of the machine learning device 10. These programs are loaded into the working memory 12 by the processor 11 and executed by the processor 11. Thereby, the functions of the individual functional components of the machine learning device 10 are realized.The mass storage 13 stores a plurality of pieces of learning data 31 and a learning model 32.FIG. 1 shows only a single processor 11, however, a plurality of processors 11 may be provided, and the plurality of processors 11 may cooperate to execute programs that realize the individual functions.*** Description of the Mode of Operation ***The operation of the machine learning device 10 according to Embodiment 1 will be described with reference to FIGS. 2 and 3.An operation flow of the machine learning device 10 according to Embodiment 1 corresponds to a machine learning method according to Embodiment 1. Further, a program implementing the operation of the machine learning device 10 according to Embodiment 1 corresponds to a machine learning program according to Embodiment 1.A processing flow of a machine learning device 10 according to Embodiment 1 will be described with reference to FIG. 2.(Step S 11: Data Division Process)A data division unit 21 reads a plurality of pieces of learning data 31 stored in the mass storage 13 into the work memory 12. the data division unit 21 divides the read plurality of pieces of learning data 31 into a constant number n of sets of pieces. n is an integer equal to or greater than 3. the data division unit 21 evenly divides the plurality of pieces of learning data 31 so that the number of pieces of learning data 31 included in each set is approximately equal, for example. In this manner, n sets of data (hereinafter referred to as learning data 33) of learning data are generated. The data division unit 21 writes the n pieces of learning data 33 into the work memory 12.(Step S 12: First Learning Process)The first learning unit 22 reads the n pieces of learning data 33 generated in step S 11 and the learning model 32 from the work memory 12. The first learning unit 22 performs training on the learning model 32 using the subject learning data 33, and generates a first learning model 34 corresponding to the subject learning data 33. Thereby, n pieces of the first learning model 34 are generated. The first learning unit 22 writes the n pieces of first learning models 34 into the work memory 12.(Step S 13: Model Integration Process)The model integration unit 23 reads the n pieces of first learning data 34 generated in step S 12 from the work memory 12. the model integration unit 23 generates an integrated model 35 by integrating m pieces of first learning models 34 selected from the n pieces of first learning models 34. m is an integer less than n. Here, the model integration unit 23 generates the integrated model 35 for each combination of the m pieces of first learning models 34 that can be selected from the n pieces of first learning models. The model integration unit 23 writes the integrated model 35 into the working memory 12 for each combination.The model integration unit 23 adds the parameters of the m pieces of the first learning models 34, and performs arithmetic processing such as additive averaging or weighted averaging for each parameter. Thereby, the model integration unit 23 generates the integrated model 35 by integrating m pieces of first learning models 34.In Embodiment 1, it is assumed that m=n-1. There are n combinations of the selection of n-1 pieces of the first learning models 34 from the n pieces of the first learning models 34. that is, there are the n combinations that are a combination of the remaining n-1 pieces of first learning models 34 in which the first first learning model 34 has been excluded from the n pieces of first learning models 34, a combination of the remaining n-1 pieces of first learning models 34 in which the second first learning model 34 has been excluded from the n pieces of first learning models 34,..., and a combination of the remaining n-1 pieces of first learning models 34 in which the nth first learning model 34 has been excluded from the n pieces of first learning models 34.Thereby, the model integration unit 23 generates the integrated model 35 by integrating for each of the n combinations of the first learning models 34 of the combination. Consequently, n pieces of integrated models 35 are generated.(Step S 14: Data Generation Process)The data generation unit 24 reads each integrated model 35 generated in step S 13 from the work memory 12.The data generation unit 24 provides, to the integrated subject model 35, subject data 36 that is learning data 33 other than the learning data 33 used for training in generating the m pieces of first learning models that form the basis of the integrated subject model 35 as input to cause the integrated subject model 35 to perform inference. The data generation unit 24 obtains a soft label that is a result of inference by the integrated subject model 35. The data generation unit 24 generates new learning data 37 by rewriting the label associated with the subject data 36 with the soft label. The data generation unit 24 aggregates pieces of new learning data 37 generated for each integrated model 35, respectively, and writes the aggregated data into the work memory 12 as a record of the new learning data 37.In Embodiment 1, the data generation unit 24 reads the n pieces of integrated models 35. the data generation unit 24 sets each of the n pieces of integrated models 35 as the subject integrated model 35.The data generation unit 24 provides, to the integrated subject model 35, subject data 36 that is learning data 33 other than the learning data 33 used for training in generating the n-1 pieces of first learning models that form the basis of the integrated model 35 as input to cause the integrated subject model 35 to perform inference. For example, when the integrated subject model 35 is generated from the combination of the remaining n-1 pieces of the first learning models 34 in which the first first learning model 34 has been excluded, the first learning model 34 is the subject data 36. When the integrated subject model 35 is generated from the combination of the remaining n-1 pieces of the first learning models 34 in which the second first learning model 34 has been excluded, the first learning model 34 is the subject data 36. the data generation unit 24 generates the new learning data 37 by rewriting the label associated with the subject data 36 with the soft label.The data generation unit 24 aggregates the new learning data 37 generated for each of the n pieces of integrated models 35 and writes the aggregated data as the record of the new learning data 37 into the work memory 12.(Step S 15: Second Learning Process)The second learning unit 25 reads, from the work memory 12, the record of the new learning data 37 and the learning model 32 generated in step S 14. the second learning unit 25 performs training on the learning model 32 using the record of the new learning data 37, and generates a second learning model 38.A concrete example of the operation of the machine learning device 10 according to Embodiment 1 will be described with reference to FIG. 3.FIG. 3 shows an example of the case where n, a division number, 3 and m are n-1.In step S 11, the data division unit 21 divides the learning data 31 including data protection information into 3 (=n) equal pieces.In this manner, learning data 33A, learning data 33B, and learning data 33C are generated.In step S 12, the first learning unit 22 sets each of the n pieces of learning data 33 as the subject learning data 33. The first learning unit 22 trains the learning model 32 using the subject learning data 33, and generates the first learning model 34 corresponding to the subject learning data 33.At this time, three pieces of learning data 33 are generated: a first learning model 34A trained on learning data 33A; a first learning model 34B trained on learning data 33B; and a first learning model 34C trained on learning data 33C.In step S 13, the model integration unit 23 sets each combination of 2 (=m=n-1) pieces of the first learning models 34 that can be selected from the three pieces of the first learning models 34 as a subject combination. The model integration unit 23 generates the integrated model 35 by integrating the two pieces of first learning models 34 included in the subject combination.Here, three pieces of integrated models 35 are created: an integrated model 35A in which the first learning model 34A and the first learning model 34B are integrated; an integrated model 35B in which the first learning model 34B and the first learning model 34C are integrated; and an integrated model 35C in which the first learning model 34A and the first learning model 34C are integrated.In step S 14, the data generation unit 24 sets each of the three pieces of integrated models 35 as the subject integrated model 35. The data generation unit 24 provides, to the integrated subject model 35, the subject data 36 which is the learning data 33 that is not used for training the two pieces of first learning models 34 that are the basis of the integrated subject model 35 as input. The integrated model 35A is provided with the learning data 33C that is not used for training the first learning model 34A and the first learning model 34B as input. The integrated model 35B is provided with the learning data 33A that is not used for training the first learning model 34B and the first learning model 34C as input. The integrated model 35C is provided with the learning data 33B that is not used for training the first learning model 34A and the first learning model 34C as input.The data generation unit 24 generates the new learning data 37 by rewriting the label assigned to the subject data 36 with the soft label that is a result obtained by the inference by the integrated subject model 35. That is, the label of the learning data 33C is rewritten with the soft label obtained by the integrated model 35A, and the new learning data 37A is generated. The label of the learning data 33A is rewritten with the soft label obtained by the integrated model 35B, and the new learning data 37B is generated. The label of the learning data 33B is rewritten with the soft label obtained by the integrated model 35C, and the new learning data 37C is generated.The data generation unit 24 aggregates the new learning data 37A, the new learning data 37B, and the new learning data 37C, and generates a record of the new learning data 37.In step S 15, the second learning unit 25 performs training on the learning model 32 using the dataset of the new learning data 37, and generates the second learning model 38.Training of the learning model 32 is effected here, for example, by deep learning (dt. Depth Learning). Training of the learning model 32 is not limited to deep learning and may be performed by, for example, arithmetic methods such as regression, decision tree learning, Bayesian learning, or clustering.*** Effects of Embodiment 1**As described above, the machine learning device 10 according to Embodiment 1 generates a plurality of first learning models 34 using the learning data 33 obtained by dividing the learning data 31 including privacy information and generates the integrated model 35 by integrating the first learning models 34. then, the machine learning device 10 generates the new learning data 37 by a soft label obtained by the integrated model 35 and generates the second learning model 38 by training the learning model 32 on the new learning data 37. that is, the second learning model 38 is generated by training the learning model 32 with the new learning data 37 from which the privacy information of the original learning data 31 has been removed.Thereby, the machine learning device 10 according to Embodiment 1 can generate the second learning model 38 having resistance to a membership inference attack. That is, the machine learning device 10 can generate the second learning model 38 having resistance to membership inference attack without editing learning data containing no privacy information as in Patent Literature 1 and Non-Patent Literature 1.Further, the machine learning device 10 according to Embodiment 1 generates the first learning model 34 for each of the plurality of learning data 33 obtained by dividing the learning data 31, and generates the integrated model 35 by aggregating the first learning models 34. that is, the machine learning device 10 does not perform additional learning as in Non-Patent Literature 2 but integrates the first learning models 34.Specifically, in order to have resistance to the membership inference attack, the machine learning device 10 requires: (1) additional training once; and (2) average calculation of the parameters of the first learning model 34 and assignment of a soft label, which is easy processing. The one-time training to be additionally performed is the training of n pieces of learning data 33 corresponding to the number of divisions n of the learning data 31. The averaging calculation of the parameters of the first learning model 34 is a calculation in the processing of integrating the first learning models 34.*** Other Configurations ***< 1>In step S 15, the second learning unit 25 may perform training using data obtained by adding the learning data 31 to the record of the new learning data 37 in a reference ratio.It is expected that learning accuracy is thereby improved. However, the higher the ratio of the learning data 31 to the new learning data 37, the lower the resistance of the second learning model 38 to a membership inference attack. Therefore, it is necessary to set the reference ratio in advance according to the required resistance to a membership inference attack.< 2>In Embodiment 1, the individual functional components are realized by software. However, modification 2 is also possible in which the individual functional components are implemented by hardware. Modification 2 will be explained with respect to differences from Embodiment 1.A configuration of a machine learning device 10 according to Embodiment 2 will be described with reference to FIG. 4.When the individual functional components are implemented by hardware, the machine learning device 10 includes an electronic circuit 15 instead of the processor 11, the working memory 12, and the mass storage 13.As the electronic circuit 15, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA may be employed. GA is an abbreviation for Gate Array (dt. Gate Arrangement). ASIC is an abbreviation for Application Specific Integrated Circuit (dt. Application Specific Integrated Circuit). FPGA is an abbreviation for field-programmable gate array (field-programmable gate array).The individual functional components may be realized by a single electronic circuit 15 or may be realized by distribution to a plurality of electronic circuits 15.< 3>A modification 3 is possible in which a part of the functional components are realized by hardware and the remaining functional components are realized by software.The processor 11, the random access memory 12, the mass storage memory 13 and the electronic circuit 15 are referred to as processing circuitry. That is, the functions of the individual functional components are implemented by the processing circuit.Embodiment 2.Embodiment 2 differs from Embodiment 1 in that retraining of the integrated model 35 is performed. In Embodiment 2, this difference will be described and a description of the same aspects will be omitted.*** Description of configuration ***A configuration of a machine learning device 10 according to Embodiment 2 will be described with reference to FIG. 5.The machine learning device 10 is different from the machine learning device 10 illustrated in FIG. 10 in that the machine learning device 10 includes a re-learning unit 26 as a functional component. The re-learning unit 26 is realized by software or hardware, as in the other functional components.*** Description of the Mode of Operation ***A processing flow of a machine learning device 10 according to Embodiment 2 will be described with reference to FIG. 6.The processes from step S 21 to step S 23 are the same as the processes from step S 11 to step S 13 in FIG. 2 ; the processes of steps S 25 and S 26 are the same as the processes of steps S 14 and S 15 in FIG. 2.However, in step S 25, the new learning data 37 is generated using the integrated model 35 that has performed retraining in step S 24.(Step S 24: Re-learning process)The re-learning unit 26 reads each integrated model 35 generated in step S 23 from the work memory 12.The re-learning unit 26 performs re-training of the integrated subject model 35 using the learning data 33 used for training in generating the m pieces of the first learning models 34 that form the basis of the integrated subject model 35. Each first learning model 34 is generated from a piece of learning data 33. Therefore, the re-learning unit 26 performs training on the m pieces of learning data 33 used in the generation of the m pieces of first learning models 34.A concrete example of the operation of the machine learning device 10 according to Embodiment 2 will be described with reference to FIG. 7.FIG. 7 shows an example of the case where n, a division number, 3 and m are n-1, as in the example of FIG. 3.The processes of steps S 21 to S 23 generate three pieces of integrated models 35 of the integrated model 35C from the integrated model 35A as in the example of FIG. 3.In step S 24, the re-learning unit 26 sets each of the three pieces of integrated models 35 as the subject integrated model 35. The data generation unit 24 retravels the integrated subject model 35 using the learning data 33 used for training the two pieces of first learning models 34 that form the basis for the integrated subject model 35.The integrated model 35A is re-trained with the learning data 33A and the learning data 33B used for training the first learning model 34A and the first learning model 34B. In this manner, the integrated model 35A' is produced. The integrated model 35B is re-trained with the learning data 33B and the learning data 33C used for training the first learning model 34B and the first learning model 34C. In this manner, the integrated model 35B' is produced. The integrated model 35C is re-trained with the learning data 33A and the learning data 33C used for training the first learning model 34A and the first learning model 34C. In this manner, the integrated model 35C' is produced.In step S 25, the data generation unit 24 sets each of the three pieces of re-trained integrated models 35 as the subject integrated model 35. That is, the data generation unit 24 sets the integrated model 35A', the integrated model 35B', and the integrated model 35C' as the subject integrated model 35, respectively. Subsequently, the data generation unit 24 generates the new learning data 37 from the integrated subject model 35 as in the example of FIG. 3.In step S 26, the second learning unit 25 performs training on the learning model 32 using the dataset of the new learning data 37, and generates the second learning model 38 as in the example of FIG. 3.*** Effects of Embodiment 2**As explained above, the machine learning device 10 according to Embodiment 2 retravels the integrated model 35. Thereby, it is possible to improve the inference accuracy of the integrated model 35 as compared with that of Embodiment 1. If the inference accuracy of the integrated model 35 is improved, it is possible to assign a soft label to the new learning data 37 with high accuracy. As a result, it is possible to generate the second learning model 38 with high inference accuracy.The term "unit" in the above description may be replaced with "circuit", "step", "sequence", "process", or "processing circuit".The embodiments and modifications of the present disclosure have been described above. Of these embodiments and modifications, some may be implemented by combination. One or some of these embodiments and modifications may alternatively be partially implemented. The present disclosure is not limited to the above-mentioned embodiments and modifications, and various modifications may be made as needed.List of reference characters10: Machine learning device; 11: processor; 12: random access memory; 13: mass storage; 15: electronic circuit; 21: data division unit; 22: first learning unit; 23: model integration unit; 24: data generation unit; 25: second learning unit; 26: re-learning unit; 31: learning data; 32: learning model; 33: learning data; 34: first learning model; 35: integrated model; 36: subject data; 37: new learning data; 38: second learning model.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedJP 2021-193533 A

[0007]

Claims

A machine learning device, comprising: a first learning unit to generate n pieces of first learning models by using each of n pieces of first learning data assigned with a label, wherein an integer n is equal to or greater than 3, as a subject, performing training using subject learning data, and generating the first learning model according to the subject learning data; a model integration unit to generate an integrated model by integrating m pieces of first learning models selected from the n pieces of first learning models generated by the first learning unit, wherein an integer m is less than n; a data generation unit for generating new learning data by rewriting a label assigned to subject data with a soft label that is a result obtained by inputting the subject data as input to the integrated model generated by the model integration unit, the subject data being learning data other than learning data used for training in generating the m pieces of first learning models that are the basis of the integrated model; and a second learning unit for generating a second learning model by performing training using the new learning data generated by the data generation unit.The machine learning device according to claim 1, wherein the model integration unit generates the integrated model for each combination of the m pieces of first learning models that can be selected from the n pieces of first learning models, and the data generation unit generates new learning data by using each integrated model generated by the model generation unit as a subject and rewriting a label assigned to the subject data with a soft label that is a result obtained by inputting the subject data as input to an integrated subject model, the subject data being learning data other than learning data used for training in generating the m pieces of first learning models that are the basis of the integrated subject model.The machine learning device according to claim 1 or 2, wherein the integer m is n-1.The machine learning device according to any one of claims 1 to 3, wherein the second learning unit generates the second learning model by performing training using data to which the learning data having a reference ratio to the new learning data is added.The machine learning device according to any one of claims 1 to 4, further comprising: a re-learning unit to perform re-training of the integrated model using learning data used for training in generating the m pieces of first learning models that form the basis of the integrated model, wherein the data generation unit generates the new learning data using the integrated model for which the re-learning unit performs re-training.A machine learning method, comprising: by a computer, generating n pieces of first learning models by using each of n pieces of first learning data to which a label is assigned, wherein an integer n is equal to or greater than 3, as a subject, performing training using subject learning data, and generating the first learning model according to the subject learning data; by the computer, generating an integrated model by integrating m pieces of first learning models selected from the n pieces of first learning models, wherein an integer m is less than n; by the computer, generating new learning data by rewriting a label assigned to the subject data with a soft label that is a result obtained by inputting the subject data as input to the integrated model, the subject data being learning data other than learning data used for training in generating the m pieces of first learning models that are the basis of the integrated model; and by the computer, generating a second learning model by performing training using the new learning data.A machine learning program for causing a computer to function as a machine learning device to execute: a first learning process to generate n pieces of first learning models by using each of n pieces of first learning data assigned with a label, an integer n being equal to or greater than 3, as a subject, performing training using subject learning data, and generating the first learning model corresponding to the subject learning data; a model integration process to generate an integrated model by integrating m pieces of first learning models selected from the n pieces of first learning models generated by the first learning process, an integer m being less than n; a data generation process for generating new learning data by rewriting a label assigned to the subject data with a soft label that is a result obtained by inputting the subject data as input to the integrated model generated by the model integration process, the subject data being learning data other than learning data used for training in generating the m pieces of the first learning models that are the basis of the integrated model; and a second learning process for generating a second learning model by performing training using the new learning data generated by the data generation process.

Images