Machine learning device, machine learning method, and machine learning program

Abstract

The first learning unit (22) generates a first learning model by training the n pieces of learning data as objects using the object learning data. A model integration unit (23) integrates m first learning models selected from among the n first learning models to generate an integrated model. A data generation unit (24) generates new learning data by replacing a tag added to the target data with a soft tag that is a result of providing the target data as an input to the integrated model. The object data is learning data other than learning data used in training when generating m first learning models which are the basis of the integrated model. A second learning unit (25) performs training using the new learning data and generates a second learning model.

Description

Technical Field

[0001] The present disclosure relates to a method for training a machine learning model. Background Art

[0002] As a privacy issue in machine learning, the existence of member inference attacks is pointed out. Member inference attacks are attacks that provide normal input data to the machine learning model (hereinafter referred to as the target model) that is the target of the attack, and observe the inference results of the target model response. In this way, it is determined whether the input data is included in the learning data of the target model (= whether it is a member).

[0003] Patent Document 1 and Non-Patent Documents 1 and 2 describe countermeasures against member inference attacks.

[0004] In Patent Document 1, learning data is divided into learning data containing private information and learning data not containing private information. The learning data containing private information is used to train only the input layer of the machine learning model. The learning data not containing private information is used to retrain all layers of the machine learning model. The retrained machine learning model has resistance to member inference attacks.

[0005] In non-patent document 1, a machine learning model trained with learning data containing private information is used to assign labels (hereinafter referred to as soft labels) to unlabeled learning data that does not contain private information. The learning data that does not contain private information is, for example, public data. Other machine learning models are trained with the learning data to which the soft labels are assigned. The trained machine learning model has resistance to member inference attacks.

[0006] In non-patent document 2, the initial learning data is divided into a constant number of n sets. For each of the n sets, n-1 sets other than the set are set as learning data. That is, n learning data having n-1 sets are set. The n learning data are used for training to generate machine learning models corresponding to the n learning data. The n machine learning models are taken as objects, and sets not included in the learning data used in the training of the machine learning models are provided as input to the machine learning models as objects to obtain soft labels. The labels of the sets provided as input are replaced with soft labels to become new learning data. Then, the machine learning model is trained using the new learning data. The trained machine learning model has resistance to member inference attacks.

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1: Japanese Patent Application Publication No. 2021-193533

[0010] Non-patent literature

[0011] Non-patent literature 1: Virat Shejwalkar et al, "Membership Privacy for Machine Learning Models Through Knowledge Transfer", AAAI2021

[0012] Non-patent literature 2: Rishav Chourasia et al, “Knowledge Cross-Distillation for Membership Privacy”, PETS2022 Summary of the invention

[0013] Problems to be solved by the invention

[0014] The measures described in Patent Document 1 and Non-Patent Document 1 require learning data that does not contain private information. In fields such as medicine and finance, sensitive data may be used in machine learning, and therefore it is difficult to prepare learning data that does not contain private information.

[0015] The countermeasure described in non-patent document 2 does not require learning data that does not contain private information. However, the countermeasure described in non-patent document 2 requires additional training of n×(n-1) learning data according to the number of divisions n of the learning data. Therefore, the amount of calculation based on the countermeasure is large compared with other existing countermeasures.

[0016] The present disclosure aims to eliminate the need for learning data that does not contain private information and to suppress the amount of calculation so as to provide resistance to member inference attacks.

[0017] Means for solving problems

[0018] The machine learning device disclosed in the present invention comprises: a first learning unit, which, with respect to an integer n greater than 3, takes n learning data to which labels are attached as objects, respectively, and uses the object learning data for training to generate a first learning model corresponding to the object learning data, thereby generating the n first learning models; a model integration unit, which, with respect to an integer m less than n, integrates m first learning models selected from the n first learning models generated by the first learning unit to generate an integrated model; a data generation unit, which replaces a label attached to the object data with a soft label to generate new learning data, wherein the soft label is a result of providing the object data as an input to the integrated model generated by the model integration unit, and the object data is learning data other than the learning data used in the training when generating the m first learning models that serve as the basis of the integrated model; and a second learning unit, which uses the new learning data generated by the data generation unit for training to generate the second learning model.

[0019] Effects of the Invention

[0020] In the present disclosure, the first learning model generated by training using the object learning data is integrated to generate an integrated model, and new learning data is generated by the integrated model. This eliminates the need for learning data that does not contain private information, reduces the amount of calculation, and provides resistance to member inference attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a configuration diagram of the machine learning device 10 according to the first embodiment.

[0022] Figure 2 This is a flowchart showing the flow of processing performed by the machine learning device 10 according to the first embodiment.

[0023] Figure 3 This is an explanatory diagram of a specific example of the operation of the machine learning device 10 according to the first embodiment.

[0024] Figure 4 This is a structural diagram of the machine learning device 10 according to the second modification.

[0025] Figure 5 This is a structural diagram of the machine learning device 10 according to the second embodiment.

[0026] Figure 6 This is a flowchart showing the flow of processing by the machine learning device 10 according to the second embodiment.

[0027] Figure 7 This is an explanatory diagram of a specific example of the operation of the machine learning device 10 according to the second embodiment. DETAILED DESCRIPTION

[0028] Implementation Method 1

[0029] ***Description of the structure***

[0030] Reference Figure 1 The structure of the machine learning device 10 according to the first embodiment will be described.

[0031] The machine learning device 10 is a computer.

[0032] The machine learning device 10 includes hardware such as a processor 11, a memory 12, and a storage 13. The processor 11 is connected to other hardware via a signal line and controls the other hardware.

[0033] The processor 11 is an IC that performs processing. IC is the abbreviation of Integrated Circuit. As a specific example, the processor 11 is a CPU, a DSP, or a GPU. CPU is the abbreviation of Central Processing Unit. DSP is the abbreviation of Digital Signal Processor. GPU is the abbreviation of Graphics Processing Unit.

[0034] The memory 12 is a storage device for temporarily storing data. As a specific example, the memory 12 is SRAM or DRAM. SRAM is the abbreviation of Static Random Access Memory. DRAM is the abbreviation of Dynamic Random Access Memory.

[0035] The memory 13 is a storage device for storing data. As a specific example, the memory 13 is an HDD. HDD is the abbreviation of Hard Disk Drive. In addition, the memory 13 can also be a mobile recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash memory, floppy disk, optical disk, high-density disk, Blu-ray (registered trademark) disk, DVD. SD is the abbreviation of Secure Digital. DVD is the abbreviation of Digital Versatile Disk.

[0036] The machine learning device 10 includes, as functional components, a data segmentation unit 21, a first learning unit 22, a model integration unit 23, a data generation unit 24, and a second learning unit 25. The functions of the functional components of the machine learning device 10 are implemented by software.

[0037] The memory 13 stores a program for realizing the functions of each functional component of the machine learning device 10. The program is read into the memory 12 by the processor 11 and executed by the processor 11. Thus, the functions of each functional component of the machine learning device 10 are realized.

[0038] The memory 13 stores a learning model 32 and a plurality of learning data 31. Each learning data 31 is labeled and contains private information.

[0039] exist Figure 1 In FIG. 1 , only one processor 11 is shown. However, there may be multiple processors 11, and the multiple processors 11 may cooperate to execute programs that realize each function.

[0040] ***Description of the action***

[0041] Reference Figure 2 and Figure 3 The operation of the machine learning device 10 according to the first embodiment will be described.

[0042] The operation procedure of the machine learning device 10 in Embodiment 1 corresponds to the machine learning method in Embodiment 1. In addition, a program that realizes the operation of the machine learning device 10 in Embodiment 1 corresponds to the machine learning program in Embodiment 1.

[0043] Reference Figure 2 The flow of processing performed by the machine learning device 10 according to the first embodiment will be described.

[0044] (Step S11: Data division processing)

[0045] The data segmentation unit 21 reads the plurality of learning data 31 stored in the storage 13 into the memory 12. The data segmentation unit 21 divides the plurality of learning data 31 read into a constant number of n sets. n is an integer greater than 3. For example, the data segmentation unit 21 divides the plurality of learning data 31 into equal parts so that the number of learning data 31 contained in each set is approximately equal. Thus, a data set of n learning data (hereinafter referred to as learning data 33) is generated. The data segmentation unit 21 writes the n learning data 33 into the memory 12.

[0046] (Step S12: First Learning Process)

[0047] The first learning unit 22 reads the learning model 32 and the n learning data 33 generated in step S11 from the memory 12. The first learning unit 22 sets the n learning data 33 as the target learning data 33. The first learning unit 22 trains the learning model 32 using the target learning data 33, and generates a first learning model 34 corresponding to the target learning data 33. Thus, n first learning models 34 are generated. The first learning unit 22 writes the n first learning models 34 into the memory 12.

[0048] (Step S13: Model integration processing)

[0049] The model integration unit 23 reads the n first learning models 34 generated in step S12 from the memory 12. The model integration unit 23 integrates the m first learning models 34 selected from the n first learning models 34 to generate an integrated model 35. m is an integer less than n. Here, the model integration unit 23 generates an integrated model 35 for each combination of the m first learning models 34 that can be selected from the n first learning models. The model integration unit 23 writes the integrated model 35 for each combination into the memory 12.

[0050] The model integration unit 23 totals the parameters of the m first learning models 34 and performs arithmetic processing such as arithmetic averaging or weighted averaging for each parameter. Thus, the model integration unit 23 integrates the m first learning models 34 to generate an integrated model 35 .

[0051] In the first embodiment, m=n-1. There are n combinations of n-1 first learning models 34 selected from the n first learning models 34. That is, there are n combinations of n-1 first learning models 34 except the first first learning model 34, n-1 first learning models 34 except the second first learning model 34, ... n-1 first learning models 34 except the nth first learning model 34.

[0052] Therefore, the model integration unit 23 integrates the first learning models 34 of each of the n combinations to generate an integrated model 35. As a result, n integrated models 35 are generated.

[0053] (Step S14: Data generation processing)

[0054] The data generation unit 24 reads each integrated model 35 generated in step S13 from the memory 12. The data generation unit 24 sets each integrated model 35 as a target integrated model 35.

[0055] The data generation unit 24 provides the object data 36 as input to the object integration model 35 for inference. The object data 36 is learning data 33 other than the learning data 33 used in training when generating the m first learning models that are the basis of the object integration model 35. The data generation unit 24 obtains the soft label, which is the result of the inference by the object integration model 35. The data generation unit 24 replaces the label attached to the object data 36 with the soft label to generate new learning data 37. The data generation unit 24 collects the new learning data 37 generated for each integration model 35 and writes it into the memory 12 as a data set of the new learning data 37.

[0056] In the first embodiment, the data generation unit 24 reads n integrated models 35 . The data generation unit 24 sets each of the n integrated models 35 as a target integrated model 35 .

[0057] The data generation unit 24 provides the object data 36 as input to the object integration model 35 for inference, and the object data 36 is learning data 33 other than the learning data 33 used in training when the n-1 first learning models that are the basis of the object integration model 35 are generated. For example, when the object integration model 35 is generated based on the combination of the remaining n-1 first learning models 34 other than the first first learning model 34, the first first learning model 34 becomes the object data 36. Similarly, when the object integration model 35 is generated based on the combination of the remaining n-1 first learning models 34 other than the second first learning model 34, the second first learning model 34 becomes the object data 36. The data generation unit 24 replaces the label attached to the object data 36 with the soft label, and generates new learning data 37.

[0058] The data generating unit 24 collects the new learning data 37 generated for each of the n integrated models 35 , and writes the data into the memory 12 as a data set of the new learning data 37 .

[0059] (Step S15: Second Learning Process)

[0060] The second learning unit 25 reads the learning model 32 and the data set of the new learning data 37 generated in step S14 from the memory 12. The second learning unit 25 trains the learning model 32 using the data set of the new learning data 37, and generates a second learning model 38.

[0061] Reference Figure 3 A specific example of the operation of the machine learning device 10 according to the first embodiment will be described.

[0062] exist Figure 3 , an example is shown in which the number of divisions, that is, n is 3 and m is n-1.

[0063] In step S11 , the data dividing unit 21 divides the learning data 31 including the private information into three (=n) equal parts.

[0064] Thus, learning data 33A, learning data 33B, and learning data 33C are generated.

[0065] In step S12 , the first learning unit 22 sets the three learning data 33 as the target learning data 33 . The first learning unit 22 trains the learning model 32 using the target learning data 33 , and generates a first learning model 34 corresponding to the target learning data 33 .

[0066] Thus, three learning data 33 are generated: a first learning model 34A trained with the learning data 33A, a first learning model 34B trained with the learning data 33B, and a first learning model 34C trained with the learning data 33C.

[0067] In step S13 , the model integration unit 23 sets each combination of two (=m=n-1) first learning models 34 selectable from the three first learning models 34 as a target combination. The model integration unit 23 integrates the two first learning models 34 included in the target combination to generate an integrated model 35 .

[0068] Thus, three integrated models 35 are generated: an integrated model 35A that integrates the first learning model 34A and the first learning model 34B, an integrated model 35B that integrates the first learning model 34B and the first learning model 34C, and an integrated model 35C that integrates the first learning model 34A and the first learning model 34C.

[0069] In step S14, the data generation unit 24 sets the three integrated models 35 as the target integrated models 35, respectively. The data generation unit 24 provides the target integrated model 35 with the target data 36 as input, which is the learning data 33 not used in the training of the two first learning models 34 that are the basis of the target integrated model 35. The integrated model 35A is provided with the learning data 33C not used in the training of the first learning model 34A and the first learning model 34B as input. The integrated model 35B is provided with the learning data 33A not used in the training of the first learning model 34B and the first learning model 34C as input. The integrated model 35C is provided with the learning data 33B not used in the training of the first learning model 34A and the first learning model 34C as input.

[0070] The data generation unit 24 generates new learning data 37 by replacing the label added to the object data 36 with a soft label, which is a result of inference by the object integrated model 35. That is, the label of the learning data 33C is replaced by the soft label obtained by the integrated model 35A, and the new learning data 37A is generated. The label of the learning data 33A is replaced by the soft label obtained by the integrated model 35B, and the new learning data 37B is generated. The label of the learning data 33B is replaced by the soft label obtained by the integrated model 35C, and the new learning data 37C is generated.

[0071] The data generating unit 24 collects the new learning data 37A, the new learning data 37B, and the new learning data 37C to generate a data set of the new learning data 37 .

[0072] In step S15 , the second learning unit 25 trains the learning model 32 using the data set of the new learning data 37 , and generates the second learning model 38 .

[0073] Here, the training of the learning model 32 is performed by deep learning, for example. In addition, the training of the learning model 32 is not limited to deep learning, and may be performed by calculations such as regression, decision tree learning, Bayesian method, clustering, etc., for example.

[0074] ***Effects of Implementation Method 1***

[0075] As described above, the machine learning device 10 of the first embodiment generates a plurality of first learning models 34 using the learning data 33 obtained by segmenting the learning data 31 containing the private information, and integrates the first learning models 34 to generate an integrated model 35. Furthermore, the machine learning device 10 generates new learning data 37 using the soft labels obtained from the integrated model 35, trains the learning model 32 using the new learning data 37, and generates a second learning model 38. That is, the learning model 32 is trained using the new learning data 37 from which the private information of the original learning data 31 is removed, and the second learning model 38 is generated.

[0076] Thus, the machine learning device 10 of Embodiment 1 can generate the second learning model 38 that is resistant to member inference attacks. That is, the machine learning device 10 can generate the second learning model 38 that is resistant to member inference attacks without preparing learning data that does not contain private information as in Patent Document 1 and Non-Patent Document 1.

[0077] Furthermore, the machine learning device 10 of the first embodiment generates first learning models 34 for each of the plurality of learning data 33 obtained by dividing the learning data 31, and integrates the first learning models 34 to generate an integrated model 35. That is, the machine learning device 10 does not perform additional learning as in the non-patent document 2, but integrates the first learning models 34. Thus, compared with the technique of the non-patent document 2, the amount of calculation can be suppressed, and the second learning model 38 having resistance to member inference attacks can be generated.

[0078] Specifically, in order to be resistant to member inference attacks, the machine learning device 10 needs to (1) perform one additional training and (2) perform average calculation of parameters of the first learning model 34 and assign soft labels as lightweight processing. The additional training is training of n learning data 33 corresponding to the number of divisions n of the learning data 31. The average calculation of parameters of the first learning model 34 is a calculation in the integrated processing of the first learning model 34.

[0079] ***Other structures***

[0080] <Modification 1>

[0081] In step S15 , the second learning unit 25 may perform training using data obtained by adding the learning data 31 at the reference ratio to the data set of the new learning data 37 .

[0082] This can improve the learning accuracy. However, the higher the ratio of the learning data 31 to the new learning data 37, the lower the member-estimated attack resistance of the second learning model 38. Therefore, it is necessary to set a reference ratio in advance according to the required member-estimated attack resistance.

[0083] <Modification 2>

[0084] In the first embodiment, each functional component is implemented by software. However, as a second modification, each functional component may be implemented by hardware. In this second modification, the differences from the first embodiment are described.

[0085] Reference Figure 4 The structure of the machine learning device 10 according to the second modification will be described.

[0086] When each functional component is realized by hardware, the machine learning device 10 includes an electronic circuit 15 instead of the processor 11, the memory 12, and the storage 13. The electronic circuit 15 is a dedicated circuit that realizes the functions of each functional component, the memory 12, and the storage 13.

[0087] As the electronic circuit 15, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, and an FPGA are envisioned. GA is the abbreviation of Gate Array. ASIC is the abbreviation of Application Specific Integrated Circuit. FPGA is the abbreviation of Field-Programmable Gate Array.

[0088] Each functional component may be realized by one electronic circuit 15 , or may be realized by distributing multiple electronic circuits 15 .

[0089] <Variation 3>

[0090] As a third modification, some of the functional components may be implemented by hardware, and the other functional components may be implemented by software.

[0091] The processor 11, the memory 12, the storage 13, and the electronic circuit 15 are referred to as a processing circuit. That is, the functions of each functional component are realized by the processing circuit.

[0092] Implementation Method 2

[0093] The difference between the second embodiment and the first embodiment is that the integrated model 35 is retrained. In the second embodiment, the difference is described, and the description of the same points is omitted.

[0094] ***Description of the structure***

[0095] Reference Figure 5 The structure of the machine learning device 10 according to the second embodiment will be described.

[0096] Machine learning device 10 and Figure 1 The machine learning device 10 shown is different in that it includes a relearning unit 26 as a functional component. The relearning unit 26 is realized by software or hardware like other functional components.

[0097] ***Description of the action***

[0098] Reference Figure 6 The flow of processing performed by the machine learning device 10 according to the second embodiment will be described.

[0099] The processing of steps S21 to S23 is Figure 2 The processing of steps S11 to S13 is the same as that of steps S25 and S26. Figure 2 The processing of step S14 and step S15 is the same.

[0100] However, in step S25 , new learning data 37 is generated using the integrated model 35 that has been retrained in step S24 .

[0101] (Step S24: Relearning Process)

[0102] The re-learning unit 26 reads each integrated model 35 generated in step S23 from the memory 12. The re-learning unit 26 sets each integrated model 35 as a target integrated model 35.

[0103] The re-learning unit 26 retrains the object integrated model 35 using the learning data 33 used in the training when the m first learning models 34 serving as the basis of the object integrated model 35 were generated. Each first learning model 34 is generated using one learning data 33. Therefore, the re-learning unit 26 performs training using the m learning data 33 used when the m first learning models 34 were generated.

[0104] Reference Figure 7 A specific example of the operation of the machine learning device 10 according to the second embodiment will be described.

[0105] exist Figure 7 In, with Figure 3 Similarly to the example, an example in which the number of divisions, that is, n is 3 and m is n-1 is shown.

[0106] Through the processing of steps S21 to S23, Figure 3 In the same manner as in the example of , three integrated models 35A to 35C are generated.

[0107] In step S24, the relearning unit 26 sets the three integrated models 35 as the target integrated model 35. The data generating unit 24 retrains the target integrated model 35 using the learning data 33 used in training the two first learning models 34 serving as the basis of the target integrated model 35.

[0108] The integrated model 35A is retrained using the learning data 33A and the learning data 33B used in the training of the first learning model 34A and the first learning model 34B. Thus, an integrated model 35A' is generated. The integrated model 35B is retrained using the learning data 33B and the learning data 33C used in the training of the first learning model 34B and the first learning model 34C. Thus, an integrated model 35B' is generated. The integrated model 35C is retrained using the learning data 33A and the learning data 33C used in the training of the first learning model 34A and the first learning model 34C. Thus, an integrated model 35C' is generated.

[0109] In step S25, the data generation unit 24 sets the three integrated models 35 after retraining as the target integrated models 35. That is, the data generation unit 24 sets the integrated model 35A', the integrated model 35B' and the integrated model 35C' as the target integrated models 35. Then, Figure 3 Similarly to the example, the data generation unit 24 generates new learning data 37 using the object integration model 35 .

[0110] In step S26, Figure 3 Similarly to the example, the second learning unit 25 uses the data set of new learning data 37 to train the learning model 32 and generate a second learning model 38.

[0111] ***Effects of Implementation Method 2***

[0112] As described above, the machine learning device 10 of the second embodiment retrains the integrated model 35. As a result, the accuracy of the inference of the integrated model 35 can be improved compared to the first embodiment. When the accuracy of the inference of the integrated model 35 is improved, soft labels can be added to the new learning data 37 with high accuracy. As a result, the second learning model 38 with high inference accuracy can be generated.

[0113] In addition, the word "unit" in the above description may be rewritten as "circuit", "process", "step", "processing" or "processing circuit".

[0114] The above describes the embodiments and variations of the present disclosure. It is also possible to implement several of these embodiments and variations in combination. In addition, it is also possible to implement a portion of any one or several embodiments and variations. In addition, the present disclosure is not limited to the above embodiments and variations, and various changes can be made as needed.

[0115] Description of symbols

[0116] 10: machine learning device; 11: processor; 12: memory; 13: storage; 15: electronic circuit; 21: data segmentation unit; 22: first learning unit; 23: model integration unit; 24: data generation unit; 25: second learning unit; 26: re-learning unit; 31: learning data; 32: learning model; 33: learning data; 34: first learning model; 35: integrated model; 36: object data; 37: new learning data; 38: second learning model.

Claims

1. A machine learning device, comprising: A first learning unit that generates the n first learning models by training the n learning data to which the labels are respectively added, with respect to an integer n greater than 3, and generating a first learning model corresponding to the object learning data; a model integration unit that integrates m first learning models selected from the n first learning models generated by the first learning unit, with respect to an integer m less than n, to generate an integrated model; a data generating unit that generates new learning data by replacing a label attached to the object data with a soft label, the soft label being a result of providing the object data as an input to the integrated model generated by the model integrating unit, the object data being learning data other than the learning data used in training when generating the m first learning models that are the basis of the integrated model; as well as The second learning unit generates a second learning model by performing training using the new learning data generated by the data generating unit.

2. The machine learning device according to claim 1, in, The model integration unit generates the integrated model for each combination of m first learning models selectable from the n first learning models. The data generation unit generates new learning data by replacing the label attached to the object data through a soft label, wherein the soft label is the result of taking each integrated model generated by the model integration unit as an object and providing the object data as input to the object integrated model, and the object data is learning data other than the learning data used in training when generating the m first learning models that serve as the basis of the object integrated model.

3. The machine learning device according to claim 1 or 2, in, The integer m is n-1.

4. The machine learning device according to any one of claims 1 to 3, in, The second learning unit generates the second learning model by performing training using data obtained by adding the learning data at a reference ratio to the new learning data.

5. The machine learning device according to any one of claims 1 to 4, in, The machine learning device further includes a re-learning unit that retrains the integrated model using learning data used in training when generating m first learning models that are the basis of the integrated model. The data generating unit generates the new learning data using the integrated model retrained by the relearning unit.

6. A machine learning method, in, The computer generates the n first learning models by using the object learning data for training, with respect to an integer n greater than 3, n learning data to which labels are respectively attached as objects, and generating a first learning model corresponding to the object learning data, thereby generating the n first learning models. The computer integrates m first learning models selected from the n first learning models for an integer m less than n to generate an integrated model, The computer generates new learning data by replacing a label attached to the object data with a soft label, wherein the soft label is a result of providing the object data as an input to the integrated model, and the object data is learning data other than the learning data used in training when generating m first learning models that are the basis of the integrated model. The computer performs training using the new learning data to generate a second learning model.

7. A machine learning program that causes a computer to function as a machine learning device that performs the following processing: A first learning process, with respect to an integer n greater than 3, takes n learning data to which labels are added as objects, performs training using the object learning data, generates a first learning model corresponding to the object learning data, thereby generating the n first learning models; A model integration process of integrating m first learning models selected from the n first learning models generated by the first learning process, with respect to an integer m less than n, to generate an integrated model; A data generation process for generating new learning data by replacing a label attached to the object data with a soft label, the soft label being a result of providing the object data as an input to the integrated model generated by the model integration process, the object data being learning data other than the learning data used in training when generating the m first learning models that are the basis of the integrated model; as well as The second learning process performs training using the new learning data generated by the data generation process to generate a second learning model.

Images