Synthesis device, synthesis method, and synthesis program
The synthesis device addresses the challenge of using models trained on different datasets by determining the flatness of the weight loss function and averaging model weights, resulting in a composite model that operates effectively across both datasets, enhancing accuracy and efficiency.
Patent Information
- Application Number
- PCT/JP2023/017642
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-14
Smart Images

Figure JP2023017642_14112024_PF_FP_ABST
Abstract
Description
Synthesis device, synthesis method, and synthesis program
[0001] The present invention relates to a synthesis device, a synthesis method, and a synthesis program.
[0002] Conventionally, when dataset A and dataset B are acquired independently, methods are known for generating a model that performs well with dataset A and dataset B. For example, continuous learning (see Non-Patent Document 1) is known for expanding the scope of application of data, federated learning (see Non-Patent Document 2) is known for learning while distributing data in consideration of privacy, and a method for synthesizing the same data (see Non-Patent Document 3) is known.
[0003] “Introduction to Continual Learning”, [online], ∞ Wiki, [Retrieved April 14, 2023], Internet <URL: https: / / wiki.continualai.org / the-continualai-wiki / introduction-to-continual-learning> “Federated learning”, [online], WIKIPEDIA, [Retrieved April 14, 2023], Internet <URL: https: / / en.wikipedia.org / wiki / Federated_learning> Samuel K. Ainsworth, Jonathan Hayase, Siddhartha Srinivasa, “GIT RE-BASIN: MERGING MODELS MODULO PERMUTATION SYMMETRIES”, 2023
[0004] However, with conventional technology, it may be difficult to generate a model that can be used with both dataset A and dataset B from model A trained with dataset A and model B trained with dataset B. For example, with conventional technology, it is not possible to independently perform training using dataset A and training using dataset B, which creates a problem of operational constraints.
[0005] The present invention has been made in consideration of the above, and aims to generate a model that can be used with both dataset A and dataset B from model A trained with dataset A and model B trained with dataset B.
[0006] In order to solve the above-mentioned problems and achieve the object, the synthesis device of the present invention is characterized by having a learning unit that uses first training data to learn a first model based on the flatness of the gradient of a loss function of the weight of the first model with respect to input data of the first model, and that uses second training data to learn a second model based on the flatness of the gradient of a loss function of the weight of the second model, and an identification unit that uses the weight of the first model and the weight of the second model to identify the weight of a synthesized model obtained by synthesizing the first model and the second model.
[0007] According to the present invention, it is possible to generate a model that can be used with both dataset A and dataset B from model A trained with dataset A and model B trained with dataset B.
[0008] FIG. 1 is a diagram for explaining an overview of a synthesis device. FIG. 2 is a diagram for explaining an overview of a synthesis device. FIG. 3 is a diagram for explaining an overview of a synthesis device. FIG. 4 is a schematic diagram illustrating an example of a synthesis device. FIG. 5 is a flowchart showing a synthesis processing procedure. FIG. 6 is a diagram for explaining an example. FIG. 7 is a diagram for explaining an example. FIG. 8 is a diagram illustrating a computer that executes a synthesis program.
[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.
[0010] [Outline of the Synthesis Device] The synthesis device of this embodiment generates a synthesis model that can be used with both dataset A and dataset B, using model A trained with dataset A and model B trained with dataset B.
[0011] First, a composite data set D of data set A and data set B is defined as in the following equation (1).
[0012]
[0013] In supervised learning, L is the loss function, w is the weight of the model input data, and the weight of model A is A , weight of Model B B , weight of the synthetic model trained on the synthetic dataset AB are expressed by the following equations (2), (3), and (4), respectively.
[0014]
[0015] Then, if we can identify the operation ★ that satisfies the following equation (5), it will be possible to learn dataset A and dataset B separately, and combine the learned weights to generate a composite model that can be used for datasets A + B.
[0016]
[0017] In Deep Learning, there is symmetry in the operation of the network structure. Therefore, the synthesis device utilizes the symmetry in the rearrangement of weights to perform permutation, which rearranges the weights so as not to change the output of the model.
[0018] In this case, the following equation (6) holds true.
[0019]
[0020] By replacing the above formula (6) with the rule shown in the following formula (7), the following formula (8) is realized.
[0021]
[0022]
[0023] Then, as will be described below, the synthesis device generates a synthesis model by specifying the average of the weight of model A and the rearranged weight of model B as the weight of the synthesis model.
[0024] For example, a method called weight matching is known that allows high-speed rearrangement without using data. Weight matching optimizes the following equation (9).
[0025]
[0026] In this case, it is known that model synthesis may or may not be successful depending on the gradient of the weight loss function. Here, FIGS. 1 to 3 are diagrams for explaining an overview of the synthesis device. For example, FIG. 1 illustrates an example of failure, and FIG. 2 illustrates an example of success. Furthermore, FIGS. 1(a) and 2(a) show the relationship between λ and the loss function, and FIGS. 1(b) and 2(b) show the relationship between λ and accuracy.
[0027] Here, the weight of the composite model is expressed by the following formula (10): In this formula (10), λ=0 represents the optimal weight for data set B, and λ=1 represents the optimal weight for data set A.
[0028]
[0029] In the successful example shown in Figure 2(a), the gradient of the loss function of each model is flat, resulting in a small loss function for the composite model, and therefore a high accuracy of the composite function, as shown in Figure 2(b).
[0030] Therefore, the synthesis device performs weight matching using a gradient method so that the gradient of the loss function of the weight of each model before synthesis becomes flatter. For example, the loss function is optimized so that the size of the double-headed arrow in Figure 3 becomes smaller. Specifically, the loss function is optimized by reducing the underlined part in the following equation (11) to induce it to be flat.
[0031]
[0032] The underlined portion of the above formula corresponds to the double-headed arrow in Fig. 3. By subsequently combining the two models, the combining device can generate a highly accurate combined model.
[0033] [Configuration of the Synthesis Device] Fig. 4 is a schematic diagram illustrating the general configuration of the synthesis device. As illustrated in Fig. 4, the synthesis device 10 is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.
[0034] The input unit 11 is realized by input devices such as a keyboard and a mouse, and inputs various instruction information such as a command to start processing to the control unit 15 in response to input operations by an operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, or the like.
[0035] The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between an external device such as a server via a network and the control unit 15. For example, the communication control unit 13 controls communication between the control unit 15 and a management device or the like that manages various information such as datasets to be subjected to synthesis processing, which will be described later, and trained models.
[0036] The storage unit 14 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores in advance the processing program that operates the synthesis device 10 and data used during execution of the processing program, or temporarily stores the data each time processing is performed. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.
[0037] The control unit 15 is realized using a CPU (Central Processing Unit), an NP (Network Processor), an FPGA (Field Programmable Gate Array), or the like, and executes a processing program stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, a learning unit 15b, and an identification unit 15c, as exemplified in FIG. 4, and executes the synthesis determination process. Note that these functional units may each be implemented in different hardware. The control unit 15 may also include other functional units.
[0038] The acquiring unit 15a acquires first training data and second training data. For example, the acquiring unit 15a acquires a data set A used in a combining process described later and a data set B collected thereafter.
[0039] The acquiring unit 15a acquires these pieces of data via the input unit 11 or from a management device that manages various pieces of information via the communication control unit 13. The acquiring unit 15a may also store the acquired data in the storage unit 14. The acquiring unit 15a may also transfer the information to the learning unit 15b described below without storing it in the storage unit 14.
[0040] The learning unit 15b uses first learning data to learn a first model based on the flatness of the gradient of a loss function of weight for input data of the first model, and also uses second learning data to learn a second model based on the flatness of the gradient of a loss function of weight for the second model.
[0041] For example, the learning unit 15b learns a model A using a data set A and puts it into operation. The learning unit 15b also learns a model B using a data set B collected thereafter. At this time, the learning unit 15b optimizes and learns each of the models A and B so that the underlined part of the above formula (11) becomes smaller.
[0042] The specifying unit 15c specifies the weight of a combined model obtained by combining the first model and the second model, using the weight of the first model for the input data and the weight of the second model.
[0043] Specifically, the specification unit 15c specifies the weight of the combined model by rearranging the weights without changing the output of the second model and averaging the rearranged weights and the weights of the first model.
[0044] For example, the specification unit 15c performs the optimization of the above formula (9) to perform the sorting at high speed by weight matching without using data.
[0045] In this way, the specifying unit 15c generates a composite model by specifying the weight of the composite model using the weight of each model.
[0046] The synthesis process of this embodiment can be applied to, for example, general AI. In particular, it has high affinity with image recognition, natural language processing, voice recognition, etc., which are areas in which Deep Learning excels, and can be used in facial recognition systems, etc. For example, if company A merges with company B after generating facial recognition model A for employees of company A, it is possible to generate a synthetic model that can be used by both companies A and B so that employees of company B can also use the facial recognition system.
[0047] [Combining Process] Next, the combining process performed by the combining device 10 according to this embodiment will be described with reference to Fig. 5. The combining process according to this embodiment includes a detection process and a search process. The flowchart in Fig. 5 starts, for example, when an operation input instructing the start of the combining process is made.
[0048] First, the acquisition unit 15a acquires a data set A. Then, the learning unit 15b uses the data set A to learn the model A based on the flatness of the gradient of the weight loss function for the input data of the model A (step S1). Specifically, the learning unit 15b optimizes and learns the weight loss function of the model A so that the underlined part of the above formula (11) becomes small and flat.
[0049] The acquiring unit 15a also acquires a dataset B collected thereafter. The learning unit 15b then uses dataset B to learn model B based on the flatness of the gradient of the weight loss function for the input data of model B (step S2). Specifically, similar to the learning of model A, the learning unit 15b optimizes and learns the weight loss function of model B so that the underlined part in the above formula (11) becomes smaller and flatter.
[0050] Then, the specifying unit 15c specifies the weight of a combined model obtained by combining the model A and the model B, using the weight of the model A and the weight of the model B (step S3).
[0051] Specifically, the specification unit 15c specifies the weight of the composite model by rearranging the weights without changing the output of model B and averaging the rearranged weights and the weight of model A. In this way, the composite model is generated.
[0052] Furthermore, the specifying unit 15c outputs the generated composite model to the operation device via, for example, the output unit 12. This completes the series of composite processing.
[0053] [Effects] As described above, the learning unit 15b uses first learning data to learn a first model based on the flatness of the gradient of a loss function of the weight of the first model with respect to input data, and uses second learning data to learn a second model based on the flatness of the gradient of a loss function of the weight of the second model. The identification unit 15c uses the weight of the first model and the weight of the second model to identify the weight of a combined model obtained by combining the first model and the second model.
[0054] Specifically, the specification unit 15c specifies the weight of the combined model by rearranging the weights without changing the output of the second model and averaging the rearranged weights and the weights of the first model.
[0055] This makes it possible to generate a model that can be used with both dataset A and dataset B from model A trained with dataset A and model B trained with dataset B.
[0056] 6 and 7 are diagrams for explaining an example. In this example, a composite model was generated by the synthesis process of the above embodiment using model A trained with MNIST and model B trained with FashionMNIST. The model was MLP, and the datasets used were MNIST and FashionMNIST. Source code created with reference to "https: / / github.com / samuela / git-re-basin" was also used. The accuracy and loss of the composite model using a synthetic dataset combining both MNIST and FashionMNIST were evaluated.
[0057] The vertical axis in FIGS. 6 and 7 indicates the ratio λ of the combination of model A trained by MNIST and model B trained by FashionMNIST, and the vertical axis indicates the accuracy of the combined model.
[0058] Here, Model A exhibits an accuracy of nearly 100% for MNIST, but an accuracy of nearly 0% for FashionMNIST. Therefore, in Figures 6 and 7, Model A has an accuracy of approximately 50% for the synthetic data set. Furthermore, Model B exhibits an accuracy of nearly 100% for FashionMNIST, but an accuracy of nearly 0% for MNIST. Therefore, in Figures 6 and 7, Model B also exhibits an accuracy of approximately 50% for the synthetic data set.
[0059] 6 illustrates two cases where the flatness of each model is not taken into consideration: one where permutation is not performed (shown by the dashed line) and one where permutation is performed (shown by the solid line). For each case, examples are shown for training (Train, thick line) and operation (Test).
[0060] Furthermore, when considering the flatness of each model, Figure 7 illustrates three patterns of ensemble results, taking the average of models A and B, shown by the dashed-dotted line, in addition to the two patterns shown in Figure 6. Note that while ensemble is a simple and powerful technique, it increases inference costs.
[0061] In both FIG. 6 and FIG. 7, it was confirmed that the accuracy of the synthesized model was higher when permutation was performed, as indicated by the solid line, than when no permutation was performed, as indicated by the dashed line.
[0062] Furthermore, when the flatness of each model shown in Figure 7 was taken into consideration, the accuracy was 61%, which confirmed that the accuracy of the composite model was higher than the accuracy of 57% when the flatness of each model shown in Figure 6 was not taken into consideration.
[0063] [Program] A program describing the processing executed by the synthesis device 10 according to the above embodiment in a computer-executable language can also be created. In one embodiment, the synthesis device 10 can be implemented by installing a synthesis program that executes the above synthesis processing as package software or online software on a desired computer. For example, by executing the synthesis program on an information processing device, the information processing device can function as the synthesis device 10. Other examples of information processing devices include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants). The functions of the synthesis device 10 may also be implemented on a cloud server.
[0064] 8 is a diagram showing an example of a computer that executes a synthesis program. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.
[0065] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example. The video adapter 1060 is connected to a display 1061, for example.
[0066] Here, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. The various pieces of information described in the above embodiments are stored in the hard disk drive 1031 or memory 1010, for example.
[0067] The merging program is stored in the hard disk drive 1031 as a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the merging device 10 described in the above embodiment is written is stored in the hard disk drive 1031.
[0068] Data used for information processing by the synthesis program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-described procedures.
[0069] Note that the program module 1093 and program data 1094 related to the merging program are not limited to being stored in the hard disk drive 1031, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1041. Alternatively, the program module 1093 and program data 1094 related to the merging program may be stored in another computer connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.
[0070] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.
[0071] REFERENCE SIGNS LIST 10 Synthesis device 11 Input unit 12 Output unit 13 Communication control unit 14 Storage unit 15 Control unit 15a Acquisition unit 15b Learning unit 15c Identification unit
Claims
1. A synthesis device comprising: a learning unit that uses first learning data to learn a first model based on the flatness of the gradient of a loss function of the weight of the first model relative to input data, and that uses second learning data to learn a second model based on the flatness of the gradient of the loss function of the weight of the second model; and an identification unit that uses the weight of the first model and the weight of the second model to identify the weight of a synthesized model obtained by synthesizing the first model and the second model.
2. The synthesis device described in claim 1, characterized in that the identification unit identifies the weight of the synthesized model by rearranging the weights without changing the output of the second model and averaging the rearranged weights and the weight of the first model.
3. A synthesis method executed by a synthesis device, comprising: a learning step of learning a first model using first learning data based on the flatness of the gradient of a loss function of the weight of a first model with respect to input data, and learning a second model using second learning data based on the flatness of the gradient of a loss function of the weight of a second model; and an identification step of identifying the weight of a synthesized model obtained by synthesizing the first model and the second model using the weight of the first model and the weight of the second model.
4. A synthesis program for causing a computer to execute the steps of: learning a first model using first learning data based on the flatness of the gradient of a loss function of the weight of a first model with respect to input data of the first model; and learning a second model using second learning data based on the flatness of the gradient of a loss function of the weight of a second model; and identifying the weight of a synthesized model obtained by synthesizing the first model and the second model using the weight of the first model and the weight of the second model.
Citation Information
Patent Citations
Integration device, learning device, and integration method
JP2022191762A
Training device, training method, and training program
WO2023062742A1
