Vertical federated learning method, guest device, and program

Abstract

The present invention helps effectively utilize all of the data held by a guest device by vertical federated learning. Therefore, the guest device of a vertical federated learning system according to the disclosed technology is provided with an ID coordination unit and a model calculation unit. Assuming that a neural network is divided into a plurality of first models including an input layer and a second model including an output layer, and that a plurality of guest devices holding a data table comprising a data ID and a feature amount each train the first models and a host device trains the second model, the ID coordination unit generates a similarity ranking between data IDs regarding the data table that the host device holds and transmits the similarity ranking to the other guest devices, acquires similarity rankings generated by the other guest devices, and generates from the held data table an extension table including all of the data IDs present in any of the guest devices. The model calculation unit trains the first models on the basis of the extension table.

Description

Vertical federated learning method, guest device, and program

[0001] The disclosed technology relates to a method and apparatus for effectively utilizing data in vertical federated learning.

[0002] [Federated learning] Federated learning is a technology that can generate machine learning models that reflect the characteristics of each dataset by sharing only the information necessary to train a machine learning model, without sharing the datasets (collections of data) managed by each organization (i.e., while protecting the confidentiality / privacy of the data).

[0003] [Vertical Federated Learning] Figure 1 is a functional block diagram of a federated learning system of a type called "vertical federated learning" described in Non-Patent Document 1. In the federated learning system 1, a host device 101 and multiple guest devices 111 to 131 cooperate with each other via, for example, a communication network 141. The host device 101 includes an ID selection unit 102 and a top model calculation unit 103. The guest devices 111 to 131 include ID selection units 112 to 132, bottom model calculation units 113 to 133, and recording units 114 to 134, respectively.

[0004] Each recording unit of each guest device has a data table as shown in Figure 2. Each record in each data table consists of a data ID and one or more features. The neural network trained on each guest device is called the bottom model, and the neural network trained on the host device is called the top model. Multiple bottom models and one top model together form a single neural network. The bottom model side has an input layer, and the top model side has an output layer. Figure 3 is a flowchart showing the operation of vertical associative learning system 1. Vertical associative learning will be explained below using Figures 1, 2, and 3.

[0005] The host device and the guest device identify data IDs that are common to all the guest devices and determine the order of the data IDs to be used for learning (step S301). In the example of Figure 2, IDs 3, 5, and 7 are data IDs that are common to all the guest devices.

[0006] Each guest device inputs data to the bottom model in the order of the determined data IDs and calculates the bottom model output (step S302). Each guest device transmits the bottom model output to the host device (step S303).

[0007] The host device inputs the bottom model outputs to the top model in the order of the determined data IDs, and calculates the output of the top model (step S304).

[0008] The host device updates the top model using the top model output and training data (for example, actual sales data if the model is used to predict the sales of a certain product) by backpropagation (step S305).The host device transmits the error that has reached the initial layer of the top model to the guest device (step S306).

[0009] The guest device updates the bottom model by backpropagation (step S307).

[0010] The above process is repeated until a predetermined termination condition is met, and a final set of top and bottom models is obtained.

[0011] Kang Wei et al., "Vertical Federated Learning: Challenges, Methodologies and Experiments", https: / / arxiv.org / abs / 2202.04309, 2022.

[0012] As mentioned above, in vertical federated learning, before learning begins, each guest device participating in federated learning checks whether it has data with the same ID. As a result, as shown in Figure 2, only data with IDs that are common to all guest devices can be used for learning, and the entire data set cannot be used effectively.

[0013] To solve the above problems, a guest device in a vertical federated learning system according to the disclosed technology includes an ID federation unit and a model calculation unit. The neural network is divided into multiple first models including an input layer and a second model including an output layer. Multiple guest devices each holding a data table consisting of data IDs and features each learn the first model, while the host device learns the second model. The ID federation unit generates a similarity ranking between data IDs for the data table held by the host device and transmits it to other guest devices. The ID federation unit then acquires the similarity rankings generated by the other guest devices. Based on the acquired similarity rankings, the guest device generates an extended table from the data table held by the host device that includes all data IDs present in any of the guest devices. The model calculation unit learns the first model based on the extended table.

[0014] Data in which IDs exist only in some guest devices can also be used for learning, making it possible to effectively utilize the entire wide variety of data possessed by guest devices.

[0015] FIG. 1 is a functional block diagram of a vertical federated learning system according to the prior art. FIG. 2 is a diagram showing an example of a data table held by a guest device. FIG. 3 is a flowchart explaining the operation of a vertical federated learning system according to the prior art. FIG. 4 is a functional block diagram of a vertical federated learning system according to a first embodiment. FIG. 5 is a flowchart explaining the operation of an ID linking unit according to the first embodiment. FIG. 6 is a diagram showing examples of a data table, a similarity ranking, and an extended data table according to the first embodiment. FIG. 7 is a diagram showing an example of the functional configuration of a computer.

[0016] Hereinafter, embodiments of the disclosed technology will be described in detail. Note that components having the same functions are assigned the same numbers, and duplicated descriptions will be omitted.

[0017] [First Embodiment] FIG. 4 is a functional block diagram showing an example of the configuration of a vertical federated learning system 4 according to the first embodiment. For simplicity, an example using two guest devices will be described. The vertical federated learning system 4 is similar to the vertical federated learning system 1, except that the ID selection unit is replaced with ID linkage units 401-403. The operation of the vertical federated learning system 4 is similar to that shown in FIG. 3, except that step S301 in FIG. 3 is replaced with "data linkage processing." FIG. 5 shows an example of the flow of "data linkage processing" performed by the ID linkage unit. FIG. 6 shows examples of data tables held by each guest device (FIGS. 6(a) and (b)), similarity rankings generated by the ID linkage unit for each guest device (FIGS. 6(c) and (d)), and extended data tables (FIGS. 6(e) and (f)). The following description will be made using FIGS. 4, 5, and 6.

[0018] <Similarity Calculation> The ID linking unit of each guest device calculates the similarity between data IDs for the data table held by each guest device and ranks them (step S501). Specifically, for example, the similarity S between ID=1 and ID=2 in the data table of guest device 1 (FIG. 6A) is calculated as follows: 12 is calculated as follows: S 13 , S 14 The row of ID=1 in Fig. 6(c) shows that the result of the similarity calculation is S 12 >S 14 >S 13 The similarity is calculated and ranked in the same way for ID=2, 3, and 4.

[0019] The similarity ranking is similarly calculated for the data table (FIG. 6B) of the guest device 2. FIG. 6D shows an example of the calculation result.

[0020] In general, when the number of features in the data table held by the guest device is M, the m-th feature of the n-th record is expressed as a nm The similarity S between data ID=i and data ID=j is ij is calculated as follows:

[0021] <Data Linkage (Table Expansion)> The guest device 1 transmits the calculated similarity ranking to the guest device 2 and acquires the similarity ranking calculated by the guest device 2 (step S502). Note that even if the similarity ranking of the guest device 2 is acquired, the information that the guest device 1 can know about the data table of the guest device 2 is extremely limited.

[0022] The ID linking unit 402 creates an extension table using the data table and the similarity ranking (step S503). In principle, the table is extended so that the data ID becomes the union of the data IDs held by guest device 1 and guest device 2. Hereinafter, the data ID in the data table of guest device 1 will be referred to as the existing ID, and the data ID in the data table of guest device 2 that is different from the existing ID will be referred to as the complement target ID.

[0023] The guest device 1 recognizes IDs 5 and 6 as IDs to be complemented. According to the similarity ranking calculated by the guest device 2 (FIG. 6(d)), the existing ID most similar to ID 5 is 4. Therefore, the features A and B of ID 4, i.e., a4 and b4, are set as the features A and B of ID 5 in the extension table (e). According to the similarity ranking calculated by the guest device 2 (FIG. 6(d)), the existing ID most similar to ID 6 is 1. Therefore, the features A and B of ID 1, i.e., a1 and b1, are set as the features A and B of ID 6 in the extension table (e).

[0024] The guest device 2 obtains the similarity ranking from the guest device 1. In this case, the information that the guest device 2 can know about the data table of the guest device 1 is also extremely limited. This time, the data IDs in the data table of the guest device 2 are existing IDs, and the data IDs in the data table of the guest device 1 that are different from the existing IDs are IDs to be complemented.

[0025] The guest device 2 recognizes IDs 2 and 3 as IDs to be complemented. According to the similarity ranking calculated by the guest device 1 ( FIG. 6(c) ), the existing ID most similar to ID=2 is 1. Therefore, features C and D of ID=1, i.e., c1 and d1, are set as features C and D of ID=2 in the extension table (f). According to the similarity ranking calculated by the guest device 1 ( FIG. 6(c) ), the existing ID most similar to ID=3 is 4. Therefore, features C and D of ID=4, i.e., c4 and d4, are set as features C and D of ID=3 in the extension table (f).

[0026] Once the expansion table is complete, the host device and guest device determine the order of the data IDs to be used for learning (step S504).

[0027] <Implementation of Vertical Federated Learning> The guest devices 1 and 2 and the host device implement vertical federated learning using the extended data table. The implementation procedure is the same as steps S302 to S307 in FIG.

[0028] This concludes the description of the first embodiment. In the above description, the similarity calculation is performed using Euclidean distance, but Manhattan distance, Chebyshev distance, cosine similarity, Jaccard coefficient, or anything similar thereto may also be used.

[0029] The above embodiment can also be applied to a system with three or more guest devices. In this case, each guest device shares all calculated similarity rankings. If multiple ranking results for the ID to be complemented conflict (for example, if guest device 1 already has IDs 1 and 2 and wants to complement ID 5, and according to guest device 2, ID 1 is the most similar to ID 5, while according to guest device 3, ID 2 is the most similar to ID 5), a rule can be established to select the candidate with the highest similarity. Note that the extension table only needs to be a union of data IDs, so the rules for handling conflicts in ranking results can be different for each guest device.

[0030] Furthermore, in the above embodiment, similarity rankings are mutually acquired, but for example, for data IDs held by guest device 1, the holding status of guest device 2 may be checked one by one, and if an ID not held by guest device 2 is found, guest device 1 may present similar ID information to complement the feature quantities of the extension table of guest device 2. The same applies to the extension table of guest device 1.

[0031] [Second Embodiment] In the first embodiment, the feature values ​​of the ID to be complemented are supplemented with the feature values ​​of the existing IDs according to the similarity ranking. However, when the similarity between data IDs is low, it is not appropriate to use the feature values ​​of the existing IDs as the feature values ​​of the ID to be complemented. Therefore, a judgment criterion may be set for the similarity, and if the criterion is not met, statistics calculated from the feature values ​​of all existing IDs may be used as the feature values ​​of the ID to be complemented instead of the feature values ​​of a specific existing ID. The judgment criterion may be, for example, a similarity of X or more, the top Y% of the ranking, or within Z rankings. The statistics may be, for example, the average, median, maximum, or minimum value of all feature values. Note that when similarity rankings are exchanged, the judgment and calculation of statistics are performed by the device that receives the similarity ranking. When similar ID information is presented each time, the judgment and calculation of statistics are performed by the device that presents the similar ID information.

[0032] [Third Embodiment] In the first and second embodiments, in principle, the feature values ​​of the ID to be complemented are supplemented with the feature values ​​of the existing ID with the highest similarity ranking among the available existing IDs. In the third embodiment, existing IDs that are not the highest in similarity ranking are also used among the available existing IDs. Specifically, the similarity ranking is converted into probability magnitude to select the existing ID. Furthermore, similar ID information is disclosed on a case-by-case basis, rather than by exchanging the entire ranking. This will be explained in detail below.

[0033] Assume that guest device 1 obtains information from guest device 2 and complements the features of the ID to be complemented on guest device 1. Assume that the similarity rankings for a certain ID to be complemented on guest device 2 are ID-a, ID-b, ID-c, and ID-d, in descending order. In the previous embodiments, guest device 2 presents ID-a to guest device 1. However, in the third embodiment, guest device 2 presents ID-a to guest device 1 with a 60% probability, ID-b with a 30% probability, ID-c with an 8% probability, and ID-d with a 2% probability. Here, the probabilities such as 60% and 30% may be arbitrarily determined while matching the high and low rankings with the magnitude of the probabilities. According to the third embodiment, it is impossible to know which IDs are linked between guest device 1 and guest device 2. This further limits the information that may be leaked when similar ID information is presented, improving security. Furthermore, it prevents the creation of data that is never handled due to low similarity.

[0034] [Program, Recording Medium] The functions realized by the components described in this specification may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to realize the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in a memory.

[0035] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.

[0036] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0037] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 7, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.

[0038] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.

[0039] The program may be distributed by, for example, selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to other computers via a network, thereby distributing the program.

[0040] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process at the terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of a server computer along with the program. In this embodiment, the program includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that dictate computer processing).

[0041] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0042] 1, 4 Vertical federated learning system 101 Host device 102 ID selection unit 103 Top model calculation unit 111, 121, 131 Guest device 112, 122, 132 ID selection unit 113, 123, 133 Bottom model calculation unit 114, 124, 134 Recording unit 401, 402, 403 ID linking unit 2000 Computer 2010 Control unit 2020 Recording unit 2030 Input unit 2040 Output unit 2050 Display unit

Claims

1. A vertical federated learning method in which a neural network is divided into a plurality of first models including an input layer and a second model including an output layer, a plurality of guest devices holding a data table consisting of data IDs and features each learning the first model, and a host device learning the second model, wherein each of the guest devices generates a similarity ranking between data IDs for the data table it holds and transmits it to other guest devices, acquires the similarity rankings generated by the other guest devices, and generates an extended table from the data table it holds based on the acquired similarity rankings, including all of the data IDs present in any of the guest devices, and each of the guest devices learns the first model based on the extended table.

2. A guest device in vertical federated learning in which a neural network is divided into multiple first models including an input layer and a second model including an output layer, multiple guest devices holding a data table consisting of data IDs and features each learn the first model, and a host device learns the second model, the guest device comprising: an ID linking unit that generates a similarity ranking between data IDs for the data table it holds and sends it to other guest devices, acquires the similarity rankings generated by the other guest devices, and generates an extended table from the data table it holds that includes all data IDs present in any of the guest devices based on the acquired similarity rankings; and a model calculation unit that learns the first model based on the extended table.

3. A guest device according to claim 2, wherein data IDs in said stored data table are defined as existing IDs, and data IDs in the data tables of other guest devices that are different from said existing IDs are defined as IDs to be complemented, and said ID linking unit determines existing IDs that are similar to said IDs to be complemented based on said acquired similarity ranking, and generates said extension table by using features of said determined existing IDs as features of said IDs to be complemented and / or by using statistics calculated from the features of said existing IDs as features of said IDs to be complemented.

4. A program for causing a computer to function as the guest device according to claim 2 or 3.

Images