Information processing device and information processing method

Abstract

According to the present invention, a passive participant performs a step for inputting training sample data into a local model to calculate intermediate results and a step for transmitting the difference between intermediate results for a current epoch and intermediate results for a previous epoch to an active participant. The active participant performs a step for receiving the difference between the intermediate results for the current epoch and the intermediate results for the previous epoch from the passive participant, a step for reconstructing the intermediate results for the current epoch on the basis of the difference between the intermediate results, and a step for inputting the reconstructed intermediate results into a local model to calculate the error with respect to ground truth labels.

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method.

[0002] Federated learning (FL) is a machine learning concept in which multiple entities jointly train a machine learning model in a distributed manner without sharing samples. There are two types of federated learning: horizontal federated learning (HFL) and vertical federated learning (VFL).

[0003] Regarding HFL and VFL, 3GPP (registered trademark) specifies the HFL procedure between network data analysis functions (NWDAFs) (e.g., Non-Patent Document 1). 3GPP also specifies HFL, VFL, VFL active participants, and VFL passive participants in 5G systems (e.g., Non-Patent Document 2).

[0004] One of the challenges of federated learning is the need for a large amount of bidirectional communication between active and passive participants. In particular, vertical federated learning requires bidirectional communication between active and passive participants in the forward and backpropagation processes, which increases communication data size, communication cost, and communication latency. Furthermore, in vertical federated learning, the number of features (attributes), communication bandwidth, communication cost, and processing power may differ for each communication link between active and passive participants.

[0005] Non-Patent Document 3 proposes a communication-efficient horizontal federated learning method. Specifically, Non-Patent Document 3 proposes learning model updates (e.g., deltas between global and local models) from a restricted space parameterized using a smaller number of variables, such as low rank or a random mask. After learning the complete model update, it compresses it using a combination of quantization, random rotation, and subsampling before transmitting it to the server. However, this method cannot be applied to vertical federated learning because there is no common global model between active and passive participants in vertical federated learning, making it non-trivial to calculate the difference between the global model and the local model.

[0006] Non-Patent Document 4 proposes vertical federated learning using common compression techniques such as scalar quantization, vector quantization, and top-K sparsification. Non-Patent Document 5 also proposes communication-efficient vertical federated learning based on embedding (intermediate results) and gradient sparsification. However, it is assumed that all participants use the same quantization and compression parameters, and does not disclose how to determine parameters according to the number of features (attributes), communication bandwidth, communication cost, and processing power for each communication link between active and passive participants.

[0007] 3GPP TS 23.288 V18.4.0, “Architecture enhancements for 5G System (5GS) to support network data analytics services.” https: / / www.3gpp.org / ftp / Specs / archive / 23_series / 23.288 / 23288-i40.zip3GPP TR 23.700-84 V0.2.0, “Study on Core Network Enhanced Support for Artificial Intelligence (AI) / Machine Learning (ML).” https: / / www.3gpp.org / ftp / Specs / archive / 23_series / 23.700-84 / 23700-84-020.zipKone?ny, J. et al., “Federated Learning: Strategies for Improving Communication Efficiency”. NIPS Workshop on Private Multi-Party Machine Learning. 2016. https: / / arxiv.org / pdf / 1610.05492T. Castiglia, et al., “Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data,” International Conference on Machine Learning, 2022. https: / / arxiv.org / pdf / 2206.08330Y. Inoue, et al., “SparseVFL: Communication-Efficient Vertical Federated Learning Based on Sparsification of Embeddings and Gradients,” KDD FL4Data-Mining '23: International Workshop on Federated Learning for Distributed Data Mining, 2023.https: / / openreview.net / pdf?id=BVH3-XCRoN3.

[0008] An object of the present disclosure is to provide an information processing device, an information processing method, and a program that enable vertical federated learning with high communication efficiency.

[0009] One aspect of the present disclosure is an information processing device that participates in vertical federated learning as a passive participant, comprising a processor and a storage device that stores learning sample data and a local model, wherein the processor executes the steps of inputting the learning sample data into the local model to calculate an intermediate result, and transmitting the difference between the intermediate result in the current epoch and the intermediate result in the previous epoch to active participants.

[0010] One aspect of the present disclosure is an information processing device that participates in vertical federated learning as an active participant, comprising a processor and a storage device that stores a correct label and a local model, wherein the processor executes the steps of receiving, from a passive participant, a difference between an intermediate result in a current epoch and an intermediate result in a previous epoch, reconstructing the intermediate result in the current epoch based on the difference between the intermediate results, and inputting the reconstructed intermediate result into the local model and calculating an error from the correct label.

[0011] One aspect of the present disclosure is an information processing device that participates in vertical federated learning as an active participant, comprising a processor and a storage device that stores a correct answer label and a local model, wherein the processor executes the following steps: receiving data related to an intermediate result from a passive participant; inputting the data related to the intermediate result into the local model and calculating an error from the correct answer label; backpropagating the error to calculate a gradient; updating the local model based on the gradient; and transmitting the difference between the gradient in the current epoch and the gradient in the previous epoch to the passive participant.

[0012] One aspect of the present disclosure is an information processing device that participates in vertical federated learning as a passive participant, comprising a processor and a storage device that stores learning sample data and a local model, wherein the processor executes the steps of receiving, from an active participant, a difference between a gradient in a current epoch and a gradient in a previous epoch; reconstructing a gradient in the current epoch based on the difference in gradient; and updating the local model based on the reconstructed gradient.

[0013] One aspect of the present disclosure is an information processing method performed by an information processing device that has a storage device that stores learning sample data and a local model and participates in vertical federated learning as a passive participant, the information processing method comprising the steps of inputting the learning sample data into the local model to calculate an intermediate result, and transmitting the difference between the intermediate result in the current epoch and the intermediate result in the previous epoch to active participants.

[0014] One aspect of the present disclosure is an information processing method performed by an information processing device that has a storage device that stores a correct label and a local model and participates in vertical federated learning as an active participant, the information processing method comprising the steps of receiving, from a passive participant, the difference between an intermediate result in a current epoch and the intermediate result in a previous epoch, reconstructing the intermediate result in the current epoch based on the difference between the intermediate results, and inputting the reconstructed intermediate result into the local model and calculating an error from the correct label.

[0015] One aspect of the present disclosure is an information processing method performed by an information processing device that has a storage device that stores correct labels and local models and participates in vertical federated learning as an active participant, the information processing method comprising the steps of receiving data related to intermediate results from passive participants, inputting the data related to the intermediate results into the local model and calculating an error from the correct label, backpropagating the error to calculate a gradient, updating the local model based on the gradient, and transmitting the difference between the gradient in the current epoch and the gradient in the previous epoch to the passive participants.

[0016] One aspect of the present disclosure is an information processing method performed by an information processing device that has a storage device that stores learning sample data and a local model and participates in vertical federated learning as a passive participant, the information processing method comprising the steps of receiving, from an active participant, a difference between a gradient in a current epoch and a gradient in a previous epoch, reconstructing a gradient in the current epoch based on the difference in gradient, and updating the local model based on the reconstructed gradient.

[0017] Other aspects include an information processing system composed of the above-mentioned information processing device, an information processing method corresponding to the above-mentioned information processing device, a program for causing a computer to execute the information processing method, or a computer-readable storage medium that non-temporarily stores the program.

[0018] According to the present disclosure, it is possible to reduce communication data size and perform vertical federated learning with high communication efficiency.

[0019] FIG. 1 is a sequence diagram showing an example of the operation of vertical federated learning in an information processing system. FIG. 2 is an explanatory diagram of vertical federated learning. FIG. 3 is a flowchart showing the processing performed by passive participants in forward propagation processing. FIG. 4 is a flowchart showing the processing performed by active participants in forward propagation processing. FIG. 5 is a flowchart showing the processing performed by active participants in back propagation processing. FIG. 6 is a flowchart showing the processing performed by passive participants in back propagation processing. FIG. 7 is an explanatory diagram of a 5G core system, which is an example of an information processing system that performs vertical federated learning. FIG. 8 shows an example configuration of an information processing device that constitutes a communication system.

[0020] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The configurations of the following embodiments are examples, and the present disclosure is not limited to the configurations of the embodiments.

[0021] <Overview of Vertical Federated Learning> Figure 2 is an explanatory diagram of Vertical Federated Learning (VFL). VFL is a federated learning technique that performs learning without exchanging or sharing local data held by each entity, and in which the local data held by each entity has different features of the same sample.

[0022] 2, an information processing system 200 that performs vertical federated learning is configured to include one active participant 210 and multiple (three in FIG. 2) passive participants 210A-210C. Of the multiple participants in the VFL, participants (also called parties) that have a correct answer label in machine learning are called active participants, and participants other than the active participants (participants that do not have a correct answer label) are called passive participants.

[0023] The active participant 210 has a local model 211 and a ground truth label 212 for each sample. The passive participants 220A-220C have local models 221A-221C and training sample data 222A-222C for each sample, respectively. In the following description, the passive participants 220A-220C may be referred to as passive participants 220, the local models 221A-221C may be referred to as local models 221, and the training sample data 222A-222C may be referred to as training sample data 222.

[0024] Local model 211 and local models 221A to 221C are each neural networks, and there is a one-to-one correspondence between each node in the most downstream layer of local model 211 and each node in the most upstream layer of local models 221A to 221C. Therefore, local model 211 and local models 221A to 221C form a single learning model or neural network. Local model 211 is also called the top model because it is an upstream model, and local models 221A to 221C are also called bottom models because they are downstream models.

[0025] The correct answer label 212 and the training sample data 222A-222C include correct answer labels and features (attributes) for common samples. The training sample data 222A-222C each include different features (attributes). The training sample data 222A-222C may include data for different (non-common) samples or may include overlapping features (attributes). However, when vertical training is performed, each passive participant 220A-220C performs training using different features of the common samples.

[0026] An overview of the vertical association learning process will be described below.

[0027] The passive participants 220A-220C input their respective training sample data 222A-222C to the local models 221A-221C to obtain the output of the top layer of the local models 221A-221C. This output is also referred to as an embedding or an intermediate result. In this disclosure, the output, embedding, and intermediate result of the local models 221A-221C are used interchangeably.

[0028] The passive participants 220A-220C send their intermediate results to the active participant 210. The active participant 210 inputs the intermediate results received from the passive participants 220A-220C into the local model 211 to obtain the final output, and calculates the error (loss function) by comparing the final result with the correct label. This completes the forward propagation process in the learning process.

[0029] The active participant 210 calculates the gradient based on the calculated error, sequentially calculates the gradient at each layer of the local model 211 by backpropagation, and updates the local model 211 (updates the weight parameters) based on the gradient. After calculating the gradient at the most downstream layer, the active participant 210 transmits the gradient to each of the passive participants 220A-220C. The passive participants 220A-220C also sequentially calculate the gradient at each layer of the local models 221A-221C by backpropagation, and update the local models 221A-221C (updates the weight parameters) based on the gradient. This completes the backpropagation process in the learning process.

[0030] The forward propagation process and the back propagation process complete one epoch of learning. The information processing system 200 improves the prediction accuracy of the model by performing multiple epochs of learning.

[0031] Here, the intermediate results (embeddings) sent from the passive participants 220A-220C to the active participant 210 and the gradients sent from the active participant 210 to the passive participants 220A-220C involve large amounts of data, which poses a problem of significant communication costs, communication delays, and processing times. Below, we propose a vertical federated learning method that improves communication efficiency.

[0032] <Example of Information Processing System> Figure 7 shows an example of the configuration of a 5G system, which is an example of an information processing system that performs vertical federation learning. In Figure 3, UE (User Equipment) 2 is a user (subscriber) terminal. RAN (Radio Access Network) 3 is an access network to the 5G core network (5GC). RAN 3 is composed of base stations (gNBs). The 5G network has a 5G core network (5GC) and an access network ((R)AN), and UE 2, DN 5, and AF 12 are connected to the 5G network. Each of NFs 11a to 11k is a function realized by one or more computers (information processing devices) executing a program. However, a single computer may realize two or more of NFs 11a to 11k.

[0033] 5GC is composed of a set of components with specific functions called NFs (Network Functions). Figure 3 shows the following NFs 11 that make up 5GC. In Figure 3, they are indicated by bold rectangles.

[0034] UPF (User Plane Function) 11a AMF (Access and Mobility Management Function) 11b SMF (Session Management Function) 11c PCF (Policy Control Function) 11d NEF (Network Exposure Function) 11e NRF (Network Repository Function) 11g NSSF (Network Slice Selection Function) 11h AUSF (Authentication Server) Function) 11i UDM (Unified Data Management) 11j NWDAF (Network Data Analytics Function) 11k

[0035] The active participants and passive participants participating in the vertical federated learning may be any NF, for example, the NWDAF 11k or the AF 12. Note that there may be multiple NFs in the system, and all of the active participants and passive participants participating in the federated learning may be the NWDAF 11k.

[0036] <Details of Vertical Federated Learning Process> In the following description, it is assumed that the NF 11 participates in vertical federation learning as an active participant and a passive participant. Therefore, in the following description, an active participant is also referred to as an active NF, and a passive participant is also referred to as a passive NF.

[0037] [Step 1: Data Collection and Alignment] In step 1, the active NF 110 and the passive NF 120 collect data from the data source NF 140 via the NRF 130 and perform data alignment of samples and features. Data alignment is a process of determining a common sample ID to be used in vertical association learning and determining the features to be used by each passive NF 120 in vertical association learning. The specific method of data alignment is not particularly limited, and any conventional method can be used.

[0038] [Step 2: Determining Quantization and Compression Parameters for Intermediate Result Difference] As will be described later, in step 3a, the passive NF 120 performs quantization and data compression processing on the intermediate result (embedding) difference Δe and transmits the result to the active NF 110. At this time, it is necessary for the active NF 110 and the passive NF 120 to agree on the parameters to be used for quantization and data compression.

[0039] In step 2, the passive NF 120 and the active NF 110 determine and agree on parameters for quantization and data compression to be applied to the intermediate result difference Δe. The parameters are determined using at least one of the communication bandwidth, throughput, and communication cost between the passive NF 120 and the active NF 110, the processing capacity of the passive NF 120, the processing capacity of the active NF 110, the target communication cost, the target communication delay, the target processing time in the passive NF 120, the target processing time in the active NF 110, the target prediction accuracy, and the target loss.

[0040] The parameters for the quantization process and the data compression process are determined for each combination of the active NF 110 and the passive NF 120. However, common parameters may be used for at least some of the different combinations of the active NF 110 and the passive NF 120.

[0041] The parameters of quantization and data compression are the same as or can be derived from the parameters of inverse quantization and data restoration, and therefore the parameters of quantization and data compression can also be considered as parameters of inverse quantization and data restoration.

[0042] In one embodiment, the passive NF 120 determines parameters for the quantization process and the data compression process. The parameter for the quantization process is, for example, the number of quantization bits, and the parameters for the data compression process are the selection of a data compression method and a compression rate. The passive NF 120 determines the parameters so as to satisfy at least one of the following conditions, and notifies the active NF 110 of the determined parameters:

[0043] Condition 1: The communication cost expected to be required for communicating the intermediate result difference Δe after quantization and data compression is lower than a target communication cost (first target value). Condition 2: The communication delay associated with transmitting the intermediate result difference Δe, which is derived based on the communication bandwidth or throughput with the active NF 110 and the data size of the intermediate result difference Δe after quantization and data compression, is smaller than a target communication delay (second target value). Condition 3: The processing time associated with the quantization process and data compression process in the passive NF 120, which is derived based on the data size of the intermediate result difference Δe after quantization and data compression and the processing capacity of the passive NF 120, is smaller than a target processing time (third target value).

[0044] In another embodiment, the active NF 110 determines parameters for the quantization process and the data compression process. The parameter for the quantization process is, for example, the number of quantization bits, and the parameters for the data compression process are the selection of a data compression method and a compression rate. The active NF 110 determines the parameters so as to satisfy at least one of the following conditions, and notifies the passive NF 120 of the determined parameters.

[0045] Condition 4: The processing time for the inverse quantization process and data restoration process in the active NF 110, which is derived based on the data size of the intermediate result difference Δe after quantization and data compression and the processing capacity of the active NF 110, is smaller than the target processing time (fourth target value). Condition 5: The learning accuracy of the predicted model is higher than the target accuracy (fifth target value). Condition 6: The error of the predicted model is lower than the target error (sixth target value).

[0046] The prediction of the learning accuracy and error under Conditions 5 and 6 can be performed based on a partial learning data set such as a validation learning data set, for example.

[0047] In yet another embodiment, the passive NF 120 and the active NF 110 may each determine parameter candidates, and one of these candidates may be determined as the final parameter. If the parameters determined by the passive NF 120 and the active NF 110 are the same, the parameter may be determined as the final parameter. If the parameters determined by the passive NF 120 and the active NF 110 are different, the passive NF 120 and the active NF 110 select one of the parameter candidates as the final parameter according to the priority of the performance index. Examples of the performance index include communication cost, communication delay, processing time in the passive NF, processing time in the active NF, learning accuracy, and error. The above condition 1 is related to communication cost, condition 2 is related to communication delay, condition 3 is related to processing time in the passive NF, condition 4 is related to processing time in the active NF, condition 5 is related to learning accuracy, and condition 6 is related to error. The priority of the performance index is, for example, input by an administrator to the information processing system as a setting value. For example, if the passive NF 120 determines parameter candidates based on condition 1 and communication cost has the highest priority, the parameter candidates determined by the passive NF 120 based on condition 1 are determined as the final parameters.

[0048] In still another embodiment, whether the passive NF 120 or the active NF 110 determines the parameters may be switched based on the priority of the performance index. For example, if the performance indexes of communication cost, communication delay, and processing time in the passive NF have a high priority, the passive NF 120 determines the parameters based on conditions 1 to 3. Furthermore, if the performance indexes of processing time, learning accuracy, and error in the active NF have a high priority, the active NF 110 determines the parameters based on conditions 4 to 6.

[0049] The target values ​​(first to sixth target values) of the communication cost, communication delay, processing time in the passive NF, processing time in the active NF, learning accuracy, and error may be input as setting values ​​by an administrator to the information processing system. Furthermore, these target values ​​may be values ​​defined as standards (including de jure standards and de facto standards), for example.

[0050] In one embodiment, the active NF 110 and the passive NF 120 may redetermine parameters for the quantization process and the data compression process when a predetermined trigger condition is satisfied. Examples of the trigger condition include a change in the communication bandwidth, throughput, or communication cost between the active NF 110 and the passive NF 120, a change in the processing capabilities of the active NF 110 and the passive NF 120, or a change in any of the target values ​​(the first to sixth target values).

[0051] [Step 3: Forward propagation processing] The forward propagation processing of step 3 includes calculating intermediate results in the passive NF 120, calculating intermediate result differences, and transmitting them to the active NF 110 (the above, step 3a), reconstructing the intermediate results in the active NF 110 (step 3b), and calculating errors in the active NF 110 (step 3c).

[0052] The process performed by the passive NF 120 in step 3a will be described in detail with reference to Figure 3. In step S301, the passive NF 120 inputs the training sample data 222 to the local model 221 and calculates the output of the top layer of the local model 221 by forward propagation processing. The output from the top layer of the local model 221 is called an intermediate result or embedding, as described above. In step S302, the passive NF 120 stores the intermediate result obtained in the current epoch in a storage device for use in the next epoch.

[0053] In step S303, it is determined whether the current epoch is the first epoch or the second or subsequent epoch. If it is the first epoch, the process proceeds to step S308, and if it is the second or subsequent epoch, the process proceeds to step S304.

[0054] For the second and subsequent epochs, the passive NF 120 calculates the difference Δe between the intermediate results of the previous epoch and the current epoch in step S304, quantizes the intermediate result difference Δe in step S305, and performs data compression on the quantized intermediate result difference Δe in step S306. The quantization process in step S305 and the data compression process in step S306 are performed using the parameters determined in step 2. Examples of data compression techniques that can be used include sparsification, low-rank approximation, and stochastic quantization. In step S307, the passive NF 120 transmits the intermediate result difference Δe, which has been subjected to the quantization and data compression processes and has undergone data reduction, to the active NF 110. Because the intermediate result difference Δe is the difference between the previous epoch and the current epoch, many of its values ​​are close to zero, enabling data compression at a high compression rate and effectively reducing the size of the data being transmitted. Furthermore, since the quantization and data compression parameters are determined for each pair of the active NF 110 and the passive NF 120, optimal data reduction can be achieved for each pair.

[0055] Since the difference between the intermediate result of the previous epoch and the current epoch cannot be calculated in the first epoch, the passive NF 120 transmits the intermediate result e to the active NF 110 in step S308. At this time, the intermediate result e may be subjected to the same quantization and data compression processing as in steps S305 and S306.

[0056] The processing performed by the active NF 110 in steps 3b and 3c will be described in detail with reference to Fig. 4. In Fig. 4, steps S311 to S317, especially steps S312 to S315, correspond to the reconstruction processing of the intermediate result in step 3b, and steps S318 to S319 correspond to the error calculation processing in step 3c.

[0057] In the case of the second or subsequent epoch (S311-NO), the active NF 110 receives from the passive NF 120 an intermediate result difference Δe that has been subjected to quantization processing and data compression processing and has been data-reduced. The active NF 110 reconstructs the intermediate result e of the current epoch from this data-reduced intermediate result difference Δe. Specifically, in step S313, the active NF 110 performs data restoration processing on the data-reduced intermediate result difference Δe, and in step S314, applies inverse quantization processing to the restored intermediate result difference Δe. Furthermore, in step S315, the active NF 110 obtains the intermediate result e of the current epoch from the intermediate result of the previous epoch stored in the storage device and the intermediate result difference Δe received in the current epoch.

[0058] If it is the first epoch (S311-YES), in step S316, the active NF 110 receives the intermediate result e from the passive NF 120. If this data has been subjected to quantization processing and data compression processing, the active NF 110 performs data restoration processing and inverse quantization processing.

[0059] In step S317, the active NF 110 stores the intermediate result e of the current epoch in a storage device for use in the next epoch.

[0060] In step S318, the active NF 110 inputs the intermediate result e of the current epoch to the local model 211 and obtains the final output by forward propagation processing. In step S319, the active NF 110 calculates an error (loss function) based on the final output and the correct label.

[0061] [Step 4: Determining Quantization and Compression Parameters for Gradient Difference] As will be described later, in step 5a, the active NF 110 performs quantization and data compression processing on the gradient difference Δg and transmits the quantized data to the passive NF 120. At this time, it is necessary for the active NF 110 and the passive NF 120 to agree on the parameters to be used for quantization and data compression. Note that, although this parameter determination processing is performed after step 3 in FIG. 1 , the parameters for the quantization and compression processing of the gradient difference Δg may be determined before step 3, for example, at the same timing as step 2.

[0062] In step 4, the active NF 110 and the passive NF 120 determine and agree on parameters for quantization and data compression to be applied to the gradient difference Δg. The parameters are determined using at least one of the communication bandwidth, throughput, and communication cost between the active NF 110 and the passive NF 120, the processing capacity of the active NF 110, the processing capacity of the passive NF 120, the target communication cost, the target communication delay, the target processing time in the passive NF 120, the target processing time in the active NF 110, the target prediction accuracy, and the target loss.

[0063] The parameters for the quantization process and the data compression process are determined for each combination of the active NF 110 and the passive NF 120. However, common parameters may be used for at least some of the different combinations of the active NF 110 and the passive NF 120.

[0064] The parameters of quantization and data compression are the same as or can be derived from the parameters of inverse quantization and data restoration, and therefore the parameters of quantization and data compression can also be considered as parameters of inverse quantization and data restoration.

[0065] In one embodiment, the active NF 110 determines parameters for the quantization process and the data compression process. The parameter for the quantization process is, for example, the number of quantization bits, and the parameters for the data compression process are the selection of a data compression method and a compression rate. The active NF 10 determines the parameters so as to satisfy at least one of the following conditions, and notifies the passive NF 120 of the determined parameters.

[0066] Condition 1: The communication cost expected to be required for communicating the gradient difference Δg after quantization and data compression is lower than a target communication cost (seventh target value). Condition 2: The communication delay associated with transmitting the gradient difference Δg, which is derived based on the communication bandwidth or throughput with the passive NF 120 and the data size of the gradient difference Δg after quantization and data compression, is smaller than a target communication delay (eighth target value). Condition 3: The processing time associated with the quantization process and data compression process in the active NF 110, which is derived based on the data size of the gradient difference Δg after quantization and data compression and the processing capacity of the active NF 110, is smaller than a target processing time (ninth target value).

[0067] In another embodiment, the passive NF 120 determines parameters for the quantization process and the data compression process. The parameter for the quantization process is, for example, the number of quantization bits, and the parameters for the data compression process are the selection of a data compression method and a compression rate. The passive NF 120 determines the parameters so as to satisfy at least one of the following conditions, and notifies the active NF 110 of the determined parameters.

[0068] Condition 4: The processing time for the inverse quantization process and data restoration process in the passive NF 120, which is derived based on the data size of the gradient difference Δg after quantization and data compression and the processing capacity of the passive NF 120, is shorter than the target processing time (tenth target value).

[0069] In yet another embodiment, the active NF 110 and the passive NF 120 may each determine parameter candidates, and one of these candidates may be determined as the final parameter. If the parameter determined by the passive NF 120 is the same as the parameter determined by the active NF 110, the parameter may be determined as the final parameter. If the parameter determined by the passive NF 120 is different from the parameter determined by the active NF 110, the passive NF 120 and the active NF 110 select one of the parameter candidates as the final parameter according to the priority of the performance index. Examples of the performance index include communication cost, communication delay, processing time in the passive NF, processing time in the active NF, learning accuracy, and error. The above condition 1 is related to communication cost, condition 2 is related to communication delay, condition 3 is related to processing time in the active NF, and condition 4 is related to processing time in the passive NF. The priority of the performance index is, for example, input by an administrator to the information processing system as a setting value. For example, if the active NF 110 determines parameter candidates based on condition 1 and communication cost has the highest priority, the parameter candidates determined by the active NF 110 based on condition 1 are determined as the final parameters.

[0070] In still another embodiment, whether the passive NF 120 or the active NF 110 determines the parameters may be switched based on the priority of the performance index. For example, if the performance indexes of communication cost, communication delay, and processing time in the active NF have high priority, the active NF 110 determines the parameters based on conditions 1 to 3. Also, if the performance index of processing time in the passive NF has high priority, the passive NF 120 determines the parameters based on condition 4.

[0071] The target values ​​(seventh to tenth target values) for the communication cost, communication delay, processing time in the passive NF, and processing time in the active NF may be input as setting values ​​by an administrator to the information processing system. These target values ​​may also be values ​​defined as standards (including de jure standards and de facto standards), for example.

[0072] In one embodiment, the active NF 110 and the passive NF 120 may redetermine parameters for the quantization process and the data compression process when a predetermined trigger condition is satisfied. Examples of the trigger condition include a change in the communication bandwidth, throughput, or communication cost between the active NF 110 and the passive NF 120, a change in the processing capabilities of the active NF 110 and the passive NF 120, or a change in any of the target values ​​(the seventh to tenth target values).

[0073] [Step 5: Backpropagation Processing] The forward propagation processing in step 5 includes calculating the error and gradient in the active NF 110, calculating the gradient difference and transmitting it to the passive NF 120 (the above, step 5a), reconstructing the gradient in the passive NF 120 (step 5b), and updating the local model in the active NF 110 and the passive NF 120 (step 5c).

[0074] The details of the processing performed by the active NF 110 in steps 5a and 5c will be described with reference to Fig. 5. In Fig. 5, steps S501 to S507 correspond to the gradient difference transmission processing in step 5a, and step S509 corresponds to the model update processing in step 5c.

[0075] In step S501, the active NF 110 inputs the error to the local model 221 and calculates the gradient at each node in each layer of the local model 221 by backpropagation processing. The gradient sent to the passive NF 120 is the gradient of the lowest layer of the local model 221, and in the following description, unless otherwise specified, the gradient refers to the gradient of the lowest layer of the local model 221. In step S502, the active NF 110 stores the gradient obtained in the current epoch in a storage device for use in the next epoch.

[0076] In step S503, it is determined whether the current epoch is the first epoch or the second or subsequent epoch. If it is the first epoch, the process proceeds to step S508, and if it is the second or subsequent epoch, the process proceeds to step S504.

[0077] For the second and subsequent epochs, the active NF 110 calculates the gradient difference Δg between the previous epoch and the current epoch in step S504, quantizes the gradient difference Δg in step S505, and performs data compression on the quantized gradient difference Δg in step S506. The quantization process in step S505 and the data compression process in step S506 are performed using the parameters determined in step 4. Examples of data compression techniques that can be used include sparsification, low-rank approximation, and stochastic quantization. In step S507, the active NF 110 transmits the reduced-data gradient difference Δg, which has been subjected to the quantization and data compression processes, to the passive NF 120. Because the gradient difference Δg is the difference between the previous epoch and the current epoch, many of its values ​​are close to zero, enabling data compression at a high compression rate, thereby effectively reducing the size of the transmitted data. Furthermore, since the quantization and data compression parameters are determined for each pair of the active NF 110 and the passive NF 120, optimal data reduction can be achieved for each pair.

[0078] Since the difference between the gradient of the first epoch and that of the previous epoch cannot be calculated in the first epoch, the active NF 110 transmits the gradient g to the passive NF 120 in step S508. At this time, the gradient g may be subjected to the same quantization and data compression processing as in steps S505 and S506.

[0079] In step S509, the active NF 110 updates the local model 211 based on the gradient of each layer by, for example, adjusting the weight coefficients between nodes.

[0080] The details of the processing performed by the passive NF 120 in steps 5b and 5c will be described with reference to Fig. 6. In Fig. 6, steps S511 to S517, especially S512 to S515, correspond to the gradient reconstruction processing in step 3b, and step S518 corresponds to the model update processing in step 5c.

[0081] In the case of the second or subsequent epoch (S311-NO), the passive NF 120 receives from the active NF 110 the gradient difference Δg, which has been subjected to quantization and data compression and has been reduced in data. The passive NF 120 reconstructs the gradient g of the current epoch from this reduced-data gradient difference Δg. Specifically, in step S5313, the passive NF 120 performs data restoration on the reduced-data gradient difference Δg, and in step S514, applies inverse quantization to the restored gradient difference Δg. Furthermore, in step S515, the passive NF 120 obtains the gradient of the current epoch from the gradient of the previous epoch stored in the storage device and the gradient difference Δg received for the current epoch.

[0082] If it is the first epoch (S511—YES), in step S316, the passive NF 120 receives the gradient g from the active NF 110. If this data has been subjected to quantization processing and data compression processing, the passive NF 120 performs data restoration processing and inverse quantization processing.

[0083] In step S517, the passive NF 120 stores the gradient g of the current epoch in a storage device for use in the next epoch.

[0084] In step S518, the passive NF 120 inputs the gradient g of the current epoch to the local model 221, obtains the gradient of each layer by backpropagation processing, and updates the local model 221 based on the gradient of each layer. The local model 221 is updated by, for example, adjusting the weight coefficient between nodes.

[0085] [Step 6: Repeated Processing] Learning for one epoch is performed by the processing from step 2 to step 5. The information processing system repeats the processing from step 2 to step 5 described above until a predetermined condition is met, for example, until the estimation accuracy of the model satisfies the condition, or until a predetermined number of epochs are completed.

[0086] Note that the parameter determination for the quantization process and the data compression process in steps 2 and 4 may not be performed every time the process is repeated, and the parameters determined in the previous epoch may be used. In this case, the parameters may be redetermined when a predetermined trigger condition is satisfied. Examples of the trigger condition include a change in the communication bandwidth, throughput, or communication cost between the active NF 110 and the passive NF 120, a change in the processing capabilities of the active NF 110 and the passive NF 120, or a change in any of the target values.

[0087] <Configuration example of information processing device>

[0088] 8 is a diagram showing an example of the configuration of an information processing device that can operate as each of NFs 11a to 11k, an OAM terminal, and an external server. In FIG. 2, the information processing device 20 can be configured using a dedicated or general-purpose information processing device (computer) such as a personal computer (PC), a workstation (WS), or a server machine. However, the information processing device 20 may also be a collection (cloud) of one or more computers.

[0089] The information processing device 20 includes a processor 21 as a processing unit or control unit (controller), a storage device 22, a communication interface 23 (communication IF23), an input device 24, and a display 25, all of which are interconnected via a bus 26.

[0090] The processor 21 is typically a CPU, but may be a processor other than a CPU, such as a DSP or GPU, or may be combined with a CPU. Furthermore, the processing and operations by the processor 21 (the processing and operations as each NF performed by the execution of a program by the processor 21) may be performed using a semiconductor device (hardware) such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit), or may be performed using a combination of a processor and hardware, such as an SoC (System On a Chip). The processor 21 (controller), FPGA, ASIC, and SoC are each an example of a "circuitry."

[0091] The storage device 22 includes a main storage device and an auxiliary storage device. The main storage device is used as at least one of a storage area for programs and data, a program development area, a program work area, and a buffer area for communication data. The main storage device is configured with RAM (Random Access Memory) or a combination of RAM and ROM (Read Only Memory). The auxiliary storage device is used as a storage area for data and programs. A non-volatile storage medium is used as the auxiliary storage device. Examples of non-volatile storage media include a hard disk, a solid state drive (SSD), a flash memory, and an EEPROM (Electrically Erasable Programmable Read-Only Memory). The storage device 22 may also include a drive device for a disk recording medium.

[0092] An information processing device that participates in vertical federated learning as an active participant stores a local model (top model) and the correct labels of each sample in the storage device 22. An information processing device that participates in vertical federated learning as a passive participant stores a local model (bottom model) and learning sample data in the storage device 22.

[0093] The communication IF 23 is a circuit that performs communication processing. For example, the communication IF 23 is a network interface card (NIC). The communication IF 23 may also be a wireless communication circuit that performs wireless communication (5G, wireless LAN (Wi-Fi (registered trademark)), BLE, etc.). The communication IF 23 may also be a combination of a circuit that performs wired communication processing and a wireless communication circuit.

[0094] The input device 24 includes keys, buttons, a pointing device, a touch panel, etc., and is used to input information. The display 25 is, for example, a liquid crystal display, etc., and displays information and data.

[0095] The processor 21 performs various processes by executing various programs stored in the storage device 22. By the processor 21 executing the programs stored in the storage device 22, the information processing device 20 can operate as each of the NFs 11a to 11k, an OAM terminal, and an external server.

[0096] Other Embodiments The above-described embodiments and modifications are merely examples, and the present disclosure may be modified as appropriate without departing from the spirit and scope of the present disclosure. Furthermore, the processes and means described in the present disclosure may be freely combined and implemented as long as no technical contradiction arises.

[0097] In the above embodiment, intermediate results and gradient differences are transmitted between active and passive participants during forward and back propagation processing in the learning phase. However, the above method can also be used to transmit intermediate results during forward propagation processing in the estimation phase. Furthermore, the quantization and data compression parameters used in the estimation phase may be determined anew at the time of transmission, or the parameters used in the learning phase may be stored and the same parameters as those used in the learning phase may be used.

[0098] Furthermore, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by a single device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0099] The present disclosure can also be realized by supplying a computer program that implements the functions described in the above embodiments to a computer, and having one or more processors in the computer read and execute the program. Such a computer program may be provided to the computer on a non-transitory computer-readable storage medium connectable to the computer's system bus or via a network. Non-transitory computer-readable storage media include any type of medium suitable for storing electronic instructions, such as any type of disk, including magnetic disks (e.g., floppy disks, hard disk drives (HDDs), etc.), optical disks (e.g., CD-ROMs, DVDs, Blu-ray disks), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic cards, flash memory, or optical cards.

[0100] 110: Active NF (active participant) 120: Passive NF (passive participant) 200: Information processing system 210: Active participant 211: Local model (top model) 220A to 220C: Passive participants 221A to 221C: Local model (bottom model)

Claims

1. An information processing device that participates in vertical federation learning as a passive participant, comprising: a processor; and a storage device that stores learning sample data and a local model, wherein the processor executes the steps of: inputting the learning sample data into the local model to calculate an intermediate result; and transmitting the difference between the intermediate result in the current epoch and the intermediate result in the previous epoch to active participants.

2. The information processing device according to claim 1, characterized in that in the transmitting step, the entire intermediate result is transmitted to the active participants in the initial epoch, and in the second or subsequent epoch, the difference between the intermediate result in the current epoch and the intermediate result in the previous epoch is transmitted to the active participants.

3. The information processing device according to claim 1, characterized in that the processor further executes a step of calculating a data-reduced intermediate result difference by applying at least one of a quantization process and a data compression process to the intermediate result difference, and in the transmitting step, transmits the data-reduced intermediate result difference to the active participants.

4. The information processing device of claim 3, wherein the processor further executes a step of determining parameters to be applied to at least one of the quantization process and the data compression process, and the parameters are determined so that: (1) the communication cost associated with transmitting the data-reduced intermediate result differential is smaller than a first target value; (2) the communication delay associated with transmitting the data-reduced intermediate result differential is smaller than a second target value based on the communication bandwidth or throughput between the active participant and the data size of the data-reduced intermediate result differential; or (3) the processing time associated with the quantization process and data compression process is smaller than a third target value based on the processing capacity of the passive participant and the data size of the data-reduced intermediate result differential.

5. The information processing device described in claim 4, characterized in that the processor redetermines the parameters when there is a change in at least one of the communication bandwidth, throughput, or communication cost between the active participant, the processing capacity of the passive participant, the first target value, the second target value, and the third target value.

6. The information processing apparatus according to claim 3, wherein said processor further executes the step of receiving parameters to be applied to at least one of said quantization process and said data compression process from said active participants.

7. The information processing device described in claim 3, characterized in that the processor further executes a step of determining parameters to be applied to at least one of the quantization process and the data compression process, and the parameters are determined by selecting one of a first parameter determined by the passive participant and a second parameter determined by the active participant.

8. The information processing device according to claim 7, characterized in that the first parameter and the second parameter are each associated with a performance index, and selection of either the first parameter or the second parameter is performed by selecting the parameter associated with the performance index of the highest priority.

9. An information processing device that participates in vertical federated learning as an active participant, comprising: a processor; and a storage device that stores a correct label and a local model, wherein the processor executes the steps of: receiving, from a passive participant, a difference between an intermediate result in a current epoch and an intermediate result in a previous epoch; reconstructing the intermediate result in the current epoch based on the difference between the intermediate result; and inputting the reconstructed intermediate result into the local model and calculating an error from the correct label.

10. The information processing device according to claim 9, characterized in that in the receiving step, the entire intermediate result is received from the passive participant in the initial epoch, and in the second or subsequent epoch, the difference between the intermediate result in the current epoch and the intermediate result in the previous epoch is received from the passive participant.

11. The information processing device according to claim 9, characterized in that in the receiving step, an intermediate result difference is received in which data has been reduced by applying at least one of a quantization process and a data compression process to the intermediate result difference, and the processor applies at least one of an inverse quantization process and a data restoration process to the data-reduced intermediate result difference.

12. The information processing device described in claim 11, characterized in that the processor further executes a step of determining parameters to be applied to at least one of the quantization process and the data compression process, and the parameters are determined so that (1) the processing time for the inverse quantization process and the data restoration process is smaller than a fourth target value based on the processing capacity of the active participants and the data size of the intermediate result difference after data reduction, (2) the learning accuracy is higher than a fifth target value, or (3) the learning error is lower than a sixth target value.

13. The information processing device described in claim 12, characterized in that the processor redetermines the parameters when there is a change in at least one of the processing capacity of the active participant, the fourth target value, the fifth target value, and the sixth target value.

14. The information processing device according to claim 11, wherein the processor further executes the step of receiving, from the passive participant, parameters to be applied to at least one of the inverse quantization process and the data restoration process.

15. The information processing device described in claim 11, characterized in that the processor further executes a step of determining parameters to be applied to at least one of the inverse quantization process and the data restoration process, and the parameters are determined by selecting one of a first parameter determined by the passive participant and a second parameter determined by the active participant.

16. An information processing device according to claim 15, characterized in that the first parameter and the second parameter are each associated with a performance index, and selection of either the first parameter or the second parameter is performed by selecting the parameter associated with the performance index of the highest priority.

17. An information processing device that participates in vertical federated learning as an active participant, comprising: a processor; and a storage device that stores a correct label and a local model, wherein the processor executes the following steps: receiving data related to an intermediate result from a passive participant; inputting the data related to the intermediate result into the local model and calculating an error from the correct label; back-propagating the error to calculate a gradient; updating the local model based on the gradient; and transmitting the difference between the gradient in the current epoch and the gradient in the previous epoch to the passive participant.

18. The information processing device described in claim 17, characterized in that in the transmitting step, the entire gradient is transmitted to the passive participant in the initial epoch, and in the second or subsequent epoch, the difference between the gradient in the current epoch and the gradient in the previous epoch is transmitted to the passive participant.

19. The information processing device described in claim 17, characterized in that the processor further executes a step of applying at least one of a quantization process and a data compression process to the gradient difference to calculate a data-reduced gradient difference, and in the transmitting step, transmits the data-reduced gradient difference to the passive participant.

20. The information processing device of claim 19, wherein the processor further executes a step of determining parameters to be applied to at least one of the quantization process and the data compression process, and the parameters are determined so that: (1) the communication cost associated with transmitting the data-reduced gradient difference is smaller than a seventh target value; (2) the communication delay associated with transmitting the data-reduced gradient difference is smaller than an eighth target value based on the communication bandwidth or throughput with the passive participant and the data size of the data-reduced gradient difference; or (3) the processing time associated with the quantization process and the data compression process is smaller than a ninth target value based on the processing capacity of the active participant and the data size of the data-reduced gradient difference.

21. The information processing device described in claim 20, characterized in that the processor redetermines the parameters when there is a change in at least one of the communication bandwidth, throughput, or communication cost between the passive participant, the processing capacity of the active participant, the seventh target value, the eighth target value, and the ninth target value.

22. The information processing device according to claim 19, wherein the processor further performs the step of receiving, from the passive participant, parameters to be applied to at least one of the quantization process and the data compression process.

23. The information processing device described in claim 19, characterized in that the processor further executes a step of determining parameters to be applied to at least one of the quantization process and the data compression process, and the parameters are determined by selecting one of a first parameter determined by the active participant and a second parameter determined by the passive participant.

24. An information processing device according to claim 23, characterized in that the first parameter and the second parameter are each associated with a performance index, and selection of either the first parameter or the second parameter is performed by selecting the parameter associated with the performance index of the highest priority.

25. An information processing device that participates in vertical federated learning as a passive participant, comprising: a processor; and a storage device that stores learning sample data and a local model, wherein the processor executes the steps of: receiving, from an active participant, a difference between a gradient in a current epoch and a gradient in a previous epoch; reconstructing a gradient in the current epoch based on the difference in gradient; and updating the local model based on the reconstructed gradient.

26. The information processing device according to claim 25, wherein in the receiving step, the entire gradient is received from the active participant in the initial epoch, and in the second or subsequent epoch, the difference between the gradient in the current epoch and the gradient in the previous epoch is received from the active participant.

27. The information processing device according to claim 25, wherein the receiving step receives gradient differences that have been subjected to data reduction by applying at least one of quantization processing and data compression processing to the gradient differences, and the processor applies at least one of inverse quantization processing and data restoration processing to the data-reduced gradient differences.

28. The information processing device described in claim 27, characterized in that the processor further executes a step of determining parameters to be applied to at least one of the quantization process and the data compression process, and the parameters are determined based on the processing capacity of the passive participant and the data size of the data-reduced gradient difference so that the processing time for the inverse quantization process and the data restoration process is smaller than an eleventh target value.

29. The information processing device according to claim 28, wherein the processor redetermines the parameters when there is a change in at least one of the processing capacity of the passive participant and the eleventh target value.

30. The information processing device according to claim 27, wherein the processor further performs the step of receiving, from the passive participant, parameters to be applied to at least one of the inverse quantization process and the data restoration process.

31. The information processing device described in claim 27, characterized in that the processor further executes a step of determining parameters to be applied to at least one of the inverse quantization process and the data restoration process, and the parameters are determined by selecting one of a first parameter determined by the passive participant and a second parameter determined by the active participant.

32. An information processing device according to claim 31, characterized in that the first parameter and the second parameter are each associated with a performance index, and selection of either the first parameter or the second parameter is performed by selecting the parameter associated with the performance index of the highest priority.

33. An information processing method performed by an information processing device that has a storage device that stores learning sample data and a local model and that participates in vertical federated learning as a passive participant, the information processing method comprising: a step of inputting the learning sample data into the local model to calculate an intermediate result; and a step of transmitting the difference between the intermediate result in the current epoch and the intermediate result in the previous epoch to active participants.

34. An information processing method performed by an information processing device that has a storage device that stores correct labels and local models and that participates in vertical federated learning as an active participant, the information processing method comprising the steps of: receiving, from a passive participant, the difference between an intermediate result in the current epoch and the intermediate result in the previous epoch; reconstructing the intermediate result in the current epoch based on the difference between the intermediate results; and inputting the reconstructed intermediate result into the local model and calculating an error from the correct label.

35. An information processing method performed by an information processing device that has a storage device that stores correct labels and local models and that participates in vertical federated learning as an active participant, the information processing method comprising the steps of: receiving data related to intermediate results from passive participants; inputting the data related to the intermediate results into the local model and calculating an error from the correct label; backpropagating the error to calculate a gradient; updating the local model based on the gradient; and transmitting the difference between the gradient in the current epoch and the gradient in the previous epoch to passive participants.

36. An information processing method performed by an information processing device that has a storage device that stores learning sample data and a local model and participates in vertical federated learning as a passive participant, the information processing method comprising the steps of: receiving, from an active participant, a difference between a gradient in a current epoch and a gradient in a previous epoch; reconstructing a gradient in the current epoch based on the difference in gradient; and updating the local model based on the reconstructed gradient.

Images