Systems and methods for implementing vertically associated learning

Abstract

Methods and servers for performing vertical federated learning (VFL) over a communications network are disclosed. Some methods include encoder and data selection, encoder customization, and VFL configuration. Encoder and data selection includes receiving a VFL request and parsing it to determine data requirements and encoder requirements, selecting a first encoder and a second encoder based on the encoder requirements, and selecting data sources for the first encoder and the second encoder based on the data requirements. Encoder customization includes selecting a first encoder and a second encoder and customizing the first encoder by muting portions of the first encoder. VFL configuration includes receiving labeled data information and an indication of a loss function, configuring an evaluator using the labeled data information and the indication of the loss function, configuring the encoder output to a combined classifier, and configuring a privacy router.

Description

Technical Field

[0001] The present disclosure relates to the field of federated learning, and more particularly, to a system and method for performing vertical federated learning.

Background Art

[0002] In telecommunications, "6G" is the sixth generation standard currently under development. The 6G standard is for wireless communication technologies that support cellular data networks. 6G networks exhibit more heterogeneity than their predecessors such as, for example, 5G networks, and support applications beyond current mobile usage scenarios such as virtual reality and augmented reality, ubiquitous instant communication, universal intelligence, and the Internet of Things (IoT). Mobile network operators can adopt a flexible distributed business model for 6G networks using local spectrum licensing, spectrum sharing, infrastructure sharing, and intelligent automation supported by mobile edge computing, artificial intelligence (AI), short packet communication, and blockchain technology.

[0003] Federated learning (FL) is one of the various machine learning (ML) techniques that can be utilized to enable 6G network capabilities. Broadly speaking, FL is an ML technique that trains one or more AI models across multiple distributed participants (e.g., edge devices or servers) using their local datasets without raw data exchange between them. This approach is in contrast to conventional "centralized" ML techniques that assume either all local datasets are uploaded to a single executor or sharing of raw local data samples among distributed participants.

[0004] In other words, FL enables multiple participants to build combined AI models without sharing data, and therefore allows them to address important issues such as data privacy, data security, data access rights, and access to heterogeneous data. There are two main methods for performing FL: horizontal federated learning (HFL) and vertical federated learning (VFL).

[0005] Broadly speaking, HFL is used for datasets of multiple participants that share the same feature space but different identity spaces. On the one hand, the feature space represents the set of attributes / dimensions of each data entity / sample within a single dataset. On the other hand, the identity space represents the identity of each data entity. For example, in a bank dataset, the feature space could be (account balance, mortgage amount, credit score, etc.), and the identity space could be the user list for all accounts.

[0006] In contrast to HFL, VFL is used for datasets of multiple participants that share the same ID space but different feature spaces. For example, Bell TM and Amazon TM It has many overlapping entities in the ID space (e.g., the same user in the real world), but with different data characteristics (e.g., Bell TM Mobile usage features hosted by Amazon TM The participants have online shopping features hosted by [platform name]. If one or both of these participants want to train a combined ML model to evaluate a given user's credit score by considering the influence of both mobile usage features and online shopping features, VFL can be used to train the combined ML model without leaking raw data and local model information between the two participants.

[0007] Since VFL participants may not belong to the same business or application category, their local ML models typically differ in structure and hyperparameters. Therefore, a better VFL architecture is needed that can be deployed for and used by a wide variety of participants. [Overview of the Initiative]

[0008] The purpose of this technology is to improve at least some of the shortcomings present in conventional VFL architectures.

[0009] Typical VFL architecture Referring to Figure 1, a typical VFL architecture 100 is shown, having a first participant having a first ML model 102 and a second participant having a second ML model 104. The first ML model 102 and the second ML model 104 together form a combined model 110.

[0010] In contrast to HFL, two specific features may be required to train and use the VFL architecture 100, namely, an intermediate result exchange 106 between the ML model and the collaborator 108.

[0011] According to the VFL paradigm, in each training step and each in-use step of the combined model 110, the intermediate output or computation results of the first local ML model 102 and the second local ML model 104 may first need to be encrypted by homomorphic encryption and then transmitted to the other of the first local ML model 102 and the second local ML model 104 so that they can perform local computations. In the intermediate result exchange process 106, each of the combined first local ML model 102 and the second local ML model 104 does not decrypt the intermediate results from the other of the first local ML model 102 and the second local ML model 104; that is, they perform local computations on the homomorphic encrypted data.

[0012] To ensure data confidentiality during the training process, a collaborator 108 is required to perform VFL. For example, collaborator 108 may be a third-party entity trusted by the first and second participants. In non-limiting examples, collaborator 108 may be embodied as a secure computing node. Collaborator 108 creates an encryption pair and sends the public key to the first and second participants to enable the encrypted intermediate result exchange process 106. Once the intermediate result exchange process 106 is completed for each training and use step of the combined model 110, the first and second participants send their encrypted outputs and locally generated masks to collaborator 108. Collaborator 208 decrypts their masked outputs with its private key, calculates the decrypted gradients and losses, and returns this information to each participant. Each participant unmasks the gradients and losses from collaborator 208 and updates its respective local ML model parameters accordingly.

[0013] Developers recognize that VFL may need to be supported by the NET4AI architecture to provide native support for the training and in-use phases of AI-based computing services. Generally speaking, NET4AI is a service-oriented architecture for 6G wireless systems that provides end-to-end support for AI applications from deployment to operation. The NET4AI architecture is generally described in the paper “Nine Challenges in Artificial Intelligence and Wireless Communications for 6G” by Wen Tong and Geoffry Ye Li, published in 2021 (DOI 10.1109 / MWC.006.2100543), the entirety of which is incorporated herein by reference. Supporting VFL in the NET4AI architecture is a challenging task.

[0014] Developers recognize that in existing VFL paradigms, VFL customers can access information indicating the number of other VFL participants, the data characteristics of other VFL participants, the datasets of other VFL participants, labeled data locations / ownership, and the encoder structures of other VFL participants. However, when implementing automated VFL systems, VFL customers may not have access to information about other VFL participants. Therefore, the developers of this technology recognize the need for a system and method that allows the NET4AI architecture to select datasets and encoders with the necessary functionality for VFL customers.

[0015] Developers also recognize that in the existing VFL paradigm, labeled data is provided by a single VFL participant, and the loss function can be split without violating homomorphic encryption. As a result, a privacy-preserving intermediate result exchange process can be performed to compute the combined loss. However, in some cases, the labeled data may be owned, at least partially, by multiple participants. Also, some loss functions may be different from linear polynomial functions that ensure homomorphic encryption after computation. Therefore, developers recognize that it may be necessary to configure combined classifiers and evaluators for different encoders. This may simplify the interaction between local classifiers of different VFL participants, and / or support loss functions that violate homomorphic encryption, and / or allow labeled data to be provided by multiple data sources.

[0016] Developers also recognize that in existing VFL paradigms, the encoder structure remains the same, maintained by the corresponding AI enabler. These solutions lack procedures for customizing the encoder structure in specific cases, such as when collaboratively training and using multiple encoders to form a combined model. However, when applying VFL in the NET4AI architecture or other architectures, the training data of selected participants may have overlapping data features.

[0017] For example, in a VFL model with a "bank" participant and a "mobile operator" participant, the data from both participants may share a common feature named "mobile bill payment record." In this example, if the VFL combined model is formed by the original encoders of the two participants, the weight of the feature "mobile bill payment record" in the combined output may increase significantly due to the overlap in the inputs of both models. In the same example, the weights of other features may decrease in the combined output without being duplicated.

[0018] Developers recognize that while setting strict convergence requirements can mitigate the problem of relatively increased weights from overlapping features, this can significantly increase the overall training cost. Therefore, developers recognize that customizing the encoder structure may be desirable to mitigate drawbacks such as relatively increased weights and slow convergence speeds caused, for example, by overlapping data features.

[0019] Typical encoder-classifier model Referring to Figure 2, a simplified representation of a typical local ML model 200 is shown. For example, the local ML model 200 could be a deep neural network (DNN) run by each computing service (e.g., a participant).

[0020] The local ML model 200 includes an encoder 202 and a classifier 204, each containing numerous layers. As seen in Figure 2, layers 2-4 are sometimes referred to as "hidden" layers. The encoder 202 includes an input layer 206 and numerous hidden layers, while the classifier 204 includes the remaining hidden layers and an output layer.

[0021] The encoder 202 captures the low-order features of the local ML model 200, which can be shared with other local ML models implemented in a similar manner (e.g., local ML models aimed at solving similar problems). The classifier 204 captures the higher-order features of the local ML model 200, which are typically specific to the target. For the input layer 206 of the encoder 202, each neuron corresponds to one feature of the input data. The dimension of the input layer 206, i.e., the number of neurons in the input layer 206, is equal to the number of features associated with the input data of the local ML model 200.

[0022] The developers of this technology recognize that when performing automated VFL, the encoder can be a network-supplied encoder, i.e., it can be provided by the network and implemented by a network entity called an "AI Enabler". During the execution of automated VFL, the classifier may be at least one of the following: (i) a customer-supplied classifier, i.e., a local classifier provided by a participant in the corresponding ML Enabler, and (ii) a network-supplied classifier, i.e., a combined classifier provided by an orchestrator in the NET4AI architecture or other architecture. Both customer-supplied and network-supplied classifiers may be implemented via the service functions of a computing service without deviating from the scope of this technology.

[0023] The developers of this technology recognize that neural network (NN) pruning techniques can be used to reduce the size of a neural network (NN) by "muting" specific neurons or links within the NN while maintaining the same or similar performance as the corresponding full-size NN. Muting a neuron can be achieved by setting its output to zero for any of its inputs. Muting a link can be achieved by setting the link's weight value to zero.

[0024] Typically, during NN pruning, neurons in a given input layer are not muted. However, in some embodiments of the present technology, at least some neurons in at least some input layers of each VFL participant's encoder may be muted to reduce the harmful effects caused by overlapping data features. The process of changing the encoder's structure by muting specific neurons or links in a given layer (potentially the input layer) is sometimes referred to as encoder customization procedure or encoder pruning.

[0025] In some embodiments of the present technology, a VFL participant includes an encoder, a local classifier (if available), and a corresponding data source. The data source of a VFL participant may include multiple data sets realized at different network locations. All data entities provided by the data source of a VFL participant may have the same format and the same features. Since each encoder is associated with an application category, the data source of a VFL participant may be associated with the same application category as the VFL participant's encoder.

[0026] The present technology In a first broad aspect of the present technology, a step of receiving a VFL request from a vertical federated learning (VFL) customer by an encoder selector, and a step of determining, by the encoder selector, data requirements and encoder requirements for solving a VFL task according to the VFL request, where the VFL task is divisible into a plurality of subtasks, and the plurality of subtasks includes at least a first subtask and a second subtask, a step of selecting, by the encoder selector, a first encoder and a second encoder from a pool of encoders in a communication network based on the encoder requirements, where the first encoder is configured to process a first set of features for solving the first subtask, and the second encoder is configured to process a second set of features for solving the second subtask, and a step of selecting, by the encoder selector, a first data source for the first encoder and a second data source for the second encoder based on the data requirements, where the selected data sources are from a pool of data sources, the first data source includes the first set of features, and the second data source includes the second set of features, are provided.

[0027] In some embodiments of the method, the data requirements indicate requirements for selecting a first data source for solving the first subtask and a second data source for solving the second subtask.

[0028] In some embodiments of the method, the encoder requirements indicate a first encoder structure to be used for solving the first subtask and a second encoder structure to be used for solving the second subtask.

[0029] In some embodiments of the method, the method further includes a step of dividing the VFL task into a plurality of subtasks by the encoder selector.

[0030] In some embodiments of the method, selecting a first encoder involves an encoder selector determining an application category for a first subtask based on encoder requirements, the application category being stored in memory in association with the first encoder.

[0031] In some embodiments of the method, selecting a first encoder includes using an encoder selector to identify a pool of encoders having a first encoder structure, and using an encoder selector to select a first encoder from the pool of encoders having a first encoder structure.

[0032] In some embodiments of the method, selecting a first data source and a second data source includes the encoder selector sending a request for data source information to a data source manager, the request including data requirements; the encoder selector receiving data source information from the data source manager indicating the quality and characteristics of the data contained in each of the pools of data sources; and the encoder selector using the data source information to select a first data source for a first encoder and a second data source for a second encoder.

[0033] In some embodiments of the method, the first data source and the second data source are associated with their respective IDs, and the method further includes the steps of: transmitting the respective IDs of the first data source and the second data source to a data source manager via an encoder selector; and receiving the network locations of the first data source and the second data source from the data source manager via the encoder selector.

[0034] In a second broader aspect of the present technology, a method is provided comprising the step of customizing a first encoder by muting a portion of the first encoder using an encoder customizer, thereby generating a customized first encoder. The first and second encoders are selected from a pool of encoders in a communication network based on encoder requirements, which are determined to solve a VFL task according to a Vertical Federated Learning (VFL) requirement. The VFL task can be divided into a plurality of subtasks, each subtask including at least a first subtask and a second subtask. The first encoder is configured to process a first set of features for solving the first subtask, and the second encoder is configured to process a second set of features for solving the second subtask.

[0035] In some embodiments of the method, customization includes determining overlapping features between a first set of features and a second set of features, and muting a portion of the first encoder that processes the overlapping features.

[0036] In some embodiments of the method, customization includes determining, by an encoder customizer, that the first encoder should be customized.

[0037] In some embodiments of the method, the encoder customizer's determination that the first encoder should be customized includes the encoder customizer's determination that the first encoder should be customized based on data feature weights, the data feature weights being received from a reference AI enabler.

[0038] In some embodiments of the method, customizing the first encoder includes muting a portion of the first encoder that processes overlapping features by the encoder customizer. Muting includes muting at least one of the neurons and links of the first encoder.

[0039] A third broad aspect of the present technology provides a method comprising the steps of configuring at least one of an evaluator and a combined classifier using a Vertical Federated Learning (VFL) configurator, and configuring a privacy router using a VFL configurator to determine at least one of the following: a data exchange path between the combined classifier and the evaluator, a labeled data merging path between the evaluator and one or more data sources, and a backpropagation path between the combined classifier and a customized first encoder and a second encoder for backpropagating gradient values, wherein the combined classifier, evaluator and privacy router are configured to perform VFL.

[0040] In some embodiments of the method, configuring an evaluator includes: receiving labeled data information from an encoder selector using a VFL configurator; determining a loss function to be used to perform VFL using a VFL configurator; and configuring an evaluator using the labeled data information and the loss function using a VFL configurator.

[0041] In some embodiments of the method, configuring the evaluator involves the VFL configurator configuring a combination rule of labeled data and loss function resulting from data sources selected for the first and second encoders.

[0042] In some embodiments of the method, configuring a coupled classifier includes, by the VFL configurator, receiving information from an encoder customizer indicating the output layer structures of a first encoder and a second encoder, and by the VFL configurator configuring encoder outputs to the coupled classifier based on the output layer structures of the first encoder and the second encoder.

[0043] In some embodiments of the method, the second encoder is a customized second encoder.

[0044] In a fourth broader aspect of this technology, an encoder selector is provided that receives a VFL request from a Vertical Federated Learning (VFL) customer, determines data requirements and encoder requirements for solving a VFL task according to the VFL request, wherein the VFL task can be divided into multiple subtasks, each subtask including at least a first and a second subtask, selects a first encoder and a second encoder from a pool of encoders in a communication network based on the encoder requirements, wherein the first encoder is configured to process a first set of features for solving the first subtask, and the second encoder is configured to process a second set of features for solving the second subtask, and selects a first data source for the first encoder and a second data source for the second encoder based on the data requirements, wherein the selected data sources are from a pool of data sources, the first data source includes a first set of features, and the second data source includes a second set of features.

[0045] In some embodiments of the encoder selector, the data requirements indicate the requirements for selecting a first data source for solving a first subtask and a second data source for solving a second subtask.

[0046] In some embodiments of the encoder selector, the encoder requirement includes a first encoder structure to be used to solve a first subtask and a second encoder structure to be used to solve a second subtask.

[0047] In some embodiments of the encoder selector, the encoder selector is further configured to divide the VFL task into multiple subtasks.

[0048] In some embodiments of the encoder selector, selecting a first encoder includes the encoder selector being configured to determine an application category for a first subtask based on encoder requirements, and the application category being stored in memory in association with the first encoder.

[0049] In some embodiments of the encoder selector, selecting a first encoder includes the encoder selector being configured to identify a pool of encoders having a first encoder structure and to select a first encoder from the pool of encoders having a first encoder structure.

[0050] In some embodiments of the encoder selector, selecting a first data source and a second data source is configured to include the encoder selector sending a request for data source information to a data source manager, the request including data requirements; receiving data source information from the data source manager indicating the quality and characteristics of the data contained in each of the pools of data sources; and using the data source information to select a first data source for the first encoder and a second data source for the second encoder.

[0051] In some embodiments of the encoder selector, the first data source and the second data source are associated with their respective IDs, and the encoder selector is further configured to transmit the respective IDs of the first data source and the second data source to a data source manager and to receive the network locations of the first data source and the second data source from the data source manager.

[0052] In a fifth broad aspect of the present technology, an encoder customizer is provided which is configured to customize a first encoder by muting a portion of the first encoder, thereby generating a customized first encoder. The first and second encoders are selected from a pool of encoders in a communication network based on encoder requirements, which are determined to solve a VFL task according to a Vertical Federated Learning (VFL) requirement. The VFL task can be divided into multiple subtasks, each subtask including at least a first and a second subtask. The first encoder is configured to process a first set of features for solving the first subtask, and the second encoder is configured to process a second set of features for solving the second subtask.

[0053] In some embodiments of the encoder customizer, customization includes configuring the encoder customizer to determine overlapping features between a first set of features and a second set of features, and to mute a portion of the first encoder that processes the overlapping features.

[0054] In some embodiments of the encoder customizer, customization includes configuring the encoder customizer to determine that the first encoder should be customized.

[0055] In some embodiments of the encoder customizer, determining that a first encoder should be customized includes the encoder customizer being configured to determine that a first encoder should be customized based on data feature weights, the data feature weights being received from a reference AI enabler.

[0056] In some embodiments of the encoder customizer, customizing the first encoder includes configuring the encoder customizer to mute a portion of the first encoder that handles overlapping features, and muting includes muting at least one of the neurons and links of the first encoder.

[0057] In a sixth broad aspect of the technology, a VFL configurator is provided, configured to configure at least one of an evaluator and a coupled classifier, and to configure a privacy router to determine at least one of the following: a data exchange path between the coupled classifier and the evaluator, a labeled data merging path between the evaluator and one or more data sources, and a backpropagation path between the coupled classifier and a customized first encoder and a second encoder for backpropagating gradient values, wherein the coupled classifier, evaluator and privacy router are configured to perform Vertical Federated Learning (VFL).

[0058] In some embodiments of the VFL configurator, configuring an evaluator includes configuring the VFL configurator to receive labeled data information from an encoder selector, determine the loss function to be used to perform the VFL, and configure an evaluator using the labeled data information and the loss function.

[0059] In some embodiments of the VFL configurator, configuring an evaluator includes configuring the VFL configurator to configure combination rules of labeled data and loss functions resulting from data sources selected for a first encoder and a second encoder.

[0060] In some embodiments of the VFL configurator, configuring a coupled classifier includes the VFL configurator receiving information from an encoder customizer indicating the output layer structures of a first encoder and a second encoder, and configuring encoder outputs to the coupled classifier based on the output layer structures of the first encoder and the second encoder.

[0061] In some embodiments of the VFL configurator, the second encoder is a customized second encoder.

[0062] In a seventh broad aspect of this technology, the encoder selector receives a VFL request from a Vertical Federated Learning (VFL) customer; the encoder selector determines data requirements and encoder requirements for solving a VFL task according to the VFL request, wherein the VFL task can be divided into multiple subtasks, each subtask including at least a first subtask and a second subtask; the encoder selector selects a first encoder and a second encoder from a pool of encoders in a communication network based on the encoder requirements, wherein the first encoder is configured to process a first set of features for solving a first subtask, and the second encoder is configured to process a second set of features for solving a second subtask; and the encoder selector selects a first data source for the first encoder and a second data source for the second encoder based on the data requirements, wherein the selected data sources are selected from a pool of data sources. A method is provided which includes the steps of: a first data source including a first set of features and a second data source including a second set of features; customizing a first encoder by muting a portion of the first encoder using an encoder customizer to generate a customized encoder; configuring at least one of an evaluator and a combined classifier using a VFL configurator; and configuring a privacy router by the VFL configurator to determine at least one of a data exchange path between the combined classifier and the evaluator, a labeled data merge path between the evaluator and one or more of the selected data sources, and a backpropagation path between the combined classifier and the customized first and second encoders, respectively, for backpropagating gradient values, wherein the combined classifier, evaluator and privacy router are configured to perform VFL.

[0063] In some embodiments of the method, the data requirements indicate the requirements for selecting a first data source for solving a first subtask and a second data source for solving a second subtask.

[0064] In some embodiments of the method, the encoder requirements include a first encoder structure to be used to solve a first subtask and a second encoder structure to be used to solve a second subtask.

[0065] In some embodiments of the method, the method further includes the step of dividing a VFL task into multiple subtasks using an encoder selector.

[0066] In some embodiments of the method, selecting a first encoder involves an encoder selector determining an application category for a first subtask based on encoder requirements, the application category being stored in memory in association with the first encoder.

[0067] In some embodiments of the method, selecting a first encoder includes using an encoder selector to identify a pool of encoders having a first encoder structure, and using an encoder selector to select a first encoder from the pool of encoders having a first encoder structure.

[0068] In some embodiments of the method, selecting a first data source and a second data source includes the encoder selector sending a request for data source information to a data source manager, the request including data requirements; the encoder selector receiving data source information from the data source manager indicating the quality and characteristics of the data contained in each of the pools of data sources; and the encoder selector using the data source information to select a first data source for a first encoder and a second data source for a second encoder.

[0069] In some embodiments of the method, the first data source and the second data source are associated with their respective IDs. The method further includes the steps of: transmitting the respective IDs of the first data source and the second data source to a data source manager via an encoder selector; and receiving the network locations of the first data source and the second data source from the data source manager via the encoder selector.

[0070] In some embodiments of the method, customization includes determining overlapping features between a first set of features and a second set of features, and muting a portion of the first encoder that processes the overlapping features.

[0071] In some embodiments of the method, customization includes determining, by an encoder customizer, that the first encoder should be customized.

[0072] In some embodiments of the method, the encoder customizer determining that the first encoder should be customized includes the encoder customizer determining that the first encoder should be customized based on data feature weights, the data feature weights being received from a reference AI enabler.

[0073] In some embodiments of the method, customizing the first encoder includes muting a portion of the first encoder that processes overlapping features by an encoder customizer, and muting includes muting at least one of the neurons and links of the first encoder.

[0074] In some embodiments of the method, configuring an evaluator includes: receiving labeled data information from an encoder selector using a VFL configurator; determining a loss function to be used to perform VFL using a VFL configurator; and configuring an evaluator using the labeled data information and the loss function using a VFL configurator.

[0075] In some embodiments of the method, configuring the evaluator involves the VFL configurator configuring a combination rule of labeled data and loss function resulting from data sources selected for the first and second encoders.

[0076] In some embodiments of the method, configuring a coupled classifier includes, by the VFL configurator, receiving information from an encoder customizer indicating the output layer structures of a first encoder and a second encoder, and by the VFL configurator configuring encoder outputs to the coupled classifier based on the output layer structures of the first encoder and the second encoder.

[0077] In some embodiments of the method, the second encoder is a customized second encoder.

[0078] In an eighth broad aspect of this technology, the encoder selector receives a VFL request from a Vertical Federated Learning (VFL) customer, determines data requirements and encoder requirements for solving a VFL task according to the VFL request, wherein the VFL task can be divided into multiple subtasks, each subtask including at least a first and a second subtask, and the encoder selector selects a first encoder and a second encoder from a pool of encoders in a communication network based on the encoder requirements, wherein the first encoder is configured to process a first set of features for solving a first subtask, and the second encoder is configured to process a second set of features for solving a second subtask, and the encoder selector selects a first data source for the first encoder and a second data source for the second encoder based on the data requirements, wherein the selected data sources are from a pool of data sources. A system is provided in which a first data source includes a first set of features, a second data source includes a second set of features, an encoder customizer customizes the first encoder by muting a portion of the first encoder, thereby generating a customized encoder, a VFL configurator configures at least one of an evaluator and a combined classifier, and a privacy router is configured by the VFL configurator to determine at least one of the following: a data exchange path between the combined classifier and the evaluator, a labeled data merge path between the evaluator and one or more of the selected data sources, and a backpropagation path between the combined classifier and the customized first and second encoders, respectively, for backpropagating gradient values, wherein the combined classifier, evaluator, and privacy router are configured to run VFL.

[0079] In some embodiments of the system, the data requirements specify the requirements for selecting a first data source for solving a first subtask and a second data source for solving a second subtask.

[0080] In some embodiments of the system, the encoder requirements include a first encoder structure to be used to solve a first subtask and a second encoder structure to be used to solve a second subtask.

[0081] In some embodiments of the system, the encoder selector is further configured to divide the VFL task into multiple subtasks.

[0082] In some embodiments of the system, selecting a first encoder includes configuring an encoder selector to determine an application category for a first subtask based on encoder requirements, and the application category being stored in memory in association with the first encoder.

[0083] In some embodiments of the system, selecting a first encoder includes configuring an encoder selector to identify a pool of encoders having a first encoder structure and to select a first encoder from the pool of encoders having a first encoder structure.

[0084] In some embodiments of the system, selecting a first data source and a second data source is configured to include the encoder selector sending a request for data source information to a data source manager, the request including data requirements; receiving data source information from the data source manager indicating the quality and characteristics of the data contained in each of the pools of data sources; and using the data source information to select a first data source for the first encoder and a second data source for the second encoder.

[0085] In some embodiments of the system, the first data source and the second data source are associated with their respective IDs, and the encoder selector is further configured to transmit the respective IDs of the first data source and the second data source to the data source manager and to receive the network locations of the first data source and the second data source from the data source manager.

[0086] In some embodiments of the system, customization includes configuring the encoder customizer to determine overlapping features between a first set of features and a second set of features, and to mute a portion of the first encoder that processes the overlapping features.

[0087] In some embodiments of the system, customization is configured such that the encoder customizer obtains (i) encoder structure parameters of the first encoder and the second encoder, (ii) the location of a target AI enabler for the customized encoder, and (iii) the location of a reference AI enabler for a trained encoder; sends a request for data feature weights for the trained encoder to the reference AI enabler; receives data feature weights from the reference AI enabler; determines overlapping features between a first set of features of the first encoder and a second set of features of the second encoder; determines, based on the data feature weights, that the first encoder should be customized; and mutes the portion of the first encoder that handles the overlapping features, wherein muting includes muting at least one of the neurons and links of the first encoder.

[0088] In some embodiments of the system, configuring an evaluator includes configuring a VFL configurator to receive labeled data information from an encoder selector, determine the loss function to be used to perform the VFL, and configure an evaluator using the labeled data information and the loss function.

[0089] In some embodiments of the system, configuring the evaluator includes configuring the VFL configurator to configure combination rules of labeled data and loss functions resulting from data sources selected for the first and second encoders.

[0090] In some embodiments of the system, configuring a coupled classifier includes configuring the VFL configurator to receive information from an encoder customizer indicating the output layer structures of a first encoder and a second encoder, and to configure encoder outputs to the coupled classifier based on the output layer structures of the first encoder and the second encoder.

[0091] In some embodiments of the system, the second encoder is a customized second encoder.

[0092] Each implementation of this technology has at least one of the above-described objectives and / or embodiments, but does not necessarily have all of them. It should be understood that some embodiments of this technology resulting from attempts to achieve the above-described objectives may not satisfy these objectives and / or may satisfy other objectives not specifically described herein.

[0093] Additional and / or alternative features, embodiments, and advantages of the implementation of this technology will become apparent from the following description, the attached drawings, and the attached claims. [Brief explanation of the drawing]

[0094] Embodiments of this disclosure will be described only as examples with reference to the accompanying drawings. [Figure 1] A schematic diagram of a typical VFL architecture with two participants is shown. [Figure 2] A schematic diagram of a typical local ML model is shown. [Figure 3] A schematic diagram of an automated VFL architecture, which is possible in some embodiments of this technology, is shown. [Figure 4] Figure 3 shows a schematic diagram of the privacy router in the automated VFL architecture. [Figure 5] Figure 3 shows a schematic diagram of the encoder and data selection procedure performed by the encoder selector of the VFL architecture. [Figure 6] Figure 3 shows a schematic diagram of the encoder customization procedure performed by the encoder customizer in the VFL architecture. [Figure 7] Figure 6 shows two customization modes for the encoder customization procedure. [Figure 8] Figure 3 shows a schematic diagram of the VFL configuration procedure performed by the VFL configurator for the VFL architecture. [Figure 9] This document presents three VFL frameworks, including conventional VFL, automated VFL, and stitched single-NN. [Figure 10] Figure 9 shows a performance comparison between conventional VFL and automated VFL applied to two test datasets. [Figure 11] Figure 9 shows a performance comparison between automated VFL and stitched single NN applied to two test datasets. [Modes for carrying out the invention]

[0095] Broadly speaking, sixth-generation (6G) communication systems are end-to-end (E2E) systems that support AI-based services and applications in a variety of contexts and scenarios. For example, 6G network components can natively integrate communication, computing, and sensing capabilities to facilitate the transition from centralized intelligence in the cloud to universal intelligence on the "deep" edge. This integration can be enabled by what is referred to herein as the "Network for AI" (NET4AI) architecture. The NET4AI architecture may enable the delivery of AI as a Service (AIaaS). However, as will become apparent from the following description herein, at least some aspects of this technology may be embodied in a wide variety of AIaaS architectures.

[0096] In the context of this technology, the developers have developed a method and system for automating Vertical Federated Learning (VFL). Generally speaking, VFL is a machine learning (ML) technique that trains a combined model using heterogeneous AI models and datasets belonging to different participants. The developers of this technology recognize that supporting automated VFL in the NET4AI architecture or other architectures may be beneficial.

[0097] According to some non-limiting embodiments of this technology, automated VFL methods and systems are provided for customers to use other customers' data and computing resources, even if they belong to different organizations or even industrial sectors. As will be described in more detail below herein, VFL automation may be enabled using at least some of the following functions: encoder and data selection, encoder customization, and VFL configuration.

[0098] In further non-limiting embodiments of this technology, an automated VFL solution is provided that supports the customization of AI models. Broadly speaking, AI model customization is a function of the automated VFL solution to improve the scalability of AI models across heterogeneous devices and scenarios. As will be described in more detail below herein, AI model customization may be enabled via an encoder customization function.

[0099] Automated VFL architecture Referring to Figure 3, a non-limiting embodiment of the automated VFL architecture 300 is shown. In this embodiment, the automated VFL architecture 300 includes an application layer 302, a network layer 304, and a data layer 306.

[0100] Broadly speaking, the various layers of the Automated VFL Architecture 300 are part of a framework used to describe the functionality of a networking system that implements VFL. Different layers can be used to characterize the computing functions performed by one or more networking nodes (e.g., servers) in order to at least partially support interoperability between the hardware and software components of the Automated VFL Architecture 300.

[0101] In this embodiment, the application layer 302 includes a VFL customer 310. However, the application layer 302 may include multiple VFL customers without departing from the scope of the Art. In this embodiment, the VFL customer 310 is configured to maintain the coupling classifier 314 and provide VFL requests, such as VFL requests 312, to the components of the network layer 304.

[0102] In this embodiment, the data layer 306 includes a plurality of data sources 342 that are potentially used during automated VFL. An ID sorting function 344 may be enabled in the data layer 306. Broadly speaking, the data sources may be configured as logical network entity storage for storing and / or maintaining the raw data (as VFL model input) and labeled data (as benchmarks for calculating training losses) necessary for the Technique to train the VFL model. The entire batch of raw or labeled data necessary to train the VFL model may be provided by one or more data sources. In some embodiments, the data sources may be deployed centrally in a network entity or distributed across multiple entities without departing from the scope of the Technique.

[0103] In the non-limiting embodiments shown, the network layer 304 maintains an AI hyperparameter optimizer 320, a plurality of encoders 328, an evaluator 330, and a privacy router 332 (transfer plane). In the context of this technology, the AI ​​hyperparameter optimizer 320 is configured to perform one or more computer implementation procedures to enable the following functions: encoder and data selection, encoder customization, and VFL configuration. In the non-limiting embodiments shown, the AI ​​hyperparameter optimizer 320 includes an encoder selector 322, an encoder customizer 324, and a VFL configurator 326, respectively, to enable the above functions. The components of the AI ​​hyperparameter optimizer 320 will now be described in order.

[0104] Encoder selector In the non-limiting embodiments shown, the encoder selector 322 is configured to receive VFL requests 312 from a VFL customer 310 via an interface between the application controller and the service manager (for example, as defined in the NET4AI architecture). The VFL customer 310 is said to be an automated VFL user that provides a VFL request 312 containing a VFL problem that the network layer 304 addresses and maintains its own combined classifier 314 to compute a combined ML model output from multiple encoders 328. In some embodiments, the VFL customer 310 may be one of the VFL participants having its associated local encoders and datasets. In other embodiments, the VFL customer 310 may not have associated local encoders or datasets and may request a combined model to solve one or more VFL problems, where the VFL problem can be considered a VFL task.

[0105] In the non-limiting embodiments shown, the encoder selector 322 maps the VFL request 312 to the data requirements and encoder structure requirements that may be necessary to solve the VFL problem described in the VFL request 312. The data requirements include indications of "essential" data features and labeled data requirements (e.g., minimum amount of labeled data, labeled data format, etc.) that may need to be provided by the data sources of the VFL participant. The data requirements indicate the requirements for selecting data sources to solve the VFL problem.

[0106] The encoder requirements indicate an encoder structure that may be necessary to provide specific functionality for solving a VFL problem. For example, given a problem described as "recognition of traffic violations from surveillance video," an encoder structure "CNN" may be provided to indicate that a CNN encoder must be included by a combined model to process the video data. The encoder selector 322 can interact with the data source manager over multiple rounds of communication to complete the encoder and data source selection procedure. How the encoder and data selection procedure is implemented in some embodiments of the technology is described in further detail below herein.

[0107] Encoder Customizer In the non-limiting embodiments shown, when a set of encoders (with associated AI enablers) is selected for the VFL problem, the encoder customizer 324 is configured to "filter out" overlapping data features for the selected encoders and determine customization parameters for each encoder. To compute the customization parameters for each encoder, the weights of each data feature in the input layer of the corresponding encoder may be determined by the encoder customizer 324. The computation process requires interaction between the encoder customizer 324 and other AI enablers that maintain trained encoders having the same structure as the corresponding encoders. In some cases, after achieving encoder customization for each selected encoder, the corresponding AI enabler may feed back the customized encoder features to the encoder customizer 324, thereby completing the encoder customization procedure.

[0108] VFL configuration In the non-limiting embodiments shown, when customized encoder features are provided by the encoder customizer 324, the VFL configurator 326 configures parameters for the combined classifier 314 and evaluator 330, as well as routing rules for data exchange between the combined classifier 314, the evaluator 330, and the multiple encoders 328. This configuration of parameters and routing rules is referred to herein as the VFL configuration procedure.

[0109] Referring to Figure 4, the VFL configuration procedure includes a coupled classifier 314, an evaluator 330, and a privacy router 332. In the non-limiting embodiments shown, the coupled classifier 314 is implemented as a service function performed by a third-party VFL customer 310 on an application server and / or by the network. The coupled classifier 314 may perform intermediate result exchange and collaborator functions defined in conventional VFL frameworks, such as calculating the output of a coupled ML model from the outputs of a number of selected VFL encoders, interacting with the evaluator 330 to obtain the coupled loss, and backpropagating the gradient for each encoder.

[0110] In some non-limiting embodiments of this technology, the coupled classifier 314 may be pre-configured with a fixed structure. In these non-limiting embodiments, the VFL configurator 326 can configure the encoder output to the coupled classifier 314 according to customized encoder output layer features.

[0111] In the non-limiting embodiments shown, the evaluator 330 is implemented as a service function running on the network layer 304. The evaluator 330 receives labeled data forwarded by the privacy router 332 from one or more data sources 308 and calculates a combined loss value by applying a specific loss function to compare the combined ML model output from the combined classifier 314 with the corresponding labeled data.

[0112] In some non-limiting embodiments, the VFL configurator 326 may configure a selected loss function and labeled data format (including combination rules if the labeled data consists of data from multiple data sources) according to the selected encoder and labeled data information of the data source 308.

[0113] In the non-limiting embodiments shown, the privacy router 332 is a predefined NET4AI function. However, the privacy router 332 may be implemented as a predefined function in a different architecture without departing from the scope of this technology. In the automated VFL, the privacy router 332 enables secure data exchange between the coupled classifier 314, the evaluator 330, the selected encoder, and the labeled data source 308.

[0114] In the non-limiting embodiments shown, the data merging function 412 can be performed by the privacy router 332 to concatenate unbalanced data entities (e.g., in terms of data entity quantity) associated with the same data ID during the training phase. The data merging function 412 can provide an interface between the encoder outputs 410 and the combined classifier 314. Roughly speaking, the data merging function 412 combines multiple outputs from different encoders (each output may be in the form of a batch of data) into a single batch of combined outputs (e.g., concatenates all output batches into one) according to predefined combination rules, and then sends it to the combined classifier 314 for further training / inference. The privacy router 332 can also perform combined loss transfer 416 between the combined classifier 314 and the evaluator 330, and labeled data transfer 414 from the labeled data source 408 to the evaluator 330.

[0115] Various embodiments of this technology that may help enable at least some of the encoder and data selection functions, encoder customization functions, and VFL configuration functions of an automatic VFL architecture are described below.

[0116] Encoder and Data Selection Embodiments Referring to Figure 5, a schematic diagram of the encoder and data selection procedure 500 is shown. In an illustrated, non-limiting embodiment, the procedure begins in step 506, in which the encoder selector 504 receives a VFL request 506 from the VFL customer 502. Without departing from the scope of the Art, it is conceivable that the VFL request 506 may be received by the encoder selector 504 in the same way that the VFL request 312 is received by the encoder selector 322. As described above, the VFL request 506 includes a description of the VFL problem to be solved / the VFL problem in which the VFL should be trained to solve.

[0117] Generally, VFL issues are general issues that can be "understood" by the encoder selector 504 to determine specific data requirements and / or encoder requirements. In embodiments where the VFL customer 502 has partial and / or complete knowledge of the detailed data requirements and encoder requirements, the VFL customer 502 can directly indicate them in the VFL request 502 for the encoder selector 504 to process.

[0118] In other embodiments where the VFL customer 502 provides only a general VFL problem in the VFL request 506, the procedure may proceed to step 508 for determining the data requirements and encoder requirements. Therefore, the encoder selector 504 separates or divides the VFL task into subtasks. As an example, the subtasks include a first subtask and a second subtask. It can be understood that the number of subtasks can be more than two.

[0119] Performing a given subtask requires a list of necessary data features and requirements (e.g., quantity, content, or data) for labeled data, which will be referred to below herein as the necessary data requirements for the subtask. Each subtask also corresponds to an application category associated with a particular encoder structure, and a particular encoder structure will be referred to as the necessary encoder requirement for the subtask.

[0120] For example, the VFL task "Traffic Violation Recognition" can be broken down into a combination of the following subtasks: 1) a subtask that recognizes traffic violations by camera recording, requiring video stream data features and labeled traffic violation video clips (required data requirements) and a CNN encoder for video processing (required encoder requirements); and 2) a subtask that recognizes traffic violations by RSU data, requiring RSU sensing data features and labeled data and an encoder structure for RSU data processing. It should be noted that application categories can be predefined NET4AI concepts that describe, for example, a set of applications / problems that can be solved by a particular encoder structure. Thus, it can be said that encoder structures may be associated with and stored in relation to each application category.

[0121] In some embodiments, a VFL task can be implemented as a list of attributes representing subtask IDs and their importance (weights). The configuration and attribute values ​​for each VFL task are maintained by an encoder selector 504 and can be learned from historical data and / or existing knowledge. In the above example having a "traffic violation recognition" task, the VFL task may also be defined as [(1,0.4);(2,0.6)], where (1,0.4) corresponds to a "camera recording" subtask with an ID equal to "1" and an importance equal to "0.4", and (2,0.6) corresponds to a "RSU" subtask with an ID equal to "2" and an importance equal to "0.6".

[0122] In this embodiment, the procedure proceeds to step 508, in which the encoder selector 504 selects one or more encoders to solve the VFL task, and the one or more encoders can be selected from a pool of encoders in the communication network. Since a given encoder structure corresponds to one application category, the encoder selector 504 may select encoders to participate in the VFL according to the required encoder requirements of multiple subtasks. The number of encoders can match the number of subtasks. When the subtasks include a first subtask and a second subtask, the encoder requirements indicate a first encoder structure to be used to solve the first subtask and a second encoder structure to be used to solve the second subtask.

[0123] In some embodiments, the encoder selector 504 may select an encoder having a relatively simpler structure to accelerate the convergence speed of the combined model training in order to ensure a predetermined effective threshold (e.g., classification / prediction accuracy).

[0124] In other embodiments, the encoder selector 504 may need to ensure that the selected set of encoders includes the encoder structures necessary to solve all subtasks of the VFL task. For example, in the “traffic violation recognition” task presented above, the selected set of encoders may need to include at least a CNN for solving the “camera recording” subtask and a DNN for solving the “RSU data” subtask. The CNN can be considered a first encoder, and the DNN can be considered a second encoder. The first encoder is configured to process a first set of features for solving a first subtask, and the second encoder is configured to process a second set of features for solving a second subtask.

[0125] In the non-limiting embodiments shown, the procedure follows step 510, which involves selecting data to be used to solve the VFL task. Given the selected encoders in step 508, the encoder selector 504 is configured to determine data sources for each selected encoder. The encoder selector 504 is configured to select a first data source from a pool of data sources to solve a first subtask, the first data source containing a first set of features, and is also configured to select a second data source from a pool of data sources to solve a second subtask, the second data source containing a second set of features.

[0126] In embodiments where all data sources are owned by the network, the encoder selector 504 can directly perform the data selection process according to complete information on all available data sources. However, in other embodiments where the available data sources are not owned by the network, the encoder selector 504 may interact with a data source manager to access information on the data sources available for data selection, including the location of the data sources, data quality, and the data characteristics that each data source can provide.

[0127] For example, a data source manager may be a data analytics and management (DAM) function or entity that monitors data sources and provides data analysis capabilities to different network applications (including VFL applications).

[0128] In this example, the data selection process can be completed by the following set of interactions between the data source manager and the encoder selector 504. The first interaction may include the encoder selector 504 sending all necessary data requirements to the data source manager to request available data source information, the data requirements being included in the request sent to the data source manager by the encoder selector 504. The second interaction may include the data source manager feeding back data quality and data characteristic information for all available data sources in the pool of data sources.

[0129] Depending on the available data source information received, the encoder selector 504 may select detailed data sources for each selected encoder and data sources for labeled data. During data source selection, the encoder selector 504 may need to ensure that (i) the set of selected data sources provides all the necessary data features required by the subtask, (ii) the selected data sources have a minimum number of overlapping features, and (iii) the labeled data can be original labeled data from one or more selected sources, or combined labeled data consisting of labeled data with partial features from multiple data sources.

[0130] In this example, the third interaction may include the encoder selector 504 sending all selected data source IDs to the DAM tool. The fourth interaction may include the DAM tool providing feedback on the location of all datasets containing the selected data sources. It should be noted that each data source may be partially hosted by multiple datasets at different network locations (e.g., servers, edge devices, etc.). Upon receiving the detailed dataset locations, the encoder selector 504 may determine the location (AI enabler location) for implementing the corresponding encoder that minimizes the communication cost between the dataset and the AI ​​enabler.

[0131] Encoder customization subset Referring to Figure 6, a schematic diagram of the encoder customization procedure 600 is shown. The customization procedure 600 may include communication and data transfer between the encoder selector 602, the encoder customizer 604, the reference AI enabler 606, and the target AI enabler 608. The encoder selector 602 may be implemented in a manner similar to the encoder selector 322 in Figure 3 and / or the encoder selector 504 in Figure 5. The encoder customizer 604 may be implemented in a manner similar to the encoder customizer 324 in Figure 3. The reference AI enabler 606 is a well-trained "full" encoder available on the network. The target AI enabler 608 is an encoder to be "customized" according to at least some embodiments of the present technology.

[0132] In step 610, the encoder selector 602 transmits the encoder selection result to the encoder customizer 604. For example, step 610 may be performed after the completion of the encoder and data selection procedure, as described above. In some embodiments, the customization procedure 600 may be triggered by the encoder customizer 602 upon receipt of the encoder selection result.

[0133] The content of the encoder selection result is not limited and may vary in particular depending on the various implementation methods of this technology. However, in some embodiments, the content of the encoder selection result includes the encoder structure parameters of each selected encoder (e.g., number of layers, size, etc.), the location of the target AI enabler (TE) that implements each customized selected encoder, and the location of the reference AI enabler (RE) that implements each well-trained selected encoder.

[0134] In the non-limiting embodiments shown in Figure 6, the encoder selection result includes, among other things, the encoder structure parameters of the first encoder for the first subtask, the corresponding position of TE608, and the corresponding position of RE606.

[0135] While not wishing to be bound by any particular theory, the developers recognize that the purpose of including RE606 in encoder customization procedure 600 is to compute different weights of data features in a given selected encoder, which may also be parameters necessary to perform customization decisions. The data features whose weights should be computed by RE606 are likely to be data features arising from a first data source. In at least one case, all AI enablers for different applications can all be maintained by a network layer (e.g., a NET4AI network), so the encoder selector can directly select the most trained RE606. However, to ensure security and privacy requirements, the encoder selector only "knows" the encoder structure within RE606 (the same as the selector encoder structure) and how well-trained the encoder is (e.g., the number of training / uses). The detailed values ​​(bias / weights) of the neurons and links within RE606 are not known to the encoder selector itself.

[0136] Continuing the explanation of Figure 6, in step 620, the encoder customizer 604 sends a request for data feature weights to the corresponding RE 606. In step 630, the encoder customizer 604 receives the calculated data feature weights. It should be noted that for a given encoder, the weight of each data feature in its input layer indicates the influence / importance of the corresponding data feature on the output of the given encoder. For example, the higher the weight of a given data feature, the greater the variation in the encoder output that can be observed when the given data feature value is changed.

[0137] Referring to Figure 7, an example of the structure of the first encoder 700 and the second encoder 710 selected by the encoder and data selection procedure is shown. For example, the first encoder 700 has input data features (A, B, C, D) with weights (0.3, 0.4, 0.1, 0.2), and the second encoder 710 has input data features (A, B, E, F) with weights (0.1, 0.2, 0.4, 0.3). In this example, input data features (A, B, C, D) originate from the first data source, and input data features (A, B, E, F) originate from the second data source.

[0138] The developers of this technology recognize that, according to a well-trained encoder, there are multiple methods for calculating the weights of each data feature for a customized encoder. However, the encoder customizer 604 may not have access to the well-trained values ​​of the neurons and links in the corresponding RE606 due to security and privacy requirements. Therefore, the calculation of the weights of each data feature for a given selected encoder can be completed through an interaction between the encoder customizer 604 and the corresponding RE606, which takes place between steps 620 and 630. Thus, the encoder customizer 604 sends a request for data feature weights to the corresponding RE606 (step 620), the corresponding RE606 locally calculates the data feature weights according to, for example, a known technique, and then feeds back the calculated data feature weights to the encoder customizer 604 (step 630). As an example, the calculated data feature weights include data feature weights corresponding to each feature in a first set of features and data feature weights corresponding to each feature in a second set of features, where the first set of features is contained in a first data source for a first encoder, and the second set of features is contained in a second data source for a second encoder. At least one such technique is disclosed in the paper "Problems with Shapley-value-based explanations as feature importance measures" by Kumar, I. Elizabeth et al., published on June 30, 2020, the entire contents of which are incorporated herein by reference. Other techniques are conceivable without departing from the scope of this technique.

[0139] The customization procedure 600 proceeds to step 640, where the computed data feature weights of all selected encoders by the corresponding RE606 are calculated by the encoder customizer 604, and the encoder customizer 604 is configured to select the "key features" of the combined model being constructed.

[0140] In some embodiments, non-overlapping data features may be automatically selected as primary features. Roughly speaking, non-overlapping data features include data features present in each selected encoder but not present in other selected encoders. In the example shown in Figure 7, the non-overlapping features include features C, D, E, and F.

[0141] In other embodiments, for a given overlapping data feature, the encoder customizer 604 compares its data feature weights in each selected encoder and selects the one with the highest relative weight as the primary feature. Roughly speaking, the overlapping data feature includes data features that are duplicated in at least two selected encoders. In the example shown in Figure 7, the overlapping feature includes features A and B.

[0142] For example, it should be noted that in the first encoder 700, features A and B are the primary features. Relative weights are different from the data feature weights calculated by the RE. Relative weight calculation takes into account both the data feature weights of each RE and other factors, including but not limited to the encoder structure size (larger sizes imply higher complexity, which affects output convergence performance), dependencies on other features, and the overall importance of the features. It should be noted that relative weights are used to determine which data features should be retained and which should be removed in the first encoder 700 and the second encoder 710. In this example, it is said that relative weights are used to determine whether features A and B should be removed from the first encoder 700 and from the second encoder 710. Given the selected primary features, all non-primary data features are removed by the encoder customizer from the corresponding selected encoder. By removing non-primary features, the side effects on the combined model caused by overlapping data features can be effectively mitigated.

[0143] In some embodiments of this technology, three modes of encoder customization can be selectively performed by the encoder selector 604, depending, in particular, on different application scenarios. For example, the first mode 720 can be called a minimal customization mode in which only input layer neurons corresponding to non-major features in the selected encoder are removed. This is the encoder customization mode that has the least impact on the coupled model structure.

[0144] In another example, the second mode can be called the weight-based customization mode 730, in which, for a given selected encoder having non-principal features removed from its input layer, the encoder customizer 604 sends the IDs of the removed neurons to the corresponding RE, which has a corresponding well-trained encoder. The corresponding RE removes all neurons and links in the encoder that are significantly affected by the removed features. The effect of each data feature on a particular neuron or link in the encoder can be calculated according to various techniques. At least one such technique is disclosed in the paper titled "Problems with Shapley-value-based explanations as feature importance measures". The corresponding RE then feeds back the IDs / locations of the removed neurons and links to the encoder customizer 604 to complete the encoder customization.

[0145] In a further example, a third mode can be called a learning-based customization mode, in which, for a given selected encoder having non-primary features removed from its input layer, the encoder customizer 604 applies a learning-based NN pruning solution to train a customized encoder. At least one such solution is disclosed in the paper "Learning both Weights and Connections for Efficient Neural Networks" by Han, Song et al., published in 2015, the entire contents of which are incorporated herein by reference.

[0146] The developers are aware that such a solution requires considerable computational resources during training. It should be noted that the training phase should be performed by the corresponding TE that implements the customized encoder in order to reduce the computational cost of running the encoder customizer 604.

[0147] In the non-limiting embodiments shown, in step 650, the encoder customizer 604 sends encoder customization parameters to the corresponding TE 608 to perform the encoder customization decision process. The contents of the encoder customization parameters may, in particular, depend on different encoder customization modes. For the minimal customization mode 720, the encoder customization parameters may include information indicating deleted input layer neurons (non-major features). For the weight-based customization mode 730, the encoder customization parameters may include information indicating deleted neurons and links for the full encoder. For the learning-based customization mode, the encoder customization parameters may include information indicating deleted neurons in the input layer and hyperparameters for customization training. In some embodiments of the technology, the hyperparameters may include a loss threshold for determining convergence, weight thresholds for neurons and links to be deleted, etc.

[0148] In the non-limiting embodiments shown, in step 660, the corresponding TE608 performs an encoder customization decision according to the encoder customization parameters received in step 650. The decision may be made based on relative weights, which can be calculated based on the encoder customization parameters received in step 650. For minimum and weight-based customization modes 720 and 730, the corresponding TE608 can directly delete the corresponding neurons and / or links. For learning-based customization modes, the corresponding TE608 may perform customization training until convergence without departing from the scope of the art. Customization training can be triggered in response to the reception of encoder customization parameters.

[0149] For illustrative purposes only, it is assumed that the corresponding TE608 performs encoder customization decisions according to a learning-based customization mode. In the non-limiting embodiments shown, the final customization result of the learning-based customization is obtained in the corresponding TE608, and after convergence is reached, the corresponding TE608 may, in step 660, feed back the customized encoder features (e.g., customized structure of the encoder) to the encoder customizer 604. The customized encoder features may be used, for example, for further VFL configuration.

[0150] VFL configuration embodiment Referring to Figure 8, a schematic diagram of the VFL configuration procedure 800 is shown. The VFL configuration procedure 800 includes an encoder selector 802, an encoder customizer 804, a VFL configurator 806, a coupling classifier 808, an evaluator 810, and a privacy router 332. At least some of the components involved in the VFL configuration procedure 800 may be implemented similarly to the components of the VFL architecture 300 shown in Figure 3, without departing from the scope of this technology.

[0151] In the non-limiting embodiments shown, in step 820, the encoder selector 802 transmits labeled data information to the VFL configurator 806, and in step 830, the encoder customizer 804 transmits customized encoder information to the VFL configurator 806. In some embodiments, the reception of the labeled data information and customized encoder information by the VFL configurator 806 may trigger the following steps of the VFL configuration procedure 800.

[0152] Generally, labeled data information includes the location of the dataset containing the labeled data sources and the provided data features. In other words, labeled data information indicates the location (which part of the labeled data comes from which of the first and second data sources) and how the labeled data should be combined. The loss function provided by the VFL customer may also be transmitted as part of the labeled data information. Customized encoder information includes information indicating the output layer structure of each customized encoder. For example, when encoder customizer 804 determines that the first encoder needs to be customized according to data feature weights, the customized encoder information includes information indicating the output layer structure of the customized first encoder. When encoder customizer 804 determines that both the first and second encoders need to be customized according to data feature weights, the customized encoder information includes information indicating the output layer structures of the customized first and second encoders.

[0153] In the non-limiting embodiments shown, in step 840, the VFL configurator 806 configures the encoder output to the combined classifier based on the customized encoder information. The configuration of the encoder output to the combined classifier may be carried out in various ways. In some embodiments, the output layers of multiple encoders may be connected to the input layer of the combined classifier via a chain of the output layers of multiple encoders. At least one other connection mode is disclosed in the paper entitled "SplitNN-driven vertical partitioning" by Iker Ceballos et al., published on August 7, 2020, and the entire contents of that paper are incorporated herein by reference.

[0154] In the non-limiting embodiments shown, in step 850, the VFL configurator 806 transmits coupled classifier configuration parameters to the coupled classifier 808. The contents of the coupled classifier configuration parameters are used by the coupled classifier to customize or modify its local configuration to match the encoder output (which is the input to the coupled classifier). For example, in some embodiments, encoders customized in a weight-based or learning-based manner as described above may have their output layer neurons partially muted and / or removed. The VFL configurator may include information on the muted output layer neurons (for example, for all encoders) in the coupled classifier configuration parameters. By receiving the coupled classifier configuration parameters, the coupled classifier can customize its input layer architecture (for example, by muting and / or removing neurons) to match the encoder output.

[0155] In the non-limiting embodiments shown, in step 860, the VFL configurator 806 configures the evaluator 810. As part of step 860, the VFL configurator 806 uses labeled data information to configure the combination rules for when partially labeled data from different datasets are merged in the evaluator 810. Also as part of step 860, the VFL configurator 806 configures the loss function to be used during VFL.

[0156] Partially labeled data can be described as labeled data in which the associated label information does not represent the complete label information. All partially labeled data associated with the same ID / sample should be combined to form complete labeled data. For example, complete labeled data may be represented as "[A, blue cat]", where "A" is the associated ID / sample (i.e., the represented entity) and "blue cat" is the label. In some embodiments, this labeled data may be provided from two different data sources, namely one providing "[A, blue]" and the other providing "[A, cat]". In this example, both "[A, blue]" and "[A, cat]" are named partially labeled data.

[0157] In the non-limiting embodiments shown, in step 880, the VFL configurator 806 configures an interencoder or a “privacy” router 812. The privacy router 812 may be implemented similarly to the privacy router 332 in Figure 3. As part of step 880, the VFL configurator 806 configures routing paths for data transmission for (i) data exchange paths between the combined classifier 808 and the evaluator 810, (ii) labeled data merging paths between the evaluator 810 and their respective data sources, and (iii) backpropagation paths between the combined classifier 808 and their respective encoders for backpropagation gradient values. In (iii), “each encoder” can be understood as all selected encoders selected by the encoder selector. When some of the selected encoders are customized by the encoder customizer, “each encoder” in (iii) includes the customized encoders and the remaining uncustomized encoders among all selected encoders. When the encoder selected by the encoder selector includes a first encoder and a second encoder, and the first encoder is customized, “each encoder” in (iii) includes the customized first encoder and the second encoder.

[0158] In the non-limiting embodiments shown, in step 890, the VFL configurator 806 sends privacy router configuration parameters to the privacy router 812. In at least some embodiments of the technology, the routes supported by the privacy router 812 may be those described above with reference to Figure 4.

[0159] Performance evaluation Referring to Figure 9, three simplified representations of the VFL framework are shown. To verify the effectiveness and efficiency of the automated VFL architectures that are possible in at least some embodiments of this technology, simulations were performed to compare the performance of three cases for a binary classification task: the conventional VFL 910, the automated VFL 92, and the stitched single NN 930.

[0160] Generally, the conventional VFL 910 includes two participants (P1 and P2) with different encoders (E1 and E2). The detailed parameters of E1 and E2 are shown in Table 1 below. Both participants have independent classifiers with the same structure (C0). The inputs of both participants (I1 and I2) completely overlap in the ID space and partially overlap in the feature space. The automated VFL 920 includes two participants (PA1 and PA2) with encoders and inputs for P1 and P2. The outputs of the two participants are concatenated into a combined classifier with the same structure as C0. It should be noted that all classifiers used in these three cases are identical in structure. The stitched single NN 930 concatenates I1 and I2 as an integrated input layer and stitches E1 and E2 layer by layer in a fully connected manner to form a stitched encoder. It should be noted that the remaining hidden layers of the encoder (e.g., the remaining E2 layers) are fully connected to the stitched structure. The independent classifier C0 is fully connected to the output layer of the stitch encoder. [Table 1]

[0161] The simulation experimented with two datasets: an anonymous banking dataset containing user information from Germanbank, and a network traffic dataset used for deep packet inspection (TLS22) of encrypted internet traffic.

[0162] Referring to Figure 10, a performance comparison between the conventional VFL 910 and the automated VFL 920 applied to the two test datasets described above is shown. To verify the effectiveness of encoder customization, which is possible in at least some embodiments of this technology, both the automated VFL "autoVFL_noCut" without encoder customization and the automated VFL "autoVFL" with minimal encoder customization were simulated. As shown in Graphs 1010 and 1030, in some implementations of this technology, the automated VFL 920 (with or without encoder customization) can converge to better optimal results with higher accuracy and lower variability compared to the conventional VFL 910. As shown in Graphs 1020 and 1040, the classification performance of the conventional VFL 910 and the automated VFL 920 is compared. In at least some implementations of this technology, the automated VFL 920 can improve general performance metrics compared to the conventional VFL 910, particularly in terms of accuracy (5% to 8% improvement) and precision (10% improvement).

[0163] The developers recognize that, in at least some embodiments of this technology, the better performance achieved by automated VFL can be made possible by two embodiments of automated VFL. Firstly, compared to the exchange of intermediate results in conventional VFL, the coupled classifier can enable stronger mutual influence between participating encoders, which increases the efficiency of training the coupled model. Secondly, encoder customization can further reduce the adverse effects caused by overlapping data features from the inputs of multiple participants. While both the coupled classifier and encoder customization can improve overall performance, the results shown in Figure 10 show that the coupled classifier has a greater impact than encoder customization. Furthermore, it should be noted that in the simulation in Figure 10, only the minimal encoder customization mode is applied. When more complex encoder customization algorithms are fully implemented (other modes described above herein), encoder customization can increase the performance of automated VFL without deviating from the scope of this technology.

[0164] Referring to Figure 11, a performance comparison between the automated VFL920 and the stitched single NN930 applied to the two test datasets described above is shown. For the stitched single NN930, both a single NN "singleNN_noCut" without applying encoder customization to its connected input layers and a single NN "singleNN_cut" with minimal encoder customization applied to its connected input layers were simulated. In at least some embodiments of this technology, it is conceivable that the automated VFL can achieve comparable convergence and classification performance to that of the stitched single NN without significant loss. Furthermore, as seen in Graphs 1110 and 1120, it should be noted that the stitched single NN930 suffers from considerable performance shortcomings because a simple stitched NN structure may not be suitable for certain datasets or problems. While a suitable new NN can mitigate these shortcomings, developers recognize that the redesign procedure for a new NN can be costly because clear design principles are not available and existing models cannot be reused. Therefore, in at least some embodiments of this technology, the automated VFL 920 can achieve general performance metrics equivalent to or better than those of the stitched single NN 930.

[0165] In some embodiments of this technology, an automated VFL system is provided that is configured to perform at least some of the following functions: an encoder selector, an encoder customizer, a VFL configurator, and a combined classifier. In some embodiments of this technology, the automated VFL system can be implemented for the NET4AI architecture.

[0166] In some embodiments, the automated VFL system can achieve better AI model performance compared to conventional VFL and stitched single NNs because (i) the combined classifier can enable deeper interaction between the participating encoders (compared to conventional VFL), and (ii) encoder customization can reduce the adverse effects caused by overlapping data features. In other embodiments, the automated VFL system may enable automated encoder selection and reuse. As a result, this allows for the reuse of existing encoders in the network to solve VFL tasks, and it may not be necessary to design a specific NN for that purpose. Both the selection and customization of encoders (and datasets) are automated by NET4AI. This may also avoid VFL configuration through the application layer, and as a result, VFL management overhead in the application layer may not be required. In further embodiments, the automated VFL system may solve complex tasks by supporting loss functions that violate homomorphic encryption, and it may be possible for labeled data to be provided separately by multiple data sources.

[0167] Those skilled in the art will understand that the descriptions of various embodiments are illustrative only and not intended to be limiting. Other embodiments are readily suggested to those skilled in the art who are interested in this disclosure. Furthermore, at least some of the disclosed embodiments may be customized to provide useful solutions to existing needs and problems relating to VFL. For clarity, not all of the typical features of at least some of the implementations of the disclosed embodiments are illustrated and described.

[0168] In particular, the combinations of features are not limited to those presented in the above description, as the combinations of elements enumerated in the attached claims form an integral part of this disclosure. Naturally, in developing any practical implementation of at least some of the embodiments disclosed, a number of implementation-specific decisions may need to be made to achieve the developer's specific objectives, such as compliance with application, system, and business-related constraints, and it is recognized that these specific objectives will vary from implementation to implementation and from developer to developer. Furthermore, while development efforts can be complex and time-consuming, it is nevertheless recognized as a routine engineering undertaking for those skilled in the art in the field of feedback equalization at high data rates who are of interest in this disclosure.

[0169] According to this disclosure, the components, process operations and / or data structures described herein may be implemented using various types of operating systems, computing platforms, network devices, computer programs and / or general-purpose machines. Furthermore, those skilled in the art will recognize that devices of a less general nature, such as hardwired devices, field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs), may also be used. When a method comprising a series of operations is implemented by a computer, a processor operably connected to memory, or a machine, those operations may be stored as a series of instructions readable by the machine, processor, or computer, or stored on a non-temporary tangible medium.

[0170] The systems and modules described herein may include software, firmware, hardware, or any combination of software, firmware, or hardware suitable for the purposes described herein. The software and other modules may be executed by a processor and reside in the memory of servers, workstations, personal computers, computerized tablets, personal digital assistants (PDAs), and other devices suitable for the purposes described herein. The software and other modules may be accessible via local memory, via a network, via a browser or other application, or via other means suitable for the purposes described herein. The data structures described herein may include computer files, variables, programming arrays, programming structures, or any electronic information storage scheme or method, or any combination thereof, suitable for the purposes described herein.

[0171] This disclosure is described above by non-limiting exemplary embodiments provided as examples. These exemplary embodiments may be modified at will. The claims should not be limited by the embodiments described in these examples, and the broadest interpretation consistent with the overall description should be given.

Claims

1. The encoder selector receives a VFL request from a vertically federated learning (VFL) customer, The steps include determining data requirements and encoder requirements for solving a VFL task according to the VFL request using the encoder selector, wherein the VFL task can be divided into a plurality of subtasks, and the plurality of subtasks include at least a first subtask and a second subtask, The steps include: using the encoder selector to select a first encoder and a second encoder from a pool of encoders in the communication network based on the encoder requirements, wherein the first encoder is configured to process a first set of features for solving a first subtask, and the second encoder is configured to process a second set of features for solving a second subtask; The steps include: using the encoder selector to select a first data source for the first encoder and a second data source for the second encoder based on the data requirements, wherein the selected data sources are from a pool of data sources, the first data source includes a set of first features, and the second data source includes a set of second features; and A method that includes this.

2. The method according to claim 1, wherein the data requirements indicate requirements for selecting a first data source for solving the first subtask and a second data source for solving the second subtask.

3. The method according to claim 1, wherein the encoder requirement includes a first encoder structure to be used to solve the first subtask and a second encoder structure to be used to solve the second subtask.

4. This method is The method according to claim 1, further comprising the step of dividing the VFL task into the plurality of subtasks using the encoder selector.

5. Selecting the first encoder means The encoder selector determines the application category of the first subtask based on the encoder requirements. Includes, The method according to claim 1, wherein the application category is stored in memory in association with the first encoder.

6. Selecting the first encoder means The encoder selector identifies the pool of encoders having the first encoder structure, The encoder selector selects the first encoder from the pool of encoders having the first encoder structure. The method according to claim 1, including the method described in claim 1.

7. Selecting the first data source and the second data source means The encoder selector sends a request for data source information to the data source manager, the request including the data requirements, The encoder selector receives data source information from the data source manager indicating the quality and characteristics of the data contained in each of the data source pools, The encoder selector uses the data source information to select the first data source for the first encoder and the second data source for the second encoder. The method according to claim 1, including the method described in claim 1.

8. The first data source and the second data source are associated with their respective IDs, and the method is as follows: The encoder selector transmits the IDs of the first data source and the second data source to the data source manager. The encoder selector receives the network locations of the first data source and the second data source from the data source manager. The method according to claim 7, further comprising:

9. The step of customizing the first encoder by muting a portion of the first encoder using an encoder customizer, thereby generating a customized first encoder. Includes, The first and second encoders are selected from a pool of encoders in the communication network based on encoder requirements, and the encoder requirements are determined to solve a VFL task according to a vertically federated learning (VFL) requirement. The VFL task can be divided into a plurality of subtasks, and the plurality of subtasks include at least a first subtask and a second subtask. The first encoder is configured to process a first set of features for solving the first subtask, and the second encoder is configured to process a second set of features for solving the second subtask. The aforementioned customization is, Determining the overlapping features between the first set of features and the second set of features, Muting the portion of the first encoder that processes the overlapping features Methods that include...

10. The aforementioned customization is, The encoder customizer determines that the first encoder should be customized. The method according to claim 9, including the method described in claim 9.

11. The encoder customizer determines that the first encoder should be customized. The encoder customizer determines that the first encoder should be customized based on data feature weights, and the data feature weights are received from a reference AI enabler. The method according to claim 10, including the method described in claim 10.

12. Customizing the first encoder is The encoder customizer mutes the portion of the first encoder that processes the overlapping features. Includes, The method according to claim 9, wherein the muting includes muting at least one of the neurons and links of the first encoder.

13. The steps include configuring at least one of the evaluator and the combined classifier using a vertically associative learning (VFL) configurator, The steps include configuring the privacy router to determine, using the VFL configurator, at least one of the following: a data exchange path between the combined classifier and the evaluator; a labeled data merge path between the evaluator and one or more data sources; and a backpropagation path between the combined classifier and a customized first encoder and a second encoder for backpropagating gradient values. Includes, A method wherein the combined classifier, the evaluator, and the privacy router are configured to perform VFL.

14. The above constitutes the evaluator, The VFL configurator receives labeled data information from the encoder selector, The VFL configurator determines the loss function to be used to execute VFL, The VFL configurator configures the evaluator using the labeled data information and the loss function. The method according to claim 13, including the method described in claim 13.

15. The above constitutes the evaluator, The VFL configurator configures the rules for combining labeled data and loss functions, which are generated from data sources selected for the first and second encoders. The method according to claim 13, including the method described in claim 13.

16. The configuration of the aforementioned combined classifier is The VFL configurator receives information from the encoder customizer indicating the output layer structure of the first encoder and the second encoder, The VFL configurator configures the encoder output to the coupled classifier based on the output layer structure of the first encoder and the second encoder. The method according to claim 13, including the method described in claim 13.

17. The method according to claim 13, wherein the second encoder is a customized second encoder.

18. It is a device, One or more processors, When executed by the one or more processors, the device has one or more memories that store instructions causing the device to perform the method described in any one of claims 1 to 8. A device that includes this.

19. It is a device, One or more processors, When executed by the one or more processors, the device has one or more memories that store instructions causing the device to perform the method described in any one of claims 9 to 12. A device that includes this.

20. It is a device, One or more processors, When executed by the one or more processors, the device has one or more memories that store instructions causing the device to perform the method described in any one of claims 13 to 17. A device that includes this.

21. The encoder selector receives a VFL request from a vertically federated learning (VFL) customer, The steps include determining data requirements and encoder requirements for solving a VFL task according to the VFL request using the encoder selector, wherein the VFL task can be divided into a plurality of subtasks, and the plurality of subtasks include at least a first subtask and a second subtask, The steps include: using the encoder selector to select a first encoder and a second encoder from a pool of encoders in the communication network based on the encoder requirements, wherein the first encoder is configured to process a first set of features for solving a first subtask, and the second encoder is configured to process a second set of features for solving a second subtask; The steps include: using the encoder selector to select a first data source for the first encoder and a second data source for the second encoder based on the data requirements, wherein the selected data sources are from a pool of data sources, the first data source includes a set of first features, and the second data source includes a set of second features; The steps include: customizing the first encoder by muting a portion of the first encoder using an encoder customizer, thereby generating a customized first encoder; The VFL configurator configures at least one of the evaluator and the combined classifier, The steps include configuring the privacy router to determine, using the VFL configurator, at least one of the following: a data exchange path between the combined classifier and the evaluator; a labeled data merge path between the evaluator and one or more of the selected data sources; and a backpropagation path between the combined classifier and each of the customized first and second encoders for backpropagating gradient values. Includes, A method wherein the combined classifier, the evaluator, and the privacy router are configured to perform VFL.

22. The method according to claim 21, wherein the data requirements indicate requirements for selecting a first data source for solving the first subtask and a second data source for solving the second subtask.

23. The method according to claim 21, wherein the encoder requirement includes a first encoder structure to be used to solve the first subtask and a second encoder structure to be used to solve the second subtask.

24. This method is The method according to claim 21, further comprising the step of dividing the VFL task into the plurality of subtasks using the encoder selector.

25. Selecting the first encoder means The encoder selector determines the application category of the first subtask based on the encoder requirements. Includes, The method according to claim 21, wherein the application category is stored in memory in association with the first encoder.

26. Selecting the first encoder means The encoder selector identifies the pool of encoders having the first encoder structure, The encoder selector selects the first encoder from the pool of encoders having the first encoder structure. The method according to claim 21, including the method described in claim 21.

27. Selecting the first data source and the second data source means The encoder selector sends a request for data source information to the data source manager, the request including the data requirements, The encoder selector receives data source information from the data source manager indicating the quality and characteristics of the data contained in each of the data source pools, The encoder selector uses the data source information to select the first data source for the first encoder and the second data source for the second encoder. The method according to claim 21, including the method described in claim 21.

28. The first data source and the second data source are associated with their respective IDs, and the method is as follows: The encoder selector transmits the IDs of the first data source and the second data source to the data source manager. The encoder selector receives the network locations of the first data source and the second data source from the data source manager. The method according to claim 27, further comprising:

29. The aforementioned customization is, Determining the overlapping features between the first set of features and the second set of features, Muting the portion of the first encoder that processes the overlapping features The method according to claim 21, including the method described in claim 21.

30. The aforementioned customization is, The encoder customizer determines that the first encoder should be customized. The method according to claim 21, including the method described in claim 21.

31. The encoder customizer determines that the first encoder should be customized. The encoder customizer determines that the first encoder should be customized based on data feature weights, the data feature weights being received from a reference AI enabler. The method according to claim 30, including the method described in claim 30.

32. Customizing the first encoder is The encoder customizer mutes the portion of the first encoder that processes overlapping features. Includes, The method according to claim 21, wherein the muting includes muting at least one of the neurons and links of the first encoder.

33. The above constitutes the evaluator, The VFL configurator receives labeled data information from the encoder selector, The VFL configurator determines the loss function to be used to execute VFL, The VFL configurator configures the evaluator using the labeled data information and the loss function. The method according to claim 21, including the method described in claim 21.

34. The above constitutes the evaluator, The VFL configurator configures the rules for combining labeled data generated from data sources selected for the first and second encoders with the loss function. The method according to claim 33, including the method described in claim 33.

35. The configuration of the aforementioned combined classifier is The VFL configurator receives information from the encoder customizer indicating the output layer structure of the first encoder and the second encoder, The VFL configurator configures the encoder output to the coupled classifier based on the output layer structure of the first encoder and the second encoder. The method according to claim 21, including the method described in claim 21.

36. The method according to claim 21, wherein the second encoder is a customized second encoder.

37. A system comprising an encoder selector configured to perform the method described in any one of claims 1 to 8, an encoder customizer configured to perform the method described in any one of claims 9 to 12, and a vertically associated learning (VFL) configurator configured to perform the method described in any one of claims 13 to 17.

38. A non-temporary computer-readable storage medium that stores an instruction causing the device to perform any one of claims 1 to 17 or any one of claims 21 to 36 when executed by the device.

Images