Methods, devices, apparatus, and media for performing actions

Abstract

Methods, apparatuses, devices and media for performing actions are provided. In one method, based on a first action model at a first device, a group of actions to be performed at the first device is determined from a plurality of actions. A data accumulation index associated with a group of actions is obtained, the data accumulation index indicating the amount of data to be sent from the first device to a second device associated with the first device. In response to the data accumulation index satisfying a predetermined condition, parameter data associated with a group of actions is transmitted to the second device so that the second device updates the second action model at the second device using the parameter data, the parameter data including reward data and consumption data respectively associated with the group of actions. By utilizing the exemplary implementation of the present disclosure, the rewards and consumption associated with performing actions can be fully considered, and then each action can be performed under the condition of satisfying the consumption constraints so as to increase the corresponding rewards.

Description

Technical Field

[0001] Exemplary implementations of the present disclosure generally relate to performing actions, and more particularly, to methods, apparatuses, devices, and computer-readable storage media for performing actions in a distributed environment. Background Art

[0002] With the development of federated learning and mobile computing, architectures for performing federated learning in distributed environments have been proposed. These architectures can enable multiple clients to collaboratively explore and utilize data while protecting sensitive data. For example, each action can be executed separately at multiple clients, and multiple clients can communicate with the server to gradually obtain the action model used to determine the action to be executed in a distributed manner. However, federated learning involves a large amount of data communication and computation, which leads to various overheads caused by data communication and computation that may affect the performance of machine learning. At this time, how to manage action execution in a more efficient manner and thus complete the federated learning process has become a research hotspot and difficulty. Summary of the Invention

[0003] In a first aspect of the present disclosure, a method for performing an action is provided. In the method, based on a first action model at a first device, a set of actions to be performed at the first device is determined from a plurality of actions. A data accumulation indicator associated with the set of actions is obtained, the data accumulation indicator indicating an amount of data to be sent from the first device to a second device associated with the first device. In response to the data accumulation indicator satisfying a predetermined condition, parameter data associated with the set of actions is transmitted to the second device so that the second device updates the second action model at the second device using the parameter data, the parameter data including reward data and consumption data respectively associated with the set of actions.

[0004] In a second aspect of the present disclosure, a device for performing an action is provided. The device includes: a determination module configured to determine, based on a first action model at a first device, a set of actions to be performed at the first device from a plurality of actions; an acquisition module configured to acquire a data accumulation index associated with the set of actions, the data accumulation index indicating an amount of data to be sent from the first device to a second device associated with the first device; and a transmission module configured to transmit, in response to the data accumulation index satisfying a predetermined condition, parameter data associated with the set of actions to the second device, so that the second device updates a second action model at the second device using the parameter data, the parameter data including reward data and consumption data respectively associated with the set of actions.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0007] In a fifth aspect of the present disclosure, a method for performing an action is provided. In this method, at a second device associated with a plurality of first devices, a plurality of parameter data from the plurality of first devices are respectively received, wherein the parameter data from a first device among the plurality of first devices in the plurality of parameter data is transmitted from the first device to the second device in response to a data accumulation index associated with the first device satisfying a predetermined condition, the data accumulation index indicating the amount of data to be transmitted from the first device to the second device, and the parameter data including reward data and consumption data respectively associated with a set of actions performed at the first device. Aggregated parameter data is determined based on the plurality of parameter data. The aggregated parameter data are respectively transmitted to the plurality of first devices so that the plurality of first devices update the plurality of first action models located at the plurality of first devices based on the aggregated parameter data.

[0008] In a sixth aspect of the present disclosure, a device for performing an action is provided. The device includes: a receiving module configured to receive, at a second device associated with a plurality of first devices, a plurality of parameter data from the plurality of first devices, respectively, wherein the parameter data from the first device in the plurality of first devices is transmitted from the first device to the second device in response to a data accumulation index associated with the first device satisfying a predetermined condition, the data accumulation index indicating the amount of data to be transmitted from the first device to the second device, the parameter data including reward data and consumption data respectively associated with a set of actions performed at the first device; a determining module configured to determine aggregated parameter data based on the plurality of parameter data; and a transmitting module configured to transmit the aggregated parameter data to the plurality of first devices respectively, so that the plurality of first devices respectively update the plurality of first action models located at the plurality of first devices based on the aggregated parameter data.

[0009] In a seventh aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to the fifth aspect of the present disclosure.

[0010] In an eighth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the fifth aspect of the present disclosure.

[0011] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the implementation of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of various implementations of the present disclosure will become more apparent hereinafter with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0013] Figure 1 shows a block diagram of a distributed processing environment in which one exemplary implementation according to the present disclosure may be used;

[0014] Figure 2 shows a block diagram for performing actions according to some implementations of the present disclosure;

[0015] Figure 3 A trajectory diagram illustrating an interactive process of performing actions at multiple devices according to some implementations of the present disclosure;

[0016] Figure 4 A block diagram illustrating an algorithm executed at a first device according to some implementations of the present disclosure;

[0017] Figure 5 A block diagram illustrating an algorithm executed at a second device according to some implementations of the present disclosure;

[0018] Figure 6 A block diagram illustrating a comparison of the performance of an action execution process according to some implementations of the present disclosure and an action execution process according to multiple other technical solutions;

[0019] Figure 7 A block diagram illustrating a comparison of the performance of an action execution process according to some implementations of the present disclosure and an action execution process according to multiple other technical solutions;

[0020] Figure 8 A flowchart illustrating a method for performing an action at a client according to some implementations of the present disclosure is shown;

[0021] Figure 9 A flowchart illustrating a method for performing an action at a server according to some implementations of the present disclosure is shown;

[0022] Figure 10 A block diagram illustrating an apparatus for performing an action according to some implementations of the present disclosure is shown;

[0023] Figure 11 A block diagram illustrating an apparatus for performing actions according to some implementations of the present disclosure; and

[0024] Figure 12 A block diagram is shown of a device capable of implementing various implementations of the present disclosure. DETAILED DESCRIPTION

[0025] The following describes implementations of the present disclosure in more detail with reference to the accompanying drawings. Although certain implementations of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the implementations described herein. Rather, these implementations are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and implementations of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0026] In the description of the implementation of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "an implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". The following may also include other explicit and implicit definitions. As used herein, the term "model" can represent the association relationship between various data. For example, the above-mentioned association relationship can be obtained based on a variety of technical solutions currently known and / or to be developed in the future.

[0027] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0028] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0029] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0030] As an optional but non-limiting implementation, in response to receiving a user's active request, a prompt message may be sent to the user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0031] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0032] As used herein, the term "in response to" refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of executing a subsequent action executed in response to the event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is satisfied. For example, in some cases, a subsequent action may be executed immediately upon the occurrence of the event or the satisfaction of the condition; in other cases, the subsequent action may be executed some time after the occurrence of the event or the satisfaction of the condition.

[0033] Sample Environment

[0034] Sequential decision-making problems exist in numerous real-world applications, such as data recommendation systems, internet data push systems, network parameter configuration, crowdsourcing systems, and clinical trial systems. Decision makers can perform actions in the order in which they decide in order to maximize long-term rewards. This type of problem is often modeled as a Multi-Armed Bandit (MAB) problem, where decision makers need to strike a balance between exploration and exploitation. As a distributed machine learning method, federated learning can protect user sensitive data to a certain extent, which has led to its increasing widespread application. MAB involves leveraging the collaboration of multiple clients to update the model, and can be extended to the context of federated learning. In this case, the central server can leverage distributed datasets from a large number of clients to improve the performance of the MAB algorithm while still protecting the sensitive data of each client.

[0035] However, current research on the federated game machine problem does not consider the resource constraints of executing actions (i.e., pulling the arms of the MAB), which is a key issue in the real world. Generally speaking, throughout the decision-making process, executing actions needs to satisfy resource consumption constraints and achieve the goal of maximizing rewards. For example, in a data push system, it is expected to provide users with various data pushes from data providers, and users' feedback on data pushes (e.g., clicks, etc.) may involve sensitive data protection. The federated game machine can be used to model the data push process, with the goal of optimizing the long-term data push effect based on user feedback. Each data push is associated with a certain consumption. In this case, when executing data push, the decision-making process needs to consider not only the expected rewards (e.g., clicks or conversion results), but also the consumption caused by the data push (e.g., various resources consumed).

[0036] For another example, in the example of base station parameter configuration, a federated game machine can also be used to perform modeling: the base station corresponds to the client, and each base station configuration corresponds to an arm of the MAB. Considering the user experience of the base station service, mobile network operators only allow a limited number of adjustments, which reflects resource constraints. Furthermore, a similar problem can also exist in crowdsourcing scenarios, namely that the crowdsourcing platform needs to assign multiple tasks to workers and provide corresponding rewards to the workers, thus facing budget constraints. However, existing technical solutions do not consider the consumption constraints that should be followed when pulling a certain arm.

[0037] A technical solution, MAB, has been proposed. This solution describes the problem of executing any of K actions on M clients as follows: a client can be represented as a MAB, and pulling a particular arm corresponds to executing one of the K actions. In the traditional MAB setting, arms can be represented as scalars to infer rewards from an unknown arm distribution. As a classic model, linear reward models have been widely studied in the context of game machines. MAB represents an online learning model that can achieve the inherent exploration-exploitation balance in many sequential decision-making problems. Therefore, many real-world problems are solved by modeling them as game machines.

[0038] In recent years, the problem of multi-agent and distributed game machines has received increasing attention, and technical solutions for distributed federated game machines have been proposed. Specifically, the channel selection problem in distributed wireless networks can be considered and modeled as a conflicting MAB, where zero rewards are assigned to clients that select the same arm. At the same time, there are also some technical solutions for cooperation estimation in the MAB problem, focusing on network latency and efficient communication. A distributed linear game machine algorithm has been proposed, which uses an efficient communication model; however, this algorithm does not protect sensitive data. Although a variety of combined technical solutions have been proposed, the following three major challenges make it impossible to directly use existing technical solutions.

[0039] The first challenge comes from computational cost. In classic federated learning, clients lack sufficient resources to perform complex computational tasks, requiring efficient solutions to the knapsack problem and reducing computational overhead. Existing technical solutions solve a linear programming (LP) problem in each round, resulting in high computational cost, especially when the number of clients or arms is large. Oracle-based solutions (i.e., optimal solutions) have also been proposed. However, estimating the optimal solution in the real world is difficult and requires strong assumptions about the class or distribution of the hypothesis. The second challenge comes from communication cost. The knapsack constraints in the federated learning framework significantly increase the complexity of the game machine. Proposed technical solutions reduce communication costs by setting specific communication thresholds. If the number of LP calculations is reduced, unlike in traditional federated game machines, clients will not be able to independently update their local action models in each round. Existing technical solutions only consider communication costs and not computational costs, which is not conducive to improving overall performance. The third challenge comes from protecting sensitive data. In many real-world applications, users' private information needs to be protected. Existing solutions often involve adding noise and perturbations to the original data. However, this will affect the performance of the model, and even when the noise is low, there is still a risk of sensitive data leakage. At this time, how to manage action execution in a more efficient way to complete the federated learning process has become a research hotspot and difficulty.

[0040] Summary of the process of executing an action

[0041] In order to at least partially address the deficiencies in the prior art, according to an exemplary implementation of the present disclosure, a method for executing an action is proposed. Specifically, the present disclosure solves the balance problem of collaborative exploration and exploitation by studying the knapsack problem. Assume that there are M clients, and any of K actions can be executed at each client. Under the coordination of the server, the M clients can execute the desired actions based on linear rewards and consumption, thereby minimizing the total regret. According to an exemplary implementation of the present disclosure, the process of executing an action is implemented using a Federated Linear Bandits with Knapsacks (abbreviated as FedUCBwK) based on a knapsack.

[0042] In the face of challenges in computational costs, the present disclosure takes into account the significant delays caused by high computational costs, and proposes a technical solution that can reduce the number of LP calculations and solve the problem without the need for assumptions. In the face of challenges in communication costs, the present disclosure proposes a new synchronization threshold that allows unified synchronization of communication and computation. If the client only calculates and updates the strategy without timely communication, information from other clients will be lost and the computational effect will be reduced; if only communication is performed without computation, the client will not be able to update the strategy, resulting in the transmission of useless information. In the face of challenges in protecting sensitive data, the technical solution of the present disclosure only transmits model parameters but not the original data, thereby protecting sensitive data.

[0043] See also Figure 1 Describes an overview of the distributed processing environment. Figure 1 1 shows a block diagram of a distributed processing environment 100 in which one exemplary implementation according to the present disclosure may be used. Figure 1 As shown, the distributed processing environment 100 may include a server 110 and multiple clients 120, ..., and 140. In this case, each client can be considered as having K arms MAB. Each client can pull any of the K arms (corresponding to performing a certain action). Each pulling of an arm incurs a certain cost, and the goal of pulling an arm is to maximize the reward obtained at the client. A linear programming solution process can be executed at the client to determine the action to be performed.

[0044] like Figure 1As shown, the learning process can be performed at multiple clients and servers based on a federated learning architecture. Specifically, at the client 120, the various actions to be performed can be determined based on the LP solution 126 so as to achieve the goal of maximizing the reward 122 under the constraint of the consumption 124. Further, at the client 140, the various actions to be performed can be determined based on the LP solution 146 so as to achieve the goal of maximizing the reward 142 under the constraint of the consumption 144. Parameters 130 (e.g., parameters related to arm selection, etc.) can be transmitted from the client 120 to the server 110, and parameters 150 (e.g., parameters related to arm selection, etc.) can be transmitted from the client 140 to the server 110.

[0045] Server 110 can aggregate parameters from various clients to update the parameters of the action model at the server. Furthermore, server 110 can transmit corresponding parameters 132 (e.g., aggregated parameters), ..., and 152 to clients 120, ..., and 140, respectively, to update the action model at each client. In this way, multiple constraints on rewards and costs can be considered in federated learning, thereby obtaining an action model that better matches actual needs.

[0046] Further, see Figure 2 More details about an exemplary implementation according to the present disclosure are shown. Figure 2 A block diagram 200 is shown for performing actions according to some implementations of the present disclosure. Figure 2 As shown, in a system including a first device 210 (eg, Figure 1 client shown) and a second device 220 (e.g., Figure 1 Specifically, the first device 210 may include a first action model 212, and the second device 220 may include a second action model 222. The first action model 212 and the second action model 222 may be updated based on federated learning.

[0047] During the federated learning process, a set of actions 214 to be performed at the first device 210 may be determined from a plurality of actions based on a first action model 212 at the first device 210. A data accumulation indicator 216 associated with the set of actions 214 may be obtained based on the set of actions 214. The data accumulation indicator 216 may indicate an amount of data to be sent from the first device 210 to a second device 220 associated with the first device 210. If the data accumulation indicator 216 satisfies a predetermined condition, parameter data 218 associated with the set of actions 214 (e.g., rewards and costs associated with the set of actions determined at the first device 210) may be sent from the first device 210 to the second device 220, so that the second device uses the parameter data to update a second action model 222 at the second device 220.

[0048] In other words, the data accumulation indicator 216 may indicate whether a communication round has been initiated between the first device 210 and the second device 220. In this way, the communication overhead between the two devices may be reduced, thereby increasing the efficiency of performing actions. Figure 2 Only a single first device 210 is schematically shown; other first devices may exist in a distributed environment. In this case, the second device 220 may receive parameter data from other first devices, generate aggregated parameter data 224, and transmit the aggregated parameter data 224 to each first device to update the local motion model.

[0049] According to an exemplary implementation of the present disclosure, the data accumulation indicator 216 may be used to indicate whether to send data to the second device 220. If the parameter data generated by the actions executed at the first device 210 reaches a predetermined threshold, the accumulated parameter data may be sent to the second device 220. If the amount of accumulated data does not reach the predetermined threshold, one or more actions may continue to be executed and the LP solution may continue to be performed at the first device 210, thereby continuing to accumulate parameter data to be transmitted.

[0050] The proposed technical solution can use a unified threshold condition to solve the distributed knapsack problem in each round, which can balance the regret, communication and computing costs of the client. In this way, at the first device 210, on the one hand, the goal of maximizing the reward can be achieved while satisfying the consumption constraint, and on the other hand, the communication overhead and time overhead caused by too frequent communication between the first device 210 and the second device 220 can be avoided, thereby finding a balance between exploration and utilization and improving the overall performance of federated learning. In addition, during the process of executing actions and updating action models, the collected real data (for example, user clicks and other operations) is not transmitted, so that the action model can be updated in a distributed manner while ensuring the security of sensitive data.

[0051] Detailed process of executing the action

[0052] Having described an overview of an exemplary implementation according to the present disclosure, in the following, see Figure 3 Describes more details of executing actions in a distributed manner. For ease of description, only the data push task is used as a specific application environment to describe the specific application environment in which the action is executed. For example, in a data push system, data can be pushed to a terminal device used by a user, and the action here can correspond to the action of pushing data to the user. Figure 3 A track diagram 300 is shown of an interactive process of performing actions at multiple devices according to some implementations of the present disclosure. Figure 3 As shown, actions can be performed at multiple first devices 210, ..., and 310, and corresponding action models can be updated based on the performed actions. In this case, the first device 210 and the first device 310 can respectively include corresponding first action models, and the second device 220 can include a second action model.

[0053] During the federated learning process, multiple actions can be performed at the first device 210. Here, the multiple actions can be actions of pulling a certain arm of the MAB. In the initial stage, the multiple actions can be performed 320 in sequence, and initial parameter data associated with the multiple actions can be transmitted 322 to the second device 220. Similar operations can be performed at the first device 310, for example, multiple actions can be performed 320' at the first device 310, and initial parameter data associated with the multiple actions can be transmitted 322' to the second device 220. The second device 220 can receive initial parameter data from each first device (e.g., the first device 210, ..., and 310), and then determine 324 aggregated initial parameter data. The aggregated initial parameter data can be used to update the second model at the second device 220.

[0054] Furthermore, the second device 220 may transmit 326 the aggregated initial parameter data to the first device 210. Similarly, the second device 220 may transmit 326' the aggregated initial parameter data to the first device 310. The aggregated initial parameter data may be used to update the first motion model at the first device 210; similarly, the aggregated initial parameter data may be used to update the first motion model at the first device 310. Here, the process illustrated by arrows 320 to 328' involves an initialization phase, and the respective first motion models at the first devices 210, ..., 310 and the second motion model at the second device 220 may be preliminarily updated.

[0055] After the initialization phase, the updated action model can be used at each first device to determine an action to be performed. For example, at the first device 210, a set of actions can be performed 330 based on the updated first action model (where the number of actions in the set depends on whether the data accumulation indicator meets a predetermined condition). The updated first action model can be used to determine one or more actions to be performed and these actions can be performed.

[0056] Furthermore, a data accumulation index associated with the set of actions can be determined. For example, the data accumulation index can be determined using characteristics of the actions that have already been performed. If the data accumulation index satisfies 332 a predetermined condition, a communication round can be initiated from the first device 210 to the second device 220 to transmit 334 parameter data associated with the set of actions. If the data accumulation index does not satisfy the predetermined condition, the operational flow can return to the position indicated by arrow 330 and continue to determine and execute one or more subsequent actions based on the updated first action model until the data accumulation index associated with the set of actions that have already been performed satisfies the predetermined condition.

[0057] Similar operations may be performed at the first device 310, for example, a set of actions 330' may be performed based on the updated first action model. If the data accumulation indicator associated with the set of actions satisfies 332' a predetermined condition, parameter data associated with the set of actions may be transmitted 334' to the second device 220.

[0058] Furthermore, the second device 220 may determine aggregated parameter data based on the parameter data received from each of the first devices. The aggregated parameter data may be used to update the second model at the second device 220. Furthermore, the second device 220 may transmit 338 the aggregated parameter data to the first device 210. Similarly, the second device 220 may transmit 338' the aggregated parameter data to the first device 310. At this point, the operational flow may return to the point indicated by arrows 328 and 328', and each first device may update its local first motion model using the newly received parameter data.

[0059] According to an exemplary implementation of the present disclosure, a predetermined loop end condition may be set, for example, a predetermined time length may be set, and the loop ends when the predetermined time length is reached. Figure 3 It will be understood that although Figure 3 Only operations at the first devices 210 and 310 are schematically shown; there may be more first devices, and the process performed at each first device may be similar.

[0060] Using the exemplary implementation of the present disclosure, the second action model at the second device 220 is an accurate action model obtained via federated learning technology. This action model can achieve the goal of maximizing rewards while satisfying predetermined consumption constraints. In this case, the first device 210 does not need to transmit the parameter data solved in each round of LP to the second device 220 after executing each action. Instead, the first device 210 can accumulate the parameter data generated by multiple action executions and transmit it when the data accumulation index meets predetermined conditions. In this way, it can be ensured that the federated learning process does not incur excessive communication overhead.

[0061] Having described the overview of the collaboration between the client and the server, the following will introduce the specific formulas related to the action execution process. According to an exemplary implementation of the present disclosure, FedUCBwK can be implemented in a server-client architecture. Assume that there are M clients, each of which can access K actions (arms) in d dimensions, represented as a∈[K]:={1,2…,K}. At each time t∈[T], action a can be executed at each client m∈[M]. t,m And observe the rewards and consumption. t,m The feature (i.e. context) of is represented as Denote the unknown expected reward as r t,m , the corresponding resource consumption is expressed as c t,m Assume that there is a fixed total consumption budget This budget is known to the server. B is a hard limit on resource consumption, and the algorithm terminates when B is exhausted. Alternatively and / or additionally, the algorithm may terminate when the runtime reaches a predetermined time length T.

[0062] According to an exemplary implementation of the present disclosure, a linear structure with global parameters is proposed. In order to obtain the intrinsic correlation between the rewards for performing the same action at different clients, it can be assumed that the reward r t,m Has a linear structure: in is a fixed but unknown vector, and Is for reward r t,m Independent random noise with zero mean. In addition, assuming that the consumption c t,m Has a linear structure: in is a fixed but unknown parameter vector, and is for cost c t,m with independent random noise of zero mean.

[0063] In terms of communication, assuming a central server exists in a distributed environment, clients can periodically communicate with the server. Specifically, during each communication, the client can send locally accumulated parameter data to the central server, which can then aggregate the received parameter data to update the action model and calculate the current policy. The central server can then broadcast this policy to all clients. The communication cost of the algorithm can be defined as the scalar (integer or real) number of communications between the server and the client.

[0064] In terms of computation, the computational problem can be described as a MAB problem in a linear static environment, so the present disclosure solves the original problem by decomposing it into multiple linear programming problems. The present disclosure defines the computational cost of the algorithm as the number of linear programming problems solved.

[0065] In terms of sensitive data protection, the context of each client is static, which can be understood as the characteristics and usage habits of each client will not change within a certain period of time. Although the context does not change over time, the context of each client is personal information, and the client needs to protect its own sensitive data. In addition, the reward for each action, such as the user's feedback (clicks and purchases) on the data push in the data push system, is also sensitive information on the client. The technical solution disclosed in this disclosure only requires each client to transmit the parameter data calculated in each round, without the need to directly transmit the original data.

[0066] According to an exemplary implementation of the present disclosure, in terms of regret, the objective function may be determined based on the following formula, that is, the objective function is to perform actions so as to minimize the expected regret among all clients:

[0067]

[0068] In this formula, REGRET(T) represents the objective function, OPT represents the total expected reward of the optimal policy, T represents the scheduled time length, M represents the number of clients, and a t,m represents the action executed on client m at time t, x(a t,m ) represents the characteristics of the corresponding action, and θ represents the reward-related parameters. At the same time, it is necessary to ensure that the consumption does not exceed the overall consumption budget, that is,

[0069] In terms of distributed selection, regret can represent the difference between the total reward obtained by the algorithm of the present disclosure and OPT. At this point, the problem description of the present disclosure adopts a federated setting, and the present disclosure extends the OPT-LP problem to a distributed scenario.

[0070] In terms of LP relaxation, this disclosure defines a linear programming relaxation of the expected total reward under the mixed-arm strategy. This disclosure can describe any possible expected reward and possible expected cost for each action based on the following linear operation:

[0071]

[0072] In the above formula, represents the matrix and includes x of the K arms, p i It can be expressed as the probability of pulling arm i. Let represents the LP relaxation value, and β represents the consumption-related parameter. It can be concluded that:

[0073]

[0074] is the upper bound of OPT, which is denoted as OPT-LP in the present disclosure. Therefore, the present disclosure can use OPT-LP to replace OPT in regret analysis.

[0075] In terms of distributed decomposition, when there are multiple clients, it is possible to determine how to allocate the budget among them. In the case of a static context setup, each arm has a corresponding context that does not change over time for all clients. Therefore, decomposition can be performed by evenly distributing the budget across all clients.

[0076] First, according to the LP relaxation, a budget can be allocated to each round so that the budget will reach B / T in each period t. Assuming that the budget is further divided and distributed to M clients, for each client, the distributed LP problem can be expressed in the following form:

[0077]

[0078] In the above formula, B m represents the budget allocated to client m. Based on formula 4, we can get:

[0079]

[0080] If there is an optimal solution, then for each m there exists A more intuitive understanding is that the current budget must be higher than the arm with the lowest cost. Otherwise, there is no arm that can be pulled. By using the strong duality theorem, we can get the following formula:

[0081]

[0082] Therefore, the present disclosure can further obtain the total reward of M customers:

[0083]

[0084] For all m=M, y m =y. That is, no matter how the budget is allocated, as long as there is an optimal solution, the goal achieved by the LP relaxation optimal solution is equal. For simplicity, the budget can be evenly distributed among the clients. Therefore, for any make Represent the value of the following linear programming.

[0085]

[0086] In regret analysis, one can use to replace OPT.

[0087] In order to protect the sensitive data of the client data, the present disclosure only transmits the parameter data, and each client saves the original data locally. Therefore, the calculation of the knapsack problem and arm selection is performed locally on each client. The central server is only responsible for aggregating the received parameter data and then sending it to each client. In addition, in order to meet the calculation requirements, the central server sends B / M (alternatively and / or additionally, the budget can also be distributed unevenly) instead of sending the original total budget to each client. Each client only knows the budget allocated to him, but not the total value. In this way, the sensitive data of both parties can be protected.

[0088] In order to solve the problem of computational and communication efficiency, a unified communication scheme is proposed. Specifically, within a time period, when the amount of accumulated information (i.e., the pulled arms and the corresponding feedback results) is limited, the amount of parameter data will not change much. Therefore, the present disclosure can divide the total time into different communication rounds, perform communication and synchronization at the end of each communication round, and solve the LP problem. When estimating parameters, the linear game machine problem can be handled based on "optimism in the face of uncertainty", thereby obtaining optimistic estimates of the parameters β and θ.

[0089] In the following, see Figure 4 and Figure 5 The specific steps in the description are used to describe the specific process of performing the action. Figure 4 A block diagram of an algorithm 400 executed at a first device (eg, a client) according to some implementations of the present disclosure is shown, and Figure 5 A block diagram of an algorithm 500 executed at a second device (eg, a server) according to some implementations of the present disclosure is shown. Figure 4 and Figure 5 The algorithm shown implements the processes performed at the client m and the central server.

[0090] In summary, in the initial stage, each client m can perform various actions and observe the feedback (corresponding rewards and costs) to initialize. After initialization, until the algorithm stops (i.e., when the time range T is not reached and the budget is not exhausted), each communication round can be performed one by one at the client. In each round e of the communication round in the algorithm 400, each client m can calculate the round p calculated at the end of the previous round (line 11). e Then, record the number of times action a is executed f(m,a) (line 12) and update the matrix V t,m , total reward R m,a and the total cost of each action C m,a (Lines 14-16). Then, the matrices of all actions can be added together to get the total matrix V for client m t,m This matrix can be considered as a matrix containing data up to time t (row 17). Then determine whether the following formula is true (that is, determine whether the data accumulation index meets the predetermined conditions):

[0091]

[0092] In this formula, D is the threshold used to determine whether synchronization is required. The intuitive understanding of is the amount of historical information accumulated in this round. According to an exemplary implementation of the present disclosure, D can be set to different values ​​(for example, a constant 1 or other values). If the communication round starts, the client will reset And enter a new round (line 21). Using historical data sample V m,a and the corresponding total reward R m,a and consumption, and projecting it into the x(a) dimension by applying linear regression, the parameters can be estimated and According to an exemplary implementation of the present disclosure, the context x is static, so the direction x(a t,m ) is the same at all times, so it can be uniformly represented by x(a).

[0093]

[0094]

[0095] Afterwards, all clients can send their calculated parameter data and the number of times the action was performed to the central server for aggregation. At this point, the central server only has the relevant parameters for different actions. and However, it does not have any context information of the action, so it cannot obtain the matrix. The direction is the same as x(a), so we can get the unit vector of x(a) In addition, the central server can record the number of executions for each action. Therefore, the present disclosure can summarize and calculate the aggregated parameter data in the following way:

[0096]

[0097] After receiving the above parameters from the central server, each client can maintain a confidence set for the parameters θ in round e. And maintain the confidence set for the parameter β Specifically, following the common technique used in previous work on linear context games, the confidence set Is centered on the ellipsoid The confidence set The ellipsoid is centered Can be used and V e To construct the confidence set And use and V e To construct

[0098]

[0099] In the above formula, This explains why the synchronization condition is set to Equation 9: The volume of the confidence ellipsoid for both θ and β depends on det(V t,m ). If det(V t,m ) does not change much, even if the confidence ellipsoid changes slightly, it will not affect the confidence guarantee. In addition, if the change is not large, V will always be maintained in each round e. e Therefore, det(V e ) instead of det(V t,m ) to participate in the calculation. Then optimistic estimates of these parameters can be used:

[0100]

[0101]

[0102] At this point, the confidence upper bound of the reward and the confidence lower bound of the consumption can be calculated. Each client will then calculate and update the arm selection strategy for the next round. The following linear programming problem can be solved:

[0103]

[0104] It will be appreciated that in order to satisfy the hard constraints, one can use replace to allow for some estimation error. In this case, the algorithm will not terminate prematurely due to exhaustion of the budget.

[0105] Having described the formulas involved in algorithms 400 and 500, see below. Figure 4 and Figure 5 Describes the interaction between the client and the server in more detail. Figure 4 As shown, the input data at the client may include: T, representing a predetermined time length; M, representing the number of clients; D, a predetermined threshold, indicating a predetermined condition for initiating data transmission to the second device 220; and ∈, representing a consumption constraint. During the client's initialization phase, as shown in line 1 of algorithm 400, intermediate variables e (number of rounds) and t (elapsed execution time) may be set. As shown in line 3 of algorithm 400, data B / M (representing the budget available to each client) may be received from the server.

[0106] See also Figure 5 Algorithm 500 is described, which is executed at a server. The server may receive the following input data: T, which represents a predetermined time duration; M, which represents the number of clients; and B, which represents the overall budget of resources that can be allocated (the corresponding consumption can be deducted from the overall budget when each action is performed). At line 1 of algorithm 500, data B / M, i.e., the amount of available resources allocated to each client, may be sent to each client. At line 2 of algorithm 500, a round number e for performing communication with the client may be set, and round number e may be set to 0 in the initial stage.

[0107] According to an exemplary implementation of the present disclosure, multiple actions can be executed at the client, and multiple rewards and multiple consumptions associated with the multiple actions can be obtained respectively. Figure 4 Describing the operations performed at the client, as shown in line 3 of the algorithm 400, each action a can be performed at the client, and a reward r corresponding to performing action a at the client m can be obtained. m,a and consumption c m,a According to an exemplary implementation of the present disclosure, as shown in line 4 of the algorithm 400, the data accumulation index can be determined based on the characteristics of each action in a set of actions. Here, λ and I may have predetermined values, and x(a t,m ) can represent an action a performed at time t and client m t,m In the initialization phase, the data accumulation index can be determined based on the sum of the outer products of the features of the K actions executed at the client. In the subsequent stages after the initialization stage, the outer products of the features of the actions that have been executed are continuously accumulated to update the data accumulation index.

[0108] Using the exemplary implementation of the present disclosure, the data accumulation index The size of the data accumulated by actions executed on the client can be used to determine whether the accumulated data needs to be transmitted to the server. In this way, data can be transmitted from the client to the server only when the accumulated data meets predetermined conditions, thereby reducing various communication-related overheads and improving the performance of federated learning.

[0109] According to an exemplary implementation of the present disclosure, initial parameter data associated with multiple actions can be determined based on multiple actions, multiple rewards, and multiple consumptions. Subsequently, the initial parameter data can be transmitted to a server so that the server can use the initial parameter data to update the corresponding action model. Here, the reward r m,a Can represent the action to be performed t,m The income generated, and the consumption of m,a It can represent the consumption of resources allocated to the client resulting from executing an action. In the context of data push, rewards can represent, for example, an increase in click-through rate resulting from pushing certain data, and consumption can represent the resources consumed by pushing that data. Using the exemplary implementations of this disclosure, both the reward and consumption resulting from executing an action can be fully considered, thereby maximizing the reward while satisfying the consumption constraint.

[0110] Continue to see Figure 4 To describe the specific method of determining the initial parameter data, the initial parameter data may include multiple aspects of data: reward data and consumption data. According to an exemplary implementation of the present disclosure, the reward data can be determined based on the linear calculation of actions and rewards. And consumption data can be determined based on linear calculation of action and consumption Specifically, the reward data can be determined based on the formula shown in the 5th row of the algorithm 400. and consumption data In the above formula, it is convenient to use a set of K action features x(a t,m ) and the corresponding reward r m,a and consumption c m,a The corresponding reward and consumption parameters are determined by linear operations. Furthermore, the number of times each action is executed can be counted to determine the number of times in the parameter data. During the initialization phase, the number of times each action is executed is 1. With subsequent execution, the number of times each action is executed can increase.

[0111] As shown in line 6 of algorithm 400, in the initialization phase, after each action has been performed, reward data for each action may be sent from the client to the server. and consumption data So that the aggregated parameter data can be determined by the server and the action model at the server can be updated.

[0112] In the following, see Figure 5 Detailed description of the process performed at the server is provided below. According to an exemplary implementation of the present disclosure, a plurality of initial parameter data are received at the server from a plurality of clients. The initial parameter data from each client in the plurality of initial parameter data is determined based on a plurality of actions performed at the client and a plurality of rewards and a plurality of costs associated with the plurality of actions.

[0113] Specifically, at line 3 of the algorithm 500, the server may receive the initial parameter data from each client m (i.e., the reward data determined at lines 4 and 5 of the algorithm 400). and consumption data Here, each initial parameter data may come from a corresponding client and be determined based on the K actions performed at the client and the multiple rewards and multiple costs associated with each of the K actions. In this way, the server may receive the initial parameter data from each client, perform an aggregation operation based on the initial parameter data received from each client, obtain aggregated initial parameter data, and update the action model at the server.

[0114] Furthermore, the server may determine aggregated initial parameter data based on multiple initial parameter data. Lines 4 to 6 of the algorithm 500 show specific formulas for generating aggregated initial parameter data. Here, the aggregated initial parameter data may include: aggregated cumulative data, aggregated reward data, and aggregated consumption data. Specifically, line 4 shows a formula for determining aggregated cumulative data. For example, the aggregated cumulative data V in the aggregated parameter data may be determined based on the reward data. 0 As shown in line 4 of algorithm 500, the ratio of the outer product of the reward data from each client to the square of the norm of the reward data can be determined, and then the ratio can be summed. The data from each client of the M clients can be aggregated. For example, the sums for each client can be added together to obtain the aggregated cumulative data V 0 .

[0115] Further, line 5 of the algorithm 500 shows the reward data based on and the aggregated cumulative data V 0, determine the aggregated reward data in the aggregated initial parameter data and row 6 shows the formula based on consumption data and the aggregated cumulative data V 0 , determine the aggregate consumption data in the aggregate initial parameter data In this way, in the initial stage, the motion model at the server can be updated based on the parameter data from each client, thereby improving the accuracy of the motion model in a distributed training manner.

[0116] Then, the server can transmit the aggregated initial parameter data to the multiple clients respectively, so that the multiple clients can update their local motion models based on the aggregated initial parameter data. Specifically, at line 7 of the algorithm 500, the determined aggregated initial parameter data (including V 0 、 and ) is transmitted to each client. In this way, each client can update its local action model based on the received aggregated initial parameter data, thereby achieving federated learning.

[0117] Furthermore, the client may receive aggregated initial parameter data for updating the local action model (the aggregated initial parameter data is determined by the second device based on the initial parameter data), so as to update the local action model using the aggregated initial parameter data. Figure 4 Describing further operations performed at each client, at line 7 of the algorithm 400, aggregated initial parameter data (i.e., including V 0 、 and ). Further, at the client, the vector p that satisfies the maximization condition can be solved based on the formula shown in line 8 of the algorithm 400. 0 , the vector includes K dimensions and the i-th dimension represents the confidence of performing the i-th action among the K actions. It will be understood that the confidence in row 8 can be determined based on formula 13 described above. And the value in row 8 can be determined based on formula 14 described above.

[0118] At this point, the initialization operation at the client is completed, and a set of actions in the plurality of actions can be executed in the subsequent operation process. When the cumulative data indicator associated with the executed action meets the predetermined condition, the corresponding parameter data can be transmitted from the client to the server. At line 9 of algorithm 400, the termination condition of the operation process can be set, that is, the algorithm is terminated when the execution time reaches the predetermined time length T. At line 10 of algorithm 400, it can be determined whether the resources allocated to the client have been exhausted, that is, whether the corresponding allocated budget has been exhausted. If the budget has not been exhausted, the corresponding action can be selected and executed based on the confidence of the determined execution action. If the budget has been exhausted, the loop is exited. In this way, the federated learning process can be managed in a more flexible manner and terminated when necessary.

[0119] According to an exemplary implementation of the present disclosure, the target action can be determined from multiple actions by the vector output by the updated action model. In the first loop after the initialization phase, as shown in line 11 of the algorithm 400, the target action can be determined based on the vector p e (e is 0 at this time) to determine the target action a to be executed at this time t,m , and based on this action a t,m To update the various parameters at the client. As shown in line 12 of algorithm 400, action a can be performed incrementally t,m The number of times f(m,a t,m ); As shown in line 13, the reward r associated with executing the action can be obtained respectively m,a and consumption c m,a As shown in line 14, the data accumulation index V can be updated based on the characteristics of the target action. t,m ; As shown in line 15, the reward data R at the client can be updated m,a (That is, reward r m,a As shown in line 16, the consumption data C can be updated m,a (That is, consumption c m,a ).

[0120] Furthermore, as shown in line 17, the cumulative data index V at this time can be determined. t,m Whether the predetermined condition is satisfied (i.e., whether Formula 9 described above is true). If the judgment result is "yes", the operation flow proceeds to line 18, and the client can send a synchronization signal to the server to start the communication round. If the judgment result is "no", the operation continues to the next round.

[0121] As shown in line 19, if it is determined that the communication round has been started, the operations shown in lines 20 to 26 are performed. Specifically, as shown in line 20, the parameters used as the basis for comparison in the next round may be updated. The current round e can be incremented. Further, the target action, and the target reward and target consumption associated with the target action can be used to update the parameter data. Specifically, the reward data for the eth round in the parameter data can be updated based on the formula shown in line 21. And the consumption data of the eth round in the parameter data can be updated based on the formula shown in line 22 Further, at line 23, the client may send updated parameter data (e.g., including reward data) to the server. Consumption data And the number of executions of each action f(m,a t,m )).

[0122] See also Figure 5 Describing the process executed at the server, at line 8 of algorithm 500, a determination may be made as to whether a predetermined time duration T has been reached. If the determination is "yes," the loop operations shown in lines 8 through 14 are terminated. If the determination is "no," the operation proceeds to line 9 to determine whether a communication round has been initiated at each client. If a communication round has been initiated, the steps shown in lines 10 through 14 are executed. At this point, as shown in line 10, the server may increment the communication round number e and receive updated parameter data from each client.

[0123] Specifically, parameter data from each of a plurality of clients can be received separately. Here, the parameter data from a client in a plurality of clients among the plurality of parameter data is transmitted from the client to the server in response to the data accumulation index associated with the client satisfying a predetermined condition, and the data accumulation index indicates the amount of data to be transmitted from the client to the server. In other words, the parameter data received here is the accumulated data formed after the training process at the client accumulates to a certain stage. In this way, the client does not have to transmit the parameter data from the execution of a single action one by one, but relies on specific predetermined conditions to perform a set of actions at the client, and then transmit the corresponding parameter data to the server. Thus, the number of communication rounds between the client and the server can be reduced, thereby reducing the communication overhead while ensuring the training effect.

[0124] According to an exemplary implementation of the present disclosure, the parameter data received by the server may include reward data and consumption data associated with a set of actions performed at the client. For a specific client m, the parameter data may include the reward data of the client m. and consumption data Here, reward data It can represent the benefits generated by each action in a set of actions executed at client m; and consume data It can represent the consumption of resources allocated to the client by each action in a set of actions executed at the client m. Further, the parameter data may include the number of times each action is performed f(m,a t,m The number of times data may identify the number of times an action in a set of actions executed at client m is executed. In this way, updated parameters for the action model generated at each client based on different action data may be collected from each client in a distributed manner, thereby facilitating improved training accuracy of the action model.

[0125] According to an exemplary implementation of the present disclosure, the aggregated parameter data may be determined based on the following method. For example, the aggregated cumulative data in the aggregated parameter data may be determined based on the number of times data and the reward data. Specifically, as shown in line 11 of the algorithm 500, the aggregated cumulative data V in the aggregated parameter data may be determined for the current communication round e based on the number of times data and the reward data. e .

[0126] According to an exemplary implementation of the present disclosure, the aggregated reward data in the aggregated parameter data may be determined based on the number of times data and the reward data. Specifically, as shown in line 12, the aggregated reward data in the aggregated parameter data may be determined for the current communication round e based on the number of times data and the reward data.

[0127] According to an exemplary implementation of the present disclosure, the aggregated consumption data in the aggregated parameter data may be determined based on the number of times data and the consumption data. As shown in line 13, the aggregated consumption data in the aggregated parameter data may be determined for the current communication round e based on the number of times data and the consumption data. In this way, aggregated parameter data can be determined in a simple and unambiguous manner to update the action model at the server.

[0128] Further, as shown in line 13, the server may send aggregated parameter data (eg, including aggregated cumulative data V for the current communication round e) to each client. e Aggregated reward data and aggregated consumption data At this point, these aggregated parameter data can be used to update the action model at each client, thereby continuously performing federated learning.

[0129] According to an exemplary implementation of the present disclosure, the client may receive aggregated parameter data for updating the client's local action model from the server, and the aggregated parameter data is determined by the server based on the parameter data (as shown in lines 11 to 13 of algorithm 500). Figure 4 At line 24 of algorithm 400, the client may receive the aggregated cumulative data V for the current communication round e. e Aggregated reward data and aggregated consumption data Furthermore, as shown in line 25, the received data can be used to solve and obtain the vector p describing the probability of performing each action. e In this way, the received aggregated parameter data can be continuously used to update the client's local action model.

[0130] The above has described the determination of whether to initiate a communication round of transmitting parameter data from the client to the server based on whether the data accumulation index meets the predetermined condition. In this way, a communication round can be initiated only when the data accumulation index meets the predetermined condition, thereby achieving the goal of reducing communication overhead. According to an exemplary implementation of the present disclosure, the following can be executed at multiple clients: Figure 4 The algorithm shown in FIG, and executed at the server as Figure 5 The algorithm shown in Figure 2. Multiple clients and servers can transmit parameter data through communication rounds, thereby achieving the process of using federated learning to update action models located in different locations.

[0131] According to an exemplary implementation of the present disclosure, the above process can be terminated when a predetermined condition is met (e.g., a predetermined time length is reached, or a predetermined budget is exhausted). At this point, the training process of the motion model is completed, and the trained motion model can be used to perform the corresponding task.

[0132] In the following, we provide theoretical justifications for regret, communication cost, and computational cost. For regret, we provide the following proof.

[0133] Theorem 6.1: Using Algorithms 400 and 500, where The following upper bound probability for the regret boundary can be obtained:

[0134]

[0135] Lemma 6.1: For any δ>0, with probability 1-Mδ, for all t and all m, θ is always in the constructed Within.

[0136] Lemma 6.2: Let express The sequence in , V is a d*d positive definite matrix and defines It can be concluded that

[0137]

[0138] Furthermore, if for all t there exists ||X t ||2≤L, then there exists:

[0139]

[0140] Lemma 6.3: With probability 1-Mδ, the single-step difference The boundaries are:

[0141]

[0142] Lemma 6.4: (Azuma-Hoeffding inequality). If corresponding to the filter Supermartingale (Y t :t≥0) for a constant c t Satisfy |Y t -Y t-1 |≤c t , for all t=1,…T, then for any a≥0, there exists:

[0143]

[0144] Lemma 6.5: Let A, B, and C denote positive semidefinite matrices such that A = B + C. Then there exists:

[0145]

[0146] Lemma 6.6: Define Diff(T) as:

[0147]

[0148] Then Diff(T) is given by To limit.

[0149] In the following, the proof process will be provided. First, although the same estimate is used in each round e For analytical purposes you can use To express the estimate for each time period t and each client m, that is, In this disclosure, the number of rounds will be divided according to the communication round. Assuming there are E rounds, the aggregation matrix in round e is represented as V e By using the communication threshold, we can obtain:

[0150]

[0151] Otherwise, synchronization will occur. MT pulls are all done by a client in a round-robin manner (i.e. according to a 1,1 ,a 1,2 ,…,a 1,M ,a 2,1 ,…,a T,M ) can be used. To indicate that when the client executes x(a t,m ) can be obtained. Through the algorithm disclosed in this disclosure, each client m will use the random strategy generated by the aggregated matrix in each communication round received from the central server. Therefore, the gap between the matrices can be constrained as:

[0152]

[0153] Therefore, according to Lemma 6.5 we can obtain:

[0154]

[0155] Then, we can use a single client difference bound and prove the difference. Assume Denote the set of (t,m) pairs belonging to round e. Using Lemmas 6.2 and 6.3, let You can get:

[0156]

[0157]

[0158] and It is estimated that the following properties are satisfied. In the case of probability 1-Mδ,

[0159] Properties(1):

[0160] Properties (2):

[0161]

[0162]

[0163] because Using Lemma 6.4, we can obtain with probability 1-Mδ:

[0164]

[0165] Then, according to Diff(T) in Lemma 6.6, we can use a confidence bound to constrain the gap between the true reward and the expected reward of the pulled arm under the fractional solution:

[0166]

[0167] In the same way as above, we can further obtain:

[0168]

[0169] Then you can set So far, property (2) can be obtained. According to property (1), the definition of OPT and problem decomposition can be expressed as:

[0170]

[0171] Then, using the above property (2) and formula 18, the hard constraint can be satisfied, namely:

[0172]

[0173] Therefore, the algorithm will not terminate before time T. The total real reward obtained can be expressed as REW:

[0174]

[0175] Thus, there can be:

[0176]

[0177]

[0178]

[0179] In terms of communication cost, according to Algorithm 400 and Algorithm 500, it can be determined that if there is Then the communication round begins, that is:

[0180]

[0181] Then by Lemma 6.2, we can get:

[0182]

[0183] Furthermore, there exists:

[0184]

[0185] According to an exemplary implementation of the present disclosure, only at the end of each round, when each client sends O(dK) numbers to the server and then downloads O(d2 ) numbers, communication is required. Therefore, in each round, the communication cost is O(Md(d+K)). Therefore, the total communication cost is

[0186] In terms of computational cost, in Algorithm 400 and Algorithm 500, the same threshold is used, so the linear programming is calculated when the communication round begins. The local feature vector X is private information and is unknown to the central server, so the LP calculation is only performed at each client. Therefore, the computational cost is

[0187] In the context of this disclosure, OPT, or distributed OPT, can be defined, and the feasibility of this solution is demonstrated through the distributed decomposition of LP. Here, FedUCBwK keeps the total budget private through budget allocation and performs policy calculation on the client. It transmits model update parameters instead of raw data to protect sensitive data on the client. Based on the transmitted parameters, this disclosure designs unified communication and computation thresholds to solve the distributed LP problem in each round and controls regret through the budget deflator. In addition, a trade-off can be made between regret, communication cost, and computation cost. Under the communication cost and computation cost, this disclosure can obtain a high-probability regret bound.

[0188] According to an exemplary implementation of the present disclosure, a variety of tasks can be performed on a real data set to verify the performance of the task execution process described above. The performance of multiple different technical solutions can be compared. Specifically, FedUCBwK can represent the algorithm proposed in the present disclosure; FedUCBwK-FullCom represents a technical solution in which the client performs communication and calculation in each round (D in this case can be set to a smaller value); FedUCBwK-NoCom represents a technical solution in which the client does not perform communication (D in this case can be set to a larger value); FedUCBwK-FewCom represents a technical solution in which the client only performs a small amount of communication and calculation (D in this case can be set to an intermediate value); FedUCB and FedUCB-FullCom represent existing technical solutions.

[0189] According to an exemplary implementation of the present disclosure, the technical solution of the present disclosure can be verified on multiple public datasets. Specifically, it can be verified on the MovieLens-100K dataset. In order to handle sparse and incomplete rating matrices, the present disclosure uses collaborative filtering to complete the rating matrix, and then uses non-negative matrix factorization with 10 latent factors to obtain R = WH, where

[0190] According to an exemplary implementation of the present disclosure, the dataset can be used to simulate two scenarios of a data push system. The first scenario involves pushing appropriate data to a specific user group. The present disclosure considers each data push corresponding to a certain reward and cost as an action. The goal is to find the best data push for a specific user group. To this end, the k-means algorithm can be applied to the column vector of W to generate 20 categories. One category can be selected as a specific user group u, and θ u As the center of the selected user group, the present disclosure can then apply the k-means algorithm to the row vector of H to generate K=10 groups (actions). At this time, the action performed corresponds to the action of finding the best data push for a specific user group.

[0191] The second scenario involves finding the right target user for a specific category of data push. Each target user can be considered as an action, and the k-means algorithm is applied to the original vector of H to generate 20 categories, and one category is selected as the specific category of data push a. Let θ a The center of the selected data push category is randomly generated, and the corresponding consumption parameter β is randomly generated. The k-means algorithm can be applied to the row vector of W to generate K = 10 target users (actions). In the above experiment, K = 20 and d = 10 can be set. In addition, M = 10 and T = 1000 can be set. At this time, the action performed corresponds to the action of selecting an appropriate recommended target user.

[0192] In terms of human activity recognition, a dataset can be generated from records of 30 data contributors. The dataset can be used to simulate crowdsourcing tasks, each activity can be regarded as an action and the label of each activity is assumed to be the expected reward. In addition, possible consumption can be generated for each data sample, the consumption parameter β can be calculated, and the class center of each action can be calculated. The center is used as a standard feature of the action to solve the LP problem and the OPT problem. In the simulated crowdsourcing scenario, each task can be regarded as capturing the corresponding human activity characteristics. Each task will generate a certain consumption and reward in terms of resources. The goal is to learn the best task. There can be six categories of tasks, each category has 561 dimensions of features, so in the game machine experiment of the present disclosure, K=6 and d=561 can be set. In addition, M=10 and T=1000 can be set.

[0193] In terms of the results of data push recommendation and targeted user recommendation, the reason why the regret curve of the Federated Bandits with Knapsacks (FBwK) problem is different from that of the traditional game machine is explained. In the traditional game machine, the second half of the regret curve of FBwK sometimes increases sharply. When the cumulative regret of each time period is calculated based on the technical solution of the present invention, if the budget is exhausted, the algorithm will stop, while OPT continues to accumulate rewards, resulting in a sharp increase in regret. This phenomenon is particularly obvious because the existing FedUCB (DisLinUCB) and FedUCB-FullCom do not take budget constraints into account. They give priority to finding the arm that obtains the highest reward, resulting in lower regret in the early stages. When the budget is exhausted in a certain time period, the algorithm will terminate early and ultimately perform poorly.

[0194] Figure 6 A block diagram 600 illustrates a comparison of the performance of an action execution process according to some implementations of the present disclosure with that of an action execution process according to various other technical solutions. Figure 6 Block diagrams 610, 620, and 630 in FIG respectively correspond to performance statistics when ∈ is set to different predetermined values. The abscissa in the block diagram corresponds to time, and the ordinate corresponds to accumulated regret. Block diagram 640 shows a comparison of accumulated regret when ∈ is set to different predetermined values.

[0195] like Figure 6 As shown, for the data push task, the algorithm of the present invention is always close to FedUCBwK-FullCom (which can be considered as the minimum regret that can be obtained in the single-agent case) and is better than other baseline methods. FedUCBwK can find data pushes with higher rewards and lower costs, thereby overcoming the problem of early termination. The results show that FedUCBwK can significantly reduce the cumulative regret. For different settings, the total budget is set larger from top to bottom. At this time, FedUCBwK and FedUCB are closer than when the budget is scarce, because the algorithm is less affected by the constraints. When the budget increases, the regret is less likely to increase sharply. In addition, when ∈ is gradually increased, it is not easy to over-consume the budget and cause the algorithm to end early.

[0196] As you'll understand, larger ∈ isn't necessarily better; rather, it's crucial to choose an appropriate ∈. For example, in a data push task, ∈ = 0.05 is a good value. When ∈ is too large, the cost estimate becomes too small, affecting the selection of the optimal arm. While the budget is preserved, overly cautious choices can actually increase the final budget.

[0197] Figure 7A block diagram 700 illustrates a comparison of the performance of an action execution process according to some implementations of the present disclosure with that of an action execution process according to a number of other technical solutions. Figure 7 Block diagrams 710, 720, and 730 in FIG respectively correspond to performance statistics when ∈ is set to different predetermined values. The abscissa in the block diagram corresponds to time, and the ordinate corresponds to accumulated regret. Block diagram 740 shows a comparison of accumulated regret when ∈ is set to different predetermined values.

[0198] For targeted user recommendation tasks, Figure 7 shows similar results. Compared with data push recommendation, Figure 7 The target user recommendation task shown is easier and converges faster. In this task, FedUCBwK performs better when ∈ = 0.1 is chosen. Specifically, FedUCBwK can reduce regret by up to 96.82% compared to FedUCB. Furthermore, to explore experimental results in more cases, the budget can be set to fully sufficient. It can be found that in this case, FedUCB, FedUCB-FullCom, and FedUCBwK all achieve results that are basically consistent with FedUCBwK-FullCom and very close to OPT. In contrast, technical solutions with less communication and computation, or no communication and computation, cannot effectively aggregate information and update strategies, resulting in poorer results.

[0199] According to an exemplary implementation of the present disclosure, the method described above can be further applied to crowdsourcing task allocation, and the experimental results are similar. Alternatively and / or additionally, the method described above can be further applied to other application scenarios such as network parameter configuration.

[0200] Using the exemplary implementation of the present disclosure, a knapsack-based federated linear game machine solution is implemented. This technical solution can be used to select the action that maximizes the reward from K actions based on the collaboration of M clients while meeting a predetermined budget, thereby minimizing the regret involved in the solution. According to an exemplary implementation of the present disclosure, only the parameter data of the model is transmitted without transmitting the original data, thereby protecting the client's sensitive data. Furthermore, a unified threshold can be used to reduce the communication and computational costs involved during federated learning, and the overall regret can be controlled by solving the distributed knapsack problem in each round.

[0201] Example Process

[0202] Figure 8A flowchart of a method 800 for performing an action at a client according to some implementations of the present disclosure is shown. At box 810, based on a first action model at a first device, a set of actions to be performed at the first device is determined from a plurality of actions. At box 820, a data accumulation indicator associated with the set of actions is obtained, the data accumulation indicator indicating an amount of data to be sent from the first device to a second device associated with the first device. At box 830, if the data accumulation indicator satisfies a predetermined condition, the operation flow proceeds to box 840. At box 840, parameter data associated with the set of actions is transmitted to the second device so that the second device updates the second action model at the second device using the parameter data, the parameter data including reward data and consumption data respectively associated with the set of actions.

[0203] According to an exemplary implementation of the present disclosure, determining the data accumulation index includes: determining the data accumulation index based on a feature of each action in a group of actions.

[0204] According to an exemplary implementation of the present disclosure, the method further includes: determining parameter data associated with a set of actions based on a set of actions, and a set of rewards and a set of consumptions respectively associated with the set of actions, wherein the rewards in the set of rewards represent the benefits generated by performing the actions in the set of actions, and the consumption in the set of consumptions represents the consumption of resources allocated to the first device generated by performing the actions.

[0205] According to an exemplary implementation of the present disclosure, determining parameter data includes: determining reward data based on a linear calculation of a set of actions and a set of rewards; determining consumption data based on a linear calculation of a set of actions and a set of consumption; and determining number data in the parameter data based on the number of executions associated with a set of actions.

[0206] According to an exemplary implementation of the present disclosure, the method further includes: determining a target action from multiple actions based on a first action model; updating a data accumulation indicator based on characteristics of the target action; and updating parameter data using the target action, and the target reward and target consumption associated with the target action.

[0207] According to an exemplary implementation of the present disclosure, the method further includes: receiving aggregated parameter data for updating the first action model from the second device, the aggregated parameter data being determined by the second device based on the parameter data; and updating the first action model using the aggregated parameter data.

[0208] According to an exemplary implementation of the present disclosure, the aggregated parameter data includes aggregated reward data, aggregated consumption data, and aggregated accumulation data.

[0209] According to an exemplary implementation of the present disclosure, the method further includes: determining a target action to be performed at the first device from a plurality of actions using the updated first action model.

[0210] According to an exemplary implementation of the present disclosure, the method further includes: performing multiple actions at a first device before determining a set of actions; respectively obtaining multiple rewards and multiple consumptions associated with the multiple actions; determining initial parameter data associated with the multiple actions based on the multiple actions, the multiple rewards, and the multiple consumptions; and transmitting the initial parameter data to a second device so that the second device uses the initial parameter data to update a second action model.

[0211] According to an exemplary implementation of the present disclosure, the method further includes: receiving aggregated initial parameter data for updating the first action model from the second device, the aggregated initial parameter data being determined by the second device based on the initial parameter data; and updating the first action model using the aggregated initial parameter data.

[0212] According to an exemplary implementation of the present disclosure, the method further includes: terminating the method in response to at least any one of the following: the time length for executing the method reaches a threshold time length; the consumption associated with at least one action that has been performed at the first device reaches a threshold consumption.

[0213] According to an exemplary implementation of the present disclosure, the first device is a client device for performing federated learning, and the second device is a server device for performing federated learning.

[0214] According to an exemplary implementation of the present disclosure, the multiple actions include at least any one of the following: a data push action, a user selection action, a crowdsourcing task allocation action, and a network parameter setting action.

[0215] Figure 9 A flowchart of a method 900 for performing an action at a server according to some implementations of the present disclosure is shown. At box 910, at a second device associated with a plurality of first devices, a plurality of parameter data from the plurality of first devices is received, respectively, wherein the parameter data from a first device among the plurality of first devices in the plurality of parameter data is transmitted from the first device to the second device in response to a data accumulation indicator associated with the first device satisfying a predetermined condition, the data accumulation indicator indicating the amount of data to be transmitted from the first device to the second device, the parameter data including reward data and consumption data respectively associated with a set of actions performed at the first device. At box 920, aggregated parameter data is determined based on the plurality of parameter data. At box 930, the aggregated parameter data is transmitted to the plurality of first devices, respectively, so that the plurality of first devices update the plurality of first action models located at the plurality of first devices based on the aggregated parameter data.

[0216] According to an exemplary implementation of the present disclosure, multiple parameter data include: reward data of the first device, the reward data representing the benefits generated by a set of actions performed at the first device; consumption data of the first device, the consumption data representing the consumption of resources allocated to the first device generated by a set of actions performed at the first device; and number data of the first device, the number data representing the number of times an action in a set of actions is performed.

[0217] According to an exemplary implementation of the present disclosure, determining the aggregated parameter data includes: determining the aggregated cumulative data in the aggregated parameter data based on the number data and the reward data; determining the aggregated reward data in the aggregated parameter data based on the number data and the reward data; and determining the aggregated consumption data in the aggregated parameter data based on the number data and the consumption data.

[0218] According to an exemplary implementation of the present disclosure, the method further includes: updating a second motion model at the second device based on the plurality of parameter data.

[0219] According to an exemplary implementation of the present disclosure, the method further includes: determining an action to be performed using the updated second action model.

[0220] According to an exemplary implementation of the present disclosure, the method further includes: before respectively receiving multiple parameter data from multiple first devices, receiving multiple initial parameter data respectively from the multiple first devices at the second device, the initial parameter data from the first device in the multiple initial parameter data being determined based on multiple actions performed at the first device and multiple rewards and multiple consumptions respectively associated with the multiple actions; and determining aggregated initial parameter data based on the multiple initial parameter data; and transmitting the aggregated initial parameter data to the multiple first devices respectively, so that the multiple first devices update the multiple first action models at the multiple first devices based on the aggregated initial parameter data.

[0221] According to an exemplary implementation of the present disclosure, the method further includes: updating a second motion model at the second device based on a plurality of initial parameter data.

[0222] According to an exemplary implementation of the present disclosure, the first device is a client device for performing federated learning, and the second device is a server device for performing federated learning.

[0223] According to an exemplary implementation of the present disclosure, the multiple actions include at least any one of the following: a data push action, a user selection action, a crowdsourcing task allocation action, and a network parameter setting action.

[0224] Example devices and equipment

[0225] Figure 10 A block diagram of an apparatus 1000 for performing an action according to some implementations of the present disclosure is shown. The apparatus includes: a determination module 1010 configured to determine, from a plurality of actions, a set of actions to be performed at a first device based on a first action model at the first device; an acquisition module 1020 configured to acquire a data accumulation indicator associated with the set of actions, the data accumulation indicator indicating an amount of data to be sent from the first device to a second device associated with the first device; and a transmission module 1030 configured to transmit parameter data associated with the set of actions to the second device in response to the data accumulation indicator satisfying a predetermined condition, so that the second device updates a second action model at the second device using the parameter data, the parameter data including reward data and consumption data respectively associated with the set of actions.

[0226] According to an exemplary implementation of the present disclosure, determining the data accumulation index includes: determining the data accumulation index based on a feature of each action in a group of actions.

[0227] According to an exemplary implementation of the present disclosure, the device further includes: a parameter determination module, configured to determine parameter data associated with a set of actions based on a set of actions, and a set of rewards and a set of consumptions respectively associated with the set of actions, wherein the rewards in the set of rewards represent the benefits generated by performing the actions in the set of actions, and the consumption in the set of consumptions represents the consumption of resources allocated to the first device generated by performing the actions.

[0228] According to an exemplary implementation of the present disclosure, the parameter determination module includes: a first determination module, configured to determine reward data based on a linear calculation of a set of actions and a set of rewards; a second determination module, configured to determine consumption data based on a linear calculation of a set of actions and a set of consumption; and a third module, configured to determine the number data in the parameter data based on the number of executions associated with a set of actions.

[0229] According to an exemplary implementation of the present disclosure, the device further includes: a target action determination module, configured to determine a target action from multiple actions based on a first action model; an indicator update module, configured to update a data accumulation indicator based on characteristics of the target action; and a parameter update module, configured to update parameter data using the target action, and the target reward and target consumption associated with the target action.

[0230] According to an exemplary implementation of the present disclosure, the apparatus further includes: a receiving module configured to receive aggregated parameter data for updating the first action model from the second device, the aggregated parameter data being determined by the second device based on the parameter data; and a model updating module configured to update the first action model using the aggregated parameter data.

[0231] According to an exemplary implementation of the present disclosure, the aggregated parameter data includes aggregated reward data, aggregated consumption data, and aggregated accumulation data.

[0232] According to an exemplary implementation of the present disclosure, the apparatus further includes: a target action determination module configured to determine a target action to be performed at the first device from a plurality of actions using the updated first action model.

[0233] According to an exemplary implementation of the present disclosure, the apparatus further includes: an execution module configured to execute multiple actions at a first device before determining a set of actions; a data acquisition module configured to respectively acquire multiple rewards and multiple consumptions associated with the multiple actions; an initial parameter determination module configured to determine initial parameter data associated with the multiple actions based on the multiple actions, the multiple rewards, and the multiple consumptions; and an initial parameter transmission module configured to transmit the initial parameter data to the second device so that the second device uses the initial parameter data to update the second action model.

[0234] According to an exemplary implementation of the present disclosure, the apparatus further includes: a receiving module configured to receive aggregated initial parameter data for updating the first action model from the second device, the aggregated initial parameter data being determined by the second device based on the initial parameter data; and a model updating module configured to update the first action model using the aggregated initial parameter data.

[0235] According to an exemplary implementation of the present disclosure, the apparatus further includes: a termination module configured to terminate the method in response to at least any one of the following: the time length for executing the method reaches a threshold time length; the consumption associated with at least one action already performed at the first device reaches a threshold consumption.

[0236] According to an exemplary implementation of the present disclosure, the first device is a client device for performing federated learning, and the second device is a server device for performing federated learning.

[0237] According to an exemplary implementation of the present disclosure, the multiple actions include at least any one of the following: a data push action, a user selection action, a crowdsourcing task allocation action, and a network parameter setting action.

[0238] Figure 11 A block diagram of an apparatus 1100 for performing an action according to some implementations of the present disclosure is shown. The apparatus 1100 includes: a receiving module 1110 configured to receive, at a second device associated with the plurality of first devices, a plurality of parameter data from the plurality of first devices, respectively, wherein the parameter data from a first device among the plurality of first devices in the plurality of parameter data is transmitted from the first device to the second device in response to a data accumulation indicator associated with the first device satisfying a predetermined condition, the data accumulation indicator indicating an amount of data to be transmitted from the first device to the second device, the parameter data including reward data and consumption data respectively associated with a set of actions performed at the first device; a determining module 1120 configured to determine aggregated parameter data based on the plurality of parameter data; and a transmitting module 1130 configured to transmit the aggregated parameter data to the plurality of first devices, respectively, so that the plurality of first devices update the plurality of first action models located at the plurality of first devices based on the aggregated parameter data.

[0239] According to an exemplary implementation of the present disclosure, multiple parameter data include: reward data of the first device, the reward data representing the benefits generated by a set of actions performed at the first device; consumption data of the first device, the consumption data representing the consumption of resources allocated to the first device generated by a set of actions performed at the first device; and number data of the first device, the number data representing the number of times an action in a set of actions is performed.

[0240] According to an exemplary implementation of the present disclosure, the determination module includes: a first determination module, configured to determine the aggregated cumulative data in the aggregated parameter data based on the number data and the reward data; a second determination module, configured to determine the aggregated reward data in the aggregated parameter data based on the number data and the reward data; and a third determination module, configured to determine the aggregated consumption data in the aggregated parameter data based on the number data and the consumption data.

[0241] According to an exemplary implementation of the present disclosure, the apparatus further includes: updating a second motion model at the second device based on a plurality of parameter data.

[0242] According to an exemplary implementation of the present disclosure, the apparatus further includes: determining an action to be performed using the updated second action model.

[0243] According to an exemplary implementation of the present disclosure, the apparatus further includes: an initial parameter receiving module, configured to receive, at a second device, a plurality of initial parameter data respectively from a plurality of first devices before respectively receiving a plurality of parameter data from the plurality of first devices, the initial parameter data from the first device in the plurality of initial parameter data being determined based on a plurality of actions performed at the first device and a plurality of rewards and a plurality of consumptions respectively associated with the plurality of actions; and an initial aggregate parameter determination module, configured to determine aggregated initial parameter data based on the plurality of initial parameter data; and an initial aggregate parameter transmission module, configured to transmit the aggregated initial parameter data to the plurality of first devices respectively, so that the plurality of first devices respectively update the plurality of first action models at the plurality of first devices based on the aggregated initial parameter data.

[0244] According to an exemplary implementation of the present disclosure, the apparatus further includes: updating a second motion model at the second device based on a plurality of initial parameter data.

[0245] According to an exemplary implementation of the present disclosure, the first device of the apparatus is a client device for performing federated learning, and the second device is a server device for performing federated learning.

[0246] According to an exemplary implementation of the present disclosure, the multiple actions include at least any one of the following: a data push action, a user selection action, a crowdsourcing task allocation action, and a network parameter setting action.

[0247] Figure 12 FIG1 shows a block diagram of a device 1200 capable of implementing multiple implementations of the present disclosure. It should be understood that Figure 12 The illustrated computing device 1200 is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. Figure 12 The illustrated computing device 1200 may be used to implement the methods described above.

[0248] like Figure 12 As shown, computing device 1200 is in the form of a general-purpose computing device. Components of computing device 1200 may include, but are not limited to, one or more processors or processing units 1210, memory 1220, storage device 1230, one or more communication units 1240, one or more input devices 1250, and one or more output devices 1260. Processing unit 1210 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 1220. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 1200.

[0249] The computing device 1200 typically includes a plurality of computer storage media. Such media can be any available media accessible to the computing device 1200, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 1220 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 1230 can be a removable or non-removable medium and can include a machine-readable medium such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the computing device 1200.

[0250] The computing device 1200 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 12 As shown in FIG, a magnetic disk drive for reading from or writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. Memory 1220 may include a computer program product 1225 having one or more program modules configured to perform various methods or actions of various implementations of the present disclosure.

[0251] The communication unit 1240 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 1200 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 1200 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.

[0252] Input device 1250 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 1260 may be one or more output devices, such as a display, a speaker, or a printer. Computing device 1200 may also communicate with one or more external devices (not shown) via communication unit 1240, as needed, such as storage devices, display devices, or the like, with one or more devices that allow a user to interact with computing device 1200, or with any device that allows computing device 1200 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0253] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.

[0254] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0255] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0256] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0257] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0258] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for performing an action, comprising: determining, based on a first action model at a first device, a set of actions to be performed at the first device from a plurality of actions; obtaining a data accumulation indicator associated with the set of actions, the data accumulation indicator indicating an amount of data to be sent from the first device to a second device associated with the first device; In response to the data accumulation indicator satisfying a predetermined condition, determining parameter data associated with the set of actions based on the set of actions, and a set of rewards and a set of consumptions respectively associated with the set of actions, wherein a reward in the set of rewards represents a benefit generated by performing an action in the set of actions, and a consumption in the set of consumptions represents a consumption of resources allocated to the first device generated by performing the action; as well as The parameter data including reward data and expenditure data respectively associated with the set of actions is transmitted to the second device such that the second device updates a second action model at the second device using the parameter data.

2. The method according to claim 1, wherein determining the data accumulation index comprises: The data accumulation index is determined based on characteristics of each action in the set of actions.

3. The method of claim 1 , wherein determining the parameter data comprises: determining the reward data based on a linear calculation of the set of actions and the set of rewards; determining the consumption data based on a linear calculation of the set of actions and the set of consumptions; as well as The number of times data in the parameter data is determined based on the number of times of execution associated with the set of actions.

4. The method according to claim 1, further comprising: determining a target action from the plurality of actions based on the first action model; updating the data accumulation indicator based on the characteristics of the target action; as well as The parameter data is updated using the target action, and the target reward and target cost associated with the target action.

5. The method according to claim 1, further comprising: receiving aggregated parameter data for updating the first motion model from the second device, the aggregated parameter data being determined by the second device based on the parameter data; as well as The first motion model is updated using the aggregated parameter data. The method according to claim 5 , wherein the aggregated parameter data comprises aggregated reward data, aggregated consumption data and aggregated accumulation data.

7. The method according to claim 5, further comprising: Using the updated first action model, a target action to be performed at the first device is determined from the plurality of actions.

8. The method according to claim 1, further comprising: Before determining the set of actions, performing the plurality of actions at the first device; Respectively obtaining a plurality of rewards and a plurality of consumptions associated with the plurality of actions; determining initial parameter data associated with the plurality of actions based on the plurality of actions, the plurality of rewards, and the plurality of expenditures; as well as The initial parameter data is transmitted to the second device, so that the second device updates the second motion model using the initial parameter data.

9. The method according to claim 8, further comprising: receiving aggregated initial parameter data for updating the first motion model from the second device, the aggregated initial parameter data being determined by the second device based on the initial parameter data; as well as The first motion model is updated using the aggregated initial parameter data.

10. The method according to claim 1, further comprising: The method is terminated in response to at least any of the following: The length of time for executing the method reaches a threshold length of time; A consumption associated with at least one action having been performed at the first device reaches a threshold consumption. 11 . The method of claim 1 , wherein the first device is a client device for performing federated learning, and the second device is a server device for performing the federated learning.

12. The method according to claim 1, wherein the multiple actions include at least any one of the following: a data push action, a user selection action, a crowdsourcing task allocation action, and a network parameter setting action.

13. A device for performing an action, comprising: a determining module configured to determine, based on a first action model at the first device, a set of actions to be performed at the first device from a plurality of actions; an acquisition module configured to acquire a data accumulation indicator associated with the set of actions, the data accumulation indicator indicating an amount of data to be sent from the first device to a second device associated with the first device; a parameter determination module configured to, in response to the data accumulation indicator satisfying a predetermined condition, determine parameter data associated with the set of actions based on the set of actions, and a set of rewards and a set of consumptions respectively associated with the set of actions, wherein a reward in the set of rewards represents a benefit generated by performing an action in the set of actions, and a consumption in the set of consumptions represents a consumption of resources allocated to the first device generated by performing the action; as well as A sending module is configured to transmit the parameter data to the second device so that the second device uses the parameter data to update the second action model at the second device, wherein the parameter data includes reward data and consumption data respectively associated with the set of actions.

14. An electronic device comprising: at least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 12 when executed by the at least one processing unit. 15 . A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to claim 1 .

16. A method for performing an action, comprising: receiving, at a second device associated with a plurality of first devices, a plurality of parameter data respectively from the plurality of first devices, wherein the parameter data from a first device among the plurality of first devices in the plurality of parameter data is transmitted from the first device to the second device in response to a data accumulation indicator associated with the first device satisfying a predetermined condition, the data accumulation indicator indicating an amount of data to be transmitted from the first device to the second device, the parameter data comprising reward data and consumption data respectively associated with a set of actions performed at the first device, the parameter data being determined based on the set of actions and a set of rewards and a set of consumptions respectively associated with the set of actions, the rewards in the set of rewards representing benefits generated by performing an action in the set of actions, and the consumption in the set of consumptions representing consumption of resources allocated to the first device generated by performing the action; determining aggregated parameter data based on the plurality of parameter data; as well as The aggregated parameter data are transmitted to the plurality of first devices respectively, so that the plurality of first devices update a plurality of first motion models located at the plurality of first devices respectively based on the aggregated parameter data.

17. The method according to claim 16, wherein the plurality of parameter data comprises: reward data for the first device, the reward data representing a benefit generated by the set of actions performed at the first device; consumption data of the first device, the consumption data representing consumption of resources allocated to the first device caused by the set of actions performed at the first device; as well as The number data of the first device indicates the number of times an action in the set of actions is performed.

18. The method of claim 17, wherein determining the aggregated parameter data comprises: determining aggregated cumulative data in the aggregated parameter data based on the number of times data and the reward data; determining aggregated reward data in the aggregated parameter data based on the number data and the reward data; as well as Based on the number data and the consumption data, aggregated consumption data in the aggregated parameter data is determined.

19. The method of claim 16, further comprising: A second motion model at the second device is updated based on the plurality of parameter data.

20. The method of claim 19, further comprising: The action to be performed is determined using the updated second action model.

21. The method of claim 16, further comprising: Before respectively receiving a plurality of parameter data from the plurality of first devices, receiving, at the second device, a plurality of initial parameter data respectively from the plurality of first devices, the initial parameter data from the first device among the plurality of initial parameter data being determined based on a plurality of actions performed at the first device and a plurality of rewards and a plurality of consumptions respectively associated with the plurality of actions; as well as determining aggregated initial parameter data based on the plurality of initial parameter data; as well as The aggregated initial parameter data are transmitted to the plurality of first devices respectively, so that the plurality of first devices update a plurality of first motion models at the plurality of first devices respectively based on the aggregated initial parameter data.

22. The method according to claim 21, further comprising: A second motion model at the second device is updated based on the plurality of initial parameter data.

23. The method of claim 16, wherein the first device is a client device for performing federated learning, and the second device is a server device for performing the federated learning.

24. The method according to claim 16, wherein the multiple actions include at least any one of the following: a data push action, a user selection action, a crowdsourcing task allocation action, and a network parameter setting action.

25. A device for performing an action, comprising: a receiving module configured to receive, at a second device associated with a plurality of first devices, a plurality of parameter data respectively from the plurality of first devices, wherein the parameter data from a first device among the plurality of first devices in the plurality of parameter data is transmitted from the first device to the second device in response to a data accumulation indicator associated with the first device satisfying a predetermined condition, the data accumulation indicator indicating an amount of data to be transmitted from the first device to the second device, the parameter data comprising reward data and consumption data respectively associated with a set of actions performed at the first device, the parameter data being determined based on the set of actions and a set of rewards and a set of consumptions respectively associated with the set of actions, the rewards in the set of rewards representing benefits generated by performing actions in the set of actions, and the consumption in the set of consumptions representing consumption of resources allocated to the first device generated by performing the actions; a determination module configured to determine aggregated parameter data based on the plurality of parameter data; as well as The transmission module is configured to transmit the aggregated parameter data to the multiple first devices respectively, so that the multiple first devices respectively update the multiple first action models located at the multiple first devices based on the aggregated parameter data.

26. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 16 to 24 when executed by the at least one processing unit.

27. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to any one of claims 16 to 24.

Images