A distributed online optimization method based on differential privacy mechanism

Abstract

The application discloses a kind of distributed online optimization methods based on differential privacy mechanism, it is related to multi-agent collaborative control technical field, builds distributed online optimization problem mathematical model;According to the difference between the cost generated by the current decision of all nodes and the cost generated by the after-optimal fixed decision, the performance evaluation index of online optimization method is constructed;Differential privacy mechanism and bandit feedback mechanism are introduced to design distributed online optimization algorithm, and the distributed optimization model is iteratively solved according to the designed distributed online algorithm. Using bandit feedback mechanism to solve a distributed online problem with time-varying cost function, the real gradient information is estimated, so as to guide the update of decision variable, so that it is applicable to the application of expensive or impossible gradient calculation;Differential privacy mechanism makes the information exchanged between nodes be perturbed by random noise conforming to Laplace distribution, effectively solving the privacy problem in distributed online optimization.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent collaborative control, and in particular to a distributed online optimization method based on a differential privacy mechanism. Background Art

[0002] Distributed optimization problems have attracted widespread attention in recent years due to their applications in machine learning, smart grids, and robust estimation. Distributed optimization methods are needed to address these problems. Unlike traditional centralized optimization methods, distributed optimization methods involve multiple nodes, each of which can find the optimal solution to the global objective by accessing only local information and interacting with its neighbors, without relying on any central node.

[0003] Distributed optimization methods have been widely used for static distributed optimization problems. A. Nedic and A. Ozdaglar proposed a distributed optimization algorithm based on gradient descent in their paper "Distributed subgradient methods for multi-agent optimization" published in IEEE Transactions on Automatic Control (54 (2009), no. 1, 48–61). This method requires that the cost function of each node in the network is time-invariant. However, in practical applications, distributed optimization often occurs in uncertain environments. For example, in the problem of tracking a moving target in a sensor network, the cost function optimized by each sensor is often time-varying. Such problems can be considered distributed online optimization problems, and static regret is often used to measure the performance of online methods. In their paper "An Accelerated Distributed Online Optimization Method Based on Conditional Gradients," Shen Xiuyu et al. proposed an accelerated distributed online conditional gradient optimization method based on conditional gradients, which effectively addresses the high time complexity of distributed online optimization algorithms. However, like most distributed online optimization methods, this method requires both gradient and subgradient information of the cost function. However, this information can be difficult to obtain in some situations, such as adversarial training, reinforcement learning, and human-in-the-loop control. Bandit feedback can effectively solve this problem by estimating the true gradient information.

[0004] In addition, most distributed online optimization methods require nodes to exchange their state information with their neighboring nodes during the update and iteration process. This distributed information sharing process may lead to the leakage of node privacy information, which may even be exploited by adversaries to cause losses to the system. Therefore, the issue of privacy protection in distributed optimization also needs to be paid attention to. For example, Wang Wei, Li Dequan, et al. proposed a distributed online optimization algorithm based on homomorphic encryption in "Distributed Online Optimization Algorithm with Privacy Preservation". However, this method has the disadvantages of high communication cost and high computational complexity, making it unsuitable for large-scale networks. Therefore, a distributed online optimization method based on differential privacy mechanism is proposed. Summary of the Invention

[0005] The purpose of this application is to provide a distributed online optimization method based on a differential privacy mechanism to solve the problems raised in the above background technology.

[0006] To achieve the above objectives, this application provides the following technical solution: a distributed online optimization method based on a differential privacy mechanism, comprising the following steps:

[0007] S1. Use multi-agent systems to build a mathematical model for distributed online optimization problems;

[0008] S2. Constructing a performance evaluation index for the online optimization method based on the difference between the cost generated by the current decision of all nodes and the cost generated by the optimal fixed decision after the fact;

[0009] S3. Introduce a differential privacy mechanism and a bandit feedback mechanism to design a distributed online optimization algorithm, and iteratively solve the distributed optimization model according to the designed distributed online algorithm.

[0010] Preferably, in S1, the multi-agent system includes multiple agents, and a communication topology graph consisting of nodes and edges is established based on the communication relationship between each agent. , each of the intelligent agents is regarded as a node in the communication topology graph, where , Respectively represent the graph The node set and edge set of , build a mathematical model of distributed online optimization problem, the expression of the mathematical model of distributed online optimization problem is:

[0011]

[0012] in, For nodes At the moment Status information, For nodes At the moment The cost function, is the number of nodes in the multi-agent network, is the total iteration time, The domain formed by the node state.

[0013] Preferably, in S2, the performance evaluation index of the online optimization method includes individual regret , used to reflect the node The difference between the total cost of the decision made and the total cost of the optimal decision. Formally, individual regret is defined as:

[0014]

[0015] in, is the optimal node state after the iteration. When the individual regret of an online optimization method is If the function is sublinear, the algorithm can effectively solve the distributed online optimization problem.

[0016] Preferably, in S3, the state of each node is perturbed based on the differential privacy mechanism, and the state of each node is perturbed based on different privacy levels. Generate the corresponding random noise and use it to perturb the state of the node. The expression is:

[0017]

[0018] in, is a random noise that conforms to the Laplace distribution, The state of the node after perturbation, the state after perturbation is used between nodes to communicate.

[0019] Preferably, in S3, a gradient estimator is constructed according to the bandit feedback mechanism, which is defined as follows:

[0020]

[0021] in, Represents dimension, is the exploration parameter, is a unit random variable with mean zero.

[0022] Preferably, in S3, the iterative formula of the distributed online optimization algorithm designed based on the differential privacy mechanism and the bandit feedback mechanism is as follows:

[0023]

[0024] in, is the updated node status; express In the collection projection on; It is a small positive number and plays a key role in the convergence of the algorithm; Represents the step length, satisfying ; For nodes At the moment The auxiliary variable of , its iteration rule is as follows:

[0025]

[0026] and is the row random adjacency matrix and column random adjacency matrix The elements in , the construction rules are as follows:

[0027]

[0028]

[0029] in, is a node The number of inner neighbors, is a node The number of outer neighbors increases with the number of iterations. Finally, accurate optimization results can be obtained.

[0030] In summary, the technical effects and advantages of the present invention are:

[0031] 1. The present invention uses a bandit feedback mechanism to solve a distributed online problem with a time-varying cost function. The bandit feedback mechanism is used to estimate the true gradient information, thereby guiding the update of decision variables, making it suitable for application of expensive or impossible gradient calculations.

[0032] 2. This invention introduces a differential privacy mechanism, so that the information exchanged between nodes is perturbed by random noise that conforms to the Laplace distribution, effectively solving the privacy problem in distributed online optimization.

[0033] 3. The present invention establishes a small error term and a duration At a fixed privacy level, the algorithm achieves sublinear regret for strongly connected directed graph sequences. Therefore, in this algorithm, the performance is not seriously affected while maintaining differential privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 This is a flowchart of the distributed online optimization method based on the differential privacy mechanism in this embodiment;

[0036] Figure 2 A communication topology diagram of a multi-agent system in an embodiment;

[0037] Figure 3 This is a graph showing the impact of different privacy levels on individual regret in the embodiment;

[0038] Figure 4 A state evolution curve diagram of each intelligent agent in the embodiment;

[0039] Figure 5 This is a graph showing the impact of different dimensions on individual regret in the embodiment;

[0040] Figure 6 The figure is a performance comparison graph of the proposed method in the embodiment and the other two methods. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] Example: Reference Figure 1 A distributed online optimization method based on a differential privacy mechanism is shown, comprising the following steps:

[0043] S1. Use multi-agent systems to build a mathematical model for distributed online optimization problems;

[0044] Among them, the multi-agent system includes multiple agents, and a communication topology consisting of nodes and edges is established based on the communication relationship between each agent. , where all nodes can only collaboratively solve the optimization problem by communicating with their neighbors, and each agent is regarded as a node in the communication topology graph, where , Respectively represent the graph The node set and edge set of , build a mathematical model of distributed online optimization problem, the expression of the mathematical model of distributed online optimization problem is:

[0045]

[0046] in, For nodes At the moment Status information, For nodes At the moment The cost function, is the number of nodes in the multi-agent network, is the total iteration time, The domain formed by the node state.

[0047] For example, a multi-agent system consists of 10 agents. According to the communication relationship between each agent, a communication topology consisting of nodes and edges is established. ,like Figure 2 As shown in Figure 2, each agent in a multi-agent network is considered as a node in the communication topology graph. , Respectively represent the graph The node set and edge set of . Build a mathematical model for the distributed online optimization problem. The mathematical model expression of this problem is:

[0048]

[0049] in, For nodes At the moment Status information, For nodes At the moment The cost function, is the number of nodes in the multi-agent network, is the total iteration time, The domain formed by the node state.

[0050] S2. Constructing a performance evaluation index for the online optimization method based on the difference between the cost generated by the current decision of all nodes and the cost generated by the optimal fixed decision after the fact;

[0051] Among them, the performance evaluation indicators of online optimization methods include individual regret , used to reflect the node The difference between the total cost of the decision made and the total cost of the optimal decision. Formally, individual regret is defined as:

[0052]

[0053] in, is the optimal node state after the iteration, when the individual regret of an online optimization method is If the function is sublinear, the algorithm can effectively solve the distributed online optimization problem.

[0054] It can be seen that the individual regret consists of a small error term and a duration At a fixed privacy level, the algorithm achieves sublinear regret for strongly connected directed graph sequences. Therefore, in this algorithm, the performance is not seriously affected while maintaining differential privacy.

[0055] S3. Introduce differential privacy mechanism and bandit feedback mechanism to design a distributed online optimization algorithm, and iteratively solve the distributed optimization model according to the designed distributed online algorithm.

[0056] Among them, the state of each node is perturbed based on the differential privacy mechanism, and different privacy levels are used. Generate the corresponding random noise and use it to perturb the state of the node. The expression is:

[0057]

[0058] in, is a random noise that conforms to the Laplace distribution, The state of the node after perturbation, the state after perturbation is used between nodes to communicate.

[0059] By introducing the differential privacy mechanism, the information interacting between nodes is perturbed by random noise that conforms to the Laplace distribution, effectively solving the privacy problem in distributed online optimization.

[0060] The gradient estimator is constructed according to the bandit feedback mechanism and is defined as follows:

[0061]

[0062] in, Represents the dimension, is the exploration parameter, is a unit random variable with mean zero.

[0063] The iterative formula of the distributed online optimization algorithm designed based on the differential privacy mechanism and the bandit feedback mechanism is as follows:

[0064]

[0065] in, is the updated node status; express In the collection projection on; It is a small positive number and plays a key role in the convergence of the algorithm; Represents the step length, satisfying ; For nodes At the moment The auxiliary variable of , its iteration rule is as follows:

[0066]

[0067] and is the row random adjacency matrix and column random adjacency matrix The elements in , the construction rules are as follows:

[0068]

[0069]

[0070] in, is a node The number of inner neighbors, is a node The number of outer neighbors increases with the number of iterations. Finally, accurate optimization results can be obtained.

[0071] A bandit feedback mechanism is used to solve a distributed online problem with a time-varying cost function. The bandit feedback mechanism is used to estimate the true gradient information to guide the update of decision variables, making it suitable for applying expensive or impossible gradient calculations.

[0072] In the Simulink environment, a multi-agent system consisting of 10 agents is simulated, where each agent has the ability to perceive and communicate.

[0073] Under the above given conditions, the method proposed by the present invention is simulated and verified and the simulation results are analyzed. The simulation results are shown in the attached Figure 3 -Attached Figure 6 As shown:

[0074] Attachment Figure 3 Reflecting the relationship between individual regret and privacy level, the results show that this method can achieve sublinear regret and there is a balance between the degree of privacy protection and optimization accuracy;

[0075] Attachment Figure 4 For each agent at a fixed privacy level The state evolution trajectory of the 10 agents can be seen. After 20 iterations, the 10 agents reach consensus, which shows that the method can solve the distributed optimization problem.

[0076] attached Figure 5 To study the dimension The influence of the algorithm convergence result, the result shows The smaller the regret of the individual, the better the convergence result of the individual;

[0077] attached Figure 6 Compared with the random gradient-free distributed projection gradient descent (RGF-DPGD) method and the directed distributed projection subgradient (D-DPS) method, the convergence result of the method is slightly worse than that of the other two methods. However, the method does not need to know the gradient information of the cost function, and only needs the function value of one point for gradient estimation in each iteration, instead of the function values of two points.

[0078] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for limiting the present application, although the foregoing embodiments of the present application are described in detail, for those skilled in the art, it still can be modified, or part of the technical features of the equivalent replacement, within the spirit and principles of the present application, any modification, equivalent replacement, improvement, etc., should be included in the protection scope of the present application.

Claims

1. A distributed online optimization method based on differential privacy mechanism, characterized by: The following steps are involved: S1. Use multi-agent systems to build a mathematical model for distributed online optimization problems; S2. Constructing a performance evaluation index for the online optimization method based on the difference between the cost generated by the current decision of all nodes and the cost generated by the optimal fixed decision after the fact; S3. Introduce differential privacy mechanism and bandit feedback mechanism to design a distributed online optimization algorithm, and iteratively solve the distributed optimization model based on the designed distributed online algorithm. Specifically: The state of each node is perturbed based on the differential privacy mechanism, according to different privacy levels Generate the corresponding random noise and use it to perturb the state of the node. The expression is: in, is a random noise that conforms to the Laplace distribution, The state of the node after perturbation, the state after perturbation is used between nodes to communicate; The gradient estimator is constructed according to the bandit feedback mechanism and is defined as follows: in, Represents dimension, is the exploration parameter, is a unit random variable with mean zero; The iterative formula of the distributed online optimization algorithm designed based on the differential privacy mechanism and the bandit feedback mechanism is as follows: in, is the updated node status; express In the collection projection on; is a positive number; Represents the step length, satisfying ; For nodes At the moment The auxiliary variable of , its iteration rules are as follows: and is the row random adjacency matrix and column random adjacency matrix The elements in , the construction rules are as follows: in, is a node The number of inner neighbors, is a node The number of outer neighbors increases with the number of iterations. Finally, accurate optimization results can be obtained.

2. The distributed online optimization method based on the differential privacy mechanism according to claim 1, characterized in that: In S1, the multi-agent system includes multiple agents, and a communication topology consisting of nodes and edges is established based on the communication relationship between each agent. , each of the said agents is regarded as a node in the communication topology graph, where , Respectively represent the graph The node set and edge set of , build a mathematical model of distributed online optimization problem, the expression of the mathematical model of distributed online optimization problem is: in, For nodes At the moment Status information, For nodes At the moment The cost function, is the number of nodes in the multi-agent network, is the total iteration time, The domain formed by the node state.

3. The distributed online optimization method based on differential privacy mechanism according to claim 2, characterized in that: In S2, the performance evaluation index of the online optimization method includes individual regret , used to reflect the node The difference between the total cost of the decision made and the total cost of the optimal decision. Formally, individual regret is defined as: in, is the optimal node state after the iteration. When the individual regret of an online optimization method is If the function is sublinear, the algorithm can effectively solve the distributed online optimization problem.

Images