Instruction optimization method and system based on privacy protection, terminal and storage medium

Abstract

The invention relates to the technical field of data instruction optimization, and discloses an instruction optimization method and system based on privacy protection, a terminal and a storage medium, and the method comprises the steps: obtaining information of a plurality of devices through an environment perception layer, taking each device as a node, taking other devices as neighbor nodes, and calculating truncation Q values of all neighbor nodes, according to the method, the global gradient in the strategy network corresponding to each device is replaced, so that the state and action of the device are optimized, the privacy of device information is ensured by dual network isolation during optimization, and finally closed-loop optimization is performed on the strategy parameters at the current moment. According to the method, the measured data and the environment network are transmitted in an isolated manner through dual network isolation, the data security is improved, the global gradient is replaced by the truncated Q value, the complexity and the cost of calculating the global gradient are reduced, the privacy of the data is protected, and the data accuracy is improved.

Description

Privacy-preserving instruction optimization methods, systems, terminals, and storage media Technical Field

[0001] This invention relates to the field of data instruction optimization technology, and in particular to an instruction optimization method, system, terminal, and computer-readable storage medium based on privacy protection. Background Technology

[0002] The core of collaborative control technology lies in enabling various intelligent agents to autonomously collaborate and jointly complete tasks through distributed decision-making and real-time communication.

[0003] In existing collaborative control systems, reinforcement learning processes often require devices to upload data from the entire scenario. This leads to data redundancy and reduces learning efficiency. Furthermore, the use of traditional protocols to transmit plaintext policy data during data transmission makes the data vulnerable to man-in-the-middle attacks, significantly reducing data security.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a privacy-preserving instruction optimization method, system, terminal, and computer-readable storage medium, aiming to solve the problems of data redundancy caused by large data transmission volume in collaborative control in the prior art, as well as insufficient security in the data transmission process.

[0006] To achieve the above objectives, this invention provides a privacy-preserving instruction optimization method, which includes the following steps: acquiring device information for each device in a topological device network; constructing a local observation vector for each device based on the neighbor relationships between each device and the corresponding device information; inputting each local observation vector into a policy network for feature extraction to obtain the policy action corresponding to each device; encrypting the policy parameters of each device to obtain corresponding encrypted policy parameters, and sending the encrypted policy parameters, parameter estimates, and action-state of each device to an optimization engine through multiple transmission channels; for each device, optimizing the policy network based on the encrypted policy parameters through the optimization engine, and inputting the local observation vector to be tested into the optimized policy network to generate a data instruction for each device; controlling the corresponding device according to each data instruction, monitoring the control process of each device to obtain deviations, and optimizing the policy parameters of each device based on the deviations.

[0007] Optionally, the privacy-preserving instruction optimization method, wherein the device information includes: action information and state information; the step of obtaining the device information of each device in the topological device network and constructing a local observation vector for each device based on the neighbor relationships between each device and the corresponding device information specifically includes: obtaining the action information and state information of each device in the topological device network; determining all neighbor devices of each device, obtaining the neighbor relationships of each device and the action information and state information of all the corresponding neighbor devices; constructing an initial observation vector and optimizing it based on the action information and state information of each device, as well as the action information and state information of all the corresponding neighbor devices, to obtain a local observation vector for each device.

[0008] Optionally, the privacy-preserving instruction optimization method, wherein the step of constructing and optimizing an initial observation vector based on the action information and state information of each device, and the action information and state information of all neighboring devices corresponding to the device, to obtain a local observation vector for each device, specifically includes: for each device, constructing an initial observation vector based on the action information and state information, and the action information and state information of all neighboring devices: ;in, This represents the initial observation vector of the i-th device at time t. This represents the state of device j at time t. This indicates the first-layer range of neighboring devices. Indicates twice the value of device i The set of neighboring devices within the range, This represents the action of device k at time t-1. Represents device i The set of neighboring devices within the range; adding preset noise to the initial observation vector to perturb the initial observation vector, thereby obtaining the local observation vector of the device: ; ;in, This represents the local observation vector of the i-th device at time t. Indicates preset noise. express Follows a Laplace distribution. Indicates the scale parameter. The privacy budget parameter represents differential privacy.

[0009] Optionally, the privacy-preserving instruction optimization method, wherein inputting each local observation vector into a policy network for feature extraction to obtain the policy action corresponding to each device, specifically includes: for each device, obtaining the policy parameters of the device and the parameter estimates of each neighboring device; obtaining the constraint violation status of all neighboring devices of the device, adjusting the policy parameters of the device according to the constraint violation status; inputting the local observation vector of the device into the policy network, performing feature extraction on the local observation vector according to the current policy parameters of the device and all the parameter estimates, obtaining the probability corresponding to the policy action, and outputting: ;in, Indicates that device i is in Make an action The probability, This represents the action of device i at time t. This represents the local observation vector of the i-th device at time t. This represents the current policy parameters for device i. express eigenvectors, This represents the parameter estimate of neighboring device j. express eigenvectors, For device i The set of neighboring devices within the range, Let A represent all possible actions for device i, and let A represent the set of possible actions. This represents the state of device i at time t. This represents the state of neighbor device j at time t.

[0010] Optionally, the privacy-preserving instruction optimization method, wherein obtaining the constraint violation status of all neighboring devices of the device and adjusting the policy parameters of the device according to the constraint violation status, specifically includes: calculating the constraint violation status of the device at the current moment based on the action information and state information of all neighboring devices. ;in, This indicates the constraint violation status of device i at time t. Represents device i The surrounding neighboring equipment set, Indicates the action-state of neighboring device k. This represents the action of neighboring device k at time t. This represents the state of neighboring device k at time t. This indicates a violation of the threshold. The norm of the gradient of device i is represented; the relationship between the constraint violation and the violation threshold is determined, and if the constraint violation is greater than the violation threshold, the policy parameters of the device are adjusted.

[0011] Optionally, the privacy-preserving instruction optimization method, wherein encrypting the policy parameters of each device to obtain corresponding encrypted policy parameters, and sending the encrypted policy parameters, parameter estimates, and action-state of each device to the optimization engine through multiple transmission channels, specifically includes: for each device, encrypting the policy parameters by superimposing Gaussian noise using an encryption function to obtain encrypted policy parameters. ;in, This represents the encryption strategy parameters for device i at time t, where SM9.Enc represents the encryption function. This represents the policy parameters of device i at time t. Indicates Gaussian noise. This represents the variance of the Gaussian noise at time t. The variance of Gaussian noise is represented. The loss function L represents the loss function with respect to the policy parameters. The norm of; the action-state is sent to the encryption engine via the first transmission channel, and the encryption policy parameters and parameter estimates of each device are sent to the encryption engine via the second transmission channel, wherein the parameter estimates represent the estimated values ​​of the policy parameters.

[0012] Optionally, the privacy-preserving instruction optimization method, wherein sending the action-state to the encryption engine via a first transmission channel, and sending the encryption policy parameters and parameter estimates of each device to the encryption engine via a second transmission channel, specifically includes: inputting the action-state into the first transmission channel, mapping the action-state through the first transmission channel, and sending the mapping result to the encryption engine; inputting the parameter estimates into the second transmission channel, mapping the parameter estimates through the second transmission channel, and sending the mapping result to the encryption engine. Where EnvNet represents the first transmission channel and LearnNet represents the second transmission channel. This represents the mapping function in EnvNet, where s represents the state and a represents the action. express The set of neighboring devices within the range, This represents the mapping function in LearnNet. Represents the policy network, Indicates parameter estimation, Indicates a network of topological devices. express The i-th node in the array.

[0013] Optionally, the privacy-preserving instruction optimization method, wherein for each device, the optimization engine optimizes the policy network according to the encryption policy parameters, and the local observation vector to be tested is input into the optimized policy network to generate data instructions for each device, specifically includes: for each device, the optimization engine calculates parameter estimates for all neighboring devices, and performs weighted fusion on each parameter estimate to update the encryption policy parameters of the device, obtaining the updated policy parameters for the next time step. ; ;in, This represents the weighted fusion result at time t+1. Let L represent the set of neighboring device nodes of device i, and let L represent the loss function of the current policy network. This represents the weight between device i and its neighbor device j. This represents the weighted fusion result at time t. This represents the update strategy parameters for device i at time t+1. Let represent the policy parameters of device i at time t; for each device's policy network, calculate the truncated Q-values ​​of all neighboring devices, and use these truncated Q-values ​​to replace the global gradients in the device's policy network: ;in, Let E represent the global gradient of the policy network for device i, and let E represent the expectation operation. Indicates that device i is in The set of neighboring devices within the range, This indicates that the Q value is truncated. This represents the error term, and k represents the neighboring device. This represents the process of approximating the global gradient using a truncated Q-value; the policy network is alternately updated using the updated policy parameters and Lagrange multipliers, and the updated policy parameters are projected into the parameter space: ;in, This represents the policy parameters of device i at time t+1. The parameter space representing the projection. This represents the policy parameters of device i at time t. The learning rate represents the policy parameter. Represents the Lagrange multipliers. This represents the constraint parameters of device i at time t. This represents the constraint violation amount of device i. This represents the constraint parameters of device i at time t+1. Represents the maximum constraint parameter. This represents the constraint parameters at time t. The learning rate represents the constraint parameter. This represents the constraint function of device i based on the policy parameters. The constraint threshold of device i is represented; the local observation vector to be measured of the device is obtained, and the local observation vector is input into the optimized policy network to output the data command of the device at the next time step of the time step to be measured.

[0014] Optionally, the privacy-preserving instruction optimization method, wherein controlling the corresponding device according to each data instruction, monitoring the control process of each device to obtain deviations, and optimizing the policy parameters of each device based on the deviations, specifically includes: for each device, inputting the data instruction into the device, and calculating the deviation of the device's adjustment according to the data instruction: ;in, This indicates the deviation of device i at time t. This represents the parameter estimate of device i for its neighbor device j. Indicates an indicator function, Indicates the measurement parameters of device i. This represents the median of the parameter estimates for the neighboring device set. This represents the parameter estimate of neighboring device j. This indicates a deviation from the threshold; if the deviation exceeds the threshold, the current policy parameter corresponding to the device is defined as an abnormal parameter, and the abnormal parameter is restored to the initial policy parameter of the previous moment, so as to re-optimize the initial policy parameter.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a privacy-preserving instruction optimization system, wherein the privacy-preserving instruction optimization system includes: a data acquisition module, used to acquire device information of each device in a topological device network, and construct a local observation vector for each device based on the neighbor relationships between each device and the corresponding device information; an action prediction module, used to input each local observation vector into a policy network for feature extraction to obtain a policy action corresponding to each device; a parameter transmission module, used to encrypt the policy parameters of each device to obtain corresponding encrypted policy parameters, and send the encrypted policy parameters, parameter estimates, and action-state of each device to an optimization engine through multiple transmission channels; an instruction generation module, used to optimize the policy network for each device based on the encrypted policy parameters through the optimization engine, and input the local observation vector to be tested into the optimized policy network to generate a data instruction for each device; and a closed-loop optimization module, used to control the corresponding device according to each data instruction, monitor the control process of each device to obtain deviations, and optimize the policy parameters of each device based on the deviations.

[0016] The privacy-preserving instruction optimization system includes a data acquisition module comprising: a device information acquisition unit for acquiring action and status information of each device in the topological device network; a neighbor information acquisition unit for determining all neighboring devices of each device, obtaining the neighbor relationships of each device and the corresponding action and status information of all neighboring devices; and a vector construction unit for constructing and optimizing an initial observation vector based on the action and status information of each device, as well as the action and status information of all neighboring devices corresponding to the device, to obtain a local observation vector for each device.

[0017] The privacy-preserving instruction optimization system includes a vector construction unit comprising: an initial vector construction subunit, used to construct an initial observation vector for each device based on the action information and the state information, as well as the action information and the state information of all neighboring devices. ;in, This represents the initial observation vector of the i-th device at time t. This represents the state of device j at time t. This indicates the first-layer range of neighboring devices. Indicates twice the value of device i The set of neighboring devices within the range, This represents the action of device k at time t-1. Represents device i A set of neighboring devices within the range; a local vector construction subunit, used to add preset noise to the initial observation vector to perturb the initial observation vector, thereby obtaining the local observation vector of the device: ; ;in, This represents the local observation vector of the i-th device at time t. Indicates preset noise. express Follows a Laplace distribution. Indicates the scale parameter. The privacy budget parameter represents differential privacy.

[0018] The privacy-preserving instruction optimization system includes an action prediction module comprising: a neighbor parameter acquisition unit, used to acquire the policy parameters of each device and parameter estimates of each neighboring device; a parameter adjustment unit, used to acquire constraint violation information of all neighboring devices of the device and adjust the policy parameters of the device according to the constraint violation information; and a probability calculation unit, used to input the local observation vector of the device into the policy network, extract features from the local observation vector based on the current policy parameters of the device and all parameter estimates, obtain the probability corresponding to the policy action, and output the result. ;in, Indicates that device i is in Make an action The probability, This represents the action of device i at time t. This represents the local observation vector of the i-th device at time t. This represents the current policy parameters for device i. express eigenvectors, This represents the parameter estimate of neighboring device j. express eigenvectors, For device i The set of neighboring devices within the range, Let A represent all possible actions for device i, and let A represent the set of possible actions. This represents the state of device i at time t. This represents the state of neighbor device j at time t.

[0019] The privacy-preserving instruction optimization system includes a parameter adjustment unit comprising a constraint calculation subunit, used to calculate the constraint violation status of the device at the current moment based on the action and status information of all neighboring devices. ;in, This indicates the constraint violation status of device i at time t. Represents device i The surrounding neighboring equipment set, Indicates the action-state of neighboring device k. This represents the action of neighboring device k at time t. This represents the state of neighboring device k at time t. This indicates a violation of the threshold. The norm of the gradient of device i is represented; the constraint analysis subunit is used to determine the relationship between the constraint violation and the violation threshold. If the constraint violation is greater than the violation threshold, the policy parameters of the device are adjusted.

[0020] The privacy-preserving instruction optimization system includes a parameter transmission module comprising: a parameter encryption unit, used to encrypt the policy parameters for each device by superimposing Gaussian noise onto them using an encryption function, thereby obtaining encrypted policy parameters. ;in, This represents the encryption strategy parameters for device i at time t, where SM9.Enc represents the encryption function. This represents the policy parameters of device i at time t. Indicates Gaussian noise. This represents the variance of the Gaussian noise at time t. The variance of Gaussian noise is represented. The loss function L represents the loss function with respect to the policy parameters. The norm of; a data transmission unit, used to send the action-state to the encryption engine through a first transmission channel, and to send the encryption policy parameters and parameter estimates of each device to the encryption engine through a second transmission channel, wherein the parameter estimates represent the estimated values ​​of the policy parameters.

[0021] The privacy-preserving instruction optimization system includes a data transmission unit comprising: a first information transmission subunit, configured to input the action-state into a first transmission channel, map the action-state through the first transmission channel, and send the mapping result to the encryption engine; and a second information transmission subunit, configured to input the parameter estimate into a second transmission channel, map the parameter estimate through the second transmission channel, and send the mapping result to the encryption engine. Where EnvNet represents the first transmission channel and LearnNet represents the second transmission channel. This represents the mapping function in EnvNet, where s represents the state and a represents the action. express The set of neighboring devices within the range, This represents the mapping function in LearnNet. Represents the policy network, Indicates parameter estimation, Indicates a network of topological devices. express The i-th node in the array.

[0022] The privacy-preserving instruction optimization system includes an instruction generation module comprising: a parameter optimization unit, used to calculate parameter estimates for all neighboring devices using the optimization engine for each device, and to perform weighted fusion of each parameter estimate to update the encryption policy parameters of the device, thereby obtaining the updated policy parameters for the next time step. ; ;in, This represents the weighted fusion result at time t+1. Let L represent the set of neighboring device nodes of device i, and let L represent the loss function of the current policy network. This represents the weight between device i and its neighbor device j. This represents the weighted fusion result at time t. This represents the update strategy parameters for device i at time t+1. Let represent the policy parameters of device i at time t; the gradient substitution unit is used to calculate the truncated Q-values ​​of all neighboring devices for each device's policy network, and use the truncated Q-values ​​to replace the global gradients in the device's policy network: ;in, Let E represent the global gradient of the policy network for device i, and let E represent the expectation operation. Indicates that device i is in The set of neighboring devices within the range, This indicates that the Q value is truncated. This represents the error term, and k represents the neighboring device. This represents the process of approximating the global gradient using a truncated Q-value; the parameter mapping unit is used to alternately update the policy network using the updated policy parameters and Lagrange multipliers, and project the updated policy parameters into the parameter space: ;in, This represents the policy parameters of device i at time t+1. The parameter space representing the projection. This represents the policy parameters of device i at time t. The learning rate represents the policy parameter. Represents the Lagrange multipliers. This represents the constraint parameters of device i at time t. This represents the constraint violation amount of device i. This represents the constraint parameters of device i at time t+1. Represents the maximum constraint parameter. This represents the constraint parameters at time t. The learning rate represents the constraint parameter. This represents the constraint function of device i based on the policy parameters. The constraint threshold of device i is represented; the instruction construction unit is used to obtain the local observation vector to be measured of the device, input the local observation vector into the optimized policy network, and output the data instruction of the device at the next time step of the time step to be measured.

[0023] The privacy-preserving instruction optimization system includes a closed-loop optimization module comprising: a deviation analysis unit, used to input the data instruction into each device and calculate the deviation of the device's adjustment based on the data instruction. ;in, This indicates the deviation of device i at time t. This represents the parameter estimate of device i for its neighbor device j. Indicates an indicator function, Indicates the measurement parameters of device i. This represents the median of the parameter estimates for the neighboring device set. This represents the parameter estimate of neighboring device j. The deviation threshold is indicated; the network optimization unit is used to define the current policy parameter corresponding to the device as an abnormal parameter if the deviation exceeds the threshold, and restore the abnormal parameter to the initial policy parameter at the previous moment, so as to re-optimize the initial policy parameter.

[0024] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a privacy-preserving instruction optimization program stored in the memory and executable on the processor, wherein when the privacy-preserving instruction optimization program is executed by the processor, it implements the steps of the privacy-preserving instruction optimization method as described above.

[0025] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a privacy-preserving instruction optimization program, which, when executed by a processor, implements the steps of the privacy-preserving instruction optimization method as described above.

[0026] In this invention, multiple network indicator data are acquired through a driver framework in multiple drivers, and a dynamic link resource map is constructed based on all the network indicator data. Multiple connection requests or transmission requests for data streams are acquired, each data stream is identified, and a corresponding service quality requirement tag is added based on each identification result. Based on each service quality requirement tag, a corresponding link is matched for each data stream in the current link resource map. If a data stream does not match any of the links, multiple links are aggregated to match the data stream, resulting in a routing matching scheme. This invention improves data security by using dual network isolation to isolate test data from the environmental network during transmission; it reduces the complexity and cost of calculating the global gradient by using a truncated Q-value instead of the global gradient; and it protects data privacy and improves data accuracy. Attached Figure Description

[0027] Figure 1 is a first flowchart of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 2 is a second flowchart of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 3 is a third flowchart of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 4 is a fourth flowchart of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 5 is a fifth flowchart of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 6 is a sixth flowchart of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 7 is a seventh flowchart of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 8 is a flowchart of the privacy-preserving instruction optimization method of the present invention. Figure 9 is an eighth flowchart of a preferred embodiment of the method; Figure 10 is a specific principle structure diagram of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 11 is another specific principle structure diagram of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 12 is a framework flowchart of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 13 is an overall flowchart of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 14 is an optimization flowchart of a preferred embodiment of the privacy-preserving instruction optimization method of the present invention; Figure 15 is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0029] The privacy-preserving instruction optimization method of the preferred embodiment of the present invention, as shown in Figure 1, includes the following steps: Step S10: Obtain the device information of each device in the topology device network, and construct the local observation vector of each device according to the neighbor relationship between each device and the corresponding device information.

[0030] The equipment information includes: action information and status information.

[0031] Specifically, as shown in Figure 2, step S101 involves obtaining the action information and status information of each device in the topology device network.

[0032] Step S102: Determine all neighboring devices for each device, and obtain the neighbor relationships of each device and the corresponding action information and status information of all neighboring devices.

[0033] Step S103: Construct an initial observation vector and optimize it based on the action information and state information of each device, as well as the action information and state information of all neighboring devices corresponding to the device, to obtain the local observation vector of each device.

[0034] In this process, the sensors of multiple devices collect corresponding data through the environmental perception layer to determine the adjustment process of each device in the next stage. For the action and status information of each device, the action and status information of other neighboring devices that can communicate with the device are used to adjust the policy parameters of the device in the subsequent instruction optimization process to improve global correlation.

[0035] Further, as shown in Figure 3, in step S1031, for each device, an initial observation vector is constructed based on the action information and the state information, as well as the action information and the state information of all neighboring devices: ;in, This represents the initial observation vector of the i-th device at time t. This represents the state of device j at time t. This indicates the first-layer range of neighboring devices. Indicates twice the value of device i The set of neighboring devices within the range, This represents the action of device k at time t-1. Represents device i The set of neighboring devices within the range.

[0036] In the distributed system disclosed in this invention, each device uses action information and state information at the current moment, and then collects its 2-hop data (i.e., twice the amount mentioned above) for each device. The action and status information of all neighboring devices within the range are used to construct the initial observation vector of the current device, in order to protect information privacy and avoid global information exposure.

[0037] Step S1032 adds preset noise to the initial observation vector to perturb it, thereby obtaining the local observation vector of the device: ; ;in, This represents the local observation vector of the i-th device at time t. Indicates preset noise. express Follows a Laplace distribution. Indicates the scale parameter. The privacy budget parameter represents differential privacy.

[0038] In the process of data privacy protection, to ensure that the original state and action data cannot be inferred from the observations, this invention employs the addition of Laplace noise to achieve differential privacy. By adding this noise to the initial observation vector, the observations are perturbed, making it difficult for attackers to infer the original state and action data, thus achieving differential privacy protection.

[0039] Step S20: Input each of the local observation vectors into the policy network for feature extraction to obtain the policy action corresponding to each device.

[0040] Specifically, as shown in Figure 4, step S201 involves obtaining the policy parameters of each device and the parameter estimates of each neighboring device for each device.

[0041] Step S202: Obtain the constraint violation status of all neighboring devices of the device, and adjust the policy parameters of the device according to the constraint violation status.

[0042] Before initial control is performed, it is necessary to determine whether the device's strategy parameters need to be adjusted. In the embodiments disclosed in this invention, a risk gradient is generated by detecting constraint violations of neighboring devices within a preset range, and then compared with the violation threshold to determine whether the device's strategy parameters need to be adjusted.

[0043] Further, as shown in Figure 5, step S2021: Based on the action information and status information of all the neighboring devices, calculate the constraint violation status of the device at the current moment: ;in, This indicates the constraint violation status of device i at time t. Represents device i The surrounding neighboring equipment set, Indicates the action-state of neighboring device k. This represents the action of neighboring device k at time t. This represents the state of neighboring device k at time t. This indicates a violation of the threshold. Let represent the norm of the gradient of device i.

[0044] Step S2022: Determine the relationship between the constraint violation and the violation threshold. If the constraint violation is greater than the violation threshold, adjust the policy parameters of the device.

[0045] In multi-device scenarios, this invention, for the current device, if a risk gradient is generated, it indicates that the sum of constraint violations by the current device's neighboring devices exceeds the violation threshold. Therefore, based on the current device's own policy parameters, adaptive adjustments are made to address constraint violations and ensure that the system can operate normally and meet the constraint conditions.

[0046] Step S203: Input the local observation vector of the device into the policy network, extract features from the local observation vector based on the current policy parameters of the device and all the parameter estimates, obtain the probability corresponding to the policy action, and output: ;in, Indicates that device i is in Make an action The probability, This represents the action of device i at time t. This represents the local observation vector of the i-th device at time t. This represents the current policy parameters for device i. express eigenvectors, This represents the parameter estimate of neighboring device j. express eigenvectors, For device i The set of neighboring devices within the range, Let A represent all possible actions for device i, and let A represent the set of possible actions. This represents the state of device i at time t. This represents the state of neighbor device j at time t.

[0047] In the distributed system disclosed in this invention, each device's decision-making process relies on its own policy parameters and the parameter estimates of its neighboring devices (parameter estimates represent the estimated values ​​of the policy parameters of the neighboring devices). The device's own policy parameters reflect its own characteristics, etc. Relevant features are extracted from information such as device state through a feature extraction function. Thus, the optimal action of the device at the current moment is calculated using its own policy parameters, features, and the estimated parameters and features of its neighbors. The optimal action is reflected by the action probability. The coefficient 0.5 in the above formula controls the degree of influence of the neighbor's policy on the device's own decision, thereby controlling the policy coupling depth, that is, the degree of correlation between the device's own policy and the neighbor's policy when making a decision.

[0048] Step S30: Encrypt the policy parameters of each device to obtain the corresponding encrypted policy parameters, and send the encrypted policy parameters, parameter estimates, and action-state of each device to the optimization engine through multiple transmission channels.

[0049] Specifically, as shown in Figure 6, step S301 involves encrypting the policy parameters by superimposing Gaussian noise onto them using an encryption function, thereby obtaining encrypted policy parameters. ;in, This represents the encryption strategy parameters for device i at time t, where SM9.Enc represents the encryption function. This represents the policy parameters of device i at time t. Indicates Gaussian noise. This represents the variance of the Gaussian noise at time t. The variance of Gaussian noise is represented. The loss function L represents the loss function with respect to the policy parameters. The norm of .

[0050] Step S302: Send the action-state to the encryption engine through the first transmission channel, and send the encryption policy parameters and parameter estimates of each device to the encryption engine through the second transmission channel, wherein the parameter estimates represent the estimated values ​​of the policy parameters.

[0051] In the embodiments disclosed in this invention, to further enhance the security of device data, the device's strategy parameters first and foremost follow a mean of 0 and a variance of . The Gaussian distributed noise is XORed and then encrypted using the national cryptographic SM9 algorithm to obtain the encrypted policy parameters.

[0052] Among them, the standard deviation of the noise and the loss function L with respect to the policy parameters The norm of the gradient is proportional to the noise variance. In the early stage of model training, the norm is large and the noise variance is also large, so more noise is added to increase the perturbation of the parameters. As training progresses, the norm decreases and the noise variance dynamically decays. The amount of noise added gradually decreases, so that the policy parameters tend to stabilize. This can protect the privacy of the policy parameters and balance the needs of convergence and exploration during the training process.

[0053] Further, as shown in Figure 7, step S3021 involves inputting the action-state into the first transmission channel, mapping the action-state through the first transmission channel, and sending the mapping result to the encryption engine.

[0054] Step S3022: Input the parameter estimate into the second transmission channel, map the parameter estimate through the second transmission channel, and send the mapping result to the encryption engine: Where EnvNet represents the first transmission channel and LearnNet represents the second transmission channel. This represents the mapping function in EnvNet, where s represents the state and a represents the action. express The set of neighboring devices within the range, This represents the mapping function in LearnNet. Represents the policy network, Indicates parameter estimation, Indicates a network of topological devices. express The i-th node in the array.

[0055] Specifically, state data and action data are transmitted separately from parameter estimation data through two physically isolated channels. This makes it difficult for attackers to simultaneously breach both channels to obtain complete data, thus blocking cross-network attack chains and improving system security and privacy.

[0056] Step S40: For each device, the optimization engine optimizes the policy network according to the encryption policy parameters, and inputs the local observation vector to be tested into the optimized policy network to generate data instructions for each device.

[0057] Specifically, the parameter estimates are distributed and averaged through a weight matrix to eliminate the time-varying effects of the communication topology. Then, the truncated Q-values ​​of local neighbors are used to approximate the global gradient. At the same time, the policy parameters and Lagrange multipliers are updated alternately, and the stability is ensured by projection operations.

[0058] Specifically, as shown in Figure 8, step S401 involves calculating parameter estimates for all neighboring devices using the optimization engine for each device, and then weighting and fusing each parameter estimate to update the encryption policy parameters of the device, thereby obtaining the updated policy parameters for the next time step. ; ;in, This represents the weighted fusion result at time t+1. Let L represent the set of neighboring device nodes of device i, and let L represent the loss function of the current policy network. This represents the weight between device i and its neighbor device j. This represents the weighted fusion result at time t. This represents the update strategy parameters for device i at time t+1. This represents the policy parameters of device i at time t.

[0059] In distributed systems, time-varying communication topology can affect parameter estimation. Therefore, in the embodiments disclosed in this invention, a weight matrix is ​​used to fuse the parameter estimates of all neighboring devices of the current device and use them to replace its own policy parameters to achieve distributed averaging. Even if the communication topology changes, the device can reasonably update its own parameters and the estimated values ​​of the neighboring parameters according to the situation of its neighbors, thereby eliminating the impact of time-varying communication topology on parameter estimation to a certain extent.

[0060] Step S402: For the policy network of each device, calculate the truncated Q-values ​​of all neighboring devices, and use the truncated Q-values ​​to replace the global gradients in the policy network of the device. ;in, Let E represent the global gradient of the policy network for device i, and let E represent the expectation operation. Indicates that device i is in The set of neighboring devices within the range, This indicates that the Q value is truncated. This represents the error term, and k represents the neighboring device. This represents the process of approximating the global gradient using a truncated Q-value.

[0061] In the process of optimizing the policy network, the truncated Q-value of neighboring devices is used to approximate the global gradient. This reduces computation and communication costs, protects privacy, and ensures the accuracy of the approximation because the error decays exponentially.

[0062] Step S403: Alternately update the policy network using the updated policy parameters and Lagrange multipliers, and project the updated policy parameters into the parameter space: ;in, This represents the policy parameters of device i at time t+1. The parameter space representing the projection. This represents the policy parameters of device i at time t. The learning rate represents the policy parameter. Represents the Lagrange multipliers. This represents the constraint parameters of device i at time t. This represents the constraint violation amount of device i. This represents the constraint parameters of device i at time t+1. Represents the maximum constraint parameter. This represents the constraint parameters at time t. The learning rate represents the constraint parameter. This represents the constraint function of device i based on the policy parameters. This represents the constraint threshold of device i.

[0063] Step S404: Obtain the local observation vector to be measured of the device, input the local observation vector into the optimized policy network, and output the data command of the device at the next time step of the time step to be measured.

[0064] In the embodiments disclosed in this invention, the strategy parameters are set to be updated in conjunction with the Lagrange multipliers, and the parameters in the system are continuously adjusted to optimize the objective function while satisfying the constraints. The updated parameters are then projected onto the parameter space through a projection operation to maintain the stability of the parameters.

[0065] Step S50: Control the corresponding device according to each data instruction, monitor the control process of each device to obtain the deviation, and optimize the strategy parameters of each device according to the deviation.

[0066] Specifically, after using the optimized policy network to output actions to the devices, the deviation between local parameters and the median of the neighbors is monitored. When the variance exceeds the threshold, the parameters are reset to achieve closed-loop optimization of the policy parameters of each device.

[0067] Specifically, as shown in Figure 9, step S501 involves inputting the data instruction into each device and calculating the deviation of the device's adjustment based on the data instruction: ;in, This indicates the deviation of device i at time t. This represents the parameter estimate of device i for its neighbor device j. Indicates an indicator function, Indicates the measurement parameters of device i. This represents the median of the parameter estimates for the neighboring device set. This represents the parameter estimate of neighboring device j. This indicates a deviation from the threshold.

[0068] Step S502: If the deviation exceeds the threshold, the current policy parameter corresponding to the device is defined as an abnormal parameter, and the abnormal parameter is restored to the initial policy parameter at the previous moment, so as to re-optimize the initial policy parameter.

[0069] Specifically, by calculating the deviation of the device at the current moment, the system monitors the deviation between the device's local parameters and the parameter estimates of neighboring devices, while also considering the variance of the neighboring device's estimated parameters. When the variance of the neighboring device's parameter estimates is large (i.e., large fluctuations), and the deviation of the device's local parameters from the median of the parameter estimates exceeds the deviation threshold, the indicator function is defined as 1. The deviation value is more significantly affected by the variance. This indicates that the device's current policy parameters have become abnormal, such as being affected by noise interference or deviations in the learning process. In this case, a parameter reset operation is triggered to restore the relevant parameters of the device to a certain initial state or to recalculate and adjust them to ensure the stability and normal operation of the system.

[0070] Furthermore, if the above conditions are not met, it indicates that the equipment is operating normally and there is no need to optimize the strategy parameters.

[0071] Furthermore, as shown in Figure 10, based on the above-mentioned privacy-preserving instruction optimization method, the present invention also provides a privacy-preserving instruction optimization system, wherein the privacy-preserving instruction optimization system includes: a data acquisition module 10, used to acquire device information of each device in the topology device network, and construct a local observation vector for each device based on the neighbor relationship between each device and the corresponding device information; an action prediction module 20, used to input each local observation vector into a policy network for feature extraction to obtain the policy action corresponding to each device; and a parameter transmission module 30, used to transmit the policy parameters of each device. Encryption is performed to obtain corresponding encryption strategy parameters, and the encryption strategy parameters, parameter estimates, and action-state of each device are sent to the optimization engine through multiple transmission channels. The instruction generation module 40 is used to optimize the policy network for each device according to the encryption strategy parameters through the optimization engine, and input the local observation vector to be tested into the optimized policy network to generate data instructions for each device. The closed-loop optimization module 50 is used to control the corresponding device according to each data instruction, monitor the control process of each device to obtain deviations, and optimize the policy parameters of each device according to the deviations.

[0072] As shown in Figure 11, another specific embodiment of the privacy-protected instruction optimization method and system in this invention includes: a data acquisition module 10, an action prediction module 20, a parameter transmission module 30, an instruction generation module 40, and a closed-loop optimization module 50.

[0073] As shown in Figure 12, existing collaborative control technologies cannot provide adequate security audits for end-to-end black-box decision-making. Therefore, this invention verifies the process by using variance detection in the verification loop phase to locate abnormal decision nodes in real time, thereby improving the visualization of the optimization process, accurately locating anomalies, and improving the management efficiency of equipment.

[0074] Specifically, the data acquisition module 10 includes: a device information acquisition unit 101, a neighbor information acquisition unit 102, and a vector construction unit 103; the device information acquisition unit 101 is used to acquire the action information and status information of each device in the topological device network; the neighbor information acquisition unit 102 is used to determine all neighbor devices of each device, obtain the neighbor relationship of each device and the corresponding action information and status information of all neighbor devices; the vector construction unit 103 is used to construct an initial observation vector and optimize it based on the action information and status information of each device, as well as the action information and status information of all neighbor devices corresponding to the device, to obtain the local observation vector of each device.

[0075] As shown in Figure 13, after obtaining various data information from the devices through the environmental perception layer, the data is preprocessed directly through the device security sandbox to construct a local observation vector for each device. This vector is then transmitted to the system decision engine for initial decision-making, thereby improving the efficiency of data processing.

[0076] Further, the vector construction unit 103 includes: an initial vector construction subunit 1031 and a local vector construction subunit 1032; the initial vector construction subunit 1031 is used to construct an initial observation vector for each device based on the action information and the state information, as well as the action information and the state information of all neighboring devices: ;in, This represents the initial observation vector of the i-th device at time t. This represents the state of device j at time t. This indicates the first-layer range of neighboring devices. Indicates twice the value of device i The set of neighboring devices within the range, This represents the action of device k at time t-1. Represents device i The set of neighboring devices within the range; the local vector construction subunit 1032 is used to add preset noise to the initial observation vector to perturb the initial observation vector and obtain the local observation vector of the device: ; ;in, This represents the local observation vector of the i-th device at time t. Indicates preset noise. express Follows a Laplace distribution. Indicates the scale parameter. The privacy budget parameter represents differential privacy.

[0077] In one of the embodiments disclosed in this invention, by aggregating 2 times... Real-time status of neighboring devices within range and By analyzing the previous actions of neighboring devices within the range, we can obtain the local observation vector of the device at the current moment. In this way, each device only uses information within its local range, thereby avoiding the exposure of global information.

[0078] Further, the action prediction module 20 includes: a neighbor parameter acquisition unit 201, a parameter adjustment unit 202, and a probability calculation unit 203; the neighbor parameter acquisition unit 201 is used to acquire the policy parameters of the device and the parameter estimates of each neighbor device for each device; the parameter adjustment unit 202 is used to acquire the constraint violation status of all the neighbor devices of the device, and adjust the policy parameters of the device according to the constraint violation status; the probability calculation unit 203 is used to input the local observation vector of the device into the policy network, extract features from the local observation vector according to the current policy parameters of the device and all the parameter estimates, obtain the probability corresponding to the policy action, and output: ;in, Indicates that device i is in Make an action The probability, This represents the action of device i at time t. This represents the local observation vector of the i-th device at time t. This represents the current policy parameters for device i. express eigenvectors, This represents the parameter estimate of neighboring device j. express eigenvectors, For device i The set of neighboring devices within the range, Let A represent all possible actions for device i, and let A represent the set of possible actions. This represents the state of device i at time t. This represents the state of neighbor device j at time t.

[0079] In multi-device collaborative or distributed systems, the decision-making process of a device has a specific mechanism that relies on its own policy parameters and the parameter estimates of neighboring devices. The policy parameters can reflect the device's own characteristics, preferences, or learned policy information, while the parameter estimates can affect the device's specific decisions. Through feature extraction functions, feature vectors that reflect the essence of the device's state are extracted from the device's action information and state information for subsequent decision calculations.

[0080] Furthermore, during the probability calculation of policy actions, a policy coupling depth is introduced to limit the impact of neighbor parameter estimation on the device's own policy parameters. If the weight of neighbor-related terms increases, the device's decision will be more influenced by the neighbor's policy, and the policy coupling depth will increase; conversely, it will decrease. This also prevents excessive interference from neighboring devices while enhancing connectivity between devices, thus improving the accuracy of data prediction.

[0081] Further, the parameter adjustment unit 202 includes: a constraint calculation subunit 2021 and a constraint analysis subunit 2022; the constraint calculation subunit 2021 is used to calculate the constraint violation status of the device at the current moment based on the action information and status information of all the neighboring devices. ;in, This indicates the constraint violation status of device i at time t. Represents device i The surrounding neighboring equipment set, Indicates the action-state of neighboring device k. This represents the action of neighboring device k at time t. This represents the state of neighboring device k at time t. This indicates a violation of the threshold. The norm of the gradient of device i is represented; the constraint analysis subunit 2022 is used to determine the relationship between the constraint violation and the violation threshold. If the constraint violation is greater than the violation threshold, the policy parameters of the device are adjusted.

[0082] Further, the parameter transmission module 30 includes: a parameter encryption unit 301 and a data transmission unit 302; the parameter encryption unit 301 is used to encrypt the policy parameters for each device by superimposing Gaussian noise onto them using an encryption function, thereby obtaining encrypted policy parameters: ;in, This represents the encryption strategy parameters for device i at time t, where SM9.Enc represents the encryption function. This represents the policy parameters of device i at time t. Indicates Gaussian noise. This represents the variance of the Gaussian noise at time t. The variance of Gaussian noise is represented. The loss function L represents the loss function with respect to the policy parameters. The norm of; the data transmission unit 302 is used to send the action-state to the encryption engine through the first transmission channel, and to send the encryption policy parameters and parameter estimates of each device to the encryption engine through the second transmission channel, wherein the parameter estimates represent the estimated values ​​of the policy parameters.

[0083] Among them, SM9.Enc is an encryption function based on the SM9 algorithm (a cryptographic algorithm with high security, used to protect the privacy of policy parameters during transmission or storage). It encrypts the parameters after XOR operation to obtain the encrypted policy parameters.

[0084] This approach protects the privacy of policy parameters through SM9 encryption, a national standard, and balances the needs of exploration (adding more noise allows for a wider range of parameter variations, which helps in exploring different policy spaces) and convergence (the noise decays as convergence progresses, gradually stabilizing the parameters) during training by adding adaptive Gaussian noise.

[0085] Further, the data transmission unit 302 includes: a first information transmission subunit 3021 and a second information transmission subunit 3022; the first information transmission subunit 3021 is used to input the action-state into a first transmission channel, map the action-state through the first transmission channel, and send the mapping result to the encryption engine; the second information transmission subunit 3022 is used to input the parameter estimate into a second transmission channel, map the parameter estimate through the second transmission channel, and send the mapping result to the encryption engine. Where EnvNet represents the first transmission channel and LearnNet represents the second transmission channel. This represents the mapping function in EnvNet, where s represents the state and a represents the action. express The set of neighboring devices within the range, This represents the mapping function in LearnNet. Represents the policy network, Indicates parameter estimation, Indicates a network of topological devices. express The i-th node in the array.

[0086] EnvNet stands for Environment Network, which is the channel for transmitting state and action data. In systems with high security requirements, state data and action data are transmitted separately from parameter estimation data through two physically isolated channels. This makes it difficult for attackers to break through both channels at the same time to obtain complete data, thus blocking cross-network attack chains and improving system security and privacy.

[0087] LearnNet is a learning network that transmits parameter estimation data. The parameter estimation data is processed by a mapping function and transmitted in the learning network to a specific set or region related to the device for subsequent learning or policy updates.

[0088] Further, the instruction generation module 40 includes: a parameter optimization unit 401, a gradient substitution unit 402, a parameter mapping unit 403, and an instruction construction unit 404; the parameter optimization unit 401 is used to calculate parameter estimates of all neighboring devices for each device through the optimization engine, and to perform weighted fusion of each parameter estimate to update the encryption policy parameters of the device, thereby obtaining the updated policy parameters for the next time step. ; ;in, This represents the weighted fusion result at time t+1. Let L represent the set of neighboring device nodes of device i, and let L represent the loss function of the current policy network. This represents the weight between device i and its neighbor device j. This represents the weighted fusion result at time t. This represents the update strategy parameters for device i at time t+1. This represents the policy parameters of device i at time t; the gradient substitution unit 402 is used to calculate the truncated Q-values ​​of all neighboring devices for each device's policy network, and use the truncated Q-values ​​to replace the global gradients in the device's policy network. ;in, Let E represent the global gradient of the policy network for device i, and let E represent the expectation operation. Indicates that device i is in The set of neighboring devices within the range, This indicates that the Q value is truncated. This represents the error term, and k represents the neighboring device. This represents the process of approximating the global gradient using a truncated Q-value; the parameter mapping unit 403 is used to alternately update the policy network using the updated policy parameters and Lagrange multipliers, and project the updated policy parameters into the parameter space: ;in, This represents the policy parameters of device i at time t+1. The parameter space representing the projection. This represents the policy parameters of device i at time t. The learning rate represents the policy parameter. Represents the Lagrange multipliers. This represents the constraint parameters of device i at time t. This represents the constraint violation amount of device i. This represents the constraint parameters of device i at time t+1. Represents the maximum constraint parameter. This represents the constraint parameters at time t. The learning rate represents the constraint parameter. This represents the constraint function of device i based on the policy parameters. The constraint threshold of device i is represented; the instruction construction unit 404 is used to obtain the local observation vector to be measured of the device, input the local observation vector into the optimized policy network, and output the data instruction of the device at the next time step of the time step to be measured.

[0089] As shown in Figure 14, the observation vectors output by the environmental perception layer drive the decision engine to generate actions; these actions are securely transmitted through the privacy layer; the optimization engine updates its strategy based on encrypted parameters; and the verification module monitors system stability. During the strategy update process of the optimization engine, when estimating and updating the strategy parameters for each device, temporal changes due to communication topology exist. That is, the connection relationships and communication capabilities between devices may change over time. Therefore, for the strategy parameters of each device, the update is constrained by a weighted fusion result of the parameter estimates from neighboring devices to mitigate the impact of time-varying communication topology on parameter estimation.

[0090] Furthermore, the closed-loop optimization module 50 includes: a deviation analysis unit 501 and a network optimization unit 502; the deviation analysis unit 501 is used to input the data instruction into each device and calculate the deviation of the device's adjustment according to the data instruction. ;in, This indicates the deviation of device i at time t. This represents the parameter estimate of device i for its neighbor device j. Indicates an indicator function, Indicates the measurement parameters of device i. This represents the median of the parameter estimates for the neighboring device set. This represents the parameter estimate of neighboring device j. The deviation threshold is indicated; the network optimization unit 502 is used to define the current policy parameter corresponding to the device as an abnormal parameter if the deviation exceeds the threshold, and restore the abnormal parameter to the initial policy parameter at the previous moment, so as to re-optimize the initial policy parameter.

[0091] The deviation is determined based on subsequent calculations and is used to determine whether a policy update operation needs to be triggered. The system monitors the deviation between the local parameters of the monitoring device and the median of neighboring devices by calculating the deviation. When the deviation exceeds the deviation threshold, it indicates that the device's policy parameters are abnormal and need to be updated. At this time, a parameter reset operation will be triggered to restore the relevant parameters of the device to a certain initial state or to recalculate and adjust them to ensure the stability and normal operation of the system.

[0092] Furthermore, the application scenarios of the privacy-preserving instruction optimization method and system disclosed in this invention are illustrated by the following embodiments, including but not limited to: For multiple collaborative robots in automotive manufacturing or electronic engineering, where assembly paths need to be adjusted in real time, the pose state of each device (e.g., the robot's joint angles) can be collected through the environmental perception layer. Then, the collaborative decision engine estimates and generates collision avoidance trajectories (e.g., the path traversed by the robotic arm of each device) based on the parameters of the neighboring devices. The collision avoidance trajectory is then optimally verified, and the path parameters are reset based on confidence interval detection. Furthermore, when a new device is introduced (i.e., when a new device is added to the topology device network), all original policy parameters can be updated to adapt to the new topology device network, improving device control efficiency and applicability.

[0093] Furthermore, based on traffic flow optimization scenarios, it can be used to coordinate the timing of traffic lights at multiple intersections. By acquiring road network data (including traffic density, queue length, etc. at the intersection) through the environmental perception layer, the distributed optimization engine calculates the truncated gradient (truncated Q value) to replace the policy parameters of the corresponding traffic lights at the intersection and dynamically updates the parameters. During parameter transmission, the roadside unit exchanges encrypted policy parameters through the national cryptographic standard SM9, thereby improving the road network throughput.

[0094] Furthermore, for the regulation of distributed energy microgrids, such as wind-solar-storage integrated power station clusters, it is necessary to balance power generation and load fluctuations. The output curve is adjusted based on the parameter estimation of neighboring equipment by the collaborative decision engine, and then a dual update is performed. Through the mapping process, the voltage over-limit rate is ensured to be lower than the preset value, which significantly reduces the curtailment rate of power generation.

[0095] Furthermore, as shown in Figure 15, based on the aforementioned privacy-preserving instruction optimization method and system, the present invention also provides a terminal, which includes a processor 100, a memory 200, and a display 300. Figure 15 only shows some components of the terminal; however, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented alternatively.

[0096] In some embodiments, the memory 200 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 200 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 200 may include both internal and external storage devices. The memory 200 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 200 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 200 stores a privacy-preserving instruction optimization program 400, which can be executed by the processor 100 to implement the privacy-preserving instruction optimization method of this application.

[0097] In some embodiments, the processor 100 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 200 or process data, such as executing the privacy-preserving instruction optimization method.

[0098] In some embodiments, the display 300 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 300 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other via a system bus.

[0099] In one embodiment, when the processor 100 executes the privacy-preserving instruction optimization program 400 in the memory 200, the following steps are implemented: acquiring device information for each device in the topology device network; constructing a local observation vector for each device based on the neighbor relationships between each device and the corresponding device information; inputting each local observation vector into a policy network for feature extraction to obtain the policy action corresponding to each device; encrypting the policy parameters of each device to obtain corresponding encrypted policy parameters, and sending the encrypted policy parameters, parameter estimates, and action-state of each device to an optimization engine through multiple transmission channels; for each device, optimizing the policy network based on the encrypted policy parameters through the optimization engine, and inputting the local observation vector to be tested into the optimized policy network to generate data instructions for each device; controlling the corresponding device according to each data instruction, monitoring the control process of each device to obtain deviations, and optimizing the policy parameters of each device based on the deviations.

[0100] The device information includes action information and status information. The process of obtaining device information for each device in the topological device network and constructing a local observation vector for each device based on the neighbor relationships between each device and the corresponding device information specifically includes: obtaining the action information and status information of each device in the topological device network; determining all neighbor devices of each device to obtain the neighbor relationships of each device and the corresponding action information and status information of all neighbor devices; constructing and optimizing an initial observation vector based on the action information and status information of each device, as well as the action information and status information of all neighbor devices corresponding to the device, to obtain the local observation vector for each device.

[0101] Specifically, the step of constructing and optimizing an initial observation vector based on the action information and state information of each device, as well as the action information and state information of all neighboring devices corresponding to that device, to obtain a local observation vector for each device, includes: for each device, constructing an initial observation vector based on the action information and state information, as well as the action information and state information of all neighboring devices. ;in, This represents the initial observation vector of the i-th device at time t. This represents the state of device j at time t. This indicates the first-layer range of neighboring devices. Indicates twice the value of device i The set of neighboring devices within the range, This represents the action of device k at time t-1. Represents device i The set of neighboring devices within the range; adding preset noise to the initial observation vector to perturb the initial observation vector, thereby obtaining the local observation vector of the device: ; ;in, This represents the local observation vector of the i-th device at time t. Indicates preset noise. express Follows a Laplace distribution. Indicates the scale parameter. The privacy budget parameter represents differential privacy.

[0102] Specifically, the step of inputting each local observation vector into a policy network for feature extraction to obtain the policy action corresponding to each device includes: for each device, obtaining the policy parameters of the device and the parameter estimates of each neighboring device; obtaining the constraint violation status of all neighboring devices of the device, adjusting the policy parameters of the device according to the constraint violation status; inputting the local observation vector of the device into the policy network, extracting features from the local observation vector according to the current policy parameters of the device and all the parameter estimates, obtaining the probability corresponding to the policy action, and outputting: ;in, Indicates that device i is in Make an action The probability, This represents the action of device i at time t. This represents the local observation vector of the i-th device at time t. This represents the current policy parameters for device i. express eigenvectors, This represents the parameter estimate of neighboring device j. express eigenvectors, For device i The set of neighboring devices within the range, Let A represent all possible actions for device i, and let A represent the set of possible actions. This represents the state of device i at time t. This represents the state of neighbor device j at time t.

[0103] The step of obtaining constraint violation information of all neighboring devices and adjusting the policy parameters of the device based on the constraint violation information specifically includes: calculating the constraint violation status of the device at the current moment based on the action information and state information of all neighboring devices. ;in, This indicates the constraint violation status of device i at time t. Represents device i The surrounding neighboring equipment set, Indicates the action-state of neighboring device k. This represents the action of neighboring device k at time t. This represents the state of neighboring device k at time t. This indicates a violation of the threshold. The norm of the gradient of device i is represented; the relationship between the constraint violation and the violation threshold is determined, and if the constraint violation is greater than the violation threshold, the policy parameters of the device are adjusted.

[0104] Specifically, the process of encrypting the policy parameters of each device to obtain corresponding encrypted policy parameters, and then sending the encrypted policy parameters, parameter estimates, and action-state of each device to the optimization engine through multiple transmission channels, includes: for each device, encrypting the policy parameters by adding Gaussian noise to the policy parameters using an encryption function to obtain encrypted policy parameters. ;in, This represents the encryption strategy parameters for device i at time t, where SM9.Enc represents the encryption function. This represents the policy parameters of device i at time t. Indicates Gaussian noise. This represents the variance of the Gaussian noise at time t. The variance of Gaussian noise is represented. The loss function L represents the loss function with respect to the policy parameters. The norm of; the action-state is sent to the encryption engine via the first transmission channel, and the encryption policy parameters and parameter estimates of each device are sent to the encryption engine via the second transmission channel, wherein the parameter estimates represent the estimated values ​​of the policy parameters.

[0105] Specifically, sending the action-state to the encryption engine via a first transmission channel and sending the encryption policy parameters and parameter estimates of each device to the encryption engine via a second transmission channel includes: inputting the action-state into the first transmission channel, mapping the action-state through the first transmission channel, and sending the mapping result to the encryption engine; inputting the parameter estimates into the second transmission channel, mapping the parameter estimates through the second transmission channel, and sending the mapping result to the encryption engine. Where EnvNet represents the first transmission channel and LearnNet represents the second transmission channel. This represents the mapping function in EnvNet, where s represents the state and a represents the action. express The set of neighboring devices within the range, This represents the mapping function in LearnNet. Represents the policy network, Indicates parameter estimation, Indicates a network of topological devices. express The i-th node in the array.

[0106] Specifically, for each device, the optimization engine optimizes the policy network based on the encryption policy parameters, and the local observation vector to be tested is input into the optimized policy network to generate data instructions for each device. This includes: for each device, the optimization engine calculates parameter estimates for all neighboring devices, and weights and fuses each parameter estimate to update the encryption policy parameters of the device, obtaining the updated policy parameters for the next time step. ; ;in, This represents the weighted fusion result at time t+1. Let L represent the set of neighboring device nodes of device i, and let L represent the loss function of the current policy network. This represents the weight between device i and its neighbor device j. This represents the weighted fusion result at time t. This represents the update strategy parameters for device i at time t+1. Let represent the policy parameters of device i at time t; for each device's policy network, calculate the truncated Q-values ​​of all neighboring devices, and use these truncated Q-values ​​to replace the global gradients in the device's policy network: ;in, Let E represent the global gradient of the policy network for device i, and let E represent the expectation operation. Indicates that device i is in The set of neighboring devices within the range, This indicates that the Q value is truncated. This represents the error term, and k represents the neighboring device. This represents the process of approximating the global gradient using a truncated Q-value; the policy network is alternately updated using the updated policy parameters and Lagrange multipliers, and the updated policy parameters are projected into the parameter space: ;in, This represents the policy parameters of device i at time t+1. The parameter space representing the projection. This represents the policy parameters of device i at time t. The learning rate represents the policy parameter. Represents the Lagrange multipliers. This represents the constraint parameters of device i at time t. This represents the constraint violation amount of device i. This represents the constraint parameters of device i at time t+1. Represents the maximum constraint parameter. This represents the constraint parameters at time t. The learning rate represents the constraint parameter. This represents the constraint function of device i based on the policy parameters. The constraint threshold of device i is represented; the local observation vector to be measured of the device is obtained, and the local observation vector is input into the optimized policy network to output the data command of the device at the next time step of the time step to be measured.

[0107] Specifically, the step of controlling the corresponding device according to each data instruction, monitoring the control process of each device to obtain deviations, and optimizing the strategy parameters of each device based on the deviations includes: for each device, inputting the data instruction into the device, and calculating the deviation of the device's adjustment based on the data instruction. ;in, This indicates the deviation of device i at time t. This represents the parameter estimate of device i for its neighbor device j. Indicates an indicator function, Indicates the measurement parameters of device i. This represents the median of the parameter estimates for the neighboring device set. This represents the parameter estimate of neighboring device j. This indicates a deviation from the threshold; if the deviation exceeds the threshold, the current policy parameter corresponding to the device is defined as an abnormal parameter, and the abnormal parameter is restored to the initial policy parameter of the previous moment, so as to re-optimize the initial policy parameter.

[0108] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a privacy-preserving instruction optimization program, which, when executed by a processor, implements the steps of the privacy-preserving instruction optimization method as described above.

[0109] In summary, this invention provides a privacy-preserving instruction optimization method and related equipment. The method includes: acquiring multiple network indicator data in multiple drivers through a driver framework, and constructing a dynamic link resource map based on all the network indicator data; acquiring connection requests or transmission requests of multiple data streams, identifying each data stream, and adding a corresponding service quality requirement tag based on each identification result; matching a corresponding link for each data stream in the current link resource map based on each service quality requirement tag; if a data stream does not match any of the links, aggregating multiple links to match the data stream to obtain a routing matching scheme. This invention improves data security by using dual network isolation to isolate test data from the environmental network during transmission; it reduces the complexity and cost of calculating the global gradient by using a truncated Q-value instead of the global gradient; and it protects data privacy and improves data accuracy.

[0110] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0111] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0112] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A privacy-preserving instruction optimization method, characterized in that, The privacy-preserving instruction optimization method includes: acquiring device information for each device in the topological device network; constructing a local observation vector for each device based on the neighbor relationships between each device and the corresponding device information; inputting each local observation vector into a policy network for feature extraction to obtain the policy action corresponding to each device; encrypting the policy parameters of each device to obtain corresponding encrypted policy parameters, and sending the encrypted policy parameters, parameter estimates, and action-state of each device to an optimization engine through multiple transmission channels; for each device, optimizing the policy network based on the encrypted policy parameters through the optimization engine, and inputting the local observation vector to be tested into the optimized policy network to generate data instructions for each device; controlling the corresponding device according to each data instruction, monitoring the control process of each device to obtain deviations, and optimizing the policy parameters of each device based on the deviations.

2. The privacy-preserving instruction optimization method according to claim 1, characterized in that, The device information includes action information and status information. The process of obtaining device information for each device in the topological device network, and constructing a local observation vector for each device based on the neighbor relationships between each device and the corresponding device information, specifically includes: obtaining the action information and status information of each device in the topological device network; determining all neighbor devices of each device, obtaining the neighbor relationships of each device and the corresponding action information and status information of all neighbor devices; constructing and optimizing an initial observation vector based on the action information and status information of each device, as well as the action information and status information of all neighbor devices corresponding to the device, to obtain the local observation vector for each device.

3. The privacy-preserving instruction optimization method according to claim 2, characterized in that, The step of constructing and optimizing an initial observation vector based on the action information and state information of each device, as well as the action information and state information of all neighboring devices corresponding to that device, to obtain a local observation vector for each device, specifically includes: for each device, constructing an initial observation vector based on the action information and state information, as well as the action information and state information of all neighboring devices: ;in, This represents the initial observation vector of the i-th device at time t. This represents the state of device j at time t. This indicates the first-layer range of neighboring devices. Indicates twice the value of device i The set of neighboring devices within the range, This represents the action of device k at time t-1. Represents device i The set of neighboring devices within the range; adding preset noise to the initial observation vector to perturb the initial observation vector, thereby obtaining the local observation vector of the device: ； ;in, This represents the local observation vector of the i-th device at time t. Indicates preset noise. express Follows a Laplace distribution. Indicates the scale parameter. The privacy budget parameter represents differential privacy.

4. The privacy-preserving instruction optimization method according to claim 1, characterized in that, The step of inputting each local observation vector into a policy network for feature extraction to obtain the policy action corresponding to each device specifically includes: for each device, obtaining the policy parameters of the device and the parameter estimates of each neighboring device; obtaining the constraint violation status of all neighboring devices of the device, and adjusting the policy parameters of the device according to the constraint violation status; inputting the local observation vector of the device into the policy network, extracting features from the local observation vector according to the current policy parameters of the device and all the parameter estimates, obtaining the probability corresponding to the policy action, and outputting: ;in, Indicates that device i is in Make an action The probability, This represents the action of device i at time t. This represents the local observation vector of the i-th device at time t. This represents the current policy parameters for device i. express eigenvectors, This represents the parameter estimate of neighboring device j. express eigenvectors, For device i The set of neighboring devices within the range, Let A represent all possible actions for device i, and let A represent the set of possible actions. This represents the state of device i at time t. This represents the state of neighbor device j at time t.

5. The privacy-preserving instruction optimization method according to claim 4, characterized in that, The step of obtaining constraint violation information of all neighboring devices of the device and adjusting the policy parameters of the device based on the constraint violation information specifically includes: calculating the constraint violation status of the device at the current moment based on the action information and state information of all neighboring devices. ;in, This indicates the constraint violation status of device i at time t. Represents device i The surrounding neighboring equipment set, Indicates the action-state of neighboring device k. This represents the action of neighboring device k at time t. This represents the state of neighboring device k at time t. This indicates a violation of the threshold. The norm of the gradient of device i is represented; the relationship between the constraint violation and the violation threshold is determined, and if the constraint violation is greater than the violation threshold, the policy parameters of the device are adjusted.

6. The privacy-preserving instruction optimization method according to claim 1, characterized in that, The process of encrypting the policy parameters of each device to obtain corresponding encrypted policy parameters, and then sending the encrypted policy parameters, parameter estimates, and action-state of each device to the optimization engine through multiple transmission channels, specifically includes: for each device, adding Gaussian noise to the policy parameters using an encryption function to encrypt the policy parameters, thereby obtaining encrypted policy parameters. ;in, This represents the encryption strategy parameters for device i at time t, where SM9.Enc represents the encryption function. This represents the policy parameters of device i at time t. Indicates Gaussian noise. This represents the variance of the Gaussian noise at time t. The variance of Gaussian noise is represented. The loss function L represents the loss function with respect to the policy parameters. The norm of; the action-state is sent to the encryption engine via the first transmission channel, and the encryption policy parameters and parameter estimates of each device are sent to the encryption engine via the second transmission channel, wherein the parameter estimates represent the estimated values ​​of the policy parameters.

7. The privacy-preserving instruction optimization method according to claim 6, characterized in that, The step of sending the action-state to the encryption engine via a first transmission channel and sending the encryption policy parameters and parameter estimates of each device to the encryption engine via a second transmission channel specifically includes: inputting the action-state into the first transmission channel, mapping the action-state through the first transmission channel, and sending the mapping result to the encryption engine; inputting the parameter estimates into the second transmission channel, mapping the parameter estimates through the second transmission channel, and sending the mapping result to the encryption engine. Where EnvNet represents the first transmission channel and LearnNet represents the second transmission channel. This represents the mapping function in EnvNet, where s represents the state and a represents the action. express The set of neighboring devices within the range, This represents the mapping function in LearnNet. Represents the policy network, Indicates parameter estimation, Indicates a network of topological devices. express The i-th node in the array.

8. The privacy-preserving instruction optimization method according to claim 1, characterized in that, For each device, the optimization engine optimizes the policy network based on the encryption policy parameters, and the local observation vector to be tested is input into the optimized policy network to generate data instructions for each device. Specifically, this includes: for each device, the optimization engine calculates parameter estimates for all neighboring devices, and performs weighted fusion on each parameter estimate to update the encryption policy parameters of the device, obtaining the updated policy parameters for the next time step. ； ;in, This represents the weighted fusion result at time t+1. Let L represent the set of neighboring device nodes of device i, and let L represent the loss function of the current policy network. This represents the weight between device i and its neighbor device j. This represents the weighted fusion result at time t. This represents the update strategy parameters for device i at time t+1. Let represent the policy parameters of device i at time t; for each device's policy network, calculate the truncated Q-values ​​of all neighboring devices, and use these truncated Q-values ​​to replace the global gradients in the device's policy network: ;in, Let E represent the global gradient of the policy network for device i, and let E represent the expectation operation. Indicates that device i is in The set of neighboring devices within the range, This indicates that the Q value is truncated. This represents the error term, and k represents the neighboring device. This represents the process of approximating the global gradient using a truncated Q-value; the policy network is alternately updated using the updated policy parameters and Lagrange multipliers, and the updated policy parameters are projected into the parameter space: ;in, This represents the policy parameters of device i at time t+1. The parameter space representing the projection. This represents the policy parameters of device i at time t. The learning rate represents the policy parameter. Represents the Lagrange multipliers. This represents the constraint parameters of device i at time t. This represents the constraint violation amount of device i. This represents the constraint parameters of device i at time t+1. Represents the maximum constraint parameter. This represents the constraint parameters at time t. The learning rate represents the constraint parameter. This represents the constraint function of device i based on the policy parameters. The constraint threshold of device i is represented; the local observation vector to be measured of the device is obtained, and the local observation vector is input into the optimized policy network to output the data command of the device at the next time step of the time step to be measured.

9. The privacy-preserving instruction optimization method according to claim 8, characterized in that, The process of controlling the corresponding device according to each data instruction, monitoring the control process of each device to obtain deviations, and optimizing the strategy parameters of each device based on the deviations specifically includes: for each device, inputting the data instruction into the device, and calculating the deviation of the device's adjustment based on the data instruction. ;in, This indicates the deviation of device i at time t. This represents the parameter estimate of device i for its neighbor device j. Indicates an indicator function, Indicates the measurement parameters of device i. This represents the median of the parameter estimates for the neighboring device set. This represents the parameter estimate of neighboring device j. This indicates a deviation from the threshold; if the deviation exceeds the threshold, the current policy parameter corresponding to the device is defined as an abnormal parameter, and the abnormal parameter is restored to the initial policy parameter of the previous moment, so as to re-optimize the initial policy parameter.

10. A privacy-preserving instruction optimization system, characterized in that, The privacy-preserving instruction optimization system is used to implement the privacy-preserving instruction optimization method as described in any one of claims 1-9, comprising: a data acquisition module, used to acquire device information of each device in the topology device network, and construct a local observation vector for each device based on the neighbor relationship between each device and the corresponding device information; an action prediction module, used to input each local observation vector into a policy network for feature extraction to obtain a policy action corresponding to each device; a parameter transmission module, used to encrypt the policy parameters of each device to obtain corresponding encrypted policy parameters, and send the encrypted policy parameters, parameter estimates, and action-state of each device to an optimization engine through multiple transmission channels; an instruction generation module, used to optimize the policy network for each device based on the encrypted policy parameters through the optimization engine, and input the local observation vector to be tested into the optimized policy network to generate a data instruction for each device; and a closed-loop optimization module, used to control the corresponding device according to each data instruction, monitor the control process of each device to obtain deviations, and optimize the policy parameters of each device based on the deviations.

11. The privacy-preserving instruction optimization system according to claim 10, characterized in that, The data acquisition module includes: a device information acquisition unit, used to acquire the action information and status information of each device in the topological device network; a neighbor information acquisition unit, used to determine all neighbor devices of each device, obtain the neighbor relationship of each device and the corresponding action information and status information of all neighbor devices; and a vector construction unit, used to construct an initial observation vector and optimize it based on the action information and status information of each device, as well as the action information and status information of all neighbor devices corresponding to the device, to obtain the local observation vector of each device.

12. The privacy-preserving instruction optimization system according to claim 11, characterized in that, The vector construction unit includes an initial vector construction subunit, used to construct an initial observation vector for each device based on the action information and the state information, as well as the action information and state information of all neighboring devices. ;in, This represents the initial observation vector of the i-th device at time t. This represents the state of device j at time t. This indicates the first-layer range of neighboring devices. Indicates twice the value of device i The set of neighboring devices within the range, This represents the action of device k at time t-1. Represents device i A set of neighboring devices within the range; a local vector construction subunit, used to add preset noise to the initial observation vector to perturb the initial observation vector, thereby obtaining the local observation vector of the device: ； ;in, This represents the local observation vector of the i-th device at time t. Indicates preset noise. express Follows a Laplace distribution. Indicates the scale parameter. The privacy budget parameter represents differential privacy.

13. The privacy-preserving instruction optimization system according to claim 10, characterized in that, The action prediction module includes: a neighbor parameter acquisition unit, used to acquire the policy parameters of each device and parameter estimates of each neighbor device for each device; a parameter adjustment unit, used to acquire the constraint violation status of all neighbor devices of the device and adjust the policy parameters of the device according to the constraint violation status; and a probability calculation unit, used to input the local observation vector of the device into the policy network, extract features from the local observation vector according to the current policy parameters of the device and all parameter estimates, obtain the probability corresponding to the policy action, and output: ;in, Indicates that device i is in Make an action The probability, This represents the action of device i at time t. This represents the local observation vector of the i-th device at time t. This represents the current policy parameters for device i. express eigenvectors, This represents the parameter estimate of neighboring device j. express eigenvectors, For device i The set of neighboring devices within the range, Let A represent all possible actions for device i, and let A represent the set of possible actions. This represents the state of device i at time t. This represents the state of neighbor device j at time t.

14. The privacy-preserving instruction optimization system according to claim 13, characterized in that, The parameter adjustment unit includes a constraint calculation subunit, used to calculate the constraint violation status of the device at the current moment based on the action information and status information of all neighboring devices. ;in, This indicates the constraint violation status of device i at time t. Represents device i The surrounding neighboring equipment set, Indicates the action-state of neighboring device k. This represents the action of neighboring device k at time t. This represents the state of neighboring device k at time t. This indicates a violation of the threshold. The norm of the gradient of device i is represented; the constraint analysis subunit is used to determine the relationship between the constraint violation and the violation threshold. If the constraint violation is greater than the violation threshold, the policy parameters of the device are adjusted.

15. The privacy-preserving instruction optimization system according to claim 10, characterized in that, The parameter transmission module includes a parameter encryption unit, used to encrypt the policy parameters for each device by superimposing Gaussian noise onto them using an encryption function, thereby obtaining encrypted policy parameters. ;in, This represents the encryption strategy parameters for device i at time t, where SM9.Enc represents the encryption function. This represents the policy parameters of device i at time t. Indicates Gaussian noise. This represents the variance of the Gaussian noise at time t. The variance of Gaussian noise is represented. The loss function L represents the loss function with respect to the policy parameters. The norm of; a data transmission unit, used to send the action-state to the encryption engine through a first transmission channel, and to send the encryption policy parameters and parameter estimates of each device to the encryption engine through a second transmission channel, wherein the parameter estimates represent the estimated values ​​of the policy parameters.

16. The privacy-preserving instruction optimization system according to claim 15, characterized in that, The data transmission unit includes: a first information transmission subunit, configured to input the action-state into a first transmission channel, map the action-state through the first transmission channel, and send the mapping result to the encryption engine; and a second information transmission subunit, configured to input the parameter estimate into a second transmission channel, map the parameter estimate through the second transmission channel, and send the mapping result to the encryption engine. Where EnvNet represents the first transmission channel and LearnNet represents the second transmission channel. This represents the mapping function in EnvNet, where s represents the state and a represents the action. express The set of neighboring devices within the range, This represents the mapping function in LearnNet. Represents the policy network, Indicates parameter estimation, Indicates a network of topological devices. express The i-th node in the array.

17. The privacy-preserving instruction optimization system according to claim 10, characterized in that, The instruction generation module includes a parameter optimization unit, used to calculate parameter estimates for all neighboring devices using the optimization engine for each device, and to perform weighted fusion of each parameter estimate to update the encryption policy parameters of the device, thereby obtaining the updated policy parameters for the next time step. ； ;in, This represents the weighted fusion result at time t+1. Let L represent the set of neighboring device nodes of device i, and let L represent the loss function of the current policy network. This represents the weight between device i and its neighbor device j. This represents the weighted fusion result at time t. This represents the update strategy parameters for device i at time t+1. Let represent the policy parameters of device i at time t; the gradient substitution unit is used to calculate the truncated Q-values ​​of all neighboring devices for each device's policy network, and use the truncated Q-values ​​to replace the global gradients in the device's policy network: ;in, Let E represent the global gradient of the policy network for device i, and let E represent the expectation operation. Indicates that device i is in The set of neighboring devices within the range, This indicates that the Q value is truncated. This represents the error term, and k represents the neighboring device. This represents the process of approximating the global gradient using a truncated Q-value; the parameter mapping unit is used to alternately update the policy network using the updated policy parameters and Lagrange multipliers, and project the updated policy parameters into the parameter space: ;in, This represents the policy parameters of device i at time t+1. The parameter space representing the projection. This represents the policy parameters of device i at time t. The learning rate represents the policy parameter. Represents the Lagrange multipliers. This represents the constraint parameters of device i at time t. This represents the constraint violation amount of device i. This represents the constraint parameters of device i at time t+1. Represents the maximum constraint parameter. This represents the constraint parameters at time t. The learning rate represents the constraint parameter. This represents the constraint function of device i based on the policy parameters. The constraint threshold of device i is represented; the instruction construction unit is used to obtain the local observation vector to be measured of the device, input the local observation vector into the optimized policy network, and output the data instruction of the device at the next time step of the time step to be measured.

18. The privacy-preserving instruction optimization system according to claim 10, characterized in that, The closed-loop optimization module includes a deviation analysis unit, used to input the data instruction into each device and calculate the deviation of the device's adjustment based on the data instruction. ;in, This indicates the deviation of device i at time t. This represents the parameter estimate of device i for its neighbor device j. Indicates an indicator function, Indicates the measurement parameters of device i. This represents the median of the parameter estimates for the neighboring device set. This represents the parameter estimate of neighboring device j. The deviation threshold is indicated; the network optimization unit is used to define the current policy parameter corresponding to the device as an abnormal parameter if the deviation exceeds the threshold, and restore the abnormal parameter to the initial policy parameter at the previous moment, so as to re-optimize the initial policy parameter.

19. A terminal, characterized in that, The terminal includes: a memory, a processor, and a privacy-preserving instruction optimization program stored in the memory and executable on the processor, wherein when the privacy-preserving instruction optimization program is executed by the processor, it implements the steps of the privacy-preserving instruction optimization method as described in any one of claims 1-9.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a privacy-preserving instruction optimization program, which, when executed by a processor, implements the steps of the privacy-preserving instruction optimization method as described in any one of claims 1-9.

Images