Decision model training method and device oriented to privacy protection and based on reinforcement learning

Abstract

The invention discloses a privacy protection-oriented decision model training method and device based on reinforcement learning, and the method comprises the steps: carrying out the offline pre-training of a decision model in a federal learning mode through each client based on local historical interaction data, the decision model is decoupled into an embedded layer and an output layer which are deployed on a client and an intermediate computing layer deployed on a computing server; each client performs asymmetric reward training on the Critic network in a federated learning mode based on local historical interaction data, and the Critic network adopts a decoupling structure which is the same as that of a decision model in the offline pre-training stage; and each client performs collaborative decision-making and joint optimization on the decision-making model after offline pre-training and the Critic network after asymmetric reward training under a reinforcement learning framework based on online user interaction data to obtain a final decision-making model. The method can achieve the consideration of model performance and privacy protection, and is suitable for various application scenes such as a recommendation system and automatic driving.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and apparatus for training a decision model based on reinforcement learning with privacy protection in mind. Background Technology

[0002] In recent years, reinforcement learning-based decision-making models have attracted widespread attention in many fields, especially in recommender systems, due to their ability to dynamically learn user preferences. These systems typically model the user sequence decision-making process as a Markov decision process, continuously optimizing decision strategies using user interaction data, demonstrating good long-term utility performance. Taking recommender system decision-making as an example, the Decision Transformer-based Recommendation System (DTRS) significantly improves personalized recommendation performance by introducing a state-action-reward modeling mechanism.

[0003] Because online user interaction data is needed for personalized model updates to adapt to users' ever-changing interests and preferences, real-time online user feedback data is extremely scarce. To address this issue, existing methods employ various strategies. One approach is pre-training followed by fine-tuning, where the model is first pre-trained based on existing data and then fine-tuned online. Another approach utilizes historical data to build interactive simulators that simulate user behavior, providing data support for model training. Furthermore, some research applies reinforcement learning to decision-making systems, such as recommendation systems, and introduces techniques like sequence modeling and simulated user interaction to improve system performance.

[0004] However, existing technologies still have many shortcomings. First, they lack effective privacy protection mechanisms. Methods like DTRS directly use user interaction data during model training. In the online fine-tuning phase, this direct use of user data poses an extremely high risk of user privacy leakage. Sensitive user information, such as browsing history and purchasing preferences, may be improperly obtained and used during model training and fine-tuning.

[0005] Secondly, it is vulnerable to poisoning attacks. Attackers can construct forged interaction data sequences and exploit the temporal characteristics of reinforcement learning models to mislead the decision-making system. This not only reduces the system's robustness and biases the recommendation results, but may also lead to the leakage of sensitive information, causing potential losses to users. For example, some poisoning attack methods train shadow models and poisoning models, generate poisoning data based on specific strategies, and attack the target model, seriously affecting the normal operation of the system.

[0006] Finally, security and performance are often mutually exclusive. In some scenarios, data is distributed across multiple nodes, and the decision-making system needs to ensure the security of data from all parties while maintaining good personalization. However, existing technologies often struggle to balance these two requirements. Strengthening data security may sacrifice some model performance, leading to a decline in personalized recommendation effectiveness, while pursuing high performance may neglect data security, increasing the risk of data leakage.

[0007] In summary, existing reinforcement learning-based decision-making models have significant shortcomings in terms of privacy protection, attack resistance, and the balance between security and performance. There is an urgent need for a technical solution that can take into account both decision performance and privacy protection. Summary of the Invention

[0008] In view of the above, the purpose of this invention is to provide a method and apparatus for training a reinforcement learning-based decision model with privacy protection, which enables collaborative training of the model without directly transmitting the user's original data, and suppresses potential attack risks by injecting differential privacy noise, thereby improving decision performance while effectively protecting the privacy and security of user feedback data. It is applicable to various application scenarios such as recommendation systems and autonomous driving.

[0009] To achieve the above-mentioned objectives, the present invention provides the following technical solution: In a first aspect, embodiments of the present invention provide a privacy-preserving method for training a reinforcement learning-based decision model, comprising the following steps: On each client, the decision model is pre-trained offline using federated learning based on local historical interaction data. The decision model is decoupled into an embedding layer and an output layer deployed on the client and an intermediate computing layer deployed on the computing server. On each client, the Critic network is trained with asymmetric rewards using a federated learning approach based on local historical interaction data. The Critic network adopts the same decoupled structure as the decision model in the offline pre-training stage. Based on online user interaction data on each client, collaborative decision-making and joint optimization are performed on the offline pre-trained decision model and the Critic network trained with asymmetric rewards under the reinforcement learning framework, and finally the online fine-tuned decision model is obtained.

[0010] Preferably, during the offline pre-training, asymmetric reward training, and online fine-tuning phases, data interactions between each client and the computing server, as well as the federated server in federated learning, are protected by differential privacy.

[0011] Preferably, when pre-training the decision model offline, the optimization objective is to maximize the following objective reward function. : , in, The parameters representing the decision model, Expressing expectations, These represent the action, state, and expected reward value, respectively. This represents the distribution of the sampled trajectories. Indicates the total number of samples. Indicates the sampling index. It represents the probability of producing a corresponding action under given conditions.

[0012] Preferably, when training the Critic network with asymmetric rewards, the optimization objective is to minimize the following objective loss function. : , in, The parameters representing the Critic network, This represents the asymmetry coefficient, used to control the bias of the Critic network when learning the reward distribution. Represents the identity matrix. Indicates the target return. , Let these represent the mean and standard deviation of the Critic network predictions, respectively. Representing trajectory, state, and action respectively. Indicates a time step. This represents the probability of generating the corresponding benefit under given conditions.

[0013] Preferably, in the forward inference stage of each training process, the data is embedded into an embedding vector through the client's embedding layer, then Gaussian noise is added for perturbation using differential privacy, and the data is uploaded to the intermediate computing layer of the computing server. The intermediate computing layer calculates the intermediate representation, then adds Gaussian noise for perturbation using differential privacy, and returns it to the client's output layer. The output layer obtains the inference result or loss value. During the backpropagation phase of each training process, the client's output layer first calculates the gradient, then the server calculates the gradient of the loss value relative to its intermediate calculation layer. This gradient is then perturbed by adding Gaussian noise using differential privacy and returned to the client. Based on the received protected gradient, the client further calculates the gradient of the loss relative to the embedding layer using the chain rule, thereby completing the update of the model parameters.

[0014] Preferably, during online fine-tuning, the optimization objective is to minimize the following objective loss function. : , in, This represents minimizing the target reward function during offline pre-training. , The parameters representing the decision model, This represents the target loss function during asymmetric reward training. The parameters representing the Critic network, and Indicates hyperparameters; Represents the maximum entropy regularization term: , in, Expressing expectations, These represent the action, state, and expected reward value, respectively. This represents the distribution of the sampled trajectories. Indicates the total number of samples. Indicates the sampling index. Represents the entropy function. This represents the probability of performing a corresponding action under given conditions. This represents the trajectory-level entropy.

[0015] Preferably, during online fine-tuning, each client optimizes the model parameters through asynchronous updates, as follows: , in, This represents the updated model parameters. Indicates the first The global model parameters are obtained by aggregating the model parameters of each client in a round of federated learning. This represents the learning rate of the federated servers in federated learning. Indicates the client Model parameters.

[0016] Secondly, embodiments of the present invention also provide a privacy-preserving reinforcement learning-based decision model training device, which is implemented using the above-mentioned privacy-preserving reinforcement learning-based decision model training method, including: an offline pre-training module, an asymmetric reward training module, and an online fine-tuning module; The offline pre-training module is used to pre-train the decision model offline on each client based on local historical interaction data in a federated learning manner. The decision model is decoupled into an embedding layer and an output layer deployed on the client and an intermediate computing layer deployed on the computing server. The asymmetric reward training module is used to train the Critic network with asymmetric rewards on each client based on local historical interaction data in a federated learning manner. The Critic network adopts the same decoupled structure as the decision model in the offline pre-training stage. The online fine-tuning module is used on each client to perform collaborative decision-making and joint optimization on the offline pre-trained decision model and the Critic network trained with asymmetric rewards under the reinforcement learning framework, based on online user interaction data, and finally obtain the online fine-tuned decision model.

[0017] Thirdly, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described privacy-preserving reinforcement learning-based decision model training method when executing the computer program.

[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described privacy-preserving reinforcement learning-based decision model training method.

[0019] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) This invention constructs a three-level architecture of client-computing server-federated server and combines bidirectional differential privacy protection in the forward inference and backward propagation process to achieve end-to-end privacy protection covering the entire life cycle of data, representation, gradient and model parameters. It is applicable to decision systems based on reinforcement learning and achieves policy optimization and reward modeling locally on the client for the first time. It controls the risk of systemic privacy leakage from the source and meets strict privacy protection requirements while ensuring model performance.

[0020] (2) This invention decouples the model by adopting a split federated learning architecture and adopts a training strategy that combines offline pre-training with synchronous federated aggregation of asymmetric reward training and asynchronous federated updates in the online fine-tuning stage. This achieves an effective balance between global knowledge sharing and local personalized learning, which not only utilizes the collective wisdom of federated learning but also adapts to the dynamic changes in user preferences. At the same time, it significantly reduces communication overhead and improves the overall training efficiency and scalability of the system.

[0021] (3) By placing sensitive computations (such as policy updates of decision-making models and reward modeling of Critic networks) locally on the client and implementing controlled Gaussian noise perturbation only during necessary intermediate model interactions, this invention achieves a fine trade-off between privacy protection strength and model availability, overcomes the defect of excessive noise addition leading to significant degradation of model performance in traditional methods, and achieves better model utility with a controllable privacy budget. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a privacy-preserving, reinforcement learning-based decision model training method provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of the process framework for the offline pre-training stage and the asymmetric reward training stage provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the process framework for the online fine-tuning stage provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a privacy-preserving decision model training device based on reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0025] The inventive concept of this invention is as follows: Addressing the problems of existing reinforcement learning-based decision-making models in terms of privacy protection, attack resistance, and the balance between security and performance, this invention trains reinforcement learning-based decision-making models by integrating federated learning and differential privacy mechanisms. The former, through local modeling, divides the model training process into server-side general representation learning and client-side local policy optimization, avoiding the centralization of the original dataset. The latter, through a noise mechanism, protects against disturbances during model updates and representation transmission, providing user privacy protection. Furthermore, by combining offline pre-training, asymmetric reward training, and online fine-tuning strategies, this invention achieves a comprehensive balance between privacy and security protection and improved decision-making performance, while also enhancing the system's robustness against malicious attacks.

[0026] In this embodiment, using a recommendation scenario as an example of a decision-making system, the present invention first divides the entire training lifecycle into three stages: offline pre-training, asymmetric reward training, and online fine-tuning. The first two stages need to be performed before the online fine-tuning stage. To protect privacy, the present invention performs inference and training under a federated learning paradigm, and decomposes the server into a computing server and a federated server. For efficient computation, the decision model of the present invention adopts a Decision Transformer (DT) model, and the DT model is decoupled. In the decoupled structure, the embedding layer and output layer deployed on the client are the Embedding layer and Head layer, respectively, and the intermediate computing layer deployed on the computing server is the Transformer layer. Furthermore, sensitive information interactions between the client and the computing server and the federated server are protected through differential privacy technology. It is worth mentioning that the present invention uses this decoupled mode in both the offline pre-training stage and the asymmetric reward training stage. The main difference between the two, besides the different optimization objectives, is that the model obtained in the asymmetric reward training stage is used as a guidance tool in the online fine-tuning stage, so that the online fine-tuning can be better aligned with the user's interests and preferences.

[0027] like Figure 1 As shown in the embodiment, a method for training a privacy-preserving reinforcement learning-based decision model is provided, including the following steps: S1, On each client, the decision model is pre-trained offline using federated learning based on local historical interaction data. The decision model is decoupled into an embedding layer and an output layer deployed on the client and an intermediate computing layer deployed on the computing server.

[0028] In this embodiment, each client performs offline pre-training based on local historical interaction data. Since this is within the DTRS context, the historical interaction data is treated as multiple trajectories, each trajectory consisting of multiple triples, and the content of each triple is determined by the state at that time. ,Behavior and expected reward value composition, This represents a time step. In other words, it represents a trajectory. It is represented as: , in, This indicates the total time step.

[0029] Based on several pre-collected user interaction trajectories, the optimization objective of offline pre-training is to maximize the following objective reward function. To learn a good policy network (i.e., a decision model): , in, The parameters representing the decision model, Expressing expectations, These represent the action, state, and expected reward value, respectively. This represents the distribution of the sampled trajectories. Indicates the total number of samples. Indicates the sampling index. It represents the probability of producing a corresponding action under given conditions.

[0030] Based on this, each client trains its own embedding and head layers to capture personalized user behaviors, such as... Figure 2 As shown.

[0031] First, during the forward inference phase, the data obtains the corresponding embedding vector through the client's embedding layer. (such as Token Embedding and Position Embedding), and then upload the embedding vector, which has been perturbed by Gaussian noise using differential privacy, to the computing server. The computing server further uses its Transformer layer to compute intermediate representations. Accordingly, this intermediate representation It also needs to be perturbed by adding Gaussian noise using differential privacy before being returned to the client. Finally, the inference result or loss value is obtained by the Head layer on the client. The formal expression of adding Gaussian noise is as follows: , in, This indicates that the mean is 0 and the covariance matrix is... The Gaussian distribution.

[0032] Then, during the backpropagation phase, based on the obtained loss value, the client and the computation server need to jointly perform a backpropagation process to update the model parameters. The most crucial aspect is that the chain rule of backpropagation needs to pass through the Transformer layer of the computation server. The loss value is represented by the gradient of the client's embedding layer, which is calculated as follows: , in, This represents the gradient received by the computation server, protected by differential privacy technology.

[0033] Finally, after training a batch of samples to obtain the parameters of the decision model, the various clients need to aggregate the model parameters. At this point, each client begins to communicate with the federated server, uploading differentially privacy-protected model weight information to the federated server. After receiving the weight information from all participating clients, the federated server begins to aggregate and update the parameters, as follows:

[0034] in, Indicates the current client The number of training samples available. This represents the total number of training samples owned by all clients. Indicates the total number of clients. Indicates the first The weight parameters of the global model in the round. After the federated server performs weighted aggregation in this way, it needs to output the weight parameters of the resulting global model. The training is redistributed to each client, and a new round of training begins until convergence or the maximum number of training rounds is reached.

[0035] In recommender system scenarios, after offline pre-training in this stage, a basic recommendation decision model can be obtained, which can effectively capture users' personalized interests and behavioral patterns, has the ability to make recommendations based on context state and reward objectives, and achieves good generalization through federated aggregation, providing a high-performance initialization strategy for the subsequent online fine-tuning stage.

[0036] S2, on each client, based on local historical interaction data, the Critic network is trained with asymmetric rewards using a federated learning approach. The Critic network adopts the same decoupled structure as the decision model in the offline pre-training stage.

[0037] In this embodiment, to better capture user interests and preferences, the present invention uses a Critic network to more accurately estimate the cumulative revenue that an action (i.e., the recommendation strategy in this scenario) can bring in the future. According to Bayes' theorem, the probability of an action under given conditions... It can be decomposed into the following forms: , In the known In this case, this reflects the assessment To emphasize its importance, this invention uses a Gaussian critic to approximate the reward distribution. Specifically, it combines differential privacy and federated learning strategies to train the critic network with asymmetric rewards, aiming to minimize the following objective loss function. :

[0038] in, The parameters representing the Critic network, This represents the asymmetry coefficient, used to control the bias of the Critic network when learning the reward distribution. Represents the identity matrix. , Let these represent the mean and standard deviation of the Critic network predictions, respectively. Representing trajectory, state, and action respectively. This represents the probability of generating the corresponding benefit under given conditions.

[0039] Based on this optimization objective, each client trains a decision model structurally equivalent to the offline pre-training stage, enabling the evaluation of reward distribution from user interaction data. Similarly, this invention places the Embedding and Head layers of the Critic network in the client, while placing the Transformer layer in the computation server. The federated server is also used to aggregate the weight parameters from multiple clients, such as... Figure 2 As shown.

[0040] From the perspective of data flow, the training data also first passes through the embedding layer located on the client side to obtain the embedding vector. Gaussian noise is then added, and the data is transmitted to the Transformer layer located in the computing server to obtain the corresponding intermediate representation. This intermediate representation is protected using the same differential privacy mechanism before being transmitted back to the client's Head layer to compute the result or loss value. The backpropagation stage also uses the same strategy as the offline pre-training stage.

[0041] After each training round, the client needs to transmit weight information with Gaussian noise added to it to the federated server. The federated server receives the weight information from all participating clients and aggregates it using the FedAvg federated learning algorithm to obtain the global model's weight parameters. Finally, it sends this data back to each client to update the model. This process is repeated multiple times until the model converges or reaches the maximum number of training rounds, resulting in a trained Critic network. This network is then used for auxiliary training during the online fine-tuning phase, providing value guidance for the decision-making model.

[0042] S3, based on online user interaction data on various clients, performs collaborative decision-making and joint optimization on the offline pre-trained decision model and the Critic network trained with asymmetric rewards under the reinforcement learning framework, and finally obtains the online fine-tuned decision model.

[0043] In this embodiment, to further improve the long-term benefits of the recommendation system, achieve online personalized dynamic adaptation of the model, and effectively balance exploration and exploitation strategies, this invention, based on online user interaction data, performs collaborative decision-making and joint optimization on the offline pre-trained decision model and the Critic network trained with asymmetric rewards within a reinforcement learning framework. Figure 3 As shown.

[0044] First, the client needs to interact with the user in real time, making decisions based on the decision model obtained in the offline pre-training phase, for example, at time steps. The agent observes And select an action in the following way: , in, Indicates the state Next action Long-term benefits assessed in a timely manner. It is the distribution of behavior based on the decision model. The agent executes... After that, the environment returned. And the agent is updated: , in, Indicates the first The reward value for each step.

[0045] In addition to making decisions, the client also needs to save new interaction records to a buffer for continuous fine-tuning of training. The training model at this stage includes both offline pre-trained decision models and Critic networks that evaluate rewards during the asymmetric reward training stage.

[0046] To encourage exploration and prevent premature convergence, this invention introduces a maximum entropy regularization term into the optimization objective: , in, This represents the entropy function, used to encourage diversity in action sampling. This represents the trajectory-level entropy, which is the entropy obtained by aggregating the entire trajectory segment of length K.

[0047] Therefore, the overall optimization objective during online fine-tuning is to minimize the following objective loss function. : , in, and These represent hyperparameters used to adjust the intensity of exploration and exploitation, as well as the degree of influence of the asymmetric negative log-likelihood loss.

[0048] In terms of data flow, the forward inference stage maintains the same pattern as the offline pre-training stage, but the inference results need to be stored in a buffer specific to this stage. In the backpropagation stage, the loss function is calculated using the decision models from the first two stages and the Critic network. During frequent data interactions between the local client and the same computing server, Gaussian noise is added to protect data privacy. Finally, to enable timely updates to the global model between the local client and the federated server without waiting for all clients to upload their models, this invention uses an asynchronous update method, expressed as follows: , in, This represents the updated model parameters. Indicates the first The global model parameters are obtained by aggregating the model parameters of each client in a round of federated learning. This represents the learning rate of the federated servers. Indicates the client The model parameters are updated asynchronously. This asynchronous update strategy effectively improves the responsiveness of the decision model in dynamic environments and supports continuous model optimization without the need for global synchronization.

[0049] After online fine-tuning, a final decision model optimized with privacy protection mechanisms is obtained, which provides high-value personalized services to specific users and can be directly deployed in online real-time recommendation scenarios.

[0050] Based on the same inventive concept, such as Figure 4 As shown, this embodiment of the invention also provides a privacy-preserving reinforcement learning-based decision model training device 400, including: an offline pre-training module 410, an asymmetric reward training module 420, and an online fine-tuning module 430.

[0051] The offline pre-training module 410 is used to pre-train the decision model offline on each client based on local historical interaction data in a federated learning manner. The decision model is decoupled into an embedding layer and an output layer deployed on the client and an intermediate computing layer deployed on the computing server.

[0052] The asymmetric reward training module 420 is used to train the Critic network with asymmetric rewards on each client in a federated learning manner based on local historical interaction data. The Critic network adopts the same decoupled structure as the decision model in the offline pre-training stage.

[0053] The online fine-tuning module 430 is used on various clients to perform collaborative decision-making and joint optimization on the offline pre-trained decision model and the Critic network trained with asymmetric rewards under the reinforcement learning framework, based on online user interaction data, and finally obtain the online fine-tuned decision model.

[0054] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described privacy-preserving reinforcement learning-based decision model training method when executing the computer program.

[0055] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described privacy-preserving reinforcement learning-based decision model training method.

[0056] It should be noted that the privacy-preserving reinforcement learning-based decision model training device, electronic device, and computer-readable storage medium provided in the above embodiments all belong to the same inventive concept as the privacy-preserving reinforcement learning-based decision model training method. For the specific implementation process, please refer to the embodiments of the privacy-preserving reinforcement learning-based decision model training method, which will not be repeated here.

[0057] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A privacy-preserving reinforcement learning-based decision model training method, characterized in that, Includes the following steps: On each client, the decision model is pre-trained offline using federated learning based on local historical interaction data. The decision model is decoupled into an embedding layer and an output layer deployed on the client and an intermediate computing layer deployed on the computing server. On each client, the Critic network is trained with asymmetric rewards using a federated learning approach based on local historical interaction data. The Critic network adopts the same decoupled structure as the decision model in the offline pre-training stage. Based on online user interaction data on each client, collaborative decision-making and joint optimization are performed on the offline pre-trained decision model and the Critic network trained with asymmetric rewards under the reinforcement learning framework, and finally the online fine-tuned decision model is obtained.

2. The method for training a privacy-preserving, reinforcement learning-based decision model according to claim 1, characterized in that, During the offline pre-training, asymmetric reward training, and online fine-tuning phases, data interactions between each client and the computing server, as well as the federated server in federated learning, are protected by differential privacy.

3. The method for training a privacy-preserving, reinforcement learning-based decision model according to claim 1, characterized in that, When pre-training the decision model offline, the optimization objective is to maximize the following objective reward function. : ， in, The parameters representing the decision model, Expressing expectations, These represent the action, state, and expected reward value, respectively. This represents the distribution of the sampled trajectories. Indicates the total number of samples. Indicates the sampling index. It represents the probability of producing a corresponding action under given conditions.

4. The method for training a privacy-preserving, reinforcement learning-based decision model according to claim 1, characterized in that, When training the Critic network with asymmetric rewards, the optimization objective is to minimize the following objective loss function. : ， in, The parameters representing the Critic network, This represents the asymmetry coefficient, used to control the bias of the Critic network when learning the reward distribution. Represents the identity matrix. Indicates the target return. , Let these represent the mean and standard deviation of the Critic network predictions, respectively. Representing trajectory, state, and action respectively. Indicates a time step. This represents the probability of generating the corresponding benefit under given conditions.

5. The method for training a privacy-preserving, reinforcement learning-based decision model according to claim 1 or 2, characterized in that, In the forward inference phase of each training process, the data is embedded into an embedding vector through the client's embedding layer, then Gaussian noise perturbation is added through differential privacy and uploaded to the intermediate computing layer of the computing server. The intermediate computing layer calculates the intermediate representation, then Gaussian noise perturbation is added through differential privacy and returned to the client's output layer. The output layer obtains the inference result or loss value. During the backpropagation phase of each training process, the client's output layer first calculates the gradient, then the server calculates the gradient of the loss value relative to its intermediate calculation layer. This gradient is then perturbed by adding Gaussian noise using differential privacy and returned to the client. Based on the received protected gradient, the client further calculates the gradient of the loss relative to the embedding layer using the chain rule, thereby completing the update of the model parameters.

6. The method for training a privacy-preserving, reinforcement learning-based decision model according to claim 1, characterized in that, When performing online fine-tuning, the optimization objective is to minimize the following objective loss function. : ， in, This represents minimizing the target reward function during offline pre-training. , The parameters representing the decision model, This represents the target loss function during asymmetric reward training. The parameters representing the Critic network, and Indicates hyperparameters; Represents the maximum entropy regularization term: ， in, Expressing expectations, These represent the action, state, and expected reward value, respectively. This represents the distribution of the sampled trajectories. Indicates the total number of samples. Indicates the sampling index. Represents the entropy function. This represents the probability of performing a corresponding action under given conditions. This represents the trajectory-level entropy.

7. The method for training a privacy-preserving, reinforcement learning-based decision model according to claim 1, characterized in that, During online fine-tuning, each client optimizes the model parameters through asynchronous updates, as shown below: ， in, This represents the updated model parameters. Indicates the first The global model parameters are obtained by aggregating the model parameters of each client in a round of federated learning. This represents the learning rate of the federated servers in federated learning. Indicates the client Model parameters.

8. A privacy-preserving reinforcement learning-based decision model training device, implemented using the privacy-preserving reinforcement learning-based decision model training method according to any one of claims 1 to 7, characterized in that, include: Offline pre-training module, asymmetric reward training module, online fine-tuning module; The offline pre-training module is used to pre-train the decision model offline on each client based on local historical interaction data in a federated learning manner. The decision model is decoupled into an embedding layer and an output layer deployed on the client and an intermediate computing layer deployed on the computing server. The asymmetric reward training module is used to train the Critic network with asymmetric rewards on each client based on local historical interaction data in a federated learning manner. The Critic network adopts the same decoupled structure as the decision model in the offline pre-training stage. The online fine-tuning module is used on each client to perform collaborative decision-making and joint optimization on the offline pre-trained decision model and the Critic network trained with asymmetric rewards under the reinforcement learning framework, based on online user interaction data, and finally obtain the online fine-tuned decision model.

9. An electronic device comprising a memory and one or more processors, the memory for storing a computer program, characterized in that, The processor is used to implement the privacy-preserving reinforcement learning-based decision model training method according to any one of claims 1 to 7 when executing a computer program.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the privacy-preserving reinforcement learning-based decision model training method according to any one of claims 1 to 7.

Images