A terminal-cloud collaborative training method for war game simulation platform

Abstract

The present application relates to a method for end-cloud collaborative training for a war game simulation platform. The method includes: using the simulation user end as a collaborative user end, and the simulation server constructing an end-cloud collaborative framework by acting as a cloud server; modeling the end-cloud collaborative framework, modeling the user end as an intelligent agent, and obtaining an end-cloud collaborative model; offline pre-training the end-cloud collaborative model according to the Decider algorithm and a pre-acquired offline data set, and online training the user end according to a subset of offline data, and uploading the trained local intelligent agent model to the cloud server; performing model aggregation on the cloud server to obtain a weighted average model as the global intelligent agent model after the current round of update, and broadcasting it to the user end. The use of this method can improve the training accuracy and efficiency of the war game simulation platform.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a terminal-cloud collaborative training method for a war game simulation platform. Background Art

[0002] With the advancement of the intelligent wave, research in the field of wargaming has attracted great attention from scholars. However, the current intelligent decision-making technology in wargaming still faces challenges such as low sample efficiency, limited generalization migration, and difficulty in model deployment. In addition, traditional knowledge-based or data-driven methods have shortcomings in computational complexity, training speed, and generalization ability, and the application of cloud-based centralized command and control and training models is also limited by communication delays, data security, and model deployment. Summary of the invention

[0003] Based on this, it is necessary to provide an end-cloud collaborative training method for a war game simulation platform that can improve the training accuracy and efficiency of the war game simulation platform in response to the above technical problems.

[0004] A terminal-cloud collaborative training method for a war game simulation platform, the method is applied to a war game simulation platform including a simulation server and a plurality of simulation user terminals, comprising:

[0005] The simulation client is used as the collaborative client, and the simulation server is used as the cloud server. The client shares data and performs collaborative processing with the cloud server through the interface of the cloud server to build a terminal-cloud collaborative framework.

[0006] The end-cloud collaboration framework is modeled, the user end is modeled as an intelligent agent, and the end-cloud collaboration model is obtained;

[0007] The end-cloud collaborative model is pre-trained offline based on the Decider algorithm and the pre-acquired offline data set, and the network parameters of the state value network and the target policy network corresponding to the offline agent and the offline data subset are broadcast to the deduction user end;

[0008] Perform online training on the user end according to the offline data subset and the data collected online by the user end, set the objective function of the action network and the loss function of the evaluation network corresponding to the local agent model on the user end, train the local agent model using the objective function of the action network and the loss function of the evaluation network corresponding to the local agent model, and upload the trained local agent model to the cloud server;

[0009] In the test environment of the cloud server, the model strategy output by the trained local agent model is used to perform performance testing. The model is aggregated according to the accumulated reward values ​​corresponding to different model strategies to obtain a weighted average model, which is used as the global agent model after the current round of update and broadcast to the user end.

[0010] This application proposes a more efficient and secure end-cloud collaborative training method for war game simulation platforms to solve the problem of intelligent development of tactical edge terminals. The framework takes advantage of the powerful computing power of cloud servers and the ease of storing a large number of offline experience samples, adds a hybrid training method of cloud offline reinforcement learning and user-side online reinforcement learning, and innovates the intelligent decision-making path driven by knowledge and data. At the same time, by actively screening high-value offline experience samples on the cloud and sending them to edge terminal devices, the acceleration effect of prior knowledge on the training of intelligent agents is further enhanced, enabling intelligent agents to adapt to online environments more quickly, and effectively reducing the phenomenon of strategy collapse caused by data distribution offset. The end-cloud collaborative training mode adopted by Decider does not require the intelligent agent to transmit and share original samples to the cloud server or between intelligent agents. It has good robustness to different task environments and effectively overcomes the disadvantages of the model being difficult to deploy in practice due to communication delays, data leakage, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of a terminal-cloud collaborative training method for a war game simulation platform in one embodiment;

[0012] Figure 2 Schematic diagram of a terminal-cloud collaborative training framework for a war game simulation platform in one embodiment. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0014] In one embodiment, Figure 1 As shown, a terminal-cloud collaborative training method for a war game simulation platform is provided, which is applied to a war game simulation platform including a simulation server and multiple simulation user terminals, and includes the following steps:

[0015] Step 102, the deduction user end is used as the collaborative user end, the deduction server is used as a cloud server, and the user end shares data and collaborates with the cloud server through the interface of the cloud server to build an end-cloud collaboration framework; the end-cloud collaboration framework is modeled, and the user end is modeled as an intelligent agent to obtain an end-cloud collaboration model.

[0016] Traditional war game simulation methods are often limited by the dispersion of data resources and the lack of computing and storage capabilities, making it impossible to efficiently integrate and summarize the decentralized multi-terminal simulation experience, limiting the efficiency of multi-terminal collaborative training. In order to overcome these challenges, this application proposes a terminal-cloud collaborative training framework for war game simulation platforms, such as Figure 2 As shown in the figure, this framework can be directly embedded in the war game simulation platform, relying on the interactive interface module between the simulation server and the simulation client to realize an efficient training and updating mechanism. As the "computing center", the simulation server carries multiple tasks such as simulation operation, data storage, model calculation and algorithm training to ensure the smoothness and accuracy of the simulation process. The simulation client, as the "information bridge" between the decision-making agent and the simulation server, realizes the accurate interpretation and transmission of instructions such as task setting and entity control order modification through the API interface. At the same time, the client is also responsible for collecting the experience data of the decision-making agent and providing data resources for the training and updating of the model. The design of this framework can not only improve the intelligence level of the war game simulation platform, but also provide strong support for its efficient operation in practical applications.

[0017] This application focuses on the horizontal end-cloud collaborative learning framework, that is, all user ends are in a parallel environment, the user ends are modeled as agents, and the state and action space of each agent are aligned. For similar tasks, each agent independently explores interactions in its own environment and finally obtains the optimal strategy. The framework can be formulated as follows: Agent Respectively in Environment, here the set of environments is represented as .exist The environment Each agent has a model similar to that of the agents in other environments (e.g., the state transition probabilities and reward functions are similar to those in other environments). The environment remains independent of other environments because The state transition and reward model of the environment do not depend on the state and actions of other environments. Each environment corresponds to Interact and learn the optimal strategy. The framework can be formally summarized as:

[0018] ;

[0019] in, and Respectively represent Agent and Similar state spaces observed by agents, and Respectively represent Agent and The action spaces of the two agents are similar. and It is not the same environment, and it is usually assumed that the environments are independent and identically distributed.

[0020] The end-cloud collaborative learning framework of the present application does not need to collect all the data of each participant, thus avoiding the risk that any participant may expose his or her own data to the server or other participants.

[0021] Step 104, pre-train the end-cloud collaborative model offline according to the Decider algorithm and the pre-acquired offline data set, and broadcast the network parameters of the state value network and the target strategy network corresponding to the offline agent and the offline data subset to the deduction user end.

[0022] The end-cloud collaborative reinforcement learning framework includes a cloud server and The Decider algorithm has six main processes, namely, offline pre-training stage, broadcasting offline agents and offline data subsets to the user end, online training stage of the user end, uploading the local agent model of the user end to the cloud server, cloud server model aggregation stage, and broadcasting the global aggregated model to the user end.

[0023] First, in the offline pre-training stage, reinforcement learning training is performed in a complex task environment. Due to the huge search state space and strategy space, and the existence of sparse rewards, delayed returns and other problems, it is very difficult to learn the decisions made many steps ago based on cumulative rewards. In real tasks, a group of expert decision-making process examples can often be obtained, so that the reinforcement learning model can directly imitate the expert's behavior strategy through offline sample learning to alleviate the above difficulties. This step first collects the historical trajectories collected by human experts' prior knowledge or suboptimal decision-making agents to form an offline data set. It is worth noting that the expert experience or agent strategy required for sample collection does not need to be the optimal trajectory. On the contrary, the samples collected by the strategy trained from the randomly initialized model can cover more sample distributions, thereby bringing better training results. This feature can significantly reduce the dependence on human experts' prior knowledge and sample collection costs. Then, the BPPO algorithm and offline data sets are used for offline pre-training. The algorithm trains four network models, namely the state value network, the state-action value network, the behavior clone network and the target strategy network, with four sets of network parameters.

[0024] State value function Training is performed by fitting the cumulative return of the experience sample trajectory as follows:

[0025] ;

[0026] in, express The loss function is represents the number of samples, represents the discount factor, All from offline datasets .

[0027] State-Action Value Function The update formula is:

[0028] ;

[0029] in, represents the estimate of the state-action value, Represents the learning rate, the five-tuple involved in the formula All use offline datasets Experience samples from .

[0030] Behavioral cloning network strategy Through the behavior cloning algorithm, directly based on the offline dataset Interactive experience , imitate the actions of each state in a supervised learning manner:

[0031] ;

[0032] The loss function of the target policy network is:

[0033] ;

[0034] in, Represents the target strategy, the state and action in the formula In offline dataset The sampling is obtained, represents the sampling strategy of offline samples, is the clipping function, Represents the clipping coefficient, and the BPPO algorithm directly uses the definition to calculate the advantage value, that is, ,in, and The functions are solved using state-value networks and state-action-value networks respectively.

[0035] When the cloud server completes the offline reinforcement learning pre-training, it will calculate the network parameters based on the latest state value. , through the priority experience replay module, high-value training samples in the offline data set are screened, and the evaluation index is calculated using the absolute value of the time difference error:

[0036] ;

[0037] Pick The samples with the highest values ​​constitute the offline data subset , Indicates the sampling ratio, offline data subset 、Network parameters of state value network , network parameters of the target policy network Together they constitute the output of the offline pre-training phase of the cloud server.

[0038] Step 106, perform online training on the user terminal based on the offline data subset and the data collected online by the user terminal, set the objective function of the action network and the loss function of the evaluation network corresponding to the local intelligent agent model on the user terminal, use the objective function of the action network and the loss function of the evaluation network corresponding to the local intelligent agent model to train the local intelligent agent model, and upload the trained local intelligent agent model to the cloud server.

[0039] The local training process on the user side mainly includes five modules, namely situation data processing module, online reinforcement learning training module, action decoding module, operation instruction encapsulation module and experience storage module.

[0040] The situation data processing module is used to receive the real-time battlefield situation information sent by the war game server, and clean, filter, extract, package, normalize and format the data. The online reinforcement learning training module receives the situation information data and outputs the action decision. The action decoding module receives the action decision output of the reinforcement learning module, decodes it into the unit to be executed, and converts it into an operation instruction acceptable to the war game platform by the operation instruction encapsulation module to control the corresponding unit. At the same time, the next moment environmental situation and reward value obtained by executing the new action are packaged and stored in the experience storage module together with the environmental situation information and action decision plan of the current decision. When the intelligent agent model is trained, the sample data is extracted and processed and passed to the reinforcement learning training module for online training. It is worth noting that before each round of training on the user side, the policy network parameters and state value network parameters broadcast by the cloud server need to be used to update the local intelligent agent model parameters. At the same time, the prior deduction experience samples in the offline data subset broadcast by the cloud server are also added to the local experience playback library on the user side for subsequent algorithm training.

[0041] The online reinforcement learning training module is the key link of local training on the user side. It adopts the Actor-Critic (AC) framework for strategy training. The Actor network (action network) maximizes the objective function calculate:

[0042] ;

[0043] in, is the action network parameter of the current update strategy, is the Importance Sampling (IS) coefficient, calculated as , Indicates the current update strategy, represents the historical sampling strategy, Estimated in state Take action The advantage value is calculated as:

[0044] ;

[0045] In order to prevent the difference between the historical sampling strategy and the current update strategy from being too large, the importance sampling coefficient is clipped to limit the parameter update range. The clipped objective function is

[0046] ;

[0047] Critic network (evaluation network) parameters according to The error between the function and the cumulative discounted return is updated as the loss function:

[0048] .

[0049] Step 108, in the test environment of the cloud server, the model strategy output by the trained local agent model is used for performance testing, and the model is aggregated according to the accumulated reward values ​​corresponding to different model strategies to obtain a weighted average model as the global agent model updated in the current round, and broadcast to the user end. After the local online training, the user-end agent model needs to be uploaded to the cloud server for model aggregation, so as to achieve knowledge fusion among multiple agents, improve the overall model performance, and accelerate the search speed of the optimal strategy.

[0050] The cloud server uses a synchronous communication mechanism to The parameters of the agent model on the client side All are collected to the server. Afterwards, in the server test environment, the model strategies of different users are used respectively. Perform performance testing, each model test times, and save the cumulative reward value obtained in the entire game .

[0051] In the aggregation phase of the model, firstly, the average score of the agent is calculated based on the model test results:

[0052] ;

[0053] Afterwards, the model parameters of different user terminals are weighted according to the average score of the agents, and the weighted average model parameters are calculated as the global agent model parameters after this round of update:

[0054] ;

[0055] After completing the cloud model aggregation process, the cloud server will broadcast the global aggregation model to all user terminals again and start the next round of training iteration.

[0056] The end-cloud collaborative framework of this application organically combines the characteristics of cloud and terminal decision-making capabilities and data resources, optimizes models and data transmission modes, and compared with traditional distributed reinforcement learning and federated reinforcement learning methods, this framework integrates the advantages of edge computing (end) and cloud computing (cloud), and uses cloud computing resources and real-time interaction experience with end devices to achieve collaborative improvement of end-cloud decision-making capabilities. Compared with traditional centralized training methods, it can achieve more efficient data processing and decision support while ensuring decision accuracy and timeliness. In the end-cloud collaborative intelligent decision-making framework of this article, the original interaction data collected by the terminal device is always saved locally and does not need to be uploaded to the cloud or shared with other devices. This training method effectively avoids the risk of data leakage and abuse, while reducing dependence on device communication frequency and bandwidth, which not only protects the privacy and security of sensitive data, but also improves the practicality of the system.

[0057] On the other hand, the training framework of this application adopts a distributed processing method. The terminal device is responsible for local model training, the cloud device performs global model aggregation, and uses offline reinforcement learning training to achieve prior knowledge fusion. This architecture makes the system more robust. Even if some terminals or cloud nodes fail, it will not affect the normal operation of the entire system. In addition, with the continuous addition of new devices and data, the decision-making ability of the system can be continuously enhanced.

[0058] In the above-mentioned end-cloud collaborative training method for war game simulation platform, the cloud server's powerful computing power and the advantage of easy storage of a large number of offline experience samples are used to add a hybrid training method of cloud offline reinforcement learning and user-side online reinforcement learning, and innovate the intelligent decision-making path driven by knowledge and data. At the same time, by actively screening high-value offline experience samples on the cloud and sending them to edge terminal devices, the acceleration effect of prior knowledge on agent training is further strengthened, enabling agents to adapt to online environments more quickly, effectively reducing the phenomenon of strategy collapse caused by data distribution offset. The end-cloud collaborative training mode adopted by Decider does not require agents to transmit and share original samples to cloud servers or between agents. It has good robustness to different task environments and effectively overcomes the disadvantages of models that are difficult to deploy in practice due to communication delays, data leakage, etc.

[0059] In one embodiment, the end-cloud collaboration model is pre-trained offline according to the Decider algorithm and the pre-acquired offline data set, including:

[0060] According to the pre-set loss function of the state value network of the intelligent agent, the update formula of the state-action value network, the update formula of the behavior cloning network and the loss function of the target policy network, the state value network, state-action value network, behavior cloning network and target policy network of the intelligent agent are trained respectively using the loss functions or update formulas of the four networks to obtain the trained state value network, state-action value network, behavior cloning network and target policy network of the intelligent agent;

[0061] The evaluation index selected by the data set is calculated according to the state value network of the trained intelligent agent, and samples are extracted as offline data subsets according to the preset sampling ratio in the order of the evaluation index from small to large.

[0062] In one embodiment, the loss function of the state value network of the agent is

[0063] ;

[0064] in, represents the discount factor, All from offline datasets , Represents the state of samples in the offline dataset, represents the reward value of the sample in the offline dataset, t represents the sampling time of samples in the offline dataset, represents the state value function, represents the auxiliary variable in the summation process, Indicates the maximum time step of the trajectory corresponding to the sample, Represents the samples in the dataset Seek expectations.

[0065] In one embodiment, the update formula of the state-action value network of the agent is:

[0066] ;

[0067] in, represents the state-action value function, represents the estimate of the state-action value, represents the discount factor, All from offline datasets , Represents the state of samples in the offline dataset, Indicates the action corresponding to the sample state in the offline dataset, represents the learning rate, represents the reward value of the sample in the offline dataset, Represents the sampling time of samples in the offline dataset.

[0068] In one embodiment, the update formula of the agent's behavior cloning network is:

[0069] ;

[0070] in, represents the behavior cloning network strategy, Indicates that when the strategy is When in state Take action under the conditions The probability of Represents the state of samples in the offline dataset, Indicates the action corresponding to the sample state in the offline dataset, Represents the samples in the dataset Seek expectations.

[0071] In one embodiment, the loss function of the agent's target policy network is

[0072] ;

[0073] in, Represents the samples in the offline dataset Seek hope, represents the target strategy, represents the sampling strategy of offline samples, Represents the state of samples in the offline dataset, Indicates the action corresponding to the sample state in the offline dataset, represents the learning rate, represents the reward value of the sample in the offline dataset, t represents the sampling time of samples in the offline dataset, represents the advantage value, which is obtained directly through the generalized advantage estimation, is represented as a clipping function, represents the cropping factor, Indicates that when the strategy is When in state Take action under the conditions probability.

[0074] In one embodiment, the evaluation index selected based on the state value network calculation data set of the trained intelligent agent is:

[0075] ;

[0076] in, represents the state value function, represents the discount factor, Represents the state of samples in the offline dataset, represents the reward value of the sample in the offline dataset, Represents the sampling time of samples in the offline dataset.

[0077] In one embodiment, the objective function of the action network corresponding to the local agent model on the user side is set to

[0078] ;

[0079] in, is the action network parameter of the current update strategy, It comes from the offline data subset and the collection of user-side online collected data. Indicates the current update strategy, represents the historical sampling strategy, Estimated in state Take action The advantage value of represents the state of the samples in the collection, Represents the action corresponding to the sample state in the set, Indicates that when the strategy is When in state Take action under the conditions The probability of Indicates that when the strategy is When in state Take action under the conditions The probability of is represented as a clipping function, Represents the crop factor.

[0080] In one embodiment, the loss function of the evaluation network corresponding to the local agent model on the user side is set to

[0081] ;

[0082] in, represents the network parameters of the evaluation network, represents the discount factor, It comes from the offline data subset and the collection of user-side online collected data. Represents the state of the samples in the collection, represents the reward value of the sample in the set, represents the auxiliary variable in the summation process of the loss function, The network parameters are The state value function of represents the sampling time of the samples in the set, Indicates the maximum time step of the trajectory corresponding to the sample.

[0083] In one embodiment, the model is aggregated according to the accumulated reward values ​​corresponding to different model strategies to obtain a weighted average model, including:

[0084] According to the accumulated reward values ​​corresponding to different model strategies, the model is aggregated to obtain the weighted average model:

[0085] ;

[0086] in, represents the agent model parameters, represents the average score of different agents, Indicates Model strategy for multi-agent models Conduct the The cumulative reward value obtained from the performance test is Represents the model policy of each agent model The total number of performance tests performed, Indicates the total number of clients.

[0087] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0088] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0089] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A terminal-cloud collaborative training method for a war game simulation platform. It is characterized in that The method is applied to a war game simulation platform including a simulation server and a plurality of simulation user terminals, and the method includes: The deduction client is used as a collaborative client, the deduction server is used as a cloud server, and the client shares data and performs collaborative processing with the cloud server through an interface of the cloud server to build a client-cloud collaborative framework; Modeling the end-cloud collaboration framework, modeling the user end as an intelligent agent, and obtaining an end-cloud collaboration model; The end-cloud collaborative model is pre-trained offline according to the Decider algorithm and the pre-acquired offline data set, and the network parameters of the state value network and the target strategy network corresponding to the obtained offline agent and the offline data subset are broadcasted to the deduction user end; Performing online training on the user terminal according to the offline data subset and the data collected online by the user terminal, setting the objective function of the action network and the loss function of the evaluation network corresponding to the local agent model on the user terminal, training the local agent model using the objective function of the action network and the loss function of the evaluation network corresponding to the local agent model, and uploading the trained local agent model to the cloud server; In the test environment of the cloud server, the model strategy output by the trained local agent model is used to perform performance testing, and the model is aggregated according to the accumulated reward values ​​corresponding to different model strategies to obtain a weighted average model as the global agent model after the current round of update, and broadcast to the user end; The end-cloud collaboration model is offline pre-trained according to the Decider algorithm and the pre-acquired offline data set, including: According to the pre-set loss function of the state value network of the intelligent agent, the update formula of the state-action value network, the update formula of the behavior cloning network and the loss function of the target policy network, the state value network, state-action value network, behavior cloning network and target policy network of the intelligent agent are trained respectively using the loss functions or update formulas of the four networks to obtain the trained state value network, state-action value network, behavior cloning network and target policy network of the intelligent agent; The evaluation index selected by the state value network of the trained agent is calculated, and samples are extracted as offline data subsets according to a preset sampling ratio in the order of small to large evaluation indexes; The evaluation index selected according to the state value network calculation data set of the trained intelligent agent is: ； in, represents the state value function, represents the discount factor, Represents the state of samples in the offline dataset, represents the reward value of the sample in the offline dataset, t Represents the sampling time of samples in the offline dataset; The model is aggregated according to the accumulated reward values ​​corresponding to different model strategies to obtain a weighted average model, including: According to the accumulated reward values ​​corresponding to different model strategies, the model is aggregated to obtain the weighted average model: ； in, represents the agent model parameters, represents the average score of different agents, Indicates Model strategy for multi-agent models Conduct the The cumulative reward value obtained from the performance test is Represents the model policy of each agent model The total number of performance tests performed, Indicates the total number of clients; Set the loss function of the evaluation network corresponding to the local agent model on the user side to ； in, represents the network parameters of the evaluation network, represents the discount factor, It comes from the offline data subset and the collection of user-side online collected data. Represents the state of the samples in the collection, represents the reward value of the sample in the set, represents the auxiliary variable in the summation process of the loss function, The network parameters are The state value function of t represents the sampling time of the samples in the set, Indicates the maximum time step of the trajectory corresponding to the sample.

2. The method according to claim 1, It is characterized in that The loss function of the state value network of the agent is in, represents the discount factor, All from offline datasets , Represents the state of samples in the offline dataset, represents the reward value of the sample in the offline dataset, t represents the sampling time of samples in the offline dataset, represents the state value function, represents the auxiliary variable in the summation process, Indicates the maximum time step of the trajectory corresponding to the sample, Represents the samples in the dataset Seek expectations.

3. The method according to claim 1, It is characterized in that The update formula of the state-action value network of the agent is ； in, represents the state-action value function, represents the estimate of the state-action value, represents the discount factor, All from offline datasets , Represents the state of samples in the offline dataset, Indicates the action corresponding to the sample state in the offline dataset, represents the learning rate, represents the reward value of the sample in the offline dataset, t Represents the sampling time of samples in the offline dataset.

4. The method according to claim 1, It is characterized in that The update formula of the agent's behavior cloning network is: ； in, represents the behavior cloning network strategy, Indicates that when the strategy is When in state Take action under the conditions The probability of Represents the state of samples in the offline dataset, Indicates the action corresponding to the sample state in the offline dataset, Represents the samples in the dataset Seek expectations.

5. The method according to claim 1, It is characterized in that The loss function of the target policy network of the agent is ； in, Represents the samples in the offline dataset Seek hope, represents the target strategy, represents the sampling strategy of offline samples, Represents the state of samples in the offline dataset, Indicates the action corresponding to the sample state in the offline dataset, represents the learning rate, represents the reward value of the sample in the offline dataset, t represents the sampling time of samples in the offline dataset, represents the advantage value, which is obtained directly through the generalized advantage estimation, is represented as a clipping function, represents the cropping factor, Indicates that when the strategy is When in state Take action under the conditions probability.

6. The method according to claim 1, It is characterized in that Set the objective function of the action network corresponding to the local agent model on the user side to ; in, is the action network parameter of the current update strategy, It comes from the offline data subset and the collection of user-side online collected data. Indicates the current update strategy, represents the historical sampling strategy, Estimated in state Take action The advantage value of represents the state of the samples in the collection, Represents the action corresponding to the sample state in the set, Indicates that when the strategy is When in state Take action under the conditions The probability of Indicates that when the strategy is When in state Take action under the conditions The probability of is represented as a clipping function, Represents the crop factor.

Images