Data generation and strategy optimization method and system based on CDDPM and DRL

Abstract

The invention discloses a data generation and strategy optimization method and system based on CDDPM and DRL. The system comprises a DRL module, a CDDPM module and an RD2CD module. The method comprises the steps that a DRL selects a corresponding action and interacts with the environment according to a state obtained from the environment, and experience data is obtained and stored in an experience playback pool; the empirical data are converted into CDDPM training data and input data during synthesis of the CDDPM training data through an RD2CD module; training the CDDPM by using the training data, inputting the source data and conditions into the CDDPM, and denoising to obtain synthetic data; randomly sampling data with a Batch size from actual empirical data, and updating a Q network and a target network of the DRL together with the synthesized data; through circulation, the DRL module continuously optimizes the strategy under the combined action of real and synthetic data. According to the invention, the training efficiency and decision accuracy of the reinforcement learning system under the condition of data scarcity can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making and deep reinforcement learning technology, and in particular to a data generation and policy optimization method and system based on CDDPM and DRL. Background Technology

[0002] With the widespread application of intelligent decision-making systems, dynamic changes in complex environments necessitate agents capable of autonomously adjusting their policies to adapt to environmental uncertainties and real-time requirements. While numerous policy optimization methods based on machine learning, particularly reinforcement learning (RL), have been proposed, most studies rely on large amounts of interaction data or lengthy training processes to enable the agent to learn the optimal policy through multiple trials. In real-world scenarios, systems often only have limited interaction samples or empirical data, making large-scale training difficult. In such cases, the learning efficiency of traditional reinforcement learning methods significantly decreases, leading to slow policy optimization. Therefore, achieving efficient policy learning and optimization under limited empirical data conditions has become a pressing issue in the fields of intelligent decision-making and reinforcement learning.

[0003] One reason for the slow online learning speed of Deep Reinforcement Learning (DRL) is its low data collection efficiency, making it difficult to obtain sufficient training data. Current generative models, such as Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), were initially applied to image generation and later to video, speech, and text. Huang et al. (Huang V, Ley T, Vlachou-Konchylaki M, et al. Enhanced Experience ReplayGeneration for Efficient Reinforcement Learning[M]. arXiv, 2017.) addressed the problem of slow learning speed caused by the low data sampling efficiency of DRL on real systems by proposing an Enhanced Generative Adversarial Network (EGAN) to synthesize training data. The results show that EGAN effectively improves the learning speed in the early stages of the training process. CTGAN (Conditional Tabular Generative Adversarial Network) effectively solves the data imbalance problem through pattern-specific normalization and conditional generators, and has good generation performance for tabular data (Xu L, Skoularidou M, Cuesta-Infante A, et al. Modeling Tabular data using Conditional GAN[C] / / Advances in Neural Information Processing Systems: Vol. 32. Curran Associates, Inc., 2019.).Chen (Chen Q, Kiami S, Gupta A, et al. GenAug: Retargeting behaviors to unseensituations via Generative Augmentation[C] / / Robotics: Science and Systems XIX.Robotics: Science and Systems Foundation, 2023.) and Yu (Yu T, Xiao T, Tompson J, et al. Scaling Robot Learning with Semantically Imagined Experience[C] / / Robotics: Science and Systems XIX: Vol. 19. 2023.) addressed the problem of excessively high costs in collecting training data for robots in different scenarios by using a text-to-image generative model. This model synthesizes training data of the same actions in different scenarios based on data sources other than text descriptions and experience, thereby improving the generalization ability of DRL.

[0004] The Denoising Diffusion Probabilistic Model (DDPM) outperforms existing generative models such as GANs and VAEs in various visual tasks, and also demonstrates good generative capabilities in tasks with few samples. Some researchers have combined it with DRL as a policy network, achieving good results in trajectory planning tasks. Other researchers have used DDPM to generate empirical data for DRL learning. Synther (Lu C, Ball PJ, Teh YW, et al. Synthetic experience replay[C] / / Proceedings of the 37th International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc., 2024: 46323-46344.) uses a diffusion model to generate data based on existing experience and can generate new actions and reward labels beyond the input data, significantly improving the learning performance of both offline and online reinforcement learning. To address the challenges in multi-task reinforcement learning, MTDiff (He H, Bai C, Xu K, et al. Diffusionmodel is an effective planner and data synthesizer for multi-task reinforcement learning[C] / / Proceedings of the 37th International Conference on Neural Information Processing Systems. Red Hook, NY, USA: CurranAssociates Inc., 2024: 64896-64917.) replaced the UNet network in the diffusion model with GPT2, improving the diffusion model's ability to simulate trajectory sequences. Simultaneously, it incorporated cue learning, using historical experience data as cues to eliminate ambiguities between different tasks during training and inference. To further leverage the powerful ability of the diffusion model to simulate complex policies and trajectories, MTDiff was applied to both action sequence planning and experience data synthesis tasks, achieving good results in both.EAQ (Oh J, Kim S, Kim G, et al. Diffusion-based Episodes Augmentation for Offline Multi-Agent Reinforcement Learning[C] / / ICML 2024 Workshop on Structured Probabilistic Inference {&} Generative Modeling. 2024.) addresses the problem of missing training data in multi-agent reinforcement learning by proposing an episode augmentation method guided by Q-total loss based on a diffusion model. Q-total loss refers to the difference between the generated trajectory reward and the maximum reward of the true trajectory. By incorporating Q-total loss into the loss function, the reward of the generated data is maximized, ensuring consistency between the diffusion model and the reinforcement learning objective. This results in the generated data containing more actions of multi-agent cooperation, improving the collaborative ability of multiple agents.

[0005] Most of the methods mentioned above rely on sufficient training data to enhance the learning performance of offline DRL, but they are not suitable for online learning or situations with limited sample data. Therefore, there is an urgent need to invent a method to accelerate the online learning speed of reinforcement learning, enabling it to generate data that meets specified conditions and improve the accuracy of the generated data. Summary of the Invention

[0006] The purpose of this invention is to provide a data generation and strategy optimization method and system based on CDDPM and DRL that has low data requirements, fast learning speed, and high accuracy.

[0007] The technical solution to achieve the purpose of this invention is: a data generation and strategy optimization method based on CDDPM and DRL, comprising the following steps:

[0008] Step 1: Construct an interactive environment model;

[0009] Step 2: Describe the decision-making process of the agent in the environment;

[0010] Step 3: DRL selects the appropriate action to interact with the environment based on the state information obtained from the environment, obtains experience data, and stores it in the experience replay pool;

[0011] Step 4: Convert the obtained empirical data into two parts using the RD2CD module: CDDPM training data and input data for synthesizing data.

[0012] Step 5: First, train CDDPM using training data to learn the distribution of empirical data; then, CDDPM undergoes continuous denoising to obtain synthetic data.

[0013] Step 6: Randomly sample data of batch size from actual experience data, update the Q network of DRL together with the synthetic data, and update the target network of DRL after a set number of updates;

[0014] Step 7: The interactive environment dynamically adjusts the state parameters based on the interaction results. DRL interacts with the environment based on the latest strategy and continuously optimizes its own strategy through the collected experience and data synthesized by CDDPM, thereby achieving strategy learning and adaptive decision-making.

[0015] Furthermore, the construction of the interactive environment model described in step 1 is as follows:

[0016] Step 1.1: Establish an interactive environment model. The environment is used to simulate the dynamic changes of the state in the task scenario. The agent interacts with the environment by perceiving the environmental state and selecting corresponding actions. The state consists of two parameters. and Composition, used to describe the current characteristics of the environment or the operating status of the system; the actions are... , , It consists of three parameters, which are used to control the response or operation instructions of the intelligent agent in the environment; the environment provides new state information based on the actions of the intelligent agent, forming a closed-loop interaction process; the intelligent agent continuously adjusts its own strategy based on the feedback from the environment until the preset task goal or termination condition is achieved.

[0017] Step 1.2: Establish the interaction process between the agent and the environment as a Markov Decision Process (MDP). An MDP is defined as a sequence. ,in It is a state space. It is the action space. It is a transfer function. It is a reward function that can be arbitrarily transferred. It is the initial state distribution. It is a discount factor; at any time step The agent uses a strategy Select Action Perform this action Then, obtain the next state. and rewards ;

[0018] Step 1.3: Implement a decision-making system using deep reinforcement learning, with the goal of learning a strategy by maximizing rewards. .

[0019] Furthermore, the problem described in step 2 is as follows:

[0020] Step 2.1: The agent interacts with the environment to obtain one piece of real data at a time. , The action was recorded. State Rewards :

[0021] (1)

[0022] Step 2.2: During the forward noise addition process of CDDPM, in Noise is gradually added to the data in each step. In the middle, after After adding noise, the result is The forward noise addition process satisfies:

[0023] (2)

[0024] (3)

[0025] in, It is a predefined noise variance. It is the identity matrix;

[0026] Step 2.3: In the inverse denoising process of CDDPM, the role of CDDPM is based on... Based on past experience, synthesize movements as accurately as possible. For similar states Data, i.e., synthetic data With real data As similar as possible;

[0027] CDDPM utilizes neural networks Fitting the inverse process ,in These are the conditions input into CDDPM; the reverse process satisfies:

[0028] (4) (5)

[0029] Step 2.4: CDDPM ultimately obtains the final result through continuous noise reduction. And by minimizing Data with improved synthesis accuracy;

[0030] Step 2.5: Add the synthesized data Compared with real datasets Used together as training data Used for DRL updates:

[0031] (6)

[0032] Furthermore, in step 4, the obtained empirical data is converted into two parts using the RD2CD module: CDDPM training data and input data for synthesizing data, as detailed below:

[0033] Step 4.1, the composition of each DRL experience data is as follows: ,in For state, For action, It is the next state after the action is executed. It is the reward obtained by performing an action;

[0034] Step 4.2: Perform offline data transformation on the existing DRL offline experience data to convert it into CDDPM offline training data. For a single DRL offline experience data set... This data serves as the source data input into CDDPM. , will the state Replace with other values In the state and actions Keep it unchanged, and find the corresponding data in the existing data. As the target of CDDPM generated data ,in The first five elements are used as generation conditions ;

[0035] CDDPM's offline training data is provided by Composition; Adding constraints to the input It can limit the scope of data generation while providing guidance for synthesizing data, resulting in highly accurate data; input With the goal They are the same action Empirical data on similar states;

[0036] Step 4.3, Settings , Still The first five elements Convert DRL online data into CDDPM online training data;

[0037] Step 4.4: RD2CD selects successful experience data, i.e. The data is used as input when synthesizing CDDPM data. ,Will The state in The parameters are replaced with other values, and the replaced states and actions are used as composition conditions input into CDDPM. The successful experiences already obtained are transformed into successful experiences similar to their current state, thus obtaining the input data for CDDPM synthesis data.

[0038] Furthermore, step 5 is specifically as follows:

[0039] Step 5.1: Train CDDPM using training data to learn the distribution of empirical data;

[0040] Step 5.2: Transfer the source data and conditions The data is input into CDDPM and undergoes continuous noise reduction to obtain the synthesized data.

[0041] Furthermore, step 5.1, which involves training the CDDPM using training data to learn the distribution of empirical data, is detailed as follows:

[0042] Step 5.1.1, for the original data In the Step 1: Add noise using the following formula:

[0043] (7)

[0044] in, It is a step with time The relevant parameters determine the degree of noise added. It is independent and identically distributed Gaussian noise;

[0045] Step 5.1.2, Original data after adding noise and conditions Embedding is performed through an MLP layer to obtain... and :

[0046] (8)

[0047] (9)

[0048] Step 5.1.3, in Add time step encoding After that, with By piecing them together, we obtain :

[0049] (10)

[0050] Step 5.1.4, for Add position encoding Then, it is passed into GPT2 to obtain :

[0051] (11)

[0052] Step 5.1.5 Synthetic data is obtained through a Generation Head consisting of a three-layer fully connected network. :

[0053] (12)

[0054] Step 5.1.6: Calculate the actual data With synthetic data The loss is then fed into the optimizer to update the CDDPM network.

[0055] Furthermore, step 5.2 describes the process of transferring the source data... and conditions The data is input into CDDPM and undergoes continuous denoising to obtain the synthesized data, as follows:

[0056] Step 5.2.1: Add noise to the source data. and conditions The data is input into the CDDPM network and synthesized through continuous denoising. ;

[0057] Since the source data is based on existing data, in order for CDDPM to learn the distribution of similar data and master basic synthesis capabilities before online learning begins, CDDPM is pre-trained using existing data. Existing data refers to the experience data of the DRL decision system for other states besides the states that need to be learned online.

[0058] In the initial stage, if the amount of data collected by DRL is too small and the number of updates is too high, overfitting may occur. Conversely, if the number of updates is too low, the learning speed of CDDPM will be slow, thus affecting DRL updates. Therefore, the number of CDDPM updates should be set based on the collected empirical data. :

[0059] (13)

[0060] in It's a hyperparameter;

[0061] By adding conditions and setting time steps, CDDPM can make the generated data similar to real data. However, since the generated data cannot be exactly the same as real data, CDDPM can only be used to accelerate the early stages of DRL online learning. As more successful experiences are collected, when a certain... When all states corresponding to the parameters have collected successful experiences, CDDPM stops training and generating the data to avoid negative updates to reinforcement learning.

[0062] A data generation and strategy optimization system based on CDDPM and DRL is provided. This system implements the aforementioned data generation and strategy optimization method based on CDDPM and DRL, and includes a DRL module, a CDDPM module, and an RD2CD module, wherein:

[0063] The DRL module is implemented using the DDQN algorithm, which intelligently adjusts the decision-making strategy according to environmental changes, giving the system adaptive capabilities.

[0064] The CDDPM module is used to learn from the real dataset of DRL, synthesize data, expand the DRL training data, and improve the learning speed of DRL.

[0065] The RD2CD module is used as input data when converting DRL training data into CDDPM training data and CDDPM synthetic data.

[0066] A mobile terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the data generation and strategy optimization method based on CDDPM and DRL.

[0067] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data generation and strategy optimization method based on CDDPM and DRL.

[0068] Compared with the prior art, the significant advantages of this invention are:

[0069] (1) A data generation and policy optimization architecture combining CDDPM and DRL is adopted. The training data required by DRL is synthesized by CDDPM, which reduces the number of interactions between DRL and the environment, so that DRL can quickly learn the optimal policy applicable to multiple states, thereby improving the learning efficiency and real-time performance of the system.

[0070] (2) Different processing methods were developed for offline and online experience data of DRL to meet the training requirements of CDDPM at different stages, and CDDPM was pre-trained using offline DRL data so that CDDPM could master the ability to synthesize relevant data before online training.

[0071] (3) CDDPM generates data from the source data after adding noise, so that it can balance accuracy and diffusion when synthesizing data. At the same time, the state and action of the target data are input into CDDPM as conditional information, which improves the accuracy of CDDPM synthesized data. Attached Figure Description

[0072] Figure 1 This is a schematic diagram of the structure of a data generation and strategy optimization system based on CDDPM and DRL according to the present invention.

[0073] Figure 2 This is a schematic diagram of the interactive environment structure established in this invention.

[0074] Figure 3 This is a schematic diagram of the RD2CD module in this invention.

[0075] Figure 4 This is a schematic diagram of the structure during CDDPM training and generation used in this invention.

[0076] Figure 5 This is an algorithm flowchart of a data generation and strategy optimization method based on CDDPM and DRL according to the present invention.

[0077] Figure 6a The graph shows the test success rate curves after each round of training for different methods.

[0078] Figure 6b This is a graph showing the change in loss during the online learning process.

[0079] Figure 6c Histograms of rewards and running times for different methods.

[0080] Figure 7a Success rate curves for different algorithms.

[0081] Figure 7b The graph shows the loss curves for different algorithms.

[0082] Figure 7c Histograms of rewards and running times for different algorithms.

[0083] Figure 8a This is a success rate curve for different networks.

[0084] Figure 8b The graph shows the loss curves for different networks.

[0085] Figure 8c Histograms of rewards and running times for different networks.

[0086] Figure 9aThis is a success rate curve for different input data and time steps in an embodiment of the present invention.

[0087] Figure 9b The diagram shows the loss curves for different input data and time steps in this embodiment of the invention.

[0088] Figure 9c This is a histogram of rewards and running time for different input data and time steps in an embodiment of the present invention. Detailed Implementation

[0089] like Figure 1 As shown, the present invention provides a data generation and strategy optimization system based on CDDPM and DRL, including a DRL module, a CDDPM module, and an RD2CD module;

[0090] The DRL module is implemented using the DDQN algorithm, which intelligently changes the strategy according to environmental changes, giving the decision-making system adaptive capabilities.

[0091] The CDDPM module is used to learn from the real dataset of DRL, synthesize data, expand the DRL training data, and improve the learning speed of DRL.

[0092] The RD2CD module is used as input data when converting DRL training data into CDDPM training data and CDDPM synthetic data.

[0093] This invention also provides a data generation and strategy optimization method based on CDDPM and DRL, comprising the following steps:

[0094] Step 1: Establish an interactive environment model, as follows:

[0095] Step 1.1: Establish an interactive environment model, such as a drone navigation model, e.g. Figure 2 As shown, the environment is used to simulate the dynamic changes in state within a task scenario. The agent interacts with the environment by perceiving its state and selecting appropriate actions. The state is determined by two parameters. and Composition, used to describe the current characteristics of the environment or the operating status of the system; the actions are... , , It consists of three parameters, used to control the agent's response or operational commands in the environment. The environment provides new state information based on the agent's actions, forming a closed-loop interaction process. The agent continuously adjusts its strategy based on environmental feedback until it reaches the preset task objective or termination condition;

[0096] Step 1.2: Establish the interaction process between the agent and the environment as a Markov Decision Process (MDP). An MDP can be defined as a sequence. ,in It is a state space. It is the action space. It is a transfer function. It is a reward function that can be arbitrarily transferred. It is the initial state distribution. It is a discount factor; at any time step The agent uses a strategy Select Action After performing this action, the next state is obtained. and rewards ;

[0097] Step 1.3: Implement a decision-making system using deep reinforcement learning, with the goal of learning a strategy by maximizing rewards. .

[0098] Step 2: Describe the problems in the data generation and strategy optimization process, such as the problem description of the UAV navigation process, as follows:

[0099] Given the low efficiency of online data collection for DRL, this invention uses CDDPM to learn real-world datasets for DRL. Synthetic data Expand the training data for DRL and improve the learning speed of DRL.

[0100] Specifically, CDDPM uses existing data and some newly collected online data to learn the distribution of real-world experience data, thereby synthesizing highly accurate data on previously uncollected similar states. Similar states refer to... Same parameters States with different parameters.

[0101] Step 2.1: One piece of data obtained from a single interaction between the agent and the environment. This data record shows the action. State reward results :

[0102] (1)

[0103] Step 2.2: During the forward noise addition process of CDDPM, in Noise is gradually added to the data in each step. In the middle, after After adding noise, the result is The forward noise addition process satisfies:

[0104] (2)

[0105] (3)

[0106] in, It is a predefined noise variance. It is the identity matrix;

[0107] Step 2.3: In the reverse denoising process of CDDPM, the role of CDDPM is to synthesize actions as accurately as possible based on the data and past experience. For similar states Data, i.e., synthetic data With real data As similar as possible;

[0108] CDDPM utilizes neural networks Fitting the inverse process ,in These are the conditions input into CDDPM; the reverse process satisfies:

[0109] (4)

[0110] (5)

[0111] Step 2.4: CDDPM ultimately obtains the final result through continuous noise reduction. And by minimizing Synthesize data with high accuracy;

[0112] Step 2.5: Add the synthesized data Compared with real datasets Used together as training data Used for DRL updates:

[0113] (6)

[0114] Step 3: DRL uses the state obtained from the environment. Navigation parameters, i.e., action selection. Interact with the environment to obtain empirical data And store it in the experience replay pool;

[0115] Step 4: Convert the obtained empirical data into two parts using the RD2CD module: CDDPM training data and input data used for synthesizing the data, as detailed below:

[0116] The specific structure of the RD2CD module is as follows: Figure 3 As shown, the RD2CD module's processing of DRL experience data includes two parts: the transformation of offline and online data.

[0117] Step 4.1, the composition of each DRL experience data is as follows: ,in For state, For action, It is the next state after the action is executed. It is the reward obtained by performing an action;

[0118] Step 4.2: Perform offline data transformation on the existing DRL offline experience data to convert it into CDDPM offline training data. For a specific DRL offline experience data point... This data serves as the source data input into CDDPM. , will the state Replace with other values In the state and actions Keep it unchanged, and find the corresponding data in the existing data. As the target of CDDPM generated data ,in The first five elements are used as generation conditions ;

[0119] CDDPM's offline training data is provided by Composition; Adding constraints to the input It can limit the scope of data generation while providing guidance for synthesizing data, resulting in highly accurate data; input With the goal They are the same action For empirical data of similar states, RD2CD is used to generate training data, enabling CDDPM to master basic data transformation capabilities during the pre-training stage, which facilitates the online synthesis and generation of state data based on existing data.

[0120] Step 4.3: The principles of online and offline data transformation are largely the same. The difference lies in that DRL can only collect one data point per online interaction, and finding data with the same action and similar state for any given online data point is time-consuming. Furthermore, CDDPM already possesses basic data transformation capabilities during the pre-training phase; during the online learning phase, it only needs to adapt to the new data distribution. Therefore, setting... , Still The first five elements Convert DRL online data into CDDPM online training data;

[0121] Step 4.4: RD2CD selects successful experience data, i.e. The data is used as input when synthesizing CDDPM data. ,Will The state in The parameters are replaced with other values, and the replaced states and actions are used as composition conditions input into CDDPM. The successful experiences already obtained are transformed into successful experiences similar to their current state, thus obtaining the input data for CDDPM synthesis data.

[0122] The parameter settings for training CDDPM and DRL are shown in Table 1.

[0123] Table 1 Overall Parameters

[0124] Step 5: First, train CDDPM using training data to learn the distribution of empirical data; then, use the source data... and conditions The data is input into CDDPM and undergoes continuous denoising to obtain synthetic data, such as... Figure 4 As shown, the details are as follows:

[0125] The use of CDDPM instead of DDPM allows for more precise control over data synthesis through input conditional information. DDPM, initially used in the image domain, employs the Unet network, which is better suited for processing image data and less suitable for generating the data required in this paper. Transformers, using self-attention mechanisms, can capture complex dependencies in sequential data, while GPT2, employing multiple stacked Transformer Decoder layers, offers superior data generation capabilities. Since the synthesized data required in this invention is similar to sequential data, GPT2 is used as a crucial network component.

[0126] The network input includes and Two parts, CDDPM during data generation It's real data with added noise, not pure noise. The purpose of using CDDPM is to transform existing data into uncollected data. On one hand, the real data with added noise retains some of the original features, allowing the model to generate similar data more accurately. This makes the generated data more similar to the real data, guiding the RL update and optimization in the right direction. On the other hand, the added noise allows the generated data to spread to some extent, preventing it from being too similar to the input data.

[0127] Step 5.1: Train CDDPM using training data to learn the distribution of experience data, as follows:

[0128] Step 5.1.1, for the original data In the Step 1: Add noise using the following formula:

[0129] (7)

[0130] in, It is a step with time The relevant parameters determine the degree of noise added. It is independent and identically distributed Gaussian noise;

[0131] Step 5.1.2, Original data after adding noise and conditions Embedding is performed through an MLP layer to obtain... and :

[0132] (8)

[0133] (9)

[0134] Step 5.1.3, in Add time step encoding After that, with By piecing them together, we obtain :

[0135] (10)

[0136] Step 5.1.4, for Add position encoding Then, it is passed into GPT2 to obtain :

[0137] (11)

[0138] Step 5.1.5 Synthetic data is obtained through a Generation Head consisting of a three-layer fully connected network. :

[0139] (12)

[0140] Step 5.1.6: Calculate the actual data With synthetic data The loss is then fed into the optimizer to update the CDDPM network.

[0141] Step 5.2: Transfer the source data and conditions The data is input into CDDPM and undergoes continuous denoising to obtain the synthesized data, as follows:

[0142] Step 5.2.1: Add noise to the source data. and conditions The data is input into the CDDPM network and synthesized through continuous denoising. ;

[0143] Since the source data is based on existing data, in order for CDDPM to learn the distribution of similar data and master basic synthesis capabilities before online learning begins, CDDPM is pre-trained using existing data. Existing data refers to the experience data of the DRL decision system for other states besides the states that need to be learned online.

[0144] In the initial stage, if the amount of data collected by DRL is too small and the number of updates is too high, overfitting may occur. Conversely, if the number of updates is too low, the learning speed of CDDPM will be slow, thus affecting DRL updates. Therefore, the number of CDDPM updates should be set based on the collected empirical data. :

[0145] (13)

[0146] in It's a hyperparameter;

[0147] By adding conditions and setting time steps, CDDPM can make the generated data similar to real data. However, since the generated data cannot be exactly the same as real data, CDDPM can only be used to accelerate the early stages of DRL online learning. As more successful experiences are collected, when a certain parameter... Once all corresponding states have collected successful experiences, CDDPM stops training and generating that data to avoid negative updates to reinforcement learning.

[0148] Step 6: Randomly sample batch-sized data from actual experience data and update the Q network of DRL together with the synthetic data. After updating a certain number of times, update the target network of DRL.

[0149] Step 7: The interactive environment dynamically adjusts state parameters based on the interaction results. DRL interacts with the environment based on the latest strategy and continuously optimizes its own strategy through collected experience and data synthesized by CDDPM, achieving efficient strategy learning and adaptive decision-making. For example, the UAV adjusts its navigation parameters according to the environment, and DRL interacts with the environment based on the latest strategy, continuously optimizing its own strategy through collected experience and data synthesized by CDDPM, achieving efficient UAV navigation.

[0150] The method proposed in this paper is as follows: Figure 5 As shown.

[0151] The present invention also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data generation and strategy optimization method based on CDDPM and DRL.

[0152] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps in the data generation and strategy optimization method based on CDDPM and DRL.

[0153] In one embodiment, a method for generating UAV navigation data and optimizing strategies based on CDDPM and DRL is provided, including the following steps:

[0154] Step 1: Construct a 3D dynamic navigation environment model for the UAV.

[0155] The environment model includes static elements (such as buildings, no-fly zones, and terrain) and dynamic elements (such as other aircraft, sudden obstacles, and wind interference). The environment provides the UAV with status information, including the UAV's own pose, speed, sensor data, and obstacle information in the environment.

[0156] Step 2: Describe the problem of the decision-making process for UAV navigation.

[0157] The drone is defined as an intelligent agent, whose actions are flight control commands (such as pitch, roll, yaw rate and throttle); its state is environmental perception information; and the reward function is designed based on factors such as flight safety, path smoothness, target proximity and energy consumption.

[0158] Step 3: Deep reinforcement learning interaction and experience collection.

[0159] The DRL policy network selects actions (control commands) based on the current environmental state (such as the drone's position and the point cloud of surrounding obstacles) and interacts with the navigation environment. The experience data (state, action, reward, new state) generated by the interaction is stored in the experience replay pool.

[0160] Step 4: Navigation experience data conversion.

[0161] The collected raw empirical data is transformed into two parts using a Raw Data to Conditional Data (RD2CD) module:

[0162] CDDPM training data: The data distribution used to train the diffusion model to learn successful navigation paths.

[0163] Input data for the synthesis conditions: These serve as the conditions guiding CDDPM to generate new navigation trajectories (such as start point, end point, and threat area information).

[0164] Step 5: Navigation trajectory generation and data augmentation based on CDDPM.

[0165] Training phase: The CDDPM is trained using the training data obtained in step 4, enabling it to learn the data distribution of safe and efficient navigation trajectories under given conditions (such as specific mission objectives and environmental constraints).

[0166] Synthesis phase: The trained CDDPM generates diverse synthetic navigation trajectory data (including state sequences and action sequences) that conform to the distribution of real data through a continuous denoising process based on new task conditions (such as new take-off and landing points and new obstacle layouts).

[0167] Step 6: Hybrid data-driven DRL strategy optimization.

[0168] A batch of data is randomly sampled from the experience replay pool of actual interactions, and another batch of data is sampled from the synthetic navigation data generated by CDDPM. These two datasets are then used together to update the Q network (or Critic network) of the DRL. After the Q network has been updated a set number of times, the parameters of the target network are synchronized using either a soft or hard update method.

[0169] Step 7: Dynamic environment adaptive navigation.

[0170] The interactive environment can dynamically adjust the difficulty or introduce new challenges (such as changing the wind field or moving obstacles) based on the drone's performance. The DRL strategy makes decisions based on the latest environmental conditions and continuously optimizes the navigation strategy by comprehensively utilizing real-world interaction experience and synthetic experience generated by CDDPM, ultimately achieving robust and adaptive navigation for the drone in complex and dynamic environments.

[0171] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0172] Example

[0173] This embodiment is implemented using PyTorch, equipped with an Intel Core i7 CPU, 16GB of RAM, and an NVIDIA GeForce 4060 Ti GPU. DRL uses the DDQN algorithm and RMSE loss function, while CDDPM uses the L1Loss loss function. Both use the Adam optimizer.

[0174] The dataset in this embodiment was obtained through simulation, and the experimental objective is to analyze four datasets. The parameter values ​​are 5. Twenty states, composed of parameter values, enable online policy learning based on deep reinforcement learning. Prior to this, DRL had successfully learned another 15 states and collected sufficient empirical data. During online learning, the environment can intelligently adjust the states. In each learning round, the agent executes at most 20 actions; if the agent achieves the optimal result in each decision, the system can reach a stable state before 20 actions. Based on the quality of the agent's action choices in each state, tiered reward values ​​are set: -800, 200, 400, 600, and 800. This embodiment will illustrate the effectiveness and advantages of the proposed method through the following experiments. For fair comparison, this embodiment presents the average experimental results under 10 random seeds.

[0175] (1) Architecture comparison experiment

[0176] This invention combines DRL with CDDPM to improve the online convergence speed of reinforcement learning. To demonstrate the effectiveness of CDDPM, this embodiment compares it with other methods, including using only DRL, using artificially constructed data to update DRL+Arti-Cons, and combining DRL with DDPM (DRL+DDPM). Here, DRL+Arti-Cons refers to directly using real data... The values ​​of the parameters are replaced with other values ​​to construct the same... Empirical data on similar states under the given parameters.

[0177] Experimental results are as follows Figures 6a-6c As shown, Figure 6a The task success rates of different methods after each training round are shown. Figure 6b This demonstrates the changes in loss during the online learning process. Figure 6c The rewards and runtimes of different methods are shown. Since the DRL model has some prior learning on existing states, it achieves a 50% task success rate in the initial stage. A longer loss curve indicates more training iterations, a lower success rate, and poorer learning performance. Figures 6a-6c As can be seen, compared with other methods, the method of this invention has the fastest learning speed, reaches the convergence state after about 175 disturbances, the task success rate stabilizes at 100% first, and the reward is the highest. Compared with the traditional DRL method, the success rate is improved by about 25%.

[0178] The high initial loss of DRL+CDDPM is due to the limited data available at the beginning of training, preventing accurate data generation. As training data increases, the accuracy of CDDPM's synthesized data gradually improves, leading to a rapid decrease in DRL loss. DRL+CDDPM is more computationally intensive than DRL and DRL+Arti-Cons, resulting in a longer runtime. DRL+Arti-Cons simply constructs data without considering the differences in the same action across different states. While the constructed data can guide positive RL updates in the initial stages, it cannot provide accurate optimization directions. Therefore, its performance fluctuates significantly in later stages. DDPM lacks constraints, causing significant discrepancies between generated and actual data, negatively impacting DRL updates. DRL exhibits overfitting with limited data, resulting in slow learning speeds. As data volume increases, DRL updates in the correct direction, and its test success rate steadily improves in later training stages, but achieving the desired results requires more time.

[0179] In summary, the CDDPM used in this invention can leverage its advantages to synthesize highly accurate empirical data even with limited data volume, thereby improving the learning speed of DRL.

[0180] (2) Comparison Experiment of Generation Algorithms

[0181] The empirical data format used in DRL training is similar to tabular data. CTGAN uses GAN to generate tabular data and is a classic algorithm for tabular data generation. Synther is a method that uses DDPM to accelerate the online learning speed of DRL, which is similar to the problem that this invention aims to solve. Therefore, this invention is compared with the above algorithms.

[0182] The results of comparing the task success rate, loss curves, rewards, and running time of different algorithms are as follows: Figures 7a-7c As shown in Table 2, the number of parameters and computational cost of the three algorithms are presented.

[0183] Table 2. Parameter count and computational cost of different algorithms

[0184] algorithm Number of parameters (M) Computational cost in FLOPs (M) Our 0.92 0.95 Synther 6.47 6.48 CTGAN 0.18 0.18

[0185] Since all algorithms are based on DRL, the parameter and computational costs only include the generation algorithm. The method of this invention converges approximately 100 times earlier than other algorithms and offers higher rewards. Synther uses a multi-layer residual MLP as its denoising network, resulting in the largest parameter and computational costs, seven times that of the method of this invention. The success rate of the method of this invention is similar to that of Synther in the initial stage. As the number of training rounds increases and more experience data is collected, the method of this invention can synthesize data with higher accuracy. This is because the method of this invention inputs the states and actions of successful experiences as conditional information into the CDDPM for synthesizing data, and uses the noisy source data as input. The input conditions and the noisy source data provide more clues for data synthesis, and the conditions further limit the range of synthesized data, thereby improving the accuracy of synthesized data and promoting rapid learning of DRL. CTGAN's data synthesis capability is inferior to that of diffusion models, with a maximum task success rate of only 80%. Although it has the smallest parameter and computational costs, in addition to DRL, it also requires training a generator and a discriminator. Furthermore, the DRL learning effect is poor, resulting in a runtime five times longer than the method of this invention.

[0186] Through multi-faceted comparative analysis, the CDDPM algorithm used in this invention, along with specially designed input source data and conditions, effectively improves the learning speed of DRL.

[0187] (3) Comparative experiment of CDDPM network

[0188] Considering that the DRL training data required for this invention is similar to sequence and text data, a Transformer network structure is suitable. GPT2 consists of TransformerDecoder layers and has strong data generation capabilities. Taking all the above factors into account, this invention uses GPT2 as the CDDPM network. To demonstrate the advantages of GPT2 in handling the task presented in this paper, it is compared with Unet, LSTM, GRU, and BERT networks. Unet is the original network used in DDPM; GRU and LSTM can capture the semantic relationships between sequence data; BERT also uses a Transformer structure and is suitable for processing text data.

[0189] The success rate, loss variation, reward, and runtime of tasks on different networks are as follows: Figures 8a-8c As shown in Table 3, the number of parameters and computational cost for different networks are as follows.

[0190] Table 3. Parameter quantity and computational cost of different networks

[0191] network Number of parameters (M) Computational cost in FLOPs (M) Our 0.92 0.95 GRU 1.12 3.52 LSTM 1.18 4.38 Bert 1.33 6.08 Unet 0.17 1.58

[0192] from Figures 8a-8cAs can be seen, the network used in this invention has the least computational cost, the fastest convergence speed, and the highest reward, showing a significant advantage over other networks. The learning performance of other networks is similar, with a final task success rate of around 85%. Among them, Unet is more suitable for processing image data, but its success rate fluctuates greatly in the early stages, its convergence speed is slower, and its running time is longer, about 10 times that of the method in this invention. The RL data generated by this invention is transition data, not trajectory data composed of multiple interaction records within a single round. LSTM and GRU are more suitable for long-term sequence data; therefore, both are insufficient in generating the data required by this invention. Although BERT is also based on Transformer, its Encoder layer has a lower data generation capability than its Decoder layer. Using it as a CDDPM network results in poor DRL learning performance. In summary, using GPT2 as the CDDPM network in this invention can significantly improve the quality of CDDPM synthesized data and accelerate the learning speed of DRL.

[0193] (4) Ablation test

[0194] Input data and time step

[0195] To balance the accuracy and diffusion of the generated data, this embodiment uses noisy real data as input and selects an appropriate time step. To explore the impact of input data type and time step on the experimental results, ablation experiments were conducted on both. Since the data to be synthesized in this embodiment is relatively concentrated and the data range is small, four time steps of 10, 20, 40, and 80 were selected for the experiments. The numerical part represents the time step, src indicates that the input is noisy real data, and noise indicates that the input is pure noise. For example, 10_src indicates a time step of 10 and input data is noisy real data.

[0196] Figures 9a-9c The results show the task success rate, loss variation, and a comparison of rewards and runtime for each combination. Figures 9a-9c The results show that when the input data is noisy source data and the time step is 20, DRL converges the fastest, yields the highest reward, has the shortest running time, and achieves the best learning effect. When the input data is noisy real data, DRL generally performs better than when the input is pure noise. This is because noisy real data retains more information and features compared to pure noise, making it easier for CDDPM to synthesize data.

[0197] Choosing an appropriate diffusion step size is equally important when using noisy real data as the input. If the time step is too small, for example, 10, too much of the original information is retained in the input data, causing the data synthesized by CDDPM to be too similar to the original data, making it difficult to synthesize the target data. This results in poor DRL learning performance and longer runtime. As the time step increases, CDDPM inverse denoising takes longer, increasing runtime; the information retained in the input data also decreases, making CDDPM more prone to diffusion, leading to decreased accuracy of the synthesized data and affecting DRL updates.

[0198] Based on the above analysis, the input data and time step used in this invention can effectively balance the accuracy and diffusion of the synthesized data, thereby enhancing the learning effect of DRL.

[0199] As can be seen from the above embodiments, this invention addresses the slow data collection speed and low learning efficiency of online learning strategies in DRL-based decision systems by proposing a data generation and strategy optimization device and method based on CDDPM and DRL. To enable CDDPM to transform acquired data into unacquired data, this invention proposes an RD2CD module to convert RL experience data into CDDPM training data, using target data with similar states and actions to the source data as the synthesis target. During data synthesis, the state and actions of the target data are used as synthesis conditions, inputting them along with the noisy source data into the network. This ensures the diffusion of CDDPM synthesized data while improving synthesis accuracy. Furthermore, GPT2 is used as part of the CDDPM network to enhance its data synthesis capabilities. Experimental results show that the construction of CDDPM input data and conditions in this invention is reasonable, GPT2 has stronger synthesis capabilities compared to other networks, and the accuracy of the CDDPM synthesized data is higher, effectively improving the online learning speed and accuracy of DRL.

[0200] This invention can be directly applied to fields such as robotics and autonomous driving. For example, in robot manipulation and grasping, it simulates grasping scenarios with various object shapes, weights, and friction coefficients using synthetic data, allowing robots to learn more robust operating strategies. In autonomous driving, it can synthesize data from various rare "edge case" scenarios, such as extreme weather and sudden traffic accidents, to train the decision-making modules of autonomous driving systems, improving their ability to cope with dangerous situations. In drone navigation, it learns efficient and safe obstacle avoidance and path planning strategies in complex urban canyons or forest environments through synthetic data. Furthermore, this invention can be widely applied to cutting-edge fields such as game AI and intelligent decision-making, fintech and algorithmic trading, healthcare and drug development, industrial manufacturing and process control, and network and information security.

[0201] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data generation and strategy optimization method based on CDDPM and DRL, characterized in that, Includes the following steps: Step 1: Construct an interactive environment model; Step 2: Describe the decision-making process of the agent in the environment; Step 3: DRL selects the appropriate action to interact with the environment based on the state information obtained from the environment, obtains experience data, and stores it in the experience replay pool; Step 4: Convert the obtained empirical data into two parts using the RD2CD module: CDDPM training data and input data for synthesizing data. Step 5: First, train CDDPM using training data to learn the distribution of empirical data; then, CDDPM undergoes continuous denoising to obtain synthetic data. Step 6: Randomly sample data of batch size from actual experience data, update the Q network of DRL together with the synthetic data, and update the target network of DRL after a set number of updates; Step 7: The interactive environment dynamically adjusts the state parameters based on the interaction results. DRL interacts with the environment based on the latest strategy and continuously optimizes its own strategy through the collected experience and data synthesized by CDDPM, thereby achieving strategy learning and adaptive decision-making.

2. The data generation and strategy optimization method based on CDDPM and DRL according to claim 1, characterized in that, The construction of the interactive environment model described in step 1 is as follows: Step 1.1: Establish an interactive environment model. The environment is used to simulate the dynamic changes of the state in the task scenario. The agent interacts with the environment by perceiving the environmental state and selecting corresponding actions. The state consists of two parameters. and Composition, used to describe the current characteristics of the environment or the operating status of the system; the actions are... , , It consists of three parameters, which are used to control the response or operation instructions of the intelligent agent in the environment; the environment provides new state information based on the actions of the intelligent agent, forming a closed-loop interaction process; the intelligent agent continuously adjusts its own strategy based on the feedback from the environment until the preset task goal or termination condition is achieved. Step 1.2: Establish the interaction process between the agent and the environment as a Markov Decision Process (MDP). An MDP is defined as a sequence. ,in It is a state space. It is the action space. It is a transfer function. It is a reward function that can be arbitrarily transferred. It is the initial state distribution. It is a discount factor; at any time step The agent uses a strategy Select Action Perform this action Then, obtain the next state. and rewards ; Step 1.3: Implement a decision-making system using deep reinforcement learning, with the goal of learning a strategy by maximizing rewards. .

3. The data generation and strategy optimization method based on CDDPM and DRL according to claim 2, characterized in that, The problem described in step 2 is as follows: Step 2.1: The agent interacts with the environment to obtain one piece of real data at a time. , The action was recorded. State Rewards : （1） Step 2.2: During the forward noise addition process of CDDPM, in Noise is gradually added to the data in each step. In the middle, after After adding noise, the result is The forward noise addition process satisfies: （2） （3） in, It is a predefined noise variance. It is the identity matrix; Step 2.3: In the inverse denoising process of CDDPM, the role of CDDPM is based on... Based on past experience, synthesize movements as accurately as possible. For similar states Data, i.e., synthetic data With real data As similar as possible; CDDPM utilizes neural networks Fitting the inverse process ,in These are the conditions input into CDDPM; the reverse process satisfies: （4） （5） Step 2.4: CDDPM ultimately obtains the final result through continuous noise reduction. And by minimizing Data with improved synthesis accuracy; Step 2.5: Add the synthesized data Compared with real datasets Together as training data Used for DRL updates: （6）。 4. The data generation and strategy optimization method based on CDDPM and DRL according to claim 3, characterized in that, Step 4 involves converting the obtained empirical data into two parts using the RD2CD module: CDDPM training data and input data for synthesizing the data. The details are as follows: Step 4.1, the composition of each DRL experience data is as follows: ,in For state, For action, It is the next state after the action is executed. It is the reward obtained by performing an action; Step 4.2: Perform offline data transformation on the existing DRL offline experience data to convert it into CDDPM offline training data. For a single DRL offline experience data set... This data serves as the source data input into CDDPM. , will the state Replace with other values In the state and actions Keep it unchanged, and find the corresponding data in the existing data. As the target of CDDPM generated data ,in The first five elements are used as generation conditions ; CDDPM's offline training data is provided by Composition; Adding constraints to the input It can limit the scope of data generation while providing guidance for synthesizing data, resulting in highly accurate data; input With the goal They are the same action Empirical data on similar states; Step 4.3, Settings , Still The first five elements Convert DRL online data into CDDPM online training data; Step 4.4: RD2CD selects successful experience data, i.e. The data is used as input when synthesizing CDDPM data. ,Will The state in The parameters are replaced with other values, and the replaced states and actions are used as composition conditions input into CDDPM. The successful experiences already obtained are transformed into successful experiences similar to their current state, thus obtaining the input data for CDDPM synthesis.

5. The data generation and strategy optimization method based on CDDPM and DRL according to claim 4, characterized in that, Step 5 is described in detail below: Step 5.1: Train CDDPM using training data to learn the distribution of empirical data; Step 5.2: Transfer the source data and conditions The data is input into CDDPM and undergoes continuous noise reduction to obtain the synthesized data.

6. The data generation and strategy optimization method based on CDDPM and DRL according to claim 5, characterized in that, Step 5.1 describes training CDDPM using training data to learn the distribution of empirical data, as detailed below: Step 5.1.1, for the original data In the Step 1: Add noise using the following formula: （7） in, It is a step with time The relevant parameters determine the degree of noise added. It is independent and identically distributed Gaussian noise; Step 5.1.2, Original data after adding noise and conditions Embedding is performed through an MLP layer to obtain... and : （8） （9） Step 5.1.3, in Add time step encoding After that, with By piecing them together, we obtain : （10） Step 5.1.4, for Add position encoding Then, it is passed into GPT2 to obtain : （11） Step 5.1.5 Synthetic data is obtained through a Generation Head consisting of a three-layer fully connected network. : （12） Step 5.1.6: Calculate the actual data With synthetic data The loss is then fed into the optimizer to update the CDDPM network.

7. The data generation and strategy optimization method based on CDDPM and DRL according to claim 6, characterized in that, Step 5.2 describes transferring the source data. and conditions The data is input into CDDPM and undergoes continuous denoising to obtain the synthesized data, as follows: Step 5.2.1: Add noise to the source data. and conditions The data is input into the CDDPM network and synthesized through continuous denoising. ; Since the source data is based on existing data, in order for CDDPM to learn the distribution of similar data and master basic synthesis capabilities before online learning begins, CDDPM is pre-trained using existing data. Existing data refers to the experience data of the DRL decision system for other states besides the states that need to be learned online. In the initial stage, if the amount of data collected by DRL is too small and the number of updates is too high, overfitting may occur. Conversely, if the number of updates is too low, the learning speed of CDDPM will be slow, thus affecting DRL updates. Therefore, the number of CDDPM updates should be set based on the collected empirical data. : （13） in It's a hyperparameter; By adding conditions and setting time steps, CDDPM can make the generated data similar to real data. However, since the generated data cannot be exactly the same as real data, CDDPM can only be used to accelerate the early stages of DRL online learning. As more successful experiences are collected, when a certain... When all states corresponding to the parameters have collected successful experiences, CDDPM stops training and generating the data to avoid negative updates to reinforcement learning.

8. A data generation and strategy optimization system based on CDDPM and DRL, characterized in that, This system is used to implement the data generation and strategy optimization method based on CDDPM and DRL as described in any one of claims 1 to 7, comprising a DRL module, a CDDPM module, and an RD2CD module, wherein: The DRL module is implemented using the DDQN algorithm, which intelligently adjusts the decision-making strategy according to environmental changes, giving the system adaptive capabilities. The CDDPM module is used to learn from the real dataset of DRL, synthesize data, expand the DRL training data, and improve the learning speed of DRL. The RD2CD module is used as input data when converting DRL training data into CDDPM training data and CDDPM synthetic data.

9. A mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data generation and strategy optimization method based on CDDPM and DRL as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the data generation and strategy optimization method based on CDDPM and DRL as described in any one of claims 1 to 7.

Images