A decision-making Transformer offline reinforcement learning method based on diffusion state planning

Abstract

This invention discloses a decision transformer offline reinforcement learning method based on diffusion state planning, belonging to the fields of offline reinforcement learning and sequential decision technology. First, given that existing offline reinforcement learning methods for conditional sequence modeling lack trajectory splicing capabilities and have difficulty in reasonably assigning credit based on driver rewards, high-reward sub-goal states are introduced to reconstruct trajectory representations. Then, a diffusion model without classifier guidance is used to plan state sequences with high rewards. Finally, a decision transformer is used to output high-quality action sequences based on the existing trajectory and planned states. This invention can significantly improve the trajectory splicing capabilities of sequence modeling methods, thereby enabling reasonable credit assignment.

Description

Technical Field

[0001] The present invention belongs to the technical field of offline reinforcement learning and sequential decision making, and in particular relates to a decision transformer offline reinforcement learning method based on diffusion state planning. Background Art

[0002] Reinforcement learning can effectively solve sequential decision-making problems through trial and error, and has achieved remarkable success in fields such as gaming. However, reinforcement learning relies on online interaction with the environment, which limits its application in the real world. In some scenarios, online interaction is inefficient and may pose safety risks. For example, training a robot to walk in a real-world environment can take weeks, and human intervention is required to restore the robot after each fall. In autonomous driving tasks, smart cars may pose a danger to pedestrians or other vehicles while learning. In contrast, offline reinforcement learning does not require real-time interaction with the environment and can only learn policies from a static dataset of experience, thus opening up more possibilities for the application of reinforcement learning in real-world scenarios.

[0003] Decision Transformers, using conditional sequence modeling techniques, propose a new supervised learning paradigm for solving offline reinforcement learning problems. Leveraging autoregression and attention mechanisms, they maximize the probability of conditional actions based on historical information, thereby enabling a transition from offline reinforcement learning to supervised sequence modeling. By considering diverse historical information, conditional sequence modeling methods can predict actions and relax the strict Markov assumptions of traditional dynamic programming. This approach has the potential to address suboptimal policies, long horizons, and sparse rewards, while also providing a method for industrial applications.

[0004] However, existing conditional sequence modeling methods only associate reward-driven token values with a single trajectory, focusing on actions that yield high rewards while ignoring the potential contributions of other trajectories to the optimal trajectory. Consequently, they lack the ability to stitch together trajectories. This limitation can lead to ineffective stitching of optimal trajectories in practical industrial applications. Therefore, enhancing the trajectory stitching capabilities of conditional sequence modeling methods has become a key challenge that needs to be addressed. Summary of the Invention

[0005] Purpose of the invention: To address the above problems, the present invention proposes a decision-making Transformer offline reinforcement learning method based on diffusion state planning, which uses a diffusion model without classifier guidance to plan high-reward sub-goal states, and guides the generation of optimal action sequences through the existing trajectories and high-reward sub-goal states.

[0006] Technical solution: To achieve the purpose of the present invention, the technical solution adopted by the present invention is: a decision transformer offline reinforcement learning method based on diffusion state planning, comprising the following steps:

[0007] The state trajectory of the agent Input to the trained diffusion model without classifier guidance Plan a high-reward sub-goal state sequence ,in, Indicates the accumulation of rewards. Indicates status, represents the sequence length, represents the high-reward sub-goal state;

[0008] The high reward sub-goal state sequence and state trajectory Perform splicing to obtain the spliced state trajectory ; Indicates action,

[0009] The spliced state trajectory Input to the trained decision Transformer model , generating the optimal action sequence.

[0010] Furthermore, the diffusion model without classifier guidance The system comprises two hidden layers, the structures of the two hidden layers are the same, and each hidden layer adopts a multi-layer perceptron structure with 256 neurons.

[0011] Furthermore, the decision Transformer network The system includes four hidden layers, all of which have the same structure. Each hidden layer adopts a causal Transformer structure with 1024 neurons.

[0012] Furthermore, the diffusion model without classifier guidance The training process is:

[0013] First, initialize the diffusion model network without classifier guidance Network parameters ;

[0014] Then use the experience pool State trajectory samples in The network parameters Update is performed by minimizing the following loss function through gradient descent method :

[0015] ℒ plan ( i ) =  i ~  , ϵ ~  ( 0 , I ) , t s ~  [ ‖ ϵ − ϵ i ( t s i , y ′ ( t s ) , i ) ‖ 2 ]

[0016] in, represents the loss function of the diffusion model network without classifier guidance, Indicates the expectation symbol, Represents a discrete set The uniform distribution on represents the number of diffusion steps, represents predefined noise, ~ represents sampling operation, represents the standard Gaussian distribution, Indicates the The state trajectory sample of the step diffusion, condition By probability Leave it blank and record it as:

[0017]

[0018] in, Represents the cumulative return corresponding to the state trajectory sample , represents the empty set, Indicates the accumulation of rewards.

[0019] Furthermore, the decision transformer model The training process is:

[0020] First, initialize the decision Transformer network Network parameters ;

[0021] Experience Pool State trajectory samples in Splice with the high-reward sub-goal state to obtain the spliced trajectory sample , based on the spliced trajectory samples, the network parameters Make updates;

[0022] The updating method is to minimize the following loss function by gradient descent method. :

[0023] ℒ D2SP ( ϕ ) =  t ¯ ~  [ 1 K ∑ j = t − K + 1 t ( a j − π ϕ ( t ¯ ) j ) ]

[0024] in, represents the loss function of the decision Transformer network, Indicates the trajectory Find the expected value of , represents the sequence length, Indicates in The action sampled from the experience pool in time steps, Indicates the decision transformer network in the The action output of each time step.

[0025] Furthermore, the network parameters The number of updates is times, the network parameters The number of updates is Second-rate.

[0026] Furthermore, the intelligent agent is an unmanned vehicle; the state Including vehicle information and surrounding environment status;

[0027] The vehicle's own information at least includes the vehicle's position in the latitude and longitude coordinate system ( ), vehicle speed in the latitude and longitude coordinate system ( ), vehicle movement direction, steering wheel rotation angle The surrounding environment state includes at least the distance and moving speed of other objects in the environment relative to the vehicle, the road structure, and traffic lights;

[0028] action At least including vehicle lateral and longitudinal control systems Among them, the steering wheel rotation angle To correspond to the lateral offset of the vehicle body; Indicates the throttle depth to correspond to the vehicle acceleration information.

[0029] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0030] To address the problem of insufficient trajectory splicing capabilities of conditional sequence modeling methods in offline reinforcement learning, this paper proposes a decision transformer offline reinforcement learning method based on diffusion state planning. (1) This method reconstructs the trajectory representation and introduces high-reward sub-goal states into the trajectory, enabling the decision transformer to generate sequences based on the existing trajectory and high-reward sub-goal states. (2) This method uses a diffusion model without classifier guidance to generate high-reward sub-goal states, which are used to guide the decision transformer to generate action sequences. (3) When faced with a low-quality dataset, this method can generate high-quality action sequences based on the high-reward sub-goal states through the decision transformer. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a structural diagram of the decision-making Transformer offline reinforcement learning method based on diffusion state planning. DETAILED DESCRIPTION

[0032] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0033] The present invention is based on the decision transformer offline reinforcement learning method of diffusion state planning, the process is as follows Figure 1 Given that the existing decision transformer method for conditional sequence modeling lacks trajectory splicing capabilities and has difficulty in making reasonable credit assignments, we reconstruct the trajectory representation; then, we use a diffusion model without classifier guidance to plan a high-reward state sequence, thereby obtaining high-reward sub-goal states; finally, we use a decision transformer to generate the optimal action sequence based on the existing trajectory and high-reward sub-goal states.

[0034] The present invention provides a decision transformer offline reinforcement learning method based on diffusion state planning, which includes the following specific steps:

[0035] Step 1: Build a diffusion model network without classifier guidance and decision Transformer network , and initialize the network parameters.

[0036] Building a diffusion model network without classifier guidance , Decision Transformer Network ; Initialize the diffusion model network Network parameters , initialize the decision Transformer network Network parameters ; Diffusion model network and decision Transformer network The network parameters are initialized randomly.

[0037] Diffusion model without classifier guidance It includes two hidden layers, the structures of the two hidden layers are the same, and each hidden layer adopts a multi-layer perceptron structure with 256 neurons.

[0038] The decision Transformer network It includes four hidden layers, and the structures of the four hidden layers are the same. Each hidden layer adopts a causal Transformer structure with 1024 neurons. In the application field of unmanned driving, in this specific embodiment, the intelligent agent is an unmanned vehicle, and the state Including vehicle information and surrounding environment status;

[0039] The vehicle's own information at least includes the vehicle's position in the latitude and longitude coordinate system ( ), vehicle speed in the latitude and longitude coordinate system ( ), vehicle movement direction (0~259°), steering wheel rotation angle The surrounding environment state includes at least the distance and moving speed of other objects in the environment relative to the vehicle, the road structure, and traffic lights. When the vehicle is started, all sensors are powered on and the angle obtained by initialization is defined as the vehicle body moving direction as 0.

[0040] action At least including vehicle lateral and longitudinal control systems Among them, the steering wheel rotation angle To correspond to the lateral offset of the vehicle body; Indicates the throttle depth to correspond to the vehicle acceleration information.

[0041] Step 2: Update the diffusion model network without classifier guidance Network parameters .

[0042] Adopting experience pool State trajectory samples in Diffusion model network without classifier guidance Network parameters Update is performed by minimizing the following loss function through gradient descent method :

[0043] ℒ plan ( i ) =  i ~  , ϵ ~  ( 0 , I ) , t s ~  [ ‖ ϵ − ϵ i ( t s i , y ′ ( t s ) , i ) ‖ 2 ]

[0044] in, represents the loss function of the diffusion model network without classifier guidance, Indicates the expectation symbol, Represents a discrete set The uniform distribution on represents the number of diffusion steps, represents predefined noise, ~ represents sampling operation, represents the standard Gaussian distribution, Indicates the The state trajectory sample of the step diffusion, ∈ ,condition By probability Leave it blank and record it as:

[0045]

[0046] in, Represents the cumulative return corresponding to the state trajectory sample , represents the empty set, Indicates the accumulation of rewards.

[0047] Step 3: Update the decision transformer network Network parameters .

[0048] Experience Pool State trajectory samples in Splice with the high-reward sub-goal state to obtain the spliced trajectory sample , based on the spliced trajectory samples, the decision Transformer network Network parameters Make updates;

[0049] The updating method is to minimize the following loss function by gradient descent method. :

[0050] ℒ D2SP ( ϕ ) =  t ¯ ~  [ 1 K ∑ j = t − K + 1 t ( a j − π ϕ ( t ¯ ) j ) ]

[0051] in, represents the loss function of the decision Transformer network, Indicates the trajectory Find the expected value of , represents the sequence length, Indicates in The action sampled from the experience pool in time steps, Indicates the decision transformer network in the The action output of each time step.

[0052] Step 4: Repeat steps 2 to 3 until the set number of times is reached to obtain the optimal strategy.

[0053] Update network parameters The updated diffusion model without classifier guidance is used to plan the high-reward target sub-state, thereby guiding the decision-making Transformer network to generate the optimal action sequence.

[0054] The method of the present invention is an offline reinforcement learning method for sequence modeling. This method can use the high-reward target sub-states planned by a diffusion model without classifier guidance to guide the decision transformer network to generate the optimal action sequence. Specifically, first, given that the existing decision transformer method for conditional sequence modeling lacks trajectory splicing capabilities and has difficulty in making reasonable credit allocations, the trajectory representation is reconstructed; then, a diffusion model without classifier guidance is used to plan the high-reward state sequence, thereby obtaining the high-reward sub-target state; finally, a decision transformer is used to generate the optimal action sequence based on the existing trajectory and high-reward sub-target state.

[0055] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A decision transformer offline reinforcement learning method based on diffusion state planning, characterized by: The following steps are involved: The state trajectory of the agent Input to the trained diffusion model without classifier guidance In the process, a high-reward sub-goal state sequence is planned ,in, Indicates the accumulation of rewards. Indicates status, represents the sequence length, represents the high-reward sub-goal state; The high reward sub-goal state sequence and state trajectory Perform splicing to obtain the spliced state trajectory ; Indicates action, The spliced state trajectory Input to the trained decision Transformer model , generate the optimal action sequence; The diffusion model without classifier guidance The training process is: First, initialize the diffusion model network without classifier guidance Network parameters ; Then use the experience pool State trajectory samples in The network parameters Update is performed by minimizing the following loss function through gradient descent method: : ； in, represents the loss function of the diffusion model network without classifier guidance, Indicates the expectation symbol, Represents a discrete set The uniform distribution on represents the number of diffusion steps, represents predefined noise, ~ represents sampling operation, represents the standard Gaussian distribution, Indicates the The state trajectory sample of the step diffusion, condition By probability Leave it blank and record it as: ； in, Represents the cumulative return corresponding to the state trajectory sample , represents the empty set, Indicates the accumulation of rewards; For the decision Transformer model The training process is: First, initialize the decision Transformer network Network parameters ; Experience Pool State trajectory samples in Splice with the high-reward sub-goal state to obtain the spliced trajectory sample , based on the spliced trajectory samples, the network parameters Make updates; The updating method is to minimize the following loss function by gradient descent method. : ； in, represents the loss function of the decision Transformer network, Indicates the trajectory Find the expected value of , represents the sequence length, Indicates in The action sampled from the experience pool in time steps, Indicates the decision transformer network in the The action output of each time step.

2. The decision transformer offline reinforcement learning method based on diffusion state planning according to claim 1, characterized in that: The diffusion model without classifier guidance The system comprises two hidden layers, the structures of the two hidden layers are the same, and each hidden layer adopts a multi-layer perceptron structure with 256 neurons.

3. The decision transformer offline reinforcement learning method based on diffusion state planning according to claim 1, characterized in that: The decision Transformer network The system includes four hidden layers, all of which have the same structure. Each hidden layer adopts a causal Transformer structure with 1024 neurons.

4. The decision transformer offline reinforcement learning method based on diffusion state planning according to claim 1, characterized in that: The network parameters The number of updates is Second-rate.

5. The decision transformer offline reinforcement learning method based on diffusion state planning according to claim 1, characterized in that: The network parameters The number of updates is Second-rate.

6. The decision transformer offline reinforcement learning method based on diffusion state planning according to claim 1, characterized in that: The intelligent agent is an unmanned vehicle; the state Including vehicle information and surrounding environment status; The vehicle's own information at least includes the vehicle's position in the latitude and longitude coordinate system ( ), vehicle speed in the latitude and longitude coordinate system ( ), vehicle movement direction, steering wheel rotation angle ; The surrounding environment state includes at least the distance and moving speed of other objects in the environment relative to the vehicle, the road structure, and traffic lights; action At least including vehicle lateral and longitudinal control systems ; Among them, the steering wheel rotation angle To correspond to the lateral offset of the vehicle body; Indicates the throttle depth, corresponding to the vehicle acceleration information.

Images