Error correction method based on Delta robot virtual-real migration
By training the Delta robot model in a virtual environment and using the PPO2 algorithm and OPC communication protocol, errors are corrected in real time, solving the operating accuracy and stability of DELTA robots in a dynamic environment, achieving efficient and reliable error correction.
Patent Information
- Application Number
- CN202510743630.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-05
- Publication Date
- 2025-08-29
AI Technical Summary
Traditional error correction methods are difficult to adapt to dynamically changing environments in real time, and are costly, resulting in a decrease in operating accuracy and stability of DELTA robots in virtual and real migration.
The Delta robot model is built in a virtual environment, the PPO2 algorithm is used to train the agent, and the real environment information is perceived in real time with the OPC communication protocol, and dynamic error correction is achieved through sparse rewards and intermediate reward correction.
Perceive environmental changes in real time, reduce control deviations after virtual and real migration, improve the operation accuracy and stability of Delta robots in the actual environment, and reduce training costs.
Smart Images
Figure CN120552057A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of robot virtual-reality migration path planning error correction, and in particular relates to an error correction method based on Delta robot virtual-reality migration. Background Art
[0002] With the rapid development of industrial automation technology, DELTA robots, thanks to their high speed, high precision, and high flexibility, have been widely used in industrial automation scenarios such as sorting, packaging, and assembly. To improve production efficiency and reduce commissioning time, digital twin technology has been introduced into the DELTA robot's control system. By building a digital twin model of the robot in a virtual environment, its motion trajectory and operation algorithms are pre-trained. The optimized algorithms are then directly applied to the actual production line through virtual-to-real migration technology. This approach significantly shortens the production line commissioning cycle and improves the robot's operational efficiency.
[0003] However, in actual operation, the DELTA robot is affected by environmental factors such as air resistance and friction, as well as the precision limitations of machine vision technology. This results in an error between the actual landing point of the end effector and the target position. This error not only reduces the robot's operational accuracy and stability, but also causes the control algorithm after virtual-to-real migration to deviate in the real environment, making it difficult to fully achieve the optimization effect in the virtual environment. Traditional error correction methods often rely on offline calibration or complex sensor feedback, making it difficult to adapt to dynamically changing environments in real time and are relatively costly. Summary of the Invention
[0004] In view of the shortcomings of the existing technology, the present invention provides an error correction method based on the virtual-reality migration of a Delta robot to solve the problems that traditional error correction methods are difficult to adapt to dynamically changing environments in real time and are relatively costly.
[0005] The technical solution of the present invention is:
[0006] An error correction method based on virtual-reality migration of a Delta robot includes the following steps:
[0007] Constructing a virtual environment; the virtual environment includes a workspace and a Delta robot model; the workspace includes a plurality of obstacles, the obstacles having the same size and position as obstacles in the real environment; the Delta robot model is used to simulate the movement and position of the Delta robot in the real environment in the workspace;
[0008] Treat the Delta robot model in the virtual environment as an intelligent agent and define the state space, action space and reward function;
[0009] Based on the state space, action space and reward function, the PPO2 algorithm is used to train the intelligent agent in a virtual environment until the loss function converges, and the trained intelligent agent is obtained;
[0010] Combined with the Delta robot in the real environment, the PPO2 algorithm is used to further train the trained intelligent agent to achieve error correction of the trained intelligent agent and obtain the final trained intelligent agent;
[0011] The final trained agent is used for path planning to obtain the path planning results, and the Delta robot in the real environment executes according to the path planning results.
[0012] Furthermore, the virtual environment receives visual information sent in real time from the real environment through the OPC communication protocol, including the size and position of obstacles in the real environment, the three-dimensional coordinates of the center point of the Delta robot's moving platform, and the target point position. The workspace is constructed according to the size and position of obstacles in the real environment, and the Delta robot model in the virtual environment is constructed according to the physical parameters of the Delta robot in the real environment.
[0013] Furthermore, the action space is a=[n0, n1, n2], where n0, n1 and n2 respectively represent the rotation angles of the Delta robot model on three axes in the virtual environment.
[0014] Furthermore, the state space is s=[n0, n1, n2, x, y, z], where (x, y, z) represents the three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the current virtual environment.
[0015] Furthermore, the reward function is constructed as follows:
[0016] First, set up sparse rewards, which are divided into four cases:
[0017] 1. The three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment exceed the specified working area;
[0018] 2. The moving platform of the Delta robot model in the virtual environment collides with an obstacle in the workspace;
[0019] 3. The three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment reach the target range; the target range is the area at a set distance from the target point;
[0020] 4. After the Delta robot completes a path planning in a real environment, there is an error between the final landing point of the center point of the Delta robot's moving platform in the real environment and the center point of the target range;
[0021] The sparse reward is expressed as:
[0022] R s =r1+r2+r3+r4
[0023] Among them, R s represents the sparse reward, r1 represents the penalty obtained when the three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment exceed the specified working area; r2 represents the penalty obtained when the moving platform of the Delta robot model in the virtual environment collides with an obstacle in the working area; r3 represents the reward obtained when the three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment reach the target range; r4 represents the error penalty when the final landing point of the center point of the moving platform of the Delta robot in the real environment has an error with the center point of the target range. The formula is as follows:
[0024] r4=-f·A·([l a ,l n ]·l e ) 3
[0025]
[0026] Among them, f is used to distinguish the operating environment. When the agent runs in the real environment, f is 1, and when the agent runs in the virtual environment, f is 0; A is the amplification coefficient, [l a ,l b ] is a multiplier, l e The difference between the two-dimensional coordinates of the center point of the target range and the two-dimensional coordinates of the final landing point of the center point of the moving platform of the Delta robot in the real environment;
[0027] Second, set the intermediate reward to be:
[0028]
[0029] Among them, R t represents the intermediate reward, D represents the Euclidean distance between the center of the Delta robot's moving platform and the center of the target range, r s is the step penalty; (x0, y0, z0) represents the three-dimensional coordinates of the center point of the starting moving platform; (x d ,y d ,z d ) represents the three-dimensional coordinates of the target point, μ is the weight value, β is the threshold, r e represents error penalty;
[0030] The error penalty is expressed as:
[0031] r e =f·Ceλj ([l a ,l b ,l c ]l) 3
[0032] Where C represents the proportional coefficient, j represents the jth action in the action sequence, and λ is the adjustment factor; [l a ,l b ,l c ] is the multiplier, and l is the error vector for each step:
[0033]
[0034] Among them, (x', y', z') is the three-dimensional coordinate of the center point of the moving platform of the Delta robot in the real environment;
[0035] Finally, based on sparse rewards, intermediate rewards, and error penalties, the reward function is defined as:
[0036] R=R s +R t .
[0037] Furthermore, when the PPO2 algorithm is used to train the agent in the virtual environment, the error penalty r4 when the final landing point of the center point of the moving platform of the Delta robot in the real environment and the center point of the target range are not considered in the sparse reward, and the error penalty r e .
[0038] Furthermore, when the PPO2 algorithm is used to further train the trained agent in combination with the Delta robot in the real environment, the Delta robot model in the virtual environment and the Delta robot in the real environment simultaneously execute the action output by the agent, and the error penalty r4 and error penalty r are considered when the final landing point of the center point of the moving platform of the Delta robot in the real environment and the center point of the target range have an error in the sparse reward. e .
[0039] Compared with the prior art, the present invention has the following beneficial effects:
[0040] This paper uses machine vision technology to construct a high-precision DELTA robot model in a virtual environment and utilizes the OPC communication protocol to achieve real-time communication between the virtual and real environments. Furthermore, a dynamic error correction algorithm is proposed that can sense environmental changes and adjust control parameters in real time, effectively reducing control deviations after virtual-to-real migration and improving the operational accuracy and stability of the Delta robot in a real-world environment. This invention not only fills a technological gap in error correction technology for virtual-to-real migration, but also provides a more efficient and reliable solution for industrial automation production. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 This is a flow chart of an error correction method based on virtual-reality migration of a Delta robot in an embodiment of the present invention. DETAILED DESCRIPTION
[0042] An error correction method based on virtual-real migration of Delta robots, such as Figure 1 As shown, the following steps are included:
[0043] Step 1: Build a virtual environment;
[0044] The virtual environment includes a workspace and a Delta robot model. The workspace includes a plurality of obstacles, and the obstacles have the same size and position as the obstacles in the real environment. The Delta robot model is used to simulate the movement and position of the Delta robot in the real environment in the workspace.
[0045] The virtual environment receives visual information sent in real time from the real environment through the OPC communication protocol, including the size and position of obstacles in the real environment, the three-dimensional coordinates of the center point of the moving platform of the Delta robot, and the position of the target point. The workspace is constructed according to the size and position of the obstacles in the real environment, and the Delta robot model in the virtual environment is constructed according to the physical parameters (such as size) of the Delta robot in the real environment.
[0046] The method for obtaining visual information is as follows: for the virtual-real migration of the Delta robot, a monocular camera above the Delta robot in the real environment and a binocular camera outside the robot are used in combination with a machine vision algorithm to perceive the real environment and identify visual information; the machine vision algorithm described in this embodiment includes edge detection, target recognition, and geometric feature measurement algorithms; in this method, virtual-real migration provides the three-dimensional center coordinates of the Delta robot's moving platform in the real environment with low latency, providing basic data for subsequent error correction methods; using the OPC communication protocol, communication between the virtual environment and the real environment is realized in Unity, so that instructions in the virtual environment can also be executed in the real environment, facilitating the subsequent migration of the results of training in the virtual environment to the real environment, and realizing virtual-real migration. By training the Delta robot model in the virtual environment through virtual-real migration and then migrating it back to the real environment, the Delta robot in the real environment can basically achieve the training task (in this embodiment, the task is for the center of the Delta robot's moving platform to reach the target point). This effectively avoids the body loss caused by directly using a real robot for training, improves the efficiency of training while reducing the training cost, but there is still a certain error, and a dynamic error correction algorithm will be used to eliminate this error later.
[0047] Step 2: Treat the Delta robot model in the virtual environment as an intelligent agent and define the state space, action space, and reward function;
[0048] The error correction method of this embodiment enables the robot to accurately circumvent obstacles and reach the target point in the obstacle avoidance path planning task implemented by reinforcement learning. It uses the PPO2 algorithm, adds a landing error penalty to the sparse reward, and adds an intermediate error penalty to the reward function to eliminate errors caused by equipment and other factors.
[0049] The Delta robot has three degrees of freedom. The Delta robot can rotate -1 degree, 0 degree, or 1 degree on any axis at its current position. Therefore, the action space in the reinforcement learning algorithm is set to a = [n0, n1, n2], where n0, n1, and n2 represent the angles of rotation of the Delta robot model on the three axes in the virtual environment. If an action vector is [1, 0, -1], it means that the Delta robot rotates 1 degree on the first axis at its current position, the second axis remains stationary, and the third axis rotates -1 degree.
[0050] Define the state space as s = [n0, n1, n2, x, y, z], where (x, y, z) represents the three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the current virtual environment;
[0051] In the reward function setting, we first set the sparse reward, which is divided into four cases:
[0052] 1. The three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment exceed the specified working area;
[0053] 2. The moving platform of the Delta robot model in the virtual environment collides with an obstacle in the workspace;
[0054] 3. The three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment reach the target range; the target range is the area at a set distance from the target point;
[0055] 4. After the Delta robot completes a path planning in a real environment, there is an error between the final landing point of the center point of the Delta robot's moving platform in the real environment and the center point of the target range;
[0056] The sparse reward is expressed as:
[0057] R s =r1+r2+r3+r4
[0058] Among them, R s represents the sparse reward, r1 represents the penalty obtained when the three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment exceed the specified working area; r2 represents the penalty obtained when the moving platform of the Delta robot model in the virtual environment collides with an obstacle in the working area; r3 represents the reward obtained when the three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment reach the target range; r4 represents the error penalty when the final landing point of the center point of the moving platform of the Delta robot in the real environment has an error with the center point of the target range. The formula is as follows:
[0059] r4=-f·A·([l a ,l b ]·l e ) 3
[0060]
[0061] Among them, f is used to distinguish the operating environment. When the agent runs in the real environment, f is 1, and when the agent runs in the virtual environment, f is 0; A is the amplification coefficient, [l a ,l b ] is a multiplier used to correct the error, l eThe difference between the two-dimensional coordinates of the center point of the target range and the two-dimensional coordinates of the final landing point of the center point of the Delta robot's moving platform in the real environment. At this time, the center point of the Delta robot's moving platform in the real environment falls on the horizontal plane, so the coordinates of the center point of the moving platform are two-dimensional coordinates. When running in a real environment, the landing point error of a single action sequence is most obvious. Machine vision technology can effectively detect errors. Including a landing point error penalty in the sparse reward during training can significantly improve training efficiency.
[0062] At the same time, in order to avoid insufficient driving force due to sparse rewards, intermediate rewards are added during the exploration process of the Delta robot to guide the Delta robot to approach the target range. The intermediate rewards are expressed as:
[0063]
[0064] Among them, R t represents the intermediate reward, D represents the Euclidean distance between the center of the Delta robot's moving platform and the center of the target range; in order to allow the Delta robot to gradually learn a path with fewer steps, a part of the score of the Delta robot should be reduced for each step, that is, the step penalty r s ; (x0, y0, z0) represents the three-dimensional coordinates of the center point of the starting moving platform; (x d ,y d ,z d ) represents the three-dimensional coordinates of the target point, μ is the weight value, β is the threshold, r e is the error penalty;
[0065] In order to further eliminate the errors between virtual environment and real environment training caused by equipment and other reasons, and to achieve higher accuracy, the score reduced by the error needs to be subtracted from the intermediate reward link. The error penalty is expressed as:
[0066] r e =f·Ce λj ([l a ,l b ,l c ]l) 3
[0067] Among them, f means that r is not added during training in the virtual environment e Training is performed, while adding r when training in the real environment e Training is performed; C represents the proportional coefficient; an action sequence is generated during training, and j represents the jth action in the action sequence. As the number of actions in the action sequence increases, the error caused by equipment and other factors becomes larger and larger, and the accuracy of the camera is limited. The error of the later steps in the action sequence is easier to identify, so e is introduced. λj To amplify this error, where λ is the adjustment factor; [la ,l b ,l c ] is a multiplier used to correct errors. It is multiplied by the cube to weaken the small errors caused by the accuracy of the visual technology, and it also amplifies the larger errors in the second half of the sequence. l is the error vector of each step:
[0068]
[0069] Among them, (x', y', z') is the three-dimensional coordinate of the center point of the moving platform of the Delta robot in the real environment;
[0070] Each time the agent makes a decision, the reward function formula it obtains is as follows:
[0071] R=R s +R t
[0072] Step 3: Based on the state space, action space, and reward function, the agent is trained using the PPO2 algorithm in a virtual environment until the loss function converges, resulting in a trained agent.
[0073] When training in a virtual environment, the error penalty r4 is not considered when the final landing point of the center point of the moving platform of the Delta robot in the real environment and the center point of the target range have an error. At the same time, the error penalty r is not considered. e ;
[0074] The PPO2 algorithm improves training stability by simultaneously clipping the policy and value functions, and can control the update amplitude of the model to avoid training instability caused by excessive updates. At the same time, the PPO2 algorithm performs multiple updates on each data batch, which can more effectively utilize each sample and improve sample efficiency.
[0075] The Delta robot's obstacle avoidance path planning is performed using the PPO2 algorithm, and a fully connected neural network with two hidden layers is used to represent the state value function.
[0076] The PPO2 algorithm update method is the gradient descent method, and the loss function formula is as follows:
[0077]
[0078]
[0079] Among them, Loss ppo2 is the loss function of PPO2, and the strategy is updated by minimizing the loss through gradient descent; n is the sample number; N is the number of samples in the batch data; t is the time step; T n is the total number of time steps; For generalized advantage estimation, measure the action In state Advantage value under ; For the current strategy in state Select Action probability; For the old policy in state Select Action probability; is the probability ratio of the new and old strategies, which is used to measure the magnitude of the strategy update; clip is the truncation function; ε is the truncation threshold, which limits the range of the probability ratio; θ represents the current parameters of the strategy network; θ' represents the old parameters of the strategy network; s t is the state; b is the offset of the future time step; γ is the discount factor; λ is the GAE hyperparameter; is the time series difference residual; V θ (s t ) is the value function;
[0080] Step 4: Combined with the Delta robot in the real environment, the PPO2 algorithm is used to further train the trained intelligent agent to correct the errors of the trained intelligent agent and obtain the final trained intelligent agent;
[0081] When the trained agent is further trained by combining the Delta robot in the real environment, the Delta robot model in the virtual environment and the Delta robot in the real environment simultaneously execute the actions output by the agent. Since the Delta robot in the real environment is affected by environmental factors such as air resistance and friction as well as the accuracy limitation of machine vision technology, the error penalty r4 and error penalty r4 are considered when the final landing point of the center point of the moving platform of the Delta robot in the real environment and the center point of the target range have an error in the sparse reward. e ;
[0082] Step 5: Use the final trained agent to perform path planning and obtain the path planning result. The Delta robot in the real environment executes according to the path planning result. This path planning result takes into account the influence of environmental factors such as air resistance and friction on the Delta robot as well as the accuracy limitations of machine vision technology, and realizes error correction.
[0083] Example 1: The reward function in the single-step PPO2 algorithm is calculated based on the following situation when the trained agent is further trained by a Delta robot in a real environment:
[0084] In this embodiment, A=0.0125, C=0.00093, μ=1.50, λ=-0.008, [l a,l b ,l c ]=5.82,4.69,3.25],[l a ,l b ]=[5.82,4.69],f=1,β=1,reach target range r3=2000,step penalty r s = 20, in one training round, (x0, y0, z0) = (0, 0, 0), (x d ,y d ,z d )=(37.54,-55.87,-120);
[0085] 1. At the 33rd step, the vehicle did not exceed the working area, collide with any obstacles, or reach the target area;
[0086]
[0087] Obviously, r1=r2=r3=r4=0, so R s =0, calculated Substitute into the formula r e =f·Ce λj ([l a ,l b ,l c ]l) 3 r e =22.00;
[0088] Substituting into the formula:
[0089]
[0090] R t =-29.97, R =-17.94;
[0091] 2. At the 40th step, the target range is reached without exceeding the working area or colliding with any obstacles;
[0092]
[0093] Obviously, r1=r2=0, r3=2000, so we can get Substitute into the formula r4=-f·A·([l a ,l b ]·l e ) 3 We can get r4=-614.85, and R s =1385.15; calculated Substitute into the formula r e =f·Ce λj ([l a ,l b,l c ]l) 3 r e =36.60;
[0094] Substituting into the formula:
[0095]
[0096] R t =149.79, R=1534.94.
Claims
1. An error correction method based on virtual-real migration of Delta robots, characterized in that: The following steps are involved: Constructing a virtual environment; the virtual environment includes a workspace and a Delta robot model; the workspace includes a plurality of obstacles, the obstacles having the same size and position as obstacles in the real environment; the Delta robot model is used to simulate the movement and position of the Delta robot in the real environment in the workspace; Treat the Delta robot model in the virtual environment as an intelligent agent and define the state space, action space and reward function; Based on the state space, action space and reward function, the PPO2 algorithm is used to train the intelligent agent in a virtual environment until the loss function converges, and the trained intelligent agent is obtained; Combined with the Delta robot in the real environment, the PPO2 algorithm is used to further train the trained intelligent agent to achieve error correction of the trained intelligent agent and obtain the final trained intelligent agent; The final trained agent is used for path planning to obtain the path planning results, and the Delta robot in the real environment executes according to the path planning results.
2. The error correction method based on virtual-real migration of a Delta robot according to claim 1, characterized in that: The virtual environment receives visual information sent in real time from the real environment through the OPC communication protocol, including the size and position of obstacles in the real environment, the three-dimensional coordinates of the center point of the Delta robot's moving platform, and the target point position. The workspace is constructed according to the size and position of obstacles in the real environment, and the Delta robot model in the virtual environment is constructed according to the physical parameters of the Delta robot in the real environment.
3. The error correction method based on virtual-real migration of a Delta robot according to claim 1, characterized in that: The action space is a=[n0, n1, n2], where n0, n1 and n2 respectively represent the rotation angles of the Delta robot model on three axes in the virtual environment.
4. The error correction method based on virtual-real migration of a Delta robot according to claim 1, characterized in that: The state space is s=[n0, n1, n2, x, y, z], where (x, y, z) represents the three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the current virtual environment.
5. The error correction method based on virtual-real migration of a Delta robot according to claim 1, characterized in that: The construction process of the reward function is as follows: First, set up sparse rewards, which are divided into four cases: (1) The three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment exceed the specified working area; (2) The moving platform of the Delta robot model in the virtual environment collides with an obstacle in the workspace; (3) The three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment reach the target range; the target range is an area at a set distance from the target point; (4) After the Delta robot in the real environment completes a path planning, an error occurs between the final landing point of the center point of the moving platform of the Delta robot in the real environment and the center point of the target range; The sparse reward is expressed as: R s ＝r1+r2+r3+r4 Among them, R s represents the sparse reward, r1 represents the penalty obtained when the three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment exceed the specified working area; r2 represents the penalty obtained when the moving platform of the Delta robot model in the virtual environment collides with an obstacle in the working area; r3 represents the reward obtained when the three-dimensional coordinates of the center point of the moving platform of the Delta robot model in the virtual environment reach the target range; r4 represents the error penalty when the final landing point of the center point of the moving platform of the Delta robot in the real environment has an error with the center point of the target range. The formula is as follows: r4=-f·A·([l a ,L b ]·L e ) 3 Among them, f is used to distinguish the operating environment. When the agent runs in the real environment, f is 1, and when the agent runs in the virtual environment, f is 0; A is the amplification coefficient, [l a ,l b ] is a multiplier, l e The difference between the two-dimensional coordinates of the center point of the target range and the two-dimensional coordinates of the final landing point of the center point of the moving platform of the Delta robot in the real environment; Second, set the intermediate reward to be: Among them, R t represents the intermediate reward, D represents the Euclidean distance between the center of the Delta robot's moving platform and the center of the target range, r s is the step penalty; (x0, y0, z0) represents the three-dimensional coordinates of the center point of the starting moving platform; (x d ,y d ,z d ) represents the three-dimensional coordinates of the target point, μ is the weight value, β is the threshold, r e represents error penalty; The error penalty is expressed as: R e =f·What λj ([it a ,it b ,it c ]it) 3 Where C represents the proportional coefficient, j represents the jth action in the action sequence, and λ is the adjustment factor; a ,l b ,l c ] is the multiplier, and l is the error vector for each step: Among them, (x', y', z') is the three-dimensional coordinate of the center point of the moving platform of the Delta robot in the real environment; Finally, based on sparse rewards, intermediate rewards, and error penalties, the reward function is defined as: R=R s +R t 。 6. The error correction method based on virtual-real migration of a Delta robot according to claim 1, characterized in that: When the PPO2 algorithm is used to train the agent in the virtual environment, the error penalty r4 when the final landing point of the center point of the moving platform of the Delta robot in the real environment and the center point of the target range are not considered in the sparse reward, and the error penalty r is not considered. e .
7. The error correction method based on virtual-real migration of a Delta robot according to claim 1, characterized in that: When the PPO2 algorithm is used to further train the trained agent in combination with the Delta robot in the real environment, the Delta robot model in the virtual environment and the Delta robot in the real environment simultaneously execute the action output by the agent, and the error penalty r4 and error penalty r are considered when the final landing point of the center point of the moving platform of the Delta robot in the real environment and the center point of the target range have an error in the sparse reward. e .