Target-driven reward generation system and method for reinforcement learning
By generating target condition videos and combining video-level and frame-level reward signals, the problems of sparse reward signals and feedback delays in existing technologies are solved, enabling reinforcement learning agents to learn efficiently, robustly, and autonomously in complex visual tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-01
- Publication Date
- 2026-04-03
AI Technical Summary
The lack of existing technologies has led to the inefficiency of reinforcement learning agents in learning complex visual tasks, making it difficult to achieve efficient, robust and autonomous learning.
The target conditional video is generated by a pre-trained video generation model. The correlation between the agent's observation sequence and the target video is calculated. The video-level and frame-level reward signals are combined and the environmental reward signal is fused to generate a comprehensive reward signal. The reward is calculated using a forward-backward representation learning mechanism.
It achieves high-level behavior guidance and fine-grained reward feedback, reduces reliance on domain expert knowledge, improves the scalability and learning efficiency of reinforcement learning in complex tasks, ensures that policy optimization does not deviate from the task objective, and makes the learning process more stable and efficient.
Smart Images
Figure CN121787500A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of reinforcement learning technology, and more specifically, to a goal-driven reward generation system and method for reinforcement learning. Background Technology
[0002] In reinforcement learning, the reward function is the core mechanism guiding an agent to learn a desired policy. However, designing an effective reward function has long been a significant challenge. In current practice, reward functions mostly rely on the knowledge of domain experts for manual design and tuning. This approach is not only time-consuming and labor-intensive, but also highly subjective, and it is difficult to guarantee its effectiveness in complex or novel tasks. Manually designed rewards are often sparse (e.g., rewarded only upon task completion), making it difficult for the agent to obtain effective learning signals in the early stages of exploration, resulting in a slow and inefficient learning process.
[0003] To reduce reliance on manual design, existing technologies have proposed several improved methods. For example, one type of method utilizes expert demonstration data to construct rewards by calculating the similarity between the agent's state and the expert's state. Another type attempts to leverage pre-trained generative models (such as image diffusion models) to evaluate the consistency between the current state and the goal. However, these methods have significant limitations. First, they mostly rely on static single-frame images or state features, failing to fully utilize video sequences containing rich temporal logic and causal relationships as target references, making it difficult to define and evaluate complex, temporally dependent behaviors. Second, the reward signals provided by these methods are often single-level, either macroscopic but sparse task completion signals or fragmented but lacking long-term planning local rewards, failing to provide coordinated guidance for both the agent's long-term strategic planning (video level) and short-term action execution (frame level).
[0004] Therefore, existing technologies lack a systematic solution capable of automatically parsing high-level task intent, generating realistic target video sequences as a macroscopic blueprint, and decomposing key sub-targets based on this blueprint to provide dense and hierarchical reward signals. This deficiency makes it difficult for reinforcement learning agents to achieve efficient, robust, and autonomous learning when faced with complex visual tasks requiring long-term planning and fine control. Summary of the Invention
[0005] The technical problem to be solved by this invention is how to enable reinforcement learning agents to learn efficiently, robustly and autonomously when faced with complex visual tasks that require long-range planning and fine control. In order to overcome the defects of the above-mentioned prior art (or related technologies), this invention provides a goal-driven reward generation system and method for reinforcement learning.
[0006] This invention provides a goal-driven reward generation method for reinforcement learning, comprising the following steps: Step S1: Based on the task description provided by the user, generate the target conditional video using a pre-trained video generation model; Step S2: Calculate the correlation between the agent observation sequence in the latent space and the target conditional video to obtain the video-level reward signal; Step S3: Select key frames from the target condition video, and calculate the frame-level reward signal related to the key frames in the target condition video based on the forward-backward representation learning mechanism; Step S4: The video-level reward signal, the frame-level reward signal, and the original task reward signal provided by the environment are fused to generate a comprehensive reward signal for policy optimization of the reinforcement learning agent.
[0007] Compared with existing technologies, the goal-driven reward generation method for reinforcement learning proposed in this invention has the following advantages: This invention provides both high-level behavior guidance and fine-grained single-step evaluation by combining video-level and frame-level reward signals, effectively solving the problems of sparsity and feedback delay in traditional reward functions. It utilizes a pre-trained video generation model to understand the task description and automatically generate target condition videos, greatly reducing reliance on domain expert knowledge and manual debugging, and improving the scalability of reinforcement learning in complex tasks. By fusing the original reward signals from the environment, it ensures that policy optimization does not deviate from the basic task objective, while using generated rewards for guidance makes the learning process more stable and efficient. This enables reinforcement learning agents to learn efficiently, robustly, and autonomously when facing complex visual tasks requiring long-range planning and fine control.
[0008] In one possible implementation, in step S1, a large-scale video dataset corresponding to the task description is obtained and input into the video generation model CogVideoX to align the video-level target and the frame-level target to generate the target conditional video.
[0009] Compared with existing technologies, the above-mentioned technical solution can utilize large-scale video datasets and the powerful CogVideoX video generation model to generate high-quality target condition videos that are highly consistent with the task description, providing an accurate and reliable reference benchmark for subsequent reward calculation and ensuring the quality of the reward signal from the source.
[0010] In one possible implementation, the video generation model CogVideoX used in step S1 is constructed through a video diffusion transformer architecture. The video diffusion transformer architecture uses a three-dimensional variational autoencoder to map the large-scale video dataset into segmented video latent features, and performs a diffusion process in the latent space based on the video latent features to train the pre-trained video generation model.
[0011] Compared with existing technologies, the above-mentioned technical solution efficiently processes and generates video data through a video diffusion transformer architecture and a three-dimensional variational autoencoder, and optimizes it in a semantically rich latent space. This makes the generated target conditional video not only temporally coherent, but also contains rich world knowledge, laying the foundation for extracting meaningful reward signals.
[0012] In one possible implementation, in step S2, a three-dimensional causal variational autoencoder is used to map the agent's observation sequence and the target conditional video into the latent space to obtain the observation latent vector and the target video latent vector, respectively. Then, the cosine similarity between the observation latent vector and the target video latent vector is calculated as the video-level reward signal.
[0013] Compared with existing technologies, the above-mentioned technical solution can calculate cosine similarity in the latent space rather than the pixel space. The reward signal is not sensitive to changes in visual appearance, but focuses more on the similarity of high-level semantics and behavior. This greatly improves the robustness and discriminative ability of the reward signal, guiding the agent to learn the essence of behavior rather than the surface pixel patterns.
[0014] In one possible implementation, in step S3, the similarity between each video frame in the target conditional video and the task description is calculated using a pre-trained cross-modal model, and the video frame with the highest similarity is selected as the key frame.
[0015] Compared with existing technologies, the above technical solution can automatically and efficiently locate the most critical target moment from the target condition video. By calculating the similarity with the task description, it ensures that the selected keyframes are strongly correlated with the final task target, avoiding the subjectivity and tediousness of manual selection.
[0016] In one possible implementation, the cross-modal model used in step S3 is the OpenCLIP model.
[0017] In one possible implementation, the calculation process of the frame-level reward signal in step S3 includes: Step A1: Encode the keyframe and the agent observation sequence to obtain semantic representation feature vector and model state feature vector respectively, and input the semantic representation feature vector and the model state feature vector into the pre-constructed forward-backward representation network for learning; Step A2: Optimize the forward-backward representation network by minimizing the Bellman residual, and obtain the frame-level reward signal based on the learned forward-backward reward.
[0018] Compared with existing technologies, the above-mentioned technical solution can quantify the probability of achieving the target state by learning the forward-backward representation network. This provides extremely fine-grained and accurate real-time feedback for each decision step, clearly informing the agent which action is more conducive to achieving the goal, thereby significantly accelerating the convergence process of the policy and helping the agent escape local optima.
[0019] In one possible implementation, in step A1, the keyframe is encoded using the DINOv3 model, and the agent's observation sequence is encoded using the DreamerV3 encoder.
[0020] In one possible implementation, in step S4, the comprehensive reward signal is obtained using the following calculation formula: in, This represents the comprehensive reward signal; This represents the first adjustable weighting coefficient; This represents the video-level reward signal; This represents the second adjustable weighting coefficient; This refers to the frame-level reward signal; This indicates the original task reward signal.
[0021] This invention also provides a goal-driven reward generation system for reinforcement learning, which applies the above-described goal-driven reward generation method and includes: A target video generation module is used to generate target conditional videos based on user-provided task descriptions using a pre-trained video generation model. A video-level reward module, connected to the target video generation module, is used to calculate the correlation between the agent's observation sequence in the latent space and the target conditional video to obtain a video-level reward signal; A frame-level reward module, connected to the video-level reward module, is used to select key frames from the target condition video and calculate the frame-level reward signal related to the key frames in the target condition video based on a forward-backward representation learning mechanism. A reward fusion module, connected to the frame-level reward module, is used to fuse the video-level reward signal, the frame-level reward signal, and the original task reward signal provided by the environment to generate a comprehensive reward signal for policy optimization of the reinforcement learning agent.
[0022] Compared with existing technologies, the goal-driven reward generation system for reinforcement learning proposed in this invention has the following advantages: This invention provides both high-level behavior guidance and fine-grained single-step evaluation by combining video-level and frame-level reward signals, effectively solving the problems of sparsity and feedback delay in traditional reward functions. It utilizes a pre-trained video generation model to understand the task description and automatically generate target condition videos, greatly reducing reliance on domain expert knowledge and manual debugging, and improving the scalability of reinforcement learning in complex tasks. By fusing the original reward signals from the environment, it ensures that policy optimization does not deviate from the basic task objective, while using generated rewards for guidance makes the learning process more stable and efficient. This enables reinforcement learning agents to learn efficiently, robustly, and autonomously when facing complex visual tasks requiring long-range planning and fine control. Attached Figure Description
[0023] Figure 1 This is a flowchart of the method steps of the present invention; Figure 2 This is a diagram of the reinforcement learning reward framework of the present invention; Figure 3 This is a flowchart illustrating the frame-level reward signal calculation process of the present invention. Figure 4 This is a schematic diagram of the system structure of the present invention. Detailed Implementation
[0024] First, those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Those skilled in the art can make adjustments as needed to adapt to specific application scenarios.
[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0026] See Figure 1 and Figure 2 This invention discloses a goal-driven reward generation method for reinforcement learning, comprising: Step S1: Based on the task description provided by the user, generate the target conditional video using a pre-trained video generation model; Step S2: Calculate the correlation between the agent observation sequence in the latent space and the target condition video to obtain the video-level reward signal; Step S3: Select key frames from the target conditional video and calculate the frame-level reward signal related to the key frames in the target conditional video based on the forward-backward representation learning mechanism. Step S4: Fuse the video-level reward signal, the frame-level reward signal, and the original task reward signal provided by the environment to generate a comprehensive reward signal for policy optimization of the reinforcement learning agent.
[0027] In this embodiment of the invention, a large-scale video dataset corresponding to the task description is input into the pre-trained video generation model CogVideoX to generate target conditional videos, so as to achieve accurate alignment of video-level targets and frame-level targets at the same time.
[0028] In this embodiment of the invention, the video generation model CogVideoX is built on a VideoDiffusion Transformers (DiTs) architecture, which uses a three-dimensional variational autoencoder to map the original video data, i.e., the large-scale video dataset, into segmented video latent features. Furthermore, the diffusion process is trained within this potential space, and task-specific conditional information is introduced when fine-tuning the pre-trained video diffusion transformer architecture. ,in This represents the text embedding vector extracted from the text prompts in the task description. This represents the image embedding vector extracted from the image guidance of the task description. During the inference process, the generation result is obtained through the inverse process of the diffusion model. First, Gaussian noise is randomly sampled from the normal distribution, and then the denoised result is restored through iteration.
[0029] In this embodiment of the invention, given that the video encoder of the pre-trained video generation model contains world knowledge, which is helpful for understanding video content, a three-dimensional causal variational autoencoder is used to compress a large-scale video dataset into a latent space. The three-dimensional causal variational autoencoder maps pixel-level inputs to latent feature representations and introduces a Kullback-Leibler (KL) divergence regularization term to enable the model to learn a latent space with a clear semantic structure and good distribution characteristics. Furthermore, in this invention, the agent's observation sequence and the target conditional video are mapped into the latent space to obtain the observation latent vector and the target video latent vector, respectively. Then, the cosine similarity between the observation latent vector and the target video latent vector is calculated as a video-level reward signal. The calculation formula is as follows: in, Indicates a video-level reward signal. Represents the observed latent vector. This represents the latent vector of the target video.
[0030] In this embodiment of the invention, the video-level reward signal, by measuring the similarity between the agent's observation sequence in the latent space and the target conditional video, incentivizes the agent to approach the expert-level behavior exhibited by the target conditional video in terms of temporal evolution and behavioral patterns, thereby promoting the achievement of the video-level goal.
[0031] In this embodiment of the invention, in step S3, the most relevant frames are extracted from the generated target condition video, and the state distribution accessed by the control strategy is aligned with the target state to achieve frame-level behavior optimization. This embodiment uses the OpenCLIP model to calculate the similarity between the video frames of the target condition video and the task description, and selects the video frame with the highest similarity score as the key frame. .
[0032] See Figure 3 In this embodiment of the invention, the calculation process of the frame-level reward signal in step S3 includes: Step A1: Encode the keyframes and agent observation sequences to obtain semantic representation feature vectors and model state feature vectors, respectively, and input the semantic representation feature vectors and model state feature vectors into the pre-constructed forward-backward representation network for learning; Step A2: Optimize the forward-backward representation network by minimizing the Bellman residual, and obtain the frame-level reward signal based on the learned forward-backward reward.
[0033] In this embodiment of the invention, to achieve good generalization ability of the frame-level reward signal under different target conditions, this embodiment learns a forward-backward representation to effectively decompose the long-term state occupancy distribution under arbitrary policies. Before training the forward-backward representation network, the DINOv3 model is used for keyframes. Encode to obtain semantic representation feature vectors Simultaneously, the DreamerV3 encoder was used to observe sequences from the agent. Extracting model state feature vectors The extracted feature vectors are input into the forward-backward representation network for learning.
[0034] In this embodiment of the invention, by minimizing the Bellman residual Optimize the forward-backward representation network, with the optimization objective being: in, Indicates a given z The following is about ( s , a The vector field of ) Indicates backward representation, Represents the distribution of the training data. Indicates the probability of environmental transition. Indicates the discount factor. Indicating in strategy The state transition probability under the following conditions and This represents the target network used for training. Indicates from data distribution The expectation of extracting the transition triplet from the middle. Indicates time t The model state feature vector, Indicates time t The model action feature vector, Indicates time t+1 The model action feature vector, This indicates that sampling is performed from a Gaussian distribution for exploration. The semantic representation of the feature vector is... Indicates based on time t+1 The model action feature vector adopted by the model action feature vector. This represents a constant term that is independent of all learnable parameters.
[0035] In this embodiment of the invention, based on the learned forward-backward representation reward, the calculation formula for the frame-level reward signal is defined as follows: in, Indicates frame-level reward signal, Represents the model state feature vector. Represents the model's action feature vector. Indicates a keyframe. The semantic representation is the feature vector.
[0036] In this embodiment of the invention, the video-level reward signal, the frame-level reward signal, and the original task reward signal provided by the environment are fused using a linear weighting method. The calculation formula is as follows: in, This indicates a comprehensive reward signal; This represents the first adjustable weighting coefficient; Indicates a video-level reward signal; This represents the second adjustable weighting coefficient; Indicates a frame-level reward signal; This represents the original task reward signal. The first and second adjustable weight coefficients are used to balance the contribution of different reward components to strategy optimization.
[0037] See Figure 4 In this embodiment of the invention, a goal-driven reward generation system for reinforcement learning is also disclosed, applied to the above-mentioned goal-driven reward generation method, comprising: A target video generation module is used to generate target conditional videos based on user-provided task descriptions using a pre-trained video generation model. A video-level reward module, connected to the target video generation module, is used to calculate the correlation between the agent's observation sequence in the latent space and the target conditional video to obtain a video-level reward signal; A frame-level reward module, connected to the video-level reward module, is used to select keyframes from the target condition video and calculate the frame-level reward signal related to the keyframes in the target condition video based on the forward-backward representation learning mechanism. A reward fusion module, connected to the frame-level reward module, is used to fuse video-level reward signals, frame-level reward signals, and the original task reward signals provided by the environment to generate a comprehensive reward signal for policy optimization of the reinforcement learning agent.
[0038] In this embodiment of the invention, the video-level reward module uses a fine-tuned video generation model to generate a target conditional video based on the task description, and extracts the latent representations of the agent's observation sequence and the target conditional video through the video encoder of the pre-trained video generation model, and calculates the correlation between the two as a video-level reward signal to achieve the video-level objective.
[0039] In this embodiment of the invention, the frame-level reward module, combined with the cross-modal understanding capability of the CLIP model, automatically selects the keyframes most relevant to the task description. It achieves state reachability through a forward-backward representation learning mechanism, providing immediate reward feedback for single-step decisions and significantly accelerating policy convergence. The collaboration between the video-level and frame-level reward modules enables reinforcement learning agents to obtain more accurate, richer, and interpretable reward signals in complex visual environments. This results in higher sample efficiency, faster learning speed, and better policy performance in goal-driven long-range decision-making tasks, demonstrating broad application prospects and value. This system can be applied to fields requiring precise goal understanding, such as robot vision manipulation, autonomous driving planning, and agent navigation, helping agents make more flexible and accurate decisions when facing diverse tasks, significantly improving the agent's autonomous adaptability and task success rate.
[0040] In this embodiment of the invention, a two-layer target-driven reward generation framework based on a pre-trained video generation model is constructed. Unlike traditional manual design of reward functions or sparse reward schemes based on state distance, this invention proposes for the first time to use the pre-trained video generation model as a reward generator. By fine-tuning it to adapt to the target task, reward signals are automatically generated at both the video level and the frame level. The video-level reward signal is calculated by using the correlation between the latent variables of the generated target condition video and historical visual observations in the latent space of the diffusion model as a dense reward. The frame-level reward signal selects the most relevant video frame from the target condition video as a sub-target to further refine the reward. This technique solves the technical problems in reinforcement learning where reward function design relies on expert experience, is highly subjective, has high sparsity, and is difficult to balance long-term progress and short-term optimization. It achieves end-to-end automated mapping from high-level visual targets to multi-granularity reward signals.
[0041] In this embodiment of the invention, target achievement is measured using latent spatial correlation. Instead of directly measuring similarity in pixel space or state space, the method innovatively uses the deep encoder of a fine-tuned video generation model to map observations and targets to a semantically rich latent space. The reward value is calculated through cosine similarity or other correlation measures. This method overcomes the shortcomings of pixel-level measurement being sensitive to noise and state space measurement relying on manual feature engineering. It improves the ability and robustness of the reward signal to distinguish semantic target achievement and solves the problem that reward calculation is easily interfered with and difficult to capture high-level semantics in the prior art.
[0042] In this embodiment of the invention, a forward-backward representation frame-level reward modeling technique that integrates action information explicitly introduces forward and backward dynamic representations of state-action pairs into the frame-level reward signal. By predicting the probability distribution of achieving the target frame after performing an action from the current state, a more action-sensitive reward function is constructed. Unlike reward designs based solely on state distance, this technique solves the technical problems of insufficient differentiation of key actions by traditional rewards and difficulty in guiding agents to learn efficient strategies. It achieves fine-grained goal achievement evaluation and action optimization, significantly enhancing the ability to guide key decision points.
[0043] In the description of this invention, the references to "one embodiment," "some embodiments," "in this embodiment," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0044] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A goal-driven reward generation method for reinforcement learning, characterized in that, Includes the following steps: Step S1: Based on the task description provided by the user, generate the target conditional video using a pre-trained video generation model; Step S2: Calculate the correlation between the agent observation sequence in the latent space and the target conditional video to obtain the video-level reward signal; Step S3: Select key frames from the target condition video, and calculate the frame-level reward signal related to the key frames in the target condition video based on the forward-backward representation learning mechanism; Step S4: The video-level reward signal, the frame-level reward signal, and the original task reward signal provided by the environment are fused to generate a comprehensive reward signal for policy optimization of the reinforcement learning agent.
2. The goal-driven reward generation method according to claim 1, characterized in that, In step S1, a large-scale video dataset corresponding to the task description is obtained and input into the video generation model CogVideoX to align video-level targets and frame-level targets to generate the target conditional video.
3. The goal-driven reward generation method according to claim 2, characterized in that, The video generation model CogVideoX used in step S1 is constructed through a video diffusion transformer architecture. The video diffusion transformer architecture uses a three-dimensional variational autoencoder to map the large-scale video dataset into segmented video latent features, and performs a diffusion process in the latent space based on the video latent features to train the pre-trained video generation model.
4. The goal-driven reward generation method according to claim 3, characterized in that, In step S2, a three-dimensional causal variational autoencoder is used to map the agent's observation sequence and the target conditional video into the latent space to obtain the observation latent vector and the target video latent vector, respectively. Then, the cosine similarity between the observation latent vector and the target video latent vector is calculated as the video-level reward signal.
5. The goal-driven reward generation method according to claim 1, characterized in that, In step S3, the similarity between each video frame in the target condition video and the task description is calculated using a pre-trained cross-modal model, and the video frame with the highest similarity is selected as the key frame.
6. The goal-driven reward generation method according to claim 5, characterized in that, The cross-modal model used in step S3 is the OpenCLIP model.
7. The goal-driven reward generation method according to claim 1, characterized in that, The calculation process of the frame-level reward signal in step S3 includes: Step A1: Encode the keyframe and the agent observation sequence to obtain semantic representation feature vector and model state feature vector respectively, and input the semantic representation feature vector and the model state feature vector into the pre-constructed forward-backward representation network for learning; Step A2: Optimize the forward-backward representation network by minimizing the Bellman residual, and obtain the frame-level reward signal based on the learned forward-backward reward.
8. The goal-driven reward generation method according to claim 7, characterized in that, In step A1, the keyframes are encoded using the DINOv3 model, and the agent's observation sequence is encoded using the DreamerV3 encoder.
9. The goal-driven reward generation method according to claim 1, characterized in that, In step S4, the comprehensive reward signal is obtained using the following calculation formula: in, This represents the comprehensive reward signal; This represents the first adjustable weighting coefficient; This represents the video-level reward signal; This represents the second adjustable weighting coefficient; This refers to the frame-level reward signal; This indicates the original task reward signal.
10. A goal-driven reward generation system for reinforcement learning, characterized in that, The goal-driven reward generation method described in any one of claims 1-9 includes: A target video generation module is used to generate target conditional videos based on user-provided task descriptions using a pre-trained video generation model. A video-level reward module, connected to the target video generation module, is used to calculate the correlation between the agent's observation sequence in the latent space and the target conditional video to obtain a video-level reward signal; A frame-level reward module, connected to the video-level reward module, is used to select key frames from the target condition video and calculate the frame-level reward signal related to the key frames in the target condition video based on a forward-backward representation learning mechanism. A reward fusion module, connected to the frame-level reward module, is used to fuse the video-level reward signal, the frame-level reward signal, and the original task reward signal provided by the environment to generate a comprehensive reward signal for policy optimization of the reinforcement learning agent.