Robot operation reward generation system and method based on video generation

Abstract

The invention discloses a robot operation reward generation system and method based on video generation, and the method comprises the steps: generating an image coding network of hidden space coding images according to a visual observation image of a robot, and the hidden space coding images comprise an initial hidden space coding image and a current hidden space coding image; predicting a video generation diffusion model network of a robot moving trajectory video according to the generated initial hidden space coding image; according to the state of the robot on the current hidden space coding image and the state of the robot on the predicted robot moving track video, the award of each robot state on the current hidden space coding image is calculated, the intelligent agent learns a strategy of maximizing the expected accumulated award according to the calculated award, and the award of the current visual observation image is obtained; and the reward module guides task execution of the robot. The method is used for calculating rewards of robot visual observation images, does not depend on environment feedback of robot task execution, and is suitable for task execution of various robots.

Description

Technical Field

[0001] The present invention relates to the technical field of robot task execution, and in particular to a robot operation reward generation system and method based on video generation. Background Art

[0002] In recent years, the execution of robot tasks through reinforcement learning has been a hot topic in the field of robotics research. In the framework of reinforcement learning, the intelligent agent usually obtains the robot's state and interaction with the environment based on the robot's running trajectory, and adjusts the robot's state strategy based on the reward feedback obtained, thereby optimizing the execution of the robot's tasks.

[0003] In order to improve the performance of reinforcement learning, reward design is crucial. The design of rewards directly affects the learning efficiency and final performance of the agent. The existing reward design is based on the robot's running trajectory and depends on the robot's task scenario. The reward function needs to be manually designed, which is usually very complex and requires an in-depth understanding of the details of the robot's task to determine the reward in the state space. The process is cumbersome and error-prone. Especially in robot tasks, improperly designed rewards may cause the agent to learn bad strategies, thereby causing errors in the execution of robot tasks. Summary of the invention

[0004] In response to the problems existing in the prior art, the present invention provides a robot operation reward generation system and method based on video generation, which generates a video of predicted robot operation trajectory through a video generation diffusion model network, which is used to calculate the reward of the robot's visual observation image, provide rich learning signals for the reinforcement learning agent, effectively guide the reinforcement learning agent to learn the strategy of complex robot tasks, and better optimize the robot task execution process.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solutions: a robot operation reward generation system based on video generation, comprising: an image encoding network, a video generation diffusion model network and a reward module; The image coding network is used to generate a latent space coding image according to the visual observation image of the robot, including an initial latent space coding image generated by the initial visual observation image and a current latent space coding image generated by the current visual observation image; The video generation diffusion model network predicts the robot running trajectory video based on the generated initial latent space encoding image and the initial visual observation image; The reward module calculates the reward of each robot state on the current latent space encoded image based on the robot state on the current latent space encoded image and the robot state on the predicted robot trajectory video. The intelligent agent learns the strategy of maximizing the expected cumulative reward based on the calculated reward, obtains the reward of the current visual observation image, and guides the robot's task execution.

[0006] Furthermore, the image coding network consists of an encoder and a decoder, wherein the encoder decomposes the visual observation image into a number of discrete vector representations, and the decoder converts the vector representation back to a high-dimensional image space to obtain a latent space encoded image.

[0007] Furthermore, before the image coding network generates a latent space coding image according to the robot's visual observation image, the image coding network needs to be trained using a generative adversarial network. The specific process is as follows: i. Collect visual observation images of the robot; ii. Input the collected visual observation images of the robot into the encoder of the image coding network and decompose them into several discrete vector representations; iii. Input the discrete vector representation into the generator in the decoder, reconstruct the vector representation back to the high-dimensional image space through encoding, evaluate the image quality converted back to the high-dimensional image space through the discriminator in the decoder, and update the parameters of the image coding network; iv. Repeat steps ii-iii until the evaluation function of the image quality converted back to the high-dimensional image space converges, thus completing the training of the image coding network.

[0008] Furthermore, the image quality evaluation function converted back to the high-dimensional image space is Specifically: , in, represents the robot’s visual observation image, The robot's visual observation image is converted into a low-dimensional vector representation through the encoder. Indicates that The discrete vector representation obtained by mapping to a vector set, Indicates that Convert back to high-dimensional image space to get latent space encoded image, represents the two-norm.

[0009] Furthermore, the video generation diffusion model network adds noise to the initial latent space encoded image, uses the initial visual observation image and the task language description corresponding to the initial visual observation image as learning conditions of the video generation diffusion model network, learns to remove the noise on the initial latent space encoded image, and predicts the robot operation trajectory video.

[0010] Furthermore, before predicting the robot's running trajectory video, the video generation diffusion model network optimizes the parameters of the video generation diffusion model network by maximizing the log-likelihood estimate of the robot's running trajectory video.

[0011] Furthermore, the reward module includes: a distance reward module, a progress reward module, an exploration reward module and a reward synthesis module; The distance reward module calculates the distance reward for each robot state on the current latent space encoding image according to the robot state on the current latent space encoding image and the robot state on the robot running trajectory video; The progress reward module calculates the progress reward for each robot state on the current latent space encoding image according to the robot state on the current latent space encoding image and the robot state on the robot running trajectory video; The exploration reward module determines the exploration reward for each robot state on the current latent space encoding image according to the robot state on the current latent space encoding image; The reward synthesis module is used to add the distance reward, progress reward and exploration reward to obtain the reward of each robot state on the current latent space encoding image.

[0012] Furthermore, the current latent space encodes the reward of each robot state on the image The calculation process is: , in, Indicates the current latent space encoding image t The distance reward under the robot state, , represents the similarity function, represents the first t robot status, Indicates the predicted robot trajectory video robot status, , Represents the number of robot states on the predicted robot trajectory video, h express The index of Indicates the predicted robot trajectory video h robot status; Indicates the current latent space encoding image t Progress reward in robot state, ; Indicates the current latent space encoding image t Exploration reward under each robot state.

[0013] Furthermore, the calculation process of the expected cumulative reward is: , in, C represents the expected cumulative reward, Express expectations, Indicates the current latent space encoding image t The discount factor corresponding to each robot state.

[0014] Furthermore, the present invention also provides a robot operation reward generation method of the robot operation reward generation system based on video generation, comprising the following steps: Step S1: After the environment is initialized, the initial visual observation image of the robot in the environment is input into the image encoder network to generate an initial latent space encoding image; Step S2, inputting the initial latent space encoding image and the initial visual observation image into the video generation diffusion model network to predict the robot running trajectory video; Step S3: inputting the visual observation image of the robot in the current environment into the image encoder network to generate the current latent space encoded image; Step S4: Input the robot state on the current latent space encoding image and the robot state on the predicted robot trajectory video into the reward module, calculate the reward of each robot state on the current visual observation image, and the intelligent agent learns the strategy of maximizing the expected cumulative reward based on the calculated reward, obtains the reward of the current visual observation image, and guides the robot's task execution.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The robot operation reward generation system and method based on video generation of the present invention guides the video generation diffusion model network through the robot's initial visual observation image and the corresponding task language description to predict the robot operation trajectory video that meets the task language description, which is used to calculate the reward of the robot's visual observation image. The reward calculation process is only applied to the image space, which can adapt to the general robot architecture and is not affected by the different spatial structures of heterogeneous robots, thus realizing a more general application scenario. The reward module in the robot operation reward generation system and method based on video generation of the present invention adds the distance reward, progress reward and exploration reward, wherein the distance reward can guide the robot strategy to learn in the direction of a better match between the robot's observed state and the predicted robot operation trajectory video; the progress reward can evaluate the completion progress of the robot task and ensure the correct execution of the robot task; the exploration reward encourages the intelligent agent of reinforcement learning to quickly learn how to recover from suboptimal or poor states, thereby improving learning efficiency; the reward for each robot state on the current latent space encoding image is obtained by adding the three rewards, and the intelligent agent learns the strategy of maximizing the expected cumulative reward based on the calculated reward, thereby obtaining the reward of the robot's visual observation image and guiding the robot's task execution; The robot operation reward generation system and method based on video generation of the present invention designs a universal reward in the reward module, provides a dense and stable reward signal for the intelligent agent, reduces the number of samples required for the intelligent agent in the exploration process, and accelerates the learning speed; at the same time, it does not need to rely on the environmental feedback of the robot task, can be applied to various robot tasks, can provide consistent learning signals in different environments, and improves the robustness and adaptability of the intelligent agent when facing different tasks and environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of a robot operation reward generation system based on video generation according to the present invention; Figure 2 Schematic diagram of the reward module in the present invention; Figure 3 The present invention is a flow chart of a method for generating robot operation rewards based on video generation. DETAILED DESCRIPTION

[0017] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0018] like Figure 1 A schematic diagram of a robot operation reward generation system based on video generation of the present invention, wherein the robot operation reward generation system comprises: an image encoding network, a video generation diffusion model network and a reward module; The image coding network is used to generate a latent space coding image according to the visual observation image of the robot, including an initial latent space coding image generated by the initial visual observation image and a current latent space coding image generated by the current visual observation image; The video generation diffusion model network predicts the robot's running trajectory video based on the generated initial latent space encoding image and the initial visual observation image; The reward module calculates the reward for each robot state on the current latent space encoded image based on the robot state on the current latent space encoded image and the robot state on the predicted robot trajectory video. Based on the calculated reward, the intelligent agent learns the strategy of maximizing the expected cumulative reward, obtains the reward of the current visual observation image, and guides the robot's task execution.

[0019] The reward calculation process in the present invention only involves the robot's visual observation images and the robot's running trajectory video predicted by the visual observation images, which can avoid dependence on environmental rewards and solve the difficulty of manual design of environmental rewards. At the same time, since the reward calculation process is only applied to the image space, it can adapt to the general robot architecture and is not affected by the different spatial structures of heterogeneous robots, realizing a more general application scenario.

[0020] It should be noted that the robot tasks applicable to the present invention can be performed by multiple robots in free combination, in a permuted order or individually, and do not need to rely on or depend on a fixed execution order.

[0021] The image coding network consists of an encoder and a decoder. The encoder decomposes the visual observation image into several discrete vector representations, and its purpose is to extract the feature information of the visual observation image layer by layer. In one technical solution of the present invention, the image encoder network may include convolutional layers and pooling layers to gradually extract the features of the visual observation image and convert it into a low-dimensional discrete vector representation through an embedding layer or a fully connected layer. The decoder converts the vector representation back to a high-dimensional image space, and its purpose is to restore the feature information of the image layer by layer to obtain a latent space encoded image with similar content and structure to the input visual observation image.

[0022] Before the image coding network generates latent space coding images based on the robot's visual observation images, the image coding network needs to be trained using a generative adversarial network, where the generator can use deconvolution or upsampling layers to gradually reconstruct the image, and the discriminator uses convolution layers to judge the authenticity of the reconstructed image, and trains the image coding network through a binary classification task. During the training process, the goal of the image encoder network is to minimize the image reconstruction error and the adversarial loss of the discriminator, while using vector quantization technology to ensure the discreteness and compactness of the coding. The image reconstruction error can use the mean square error to measure the difference between the original image and the reconstructed image, and the discriminator can use the cross entropy loss as the loss function. The optimizer usually uses the Adam optimizer, which iteratively updates the historical visual observation images in the static data set, and gradually optimizes the parameters of the image encoder network, thereby obtaining an efficient and high-performance image encoder network.

[0023] The specific process of image coding network training is: i. Collect the robot's historical visual observation images and the task language descriptions corresponding to the historical visual observation images to form a static data set, and uniformly sample the robot's visual observation images in the static data set to obtain an image sequence of fixed length; ii. Input the fixed-length image sequence into the encoder of the image coding network and decompose it into several discrete vector representations; iii. Input the discrete vector representation into the generator in the decoder, reconstruct the vector representation back to the high-dimensional image space through encoding, evaluate the image quality converted back to the high-dimensional image space through the discriminator in the decoder, and update the parameters of the image coding network; The evaluation function of the image quality converted back to the high-dimensional image space in the present invention Specifically: , in, represents the robot’s visual observation image, The robot's visual observation image is converted into a low-dimensional vector representation through the encoder. Indicates that The discrete vector representation obtained by mapping to a vector set, Indicates that Convert back to high-dimensional image space to get latent space encoded image, represents the two-norm; iv. Repeat steps ii-iii until the evaluation function of the image quality converted back to the high-dimensional image space converges, thus completing the training of the image coding network.

[0024] The video generation diffusion model network generates realistic video frame sequences by gradually adding and removing noise from the initial latent space encoded image. Specifically, the video generation diffusion model network gradually adds noise to the initial latent space encoded image through a multi-step forward diffusion process to generate a series of intermediate states; then through a backward diffusion process, the initial visual observation image and the task language description corresponding to the initial visual observation image are used as the learning conditions of the video generation diffusion model network to learn to remove the noise on the initial latent space encoded image and predict the robot's trajectory video. The video generation diffusion model network ensures the consistency of the predicted robot's trajectory video in space and time by capturing the temporal relationship between video frames, and can handle the multimodality and complexity of image sequences in video data, which is suitable for video generation tasks in various scenarios.

[0025] The video generation diffusion model network maximizes the log-likelihood estimate of the robot's trajectory video before predicting the robot's trajectory video. To optimize the parameters of the video generation diffusion model network and improve the generation ability and accuracy of the video generation diffusion model network.

[0026] In one technical solution of the present invention, the log-likelihood estimation of the robot running trajectory video can be maximized. The process is converted into a loss function that enables the video to generate a diffusion model network Convergent iterative solution process, loss function of the video generation diffusion model network It is expressed as: , in, represents the parameters of the video generation diffusion model network, represents the visual observation images in the static dataset, express The corresponding task language description, K represents the maximum diffusion step, represents Gaussian noise, Represents a static dataset, Indicates passing k The visual observation image after diffusion steps is represents the original noise added, Express expectations, Represents the noise term predicted by the video generative diffusion model network.

[0027] like Figure 2 ,The reward modules include: distance reward module, progress reward module, exploration reward module and reward synthesis module.

[0028] The distance reward module calculates the distance reward of each robot state on the current latent space encoding image based on the robot state on the current latent space encoding image and the robot state on the robot running trajectory video. The distance reward can guide the robot strategy to learn in the direction of a better match between the robot's observed state and the predicted robot running trajectory video. The calculation process of the distance reward is: ,in, Indicates the current latent space encoding image t The distance reward under the robot state, represents the similarity function, represents the first t robot status, Indicates the predicted robot trajectory video robot status, , Represents the number of robot states on the predicted robot trajectory video, h express The index of Indicates the predicted robot trajectory video h The robot status.

[0029] The progress reward module calculates the progress reward of each robot state on the current latent space encoding image based on the robot state on the current latent space encoding image and the robot state on the robot running trajectory video, which can evaluate the completion progress of the robot task. The reward of reinforcement learning should guide the robot to execute each key step sequence to ensure the correct execution of the robot task. In some embodiments, in order to reduce the tendency of the robot to skip important intermediate links and jump directly to the end of the trajectory, a concept of "arrival state" is introduced. This mechanism uses a similarity threshold to evaluate progress. , in the sequence of interactions between the reinforcement learning agent and the environment in order to learn to predict trajectories, where Mark robot status , Represents the number of robot states on the predicted robot trajectory video, h express The index of Indicates the predicted robot trajectory video h The robot state is the state that is farthest along the image of the predicted trajectory in the time trajectory. In some embodiments, the progress reward is defined as ,in, Indicates the current latent space encoding image t Progress reward in robot state.

[0030] The exploration reward module determines the exploration reward for each robot state on the current latent space encoding image based on the robot state on the current latent space encoding image. , which encourages the reinforcement learning agent to quickly learn how to recover from suboptimal or undesirable states, thereby improving learning sample efficiency. In some embodiments, the exploration reward Using Random Network Distillation (RND), RND calculates the exploration reward through a fixed randomly initialized target network and a trainable prediction network. Specifically, when the agent visits a new, unseen robot state, the prediction error is larger, thus giving a higher exploration reward; while for familiar states, the prediction error is smaller, and the exploration reward is reduced accordingly.

[0031] The reward synthesis module is used to add the distance reward, progress reward and exploration reward to obtain the reward for each robot state on the current latent space encoding image. Based on the calculated rewards, the agent can reduce the number of samples required in the exploration process, speed up the learning process, and learn the strategy of maximizing the expected cumulative reward, thereby obtaining rewards for the robot's visual observation images and guiding the robot's task execution.

[0032] In some embodiments, a Q-Learning algorithm is used to train a strategy based on a reward function. Specifically, in a certain robot state, the reward of the robot state is calculated, and the agent learns a strategy to maximize the expected cumulative reward based on the calculated reward, wherein the calculation process of the expected cumulative reward is: , in, C represents the expected cumulative reward, Express expectations, Indicates the current latent space encoding image t The discount factor corresponding to each robot state.

[0033] According to an embodiment of the present application, at least one of the distance reward module, the progress reward module, the exploration reward module and the reward synthesis module can be at least partially implemented as a hardware circuit, such as an FPGA, a PLA, a system on a chip, a system on a substrate, a system on a package, an ASIC, or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware, or in an appropriate combination of any of them.

[0034] Alternatively, at least one of the distance reward module, the progress reward module, the exploration reward module and the reward synthesis module may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0035] The robot operation reward generation system based on video generation of the present invention can be a hardware device independently set in the terminal, or a software program running in the terminal. For example, when the terminal is a desktop computer, the robot operation reward generation system can be embodied as an application such as software in the desktop computer.

[0036] like Figure 3 The present invention also provides a method for generating robot operation rewards based on video generation, comprising the following steps: Step S1: After the environment is initialized, the initial visual observation image of the robot in the environment is input into the image encoder network to generate an initial latent space encoding image; Step S2, inputting the initial latent space encoding image and the initial visual observation image into the video generation diffusion model network to predict the robot running trajectory video; Step S3: inputting the visual observation image of the robot in the current environment into the image encoder network to generate the current latent space encoded image; Step S4: Input the robot state on the current latent space encoding image and the robot state on the predicted robot trajectory video into the reward module, calculate the reward of each robot state on the current visual observation image, and the intelligent agent learns the strategy of maximizing the expected cumulative reward based on the calculated reward, obtains the reward of the current visual observation image, and guides the robot's task execution.

[0037] The robot operation reward generation method based on video generation of the present invention generates a predicted robot operation trajectory video through a video generation diffusion model network, which is used to calculate the reward of the robot's visual observation image, provide rich learning signals for the reinforcement learning agent, effectively guide the reinforcement learning agent to learn the strategy of the robot's complex tasks, and better optimize the robot's task execution process.

[0038] In a technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program, wherein the computer program enables a computer to execute the robot operation reward generation method based on video generation.

[0039] In a technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating robot operation rewards based on video generation is implemented.

[0040] In a technical solution of the present invention, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method for generating robot operation rewards based on video generation is implemented.

[0041] In the embodiments disclosed in the present application, the computer storage medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of computer storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0042] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0043] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A robot operation reward generation system based on video generation, characterized in that: include: Image encoding network, video generation diffusion model network and reward module; The image coding network is used to generate a latent space coding image according to the visual observation image of the robot, including an initial latent space coding image generated by the initial visual observation image and a current latent space coding image generated by the current visual observation image; The video generation diffusion model network predicts the robot running trajectory video based on the generated initial latent space encoding image and the initial visual observation image; The reward module calculates the reward of each robot state on the current latent space encoded image based on the robot state on the current latent space encoded image and the robot state on the predicted robot trajectory video. The intelligent agent learns the strategy of maximizing the expected cumulative reward based on the calculated reward, obtains the reward of the current visual observation image, and guides the robot's task execution.

2. A robot operation reward generation system based on video generation according to claim 1, characterized in that: The image coding network consists of an encoder and a decoder. The encoder decomposes the visual observation image into a number of discrete vector representations, and the decoder converts the vector representation back to a high-dimensional image space to obtain a latent space encoded image.

3. A robot operation reward generation system based on video generation according to claim 2, characterized in that: Before the image coding network generates a latent space coding image according to the robot's visual observation image, the image coding network needs to be trained using a generative adversarial network. The specific process is as follows: i. Collect visual observation images of the robot; ii. Input the collected visual observation images of the robot into the encoder of the image coding network and decompose them into several discrete vector representations; iii. Input the discrete vector representation into the generator in the decoder, reconstruct the vector representation back to the high-dimensional image space through encoding, evaluate the image quality converted back to the high-dimensional image space through the discriminator in the decoder, and update the parameters of the image coding network; iv. Repeat steps ii-iii until the evaluation function of the image quality converted back to the high-dimensional image space converges, thus completing the training of the image coding network.

4. A robot operation reward generation system based on video generation according to claim 3, characterized in that: The image quality evaluation function converted back to the high-dimensional image space Specifically: ， in, represents the robot’s visual observation image, The robot's visual observation image is converted into a low-dimensional vector representation through the encoder. Indicates that The discrete vector representation obtained by mapping to a vector set, Indicates that Convert back to high-dimensional image space to get latent space encoded image, represents the two-norm.

5. The robot operation reward generation system based on video generation according to claim 1, characterized in that: The video generation diffusion model network adds noise to the initial latent space coding image, uses the initial visual observation image and the task language description corresponding to the initial visual observation image as the learning conditions of the video generation diffusion model network, learns to remove the noise on the initial latent space coding image, and predicts the robot operation trajectory video.

6. A robot operation reward generation system based on video generation according to claim 5, characterized in that: Before predicting the robot's running track video, the video generation diffusion model network optimizes the parameters of the video generation diffusion model network by maximizing the log likelihood estimation of the robot's running track video.

7. A robot operation reward generation system based on video generation according to claim 1, characterized in that: The reward module includes: a distance reward module, a progress reward module, an exploration reward module and a reward synthesis module; The distance reward module calculates the distance reward for each robot state on the current latent space encoding image according to the robot state on the current latent space encoding image and the robot state on the robot running trajectory video; The progress reward module calculates the progress reward for each robot state on the current latent space encoding image according to the robot state on the current latent space encoding image and the robot state on the robot running trajectory video; The exploration reward module determines the exploration reward for each robot state on the current latent space encoding image according to the robot state on the current latent space encoding image; The reward synthesis module is used to add the distance reward, progress reward and exploration reward to obtain the reward of each robot state on the current latent space encoding image.

8. A robot operation reward generation system based on video generation according to claim 7, characterized in that: The current latent space encodes the reward for each robot state on the image The calculation process is: ， in, Indicates the current latent space encoding image t The distance reward under the robot state, , represents the similarity function, Represents the first t robot status, Indicates the predicted robot trajectory video robot status, , Represents the number of robot states on the predicted robot trajectory video, h express The index of Indicates the predicted robot trajectory video h robot status; Indicates the current latent space encoding image t Progress reward in robot state, ; Indicates the current latent space encoding image t Exploration reward under each robot state.

9. A robot operation reward generation system based on video generation according to claim 8, characterized in that: The calculation process of the expected cumulative reward is: ， in, C represents the expected cumulative reward, Express expectations, Indicates the current latent space encoding image t The discount factor corresponding to each robot state.

10. A robot operation reward generation method of a robot operation reward generation system based on video generation according to any one of claims 1 to 9, characterized in that: The steps include: Step S1: After the environment is initialized, the initial visual observation image of the robot in the environment is input into the image encoder network to generate an initial latent space encoding image; Step S2, inputting the initial latent space encoding image and the initial visual observation image into the video generation diffusion model network to predict the robot running trajectory video; Step S3: inputting the visual observation image of the robot in the current environment into the image encoder network to generate the current latent space encoded image; Step S4: Input the robot state on the current latent space encoding image and the robot state on the predicted robot trajectory video into the reward module, calculate the reward for each robot state on the current visual observation image, and the intelligent agent learns the strategy of maximizing the expected cumulative reward based on the calculated reward, obtains the reward of the current visual observation image, and guides the robot's task execution.

Images