Multi-view video reward mechanism learning framework and construction method thereof

Abstract

The invention provides a multi-view video reward mechanism learning framework and method, and the framework comprises a multi-view video evaluation subsystem which employs a multi-view video learning framework MVR to evaluate robot behaviors according to multi-view video learning; the visual feedback reward feedback strategy subsystem is used for generating visual feedback according to task text description through a visual language large model; accurate reward feedback is obtained according to the latest state correlation evaluation, so that the strategy is adjusted more effectively; the visual feedback task reward balancing subsystem analyzes the importance degree of task reward feedback according to the degree that robot behaviors approach to an expected target through a task reward model, dynamically adjusts the relative size between a task reward and a visual language model reward according to state correlation, and carries out visual feedback and task reward balancing; and the multi-view video reward combination subsystem is used for providing more accurate visual feedback and learning effect in a complex robot motion task in combination with multi-view videos and task rewards.

Description

Technical Field

[0001] The present invention relates to the field of humanoid robot data analysis intelligent large model information processing technology, and more specifically, the present invention relates to a multi-view video reward mechanism learning framework and a construction method thereof. Background Art

[0002] At present, reinforcement learning (RL) has shown great potential in solving complex tasks, and reward design is a core link. In the field of robotic tasks, the traditional way of setting rewards is mainly to encode quantitative goals such as movement and distance. Taking the robot's running task as an example, the task reward will prompt the robot to track the set forward speed. In order to further optimize the robot's movement performance, researchers often draw on the principles of animal kinematics and combine task rewards with energy consumption penalties; this method can cultivate the robot's agile movement ability to a certain extent. For example, in some studies, by reasonably adjusting the weights of rewards and penalties, the robot can use energy more efficiently during movement and achieve more flexible movements.

[0003] The process of human skill learning is significantly different from the traditional robot task reward design. When guiding learners, human coaches often first evaluate the learner's movement pattern through vision and give corresponding feedback, and then introduce quantitative goals to further improve the learner's skill level. This method of relying on visual guidance and then quantitative training has inspired researchers to explore incorporating visual feedback into robot task rewards to enhance the robot's skill learning ability. With the continuous development of visual language models (VLMs), their powerful image and text processing capabilities have provided new opportunities for this research direction.

[0004] VLM-based reward methods have gradually become a research hotspot. Some studies use VLM's ability to calculate image-text similarity to guide the robot towards a state that matches the task description. For example, by building a reward mechanism based on image-text similarity, the robot can understand the visual requirements of the task to a certain extent and make corresponding actions. However, these methods have obvious defects. On the one hand, they tend to select the state with the highest image-text similarity score. This preference may lead to a failure to fully consider various related states in actual tasks. In the robot running task, different image-text similarity scores will be obtained when observing the robot's running posture from different angles. Due to the method's own bias towards high-scoring states, the robot may excessively pursue a certain set posture (such as the set leg lifting posture) and ignore other equally important postures in actual running, which is not conducive to the robot learning a comprehensive and reasonable running action.

[0005] On the other hand, most existing methods rely on images captured from a single perspective. However, in actual scenarios, a single perspective cannot cover all key aspects of robot motion. For example, in complex robot manipulation tasks, all the movement details of the robot arm and the interaction between the robot and the surrounding environment may not be observed from a fixed perspective, which limits the comprehensive understanding and accurate evaluation of the robot's motion, and thus affects the accuracy and effectiveness of the reward, making it difficult for the robot to obtain comprehensive and accurate feedback during the learning process and unable to achieve the optimal learning effect; In the field of reinforcement learning (RL), existing reward techniques based on visual language models (VLMs) have many flaws that severely restrict robots' learning and decision-making capabilities in complex tasks.

[0006] Bias towards set states: Existing technologies rely on image-text similarity to generate reward signals, which makes them generally biased towards set states. In practical applications, tasks often involve multiple states that are visually different but all relevant to the task. Taking the robot performing complex dance movement tasks as an example, different dance postures and movement sequences are important for completing the task. However, the reward mechanism based on image-text similarity will tend to favor those states that have the highest image similarity with the given text description. This means that if a set dance posture has the highest match with the description in the image presentation, even if other postures are equally critical for coherent dance movements, the robot will over-pursue the set posture, resulting in a single movement and lack of flexibility, and unable to learn a complete and smooth dance movement sequence. This bias not only affects the robot's comprehensive understanding and processing of complex tasks, but may also cause the robot to fall into a local optimal solution and fail to find the true optimal strategy.

[0007] Limitations of single-view image information: Most current methods rely only on images captured from a single viewpoint, which results in a serious lack of information. In real-world task scenarios, the robot's motion is multi-dimensional and omnidirectional, and a single viewpoint cannot capture all important aspects of the robot's motion. For example, in the case of a robot's navigation task in an indoor environment, a single viewpoint may only be able to see part of the scene in front of the robot, and cannot obtain information on its sides and rear, such as whether there are obstacles approaching, the overall layout of the surrounding environment, etc. This makes it difficult for the robot to make accurate and comprehensive judgments without sufficient basis when making decisions. In addition, different viewpoints may present completely different visual information, and relying solely on single-view images may miss key clues, resulting in inaccurate assessments of the robot's motion. For example, in a robot manipulation task, the subtle contact between the robot arm and the object may not be observed from a certain angle, making it impossible to accurately judge whether the operation is successful, which in turn affects the accurate giving of rewards, making it impossible for the robot to obtain effective feedback during the learning process, hindering its effective learning and execution of tasks.

[0008] Unable to adapt to dynamic environmental changes: Since existing technologies are mainly based on fixed image-text matching patterns, they are difficult to adapt to dynamically changing environments. In practical applications, the environment is often complex and changeable, and the position, shape, color and other attributes of objects may change at any time, and the requirements of the task may also change over time. The reward mechanism based on a fixed pattern cannot adjust the reward strategy for the robot in a timely manner. In the rescue missions in which robots participate, the environment at the disaster site may change constantly, new obstacles may appear, and the location of the rescue target may also change. At this time, the reward technology that relies on fixed image-text similarity cannot quickly adapt to these changes, resulting in the reward signal received by the robot being out of touch with the actual task requirements, and cannot effectively guide the robot to make correct decisions, reducing its ability to respond in a dynamic environment and the efficiency of task execution.

[0009] Lack of deep understanding of task semantics: Existing technologies have obvious deficiencies in understanding task semantics. Although image-text similarity can reflect the degree of match between visual information and text description to a certain extent, it cannot deeply understand the semantic logic and contextual information behind the task. This defect is particularly prominent in some tasks that require reasoning and planning. In the robot's home service task, the robot is required to find the red cup on the dining table and take it to the kitchen. The reward technology based on image-text similarity may only focus on the match between the appearance features of the cup and the red cup, but cannot understand the semantics and sequence requirements of actions such as finding and taking it to the kitchen, and it is difficult to consider contextual information such as the spatial relationship between the dining table and the kitchen. This may cause confusion when the robot performs the task, and it is unable to complete the task according to the correct logic, affecting the quality and efficiency of the task completion. These problems remain to be solved; therefore, it is necessary to propose a multi-view video reward mechanism learning framework and its construction method to at least partially solve the problems existing in the existing technology. Summary of the invention

[0010] A series of simplified concepts are introduced in the content of the invention, which will be further described in detail in the specific implementation method section; the content of the invention of the present invention does not mean to attempt to limit the key features and essential technical features of the technical solution claimed for protection, nor does it mean to attempt to determine the scope of protection of the technical solution claimed for protection.

[0011] To at least partially solve the above problems, the present invention provides a multi-view video reward mechanism learning framework, including: Multi-view video evaluation subsystem, using the multi-view video learning framework MVR, to evaluate robot behavior based on multi-view video learning; The visual feedback reward feedback strategy subsystem generates visual feedback based on the task text description through the visual language large model to improve the robot's motor skill learning; based on the latest state relevance evaluation, accurate reward feedback is obtained to adjust the strategy more effectively; The visual feedback task reward balance subsystem uses the task reward model to analyze the importance of task reward feedback based on the degree to which the robot's behavior approaches the expected goal. It dynamically adjusts the relative size between the task reward and the visual language model reward based on the state correlation to balance the visual feedback and task reward. The multi-view video reward combination subsystem combines multi-view videos and task rewards to provide more accurate visual feedback and learning effects in complex robot motion tasks.

[0012] Preferably, the multi-view video assessment subsystem comprises: Sequence acquisition dataset subsystem: When the intelligent robot performs tasks in the environment, it collects robot state sequences at a set frequency and converts the robot state sequences into multi-view videos. At the same time, it extracts the target tasks into concise text descriptions. It calculates the similarity between the video and the text through VLM and constructs a dataset D. The dataset D contains: robot state sequences, multi-view videos corresponding to the robot state sequences, and similarity scores between the videos and the text. The matching pairwise comparison subsystem, the multi-view video learning framework MVR uses the sports competition statistical model through matching pairwise comparison to keep the robot state sequence comparison results as consistent as possible with the video comparison results, and learn the state correlation more accurately; Regularization interference reduction subsystem,The multi-view video learning framework MVR performs regularized state representation to reduce the interference caused by perspective changes in multi-view videos.

[0013] Preferably, the visual feedback reward feedback strategy subsystem includes: Trajectory rendering behavior visual subsystem, multi-view video learning framework MVR During the entire online reinforcement learning process, the trajectory generated by the robot is rendered into a video according to the set rendering frequency TRender; the robot's behavior visual information is obtained in a timely manner; Similarity score video embedding subsystem, the rendered video will be input into VLM to obtain the video-text similarity score and video embedding; this new information will be used to update the dataset D to keep the dataset containing the latest robot behavior and task matching; Buffer state sequence selection subsystem,The multi-view video learning framework MVR maintains a reference buffer Dref, which specifically stores those robot state sequences with high video-text similarity scores; these robot state sequences are considered to be more relevant to the task objectives; The multi-view video learning framework MVR uses samples in the dataset to update the model used to measure state relevance at a set update frequency TUpdate, continuously optimizing the understanding and judgment of state relevance; it uses an off policy strategy algorithm, and each time the model is updated, the rewards in the robot's playback buffer need to be recalculated to maintain the consistency of the rewards, so that the robot can obtain accurate reward feedback based on the latest state relevance evaluation, and thus adjust the strategy more effectively.

[0014] Preferably, the visual feedback task reward balance subsystem includes: The reward model designed by the multi-view video learning framework MVR realizes the flexible balance between the robot task reward rtask and the visual guidance reward rVLM. The visual guidance reward adjustment subsystem first uses the visual guidance reward rVLM to let the robot obtain the correct action mode, and then relies on the quantitative target robot task reward rtask for fine adjustment; The reward model designed by the multi-view video learning framework MVR achieves a flexible balance between the robot task reward rtask and the vision-guided reward rVLM, including: the vision-guided reward rVLM encourages the robot to explore task-related states; but when the robot gradually approaches these states, the influence of rVLM should gradually decrease, and the robot task reward rtask should play a dominant role.

[0015] Preferably, the multi-view video reward combined subsystem includes: Policy relevance matching measurement subsystem, the multi-view video learning framework MVR introduces policy relevance, comprehensively considers the relevance of states and the frequency of policy visits to these states; more comprehensively measures the degree of match between policy and task; Reward-guided exploration and learning subsystem: The multi-view video learning framework MVR uses approximate computing to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal; Policy relevance includes: when learning a strategy, the multi-view video learning framework MVR strives to maximize a set objective function; this objective function contains two key parts, one is the value function of the strategy, which reflects the expected return that can be obtained by performing actions according to the strategy; the other part is related to policy relevance, and its influence is adjusted by a special function; there is a hyperparameter w, which can be adjusted according to the specific characteristics and requirements of the task to balance the effects of these two parts; as the robot's strategy gets closer and closer to the strategy that best matches the task description, the influence of this part related to state relevance will gradually weaken, realizing a dynamic transformation of the reward function from rVLM-based to rtask-based.

[0016] The present invention provides a method for constructing a multi-view video reward mechanism learning framework, comprising: S1, using the multi-view video learning framework MVR, evaluates robot behavior based on multi-view video learning; S2, through the visual language large model, generates visual feedback based on the task text description to improve the robot's motor skill learning; based on the latest state relevance evaluation, accurate reward feedback is obtained to adjust the strategy more effectively; S3, through the task reward model, when the robot behavior is close to the expected goal, according to the degree of closeness of the robot behavior to the expected goal, analyze the importance of the task reward feedback, dynamically adjust the relative size between the task reward and the visual language model reward according to the state correlation, and balance the visual feedback and task reward; S4, combines multi-view videos and task rewards to provide more accurate visual feedback and learning effects in complex robot motion tasks.

[0017] Preferably, S1 comprises: S11, when the intelligent robot performs tasks in the environment, it collects robot state sequences at a set frequency and converts the robot state sequences into multi-view videos; at the same time, it extracts the target tasks into concise text descriptions; calculates the similarity between the video and the text through VLM; and constructs a data set D; the data set D includes: robot state sequences, multi-view videos corresponding to the robot state sequences, and similarity scores between the videos and the texts; S12, the multi-view video learning framework MVR uses a sports game statistical model to keep the robot state sequence comparison results as consistent as possible with the video comparison results through matching pairwise comparisons, and learn state correlations more accurately; S13, regularized interference reduction subsystem, the multi-view video learning framework MVR performs regularized state representation to reduce the interference caused by perspective changes in multi-view videos.

[0018] Preferably, S2 comprises: S21, the multi-view video learning framework MVR, renders the trajectory generated by the robot into a video according to the set rendering frequency TRender during the entire online reinforcement learning process; and obtains the robot's behavioral visual information in a timely manner; S22, the rendered video will be input into VLM to obtain the video-text similarity score and video embedding; this new information will be used to update the dataset D to keep the dataset containing the latest robot behavior and task matching; S23, the multi-view video learning framework MVR maintains a reference buffer Dref, which specifically stores robot state sequences with higher video-text similarity scores; these robot state sequences are considered to be more relevant to the task goal; The multi-view video learning framework MVR uses samples in the dataset to update the model used to measure state relevance at a set update frequency TUpdate, continuously optimizing the understanding and judgment of state relevance; it uses an off policy strategy algorithm, and each time the model is updated, the rewards in the robot's playback buffer need to be recalculated to maintain the consistency of the rewards, so that the robot can obtain accurate reward feedback based on the latest state relevance evaluation, and thus adjust the strategy more effectively.

[0019] Preferably, S3 includes: S31, the reward model designed in the multi-view video learning framework MVR achieves a flexible balance between the robot task reward rtask and the visual guidance reward rVLM; S32, first use the visual guidance reward rVLM to let the robot acquire the correct action mode, and then rely on the quantitative target robot task reward rtask for fine adjustment; The reward model designed by the multi-view video learning framework MVR achieves a flexible balance between the robot task reward rtask and the vision-guided reward rVLM, including: the vision-guided reward rVLM encourages the robot to explore task-related states; but when the robot gradually approaches these states, the influence of rVLM should gradually decrease, and the robot task reward rtask should play a dominant role.

[0020] Preferably, S4 includes: S41, the multi-view video learning framework MVR introduces policy relevance, which comprehensively considers the relevance of states and the frequency of policy visits to these states; it more comprehensively measures the degree of match between policy and task; S42, the multi-view video learning framework MVR uses approximate computing to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal; Policy relevance includes: when learning a strategy, the multi-view video learning framework MVR strives to maximize a set objective function; this objective function contains two key parts, one is the value function of the strategy, which reflects the expected return that can be obtained by performing actions according to the strategy; the other part is related to policy relevance, and its influence is adjusted by a special function; there is a hyperparameter w, which can be adjusted according to the specific characteristics and requirements of the task to balance the effects of these two parts; as the robot's strategy gets closer and closer to the strategy that best matches the task description, the influence of this part related to state relevance will gradually weaken, realizing a dynamic transformation of the reward function from rVLM-based to rtask-based.

[0021] Compared with the prior art, the present invention has at least the following beneficial effects: The present invention provides a learning framework for a multi-view video reward mechanism and a construction method thereof. Through a multi-view video evaluation subsystem, a multi-view video learning framework MVR is used to evaluate robot behavior based on multi-view video learning; a visual feedback reward feedback strategy subsystem generates visual feedback based on a task text description through a large visual language model to improve the robot's motion skill learning; accurate reward feedback is obtained based on the latest state relevance evaluation, thereby adjusting the strategy more effectively; a visual feedback task reward balance subsystem uses a task reward model to analyze the importance of task reward feedback based on the degree to which the robot behavior approaches the expected goal when the robot behavior approaches the expected goal, dynamically adjusts the relative size between the task reward and the visual language model reward based on the state relevance, and balances the visual feedback and task reward; a multi-view video reward combination subsystem combines multi-view videos and task rewards to provide more accurate visual feedback and learning effects in complex robot motion tasks; the multi-view video learning framework MVR can be used to evaluate the robot's behavior using multi-view videos, and generate a task reward based on the task text description. Visual feedback is used to improve the learning effect of robot motion skills; through the reward function, when the robot behavior is close to the expected goal, the importance of task reward feedback can be analyzed according to the degree to which the robot behavior is close to the expected goal, and the balance problem between visual feedback and task reward can be solved; by combining multi-view video and task rewards, the multi-view video learning framework MVR can provide more accurate visual feedback in complex robot complex motion tasks and improve learning effects; based on the reinforcement learning framework of multi-view video and visual language model (VLM), the robot can master complex motion skills on the humanoid robot simulation platform; the reward function dynamically adjusts the relative size between task reward and visual language model reward according to state correlation to achieve a balance between the two; improves the learning effect of the robot in complex tasks, and enables the robot to learn the required motion skills faster and more accurately on the humanoid robot simulation platform; effectively solves the problems of biased setting of states and reliance on single-view images in the reward method based on the visual language model, comprehensively understands the robot motion, and improves the accuracy and effectiveness of rewards; it has great significance and significant effects.

[0022] The multi-view video reward mechanism learning framework and construction method thereof described in the present invention, and other advantages, objectives and features of the present invention will be partially reflected through the following description, and will also be partially understood by technicians in this field through research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1This is a diagram of an embodiment of the multi-view video reward mechanism learning framework described in the present invention.

[0024] Figure 2 This is another embodiment diagram of the multi-view video reward mechanism learning framework described in the present invention.

[0025] Figure 3 This is a diagram of an embodiment of a method for constructing a multi-view video reward mechanism learning framework according to the present invention. DETAILED DESCRIPTION

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments so that those skilled in the art can implement the invention according to the instructions. Figure 1-Figure 3 As shown, the present invention provides a multi-view video reward mechanism learning framework, including: Multi-view video evaluation subsystem, using the multi-view video learning framework MVR, to evaluate robot behavior based on multi-view video learning; The visual feedback reward feedback strategy subsystem generates visual feedback based on the task text description through the visual language large model to improve the robot's motor skill learning; based on the latest state relevance evaluation, accurate reward feedback is obtained to adjust the strategy more effectively; The visual feedback task reward balance subsystem uses the task reward model to analyze the importance of task reward feedback based on the degree to which the robot's behavior approaches the expected goal. It dynamically adjusts the relative size between the task reward and the visual language model reward based on the state correlation to balance the visual feedback and task reward. The multi-view video reward combination subsystem combines multi-view videos and task rewards to provide more accurate visual feedback and learning effects in complex robot motion tasks.

[0027] The principle and effect of the above technical solution are as follows: the present invention provides a learning framework for a multi-view video reward mechanism, including: a multi-view video evaluation subsystem, which uses a multi-view video learning framework MVR to evaluate robot behavior based on multi-view video learning; a visual feedback reward feedback strategy subsystem, which generates visual feedback based on the task text description through a large visual language model to enhance the robot's motion skill learning; obtains accurate reward feedback based on the latest state relevance evaluation, and then adjusts the strategy more effectively; a visual feedback task reward balance subsystem, which uses a task reward model to analyze the importance of task reward feedback based on the degree to which the robot's behavior approaches the expected goal, dynamically adjusts the relative size between the task reward and the visual language model reward based on the state relevance, and balances visual feedback and task rewards; a multi-view video reward combination subsystem, which combines multi-view videos and task rewards to perform complex robot motion tasks. The multi-view video learning framework MVR can be used to evaluate the robot's behavior using multi-view videos, and visual feedback can be generated based on the task text description to improve the learning effect of the robot's motion skills. The reward function can analyze the importance of task reward feedback based on the degree to which the robot's behavior approaches the expected goal, and solve the problem of balancing visual feedback and task rewards. By combining multi-view videos and task rewards, the multi-view video learning framework MVR can provide more accurate visual feedback in complex robot complex motion tasks and improve learning effects. The reinforcement learning framework based on multi-view videos and visual language models (VLM) enables robots to master complex motion skills on a humanoid robot simulation platform. The reward function dynamically adjusts the relative size between task rewards and visual language model rewards based on state correlation to achieve a balance between the two. Among them, Top-k Sampling is performed from the top k basic unit tokens, including random sampling only from the k words with the highest probability. The randomness allows other basic unit tokens with higher scores or probabilities to have a chance to be selected. It improves the learning effect of robots in complex tasks and enables robots to learn the required sports skills faster and more accurately on the humanoid robot simulation platform. It effectively solves the problems of biased setting states and reliance on single-view images in reward methods based on visual language models, fully understands robot movements, and improves the accuracy and effectiveness of rewards.

[0028] In one embodiment, a multi-view video assessment subsystem includes: Sequence acquisition dataset subsystem: When the intelligent robot performs tasks in the environment, it collects robot state sequences at a set frequency and converts the robot state sequences into multi-view videos. At the same time, it extracts the target tasks into concise text descriptions. It calculates the similarity between the video and the text through VLM and constructs a dataset D. The dataset D contains: robot state sequences, multi-view videos corresponding to the robot state sequences, and similarity scores between the videos and the text. The matching pairwise comparison subsystem, the multi-view video learning framework MVR uses the sports competition statistical model to keep the robot state sequence comparison results as consistent as possible with the video comparison results, and learn the state correlation more accurately; Regularization interference reduction subsystem,The multi-view video learning framework MVR performs regularized state representation to reduce the interference caused by perspective changes in multi-view videos.

[0029] The principle and effect of the above technical solution are as follows: a multi-view video evaluation subsystem, including: a sequence acquisition data set subsystem, in which the intelligent robot collects the robot state sequence at a set frequency during the execution of tasks in the environment, and converts the robot state sequence into a multi-view video; at the same time, the target task is refined into a concise text description; the similarity between the video and the text is calculated by VLM; a data set D is constructed; the data set D includes: the robot state sequence, the multi-view video corresponding to the robot state sequence, and the similarity score between the video and the text; a matching pairwise comparison subsystem, the multi-view video learning framework MVR uses a matching pairwise comparison method and a sports game statistical model to keep the robot state sequence comparison result and the video comparison result as consistent as possible, and learn the state correlation more accurately; the sports game statistical model includes the Bradley-Terry model; a regularization interference reduction subsystem, the multi-view video learning framework MVR performs regularized state representation to reduce the interference caused by the change of perspective in the multi-view video; Keep the robot state sequence comparison results and video comparison results as consistent as possible, and learn the state correlation more accurately, including: in actual tasks, the degree of match between any two videos and the task description will form a probabilistic comparison relationship, and the multi-view video learning framework MVR can also reflect this relationship at the robot state sequence level; for example, if one video is more consistent with the task description than another video, then the corresponding robot state sequences should also show a similar relationship; through continuous adjustment, the similarity of the robot state sequence and the video sequence is aligned as much as possible, so as to avoid the bias towards the set state caused by the image-text similarity calculation, and more comprehensively consider the correlation between different states and tasks; The multi-view video learning framework MVR regularizes the state representation to reduce the interference caused by the change of viewpoints in multi-view videos, including: the video similarity scores under different viewpoints will fluctuate, which may affect the accurate judgment of state relevance; to reduce this interference, the multi-view video learning framework MVR keeps the robot state sequence and the video embedding similarity structure matching. Specifically, the state is processed through a set function, the state in the state space is mapped to a suitable vector space, and then some operations are performed to measure the relevance of the state to the task; during the calculation process, different viewpoints are randomly and uniformly sampled to calculate the video embedding, and the state representation is adjusted based on this reference, so that the state representation can reflect the common information under different viewpoints, reduce the interference of the view setting factors, and thus learn the state relevance more stably and accurately.

[0030] In one embodiment, the visual feedback reward feedback strategy subsystem includes: Trajectory rendering behavior visual subsystem, multi-view video learning framework MVR During the entire online reinforcement learning process, the trajectory generated by the robot is rendered into a video according to the set rendering frequency TRender; the robot's behavior visual information is obtained in a timely manner; Similarity score video embedding subsystem, the rendered video will be input into VLM to obtain the video-text similarity score and video embedding; this new information will be used to update the dataset D to keep the dataset containing the latest robot behavior and task matching; Buffer state sequence selection subsystem,The multi-view video learning framework MVR maintains a reference buffer Dref, which specifically stores those robot state sequences with high video-text similarity scores; these robot state sequences are considered to be more relevant to the task objectives; The multi-view video learning framework MVR uses samples in the dataset to update the model used to measure state relevance at a set update frequency TUpdate, continuously optimizing the understanding and judgment of state relevance; an offline strategy algorithm is used, and after each model update, the rewards in the robot's playback buffer need to be recalculated to maintain the consistency of the rewards, so that the robot can obtain accurate reward feedback based on the latest state relevance evaluation, and thus adjust the strategy more effectively.

[0031] The principle and effect of the above technical solution are as follows: visual feedback reward feedback strategy subsystem, including: trajectory rendering behavior visual subsystem, multi-view video learning framework MVR renders the trajectory generated by the robot into video according to the set rendering frequency TRender during the entire online reinforcement learning process; obtains the robot's behavioral visual information in a timely manner; similarity score video embedding subsystem, the rendered video will be input into VLM to obtain the video-text similarity score and video embedding; these new information will be used to update the data set D to keep the data set containing the latest robot behavior and task matching; buffer state sequence selection subsystem, multi-view video learning framework MVR maintains a reference buffer Dref, which is specifically used to store robot state sequences with high video-text similarity scores; these robot state sequences are considered to be more relevant to the task objectives; the multi-view video learning framework MVR uses the samples in the data set to update the model used to measure state relevance at a set update frequency TUpdate, and continuously optimizes the understanding and judgment of state relevance; an offline strategy algorithm is used, and after each model update, the reward in the robot playback buffer needs to be recalculated to maintain the consistency of the reward, so that the robot can obtain accurate reward feedback based on the latest state relevance evaluation, and then adjust the strategy more effectively; MVR design idea: On the one hand, by matching paired comparisons and using sports game statistical models, the comparison results of the state sequence are kept as consistent as possible with the comparison results of the video; in actual tasks, the degree of match between any two videos and the task description will form a probabilistic comparison relationship.

[0032] The probability that a video o matches the task description l better than another video o′ is given by: in, is the sigmoid function, Represents the cosine similarity of the VLM model.

[0033] MVR expects this relationship to be reflected at the state sequence level. For example, if one video is more consistent with the task description than another video, then the corresponding state sequences should also show a similar relationship. Similarly, the probability that a state sequence s matches l better than another state sequence s′ is: Where n(s) is the length of s, Represents the state correlation function; it should be noted that the state correlation is determined based on the video, thereby overcoming the bias towards the set state; given two samples in the dataset D and , minimize the following loss function: Here, we use the property 1 - σ(x) = σ(-x); by continuous adjustment, we can align the similarity between the state sequence and the video sequence as much as possible, thereby avoiding the bias towards the set state caused by the image-text similarity calculation and more comprehensively considering the relevance of different states to the task.

[0034] On the other hand, the perturbation of view setting is independent of the underlying states and the similarity between states, since a change in view does not affect the state itself. Therefore, this perturbation can be eliminated by keeping the similarity structure between state sequences matching that of the video embedding, where the viewpoints are uniformly randomly sampled. Using the state encoder and vector to parameterize the correlation function, where d is the dimension of the state embedding; and All are normalized; the correlation of state s is calculated as follows: Where 〈·〉 represents the dot product operation; for the state sequence s, the following formula is used as its representation: where o represents the video stream; since the viewpoints are uniformly randomly sampled, this regularization produces an average state representation that reduces the noise of the view setting. Finally, the overall objective function for learning state correlation combines the matching term and the regularization term: Lrel is the overall objective function of state correlation, Lmatching is the matching term, and Lreg is the regularization term.

[0035] In one embodiment, the visual feedback task reward balance subsystem includes: The reward model designed by the multi-view video learning framework MVR realizes the flexible balance between the robot task reward rtask and the visual guidance reward rVLM. The visual guidance reward adjustment subsystem first uses the visual guidance reward rVLM to let the robot obtain the correct action mode, and then relies on the quantitative target robot task reward rtask for fine adjustment; The reward model designed by the multi-view video learning framework MVR achieves a flexible balance between the robot task reward rtask and the vision-guided reward rVLM, including: the vision-guided reward rVLM encourages the robot to explore task-related states; but when the robot gradually approaches these states, the influence of rVLM should gradually decrease, and the robot task reward rtask should play a dominant role.

[0036] The principle and effect of the above technical solution are as follows: the visual feedback task reward balance subsystem includes: a flexible balance subsystem of multiple types of rewards, and the reward model designed by the multi-view video learning framework MVR realizes a flexible balance between the robot task reward rtask and the visual guidance reward rVLM; the visual guidance reward adjustment subsystem first allows the robot to obtain the correct action mode through the visual guidance reward rVLM, and then relies on the quantitative target robot task reward rtask for fine adjustment; the reward model designed by the multi-view video learning framework MVR realizes a flexible balance between the robot task reward rtask and the visual guidance reward rVLM, including: the visual guidance reward rVLM encourages the robot to explore the states related to the task; but when the robot gradually approaches these states, the influence of rVLM should gradually decrease, and at this time the robot task reward rtask should play a leading role; The reward function designed by MVR is committed to achieving a flexible balance between the task reward rtask and the VLM-based reward rVLM. Specifically, the role of rVLM is to encourage the agent to explore task-related states, but when the agent gradually approaches these states, the influence of rVLM should gradually decrease, and rtask should play a leading role. This design concept is similar to the process of human skill learning, first allowing the agent to acquire the correct action mode through rVLM, and then relying on rtask for fine-tuning.

[0037] To achieve this balance, MVR introduces the concept of policy relevance. Simply put, it is to comprehensively consider the relevance of states and the frequency of policy visits to these states, so as to more comprehensively measure the degree of match between the policy and the task. Based on policy relevance, MVR strives to maximize a set objective function when learning a policy. This objective function contains two key parts. One part is the value function of the policy, which reflects the expected return that can be obtained by performing actions according to the policy; the other part is related to policy relevance, and its influence is adjusted by a special function. The policy relevance function can be adjusted according to the specific characteristics and needs of the task to balance the effects of these two parts. As the agent's strategy gets closer and closer to the strategy that best matches the task description, the influence of this part related to state relevance will gradually weaken, realizing a dynamic transition of the reward function from rVLM-based to rtask-based: in, Strategy The value function of reinforcement learning is: Refers to and language The best matching strategy. In order to maximize the log-likelihood, Under the BT model Better match , introduced to adjust the scale of the two terms. The function log(σ(x)) is monotonically decreasing when x ≤ 0, and maintains Getting closer hour, The impact has weakened.

[0038] When calculating rVLM, MVR uses an approximate method. Since it is difficult to sample directly from the ideal strategy, MVR uses the reference set Dref mentioned earlier for approximate sampling. The state sequences in the reference set are selected based on the video-text similarity score and are considered to be more closely related to the target strategy. By sampling from the reference set, rVLM can be approximately calculated, providing the agent with a reward signal based on state relevance, guiding the agent to explore and learn towards a state that is more consistent with the task goal: Among them, E[] represents expectation, Representative from the strategy The state of sampling, represents the correlation function, σ is the sigmoid function.

[0039] In one embodiment, the multi-view video reward combined subsystem includes: Policy relevance matching measurement subsystem, the multi-view video learning framework MVR introduces policy relevance, comprehensively considers the relevance of states and the frequency of policy visits to these states; more comprehensively measures the degree of match between policy and task; Reward-guided exploration and learning subsystem: The multi-view video learning framework MVR uses approximate computing to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal; Strategy relevance includes: when learning strategies, the multi-view video learning framework MVR strives to maximize a set objective function; this objective function contains two key parts, one is the value function of the strategy, which reflects the expected return that can be obtained by performing actions according to the strategy; the other part is related to strategy relevance, and its influence is adjusted by a special function; the strategy relevance function can be adjusted according to the specific characteristics and needs of the task to balance the effects of these two parts; as the robot's strategy gets closer and closer to the strategy that best matches the task description, the influence of this part related to state relevance will gradually weaken, realizing the dynamic transformation of the reward function from rVLM-based to rtask-based.

[0040] The principle and effect of the above technical solution are: Multi-view video reward combined subsystem, including: Policy relevance matching measurement subsystem, the multi-view video learning framework MVR introduces policy relevance, comprehensively considers the relevance of states and the frequency of policy visits to these states; more comprehensively measures the degree of match between policy and task; Reward-guided exploration and learning subsystem: The multi-view video learning framework MVR uses approximate computing to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal; Policy relevance includes: when learning a policy, the multi-view video learning framework MVR will strive to maximize a set objective function; this objective function contains two key parts, one is the value function of the policy, which reflects the expected return that can be obtained by executing actions according to the policy; the other part is related to policy relevance, and its influence is adjusted by a special function; there is a hyperparameter w, which can be adjusted according to the specific characteristics and requirements of the task to balance the effects of these two parts; as the robot's policy gets closer and closer to the policy that best matches the task description, the influence of this part related to state relevance will gradually weaken, realizing a dynamic transformation of the reward function from rVLM-based to rtask-based; The multi-view video learning framework MVR uses approximate calculations to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal. The multi-view video learning framework MVR uses an approximate calculation method to calculate rVLM, providing the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal. The multi-view video learning framework MVR uses the previously mentioned reference set Dref for approximate sampling. The robot state sequence in the reference set is selected based on the video-text similarity score and is considered to be more closely related to the target strategy. By sampling from the reference set, rVLM can be approximately calculated to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal. Multiple complex motion tasks were selected from the simulated humanoid robot benchmark for full-body motion and manipulation, including dynamic motion tasks and static posture tasks, to test the complex skills of the Unitree H1 robot; dynamic motion tasks include running, walking, climbing stairs, and sliding dynamic tasks; dynamic motion tasks include standing, balancing, and sitting static tasks; the robot can only obtain proprioceptive data such as joint position and angular velocity during task execution, which is closer to real-life application scenarios; during the experiment, one out of every nine trajectories was selected for rendering as a video to reduce computational costs and maintain data diversity; for each video, a segment of length 64 was captured, and the video-text similarity score was calculated using the ViCLIP model. The reference model was updated every 100,000 environmental steps, and an early stopping strategy was used to prevent overfitting. The optimal value of the hyperparameter w is determined by grid search from {0.01, 0.1, 0.5}. Based on the TQC algorithm, experiments are carried out in the SBX framework using the JAX version of TQC, and its parameters are fine-tuned for complex humanoid robot control tasks. Each task is trained for 10 million steps. Samples are collected from 8 parallel environments during training, and the learning rate gradually decays from the initial 6e-4 to the final 5e-5 to maintain training stability. 256 samples are sampled from the playback buffer each time to update the strategy, and 16 gradient updates are performed every 16 steps; state-dependent exploration (SDE) is enabled to enhance the exploration ability of the algorithm; the neural networks are all three-layer neural networks, each layer is 256 wide, the quantile count is set to 50, and the top 5 quantiles of each network are discarded; a variety of cutting-edge methods are selected for comparison with the multi-view video learning framework MVR; TQC, as a classic algorithm, only relies on task rewards for strategy learning; VLM-RM combines task rewards and image-text similarity scores generated by the CLIP model; FuRL is based on image-text similarity and solves the reward sparsity problem through online fine-tuning; RoboCLIP uses video-text similarity to provide rewards for online RL robots, but only gives a single reward at the end of the trajectory; in addition, the experimental results of DreamerV3 are included for comparative analysis; the results show that the multi-view video learning framework MVR surpasses all baseline methods in terms of average performance on multiple tasks; The multi-view video learning framework MVR can also identify states with high rewards but suboptimal, including: As shown in Figure 2, the multi-view video learning framework MVR identifies states with high task rewards but suboptimal; visualize the states generated by the TQC agent for the sit_hard task; the left part illustrates the task reward. The right part shows the combination of the task reward and the visual guidance reward calculated by the trained multi-view video learning framework MVR agent; the multi-view video learning framework MVR identifies several parts of the state space with high task reward values ​​as suboptimal, mainly corresponding to unstable sitting postures; the multi-view video learning framework MVR can also identify states with high rewards but suboptimal.

[0041] The present invention provides a method for constructing a multi-view video reward mechanism learning framework, comprising: S1, using the multi-view video learning framework MVR, evaluates robot behavior based on multi-view video learning; S2, through the visual language large model, generates visual feedback based on the task text description to improve the robot's motor skill learning; based on the latest state relevance evaluation, accurate reward feedback is obtained to adjust the strategy more effectively; S3, through the task reward model, when the robot behavior is close to the expected goal, according to the degree of closeness of the robot behavior to the expected goal, analyze the importance of the task reward feedback, dynamically adjust the relative size between the task reward and the visual language model reward according to the state correlation, and balance the visual feedback and task reward; S4, combines multi-view videos and task rewards to provide more accurate visual feedback and learning effects in complex robot motion tasks.

[0042] The principle and effect of the above technical solution are as follows: the present invention provides a method for constructing a learning framework of a multi-view video reward mechanism, including: using a multi-view video learning framework MVR to evaluate robot behavior based on multi-view video learning; generating visual feedback based on task text descriptions through a large visual language model to enhance robot motion skill learning; obtaining accurate reward feedback based on the latest state relevance evaluation, thereby adjusting strategies more effectively; through a task reward model, when the robot behavior approaches the expected goal, analyzing the importance of task reward feedback based on the degree to which the robot behavior approaches the expected goal, dynamically adjusting the relative size between task rewards and visual language model rewards based on state relevance, and balancing visual feedback and task rewards; combining multi-view videos and task rewards to provide more accurate visual feedback and learning effects in complex robot motion tasks; being able to use multi-view The multi-view video learning framework MVR uses multi-view videos to evaluate the robot's behavior and generates visual feedback based on the task text description to improve the learning effect of the robot's motion skills. Through the reward function, when the robot's behavior is close to the expected goal, the importance of the task reward feedback can be analyzed according to the degree to which the robot's behavior is close to the expected goal, solving the problem of balancing visual feedback and task rewards. By combining multi-view videos and task rewards, the multi-view video learning framework MVR can provide more accurate visual feedback in complex robot complex motion tasks and improve learning effects. Based on the reinforcement learning framework of multi-view videos and visual language models (VLM), the robot can master complex motion skills on a humanoid robot simulation platform. The reward function dynamically adjusts the relative size between the task reward and the visual language model reward according to the state correlation to achieve a balance between the two. Among them, Top-k Sampling is performed from the top k basic unit tokens, including random sampling only from the k words with the highest probability. The randomness allows other basic unit tokens with higher scores or probabilities to have a chance to be selected. It improves the learning effect of robots in complex tasks and enables robots to learn the required sports skills faster and more accurately on the humanoid robot simulation platform. It effectively solves the problems of biased setting states and reliance on single-view images in reward methods based on visual language models, fully understands robot movements, and improves the accuracy and effectiveness of rewards.

[0043] In one embodiment, S1 includes: S11, when the intelligent robot performs tasks in the environment, it collects robot state sequences at a set frequency and converts the robot state sequences into multi-view videos; at the same time, it extracts the target tasks into concise text descriptions; calculates the similarity between the video and the text through VLM; and constructs a data set D; the data set D includes: robot state sequences, multi-view videos corresponding to the robot state sequences, and similarity scores between the videos and the texts; S12, the multi-view video learning framework MVR uses a sports game statistical model to keep the robot state sequence comparison results as consistent as possible with the video comparison results through matching pairwise comparisons, and learn state correlations more accurately; S13, regularized interference reduction subsystem, the multi-view video learning framework MVR performs regularized state representation to reduce the interference caused by perspective changes in multi-view videos.

[0044] The principle and effect of the above technical solution are as follows: when the intelligent robot performs tasks in the environment, it collects the robot state sequence at a set frequency and converts the robot state sequence into a multi-view video; at the same time, the target task is refined into a concise text description; the similarity between the video and the text is calculated by VLM; a data set D is constructed; the data set D includes: the robot state sequence, the multi-view video corresponding to the robot state sequence and the similarity score between the video and the text; the multi-view video learning framework MVR uses a matching pairwise comparison method and a sports game statistical model to keep the robot state sequence comparison result and the video comparison result as consistent as possible, so as to learn the state correlation more accurately; the sports game statistical model includes the Bradley-Terry model; the regularization interference reduction subsystem, the multi-view video learning framework MVR performs regularized state representation to reduce the interference caused by the change of viewpoint in the multi-view video; Keep the robot state sequence comparison results and video comparison results as consistent as possible, and learn the state correlation more accurately, including: in actual tasks, the degree of match between any two videos and the task description will form a probabilistic comparison relationship, and the multi-view video learning framework MVR can also reflect this relationship at the robot state sequence level; for example, if one video is more consistent with the task description than another video, then the corresponding robot state sequences should also show a similar relationship; through continuous adjustment, the similarity of the robot state sequence and the video sequence is aligned as much as possible, so as to avoid the bias towards the set state caused by the image-text similarity calculation, and more comprehensively consider the correlation between different states and tasks; The multi-view video learning framework MVR regularizes the state representation to reduce the interference caused by the change of viewpoints in multi-view videos, including: the video similarity scores under different viewpoints will fluctuate, which may affect the accurate judgment of state relevance; to reduce this interference, the multi-view video learning framework MVR keeps the robot state sequence and the video embedding similarity structure matching. Specifically, the state is processed through a set function, the state in the state space is mapped to a suitable vector space, and then some operations are performed to measure the relevance of the state to the task; during the calculation process, different viewpoints are randomly and uniformly sampled to calculate the video embedding, and the state representation is adjusted based on this reference, so that the state representation can reflect the common information under different viewpoints, reduce the interference of the view setting factors, and thus learn the state relevance more stably and accurately.

[0045] In one embodiment, S2 includes: S21, the multi-view video learning framework MVR, renders the trajectory generated by the robot into a video according to the set rendering frequency TRender during the entire online reinforcement learning process; and obtains the robot's behavioral visual information in a timely manner; S22, the rendered video will be input into VLM to obtain the video-text similarity score and video embedding; this new information will be used to update the dataset D to keep the dataset containing the latest robot behavior and task matching; S23, the multi-view video learning framework MVR maintains a reference buffer Dref, which specifically stores robot state sequences with higher video-text similarity scores; these robot state sequences are considered to be more relevant to the task goal; The multi-view video learning framework MVR uses samples in the dataset to update the model used to measure state relevance at a set update frequency TUpdate, continuously optimizing the understanding and judgment of state relevance; an offline strategy algorithm is used, and after each model update, the rewards in the robot's playback buffer need to be recalculated to maintain the consistency of the rewards, so that the robot can obtain accurate reward feedback based on the latest state relevance evaluation, and thus adjust the strategy more effectively.

[0046] The principle and effect of the above technical solution are as follows: the multi-view video learning framework MVR renders the trajectory generated by the robot into a video according to the set rendering frequency TRender during the entire online reinforcement learning process; the robot's behavioral visual information is obtained in a timely manner; the rendered video will be input into the VLM to obtain the video-text similarity score and video embedding; these new information will be used to update the data set D to keep the data set containing the latest robot behavior and task matching; the buffer state sequence selection subsystem, the multi-view video learning framework MVR maintains a reference buffer Dref, which is specifically used to store robot state sequences with high video-text similarity scores; these robot state sequences are considered to be more relevant to the task objectives; the multi-view video learning framework MVR uses the samples in the data set to update the model used to measure state relevance at the set update frequency TUpdate, and continuously optimizes the understanding and judgment of state relevance; the off policy strategy algorithm is used, and the rewards in the robot playback buffer need to be recalculated after each model update to maintain the consistency of the rewards, so that the robot can obtain accurate reward feedback based on the latest state relevance evaluation, and then adjust the strategy more effectively; MVR design idea: On the one hand, by matching paired comparisons and using sports game statistical models, the comparison results of the state sequence are kept as consistent as possible with the comparison results of the video; in actual tasks, the degree of match between any two videos and the task description will form a probabilistic comparison relationship.

[0047] The probability that a video o matches the task description l better than another video o′ is given by: in, is the sigmoid function, Represents the cosine similarity of the VLM model.

[0048] MVR expects this relationship to be reflected at the state sequence level. For example, if one video is more consistent with the task description than another video, then the corresponding state sequences should also show a similar relationship. Similarly, the probability that a state sequence s matches l better than another state sequence s′ is: Where n(s) is the length of s, Represents the state correlation function; it should be noted that the state correlation is determined based on the video, thereby overcoming the bias towards the set state; given two samples in the dataset D and , minimize the following loss function: Here, we use the property 1 - σ(x) = σ(-x); by continuous adjustment, we can align the similarity between the state sequence and the video sequence as much as possible, thereby avoiding the bias towards the set state caused by the image-text similarity calculation and more comprehensively considering the relevance of different states to the task.

[0049] On the other hand, the perturbation of view setting is independent of the underlying states and the similarity between states, since a change in view does not affect the state itself. Therefore, this perturbation can be eliminated by keeping the similarity structure between state sequences matching that of the video embedding, where the viewpoints are uniformly randomly sampled. Using the state encoder and vector to parameterize the correlation function, where d is the dimension of the state embedding; and All are normalized; the correlation of state s is calculated as follows: Where 〈·〉 represents the dot product operation; for the state sequence s, the following formula is used as its representation: where o represents the video stream; since the viewpoints are uniformly randomly sampled, this regularization produces an average state representation that reduces the noise of the view setting. Finally, the overall objective function for learning state correlation combines the matching term and the regularization term: Lrel is the overall objective function of state correlation, Lmatching is the matching term, and Lreg is the regularization term.

[0050] In one embodiment, the visual feedback task reward balance subsystem includes: The reward model designed by the multi-view video learning framework MVR realizes the flexible balance between the robot task reward rtask and the visual guidance reward rVLM. The visual guidance reward adjustment subsystem first uses the visual guidance reward rVLM to let the robot obtain the correct action mode, and then relies on the quantitative target robot task reward rtask for fine adjustment; The reward model designed by the multi-view video learning framework MVR achieves a flexible balance between the robot task reward rtask and the vision-guided reward rVLM, including: the vision-guided reward rVLM encourages the robot to explore task-related states; but when the robot gradually approaches these states, the influence of rVLM should gradually decrease, and the robot task reward rtask should play a dominant role.

[0051] The principle and effect of the above technical solution are as follows: the visual feedback task reward balance subsystem includes: a flexible balance subsystem of multiple types of rewards, and the reward model designed by the multi-view video learning framework MVR realizes a flexible balance between the robot task reward rtask and the visual guidance reward rVLM; the visual guidance reward adjustment subsystem first allows the robot to obtain the correct action mode through the visual guidance reward rVLM, and then relies on the quantitative target robot task reward rtask for fine adjustment; the reward model designed by the multi-view video learning framework MVR realizes a flexible balance between the robot task reward rtask and the visual guidance reward rVLM, including: the visual guidance reward rVLM encourages the robot to explore the states related to the task; but when the robot gradually approaches these states, the influence of rVLM should gradually decrease, and at this time the robot task reward rtask should play a leading role; The reward function designed by MVR is committed to achieving a flexible balance between the task reward rtask and the VLM-based reward rVLM. Specifically, the role of rVLM is to encourage the agent to explore task-related states, but when the agent gradually approaches these states, the influence of rVLM should gradually decrease, and rtask should play a leading role at this time. This design concept is similar to the process of human skill learning, first allowing the agent to acquire the correct action mode through rVLM, and then relying on rtask for fine-tuning. To achieve this balance, MVR introduces the concept of policy relevance. In simple terms, it comprehensively considers the relevance of states and the frequency of policy visits to these states, so as to more comprehensively measure the degree of match between policy and task. Based on policy relevance, MVR strives to maximize a set objective function when learning a policy. This objective function contains two key parts. One part is the value function of the policy, which reflects the expected return that can be obtained by executing actions according to the policy. The other part is related to policy relevance, and its influence is adjusted by a special function. The policy relevance function can be adjusted according to the specific characteristics and requirements of the task to balance the effects of these two parts. As the agent's policy gets closer and closer to the policy that best matches the task description, the influence of this part related to state relevance will gradually weaken, realizing a dynamic transformation of the reward function from rVLM-based to rtask-based: in, Strategy The value function of reinforcement learning is: Refers to and language The best matching strategy. In order to maximize the log-likelihood, Under the BT model Better match , introduced to adjust the scale of the two terms. The function log(σ(x)) is monotonically decreasing when x ≤ 0, and maintains Getting closer hour, The impact has weakened.

[0052] When calculating rVLM, MVR uses an approximate method. Since it is difficult to sample directly from the ideal strategy, MVR uses the reference set Dref mentioned earlier for approximate sampling. The state sequences in the reference set are selected based on the video-text similarity score and are considered to be more closely related to the target strategy. By sampling from the reference set, rVLM can be approximately calculated, providing the agent with a reward signal based on state relevance, guiding the agent to explore and learn towards a state that is more consistent with the task goal: Among them, E[] represents expectation, Representative from the strategy The state of sampling, represents the correlation function, σ is the sigmoid function.

[0053] In one embodiment, S4 includes: S41, the multi-view video learning framework MVR introduces policy relevance, which comprehensively considers the relevance of states and the frequency of policy visits to these states; it more comprehensively measures the degree of match between policy and task; S42, the multi-view video learning framework MVR uses approximate computing to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal; Policy relevance includes: when learning a strategy, the multi-view video learning framework MVR strives to maximize a set objective function; this objective function contains two key parts, one is the value function of the strategy, which reflects the expected return that can be obtained by performing actions according to the strategy; the other part is related to policy relevance, and its influence is adjusted by a special function; there is a hyperparameter w, which can be adjusted according to the specific characteristics and requirements of the task to balance the effects of these two parts; as the robot's strategy gets closer and closer to the strategy that best matches the task description, the influence of this part related to state relevance will gradually weaken, realizing a dynamic transformation of the reward function from rVLM-based to rtask-based.

[0054] The principle and effect of the above technical solution are as follows: the multi-view video learning framework MVR introduces policy relevance, comprehensively considers the relevance of states and the frequency of visits of the policy to these states; more comprehensively measures the degree of match between the policy and the task; the multi-view video learning framework MVR adopts approximate calculation to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn towards a state that is more consistent with the task goal; policy relevance includes: when learning a policy, the multi-view video learning framework MVR will strive to maximize a set objective function; this objective function contains two key parts, one is the value function of the policy, which reflects the expected return that can be obtained by executing actions according to the policy; the other part is related to policy relevance, and its influence is adjusted by a special function; there is a hyperparameter w, which can be adjusted according to the specific characteristics and requirements of the task to balance the effects of the two parts; as the robot's policy gets closer and closer to the policy that best matches the task description, the influence of this part related to state relevance will gradually weaken, realizing the dynamic transformation of the reward function from rVLM-based to rtask-based; The multi-view video learning framework MVR uses approximate calculations to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal. The multi-view video learning framework MVR uses an approximate calculation method to calculate rVLM, providing the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal. The multi-view video learning framework MVR uses the previously mentioned reference set Dref for approximate sampling. The robot state sequence in the reference set is selected based on the video-text similarity score and is considered to be more closely related to the target strategy. By sampling from the reference set, rVLM can be approximately calculated to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal. Multiple complex motion tasks were selected from the simulated humanoid robot benchmark for full-body motion and manipulation, including dynamic motion tasks and static posture tasks, to test the complex skills of the Unitree H1 robot; dynamic motion tasks include: running, walking, climbing stairs, and sliding dynamic tasks; dynamic motion tasks include: standing, balancing, and sitting static tasks; the robot can only obtain proprioceptive data such as joint position and angular velocity during task execution, which is closer to real-life application scenarios; during the experiment, one out of every nine trajectories was selected to render as a video to reduce computational cost and maintain data diversity; for each video, a segment of length 64 was cut, and the video-text similarity score was calculated using the ViCLIP model. The reference model was updated every 100,000 environmental steps, and an early stopping strategy was used to prevent overfitting. The optimal value of the hyperparameter w was determined by grid search from {0.01, 0.1, 0.5}; based on the TQC algorithm, the JAX version of TQC was used to conduct experiments in the SBX framework, and its parameters were fine-tuned for complex humanoid robot control tasks. Each task is trained for 10 million steps. During the training process, samples are collected from 8 parallel environments. The learning rate gradually decays from the initial 6e-4 to the final 5e-5 to maintain training stability. Each time, 256 samples are sampled from the playback buffer to update the strategy, and 16 gradient updates are performed every 16 steps. To enhance the exploration ability of the algorithm, state-dependent exploration (SDE) is enabled. Among them, the neural network is a three-layer neural network with a width of 256 per layer, the quantile count is set to 50, and the top 5 quantiles of each network are discarded. A variety of cutting-edge methods are selected for comparison with the multi-view video learning framework MVR. TQC is a classic algorithm that only relies on task rewards for policy learning. VLM-RM combines task rewards and image-text similarity scores generated by the CLIP model. FuRL is based on image-text similarity and solves the reward sparsity problem through online fine-tuning. RoboCLIP uses video-text similarity to provide rewards for online RL robots, but only gives a single reward at the end of the trajectory. In addition, the experimental results of DreamerV3 are also included for comparative analysis; the results show that the multi-view video learning framework MVR surpasses all baseline methods in average performance on multiple tasks; The multi-view video learning framework MVR can also identify states with high rewards but suboptimal, including: As shown in Figure 2, the multi-view video learning framework MVR identifies states with high task rewards but suboptimal; visualize the states generated by the TQC agent for the sit_hard task; the left part illustrates the task reward. The right part shows the combination of the task reward and the visual guidance reward calculated by the trained multi-view video learning framework MVR agent; the multi-view video learning framework MVR identifies several parts of the state space with high task reward values ​​as suboptimal, mainly corresponding to unstable sitting postures; the multi-view video learning framework MVR can also identify states with high rewards but suboptimal.

[0055] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A multi-view video reward mechanism learning framework, characterized by: include: Multi-view video evaluation subsystem, using the multi-view video learning framework MVR, to evaluate robot behavior based on multi-view video learning; The visual feedback reward feedback strategy subsystem generates visual feedback based on the task text description through the visual language large model to improve the robot's motor skill learning; based on the latest state relevance evaluation, accurate reward feedback is obtained to adjust the strategy more effectively; The visual feedback task reward balance subsystem uses the task reward model to analyze the importance of task reward feedback based on the degree to which the robot's behavior approaches the expected goal. It dynamically adjusts the relative size between the task reward and the visual language model reward based on the state correlation to balance the visual feedback and task reward. The multi-view video reward combination subsystem combines multi-view videos and task rewards to provide more accurate visual feedback and learning effects in complex robot motion tasks.

2. The multi-view video reward mechanism learning framework according to claim 1, characterized in that: Multi-view video evaluation subsystem, including: Sequence acquisition dataset subsystem: When the intelligent robot performs tasks in the environment, it collects robot state sequences at a set frequency and converts the robot state sequences into multi-view videos. At the same time, it extracts the target tasks into concise text descriptions. It calculates the similarity between the video and the text through VLM and constructs a dataset D. The dataset D contains: robot state sequences, multi-view videos corresponding to the robot state sequences, and similarity scores between the videos and the text. The matching pairwise comparison subsystem, the multi-view video learning framework MVR uses the sports competition statistical model to keep the robot state sequence comparison results as consistent as possible with the video comparison results, and learn the state correlation more accurately; Regularization interference reduction subsystem,The multi-view video learning framework MVR performs regularized state representation to reduce the interference caused by perspective changes in multi-view videos.

3. The multi-view video reward mechanism learning framework according to claim 1, characterized in that: Visual feedback reward feedback strategy subsystem, including: Trajectory rendering behavior visual subsystem, multi-view video learning framework MVR During the entire online reinforcement learning process, the trajectory generated by the robot is rendered into a video according to the set rendering frequency TRender; the robot's behavior visual information is obtained in a timely manner; Similarity score video embedding subsystem, the rendered video will be input into VLM to obtain the video-text similarity score and video embedding; this new information will be used to update the dataset D to keep the dataset containing the latest robot behavior and task matching; Buffer state sequence selection subsystem,The multi-view video learning framework MVR maintains a reference buffer Dref, which specifically stores those robot state sequences with high video-text similarity scores; these robot state sequences are considered to be more relevant to the task objectives; The multi-view video learning framework MVR uses samples in the dataset to update the model used to measure state relevance at a set update frequency TUpdate, continuously optimizing the understanding and judgment of state relevance; an offline strategy algorithm is used, and after each model update, the rewards in the robot's playback buffer need to be recalculated to maintain the consistency of the rewards, so that the robot can obtain accurate reward feedback based on the latest state relevance evaluation, and thus adjust the strategy more effectively.

4. The multi-view video reward mechanism learning framework according to claim 1, characterized in that: Visual feedback task reward balance subsystem, including: The reward model designed by the multi-view video learning framework MVR realizes the flexible balance between the robot task reward rtask and the visual guidance reward rVLM. The visual guidance reward adjustment subsystem first uses the visual guidance reward rVLM to let the robot obtain the correct action mode, and then relies on the quantitative target robot task reward rtask for fine adjustment; The reward model designed by the multi-view video learning framework MVR achieves a flexible balance between the robot task reward rtask and the vision-guided reward rVLM, including: the vision-guided reward rVLM encourages the robot to explore task-related states; but when the robot gradually approaches these states, the influence of rVLM should gradually decrease, and the robot task reward rtask should play a dominant role.

5. The multi-view video reward mechanism learning framework according to claim 1, characterized in that: Multi-view video reward combined with subsystem, including: Policy relevance matching measurement subsystem, the multi-view video learning framework MVR introduces policy relevance, comprehensively considers the relevance of states and the frequency of policy visits to these states; more comprehensively measures the degree of match between policy and task; Reward-guided exploration and learning subsystem: The multi-view video learning framework MVR uses approximate computing to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal; Policy relevance includes: when learning a strategy, the multi-view video learning framework MVR strives to maximize a set objective function; this objective function contains two key parts, one is the value function of the strategy, which reflects the expected return that can be obtained by performing actions according to the strategy; the other part is related to policy relevance, and its influence is adjusted by a special function; there is a hyperparameter w, which can be adjusted according to the specific characteristics and requirements of the task to balance the effects of these two parts; as the robot's strategy gets closer and closer to the strategy that best matches the task description, the influence of this part related to state relevance will gradually weaken, realizing a dynamic transformation of the reward function from rVLM-based to rtask-based.

6. A method for constructing a multi-view video reward mechanism learning framework, characterized in that: include: S1, using the multi-view video learning framework MVR, evaluates robot behavior based on multi-view video learning; S2, through the visual language large model, generates visual feedback based on the task text description to improve the robot's motor skill learning; based on the latest state relevance evaluation, accurate reward feedback is obtained to adjust the strategy more effectively; S3, through the task reward model, when the robot behavior is close to the expected goal, according to the degree of closeness of the robot behavior to the expected goal, analyze the importance of the task reward feedback, dynamically adjust the relative size between the task reward and the visual language model reward according to the state correlation, and balance the visual feedback and task reward; S4, combines multi-view videos and task rewards to provide more accurate visual feedback and learning effects in complex robot motion tasks.

7. The method for constructing a multi-view video reward mechanism learning framework according to claim 6, characterized in that: S1 includes: S11, when the intelligent robot performs tasks in the environment, it collects robot state sequences at a set frequency and converts the robot state sequences into multi-view videos; at the same time, it extracts the target tasks into concise text descriptions; calculates the similarity between the video and the text through VLM; and constructs a data set D; the data set D includes: robot state sequences, multi-view videos corresponding to the robot state sequences, and similarity scores between the videos and the texts; S12, the multi-view video learning framework MVR uses a sports game statistical model to keep the robot state sequence comparison results as consistent as possible with the video comparison results through matching pairwise comparisons, and learn state correlations more accurately; S13, regularized interference reduction subsystem, the multi-view video learning framework MVR performs regularized state representation to reduce the interference caused by perspective changes in multi-view videos.

8. The method for constructing a multi-view video reward mechanism learning framework according to claim 6, characterized in that S2 include: S21, the multi-view video learning framework MVR, renders the trajectory generated by the robot into a video according to the set rendering frequency TRender during the entire online reinforcement learning process; and obtains the robot's behavioral visual information in a timely manner; S22, the rendered video will be input into VLM to obtain the video-text similarity score and video embedding; this new information will be used to update the dataset D to keep the dataset containing the latest robot behavior and task matching; S23, the multi-view video learning framework MVR maintains a reference buffer Dref, which specifically stores robot state sequences with higher video-text similarity scores; these robot state sequences are considered to be more relevant to the task goal; The multi-view video learning framework MVR uses samples in the dataset to update the model used to measure state relevance at a set update frequency TUpdate, continuously optimizing the understanding and judgment of state relevance; it uses an off policy strategy algorithm, and each time the model is updated, the rewards in the robot's playback buffer need to be recalculated to maintain the consistency of the rewards, so that the robot can obtain accurate reward feedback based on the latest state relevance evaluation, and thus adjust the strategy more effectively.

9. The method for constructing a multi-view video reward mechanism learning framework according to claim 6, characterized in that S3 include: S31, the reward model designed in the multi-view video learning framework MVR achieves a flexible balance between the robot task reward rtask and the visual guidance reward rVLM; S32, first use the visual guidance reward rVLM to let the robot acquire the correct action mode, and then rely on the quantitative target robot task reward rtask for fine adjustment; The reward model designed by the multi-view video learning framework MVR achieves a flexible balance between the robot task reward rtask and the vision-guided reward rVLM, including: the vision-guided reward rVLM encourages the robot to explore task-related states; but when the robot gradually approaches these states, the influence of rVLM should gradually decrease, and the robot task reward rtask should play a dominant role.

10. The method for constructing a multi-view video reward mechanism learning framework according to claim 6, characterized in that S4 include: S41, the multi-view video learning framework MVR introduces policy relevance, which comprehensively considers the relevance of states and the frequency of policy visits to these states; it more comprehensively measures the degree of match between policy and task; S42, the multi-view video learning framework MVR uses approximate computing to provide the robot with a reward signal based on state relevance, guiding the robot to explore and learn in a state that is more consistent with the task goal; Policy relevance includes: when learning a strategy, the multi-view video learning framework MVR strives to maximize a set objective function; this objective function contains two key parts, one is the value function of the strategy, which reflects the expected return that can be obtained by performing actions according to the strategy; the other part is related to policy relevance, and its influence is adjusted by a special function; there is a hyperparameter w, which can be adjusted according to the specific characteristics and requirements of the task to balance the effects of these two parts; as the robot's strategy gets closer and closer to the strategy that best matches the task description, the influence of this part related to state relevance will gradually weaken, realizing a dynamic transformation of the reward function from rVLM-based to rtask-based.

Images