Multi-modal reward relabeling and trajectory value training method and system, and storage medium
By employing multimodal reward relabeling and trajectory value training methods, we have solved the problems of sparse reward signals, multimodal data processing, and temporal misalignment, and constructed an efficient value model, thereby improving the robustness and safety of robot learning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Filing Date
- 2026-04-21
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies face challenges in constructing high-quality value functions, including difficulties in learning sparse reward signals, processing multimodal unstructured data, temporal misalignment, distribution offset, and uncertainty, leading to poor learning performance for robots.
We employ a multimodal reward relabeling and trajectory value training method. By freezing the pre-trained visual-language model to extract features, and combining it with robot ontology perception, we perform multimodal stochastic variational representation learning and dual-channel reward relabeling to construct a high-quality value model.
It significantly improves the training efficiency of value networks in complex long-sequence tasks, enhances the robustness and safety of robot strategies, and strengthens the ability to understand multimodal data.
Smart Images
Figure CN122392061A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence and robotics cross-control technology, specifically to a multimodal reward relabeling and trajectory value training method, system, and storage medium. Background Technology
[0002] In the field of robot learning, offline reinforcement learning is gradually becoming the mainstream paradigm. It allows agents to learn policies from large-scale static historical datasets without engaging in costly and dangerous online interactions in real-world environments. With the rise of large-scale VLA models such as RT-2, OpenVLA, π0.5, and X-VLA, how to post-train these pre-trained models to achieve better performance metrics for specific downstream tasks has become a research hotspot.
[0003] In the post-training process of VLA models, a high-quality value function or reward model is crucial. It can serve not only as an indicator for offline policy evaluation, but also as a heuristic function in the model-based planning phase, and even be used to further optimize VLA policies through reinforcement learning based on human feedback or AI feedback.
[0004] However, existing technologies face the following serious challenges in constructing high-quality value functions:
[0005] (1) Difficulty in learning sparse reward signals: Many robotic operation tasks (such as precision assembly and non-rigid object manipulation) only provide sparse binary success signals at the moment of task completion. When training a value network based on such sparse signals, the value network has difficulty capturing fine-grained state progress in long-horizon tasks, resulting in extremely large variance in value estimation and difficulty in convergence.
[0006] (2) Multimodal unstructured data processing: VLA models for real-world tasks typically include high-dimensional visual images, ontology perception, and natural language instructions as input. Traditional reward designs often rely on manual feature engineering, making it difficult to fully utilize the semantically rich and modally aligned features extracted by "visual-language" models such as SigLIP, Qwen2.5-VL, InternVL-3, and Gemma. This results in poor generalization of the reward function to unseen scenes, objects, and text instructions with similar semantics.
[0007] (3) Temporal misalignment: When using expert demonstrations to guide suboptimal trajectory learning, the speed of movement of different operators varies greatly. Traditional imitation rewards based on Euclidean distance require strict temporal alignment, which is almost impossible to meet in real data, resulting in the reward signal being full of noise.
[0008] (4) Distribution skew and uncertainty: Offline data often has limited coverage. Traditional deterministic value networks often overestimate the value distribution when faced with out-of-distribution states, and lack modeling of the value distribution, making it impossible for agents to assess their own uncertainty risk.
[0009] Therefore, to address the above issues, we need multimodal reward relabeling and trajectory value training methods, systems, and storage media. Summary of the Invention
[0010] The purpose of this invention is to provide a method, system, and storage medium for multimodal reward relabeling and trajectory value training. This invention can automatically transform offline datasets containing both success and failure samples into high-quality datasets with dense value labels; significantly improve the training efficiency of value networks in complex long-sequence tasks; and the trained value model can serve as a source of differentiable optimization targets during the post-training of VLA models.
[0011] This invention is implemented as follows:
[0012] This invention provides a multimodal reward relabeling and trajectory value training method, comprising parsing the original HDF5 data into a standardized trajectory sequence, using a pre-trained visual-language model with frozen internal gradients as a feature extractor to extract high-dimensional semantic feature vectors from RGB images, wherein the feature extractor reads the original trajectory data from the HDF5 file. Next, feature extraction is performed, loading the pre-trained InternVL3-1B as a visual-language encoder, freezing the VLM parameters, and processing each frame of multi-view RGB image. Combined with task text instructions, forward reasoning is performed to extract... The feature vector corresponding to the CLS token At the same time, adjust the joint angles of the robot body. With end pose Normalization is performed, and then mapped to MLP. .
[0013] Furthermore, the semantic feature vector is concatenated and fused with the normalized ontology perception vector to construct multimodal input features. Specifically, visual features and ontology features are concatenated to obtain the final state representation, as shown in the following expression:
[0014] .
[0015] The standardized trajectory sequence is as follows: The high-dimensional semantic feature vector is The ontology perception vector is .
[0016] Furthermore, multimodal stochastic variational representation learning is performed based on the input features to obtain a joint feature vector; specifically, the multimodal stochastic variational representation learning based on the input features to obtain the joint feature vector includes:
[0017] First, feature fusion encoding is performed, and the time step is defined. The multimodal observation dataset is expressed as follows:
[0018]
[0019] Construct a two-stream encoder and use an MLP to extract the visual-linguistic features output from the VLM. With ontological features Mapping to the same latent space and concatenating them, we obtain a joint feature vector, expressed as follows:
[0020]
[0021] Secondly, in order to capture environmental uncertainty, a variational inference structure is introduced into the design of the random latent variable layer. The network does not directly output a fixed vector, but instead predicts the latent variables. posterior distribution mean With the diagonal covariance matrix To ensure the differentiability of gradient backpropagation, reparameterized sampling is used, as shown in the following expression:
[0022]
[0023] Among the noise Then, the lower bound of the evidence is maximized through variational objective optimization. The expression for minimizing the mixture loss function is as follows:
[0024]
[0025] Among them, the prior distribution ; The coefficients of the KL divergence loss term.
[0026] Furthermore, based on the optimal transmission dual-channel reward relabeling mechanism, a reward sequence is obtained. The target value is then calculated recursively in reverse order on the relabeled reward sequence, followed by value distribution statistics and normalization. For the calculated reward sequence, a global discount value backtracking is performed. Specifically, the target value is calculated recursively in reverse order on the relabeled reward sequence. The expression for calculating the target value recursively in reverse order on the relabeled reward sequence is as follows:
[0027]
[0028] Furthermore, value distribution statistics and normalization are performed, traversing the entire dataset. Calculate the value of all time steps Find the global extreme value and ;
[0029] Apply a linear transformation to normalize the value to the output range of the target neural network. Or configured by parameters The expression is as follows:
[0030]
[0031] The normalized neural network output range is used for distributed value prediction and inference. The learned distribution is then used to calculate the expected value as the final scalar score, outputting the value model. (Normalized...) As the final regression target or input benchmark for the discretized distribution.
[0032] The aforementioned distributed value prediction and inference process first involves discretizing the distribution model, dividing the continuous value space into... Each discrete bin defines a support set. ,in Evenly distributed in the normalized interval ;
[0033] In terms of probability distribution prediction, the value network outputs a Logits vector, which, after softmax normalization, yields the probability distribution for each bin, as shown in the following expression:
[0034]
[0035] To perform distributed loss supervision, the target value Project onto the nearest bin, or construct the target distribution using two-point projection. The multi-class cross-entropy loss is used for training, as shown in the following expression:
[0036]
[0037] The expected value is calculated using the learned distribution as the final scalar score, which not only serves as a criterion for the quality of the strategy, but its variance can also be used as a measure of uncertainty, as shown in the following expression:
[0038] .
[0039] Furthermore, the dual-channel reward relabeling mechanism based on optimal transmission includes sparse and dense channels, which are then mixed and fused. First, in the sparse channel, the success or failure of the trajectory is determined, and the cumulative environmental reward for each offline trajectory τ is calculated, as shown in the following expression:
[0040]
[0041] Set success threshold (e.g., 300); Define a binary indicator function with the following expression:
[0042]
[0043] Then according to Assign basic sparse rewards to each time step ;when and The reward is This indicates positive reinforcement; when and The reward is This indicates a negative penalty; the remaining time steps Then give The time step penalty. In dense channels, using optimal transport theory, the expert trajectory distribution is defined as follows:
[0044]
[0045] The distribution of the trajectory to be measured is expressed as follows:
[0046]
[0047] First, construct the cost matrix. Calculate elements The cosine distance is expressed as follows:
[0048]
[0049] or Euclidean distance Next, the Wasserstein distance is calculated, as shown in the following expression:
[0050]
[0051] The optimal transmission cost is then transformed into a frame-by-frame dense reward, as shown in the following expression:
[0052]
[0053] Ultimately, through Calculate the mixed reward, where This is the balance coefficient.
[0054] Furthermore, this invention provides a multimodal reward relabeling and trajectory value training system, including a data parsing and VLM feature extraction module, which is used to extract high-dimensional semantic feature vectors from RGB images and concatenate and fuse them with normalized ontology perception vectors to construct multimodal input features;
[0055] The multimodal stochastic variational representation learning module maps the visual-linguistic features and ontology features output by the VLM part to the same latent space and concatenates them to obtain a joint feature vector through feature fusion encoding and MLP.
[0056] The dual-channel reward relabeling mechanism module mixes and merges sparse and dense channels to calculate the reward sequence.
[0057] The global discount value backtracking module normalizes the calculated reward sequence to the output range of the target neural network, serving as the final regression target or the input benchmark for the discretized distribution.
[0058] The distributed value prediction and reasoning module performs discretized distribution modeling and uses the learned distribution to calculate the expected value as the final scalar score.
[0059] Furthermore, the present invention provides a storage medium including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal reward relabeling and trajectory value training method described above.
[0060] Compared with the prior art, the beneficial effects of the present invention are:
[0061] 1. Maximize data value: It can extract dense signals similar to expert strategies from offline data containing a large number of failed attempts, significantly reducing the reliance on high-quality demonstration data.
[0062] 2. Temporal robustness: The Wasserstein distance-based metric enables the system to be compatible with trajectory data of different operating speeds and durations, solving the alignment problem in imitation learning.
[0063] 3. Risk sensitivity: Distributed value output provides more statistical information (such as variance and risk tails) in addition to expected value, supporting safer robot strategy deployment.
[0064] 4. Universality and scalability: By using VLM as the feature extractor, the reward model naturally possesses the ability to understand the semantics of general objects and scenes, and is easy to transfer to different VLA models. Attached Figure Description
[0065] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.
[0066] Figure 1 This is a flowchart of the method of the present invention;
[0067] Figure 2 This is a data execution flowchart of an embodiment of the present invention;
[0068] Figure 3 This is a system structure diagram of the present invention. Detailed Implementation
[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to describe selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0070] Please see Figure 1 This invention provides a multimodal reward relabeling and trajectory value training method, which includes parsing the original HDF5 data into a standardized trajectory sequence, using a pre-trained visual-language model with frozen internal gradients as a feature extractor to extract high-dimensional semantic feature vectors from RGB images, wherein the feature extractor reads the original trajectory data from the HDF5 file. Next, feature extraction is performed, loading the pre-trained InternVL3-1B as a visual-language encoder, freezing the VLM parameters, and processing each frame of multi-view RGB image. Combined with task text instructions, forward reasoning is performed to extract... The feature vector corresponding to the CLS token. At the same time, adjust the joint angles of the robot body. With end pose Normalization is performed, and then mapped using MLP. .
[0071] In this embodiment, the semantic feature vector and the normalized ontology perception vector are concatenated and fused to construct multimodal input features. Specifically, visual features and ontology features are concatenated to obtain the final state representation, as shown in the following expression:
[0072] .
[0073] The standardized trajectory sequence is as follows: The high-dimensional semantic feature vector is... The ontology perception vector is .
[0074] In this embodiment, multimodal stochastic variational representation learning is performed based on the input features to obtain a joint feature vector; specifically, the multimodal stochastic variational representation learning based on the input features to obtain the joint feature vector includes:
[0075] First, feature fusion encoding is performed, and the time step is defined. The multimodal observation dataset is expressed as follows:
[0076]
[0077] Construct a two-stream encoder and use an MLP to extract the visual-linguistic features output from the VLM. With ontological features Mapping to the same latent space and concatenating them, we obtain a joint feature vector, expressed as follows:
[0078]
[0079] Secondly, in order to capture environmental uncertainty, a variational inference structure is introduced into the design of the random latent variable layer. The network does not directly output a fixed vector, but instead predicts the latent variables. posterior distribution mean With the diagonal covariance matrix To ensure the differentiability of gradient backpropagation, reparameterized sampling is used, as shown in the following expression:
[0080]
[0081] Among the noise Then, the lower bound of the evidence is maximized through variational objective optimization. The expression for minimizing the mixture loss function is as follows:
[0082]
[0083] Among them, the prior distribution ; The coefficients of the KL divergence loss term.
[0084] In this embodiment, a reward sequence is obtained based on the optimal transmission dual-channel reward relabeling mechanism. The target value is then calculated recursively in reverse order on the relabeled reward sequence, followed by value distribution statistics and normalization. For the calculated reward sequence, a global discount value backtracking is performed. Specifically, the target value is calculated recursively in reverse order on the relabeled reward sequence. The expression for calculating the target value in reverse order on the relabeled reward sequence is as follows:
[0085]
[0086] In this embodiment, value distribution statistics and normalization are then performed, traversing the entire dataset. Calculate the value of all time steps Find the global extreme value and ;
[0087] Apply a linear transformation to normalize the value to the output range of the target neural network. Or configured by parameters The expression is as follows:
[0088]
[0089] The normalized neural network output range is used for distributed value prediction and inference. The learned distribution is then used to calculate the expected value as the final scalar score, outputting the value model. (Normalized...) As the final regression target or input benchmark for the discretized distribution.
[0090] The aforementioned distributed value prediction and inference process first involves discretizing the distribution model, dividing the continuous value space into... Each discrete bin defines a support set. ,in Evenly distributed in the normalized interval ;
[0091] In terms of probability distribution prediction, the value network outputs a Logits vector, which, after softmax normalization, yields the probability distribution for each bin, as shown in the following expression:
[0092]
[0093] To perform distributed loss supervision, the target value Project onto the nearest bin, or construct the target distribution using two-point projection. The multi-class cross-entropy loss is used for training, as shown in the following expression:
[0094]
[0095] The expected value is calculated using the learned distribution as the final scalar score, which not only serves as a criterion for the quality of the strategy, but its variance can also be used as a measure of uncertainty, as shown in the following expression:
[0096] .
[0097] In this embodiment, the dual-channel reward relabeling mechanism based on optimal transmission includes sparse and dense channels that are mixed and fused. First, in the sparse channel, the success or failure of the trajectory is determined, and the cumulative environmental reward for each offline trajectory τ is calculated, as shown in the following expression:
[0098]
[0099] Set success threshold (e.g., 300); Define a binary indicator function with the following expression:
[0100]
[0101] Then according to Assign basic sparse rewards to each time step ;when and The reward is This indicates positive reinforcement; when and The reward is This indicates a negative penalty; the remaining time steps Then give The time step penalty. In dense channels, using optimal transport theory, the expert trajectory distribution is defined as follows:
[0102]
[0103] The distribution of the trajectory to be measured is expressed as follows:
[0104]
[0105] First, construct the cost matrix. Calculate elements The cosine distance is expressed as follows:
[0106]
[0107] or Euclidean distance Next, the Wasserstein distance is calculated, as shown in the following expression:
[0108]
[0109] The optimal transmission cost is then transformed into a frame-by-frame dense reward, as shown in the following expression:
[0110]
[0111] Ultimately, through Calculate the mixed reward, where This is the balance coefficient.
[0112] In this embodiment, as Figure 3 This invention provides a multimodal reward relabeling and trajectory value training system, including a data parsing and VLM feature extraction module, which is used to extract high-dimensional semantic feature vectors from RGB images and splice and fuse them with normalized ontology perception vectors to construct multimodal input features;
[0113] The multimodal stochastic variational representation learning module maps the visual-linguistic features and ontology features output by the VLM part to the same latent space and concatenates them to obtain a joint feature vector through feature fusion encoding and MLP.
[0114] The dual-channel reward relabeling mechanism module mixes and merges sparse and dense channels to calculate the reward sequence.
[0115] The global discount value backtracking module normalizes the calculated reward sequence to the output range of the target neural network, serving as the final regression target or the input benchmark for the discretized distribution.
[0116] The distributed value prediction and reasoning module performs discretized distribution modeling and uses the learned distribution to calculate the expected value as the final scalar score.
[0117] In this embodiment, the present invention provides a storage medium including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multimodal reward relabeling and trajectory value training method described in any one of the above descriptions.
[0118] In this embodiment, as Figure 2 The specific steps are as follows:
[0119] Automatic relabeling and training system for value models based on offline trajectory data during VLA model post-training process
[0120] This embodiment is based on the Python PyTorch framework and integrates the POT (Python Optimal Transport) library with the InternVL3-1B pre-trained model.
[0121] Multimodal data parsing and feature pre-extraction: First, read the raw trajectory data from the HDF5 file. Next, feature extraction is performed, loading the pre-trained InternVL3-1B as a "visual-language" encoder. The VLM parameters are then frozen, and the multi-view RGB image for each frame is processed. Perform forward inference to extract the feature vector corresponding to the [CLS] token. .
[0122] At the same time, the robot's joint angles With end pose Normalization is performed, and then mapped to MLP. .
[0123] Finally, the visual features and ontological features are concatenated to obtain the final state representation. .
[0124] In this embodiment, through expert benchmark screening, the system traverses all recorded episodes and calculates the cumulative reward of the original environment for each trajectory. .according to Sort by value and select the top-K trajectories as the expert set. In the absence of environmental rewards, it is possible to manually specify 1-2 perfect demos as a benchmark.
[0125] Dual-channel reward relabeling based on optimal transmission involves calculating the dual-channel reward in this step. In the sparse channel (channel A), a base threshold of "sparse_relabel_reward_threshold=300.0" is set. If the total trajectory score... If the mission is deemed a failure, a final reward will be given. ,
[0126] Corresponding to "sparse_relabel_failure_reward"; if The mission is deemed successful, and a final reward will be given. The corresponding value is "sparse_relabel_success_reward", which maintains consistency with the optimal transmission distance; intermediate steps are penalized with "sparse_relabel_step_reward=-1.0".
[0127] In this embodiment, in the dense reward channel, the optimal transmission core algorithm is used to process each channel of length [missing information]. Training trajectory With length Expert trajectory .
[0128] First, build Cost matrix , of which elements It can be configured as cosine distance or Euclidean distance (controlled by "dense_relabel_cost_func_type"). Call "ot.emd" to solve for the optimal transmission plan. And calculate the total Wasserstein distance.
[0129] In this embodiment, the present invention innovatively allocates the total distance back to each time step, defining... Indicates the first The contribution of each step, and Linear mapping to a specified interval (e.g., [-1.0, 0.0]). The final fusion reward is calculated as follows: .
[0130] In this embodiment, value backtracking and global normalization are performed. Bellman reverse backtracking is executed on the relabeled reward sequence, as shown in the following expression.
[0131] Among the discount factors Set it to 0.99. After completing the backtracking, calculate the value range of the entire dataset. And perform Min-Max normalization to map it to The interval is used to facilitate subsequent probability classification training.
[0132] In this embodiment, a multimodal stochastic variational network is constructed and trained. The constructed network architecture includes an encoder and a value head. The encoder uses a two-stream structure to receive images and ontology, which are then fused and fed into a StochasticLayer for output. and The latent dimension "stochastic_encoder_dim" is set to 64. The value header receives the sampled data. The MLP outputs a Logits vector of dimension 101, named "value_head_output_bins=101".
[0133] The training objective consists of two parts. The VAE loss expression is as follows:
[0134] Among them, "encoder_kl_weight=0.01" and distributed value loss
[0135] in Discretized into soft label distribution The AdamW optimizer was used with a learning rate of "lr=5e-5" and a weight decay of "weight_decay=0.01", along with the CosineAnnealing learning rate scheduling strategy.
[0136] Offline strategy evaluation and deployment; during the online inference phase, the model outputs the probability distribution of 101 bins. Expectation value is calculated. This serves as a performance evaluation. If the score falls below a preset threshold, the system will trigger the VLA model's "SafetyStop" or switch to a conservative strategy to ensure operational safety.
[0137] Finally, a standard application execution model is trained based on the set security thresholds.
[0138] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations will be apparent to those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A multimodal reward relabeling and trajectory value training method, characterized by: include: The original HDF5 data is parsed into a normalized trajectory sequence. A pre-trained visual-language model with frozen internal gradients is used as a feature extractor to extract high-dimensional semantic feature vectors from the RGB images. The semantic feature vector is concatenated and fused with the normalized ontology perception vector to construct multimodal input features; Multimodal random variational representation learning is performed based on the input features to obtain a joint feature vector; Then, based on the optimal transmission dual-channel reward relabeling mechanism, the reward sequence is obtained; the target value is obtained by recursively calculating the relabeled reward sequence in reverse order, and then the value distribution statistics and normalization are further performed. The normalized neural network output range is used for distributed value prediction and inference. The expected value is calculated using the learned distribution as the final scalar score, and the value model is output.
2. The multimodal reward relabeling and trajectory value training method according to claim 1, characterized in that: The feature extractor reads raw trajectory data from the HDF5 file. Next, feature extraction is performed, loading the pre-trained InternVL3-1B as a visual-language encoder, freezing the VLM parameters, and processing each frame of multi-view RGB image. Combined with task text instructions, forward reasoning is performed to extract... The feature vector corresponding to the CLS token At the same time, adjust the joint angles of the robot body. With end pose Normalization is performed, and then mapped to MLP. Visual-linguistic features are concatenated with ontological features to obtain the final multimodal state representation, expressed as follows: 。 3. The multimodal reward relabeling and trajectory value training method according to claim 2, characterized in that: The standardized trajectory sequence is as follows: The high-dimensional semantic feature vector is The ontology perception vector is .
4. The multimodal reward relabeling and trajectory value training method according to claim 1, characterized in that: The aforementioned multimodal random variational representation learning based on input features to obtain a joint feature vector specifically includes: First, feature fusion encoding is performed, and the time step is defined. The multimodal observation dataset is expressed as follows: ; Construct a two-stream encoder and use an MLP to extract the visual-linguistic features output from the VLM. With ontological features Mapping to the same latent space and concatenating them, we obtain a joint feature vector, expressed as follows: ； Reparameterized sampling is used, as shown in the following expression: Noise Then, the lower bound of the evidence is maximized through variational objective optimization. The expression for minimizing the mixture loss function is as follows: Among them, the prior distribution ; The coefficients of the KL divergence loss term.
5. The multimodal reward relabeling and trajectory value training method according to claim 1, characterized in that: The dual-channel reward relabeling mechanism based on optimal transmission first determines the success or failure of a trajectory in the sparse channel and calculates the cumulative environmental reward for each offline trajectory τ, as shown in the following expression: Set success threshold Define a binary indicator function with the following expression: ; Then according to Assign basic sparse rewards to each time step ;when and The reward is This indicates positive reinforcement; when and The reward is This indicates a negative penalty; the remaining time steps Then give The time step penalty. In dense channels, using optimal transport theory, the expert trajectory distribution is defined as follows: The distribution of the trajectory to be measured is expressed as follows: .
6. The multimodal reward relabeling and trajectory value training method according to claim 5, characterized in that: First, construct the cost matrix. Calculate elements The cosine distance is expressed as follows: ; or Euclidean distance Next, the Wasserstein distance is calculated, as shown in the following expression: The optimal transmission cost is then transformed into a frame-by-frame dense reward, expressed as follows: ; Ultimately, through Calculate the mixed reward, where This is the balance coefficient.
7. The multimodal reward relabeling and trajectory value training method according to claim 1, characterized in that: The target value expression obtained by recursively calculating the relabeled reward sequence in reverse order is as follows: Then, value distribution statistics and normalization are performed, traversing the entire dataset. Calculate the value of all time steps Find the global extreme value and ; Apply a linear transformation to normalize the value to the output range of the target neural network. Or configured by parameters The expression is as follows: Normalized As the final regression target or input benchmark for the discretized distribution.
8. The multimodal reward relabeling and trajectory value training method according to claim 1, characterized in that: The aforementioned distributed value prediction and inference process first involves discretizing the distribution model, dividing the continuous value space into... Each discrete bin defines a support set. ,in Evenly distributed in the normalized interval ; In terms of probability distribution prediction, the value network outputs a Logits vector, which, after softmax normalization, yields the probability distribution for each bin, as shown in the following expression: ; To perform distributed loss supervision, the target value Project onto the nearest bin, or construct the target distribution using two-point projection. The multi-class cross-entropy loss is used for training, as shown in the following expression: ; The expected value is calculated using the learned distribution as the final scalar score, as shown in the following expression: .
9. A multimodal reward relabeling and trajectory value training system, characterized in that: include: The data parsing and VLM feature extraction module is used to extract high-dimensional semantic feature vectors from RGB images and concatenate and fuse them with normalized ontology-aware vectors to construct multimodal input features. The multimodal stochastic variational representation learning module maps the visual-linguistic features and ontology features output by the VLM part to the same latent space and concatenates them to obtain a joint feature vector through feature fusion encoding and MLP; the dual-channel reward relabeling mechanism module mixes and fuses sparse and dense channels to calculate the reward sequence. The global discount value backtracking module normalizes the calculated reward sequence to the output range of the target neural network, serving as the final regression target or the input benchmark for the discretized distribution. The distributed value prediction and reasoning module performs discretized distribution modeling and uses the learned distribution to calculate the expected value as the final scalar score.
10. A storage medium comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal reward relabeling and trajectory value training method according to any one of claims 1-8.