Method, device and system for training robot strategy network

Abstract

The embodiment of the invention discloses a method, device and system for training a robot strategy network. The specific embodiment of the method comprises the steps of determining a predicted action of a robot through a robot strategy network based on a sample state and a sample instruction; based on the predicted action and the sample state, potential characteristics of the robot state at the next moment after the robot executes the predicted action are determined through a world model, and a decoder in the world model is removed or disabled; based on the potential features, determining semantic features of the state of the robot at the next moment through a pre-trained adapter; based on the semantic features and the sample instructions, rewards are determined through a large language model; and adjusting parameters of the robot strategy network based on the rewards. According to the embodiment, the training speed of the robot strategy network can be improved.

Description

Methods, apparatus and systems for training robot policy networks Technical Field

[0001] The embodiments disclosed herein relate to the field of artificial intelligence technology, specifically to embodied AI, world models, reinforcement learning, and efficient collaborative training techniques for multimodal large models. Background Technology

[0002] In training embodied intelligent robots, reinforcement learning is a core method for enabling robots to learn complex skills. To accelerate training, the industry typically uses world models to simulate the physical environment and predict the robot's future state after performing actions.

[0003] To determine whether a robot has completed a task (such as "putting the apple on the plate"), a referee, or reward model, is needed. Currently, a Visual Language Model (VLM) is commonly used as the referee. The existing workflow is as follows: The world model internally calculates the latent features (compressed states) for the future. A large decoder must be called to restore these features to a high-resolution image visible to the human eye. Another large visual encoder (such as ViT) is then called to reread the image and extract semantic information. Finally, the large language model determines whether the task was successful.

[0004] Since the world model generates features that machines can understand, VLM also requires features that machines can understand. However, in existing technologies, the system is forced to "draw a graph" before "reading the graph," resulting in inefficient encoding and decoding loops.

[0005] Furthermore, reinforcement learning often requires millions of trial and error (sampling). Each trial and error involves the aforementioned cumbersome image generation and recognition process, resulting in extremely slow training speeds, typically consuming thousands of GPU hours. Summary of the Invention

[0006] Embodiments of this disclosure present methods, apparatus, and systems for training robot policy networks.

[0007] In a first aspect, embodiments of this disclosure provide a method for training a robot policy network, comprising: determining a predicted action of a robot using a robot policy network based on sample states and sample instructions; determining potential features of the robot's state at the next moment after the robot performs the predicted action using a world model based on the predicted action and sample states, wherein a decoder in the world model is removed or disabled; determining semantic features of the robot's state at the next moment using a pre-trained adapter based on the potential features; determining a reward using a large language model based on the semantic features and sample instructions; and adjusting the parameters of the robot policy network based on the reward.

[0008] In some embodiments, an adapter is generated through feature distillation based on a visual language model.

[0009] In some embodiments, generating an adapter based on a visual language model via feature distillation includes: acquiring training data based on the visual language model, wherein the training data includes a latent spatial feature vector of the robot state and a desired semantic feature vector of the robot state; determining a predicted semantic feature vector of the robot state through the adapter based on the latent spatial feature vector; and adjusting the network parameters of the adapter based on the difference between the predicted semantic feature vector and the desired semantic feature vector.

[0010] In some embodiments, acquiring training data based on a visual language model includes: determining the expected semantic feature vector of the robot state based on sample images of the robot state using a visual encoder in the visual language model; determining the latent spatial feature vector of the robot state based on the sample images using a variational autoencoder in the world model; and determining training data based on the expected semantic feature vector and the latent spatial feature vector.

[0011] In some embodiments, the method further includes: adjusting the parameters of the adapter based on rewards.

[0012] In some embodiments, the adapter includes a multilayer perceptron.

[0013] In a second aspect, embodiments of this disclosure provide a method for predicting robot actions, comprising: inputting task instructions and a current state image of the robot into a robot policy network trained according to any one of the methods in the first aspect, and outputting a predicted action.

[0014] Thirdly, embodiments of this disclosure provide a system for training a robot policy network, comprising: a robot policy network configured to determine a predicted action of a robot based on sample states and sample instructions; a world model configured to determine latent features of the robot's state at the next moment after the robot performs the predicted action based on the predicted action and sample states, wherein a decoder in the world model is removed or disabled; an adapter configured to determine semantic features of the robot's state at the next moment based on the latent features; and a large language model configured to determine a reward based on the semantic features and sample instructions; wherein the robot policy network adjusts its parameters based on the reward.

[0015] In some embodiments, the system further includes a visual language model configured to generate an adapter through feature distillation.

[0016] In some embodiments, the visual language model is further configured to: acquire training data, wherein the training data includes a latent spatial feature vector of the robot state and a desired semantic feature vector of the robot state; the adapter is further configured to: determine a predicted semantic feature vector of the robot state based on the latent spatial feature vector, and adjust the network parameters of the adapter based on the difference between the predicted semantic feature vector and the desired semantic feature vector.

[0017] In some embodiments, the visual language model includes a visual encoder, and the world model includes a variational autoencoder; the visual language model is further configured to: determine the desired semantic feature vector of the robot state based on sample images of the robot state using the visual encoder; the world model is further configured to: determine the latent spatial feature vector of the robot state based on sample images using the variational autoencoder.

[0018] In some embodiments, the adapter is further configured to adjust parameters based on rewards.

[0019] In some embodiments, the adapter includes a multilayer perceptron.

[0020] Fourthly, embodiments of this disclosure provide an apparatus for training a robot policy network, comprising: an action prediction unit configured to determine a predicted action of a robot through a robot policy network based on sample states and sample instructions; a state prediction unit configured to determine potential features of the robot's state at the next moment after the robot performs the predicted action through a world model based on the predicted action and sample states, wherein the decoder in the world model is removed or disabled; a semantic determination unit configured to determine the semantic features of the robot's state at the next moment through a pre-trained adapter based on the potential features; a reward determination unit configured to determine the reward through a large language model based on the semantic features and sample instructions; and a parameter adjustment unit configured to adjust the parameters of the robot policy network based on the reward.

[0021] Fifthly, embodiments of this disclosure provide an apparatus for predicting robot actions, comprising: a prediction unit configured to input task instructions and a current state image of the robot into a robot policy network trained according to any one of the methods in the first aspect, and output a predicted action.

[0022] In a sixth aspect, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors perform the method as described in any one of the first or second aspects.

[0023] In a seventh aspect, embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0024] Eighthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0025] The methods, apparatus, and systems for training robot policy networks provided in the embodiments of this disclosure aim to establish a "direct connection" between the world model and the reward model. By constructing an extremely lightweight adapter, the "internal ideas" (latent features) of the world model are directly translated into "language" (semantic features) that the reward model can understand, thereby skipping the image generation step and compressing image features more efficiently, achieving ultra-high-speed, high-throughput reinforcement learning reward calculation.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0027] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: FIG1 is a system architecture diagram of the training robot policy network of this disclosure; FIG2 is a flowchart of an embodiment of the method for training robot policy networks according to this disclosure; FIG3a and FIG3b are schematic diagrams of application scenarios of the method for training robot policy networks according to this disclosure; FIG4 is a structural schematic diagram of an embodiment of the device for training robot policy networks according to this disclosure; FIG5 is a structural schematic diagram of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0028] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] Figure 1 illustrates the architecture of a system for which the training robot policy network of this disclosure can be applied.

[0031] As shown in Figure 1, the system for training a robot policy network can include a robot policy network, a world model, an adapter, and a large language model. Specific functions are as follows: The robot policy network is configured to determine the robot's predicted action based on sample states and sample instructions; the world model is configured to determine the latent features of the robot's state at the next moment after the robot executes the predicted action, based on the predicted action and sample states, wherein the decoder in the world model is removed or disabled; the adapter is configured to determine the semantic features of the robot's state at the next moment based on the latent features; the large language model is configured to determine the reward based on the semantic features and sample instructions; and the robot policy network adjusts its parameters based on the reward.

[0032] In some optional implementations of this embodiment, the system further includes a visual language model configured to generate an adapter through feature distillation.

[0033] In some optional implementations of this embodiment, the visual language model is further configured to: acquire training data, wherein the training data includes a latent spatial feature vector of the robot state and a desired semantic feature vector of the robot state; the adapter is further configured to: determine a predicted semantic feature vector of the robot state based on the latent spatial feature vector, and adjust the network parameters of the adapter based on the difference between the predicted semantic feature vector and the desired semantic feature vector.

[0034] In some optional implementations of this embodiment, the visual language model includes a visual encoder, and the world model includes a variational autoencoder; the visual language model is further configured to: determine the expected semantic feature vector of the robot state based on sample images of the robot state through the visual encoder; the world model is further configured to: determine the latent spatial feature vector of the robot state based on sample images through the variational autoencoder.

[0035] In some alternative implementations of this embodiment, the adapter is further configured to adjust parameters based on rewards.

[0036] In some optional implementations of this embodiment, the adapter includes a multilayer perceptron.

[0037] The core of this scheme is to build a "graphless" training loop, which mainly includes two stages: 1. Stage 1: "Imitation learning" of the adapter (distillation stage). In this stage, a lightweight adapter, such as an MLP (Multi-Layer Perceptron), is trained to learn to imitate the functions of a large visual encoder.

[0038] Teacher model (standard answer): This can be a mature, large-scale visual model with frozen parameters (such as CLIP or SigLIP). It reads real images and outputs standard semantic features (tokens).

[0039] Student model (the trainee): This is the adapter. Its input is no longer an image, but the latent features generated by the world model.

[0040] Training objective: To force the adapter's output to infinitely approximate the teacher model's output. Once trained, this adapter becomes a "shadow encoder," capable of "guessing" semantic features directly from latent features without needing to look at the image.

[0041] 2. Phase Two: Efficient Reinforcement Learning Based on Potential Rewards (Application Phase)

[0042] In the robot policy training loop, the system completely abandons the image decoder: Action prediction: The robot policy network outputs an action, and the world model predicts the latent features of the next frame after the action is executed.

[0043] Instant Feature Bridging: The latent feature flows directly through the trained adapter. The adapter abandons complex network structures such as convolutional layers and attention layers, consisting of only 3-4 stacked fully connected layers. It has no redundant feature extraction or dimensionality transformation modules, and the number of network parameters is far lower than heavyweight modules like visual encoders and image decoders. Due to the extremely simple adapter structure (containing only simple matrix multiplication), the core advantage of this minimalist architecture is the absence of complex feature computation chains. The input latent features can be mapped along a direct path of "input layer → hidden layer → output layer," without additional feature preprocessing or post-processing steps. This step consumes almost no time, instantly transforming the feature into semantic features. The essence of a fully connected layer is matrix multiplication, a highly optimized parallel computing task on GPUs, enabling synchronous mapping of batch features at the hardware level. Compared to the multi-step computation in existing technologies (including time-consuming operations such as deconvolution, convolution, and pooling) of "latent features → decoder reconstructing the image → visual encoder extracting semantics," the computational complexity of matrix multiplication is reduced exponentially, and the mapping time for a single set of features can be compressed to milliseconds or even microseconds.

[0044] Direct scoring: These semantic features are directly fed into the LLM (Large Language Model). The LLM outputs a reward directly based on the instruction (such as "determine if objects are aligned").

[0045] Strategy Update: The robot's policy network optimizes its behavior based on this reward.

[0046] This application focuses on embodied intelligent tasks related to organizing everyday items, such as "placing a white plastic cup (12cm high, 8cm in diameter) in the center of a wooden square tray (30cm on each side)." The executor is a 7-DOF collaborative robotic arm equipped with a high-definition RGB camera (for capturing scene images). The training environment is based on a physics simulation platform built with PyTorch, which can accurately simulate the object's gravity, contact collisions, and the robotic arm's movement trajectory.

[0047] In one example, the robot policy network adopts a hybrid architecture of "visual feature extraction layer + fully connected decision layer", which can receive scene-related features and instruction-related features, perform fusion calculations, and finally output the corresponding robot arm movement instructions.

[0048] In one example, the world model is based on a variational autoencoder framework, retaining only the encoder module responsible for feature transformation and the state deduction module that simulates physical laws. The decoder-related computational logic has been removed at the code level, and the function of restoring latent features to pixel images has been completely disabled, outputting only the compressed feature representation of the scene.

[0049] In one example, the pre-trained adapter is a 3-layer lightweight fully connected network (i.e., a multilayer perceptron). It is trained by mimicking the semantic encoding capabilities of a pre-trained visual language model, and can quickly achieve the mapping and transformation of features of different modalities.

[0050] In one example, the Large Language Model (LLM) uses a lightweight model that has been fine-tuned for scene adaptation. It can understand the degree of matching between scene semantic features and task instructions and output a numerical reward for the corresponding task completion progress.

[0051] In one example, the optimization algorithm employs a proximal policy optimization algorithm to adjust the parameters of the robot policy network based on the reward signal, ensuring the stability and efficiency of the robot policy network training.

[0052] Referring again to Figure 2, a flow 200 of an embodiment of a method for training a robot policy network according to the present disclosure is shown. The method for training a robot policy network includes the following steps: Step 201, determining the robot's predicted action using the robot policy network based on sample states and sample instructions.

[0053] In this embodiment, the sample state and sample instruction are pre-prepared training data. For example, a camera mounted on a robotic arm captures an image of the current desktop scene, including a white plastic cup (located 10cm to the left of the tray), a wooden square tray, and the end effector of the robotic arm. This image is input into the encoder module of the world model, and after feature compression processing, a sample state representing the current scene is obtained. In one example, the task text instruction is "Place the white plastic cup in the center of the wooden square tray." Using a pre-trained text encoding tool, this text instruction is converted into a feature form that the model can process, i.e., the sample instruction.

[0054] The sample state and sample instructions are used as input data for the robot policy network. The policy network analyzes the gap between the current scene and the task objective through internal feature extraction and decision-making operations, and finally outputs a set of specific robot prediction actions, such as "the robotic arm translates to the location of the white plastic cup, the end effector descends to the height of the cup opening, and closes the clamp on the white plastic cup with appropriate force".

[0055] Step 202: Based on the predicted action and sample state, determine the potential characteristics of the robot's state at the next moment after the robot performs the predicted action using a world model.

[0056] In this embodiment, the decoder in the world model is removed or disabled. The sample state and the predicted action are input into the world model. The world model's state derivation module calculates the changes in the scene after the predicted action is executed, based on built-in physical simulation rules. For example, a robotic arm successfully grasps a white plastic cup and moves it 5cm above a wooden tray. Since the world model's decoder has been completely removed, the system does not generate any pixel-level scene images and directly outputs potential features that characterize the scene at the next moment.

[0057] Step 203: Based on the latent features, determine the semantic features of the robot's state at the next moment through a pre-trained adapter.

[0058] In this embodiment, the latent features output by the world model are input into a pre-trained adapter. The adapter directly converts the latent features into semantic features containing core semantic information of the current scene state through fast feature mapping operations, without going through the image encoding and decoding stage. These semantic features can clearly reflect the robot's state at the next moment, for example, "The end effector of the robotic arm is holding a white plastic cup, which is located on top of a wooden square tray."

[0059] Step 204: Determine the reward based on semantic features and sample instructions using a large language model.

[0060] In this embodiment, semantic features and sample instructions are input into the large language model. The large language model judges the degree of fit between semantic features and sample instructions through semantic understanding. For example, if the white plastic cup has been successfully clamped and is close to the target position in the center of the tray, which meets the task progress requirements, a positive reward of 0.6 is output (the reward range is from -1 to 1, and the larger the value, the closer it is to the target of the sample instruction).

[0061] In existing technologies, a Vision Language Model (VLM) is typically used to determine rewards. The structure of a VLM includes a large language model and a visual encoder. However, this application directly uses a large language model instead of a VLM, effectively abandoning the visual encoder, thus eliminating the need for image encoding and decoding and reducing training time.

[0062] Step 205: Adjust the parameters of the robot policy network based on the reward.

[0063] In this embodiment, the reward output by the large language model is used as the optimization signal, and the parameters of the robot policy network are adjusted through a proximal policy optimization algorithm. The algorithm first evaluates the advantage of the predicted action relative to historical actions, and then transmits the gradient corresponding to the reward signal to each layer of the policy network structure through backpropagation, updating the network weights and bias parameters, so that the robot policy network is more inclined to output actions that are closer to the command target in subsequent similar scenarios.

[0064] Repeat steps 201 to 205 above. After multiple rounds of iterative training, the parameters of the robot policy network tend to stabilize and can stably output a complete action sequence such as "grip a plastic cup → move it precisely to the center of the tray → release it smoothly", thus achieving the action optimization goal based on reward feedback.

[0065] The method provided by the above embodiments of this disclosure can achieve the following technical effects: 1. Order-of-magnitude improvement in training efficiency: The offline semantic feature extraction of the traditional path needs to perform three steps: "latent features → decoder generates high-definition image → visual encoder extracts semantics", which includes complex operations such as deconvolution, convolution, and pooling; the offline steps of this solution only require "sample image → encoder of world model outputs latent features → adapter maps to semantic features", and cross-modal alignment is directly completed through lightweight operations mainly based on matrix multiplication.

[0066] Compared to the traditional approach of "generating an image first and then examining it," this solution shortens the inference chain from a long, multi-module chain of "latent features → image → semantic features" to a single-module direct chain of "latent features → semantic features," reducing the length by over 90%. Simultaneously, the dimensionality of the latent features (256 / 512 dimensions) is significantly lower than the pixel dimension of a high-resolution image (e.g., 224×224×3=150528 dimensions), further reducing the length of the inference features. In reinforcement learning scenarios, this means the robot can perform more simulation exercises per unit time, significantly shortening the model convergence time, and compressing the mapping time of a single set of features to the microsecond level.

[0067] During the online training phase, the core of robot policy network optimization is the iterative cycle of "action prediction → state deduction → reward calculation → parameter update". Traditional solutions consume a significant amount of time in image generation and encoding per cycle, limiting the number of iterations per unit time. In this solution, the latent features output by the world model directly flow through the adapter to obtain semantic features in each cycle, eliminating the time consumption of image encoding and decoding, and increasing the number of simulation practice sessions per unit time by more than 10 times.

[0068] More iterations mean that the robot can explore more action strategies in the same amount of time, and the convergence time of the policy network is shortened from "days" to "hours", achieving an order-of-magnitude improvement in training efficiency.

[0069] 2. Lowering the hardware threshold: Traditional offline training requires caching a large number of high-definition images generated by the decoder. A single 224×224 RGB image occupies about 0.3MB of video memory. In batch training (such as batch size 64), a single batch of images occupies about 20MB of video memory. If the training set contains 100,000 images, the video memory requirement for caching intermediate images will exceed the capacity limit of consumer-grade graphics cards.

[0070] This solution's offline training steps only require caching latent feature vectors and expected semantic feature vectors, both of which have hundreds of dimensions. The memory usage per batch is only 1 / 1000th of traditional solutions. The training process does not require large amounts of GPU memory to cache high-resolution images, making it possible to deploy complex embodied intelligence training tasks on consumer-grade GPUs or edge devices.

[0071] Online reinforcement learning requires real-time state inference and reward calculation on the robot or edge device. Traditional solutions require large amounts of video memory for their image encoding and decoding modules, making them unsuitable for deployment on consumer-grade graphics cards (such as RTX 3060) or edge computing devices (such as Jetson Nano). This solution, however, only requires running a "lightweight adapter + world model encoder" online. Both modules have millions of parameters, with video memory usage kept below 1GB. It eliminates the need for large video memory caches for images, enabling the direct deployment of complex embodied intelligence training tasks on consumer-grade hardware.

[0072] 3. Purer semantic alignment: When traditional image decoders reconstruct images, pixel-level noise (such as blurred edges and color distortion) is generated due to model accuracy limitations. This noise is captured by the visual encoder, causing the extracted semantic features to contain interference information that is irrelevant to the task, affecting the alignment accuracy of the adapter.

[0073] The offline training step of this scheme directly targets the desired semantic features output by the visual language model, skipping the image reconstruction stage. The alignment process between latent features and semantic features does not involve any pixel data, avoiding pixel-level noise interference during image reconstruction. The adapter learns a pure mapping relationship of "latent features → semantic features", rather than a distorted mapping of "latent features → pixel noise → semantic features".

[0074] In online reinforcement learning, the stability of the reward signal directly determines the robustness of the policy network. In traditional schemes, image noise causes fluctuations in semantic features, which in turn leads to large oscillations in the reward signal (if the same state is judged as different semantics due to pixel noise, the difference in reward value can be more than 0.5). In the online steps of this scheme, the semantic features output by the adapter are not affected by pixel noise, the semantic representation of the same state is highly consistent, and the fluctuation amplitude of the reward signal of the corresponding large language model is controlled within 0.05.

[0075] Stable reward signals provide a clear direction for policy networks to optimize. The policies learned by the robot will not be "misjudged" due to pixel noise, thus having stronger robustness. The robot can still complete tasks stably under the complex interference of real-world scenarios, which helps the robot learn more robust policies.

[0076] In some optional implementations of this embodiment, an adapter is generated based on a visual language model through feature distillation. The specific steps are as follows: Step 1: Constructing a distillation training dataset. Data screening and preprocessing: From public image datasets and robot operation video frame datasets, clear, unobstructed images containing the target desktop object are selected. The image resolution is uniformly adjusted, and invalid data with blurriness and severe noise are removed. Extracting features (semantic features) output by the teacher model: Each preprocessed image is input into the visual encoder module of the visual language model. The parameters of this module are fixed (it does not participate in training). Through forward propagation, visual semantic features are output as the "standard answer" for feature distillation. Extracting features (latent features) output by the student model: The same preprocessed image is input into the encoder module of the world model (the structure of the world model is set as described above, only the encoder is retained). After feature compression and encoding, latent features are output as the input data for the adapter. Constructing sample pairs: The "latent features (input)" corresponding to each image are matched one-to-one with the "visual semantic features (labels)" to form multiple sets of distillation training sample pairs. One part is divided into a training set and the other part into a validation set.

[0077] Step 2: Build the adapter network structure

[0078] The adapter is built based on the PyTorch framework, with a network structure of 3 fully connected layers: Input layer: receives latent features; Hidden layer: 2 hidden layers, both using ReLU activation function to avoid gradient vanishing and enhance feature mapping capability; Output layer: outputs predicted semantic features, consistent with the semantic feature dimension output by the visual language model, ensuring the feasibility of loss calculation.

[0079] Step 3: Train the adapter via feature distillation

[0080] Initialize training parameters: Load the training and validation sets, initialize the network weights and bias parameters of the adapter, and set the training epochs and early stopping strategy (e.g., stop training if the validation set loss does not decrease for 5 consecutive epochs); Forward propagation training: Input the latent features from the training set into the adapter in batches. The adapter outputs predicted semantic features through the computation of each layer of the network; Loss calculation and backpropagation: Calculate the loss between the predicted semantic features and the standard semantic features output by the visual language model (mean squared error loss accounts for 70%, cosine similarity loss accounts for 30%), backpropagate the loss gradient to each layer of the adapter network through the Adam optimizer, update the weights and bias parameters, and minimize the difference between the predicted semantic features and the standard semantic features; Iterative training and validation: After each training epoch, use the validation set to evaluate the feature mapping accuracy of the adapter, adjust the training pace according to the validation set loss, and avoid overfitting; for example, if the validation set loss reaches its minimum and does not decrease for 5 consecutive epochs after training to the 68th epoch, trigger the early stopping strategy and stop training.

[0081] During distillation training, each training sample consists of visual compressed modal features (latent features) and semantic language modal features (semantic features) corresponding to the same robot state image. Although the features of the two modalities are different in form, they point to the same scene semantics (such as "robotic arm grasping a cup"). This "visual compression → semantic language" feature pairing breaks through the information boundary of a single modality and is therefore called a "cross-modal" dataset. Alignment operations are used to make them semantically precisely correspondent, i.e., "cross-modal alignment". By achieving this "cross-modal alignment", the subsequently trained adapter can learn the precise mapping of "visual compressed features → semantic language features", thus skipping the image encoding and decoding stage and directly completing the conversion from latent features to semantic features.

[0082] In some optional implementations of this embodiment, an adapter is generated based on a visual language model through feature distillation, including: acquiring training data based on the visual language model, wherein the training data includes the latent spatial feature vector of the robot state and the expected semantic feature vector of the robot state; determining the predicted semantic feature vector of the robot state through the adapter based on the latent spatial feature vector; and adjusting the network parameters of the adapter based on the difference between the predicted semantic feature vector and the expected semantic feature vector. The specific steps are as follows: Step 1: Acquiring training data based on the visual language model The core of the training data is the one-to-one correspondence of the "latent spatial feature vector of the robot state" and the "expected semantic feature vector of the robot state". The acquisition process is as follows: Data acquisition and preprocessing: Acquire a large number of robot operation scene images, covering the scene of the green cylinder in different positions (left, right, front, back) around the storage box, different postures (upright, tilted) and different operation stages of the robotic arm (not grasping, grasping, moving), uniformly adjust the image resolution, and remove blurry and severely occluded invalid images; Extract the expected semantic feature vector: Input each preprocessed image into the visual language model The visual encoder, with its parameters frozen, extracts high-dimensional visual semantic features through forward propagation to obtain the "expected semantic feature vector," which serves as the "standard target" for adapter training. It then extracts the latent space feature vector: inputting the same preprocessed image into the encoder module of the world model, performing feature compression, dimensionality reduction, and redundant information removal, outputting the "latest space feature vector," which is the adapter's input data. Finally, it constructs a training dataset: binding the "latest space feature vector" and "expected semantic feature vector" corresponding to each image to form multiple training sample pairs, which are divided into training and validation sets in an 8:2 ratio for adapter training and performance validation.

[0083] Step 2: Determine the predicted semantic feature vector through the adapter.

[0084] Initialize adapter parameters: Load training set data, randomly initialize the network weights and bias parameters of the adapter, set the number of training rounds, and adopt an early stopping strategy (e.g., terminate training if the validation set loss does not decrease for 6 consecutive rounds); Batch input and forward propagation: Input the latent spatial feature vectors from the training set into the adapter. The adapter processes the input features through matrix operations and activation functions of each fully connected network layer, performs cross-modal mapping, and finally outputs a "predicted semantic feature vector", which is the semantic interpretation result of the adapter of the latent features.

[0085] Step 3: Adjust the adapter's network parameters based on feature differences

[0086] Calculate feature difference (loss value): For each batch of predicted semantic feature vectors and expected semantic feature vectors, the difference between them is calculated using a preset loss function. For example, mean squared error loss focuses on element-wise deviation at the numerical level, and cosine similarity loss focuses on consistency deviation in semantic direction. The total loss value for each batch is obtained after weighted summation. The smaller the loss value, the better the alignment effect. Backpropagation and parameter update: The total loss value is used as an optimization signal. The loss gradient is backpropagated to each layer of the adapter network through the optimizer. The network weights and bias parameters are updated according to a preset learning rate (e.g., initial learning rate 1e-4, decaying to 1 / 10 of the original every 20 rounds) to gradually reduce the difference between the predicted semantic feature vectors and expected semantic feature vectors. Iterative training and validation: After each training round, the feature mapping accuracy of the adapter is evaluated using a validation set, and the validation set loss value is recorded. If the validation set loss value reaches the minimum value and does not decrease in the subsequent 6 rounds, an early stopping strategy is triggered, training is stopped, the network parameters of the current adapter are saved, and the adapter generation is completed.

[0087] In some optional implementations of this embodiment, training data is acquired based on a visual language model, including: determining the expected semantic feature vector of the robot state based on sample images of the robot state using a visual encoder in the visual language model; determining the latent spatial feature vector of the robot state based on the sample images using a variational autoencoder in the world model; and determining training data based on the expected semantic feature vector and the latent spatial feature vector. The specific steps are as follows: Step 1: Acquire and preprocess sample images of the robot state. Sample image acquisition: Using a camera mounted on the robotic arm, acquire full-scene images of the robot performing the block placement task, covering multiple key states: different initial positions of the orange block on the left, right, and front of the base; different operation stages such as the robotic arm not grasping, grasping, moving, and about to place; acquiring a large number of effective images; Image preprocessing: Preprocess the acquired images, ultimately retaining high-quality sample images.

[0088] Step 2: Determine the desired semantic feature vector using the visual encoder of the visual language model.

[0089] The preprocessed sample images are input one by one into the visual encoder module of the visual language model: Since the parameters of the visual language model are frozen, feature extraction is completed only through forward propagation without any parameter updates; The visual encoder performs semantic parsing and feature encoding on key information in the image (such as the position and pose of the orange blocks, the position of the gray base, and the state of the robotic arm), and finally outputs a high-dimensional feature vector for each sample image. This vector is the "expected semantic feature vector of the robot state", which is the "standard semantic target" for subsequent adapter training.

[0090] Step 3: Determine the latent space feature vectors using the variational autoencoder of the world model.

[0091] The same preprocessed sample image is simultaneously input into the VAE (Variational Autoencoder) of the world model: the VAE of the world model only enables the encoder function, and through multi-layer convolution and fully connected operations, it compresses, removes redundancy and abstracts features of the image's pixel information, discarding details and noise that are irrelevant to the task; finally, it outputs a compact feature vector for each sample image, which is the "latent spatial feature vector of the robot state", and is the input data for the subsequent adapter.

[0092] Step 4: Determine training data based on two types of feature vectors

[0093] Sample pair binding: The "expected semantic feature vector" and "latent space feature vector" corresponding to each sample image are bound one by one to ensure that the input features and target features of the same robot state correspond accurately, forming multiple sets of training sample pairs of "latent space feature vector - expected semantic feature vector"; Data set partitioning: The sample pairs are divided into training set and validation set. The training set is used for iterative optimization of the adapter parameters, and the validation set is used to evaluate the training effect in real time and avoid overfitting, finally forming complete adapter training data.

[0094] Step 5: Follow-up to adapter training

[0095] The completed training data is input into the pre-built adapter, and the adapter is trained iteratively according to the process of "input latent space feature vector → output predicted semantic feature vector → calculate the difference with the expected semantic feature vector → backpropagation to update parameters" until the feature mapping accuracy of the adapter meets the requirements (e.g., the average cosine similarity of the validation set is ≥0.91), and the adapter generation is completed.

[0096] In some optional implementations of this embodiment, the method further includes: adjusting the adapter parameters based on the reward. The specific steps are as follows: 1. Constructing a reward-related loss: Based on the reward value output by the LLM, calculate the difference between the "ideal reward (target state reward 1.0) and the current reward", and convert this difference into the "semantic bias loss" of the adapter. That is, the lower the reward, the greater the semantic bias loss and the greater the parameter adjustment range; 2. Joint backpropagation: Weight the semantic bias loss and the adapter's original "predicted semantic features - expected semantic features" loss to obtain the total loss of the adapter; 3. Parameter update: Backpropagate the gradient of the total loss to each layer of the fully connected network of the adapter through the optimizer, and adjust the network weights and bias parameters according to the preset learning rate - focus on optimizing the feature mapping weights related to "location semantics" to enhance the semantic transformation capability of information such as "object coordinates, tray area division" in the latent features; 4. Synchronization constraint: Fix the parameters of the VLM and the world model during the adjustment process, and only update the adapter parameters to avoid affecting the stability of the core feature extraction module.

[0097] The reward directly reflects the task adaptability of semantic features. Adjusting parameters based on the reward allows the semantic mapping capability of the adapter to "fit the actual task requirements" rather than being limited to the general alignment of offline distillation, thus solving the problem of "disconnect between offline training and online tasks".

[0098] In some optional implementations of this embodiment, the adapter includes a multilayer perceptron.

[0099] The fully connected nature of the multilayer perceptron enables nonlinear mapping between latent spatial features and semantic features. The mean cosine similarity between the predicted semantic features and the expected semantic features output by the adapter reaches 0.93, and the mean square error is controlled within 0.02, proving that the structure of the multilayer perceptron can accurately complete cross-modal feature alignment.

[0100] The lightweight structure of the multilayer perceptron requires no additional computing power and is perfectly integrated into the "graphless" training loop, continuing the overall design goal of high efficiency and low hardware threshold.

[0101] Referring again to Figures 3a and 3b, Figure 3a is a schematic diagram of an application scenario of a method for training a robot policy network in the prior art. Figure 3b is a schematic diagram of an application scenario of a method for training a robot policy network according to this embodiment.

[0102] In the application scenario shown in Figure 3a, the complete chain of "relying on image encoding and decoding to achieve task judgment" is presented: starting with the "latent features" output by the world model, the latent features are first restored to a human-visible scene image by a large and computationally intensive image decoder (as shown in the example of "robotic arm grasping a cup"). Then, another large visual encoder (ViT) rereads the image and extracts semantic features. Finally, the semantic features are input into the Large Language Model (LLM) to determine whether the task is completed and output a reward. This highlights the core defects of this solution: "redundant path, need to repeatedly call heavyweight modules, and time-consuming and labor-intensive".

[0103] In the application scenario shown in Figure 3b, an embodied intelligence training method based on latent spatial feature distillation is presented. This method mainly consists of two stages: the first stage is constructing and training a cross-modal adapter (offline preparation stage), and the second stage is efficient training of reinforcement learning based on this adapter (online application stage). The specific process steps are as follows: Step S1: Constructing a cross-modal feature-aligned dataset. First, the system needs to prepare data for training the adapter. The system acquires a series of image frames from existing robot operation video libraries or public datasets (such as ImageNet, Ego4D). For each input image, the system simultaneously performs processing on two paths: Path A (teacher side): The image is input into the visual encoder (e.g., the ViT module) of a pre-trained, parameter-frozen Visual Language Model (VLM) to extract high-dimensional visual semantic feature vectors. This is the "standard answer" expected to be learned by the adapter.

[0104] Path B (Student End): Input the same image into the encoding part of the variational autoencoder (VAE) of the world model to extract latent features. This is the input data for the adapter.

[0105] Step S2: Training the lightweight adapter (distillation process)

[0106] The system constructs an adapter consisting of multiple fully connected layers (MLP).

[0107] Forward propagation: The "latent features" obtained in step S1 are input into the adapter, and the adapter outputs a "predicted semantic feature" after processing.

[0108] Backpropagation: Calculate the difference (i.e., loss function) between the "predicted semantic features" and the "visual semantic feature vector" from the teacher's side in step S1. Update the adapter's parameters using gradient descent until the adapter can accurately map latent features to semantic features.

[0109] Training complete: Once the error converges, fix the adapter parameters and use them as part of the subsequent reinforcement learning process.

[0110] Step S3: Initialize the reinforcement learning environment

[0111] The training phase of the embodied intelligent robot begins. The robot policy network and world model are initialized. At this point, the system removes or disables the decoder module originally used to generate images in the world model to save GPU memory and computing resources.

[0112] Step S4: Predicting Action Outcomes Based on the World Model

[0113] At each time step of training, the robot policy network outputs an action (e.g., "move the palm forward") based on the current state. The world model receives the current state and the action, internally deduces the physical laws, and predicts the potential features of the next moment after the action is executed.

[0114] Note: In this step, the system does not need to restore the latent feature to a pixel image.

[0115] Step S5: Instantaneous semantic mapping of latent features

[0116] The system directly inputs the latent features predicted in step S4 into the adapter trained in step S2. The adapter directly outputs the semantic features corresponding to the state through fast matrix operations.

[0117] Step S6: Generate semantic reward signal

[0118] The system inputs the semantic features obtained in step S5, along with the text instructions for the human-preset task (e.g., "Please put the red square into the blue plate"), into the Large Language Model (LLM) module of VLM.

[0119] LLM understands the content of the current image based on semantic features (such as the positional relationships of objects).

[0120] LLM determines whether the current content meets the requirements of the text instruction and outputs a numerical score (i.e., reward) accordingly. For example, if the object is close to the target, a positive reward is given; if it fails, a negative reward is given.

[0121] Step S7: Iterative optimization of the strategy

[0122] The system feeds back the reward generated in step S6 to the robot policy network. The robot policy network adjusts its parameters (i.e., the learning process) based on this reward, tending to generate actions that yield higher rewards.

[0123] The system repeats steps S4 to S7 until the robot's strategy converges and it can skillfully complete the task.

[0124] This application also protects a method for predicting robot actions, comprising: inputting task instructions and a current state image of the robot into a robot policy network trained according to process 200, and outputting a predicted action. In one embodiment, the specific steps are as follows: Step 1: Prepare input data, including task instructions and a current state image of the robot.

[0125] - Task Instructions: Text-based instructions indicating task content, such as "Place the transparent glass into the left side of the brown wooden storage box." After preprocessing, these instructions are input into the instruction feature extraction branch of the robot policy network and converted into instruction feature vectors by a text encoder, retaining the core semantic information of "target object (transparent glass), container (brown wooden storage box), target location (left side)." - Current State Image: For example, real-time images of the current desktop scene are captured by a camera mounted on the robotic arm. The image includes the transparent glass (located 8cm in front of the storage box, with the rim facing upwards), the brown wooden storage box, the end effector of the robotic arm (in a ready-to-operate state), and the desktop background. The image is unblurred and unobstructed. After preprocessing (adjusting the resolution to 224×224, standardizing pixel values, and removing ambient light interference), the image is input into the image feature extraction branch of the robot policy network.

[0126] Step 2: Input the trained robot policy network and perform feature fusion and decision-making operations.

[0127] The preprocessed instruction feature vector and the current state image features are synchronously input into the trained robot policy network: the image feature extraction branch compresses and abstracts the features of the input image, and outputs a scene feature vector (containing key information such as the position of the glass, the position of the storage box, and the posture of the robotic arm); the instruction feature vector and the scene feature vector are fused across modal features through a fusion layer, and transformed into decision features through nonlinear mapping; finally, the output layer calculates and generates a set of 6-dimensional action vectors, corresponding to the specific operation parameters of the robotic arm.

[0128] Step 3: Output and parse the predicted action.

[0129] The 6-dimensional motion vector output by the policy network is processed by the parsing module and transformed into predicted actions that the robotic arm can execute. Examples include: translating 5cm along the x-axis (closer to the glass), translating 3cm along the y-axis (correcting the horizontal position), descending 6cm along the z-axis (aligning the end effector with the rim of the glass), rotating the end effector 0° (maintaining a horizontal gripping posture), applying a gripping force of 4N (adapting to the weight of the glass to prevent breakage), and holding for 1 second (ensuring stable gripping). This predicted action precisely corresponds to the task requirement of "the robotic arm moving to the glass position, smoothly gripping it, and preparing to move it to the left side of the storage box," meeting the instruction requirements and the current scene state.

[0130] Referring further to Figure 4, as an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an apparatus for training a robot policy network, which corresponds to the method embodiment shown in Figure 2, and the apparatus can be specifically applied to various electronic devices.

[0131] As shown in Figure 4, the device 400 for training a robot policy network in this embodiment includes: an action prediction unit 401, a state prediction unit 402, a semantic determination unit 403, a reward determination unit 404, and a parameter adjustment unit 405. The action prediction unit 401 is configured to determine the robot's predicted action based on sample states and sample instructions using the robot policy network. The state prediction unit 402 is configured to determine the potential features of the robot's state at the next moment after the robot executes the predicted action based on the predicted action and sample states using a world model, wherein the decoder in the world model is removed or disabled. The semantic determination unit 403 is configured to determine the semantic features of the robot's state at the next moment based on the potential features using a pre-trained adapter. The reward determination unit 404 is configured to determine the reward based on the semantic features and sample instructions using a large language model. The parameter adjustment unit 405 is configured to adjust the parameters of the robot policy network based on the reward.

[0132] In this embodiment, the specific processing of the action prediction unit 401, state prediction unit 402, semantic determination unit 403, reward determination unit 404 and parameter adjustment unit 405 of the device 400 for training the robot policy network can be referred to steps 201, 202, 203, 204 and 205 in the embodiment corresponding to Figure 2.

[0133] In some optional implementations of this embodiment, the apparatus 400 further includes a distillation unit (not shown in the figures) configured to generate an adapter based on a visual language model through feature distillation.

[0134] In some optional implementations of this embodiment, the distillation unit is further configured to: acquire training data based on a visual language model, wherein the training data includes a latent spatial feature vector of the robot state and a desired semantic feature vector of the robot state; determine a predicted semantic feature vector of the robot state through an adapter based on the latent spatial feature vector; and adjust the network parameters of the adapter based on the difference between the predicted semantic feature vector and the desired semantic feature vector.

[0135] In some optional implementations of this embodiment, the distillation unit is further configured to: determine the expected semantic feature vector of the robot state based on the sample image of the robot state by using a visual encoder in the visual language model; determine the latent spatial feature vector of the robot state based on the sample image by using a variational autoencoder in the world model; and determine training data based on the expected semantic feature vector and the latent spatial feature vector.

[0136] In some optional implementations of this embodiment, the parameter adjustment unit 405 is further configured to adjust the parameters of the adapter based on the reward.

[0137] In some optional implementations of this embodiment, the adapter includes a multilayer perceptron.

[0138] This disclosure provides one embodiment of a device for predicting robot movements, which can be specifically applied to various electronic devices.

[0139] In this embodiment, the device for predicting robot actions includes a prediction unit configured to input task instructions and a current state image of the robot into a robot policy network trained according to process 200, and output predicted actions.

[0140] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security and network security.

[0141] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0142] An electronic device includes: one or more processors; and a storage device having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement the method described in process 200.

[0143] A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in process 200.

[0144] Figure 5 illustrates a schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0145] As shown in Figure 5, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 can also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0146] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0147] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as road planning methods. For example, in some embodiments, the road planning method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the road planning method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the road planning method by any other suitable means (e.g., by means of firmware).

[0148] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0150] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0152] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0153] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be servers in distributed systems or servers incorporating blockchain technology. Servers can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0154] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0155] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a robot policy network, comprising: Based on the sample state and sample instructions, the robot's predicted action is determined through the robot policy network; Based on the predicted action and the sample state, the latent features of the robot's state at the next moment after the robot executes the predicted action are determined through a world model, wherein the decoder in the world model is removed or disabled; based on the latent features, the semantic features of the robot's state at the next moment are determined through a pre-trained adapter; based on the semantic features and the sample instruction, the reward is determined through a large language model; and the parameters of the robot policy network are adjusted based on the reward.

2. The method according to claim 1, wherein, The adapter is generated based on a visual language model through feature distillation.

3. The method according to claim 2, wherein, The method of generating the adapter based on a visual language model through feature distillation includes: acquiring training data based on the visual language model, wherein the training data includes a latent spatial feature vector of the robot state and a desired semantic feature vector of the robot state; determining a predicted semantic feature vector of the robot state through the adapter based on the latent spatial feature vector; and adjusting the network parameters of the adapter based on the difference between the predicted semantic feature vector and the desired semantic feature vector.

4. The method according to claim 3, wherein, The step of acquiring training data based on a visual language model includes: determining the expected semantic feature vector of the robot state based on sample images of the robot state using a visual encoder in the visual language model; determining the latent spatial feature vector of the robot state based on the sample images using a variational autoencoder in the world model; and determining the training data based on the expected semantic feature vector and the latent spatial feature vector.

5. The method according to claim 1, wherein, The method further includes adjusting the parameters of the adapter based on the reward.

6. The method according to any one of claims 1-5, wherein, The adapter includes a multilayer sensor.

7. A method for predicting robot actions, comprising: Input the task instructions and the robot's current state image into the robot policy network trained by the method according to any one of claims 1-6, and output the predicted action.

8. A system for training a robot policy network, comprising: The robot policy network is configured to determine the robot's predicted actions based on sample states and sample instructions; A world model is configured to determine latent features of the robot's state at the next moment after the robot performs the predicted action, based on the predicted action and the sample state, wherein the decoder in the world model is removed or disabled; an adapter is configured to determine semantic features of the robot's state at the next moment based on the latent features; a large language model is configured to determine a reward based on the semantic features and the sample instruction; wherein the robot policy network adjusts its parameters based on the reward.

9. The system according to claim 8, wherein, The system also includes a visual language model configured to generate the adapter through feature distillation.

10. The system according to claim 9, wherein, The visual language model is further configured to: acquire training data, wherein the training data includes a latent spatial feature vector of the robot state and a desired semantic feature vector of the robot state; the adapter is further configured to: determine a predicted semantic feature vector of the robot state based on the latent spatial feature vector, and adjust the network parameters of the adapter based on the difference between the predicted semantic feature vector and the desired semantic feature vector.

11. The system according to claim 10, wherein, The visual language model includes a visual encoder, and the world model includes a variational autoencoder; the visual language model is further configured to: determine the expected semantic feature vector of the robot state based on sample images of the robot state using the visual encoder; the world model is further configured to: determine the latent spatial feature vector of the robot state based on the sample images using the variational autoencoder.

12. The system according to claim 8, wherein, The adapter is further configured to adjust the parameters based on the reward.

13. The system according to any one of claims 8-12, wherein, The adapter includes a multilayer sensor.

14. An apparatus for training a robot policy network, comprising: The action prediction unit is configured to determine the robot's predicted action based on the sample state and sample instructions through the robot policy network; A state prediction unit is configured to determine the potential features of the robot's state at the next moment after the robot executes the predicted action, based on the predicted action and the sample state, using a world model, wherein the decoder in the world model is removed or disabled; a semantic determination unit is configured to determine the semantic features of the robot's state at the next moment, based on the potential features, using a pre-trained adapter; a reward determination unit is configured to determine the reward based on the semantic features and the sample instruction, using a large language model; and a parameter adjustment unit is configured to adjust the parameters of the robot policy network based on the reward.

15. A device for predicting robot actions, comprising: The prediction unit is configured to input task instructions and a current state image of the robot into a robot policy network trained by the method according to any one of claims 1-6, and output a predicted action.

16. An electronic device comprising: One or more processors; A storage device having stored one or more computer programs thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method as described in any one of claims 1-7.

17. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images