A robot control method based on cross-modal shared world model reinforcement learning

Abstract

The application discloses a robot control method based on cross-modal shared world model reinforcement learning. By using the world model to learn the dynamic characteristics of robot-environment interaction, and using the prediction model to generate virtual samples to increase the diversity of training data, through multiple iterations and optimization in the simulation environment, the method enables the robot intelligent system to better adapt to the uncertainty and changes in the actual environment. By using deep neural networks to model the state and dynamic system of the environment, and combining reinforcement learning algorithms for training and optimization. The application can be effectively applied to various robot action decision-making problem fields, including but not limited to robot walking, arm swinging, running and other behavior control. It allows the robot intelligent control system to efficiently learn and optimize strategies according to different situations and goals, thereby improving the accuracy and optimization efficiency of the robot control strategy.

Description

Technical Field

[0001] This invention relates to a robot control method in cross-modal scenarios, specifically a robot control method based on cross-modal shared world model reinforcement learning, which trains a dynamic system model for robot-environment interaction by simultaneously aligning robot trajectory data from the source and target modal modes into the latent space to optimize robot agent policies. Background Technology

[0002] In today's rapidly evolving technological landscape, reinforcement learning has become an effective tool for solving complex problems. For example, in the field of robot control, reinforcement learning techniques have been successfully applied to tasks such as autonomous navigation, object grasping, and manipulation. A robot can be viewed as an intelligent agent, and its environment includes its physical surroundings and the task objective. Through the guidance of reward signals, reinforcement learning enables robots to learn the optimal actions to take in different states, thereby achieving autonomous decision-making and control.

[0003] However, despite the significant breakthroughs that reinforcement learning has achieved in robot control, data scarcity remains a persistent challenge, limiting its application scope and effectiveness and causing a series of problems in its application in robot control. This is because collecting large amounts of sample data in real-world robot simulation environments can be expensive or time-consuming. Therefore, how to efficiently utilize existing data has become a key solution to the aforementioned problems. Designing an efficient reinforcement learning algorithm framework to address the data reuse problem in robot control decision-making methods is the focus of this invention. Summary of the Invention

[0004] Objective: To address the limitations of traditional reinforcement learning decision-making methods in robot control when dealing with data scarcity, this invention provides a robot control method based on a cross-modal shared world model reinforcement learning. This method, grounded in deep reinforcement learning and incorporating a cross-modal shared world model framework, enhances the intelligent control decision-making performance of robots. It utilizes data from both the source and target modalities to co-train a robot-environment interaction dynamics model. By co-training this model with robot trajectory data from both modalities, it gradually learns the transfer dynamics of the real environment, thereby improving the utilization of collected source modal trajectory data. To achieve this specific goal, this method imposes additional constraints on the alignment of the two modalities' data. Specifically, it includes explicit and implicit alignment methods to map unpaired data from both modalities to the same latent space, ultimately resulting in a reinforcement learning robot control method that efficiently utilizes collected data and rapidly generates high-quality samples.

[0005] Technical solution: A robot control method based on reinforcement learning of a cross-modal shared world model, specifically including the following steps:

[0006] Step 101: Determine the collected robot historical state-motion trajectory experience dataset D. src In addition, the robot's observation space and action space under the current robot control decision-making task, such as the observation space may include the robot's self-perceived state in the environment and the data received by the robot's sensors such as cameras and lidar, and the action space may include the set of all actions that the robot can take at that moment, such as moving, grasping, placing, rotating and swinging, etc.

[0007] Step 102: Initialize the parameters φ of all neural networks related to the source mode. src , The parameters φ of all neural networks related to the target mode tgt The parameters φ of all neural networks related to the shared world model and the parameters ξ of the discriminator-related neural networks are also considered. D The implicit regularization predictor-related neural network parameters ψ and all neural network parameters θ related to the robot motion control strategy; initialize the experience replay pool D. tgt .

[0008] Step 103: At time t after the robot's interaction with the environment begins, when the number of transfer pairs in the experience replay pool exceeds the minimum number of samples required for training the shared world model, the robot's historical trajectory dataset D is used for... src Extract the data sequence from time step k to time step k+L in a t ,r t Let D represent the state at time t, the action performed at time t, and the reward at time t, respectively, from the current experience replay pool D. tgt Extract the corresponding data sequence The specific meaning is the same as above: calculate all loss functions and update the neural network parameters mentioned in step 102 through backpropagation, including φ, φ tgt ,φ src ,ξ D ,ψ.

[0009] Step 104: At time t after the robot begins interacting with the environment, when the number of transfer pair data in the experience replay pool exceeds the minimum number of samples required for training the robot motion control policy network, a small batch of transfer pair sample data is randomly sampled from the experience replay pool to train the robot motion control policy network and update all neural network parameters θ related to the robot motion control policy.

[0010] Step 105: At time t after the robot begins interacting with the environment, calculate the current hidden state based on the robot's current state and historical trajectory information, infer the action the robot should take based on the obtained hidden state and make it execute the action, then collect all information of the next moment of interaction with the environment, and write all transition data into the experience replay pool.

[0011] Step 106: Repeat the training and interaction process of steps 103-105 until the training converges or the maximum number of iterations is reached. Finally, save all neural network models.

[0012] Step 107: At time t, after the start of the testing phase where the robot interacts with the real task environment, read the trained policy model and use the current robot motion control policy to process the received observations o. t Infer the action a t And execute that action, completing the robot's decision-making action at that moment.

[0013] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robot control method based on reinforcement learning of a cross-modal shared world model as described above.

[0014] A computer-readable storage medium storing a computer program that performs the robot control method based on cross-modal shared world model reinforcement learning as described above.

[0015] The method of this invention efficiently utilizes the collected cross-modal robot historical trajectory dataset. Based on the reinforcement learning method of world model, it inputs the source modality historical data and the target modality trajectory data into the world model for training. To achieve this goal, the method of this invention innovatively proposes a cross-modal shared world model framework. Through training and learning, it can not only reconstruct the robot trajectory data of the two modalities, but also achieve the goal of collaborative processing of training trajectory data from different modalities, which greatly improves the sample efficiency. This not only accelerates the learning process of the world model, but also accelerates the learning of robot behavior decision-making and control strategies.

[0016] This invention addresses the alignment problem of different modal data within a shared latent space by employing both explicit and implicit alignment regularization methods. These methods ensure that robot trajectory data from both modalities can efficiently achieve the goal of collaboratively training a shared world model. The explicit alignment regularization method uses a cycle-consistency-like approach to align the corresponding mappings between the source and target modalities in the latent space, further enhanced by an adversarial generative network. Conversely, the implicit alignment regularization method utilizes the forward and backward transition dynamics in the latent space to implicitly align data from both modalities.

[0017] Under the constraints of a cross-modal shared world model framework and multiple additional regularization loss functions, the method of this invention can effectively address the problem of scarce training data in traditional robot control methods. It can efficiently utilize a large amount of collected historical robot trajectory data. At the same time, by using feedback information from the current actual task environment, the parameters of each model learning part in the method are updated, improving the accuracy and generalization ability of the model, thereby accelerating the optimization of robot control strategy under the current task. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the interaction and training process of the robot control method based on cross-modal shared world model reinforcement learning in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the overall framework of the robot control method based on cross-modal shared world model reinforcement learning in an embodiment of the present invention. Detailed Implementation

[0020] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0021] like Figure 1 As shown, the process of the robot control method based on reinforcement learning of cross-modal shared world model is as follows: First, determine the collected robot state-motion trajectory experience dataset D. srcThe robot's observation space and action space under the current robot control task are also considered. For example, the observation space may include the robot's self-perceived state in the environment and the data received by the robot's sensors such as cameras and lidar. The action space may include the set of all actions that the robot can take at this moment, such as moving, grasping, placing, rotating, and swinging, etc. (step 101). The neural networks related to processing source modal data (generally using multilayer perceptron neural networks for low-dimensional data), target modal data (generally using convolutional neural networks for image data), shared world model (generally using recurrent neural networks), discriminator-related neural networks (multilayer perceptron neural networks), implicit regularization predictor-related neural networks (multilayer perceptron neural networks), and robot control strategy neural networks (multilayer perceptron neural networks) are initialized, and the experience replay pool is initialized (step 102). Then, during the robot's interaction with the environment, when the number of transfer pairs in the experience replay pool exceeds the minimum number of samples required for training the shared world model, data is retrieved from the robot's historical trajectory dataset D. src Extracting data sequences From Experience Replay Pool D tgt Extracting data sequences Perform one training iteration, calculate all loss functions, and update all relevant neural network parameters mentioned in step 102 via backpropagation (step 103). At time t after the robot-environment interaction begins, when the number of transfer pairs in the experience replay pool exceeds the minimum number of samples required for training the robot control policy network, randomly sample a small batch of transfer pairs from the experience replay pool to train the robot control policy network and update all neural network parameters related to the robot control policy (step 104). At time t after the robot-environment interaction begins, calculate the current hidden state based on the robot's current environmental state and historical trajectory information, infer the robot's current action based on the obtained hidden state, and execute the action. Then, obtain all information for the next moment through interaction with the environment, and write all transfer data into the experience replay pool (step 105). Repeat the above algorithm training and robot-environment interaction process until training converges or reaches the maximum number of iterations, complete the training, and finally save all neural network models (step 106). In the testing phase, read the trained policy model, use the current robot control policy to infer the received observations to obtain the behavioral action, and execute the action to complete the robot's decision action at that moment (step 107).

[0022] More specifically, the implementation methods for each step are as follows:

[0023] First, the collected dataset needs to be determined. This dataset is typically collected during the training phase of the robot's current task execution, but the specific format of the observation data in its trajectory data is not limited. Additionally, key task elements such as the robot's observation data space and action space for the current task need to be identified.

[0024] Next, in the model initialization phase, the parameters φ of all neural networks related to the source mode are initialized. src , The parameters φ of all neural networks related to the target mode tgt The parameters φ of all neural networks related to the shared world model and the parameters ξ of the discriminator-related neural networks are also considered. D The implicit regularization predictor-related neural network parameters ψ and all neural network parameters θ related to the robot control strategy; initialization of the experience replay pool D. tgt All network parameters are randomly initialized according to a normal distribution, while the neural network parameters related to the robot control strategy are based on the specific practices of the classic model-based reinforcement learning method Dreamer.

[0025] Next, the shared-world model is trained. The basic loss function of the shared-world model consists of the reconstruction loss. and dynamic loss It consists of two parts. Simply put, φ is used here. i Representation and observation space The relevant neural network, where i = {src, tgt}. Here, φ is used to represent the parameters of the shared world model. For the observation space... Reconstruct the loss function

[0026]

[0027] in This indicates that the normal distribution of the corresponding neural network output is substituted into the observed data. The probability density value at a given point is specifically calculated by taking the negative logarithm of the normal distribution mean and variance output by the neural network. The latter two terms are calculated in the same way, where r... t c represents the reward at time t. t The flag indicating whether to terminate at time t is c. t ∈{0,1}.

[0028] It uses negative log-likelihood as the reconstruction target, similar to a variational autoencoder. In the reconstruction loss function mentioned above... In the middle, h t Represents trajectory information. The first item is using φ i For state z t The corresponding observations The prediction. The second term is used to predict the reward and the state z. t Does this correspond to the end state?

[0029] When training the algorithm, in order to make the learned model as close as possible to the real environment, it is necessary to minimize the difference between the two state transition functions, that is, to calculate the following dynamic loss function.

[0030]

[0031] in Represents the Kullback-Leibler (KL) divergence. The two independent variables are the normal distributions of the outputs of the two corresponding neural networks, which are directly calculated from the mean and variance of the two distributions in practice.

[0032] In actual training, the reconstruction loss function and the dynamic loss function are weighted and combined according to weighting coefficients to form the base loss function:

[0033]

[0034] Among them, expectations This indicates that the average calculation is performed simultaneously on the source and target tasks, ∑ represents the summation operation from time step t = 1 to time step t = T, and β dyn This represents the weighting coefficient between the two terms. It is part of the hyperparameters and can be set and adjusted. It is usually set to 0.5.

[0035] If the algorithm relies solely on the aforementioned basic loss function, the shared-world model learns the source observation space. and target observation space When using a shared-world model, there is a possibility of eventual alignment failure. This makes it difficult for the algorithm to effectively utilize the collected dataset for training and learning in the current task. Therefore, in addition to the basic loss function, training a shared-world model requires the addition of an extra loss function.

[0036] First, we need to define the explicit alignment regularization loss function. Specifically, let's first consider the observations at time step t. Where t = 1, 2, ..., T. This is achieved through the target space encoder. That is, input and the preceding observation-motion trajectory h t To obtain the corresponding state z in the latent space t Subsequently, through the source observation spatial decoder To get Next, we will use the source observation space encoder. Input the result from the previous step and its preceding observation-motion trajectory Obtain the corresponding status so and Alignment can be achieved in the latent space, thus ensuring that all encoders and decoders can perform perfect encoding and decoding, ultimately yielding... The effect is as follows. (Note: All encoders and decoders mentioned above are implemented using the neural network architecture mentioned earlier.)

[0037] Therefore, at each time step t, calculate the following loss function for cycle-consistency.

[0038]

[0039] The calculation method is the same as above, using the form of the normal distribution density function.

[0040] To mitigate the potential failure of the method due to using only the aforementioned cycle consistency loss function, a simple generative adversarial network is used here to process the trajectory... Latest observations in China The corresponding hidden states are used as real samples, utilizing the observation space. of encoder As a generator, a discriminator is used to determine the generated z. t It is by encoder Or by encoder Generated.

[0041] In addition to the explicit alignment regularization loss function mentioned above, an implicit alignment regularization loss function is needed to assist in training. Specifically, we use a predictor... To predict forward transition dynamics and predictors To predict the backward transition dynamics, the forward predictor uses the current hidden state and the action taken in the current state as input to predict the next hidden state. The forward predictor network uses a two-layer neural network with two output heads, one corresponding to the deterministic part of the hidden state and the other to the random part of the hidden state, and then calculates the hidden state prediction loss function. The backward predictor uses both the deterministic and random parts of the hidden state as input to the network's state features, and then uses a two-layer neural network and an action prediction layer to predict the action between two states in the latent space. The implicit alignment regularization loss is then calculated by maximizing the prediction log-likelihood of the two predictors:

[0042]

[0043] Similarly, the negative log-likelihood is calculated using the density function derived from the normal distribution parameters output by the neural network.

[0044] After calculating all the loss functions above, backpropagation is performed, and gradient descent is used to update all the corresponding neural network parameters, including the parameters φ of all neural networks that handle source mode correlation. src , The parameters φ of all neural networks related to the target mode tgt The parameters φ of all neural networks related to the shared world model and the parameters ξ of the discriminator-related neural networks are also considered. D The implicit regularization predictor is related to the neural network parameter ψ. The above steps complete the training of the shared-world model.

[0045] Next, the robot's control policy model is trained. Using the shared-world model described above, a virtual trajectory is generated starting from the current observations. This trajectory includes a series of future observations, actions, and rewards, sampled from the model's predictions. Then, a critic network is used to estimate the value of the virtual trajectory, calculating the long-term cumulative reward for each state or state-action pair. The average expected reward under the current policy is then calculated, with the former subtracted from the latter as the advantage estimation function, representing the advantage of the current state or state-action pair relative to the average expected reward. Finally, an optimization algorithm (such as stochastic gradient descent) is used to update the parameters of the policy network based on the direction of the policy gradient. This step of training the robot's control policy model can be repeated iteratively multiple times.

[0046] The above steps constitute the training process of the shared-world model network in the method of this invention. During training, a small batch of samples of size N is randomly drawn from the experience replay pool for training. Next is the robot's interaction with the environment, the purpose of which is to collect data and store it in the experience replay pool.

[0047] During the robot's interaction with the environment, state-action-state transition data can be obtained and stored in the experience replay pool. t Actions at each moment a t Observation at the next moment t+1 Rewards t .

[0048] Specifically, at time t after the robot control strategy algorithm training begins, the current hidden state is calculated based on the robot's current state and historical trajectory information. Based on the obtained hidden state, the action 'a' that the robot should take is inferred. t It then performs the action, obtains all the information for the next moment by interacting with the environment, and writes the transfer data into the experience replay pool.

[0049] Then, repeat all the above network model training and robot-environment interaction processes until the training converges or the maximum number of iterations is reached. Finally, save all neural network model parameters after training is completed.

[0050] After the testing phase of a specific task begins, the trained neural network model is read, and the current robot control strategy is used to process the received observations. t Infer the action a t And execute that action, completing the robot's decision-making action at that moment.

[0051] During training, the specific hyperparameters were set as follows: the optimizer used was Adam with a learning rate of 1e-4; the optimizer's epsilon was 1e-8; the value function's learning rate was 3e-5; and the gradient clipping threshold was 100. The actor's learning rate was also 3e-5, with a gradient clipping threshold of 100. The batch size used during training was 16, and each batch length was 64. The recurrent neural network unit used was a GRU (Gated Recurrent Unit). The model's hidden state size was 512, the deterministic part size was 512, the random part size was 32, and the discrete part size was 32. Both the input and output layers were single-layered, and the recurrent depth was 1. Parameter sharing was not used. The discount factor γ for reinforcement learning was 0.997. The minimum number of samples required for training was 50,000.

[0052] Obviously, those skilled in the art should understand that the steps of the robot control method based on cross-modal shared world model reinforcement learning in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by the computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

Claims

1. A robot control method based on reinforcement learning of a cross-modal shared world model, characterized in that, Includes the following steps: Step 101: Determine the collected robot historical state-motion trajectory experience dataset. And the observation space and action space of robots under current robot control and decision-making tasks; Step 102: Initialize the neural networks related to processing source modal data, processing target modal data, processing shared world model, discriminator, implicit regularization predictor, and robot control strategy, and initialize the experience replay pool; Step 103: During the interaction between the robot and the environment, when the number of transfer pairs in the experience replay pool exceeds the minimum number of samples required for training the shared world model, relevant data is extracted for training, all loss functions are calculated, and the relevant neural network parameters mentioned in step 102 are updated through backpropagation. Step 104: During the interaction between the robot and the environment, when the number of transfer pair data in the experience replay pool exceeds the minimum number of samples required for training the robot motion control strategy network, a small batch of transfer pair sample data is randomly sampled from the experience replay pool to train the robot motion control strategy network and update all neural network parameters related to the robot motion control strategy. Step 105: During the interaction between the robot and the environment, calculate the current hidden state based on the robot's current state and historical trajectory information, infer the action that the robot should take based on the obtained hidden state and make it execute the action, then collect all the information of the next moment of interaction with the environment, and write all the transfer data into the experience replay pool. Step 106: Repeat the training and robot-environment interaction process of steps 103-105 until the training converges or the maximum number of iterations is reached. Finally, save all neural network models. Step 107: In the testing phase where the robot interacts with the real task environment, the trained policy model is read, the current robot motion control policy is used to infer the behavior action from the received observations and execute the action, thus completing the robot's decision action at that moment.

2. The robot control method based on cross-modal shared world model reinforcement learning according to claim 1, characterized in that, Identify the collected dataset, which was collected during the training phase of the current robot performing the task, without limiting the specific form of the observation data in its trajectory data, and identify the key elements of the robot's observation data space and action space for the current task.

3. The robot control method based on cross-modal shared world model reinforcement learning according to claim 1, characterized in that, During the model initialization phase, the parameters of all neural networks related to the source mode are initialized. , parameters of all neural networks related to the target mode , Processing parameters of all neural networks related to the shared world model Discriminator-related neural network parameters Implicit regularization predictor related neural network parameters All neural network parameters related to robot control strategy Initialize the experience replay pool ; All network parameters are randomly initialized according to a normal distribution.

4. The robot control method based on cross-modal shared world model reinforcement learning according to claim 1, characterized in that, In training a shared-world model, the basic loss function of the shared-world model consists of the reconstruction loss. and dynamic loss It consists of two parts; use Representation and observation space The relevant neural networks, among which ;use To represent the parameters of the shared world model; for the observation space Reconstruct the loss function: The reconstruction loss function uses negative log-likelihood as the reconstruction target. middle, Represents trajectory information. The first item is to use State The corresponding observations The first prediction, the second, is used to predict rewards and status. Is this the end state? This represents the reward at time t. This indicates whether the process should terminate at time t; When training the algorithm, minimize the difference between the two state transition functions, i.e., calculate the following dynamic loss function. During training, the reconstruction loss function and the dynamic loss function are weighted and combined according to weighting coefficients to form the base loss function: 。 5. The robot control method based on cross-modal shared world model reinforcement learning according to claim 4, characterized in that, If the algorithm relies solely on the basic loss function, the shared-world model learns from the source observation space. and target observation space When training a shared-world model, there may be a problem where alignment may not be successful. Therefore, in addition to using the basic loss function, an additional loss function needs to be added when training the shared-world model. First, we'll show the alignment regularization loss function, starting with the time step. Observations on ,in ;pass ,use The corresponding state in the latent space is obtained from the observation-action trajectory and the preceding actions. Subsequently, the observation space was used. The decoder, thus obtaining Next, through reconstruction The state is obtained from the observation-action trajectory and its preceding events. ,Right now ,in It's a prediction of the previous trajectory; thus and Alignment can be achieved in the implicit space, ultimately yielding... ; Therefore, at each time step Calculate the loss function for the following type of cycle consistency. To mitigate the potential failure of the method due to using only the aforementioned cycle consistency loss function, a simple adversarial generative network is used to process the trajectory. Latest observations in China The corresponding hidden states are used as real samples, utilizing the observation space. of encoder As a generator, a discriminator is used to determine the generated... It is by encoder Or by encoder Generated.

6. The robot control method based on cross-modal shared world model reinforcement learning according to claim 5, characterized in that, In addition to the explicit alignment regularization loss function, an implicit alignment regularization loss function is needed to assist in training; specifically, a predictor is used. To predict forward transition dynamics and predictors To predict the backward transition dynamics; the forward predictor uses the current hidden state and the action taken in the current state as input to predict the next hidden state; the forward predictor network uses a two-layer neural network with two output heads, one corresponding to the deterministic part of the hidden state and the other to the random part of the hidden state, and then calculates the hidden state prediction loss function; the backward predictor uses both the deterministic and random parts of the hidden state as input to the network's state features, and then uses a two-layer neural network and an action prediction layer to predict the action between two states in the latent space; then the implicit alignment regularization loss is calculated by maximizing the prediction log-likelihood of the two predictors: After calculating all the loss functions above, backpropagation is performed, and gradient descent is used to update the parameters of all corresponding neural networks, including the parameters of all neural networks that handle source mode-related parameters. , parameters of all neural networks related to the target mode , Processing parameters of all neural networks related to the shared world model Discriminator-related neural network parameters Implicit regularization predictor related neural network parameters .

7. The robot control method based on cross-modal shared world model reinforcement learning according to claim 1, characterized in that, In training the robot control policy model, a shared-world model is used. Starting from the current observations, a virtual trajectory is generated by sampling actions and model predictions, including a series of future observations, actions, and rewards. Then, a critic network is used to estimate the value of the virtual trajectory, calculating the long-term cumulative reward for each state or state-action pair, and then calculating the average expected reward under the current policy. The former minus the latter serves as the advantage estimation function, representing the degree of advantage of the current state or state-action pair relative to the average expected reward. Finally, an optimization algorithm is used to update the parameters of the policy network according to the direction of the policy gradient. The training of the robot control policy model can be repeated iteratively.

8. The robot control method based on cross-modal shared world model reinforcement learning according to claim 1, characterized in that, The training process of the shared-world model network involves randomly sampling a sample of size [value] from the experience replay pool during training. Small batches of samples are used for training; the purpose of the robot's interaction with the environment is to collect data and store it in the experience playback pool. During the interaction between the robot and the environment, state-action-state transition data can be obtained and stored in the experience playback pool; The experience replay pool stores observations at each moment. Actions at every moment Observation at the next moment Rewards ; Specifically, after the robot control strategy algorithm training begins At any given moment, the current hidden state is calculated based on the robot's current state and historical trajectory information. Based on the obtained hidden state, the action that the robot should take at that moment is inferred. It then performs the action, obtains all the information for the next moment by interacting with the environment, and writes the transfer data into the experience replay pool.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robot control method based on reinforcement learning of a cross-modal shared world model as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that performs the robot control method based on cross-modal shared world model reinforcement learning as described in any one of claims 1-8.

Images