A robot planning method and system based on multi-modal large model predictive control

Abstract

The application provides a robot planning method and system based on multimodal large model predictive control, which utilizes the powerful visual reasoning and visual positioning capabilities of the multimodal large model, samples candidate action sequences through target images or text instructions of robot movement and current observation images of the robot, and then predicts future state videos of the sampled candidate action sequences to guide the robot operation, complex path planning of rotation and interaction with scene objects, avoids the need for manual design of basic actions and skills, and breaks through the limitation that previous methods based on multimodal large models can only form rough trajectories without foresight, and realizes robot control planning based on future state prediction.

Description

Technical Field

[0001] This invention belongs to the field of robot control technology, and in particular relates to a robot planning method and system based on multimodal large model predictive control. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The emergence of large models has demonstrated powerful capabilities in knowledge extraction and reasoning. Consequently, exploration based on large models is rapidly progressing across multiple fields, including computer vision, AI science, healthcare, and robotics. Recently, a significant amount of work has made substantial progress in integrating large models into robotics. These works typically leverage multimodal data (including language, images, and video) and the powerful understanding and reasoning capabilities of general-purpose large models to enhance robots' perception and decision-making abilities.

[0004] To achieve knowledge transfer from large models to robots, most early work focused on robot planning, directly utilizing large language models (LLMs) to decompose high-level natural language commands and tasks into low-level and predefined basic actions or skills. While such approaches intuitively enable robots to perform complex and long-term tasks, they lack visual perception capabilities. Therefore, they heavily rely on predefined motor skills to interact with specific physical entities, limiting the flexibility of robot planning. Recent work addresses this issue by integrating large-scale visual-language models (VLMs), improving scene perception and adaptively generating trajectories for robot manipulation in complex scenarios without using predefined basic actions and skills. Although existing methods have shown fruitful results in incorporating basic models into robot manipulation, interacting with a wide variety of objects and humans in the real world remains a challenge. Specifically, because the decision loops of these methods do not adequately consider the robot's future states, the reasoning of large models is primarily based on current observations, leading to insufficient forward-looking planning. For example, when opening a drawer, the state-of-the-art VLM-based method VOXPOSER cannot directly generate an accurate trajectory to pull the drawer handle because of the lack of prediction of future states; therefore, it still requires designing specific object-level interaction actions or skills. Therefore, it is advisable to develop a robotic framework with a similar human ability to "think before you act".

[0005] Model predictive control (MPC) is a widely used control strategy in robotics. MPC possesses the excellent property of predicting the future state of a system through a predictive model. This forward-looking property allows robots to plan their actions by considering potential future scenarios, thereby enhancing their ability to dynamically interact with a variety of environments. Traditional MPC typically constructs a deterministic, complex dynamic model for a specific task and environment, which is not well-suited to the complex scenarios of the real world. Recent research has explored the use of vision-based predictive models to learn dynamic models from visual input and predict high-dimensional future states in 2D or 3D space. These methods sample and filter actions in the MPC loop based on current visual observations, enabling robots to make more rational decisions based on visual cues. However, the effectiveness of these methods is constrained by the inherent limitations of vision-based predictive models trained on limited datasets. These models struggle to accurately predict scenarios or situations involving objects they have not encountered before. This problem becomes particularly pronounced in real-world environments, which are often partially or even completely unfamiliar to robots, where models can only perform basic tasks closely related to their training data.

[0006] In summary, the existing problems in the application of large-scale models in robot perception and planning algorithms include:

[0007] 1. Existing large-scale models rely primarily on current observations for reasoning, lacking predictions of future robot and scene interactions, resulting in insufficient forward-looking planning;

[0008] 2. The performance of Model Predictive Control (MPC) in prediction and execution is constrained by the limitations of visual models trained with limited datasets, resulting in insufficient generalization ability.

[0009] The problems with existing large models mean that they still rely on predefined actions or skills in robot planning and execution, and lack prediction of future robot-environment interactions. This increases the complexity of large models in robot applications and makes it difficult to improve their generalization ability. Summary of the Invention

[0010] To overcome the shortcomings of the prior art, this invention provides a robot planning method and system based on multimodal large model predictive control. It utilizes the powerful visual reasoning and visual localization capabilities of the multimodal large model to sample candidate action sequences from the target image or text command of the robot's movement and the robot's current observation image. Then, it predicts the future state video from the sampled candidate action sequences, avoiding the need to manually design basic actions and skills, and realizing robot control planning based on future state prediction.

[0011] To achieve the above objectives, a first aspect of the present invention provides a robot planning method based on multimodal large model predictive control, comprising:

[0012] Acquire the target image or text command for the robot's movement, as well as the image currently observed by the robot;

[0013] Using the configured multimodal large model, the sampling distribution of the robot's future actions is obtained based on the target image or text command and the robot's current observed image;

[0014] The sampling distribution of the obtained robot future action samples is sampled based on Gaussian distribution to obtain multiple candidate action sequences;

[0015] Based on the candidate action sequence and the execution actions of the consecutive frames corresponding to each candidate action sequence, the trained video prediction model is used to obtain the video prediction results corresponding to the robot's future movement.

[0016] The prediction results of all videos are input into the constructed cost function, and the candidate action sequence with the best evaluation result is selected to guide the robot's movement.

[0017] A second aspect of the present invention provides a robot planning system based on multimodal large model predictive control, comprising:

[0018] Acquisition module: Acquires target images or text commands for robot movement, as well as the currently observed images of the robot;

[0019] Extraction module: Using the configured multimodal large model, the sampling distribution of the robot's future actions is obtained based on the target image or text command and the robot's current observed image;

[0020] Sampling module: Based on Gaussian distribution sampling, the sampling distribution of the obtained robot future action samples is used to obtain multiple candidate action sequences;

[0021] Video prediction module: Based on the candidate action sequence and the corresponding consecutive frame execution actions, the module uses a trained video prediction model to obtain the video prediction results corresponding to the robot's future movements.

[0022] Control module: Input the results of all video predictions into the constructed cost function, and select the candidate action sequence with the best evaluation results to guide the robot's movement.

[0023] A third aspect of the present invention provides a computer device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, a robot planning method based on multimodal large model predictive control is performed.

[0024] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs a robot planning method based on multimodal large model predictive control.

[0025] The above one or more technical solutions have the following beneficial effects:

[0026] In this invention, the powerful visual reasoning and visual localization capabilities of a multimodal large model are utilized. Candidate action sequences are sampled using target images or text commands for robot movement, as well as images currently observed by the robot. Then, future state videos are predicted from the sampled candidate action sequences, and the prediction results are evaluated using a cost function. This allows for the selection of the optimal action sequence for the future state to guide the robot's operation, rotation, and complex path planning for interaction with objects in the scene. This avoids the need to manually design basic actions and skills and overcomes the limitation of previous methods based on multimodal large models, which could only form rough trajectories without foresight. This enables robot control planning based on future state prediction.

[0027] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0028] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0029] Figure 1 This is the overall flowchart of the robot planning method based on multimodal large model predictive control in Embodiment 1 of the present invention;

[0030] Figure 2 This is a flowchart illustrating the action sampling method in Embodiment 1 of the present invention;

[0031] Figure 3 This is a flowchart illustrating the video prediction method based on action sequences in Embodiment 1 of the present invention;

[0032] Figure 4 This is the prompt instruction in Embodiment 1 of the present invention. Example image;

[0033] Figure 5 These are the experimental results in a simulation environment in Embodiment 1 of the present invention;

[0034] Figure 6 These are the experimental results in a real-world scenario in Embodiment 1 of the present invention. Detailed Implementation

[0035] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0036] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0037] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0038] Before introducing the proposed solution, let's first explain the basic concepts involved in this paper:

[0039] Large models, also known as large-scale pre-trained models, refer to models pre-trained on large-scale datasets. Depending on the dataset, large models can be categorized into different modalities, such as language large models, image large models, speech large models, and multimodal large models. Multimodal large models, in particular, can process data from multiple modalities simultaneously. Their training datasets typically include large amounts of text, images, videos, and other multimodal data from the internet. Large models acquire general knowledge representations by learning patterns and features from the data. The goal of these large models is to learn a universal representation capability for fine-tuning or transfer learning in various downstream tasks.

[0040] Example 1

[0041] This embodiment discloses a robot planning method based on multimodal large model predictive control, including:

[0042] Acquire the target image or text command for the robot's movement, as well as the image currently observed by the robot;

[0043] Using the configured multimodal large model, the sampling distribution of the robot's future actions is obtained based on the target image or text command and the robot's current observed image;

[0044] The sampling distribution of the obtained robot future action samples is sampled based on Gaussian distribution to obtain multiple candidate action sequences;

[0045] Based on the candidate action sequence and the execution actions of the consecutive frames corresponding to each candidate action sequence, the trained video prediction model is used to obtain the video prediction results corresponding to the robot's future movement.

[0046] The prediction results of all videos are input into the constructed cost function, and the candidate action sequence with the best evaluation result is selected to guide the robot's movement.

[0047] This embodiment designs an action sampling method based on a multimodal large model (VLMPC). This application pre-configures a multimodal large model, which can process input data of different modalities, such as text data and image data. By leveraging the powerful knowledge representation capabilities and robust visual reasoning and visual localization capabilities provided by the multimodal large model, action sequences are sampled, avoiding the need for manually designing basic actions and skills.

[0048] like Figure 1 As shown, in the VLMPC framework, at each time t, N candidate action sequences are sampled from a custom sampling distribution. in It contains T actions.

[0049] like Figure 2 As shown, given a target image G or a language instruction L as input to VLMPC, along with the current observation O... t The VLM (Virtual Model) can generate realistic future robot movement states and output sampling distributions for motion sampling. By inputting target images or text commands into the VLM, the model outputs optimal motion sampling distributions. The motion sequences sampled using these distributions determine whether and how to interact with scene objects.

[0050] In this embodiment, the multimodal large model can use GPT-4V. A prompt instruction suitable for VLM is designed and used... This indicates that VLM analysis of the currently observed image O t The input is either the target image G or the verbal instruction L. Generally, a large model's prompt includes a role, task description, contextual information, examples, and user instructions. The role defines a character for the large model that matches the target task; the task description provides a detailed description of the task; the contextual information provides relevant historical information; and the examples standardize the model's output and demonstrate how to correctly complete the task. However, the prompt may vary depending on the specific task. The content is as follows: First, the GPT-4V is designated as a motion generator in a robotic arm operating environment. Its task is to determine the appropriate motion direction for the robot based on observations and targets, and to provide 3 to 4 relevant examples for the GPT-4V as a reference. Finally, user commands are input, and the GPT-4V can combine the user commands with the example format for standardized output. (Prompt commands are also included.) This forces the VLM to identify and locate the robot and the objects it will interact with, infer the appropriate future interaction parties, and generate appropriate future action distributions.

[0051] The output of VLM can be expressed as:

[0052]

[0053] in, This indicates the direction of movement of the robotic arm along the x-axis at time t. This indicates the direction of movement of the robotic arm along the y-axis at time t. This indicates the direction of movement of the robotic arm along the z-axis at time t. This indicates the direction of rotation of the robotic arm along the x-axis at time t. This indicates the direction of rotation of the robotic arm along the y-axis at time t. Indicates the direction of rotation of the robotic arm along the z-axis at time t, g t ={0,1} represents the on / off state of the gripper at time t.

[0054] As a further implementation, Gaussian sampling is used to obtain candidate action sequences, mapping the output of the VLM to a sample mean at time t.

[0055]

[0056] Where, ω m and ω r It is a hyperparameter that maps the output of the VLM to the robot's motion space.

[0057] VLMPC uses the sampled mean obtained from historical information in the previous step's sequence of subsequent candidate actions. Will and The element-wise weighted sum is used to produce the final sample mean μ at time t. t :

[0058]

[0059] Where, ω VLM and ω sub These are two types of sampling mean weights.

[0060] This embodiment uses a Gaussian distribution. N samples were obtained by repeatedly sampling N times. t , where μ t Let I represent the expectation of the Gaussian distribution at time t, and let I represent the variance. These candidate action sequences are then input into a video prediction module based on robot action conditions.

[0061] This embodiment designs a video prediction module based on robot action sequences, such as... Figure 3As shown, given the current robot observations and N sampled candidate action sequences, the video prediction module outputs N videos representing future states.

[0062] A variant of DMVFN was constructed. DMVFN is an efficient dynamic multi-scale voxel flow network for video prediction. However, since DMVFN is a network that predicts video based on video, this embodiment first repeatedly stacks candidate actions into tensors of the image's width and height. Then, the stacked actions and images are concatenated along the channel dimension to calculate the total number of channels and input them into the network. The number of input channels of the fully connected layers of the DMVFN network is modified accordingly to perform video prediction based on action sequences, and this variant is named DMVFN-Act. Given two past historical frames O t-1 and O t DMVFN predicts a future frame Formulated as:

[0063]

[0064] As a further implementation, we have a sampled candidate action sequence S. t and the corresponding execution action a t-1 and a t DMVFN-Act can predict future images frame by frame based on actions. First, it selects candidate actions a... t-1 a t , Expanded to the same dimension as the image Then the expanded motion and image observation O t-1 O t spliced ​​together to get The formula is as follows:

[0065]

[0066]

[0067] As a further implementation, the stitched observations are input into DMVFN-Act to obtain the predicted robot-executed observations, as shown in the following formula:

[0068]

[0069] As a further implementation, the video predicted by a single action sequence can be represented as:

[0070]

[0071] As a further implementation, the N candidate action sequences correspond to the N predicted video segments:

[0072]

[0073] To comprehensively evaluate the video prediction results, this embodiment designs a cost function based on image pixels and a multimodal large model, providing hierarchical evaluation at the pixel level and knowledge level, respectively. This embodiment also proposes a VLM switcher, which can dynamically select one or two sub-cost functions based on observations, choosing them in an appropriate manner.

[0074] This embodiment first designs a pixel distance cost function: when the task input is a target image G, an intuitive way to evaluate video prediction is to calculate the sum of pixel distances between each future frame and the target image. The function is calculated in action-conditional video... In each future frame The L2 distance between G and then these distances are used as Pixel distance cost The formula for calculating the sum is as follows:

[0075]

[0076] As a further implementation, at time t, for V... t The cost function can be expressed as:

[0077]

[0078] As a further implementation, to achieve more complex or multi-objective tasks, this embodiment adds an auxiliary cost function based on VLM. Specifically, based on the currently observed image O... t In conjunction with the task input G or L, a prompt instruction was designed to... Indicates, such as Figure 4 , The content is as follows: First, the GPT-4V is designated as a target detector in a robotic arm operation environment. Its task is to determine the next sub-target and interfering object of the robot operation based on observation and instructions. Three to four relevant examples are provided to the GPT-4V for reference. Finally, user instructions are input, and the GPT-4V can combine the user instructions to output in the format of the examples. The driven VLM can infer and locate the next sub-target and all interfering objects, where the sub-target is typically the next object to interact with the robot. This process, in the current observation, produces the robot's end effector e at time t. t The next sub-target s t And all interfering objects I t The bounding box is defined by the following formula:

[0079] VLM(O t,G∨V|φ C )={e t ,s t ,I t}

[0080] As a further implementation, due to the predicted video V t Shared historical frames O t A lightweight vision tracker, V, can be used. T In all of e t s t and I t In each future frame of the initialized prediction video, locate the end effector. and sub-targets The formal expression is as follows:

[0081]

[0082] As a further implementation, the formula for calculating the VLM auxiliary cost function is as follows, where c(.) represents the center of the bounding box:

[0083]

[0084]

[0085] As a further implementation, in order to combine these two cost functions, a further design based on prompt instructions was created. The VLM switcher was designed with a prompt instruction. It indicates, and will The input is fed into GPT-4V, thus enabling GPT-4V to select an appropriate cost function. The content is as follows: First, the GPT-4V is designated as a cost function switcher in a robotic arm operating environment. Its task is to determine the appropriate cost function for the robot based on observations and the target, and to provide 3 to 4 relevant examples for the GPT-4V as a reference. Finally, user commands are input, and the GPT-4V can combine these commands with the example-formatted output. Based on... The GPT-4V dynamically selects one or two appropriate cost functions using w at each time t through knowledge reasoning. D This indicates that, to adapt to the current observation, 0 means selecting the pixel distance cost function, 0.5 means selecting a combination of the pixel distance cost function and the auxiliary cost function, and 1 means selecting the auxiliary cost function, thus generating the final cost C(t):

[0086] VLM(O t ,G∨V|φ D ) = w D ∈{0,0.5,1}

[0087] C(t) = w D *C P (t)+(1-w D )*C VLM (t)

[0088] As a further implementation, the cost C(t) = {C^n(t) | n ∈ {1, 2, ..., N}} is used as the evaluation for all predicted videos, and the candidate action sequence with the lowest cost is selected for subsequent processing. When the first action in this sequence is executed, subsequent actions are fed into a global average pooling layer to generate the sampled mean. This provides historical information for the next step of action sampling.

[0089] This embodiment first compares the method with the method in the public benchmark VP2 on the RoboDesk simulation environment, such as... Figure 5 As shown, VLMPC significantly outperforms VP2. For tasks involving pushing (red, green, and blue) buttons, both VP2 and VLMPC achieve high performance. However, in more challenging tasks (opening drawers, opening blackboards, and knocking over blocks), VLMPC demonstrates far superior performance compared to VP2. This demonstrates VLMPC's greater ability to complete complex tasks.

[0090] To evaluate the capabilities of the VLMPC in the real world, the robotic arm attempted four tasks: grasping a towel, placing an object in a bowl, drying a table, and turning on a lamp. The task difficulty gradually increased, and each task was repeated 30 times. The success rate was calculated. The success rate of the experiment is as follows: Figure 6 As shown, the task definition methods are divided into two types: target image and text instruction. It can be seen that VLMPC can achieve a high success rate in relatively simple tasks such as turning on the desk lamp and grabbing the towel, and can also guarantee a certain success rate in tasks such as placing the banana into the bowl and wiping the water.

[0091] Example 2

[0092] The purpose of this embodiment is to provide a robot planning system based on multimodal large model predictive control, including:

[0093] Acquisition module: Acquires target images or text commands for robot movement, as well as the currently observed images of the robot;

[0094] Extraction module: Using the configured multimodal large model, the sampling distribution of the robot's future actions is obtained based on the target image or text command and the robot's current observed image;

[0095] Sampling module: Based on Gaussian distribution sampling, the sampling distribution of the obtained robot future action samples is used to obtain multiple candidate action sequences;

[0096] Video prediction module: Based on the candidate action sequence and the corresponding consecutive frame execution actions, the module uses a trained video prediction model to obtain the video prediction results corresponding to the robot's future movements.

[0097] Control module: Uses the obtained video prediction results to control the robot's movement.

[0098] Example 3

[0099] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0100] Example 4

[0101] The purpose of this embodiment is to provide a computer-readable storage medium.

[0102] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0103] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0104] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0105] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A robot planning method based on multimodal large model predictive control, characterized in that, include: Acquire the target image or text command for the robot's movement, as well as the image currently observed by the robot; Using the configured multimodal large model, the sampling distribution of the robot's future actions is obtained based on the target image or text command and the robot's current observed image; The sampling distribution of the obtained robot future actions is sampled based on a Gaussian distribution to obtain multiple candidate action sequences; specifically: Map the sampling distribution of the obtained future action samples to the sampling mean; The final sample mean is obtained by summing the historical sample mean obtained from the subsequent candidate action sequence corresponding to the previous step and the sample mean corresponding to the future action element by element. Based on the final sample mean, and using Gaussian distribution sampling, multiple candidate action sequences are obtained; Based on the candidate action sequence and the execution actions of the consecutive frames corresponding to each candidate action sequence, the trained video prediction model is used to obtain the video prediction results corresponding to the robot's future movement. The results of all video predictions are input into the constructed cost function, and the candidate action sequence with the best evaluation result is selected to guide the robot's movement. Specifically, a pixel distance cost function is constructed based on the sum of the predicted pixel distances between each future frame and the target image; For each predicted future frame, the robot's end effector, the next sub-target, and all interfering objects are located based on the vision tracker. An auxiliary cost function for the multimodal large model is constructed based on the localization results. The final cost is obtained by dynamically selecting the pixel distance cost function and / or auxiliary cost function to adapt to the current observation; The final cost obtained is used as the evaluation result for the predicted video.

2. The robot planning method based on multimodal large model predictive control as described in claim 1, characterized in that, Using the configured multimodal large model, based on the target image or text command and the robot's current observed image, the sampling distribution of future action samples is obtained, specifically: Using a multimodal large model as a motion generator in the robotic arm's operating environment, prompts are generated based on the robot's motion direction corresponding to the observed and target images. ; According to the prompt instructions The robot uses the current observed image, target image or text command, and a multimodal large model to obtain the future action distribution.

3. The robot planning method based on multimodal large model predictive control as described in claim 1, characterized in that, The video prediction model employs a dynamic multi-scale voxel flow network. Based on the candidate action sequence and the corresponding consecutive frames of execution, the trained video prediction model obtains the video prediction results corresponding to the robot's future motion. Specifically: Expand the dimensions of the actions executed in the previous and current frames corresponding to the candidate action sequences. The expanded dimension execution action is concatenated with the previous image observation value and the current image observation frame, respectively; The stitched observations are input into the trained video prediction model to obtain the robot's observations.

4. The robot planning method based on multimodal large model predictive control as described in claim 1, characterized in that, The candidate action sequence with the lowest cost function is selected. When the first action in the selected candidate action sequence is executed, the subsequent actions are input into a global average pooling layer to obtain the historical sample mean.

5. The robot planning method based on multimodal large model predictive control as described in claim 1, characterized in that, Using a multimodal large model as a target detector in the robotic arm's operating environment, prompts are generated based on observed images, text commands, and the corresponding next sub-target and interfering objects to be operated by the robot. Based on the generated prompts Using the current observed image, as well as the target image or text command, and employing a multimodal large model, the next sub-target and interfering object to be operated by the robot are obtained.

6. A robot planning system based on multimodal large model predictive control, characterized in that, include: Acquisition module: Acquires target images or text commands for robot movement, as well as the currently observed images of the robot; Extraction module: Using the configured multimodal large model, the sampling distribution of the robot's future actions is obtained based on the target image or text command and the robot's current observed image; Sampling module: Based on Gaussian distribution sampling, the sampling distribution of the obtained robot future actions is used to obtain multiple candidate action sequences; specifically: Map the sampling distribution of the obtained future action samples to the sampling mean; The final sample mean is obtained by summing the historical sample mean obtained from the subsequent candidate action sequence corresponding to the previous step and the sample mean corresponding to the future action element by element. Based on the final sample mean, and using Gaussian distribution sampling, multiple candidate action sequences are obtained; Video prediction module: Based on the candidate action sequence and the corresponding consecutive frame execution actions, the module uses a trained video prediction model to obtain the video prediction results corresponding to the robot's future movements. Control module: Inputs the prediction results of all videos into the constructed cost function, and selects the candidate action sequence with the best evaluation result to guide the robot's movement; specifically: constructs a pixel distance cost function based on the sum of pixel distances between each predicted future frame and the target image; For each predicted future frame, the robot's end effector, the next sub-target, and all interfering objects are located based on the vision tracker. An auxiliary cost function for the multimodal large model is constructed based on the localization results. The final cost is obtained by dynamically selecting the pixel distance cost function and / or auxiliary cost function to adapt to the current observation; The final cost obtained is used as the evaluation result for the predicted video.

7. A computer device, characterized in that, include: The system includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform a robot planning method based on multimodal large model predictive control as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, performs a robot planning method based on multimodal large model predictive control as described in any one of claims 1 to 5.

Images