A robot behavior prediction and planning method fusing a large language model and a generative world model

Abstract

This invention relates to a robot behavior prediction and planning method integrating a large language model and a generative world model. The method includes: mapping observed images, robot joint and pose states, and task commands into vectors, aligning and fusing them to form a comprehensive scene representation; inputting the world model to generate an abstract representation of the current environmental context and predicting future potential state representations; concatenating task commands and the comprehensive scene representation into prompt words and inputting them into the large language model to decompose them into a sequence of sub-tasks; fusing the potential state representations and symbolic sub-objective constraints of the sub-task sequences to obtain dynamic constraints, constructing and solving the objective function, obtaining the optimal action sequence, parsing the first step of the optimal action sequence, and obtaining the robot joint trajectory or end effector pose command. Compared with existing technologies, this invention enhances the robot's long-term behavior prediction and robust planning capabilities in dynamic scenes, and is applicable to autonomous task execution in various complex service scenarios such as homes.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making for service robots, and in particular to a method for predicting and planning robot behavior that integrates a large language model and a generative world model. Background Technology

[0002] Currently, the autonomous decision-making and planning of service robots mainly rely on rule-based strategies, classical path planning algorithms, or skill models based on supervised learning. These methods perform well in structured, deterministic environments, but their limitations become apparent when facing open, dynamic, and unstructured complex real-world scenarios, such as homes, hospitals, or shopping malls. Traditional methods are poorly adaptable to environmental changes, struggle to understand high-level semantic instructions (such as "put the water glass on the table in a safe place"), and are unable to effectively predict and reason about the long-term physical consequences of their own actions, resulting in high task failure rates, rigid behavior, and safety issues.

[0003] In recent years, large language models, with their powerful world knowledge and symbolic reasoning capabilities, have provided new approaches for understanding and decomposing high-level tasks in robotics. Meanwhile, generative world models, through self-supervised learning of physical laws, can predict the future state outcomes of actions. However, existing technologies still face the following key challenges: Decision-making is disconnected from physical simulation: The planning of large language models is usually symbolic and discrete, lacking consideration of continuous physical dynamics and the feasibility of specific robot movements.

[0004] Insufficient real-time prediction and adjustment capabilities: Most systems lack a model that can quickly and accurately predict the physical consequences of actions, making it impossible to proactively assess the risks of a plan during the planning phase, or to perform rapid state corrections and strategy adjustments based on real-time observations during the execution phase.

[0005] The multimodal information fusion is not deep enough: the system fails to deeply integrate and jointly reason with semantic instructions, real-time visual observations, and robot body state within a unified framework, which limits the system's comprehensive understanding of complex scenarios.

[0006] Lack of zero-shot adaptation mechanism: When faced with new objects, new scenes or new tasks that have not appeared in the training data, the existing system has poor generalization ability and cannot carry out effective zero-shot or few-shot learning and adaptation.

[0007] Therefore, there is a need for a service robot behavior prediction and planning method that can improve the robot's long-term task planning capabilities, safety, and zero-sample adaptation capabilities in dynamic and unknown environments. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a robot behavior prediction and planning method that integrates a large language model and a generative world model. By combining the abstract reasoning of the large language model with the physical prediction of the generative world model, a closed loop from semantic instructions to physically executable actions is achieved, thereby improving the robot's long-term task planning ability, safety and zero-shot adaptation ability in dynamic and unknown environments.

[0009] The objective of this invention can be achieved through the following technical solutions: A method for predicting and planning robot behavior that integrates a large language model and a generative world model, the method comprising: By using parallel visual encoders, ontology state encoders, and language encoders, environmental visual observation images, robot joint and pose states, and natural language task commands are mapped into visual embedding vectors, state embedding vectors, and language command embedding vectors in a unified dimension. The cross-modal attention module is used to align and fuse the features of each vector to form a comprehensive scene representation. The comprehensive scene representation is input into the video encoder of the preset world model to generate an abstract representation of the current environmental context. Based on the previous video observations and robot state sequence, the potential state representation of the future is predicted in the abstract representation space. The current natural language task instruction and the natural language description of the comprehensive scene representation are concatenated into prompt words and input into a preset large language model. The prompt words are then decomposed into a sub-task sequence through chain thinking. Within the rolling time domain of the pre-built model predictive controller, dynamic constraints are obtained by fusing the latent state representation with the symbolic sub-objective constraints of the sub-task sequence. Based on the dynamic constraints and task objectives, an objective function is constructed and solved to obtain the optimal action sequence. The first step of the optimal action sequence is parsed to obtain the robot joint trajectory or end effector pose command.

[0010] Furthermore, the visual encoder adopts the Vision Transformer architecture, the ontology state encoder adopts a multilayer perceptron, and the language encoder adopts a pre-trained Transformer model; the cross-modal attention module adopts a cross-attention mechanism, using language embedding as the query and visual and state embedding as the key-value pair, to realize the task instruction's focus on and guidance of perceptual features.

[0011] Furthermore, the process of mapping environmental visual observation images, robot joint and pose states, and natural language task instructions into visual embedding vectors, state embedding vectors, and language instruction embedding vectors in a unified dimension includes: Visual encoder: Divides the environmental visual observation image into fixed-size patches, adds position encoding, and processes it through multiple layers of Transformer to extract the global visual embedding vector; Body state encoder: maps the body state of the robot joint into a state embedding vector, wherein the body state includes angle and pose; Language encoder: The natural language task instruction is segmented and the last hidden state is extracted. The language instruction embedding vector is obtained by average pooling.

[0012] Furthermore, the preset world model includes a video encoder and a predictor, and adopts a joint embedding prediction architecture. Through self-supervised learning on large-scale Internet video data, it performs inference in the abstract representation space to predict the state of occluded objects and the physical laws of the motion trajectory of objects affected by gravity.

[0013] Furthermore, the process of predicting the future latent state representation within the abstract representation space includes: The sequence of the comprehensive scene representations from historical moments is input into the video encoder of the preset world model to generate an abstract representation of the environmental context at the current moment. The candidate action sequence based on the preset large language model, together with the context abstract representation, is input into the predictor of the preset world model to predict the potential state representation of the future in the abstract semantic space.

[0014] Furthermore, during the process of decomposing prompt words into sub-task sequences through chain thinking, the pre-established large language model also assists in prompt word decomposition by retrieving successful strategies in similar situations from a pre-established long-term operating memory using vector retrieval. The pre-established long-term operating memory is a temporal graph neural network that stores and associates key decision points, multimodal scene representations, and action results in historical tasks.

[0015] Furthermore, the preset large language model includes a predefined structured function call interface, which calls the abstract state deduction function of the preset world model in the form of text instructions, and parses the returned textual deduction results to make logical decisions.

[0016] Furthermore, after decomposing the prompt words into a sequence of subtasks using chain thinking, a pre-defined world model is structurally invoked to perform forward state deduction on the candidate action sequences. Based on the deduction results, a feasibility assessment and strategy optimization are performed to obtain an initial action sequence. This initial action sequence serves as the initial iterative solution when iteratively solving the objective function and participates in the solution of the optimal action sequence.

[0017] Furthermore, the task objective is to minimize the sum of state tracking error and control cost, and the process of obtaining the dynamic constraints includes: Based on the results of the feasibility assessment, the basic physical constraints, basic robot motion constraints, and environmental conflict constraints of the current task are obtained. Based on the aforementioned fundamental physical constraints, the latent state representation is decoded into specific physical state constraints through a lightweight mapping network; By fusing the robot's basic motion constraints with the robot's current motion constraints, robot constraints are obtained. The symbolic sub-target numerical constraints output by the preset large language model when decomposing prompt words into sub-task sequences through chain thinking are obtained, and dynamic constraints are obtained by combining the environmental conflict constraints, specific physical state constraints and robot constraints.

[0018] Furthermore, after acquiring the robot joint trajectory or end effector pose command, the robot is controlled based on the joint trajectory or end effector pose command, and real-time multimodal observations are acquired simultaneously. State filtering and prediction calibration are performed using a world model. When the representation uncertainty, prediction deviation, or overall system confidence exceeds a preset threshold, a large language model is triggered to perform semantic anomaly diagnosis on the current situation and generate corrective commands based on physical understanding, achieving dynamic reprogramming and zero-shot task adaptation. Specifically, The state filtering process includes: predicting the current potential state based on real-time multimodal observations using a world model, and calculating the representation uncertainty of the current potential state using a preset uncertainty quantification module; The prediction calibration process includes: predicting the current potential state based on real-time multimodal observations using a world model, calculating the cosine distance between the predicted potential state and the encoded real observed potential state, and obtaining the prediction bias; The confidence level of the integrated system is obtained by comprehensively representing uncertainty and prediction bias through preset weights.

[0019] Compared with the prior art, the beneficial effects of the present invention include: 1. This invention achieves a deep integration of symbolic reasoning and physical simulation: by structurally combining the logical planning capability of a large language model with the physical prediction capability of a world model, high-level task planning is based on verifiable physical simulation, which greatly improves the feasibility and security of the planning scheme and solves the problem of planning in the air.

[0020] 2. This invention possesses a long-term perspective and forward-looking risk avoidance capability: by using a world model to perform efficient and rapid long-term state extrapolation in an abstract space, the system can anticipate the long-term physical consequences of different action sequences during the planning stage and proactively avoid high-risk solutions that may lead to collisions, overturning, or unstable states.

[0021] 3. This invention enhances real-time adaptation to dynamic environments and zero-shot generalization capabilities: Through uncertainty quantification in closed-loop execution and semantic anomaly diagnosis mechanisms of the large language model, the system can proactively identify prediction failures or environmental mutations, and invoke the common sense of the large language model for explanation and replanning. Combined with the general physical priors learned by the world model from massive video data, the system exhibits powerful zero-shot task adaptation capabilities.

[0022] 4. This invention optimizes the efficiency of collaborative utilization of multimodal information: the cross-modal attention mechanism enables task instructions to focus on perceptual information in real time, while the long-term operating memory stores cross-task multimodal experience in a graph structure, enabling the system to perform experience-based analogical reasoning, which significantly improves the speed and accuracy of decision-making in complex and repetitive scenarios.

[0023] 5. This invention constructs an end-to-end intelligent decision-making closed loop from semantics to actions: This invention fully covers the entire process from instruction understanding, scene perception, physical deduction, optimization solution to execution monitoring and anomaly recovery, forming an autonomous, robust and interpretable intelligent decision-making system for service robots, providing key technical support for deploying general-purpose service robots in the real open world. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart illustrating the world model workflow in this invention. Figure 3 This is a flowchart illustrating the collaborative generation of the large language model and the world model in this invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] Example 1 This embodiment discloses a method for robot behavior prediction and planning that integrates a large language model and a generative world model. The method is as follows: Figure 1As shown, steps S1-S5 are included, and each step is described in detail below: Step S1: Environmental visual observation is performed through parallel visual encoders, ontology state encoders, and language encoders. Robot joints and pose states and natural language task instructions Mapped to a unified dimension Visual embedding vectors State embedding vector With language instruction embedding vectors Furthermore, the cross-modal attention module aligns and fuses the features of each vector to form a comprehensive scene representation.

[0027] The process aims to extract and encode the perceptual features of the environment and the semantic features of the task.

[0028] The visual encoder adopts the Vision Transformer architecture, the ontology state encoder adopts a multilayer perceptron, and the language encoder adopts a pre-trained Transformer model.

[0029] The cross-modal attention module employs a cross-attention mechanism, using language embedding as the query and visual and state embedding as the key-value pair, to enable task instructions to focus on and guide perceptual features.

[0030] The process of mapping environmental visual observation images, robot joint and pose states, and natural language task commands into visual embedding vectors, state embedding vectors, and language command embedding vectors in a unified dimension includes: Visual encoder: Divides the environmental visual observation image into fixed-size patches, adds position encoding, and then processes it through multiple layers of Transformer to extract the global visual embedding vector; Body state encoder: maps the body state of the robot joint into a state embedding vector, wherein the body state includes angle and pose; Language encoder: The natural language task instruction is segmented and the last hidden state is extracted. The language instruction embedding vector is obtained by average pooling.

[0031] Specifically, the visual encoder will input the image Divided into A fixed-size patch is linearly projected into a sequence. Then position encoding is added. : The sequence was processed The Transformer encoder process consists of multiple layers, each containing a multi-head self-attention network and a feedforward network. The layer's output is calculated as follows: Finally, the output corresponding to the [CLS] marker is taken as the global visual embedding vector. The ontology state feature extraction employs a multilayer perceptron. The state vector is... Mapping to embedding: in, For learnable parameters, This is the activation function.

[0032] Language feature extraction utilizes the text encoder portion of a pre-trained large language model. (Instructions...) After word segmentation, the average pooling result of the last hidden state is used as the language instruction embedding. Finally, a cross-modal attention module is used to achieve early feature alignment and fusion, generating a comprehensive scene representation. This module uses language embedding. For queries, visual and state embedding are combined. For Key and Value: In this embodiment, the world model used is an open-source world AI model, which includes a video encoder. and predictor , Figure 2 This is a flowchart illustrating the workflow of the world model in this invention. This world model can learn physical laws by watching videos; its code and model are existing technologies and will not be elaborated upon here. The large language model used is an open-source large language model; its code and model are existing technologies and will not be elaborated upon here.

[0033] In the above S1, it should be noted that the visual encoder adopts the standard Vision Transformer architecture, the ontology state encoder is a three-layer fully connected network, the language encoder reuses the text encoder part of the large language model, and the cross-modal attention module adopts a single-head cross-attention mechanism to achieve early fusion.

[0034] Step S2: Input the comprehensive scene representation into the video encoder of the preset world model to generate an abstract representation of the current environmental context. Based on the previous video observations and robot state sequence, predict the potential future state representation in the abstract representation space.

[0035] The pre-defined world model includes a video encoder and a predictor. It adopts a joint embedding prediction architecture and performs inference in the abstract representation space by conducting self-supervised learning on large-scale Internet video data. It can predict the physical laws such as the state of occluded objects and the trajectory of objects affected by gravity.

[0036] The process of predicting the future representation of the latent state within the abstract representation space includes: The sequence of the comprehensive scene representations from historical moments is input into the video encoder of the preset world model to generate an abstract representation of the environmental context at the current moment. The candidate action sequence based on the preset large language model, together with the context abstract representation, is input into the predictor of the preset world model to predict the potential state representation of the future in the abstract semantic space.

[0037] Specifically, given a comprehensive scene representation of a historical observation sequence. The video encoder maps it to a context abstraction representation. : The planning module generates a future Candidate action sequence of the step The predictor is and Given conditions, predict the future. Potential state representation of the step : in, It is in the same abstract semantic space, and is encoded with real observations. Comparable predictive representations. For example, but not limited to, publicly available V-JEPA 2 model architectures can be used, the core of which lies in employing a joint embedding predictive architecture. V-JEPA 2 achieves this through self-supervised learning of mask modeling on large-scale video data. It can encode general physical laws such as object motion, occlusion, and the effects of gravity.

[0038] In S2 above, it should be noted that the world model, as the physical reasoning engine of this system, makes predictions in an abstract, high-semantic-level representation space, avoiding the computational burden of pixel-level reconstruction and realizing rapid simulation of the long-term consequences of actions.

[0039] In S2 above, the specific calling process of the world model is encapsulated into a standardized API that can be called by the language model, which receives the action sequence described in the text and returns the predicted state change described in the text.

[0040] Step S3: Concatenate the current natural language task instruction with the natural language description of the comprehensive scene representation into prompt words and input them into the preset large language model. Then, decompose the prompt words into a sub-task sequence through chain thinking.

[0041] The preset large language model includes a predefined structured function call interface. The structured function call interface calls the abstract state deduction function of the preset world model in the form of text commands, and parses the returned textual deduction results to make logical decisions.

[0042] In the process of decomposing prompt words into sub-task sequences through chain thinking, the pre-established large language model also assists in the decomposition of prompt words by retrieving successful strategies in similar situations from a pre-established long-term operating memory using vector retrieval. The pre-established long-term operating memory is a temporal graph neural network that stores and associates key decision points, multimodal scene representations, and action results in historical tasks.

[0043] After decomposing the prompt words into a sequence of subtasks using chain thinking, a pre-defined world model is used to perform forward state deduction on the candidate action sequences. Based on the deduction results, a feasibility assessment and strategy optimization are performed to obtain an initial action sequence. This initial action sequence serves as the initial iterative solution when iteratively solving the objective function and participates in the solution of the optimal action sequence.

[0044] Specifically, step S3 includes: First, set the current task instructions Integrated scene representation described by language Joint composition hints Inputting into a large language model, chain-like reasoning is used to decompose sub-target sequences. : At the same time, the model generates a corresponding logical reasoning chain as an intermediate output.

[0045] For the sub-goals that need to be achieved at present The large language model calls the world model for forward inference through a predefined structured interface. Specifically, the large language model generates several candidate action sequence hypotheses. And request the world model to evaluate the predicted state after executing these sequences. Large language models receive semantic descriptions of these predictions and, based on task objectives... Logically evaluate based on common sense and physics to select the optimal sequence of actions. .

[0046] In S3 mentioned above, it should be noted that the large language model, through prompting engineering combined with its inherent function calling capabilities, realizes the standard calling of the world model, which does not require additional model fine-tuning of the world model.

[0047] Step S4, in the rolling time domain of the pre-built model predictive controller Within this framework, the latent state representation is fused with the symbolic sub-objective constraints of the sub-task sequence to obtain dynamic constraints. Based on these dynamic constraints and the task objective, an objective function is constructed and solved to obtain the optimal action sequence. The first step of the optimal action sequence is then analyzed to obtain the robot joint trajectory or end effector pose command. For example... Figure 3 This is a flowchart illustrating the collaborative generation of the large language model and the world model in this invention.

[0048] The task objective is to minimize the sum of state tracking error and control cost. The process of obtaining the dynamic constraints includes: Based on the results of the feasibility assessment, the basic physical constraints, basic robot motion constraints, and environmental conflict constraints of the current task are obtained. Based on the aforementioned fundamental physical constraints, the latent state representation is decoded into specific physical state constraints through a lightweight mapping network; By fusing the robot's basic motion constraints with the robot's current motion constraints, robot constraints are obtained. The symbolic sub-target numerical constraints output by the preset large language model when decomposing prompt words into sub-task sequences through chain thinking are obtained, and dynamic constraints are obtained by combining the environmental conflict constraints, specific physical state constraints and robot constraints.

[0049] Specifically, the optimization problem that the model predictive controller solves in the rolling time domain T, i.e., the objective function, is: in, The predicted potential state is obtained from formula (8); It is a lightweight mapping network that maps latent states Decoded into specific physical quantities related to the sub-target; Provided by the large language model, corresponding to time... The specific numerical constraints of the symbolic sub-targets; and These are the weighted norms of the state tracking error and the control cost, respectively. This is the action space under the constraints of robot kinematics and dynamics. The optimal action sequence is obtained by solving equation (11). and its first element The values ​​are parsed as settings for the underlying joint controller or pose controller.

[0050] In S4 above, it should be noted that the model predictive controller maps the abstract predictive output of V-JEPA 2 back to specific physical quantities through a trainable small decoder network, thereby combining it with specific robot control objectives.

[0051] Step S5, the robot performs the action. Afterwards, new observations were obtained. Repeat step S1 to obtain a new representation of the real scene. and its encoding .

[0052] After acquiring the robot joint trajectory or end effector pose command, the robot is controlled based on the robot joint trajectory or end effector pose command and real-time multimodal observations are acquired simultaneously. State filtering and prediction calibration are performed through the world model. When the characterization uncertainty, prediction deviation or comprehensive system confidence exceeds the preset threshold, the large language model is triggered to perform semantic anomaly diagnosis on the current situation and generate correction commands based on physical understanding, so as to realize dynamic replanning and zero-shot task adaptation.

[0053] The state filtering process includes: predicting the current potential state based on real-time multimodal observations using a world model, and calculating the representation uncertainty of the current potential state using a preset uncertainty quantification module; The prediction calibration process includes: predicting the current potential state based on real-time multimodal observations using a world model, calculating the cosine distance between the predicted potential state and the encoded real observed potential state, and obtaining the prediction bias; The confidence level of the integrated system is obtained by comprehensively representing uncertainty and prediction bias through preset weights.

[0054] Specifically, calculate the confidence level of the V-JEPA 2 prediction. This involves taking the prediction from the previous time step... With current real encoding Comparison: The first term is the cosine distance, and the second term is the variance of the predicted sequence. These are the weighting coefficients. Exceeding the threshold It is then classified as a high-uncertainty event.

[0055] When a high uncertainty alert is triggered or a significant deviation occurs in task execution, the system will provide a natural language description of the current abnormal situation. Inputting historical memory into the large language model, requesting diagnosis and correction, and based on the diagnosis results and correction plan, the system updates the current sub-goal or directly jumps to step S3 for replanning, achieving zero-shot adaptation.

[0056] The system maintains a long-term operating memory. After each task is completed, a memory tuple is stored. ,in It represents the sequence of scenarios and actions at critical decision moments, with the outcome being a label indicating whether the task succeeded or failed. During S3 planning, the system embeds the current language... and initial scene representation Using the query key, a near nearest neighbor search is performed from... The K most similar historical memories were retrieved from the database. : The semantic summaries of these historical memories are provided as context to open-source large language models such as Qwen 3 (as one implementation), thereby assisting them in making more accurate and efficient plans. In S5 above, it should be noted that the uncertainty quantification module integrates the difference between prediction and observation, as well as the uncertainty of the model itself, providing the system with reliable anomaly detection metrics.

[0057] Example 2 Based on Embodiment 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, the one or more programs including instructions for executing the robot behavior prediction and planning method as described above, which integrates a large language model and a generative world model.

[0058] At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the aforementioned method for robot behavior prediction and planning that integrates a large language model and a generative world model. Of course, in addition to software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0059] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0060] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0061] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for predicting and planning robot behavior that integrates a large language model and a generative world model, characterized in that, The method includes: By using parallel visual encoders, ontology state encoders, and language encoders, environmental visual observation images, robot joint and pose states, and natural language task commands are mapped into visual embedding vectors, state embedding vectors, and language command embedding vectors in a unified dimension. The cross-modal attention module is used to align and fuse the features of each vector to form a comprehensive scene representation. The comprehensive scene representation is input into the video encoder of the preset world model to generate an abstract representation of the current environmental context. Based on the previous video observations and robot state sequence, the potential state representation of the future is predicted in the abstract representation space. The current natural language task instruction and the natural language description of the comprehensive scene representation are concatenated into prompt words and input into a preset large language model. The prompt words are then decomposed into a sub-task sequence through chain thinking. Within the rolling time domain of the pre-built model predictive controller, dynamic constraints are obtained by fusing the latent state representation with the symbolic sub-objective constraints of the sub-task sequence. Based on the dynamic constraints and task objectives, an objective function is constructed and solved to obtain the optimal action sequence. The first step of the optimal action sequence is parsed to obtain the robot joint trajectory or end effector pose command.

2. The robot behavior prediction and planning method integrating a large language model and a generative world model according to claim 1, characterized in that, The visual encoder adopts the Vision Transformer architecture, the ontology state encoder adopts a multilayer perceptron, and the language encoder adopts a pre-trained Transformer model. The cross-modal attention module adopts a cross-attention mechanism, using language embedding as the query and visual and state embedding as the key-value pair, to enable task instructions to focus on and guide perceptual features.

3. The robot behavior prediction and planning method integrating a large language model and a generative world model according to claim 1, characterized in that, The process of mapping environmental visual observation images, robot joint and pose states, and natural language task commands into visual embedding vectors, state embedding vectors, and language command embedding vectors in a unified dimension includes: Visual encoder: Divides the environmental visual observation image into fixed-size patches, adds position encoding, and processes it through multiple layers of Transformer to extract the global visual embedding vector; Body state encoder: maps the body state of the robot joint into a state embedding vector, wherein the body state includes angle and pose; Language encoder: The natural language task instruction is segmented and the last hidden state is extracted. The language instruction embedding vector is obtained by average pooling.

4. The robot behavior prediction and planning method integrating a large language model and a generative world model according to claim 1, characterized in that, The preset world model includes a video encoder and a predictor. It adopts a joint embedding prediction architecture and performs self-supervised learning on large-scale Internet video data. It performs inference in the abstract representation space to predict the state of occluded objects and the physical laws of the motion trajectory of objects affected by gravity.

5. The robot behavior prediction and planning method integrating a large language model and a generative world model according to claim 4, characterized in that, The process of predicting the future representation of the latent state within the abstract representation space includes: The sequence of the comprehensive scene representations from historical moments is input into the video encoder of the preset world model to generate an abstract representation of the environmental context at the current moment. The candidate action sequence based on the preset large language model, together with the context abstract representation, is input into the predictor of the preset world model to predict the potential state representation of the future in the abstract semantic space.

6. The robot behavior prediction and planning method integrating a large language model and a generative world model according to claim 1, characterized in that, In the process of decomposing prompt words into sub-task sequences through chain thinking, the pre-established large language model also assists in prompt word decomposition by retrieving successful strategies in similar situations from a pre-established long-term operating memory using vector retrieval. The pre-established long-term operating memory is a temporal graph neural network that stores and associates key decision points, multimodal scene representations, and action results in historical tasks.

7. The robot behavior prediction and planning method integrating a large language model and a generative world model according to claim 1, characterized in that, The preset large language model includes a predefined structured function call interface. The structured function call interface calls the abstract state deduction function of the preset world model in the form of text commands, and parses the returned textual deduction results to make logical decisions.

8. The robot behavior prediction and planning method integrating a large language model and a generative world model according to claim 1, characterized in that, After decomposing the prompt words into a sequence of subtasks using chain thinking, a pre-defined world model is called in a structured manner to perform forward state deduction on the candidate action sequences. Based on the deduction results, a feasibility assessment and strategy optimization are performed to obtain an initial action sequence. This initial action sequence serves as the initial iterative solution when iteratively solving the objective function and participates in the solution of the optimal action sequence.

9. The robot behavior prediction and planning method integrating a large language model and a generative world model according to claim 8, characterized in that, The task objective is to minimize the sum of state tracking error and control cost. The process of obtaining the dynamic constraints includes: Based on the results of the feasibility assessment, the basic physical constraints, basic robot motion constraints, and environmental conflict constraints of the current task are obtained. Based on the aforementioned fundamental physical constraints, the latent state representation is decoded into specific physical state constraints through a lightweight mapping network; By fusing the robot's basic motion constraints with the robot's current motion constraints, robot constraints are obtained. The symbolic sub-target numerical constraints output by the preset large language model when decomposing prompt words into sub-task sequences through chain thinking are obtained, and dynamic constraints are obtained by combining the environmental conflict constraints, specific physical state constraints and robot constraints.

10. The robot behavior prediction and planning method integrating a large language model and a generative world model according to claim 1, characterized in that, After acquiring the robot joint trajectory or end effector pose command, the robot is controlled based on the robot joint trajectory or end effector pose command and real-time multimodal observations are acquired simultaneously. State filtering and prediction calibration are performed through the world model. When the characterization uncertainty or prediction deviation or the confidence of the integrated system exceeds the preset threshold, the large language model is triggered to perform semantic anomaly diagnosis on the current situation and generate correction commands based on physical understanding to achieve dynamic replanning and zero-shot task adaptation. Specifically, The state filtering process includes: predicting the current potential state based on real-time multimodal observations using a world model, and calculating the representation uncertainty of the current potential state using a preset uncertainty quantification module; The prediction calibration process includes: predicting the current potential state based on real-time multimodal observations using a world model, calculating the cosine distance between the predicted potential state and the encoded real observed potential state, and obtaining the prediction bias; The confidence level of the integrated system is obtained by comprehensively representing uncertainty and prediction bias through preset weights.

Images