Task-aware multi-modal large language model and world model joint training method, device, equipment and storage medium

Abstract

The application provides a task-aware multi-modal large language model and world model joint training method and device, equipment and storage medium, related to the technical field of agents, aiming to improve the reasoning consistency and generalization ability of the model in a complex embodied environment. The method comprises: using a world model to be trained, generating K candidate future trajectories based on the semantic representation of the video observation sequence and the semantic representation of the task instruction text. According to the semantic matching degree between the K candidate future trajectories and the semantic representation of the task instruction text, the K candidate future trajectories are fused to obtain a final future trajectory. Based on the difference between the trajectory-level semantic representation of the final future trajectory and the semantic representation of the video observation sequence, an alignment loss is obtained; at least based on the alignment loss, the parameters of the multi-modal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory-level encoder to be trained are updated.

Description

Technical Field

[0001] This application relates to the field of intelligent agent technology, and more specifically, to a method, apparatus, device, and storage medium for joint training of a task-aware multimodal large language model and a world model. Background Technology

[0002] Embodied intelligence refers to the integration of perception, cognition, decision-making, and action achieved by intelligent agents (such as robots and drones) interacting with their environment in real time through physical entities. General embodied intelligent agents aim to perform various types of tasks in real-world environments within a unified framework. They need to possess the ability to understand multimodal information, transform high-level task objectives into executable internal representations, and achieve stable and reliable control and decision-making based on learned environmental dynamic models.

[0003] Multimodal Large Language Models (MLLMs) have demonstrated powerful capabilities in semantic reasoning and cross-modal understanding; World Models (WMs) excel in modeling environmental dynamics and supporting decision-making. Therefore, related technologies attempt to combine MLLMs and World Models for general embodied agents: using MLLMs as external tools for high-level planning or reward computation; or projecting the representations of MLLMs onto the World Model space through a learning mapper.

[0004] However, the above techniques all have some problems: the multimodal large language model and the world model lack unified constraints in terms of representation space and optimization objectives, making it difficult to jointly optimize them under a unified framework, which limits the consistency and robustness of the system in complex tasks. Relying on unidirectional projection, information is only transferred from the multimodal large language model to the world model, lacking a mechanism for dynamic feedback from the physical environment back to the multimodal large language model, making it difficult to achieve adaptive adjustments in long-term decision-making processes. Summary of the Invention

[0005] The present application provides a method, apparatus, device, and storage medium for joint training of a task-aware multimodal large language model and a world model, aiming to overcome the above-mentioned problems or at least partially solve them.

[0006] The first aspect of this application provides a method for jointly training a task-aware multimodal large language model and a world model, comprising: Using the task encoder to be trained, the semantic representation of the task instruction text is extracted; Using the world model to be trained, and based on the semantic representation of the video observation sequence and the semantic representation of the task instruction text, K candidate future trajectories are generated; Based on the semantic matching degree between each of the K candidate future trajectories and the semantic representation of the task instruction text, the K candidate future trajectories are fused to obtain the final future trajectory; The trajectory-level semantic representation of the final future trajectory is extracted using the trajectory-level encoder to be trained. Alignment loss is obtained based on the difference between the trajectory-level semantic representation of the final future trajectory and the semantic representation of the video observation sequence; Based at least on the alignment loss, the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory encoder to be trained are updated. The trained multimodal large language model, the trained world model, the trained task encoder, and the trained trajectory encoder constitute a task execution agent, which is used to execute tasks according to the target task instruction text.

[0007] In one optional implementation, using the world model to be trained, based on the semantic representation of the video observation sequence and the semantic representation of the task instruction text, K candidate future trajectories are generated, including: The current frame in the video observation sequence is input into the world model to be trained. The encoder of the world model to be trained is used to obtain the current potential state based on the current frame and the previous potential state and the previous action stored in the world model to be trained. The semantic representation of the video observation sequence, the current latent state, and the semantic representation of the task instruction text are mapped to a shared latent space through the fusion module to be trained, so as to obtain the current fusion representation. Using the world model to be trained, starting from the current fused representation, perturbation sampling is performed in the shared latent space to generate the K candidate future trajectories.

[0008] In one optional implementation, the K candidate future trajectories are fused based on the semantic matching degree between each of them and the semantic representation of the task instruction text to obtain the final future trajectory, including: The semantic matching degree between the K candidate future trajectories and the semantic representation of the task instruction text is evaluated to obtain the task consistency score of each of the K candidate future trajectories, and the weight of each candidate trajectory of the K candidate future trajectories is determined. Based on the weights of the K candidate future trajectories, the K candidate future trajectories are fused to obtain the final future trajectory.

[0009] In an optional implementation, the method further includes: Based on the weights of the candidate trajectories of the K candidate future trajectories, the policy losses of the K candidate future trajectories are fused to obtain the total policy loss; Based at least on the alignment loss, parameter updates are performed on the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory-level encoder to be trained, including: Based at least on the alignment loss and the total policy loss, the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory encoder to be trained are updated.

[0010] In one optional implementation, the semantic representation of the video observation sequence, the current latent state, and the semantic representation of the task instruction text are mapped to a shared latent space through the fusion module to be trained, to obtain the current fused representation, including: The semantic representation of the task instruction text is processed by a gating function to obtain the first fusion weight corresponding to the video observation sequence and the second fusion weight corresponding to the current potential state. Based on the first fusion weight and the second fusion weight, the semantic representation of the video observation sequence and the current potential state are fused to obtain the current fused representation.

[0011] In an optional implementation, after obtaining the first fusion weight corresponding to the video observation sequence and the second fusion weight corresponding to the current potential state, the method further includes: The task type is determined based on the semantic representation of the task instruction text; When the task type is a semantically dominant task type, the first fusion weight corresponding to the video observation sequence is adjusted to be greater than the second fusion weight corresponding to the current potential state; When the task type is an environment-dynamic-dominated task type, the second fusion weight corresponding to the current potential state is adjusted to be greater than the first fusion weight corresponding to the video observation sequence.

[0012] In one optional implementation, the encoder of the multimodal large language model to be trained has M layers, and the encoder of the world model to be trained also has M layers, each m layer having its own independent gating function; based on the first fusion weight and the second fusion weight, the semantic representation of the video observation sequence and the current latent state are fused to obtain the current fused representation, including: Based on the first fusion weight and the second fusion weight, the semantic representation of the video observation sequence output by the first layer of the encoder of the multimodal large language model to be trained and the current latent state output by the first layer of the encoder of the world model to be trained are fused to obtain the first layer fusion representation. Taking m sequentially from 1 to M, based on the semantic representation of the task instruction text and the independent gating function of each m layer, the first fusion weight and the second fusion weight of the m-th layer are determined. The semantic representation of the video observation sequence output by the encoder of the multimodal large language model to be trained, the current latent state output by the encoder of the m-th layer of the world model to be trained, and the fusion representation of the (m-1)-th layer are fused to obtain the fusion representation of the m-th layer. The M-th layer fusion representation is taken as the current fusion representation.

[0013] A second aspect of this application provides a joint training device for a task-aware multimodal large language model and a world model, comprising: The video semantic extraction module is used to input the video observation sequence consisting of the first frame to the tth frame into the multimodal large language model to be trained, and to extract the semantic representation of the video observation sequence using the encoder of the multimodal large language model to be trained. The task semantic extraction module is used to extract the semantic representation of the task instruction text using the task encoder to be trained; The candidate trajectory generation module is used to generate K candidate future trajectories based on the semantic representation of the video observation sequence and the semantic representation of the task instruction text, using the world model to be trained. The trajectory fusion module is used to fuse the K candidate future trajectories based on the semantic matching degree between each of the K candidate future trajectories and the semantic representation of the task instruction text, so as to obtain the final future trajectory. The trajectory semantic extraction module is used to extract the trajectory-level semantic representation of the final future trajectory through the trajectory-level encoder to be trained; The alignment loss calculation module is used to obtain the alignment loss based on the difference between the trajectory-level semantic representation of the final future trajectory and the semantic representation of the video observation sequence; An update module is used to update the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory encoder to be trained, based at least on the alignment loss. The trained multimodal large language model, the trained world model, the trained task encoder, and the trained trajectory encoder constitute a task execution agent, which is used to execute tasks according to the target task instruction text.

[0014] A third aspect of this application provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the steps of the joint training method of the task-aware multimodal large language model and world model of the first aspect of this application.

[0015] A fourth aspect of this application provides a readable storage medium storing a program or instructions, which, when executed by a processor, implements the steps of the joint training method of a task-aware multimodal large language model and a world model according to the first aspect of this application.

[0016] Beneficial technical effects: This application, in its joint training method for a task-aware multimodal large language model and a world model, utilizes the world model to be trained to generate K candidate future trajectories based on the semantic representations of video observation sequences and task instruction texts. The K candidate future trajectories are then fused according to their semantic matching degree with the semantic representations of the task instruction texts to obtain the final future trajectory. The world model can generate multiple possible future evolution paths and uses task semantics to filter and optimize them, thereby improving the robustness and semantic consistency of decision-making. An alignment loss is obtained based on the difference between the trajectory-level semantic representation of the final future trajectory and the semantic representation of the video observation sequence; at least based on this alignment loss, the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory-level encoder to be trained are updated. This application achieves a closed-loop learning process from semantic understanding and dynamic modeling to behavior generation within a unified framework, enabling semantic information not only to guide physical behavior generation but also to be inversely constrained by physical dynamics, thus significantly improving the model's reasoning consistency and generalization ability in complex embodied environments. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the steps of a task-aware multimodal large language model and world model joint training method proposed in an embodiment of this application. Figure 2 This is a schematic diagram of the dynamic joint framework of the task-aware multimodal large language model and world model joint training method proposed in an embodiment of this application; Figure 3 This is a schematic diagram of different forms of embodied intelligent agents walking in a simulation environment, based on the joint training method of a task-aware multimodal large language model and a world model proposed in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] General embodied agents aim to perform various types of tasks in real-world environments within a unified framework. These agents need to understand multimodal information, transform high-level task objectives into executable internal representations, and achieve stable and reliable control and decision-making based on learned environmental dynamics models. Compared to methods that rely on task-specific processes or are designed for single scenarios, general embodied agents aim to build a unified system architecture that can effectively generalize across different task types and diverse environmental conditions, while maintaining consistency in decision-making and stability in control behavior.

[0021] Multimodal large language models (MLLMs) demonstrate powerful capabilities in semantic reasoning and cross-modal understanding. They possess rich world knowledge and strong combinatorial generalization abilities, enabling them to understand high-level task instructions and align multimodal inputs within a unified semantic space. Meanwhile, world models (WMs) excel in modeling environmental dynamics and supporting decision-making. They capture temporal dependencies by learning latent state representations of the environment, thus supporting long-term imagining and planning. Despite the advantages of MLLMs in semantic understanding and cross-modal generalization, they lack the ability to interact with the physical environment; conversely, while world models offer accurate dynamic prediction and control capabilities, they are limited in semantic abstraction and cross-task generalization.

[0022] Therefore, integrating these two paradigms provides a promising path for building open embodied intelligence: a multimodal large language model provides semantic intent and contextual understanding, while a world model provides prediction and action modeling under physical constraints. Together, they construct a unified framework capable of reasoning, interacting, and adapting in complex real-world environments.

[0023] However, the methods that combine multimodal large language models with world models in related technologies still have obvious limitations in breadth and depth. Some works only use multimodal large language models as external tools for auxiliary functions such as high-level planning or reward calculation. Although these methods utilize the semantic priors of multimodal large language models to assist decision-making and reward design, the learning objectives between multimodal large language models and world models are inconsistent, and there is a lack of effective connection between semantic space and physical space. Some works have also attempted to map the representation of multimodal large language models to the representation space of world models by learning a mapper. However, these methods still face two key problems: (1) They rely on unidirectional projection, only transferring information from multimodal large language models to world models, while environmental interaction is still completely dominated by the world model, lacking a mechanism for feedback from physical dynamics back to multimodal large language models; (2) The learned connectors are usually task-independent, adopting a uniform alignment strategy for all tasks, and cannot adapt to the semantic differences and dynamic characteristics of different tasks. Because of the significant differences between tasks, this uniform modeling approach can lead to parameter sensitivity issues, thus limiting the generalization ability in multi-task scenarios.

[0024] To address the aforementioned issues, this application proposes a joint training method for a task-aware multimodal large language model and a world model. By introducing task embeddings (semantic representations of task instruction texts) as unified conditional information, multimodal semantic representations and environmental dynamic states are mapped to a shared latent space. A task-driven dynamic routing mechanism adaptively fuses semantic information and physical dynamics. The world model generates multiple candidate future trajectories in the latent space, constructing multi-branch environmental evolution paths. Consistency evaluation and selection of each candidate trajectory are performed in conjunction with task semantics, thereby selecting the evolutionary path that best aligns with the task objective from multiple possible futures to guide policy generation and behavioral decisions. In the feedback-based semantic correction and joint optimization module, the future trajectories generated by the world model are used to impose reverse constraints on the semantic representation, feeding back physical dynamic information to the multimodal semantic space to correct the semantic representation, making it more consistent with the physical evolution laws in the real environment.

[0025] Reference Figure 1 , Figure 1 This is a flowchart illustrating the steps of a task-aware multimodal large language model and world model joint training method proposed in an embodiment of this application. Figure 1 As shown, specifically, the method includes the following steps S11~S17: Step S11: Input the video observation sequence consisting of the first frame to the tth frame into the multimodal large language model to be trained, and use the encoder of the multimodal large language model to be trained to extract the semantic representation of the video observation sequence.

[0026] In this embodiment, the given video observation sequence The input is fed into the multimodal large language model to be trained. The encoder f of the multimodal large language model to be trained... MLLM Video observation sequence Processing is performed to capture the visual semantics, contextual relationships, and temporal structure in the video observation sequence, resulting in a semantic representation of the video observation sequence: This provides explicit conditions for subsequent fusion. Among these, the semantic representation of the video observation sequence... Represent the semantic structure and contextual relationships in temporal visual input.

[0027] Step S12: Use the task encoder to be trained to extract the semantic representation of the task instruction text.

[0028] In this embodiment, the given task instruction text is input into the task encoder to be trained, and then processed by the task encoder f... task Extract the semantic representation of the task instruction text: This provides explicit conditions for subsequent integration. Task instruction text refers to high-level task objectives described in natural language, used to tell the agent what task needs to be completed, such as a user expressing a task instruction text that the robot "walks from the starting point to the ending point." Semantic representation of task instruction text. Characterize the coding task objectives and their semantic constraints.

[0029] Step S13: Using the world model to be trained, generate K candidate future trajectories based on the semantic representation of the video observation sequence and the semantic representation of the task instruction text.

[0030] In this embodiment, the world model to be trained is based on the semantic representation of video observation sequences. Semantic representation of task instruction text K candidate future trajectories are generated, each recursively generated by the transition function of the world model to be trained. These candidate future trajectories reflect the multiple possible developments of environmental dynamics, providing a rich decision candidate space for task-driven planning.

[0031] In one implementation, step S13, "using the world model to be trained, and based on the semantic representation of the video observation sequence and the semantic representation of the task instruction text, to generate K candidate future trajectories," may specifically include the following steps S131~S133: Step S131: Input the current frame in the video observation sequence into the world model to be trained, and use the encoder of the world model to be trained to obtain the current potential state based on the current frame and the previous potential state and the previous action stored in the world model to be trained.

[0032] In this embodiment, the world model to be trained adopts a world model based on the cyclic state-space model (RSSM) to model environmental dynamics. The current frame in the video observation sequence... Input the world model to be trained. The encoder of the world model to be trained. Based on the stored previous potential state And the previous action Based on the recursive relationship, the current frame in the video observation sequence is... Mapped to the current potential state : Among them, the current potential state It comprehensively depicts information on current observations, historical states, and actions already taken, thus providing a compact representation of dynamic environmental changes.

[0033] Step S132: Through the fusion module to be trained, the semantic representation of the video observation sequence, the current latent state and the semantic representation of the task instruction text are mapped to the shared latent space to obtain the current fusion representation.

[0034] In this embodiment, the fusion module to be trained The semantic representation of video observation sequences is obtained through a unified fusion function. Current potential state Semantic representation of task instruction text Mapping to the shared latent space yields the current fused representation. By introducing task embedding (the semantic representation of task instruction text) as unified conditional information, multimodal semantic representation and dynamic environmental state are mapped to a shared latent space. The two have unified constraints on representation space and optimization objective, which improves the consistency and robustness of the system in complex tasks.

[0035] This application addresses the common practice in related technologies of combining multimodal large language models with world models that employ task-agnostic unified fusion or alignment strategies, using the same representation mapping or decision mechanism for different tasks while ignoring the differences in semantic structure and dynamic characteristics between tasks. This static modeling approach struggles to adapt to the significantly changing semantic requirements and environmental dynamics in multi-task scenarios, resulting in limited generalization ability of the model across task and environment settings. Therefore, this application further introduces a task-aware dynamic gating mechanism to enable the model to flexibly balance semantically dominant and dynamically dominant tasks.

[0036] In one implementation, step S132, "mapping the semantic representation of the video observation sequence, the current latent state, and the semantic representation of the task instruction text to a shared latent space through the fusion module to be trained, to obtain the current fusion representation," may specifically include the following steps S1321~S1322: Step S1321: Process the semantic representation of the task instruction text through a gating function to obtain the first fusion weight corresponding to the video observation sequence and the second fusion weight corresponding to the current potential state; Step S1322: Based on the first fusion weight and the second fusion weight, fuse the semantic representation of the video observation sequence and the current potential state to obtain the current fusion representation.

[0037] In this embodiment, the semantic representation of the task instruction text is... The gating function input to the fusion module to be trained The first fusion weights corresponding to the video observation sequence are obtained. And the second fusion weight 1- corresponding to the current potential state And based on this, semantic representation of the video observation sequence. and current potential state Perform weighted fusion to obtain the current fusion representation: .in, and These are used to extract dynamic branch features and semantic branch features, respectively. The model adaptively adjusts the contribution ratio of semantic and dynamic information based on task semantics, enabling it to flexibly balance semantically and dynamically dominant tasks. This addresses the limitation of generalization ability across tasks and environments caused by static modeling methods.

[0038] In one implementation, after step S1321, "processing the semantic representation of the task instruction text through a gating function to obtain the first fusion weight corresponding to the video observation sequence and the second fusion weight corresponding to the current potential state," steps S1323-S1325 are further included: Step S1323: Determine the task type based on the semantic representation of the task instruction text; Step S1324: If the task type is a semantically dominant task type, adjust the first fusion weight corresponding to the video observation sequence to be greater than the second fusion weight corresponding to the current potential state; Step S1325: If the task type is an environment-dynamic-dominated task type, adjust the second fusion weight corresponding to the current potential state to be greater than the first fusion weight corresponding to the video observation sequence.

[0039] In this embodiment, a lightweight classifier or rule function can be used to semantically represent the task instruction text. The analysis outputs a task type label: semantically dominant task type or environment-dynamically dominant task type. If the task type is semantically dominant, the first fusion weight p corresponding to the video observation sequence is... t (Corresponding semantic branch) Adjust to a weight greater than the second fusion weight 1-p corresponding to the current latent state. t (Corresponding to dynamic branches). Conversely, if the task type is an environment-driven task type, the second fusion weight 1-p corresponding to the current potential state is changed. t Adjust to a value greater than the first fusion weight p corresponding to the video observation sequence. t It enables dynamic adjustment based on task adaptation, allowing the fusion weights to change flexibly according to the task type.

[0040] For example, if the semantic representation of the task instruction text corresponds to tasks where semantic information is more important than physical dynamics, such as "object recognition (e.g., finding a red cup)" or "understanding the meaning of the instruction," then the task type is determined to be: semantically dominant task type. If the semantic representation of the task instruction text corresponds to tasks where physical dynamics are more important than semantic information, such as "walking (e.g., walking from the starting point to the ending point)," "jumping," "running," or "grabbing a moving object," then the task type is determined to be: environmentally dynamic task type.

[0041] To further enhance expressive power and training stability, this application extends the above-mentioned fusion process (fusion of semantic representations of video observation sequences, current latent states, and task instruction texts) into a hierarchical structure.

[0042] In one implementation, the encoder of the multimodal large language model to be trained has M layers, and the encoder of the world model to be trained also has M layers, with each m layers having its own independent gating function; the above step S1322 "based on the first fusion weight and the second fusion weight, fuse the semantic representation of the video observation sequence and the current latent state to obtain the current fusion representation" may specifically include the following steps S13221~S13223: Step S13221: Based on the first fusion weight and the second fusion weight, fuse the semantic representation of the video observation sequence output by the first layer of the encoder of the multimodal large language model to be trained and the current latent state output by the first layer of the encoder of the world model to be trained to obtain the first layer fusion representation. Step S13222: Take m sequentially from 1 to M, and based on the first fusion weight and the second fusion weight, fuse the semantic representation of the video observation sequence output by the encoder of the multimodal large language model to be trained, the current latent state output by the encoder of the world model to be trained, and the (m-1)th fusion representation to obtain the fusion representation of the mth layer. Step S13223: Use the Mth layer fusion representation as the current fusion representation.

[0043] In this embodiment, hierarchical fusion is employed, meaning that fusion is performed at each layer of the encoder of the multimodal large language model to be trained and the encoder of the world model to be trained, and the fusion result of each layer is passed to the next layer. This achieves layer-by-layer alignment and fine-grained interaction of semantic and dynamic information, further enhancing the model's expressive power and training stability. Therefore, at each layer... In the semantic representation of task instruction text and each layer Corresponding gating function Generate local control And progressively update the fusion representation of the m-th layer. The fusion representation of the 1st layer is: The m-th layer fusion representation: .

[0044] Specifically, in the first layer, the gating function can correspond to the gating function used above. Then, based directly on the first and second fusion weights calculated above, the semantic representation of the video observation sequence output by the encoder of the multimodal large language model to be trained is obtained. The current latent state and the output of the first layer of the encoder of the world model to be trained The data is then fused to obtain the first-layer fused representation. In the second layer, the semantic representation is based on the task instruction text. The gating function corresponding to the second layer generates the first and second fusion weights of the second layer, and is used to generate the semantic representation of the video observation sequence output by the second layer of the encoder of the multimodal large language model to be trained. The current latent state output by the second layer of the encoder of the world model to be trained. The first-level fusion representation is fused with the second-level fusion representation. Then, m is sequentially chosen from 1 to M to calculate the final M-level fusion representation, which is then used as the current fusion representation. .

[0045] Step S133: Using the world model to be trained, starting from the current fusion representation, perform perturbation sampling in the shared latent space to generate the K candidate future trajectories.

[0046] In this embodiment, unlike traditional world models that extrapolate based on only a single trajectory, the world model to be trained in this application is based on the current fusion representation. Starting from this point, perturbation sampling is performed in the shared potential space to generate K candidate future trajectories: Each candidate future trajectory is recursively generated by the transfer function of the world model to be trained. These candidate future trajectories reflect the diverse possible developments in environmental dynamics, providing a rich space of decision-making candidates for task-driven planning.

[0047] Step S14: Based on the semantic matching degree between each of the K candidate future trajectories and the semantic representation of the task instruction text, the K candidate future trajectories are fused to obtain the final future trajectory.

[0048] In this embodiment, the semantic similarity, or semantic matching degree, between each of the K candidate future trajectories and the semantic representation of the task instruction text is calculated. Based on the semantic matching degree of each candidate trajectory, the K candidate future trajectories are weighted and fused to obtain the final future trajectory. This achieves "semantic selection" among multiple possible futures, freeing the model from reliance on single-path prediction and significantly reducing decision bias caused by error accumulation.

[0049] In one implementation, step S14, "fusing the K candidate future trajectories to obtain the final future trajectory based on the semantic matching degree between each of the K candidate future trajectories and the semantic representation of the task instruction text," may specifically include the following steps S141~S142: Step S141: Evaluate the semantic matching degree between the K candidate future trajectories and the semantic representation of the task instruction text, obtain the task consistency score of each of the K candidate future trajectories, and determine the weight of each candidate trajectory of the K candidate future trajectories; Step S142: Based on the weights of the candidate trajectories of the K candidate future trajectories, fuse the K candidate future trajectories to obtain the final future trajectory.

[0050] In this embodiment, a task consistency scoring function is introduced. The semantic matching degree between each of the K candidate future trajectories and the semantic representation of the task instruction text is evaluated to obtain a task consistency score for each candidate trajectory: The higher the task consistency score of a candidate future trajectory, the greater the degree to which the candidate future trajectory conforms to the semantic constraints of the task instruction text. Scoring Function This is achieved by calculating the similarity (cosine similarity) between the semantic representation of the task instruction text and the semantic representation of the trajectory, to characterize the extent to which the trajectory conforms to the semantic constraints of the given task. Subsequently, softmax normalization is used to obtain the weights of each candidate trajectory. Based on the weights of the K candidate future trajectories, a weighted fusion is performed on the K candidate future trajectories to obtain the final future trajectory. The action that needs to be performed at the current moment in the final future trajectory is the final decision output. The final decision output can be expressed as: It achieves "semantic selection" among multiple possible futures, enabling the model to no longer rely on single-path prediction, but instead to filter and reweight multiple counterfactual trajectories through task semantics, thereby significantly reducing decision bias caused by error accumulation.

[0051] Step S15: Extract the trajectory-level semantic representation of the final future trajectory using the trajectory-level encoder to be trained.

[0052] In this embodiment, the trajectory-level encoder to be trained performs semantic aggregation and mapping on the multi-step predicted trajectories (final future trajectories) generated by the world model to be trained in the shared latent space, thereby obtaining the trajectory-level semantic representation of the final future trajectory: .

[0053] Step S16: Based on the difference between the trajectory-level semantic representation of the final future trajectory and the semantic representation of the video observation sequence, obtain the alignment loss.

[0054] In this embodiment, considering that traditional methods typically only utilize multimodal large language models to provide high-level semantic guidance while ignoring the reverse correction effect of environmental dynamics on semantic representations, leading to semantic-physical inconsistencies in complex interaction scenarios, this application calculates the difference between the trajectory-level semantic representation of the final future trajectory and the semantic representation of the video observation sequence to obtain an alignment loss for updating model parameters. Specifically, the trajectory-level semantic representation of the final future trajectory... Semantic representation of video observation sequences Alignment is performed to minimize the reconstruction error, allowing the semantic representation of the video observation sequence to gradually approximate the physically realizable state distribution. This avoids spurious reasoning or decisions that do not conform to physical constraints due to reliance solely on visual semantics. The alignment loss is: .

[0055] Step S17: Based at least on the alignment loss, update the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory encoder to be trained. The trained multimodal large language model, the trained world model, the trained task encoder, and the trained trajectory encoder form a task execution agent, which is used to execute tasks according to the target task instruction text.

[0056] In this embodiment, the alignment loss is calculated as described above. The parameters of the multimodal large language model, the world model, the task encoder, and the trajectory encoder to be trained are updated. The jointly trained multimodal large language model, world model, task encoder, and trajectory encoder constitute a task execution agent. This agent executes tasks based on target task instructions. For example, the agent is a robot; the environment is an indoor testing ground with a low step in front of it; the target task instruction is "walk to the step and then step over it." The robot will execute the task: walk to the step at a normal walking speed and then step over it. By introducing a feedback-based semantic correction mechanism, the semantic representation is back-aligned using the future trajectory generated by the world model, gradually approximating the physically realizable state distribution, thus achieving closed-loop collaborative optimization between semantics and dynamics.

[0057] In one implementation, the joint training method of the task-aware multimodal large language model and the world model further includes the following step S18. The aforementioned step S17, "updating the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory-level encoder to be trained, at least based on the alignment loss," may specifically include the following step S171: Step S18: Based on the weights of the candidate trajectories of the K candidate future trajectories, fuse the policy losses of the K candidate future trajectories to obtain the total policy loss; Step S171: Update the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory encoder to be trained, based at least on the alignment loss and the total policy loss.

[0058] In this embodiment, the strategy loss Multiple candidate future trajectories generated based on counterfactual imagination are then scored based on task consistency. Construct the weights of each of the K candidate future trajectories. The loss function is defined in the form of weighted behavior optimization, thereby guiding the policy to prioritize learning behavioral patterns that are highly consistent with the task semantics. Specifically, it is based on the weights of the candidate trajectories of the K candidate future trajectories. The policy losses of the K candidate future trajectories are fused to obtain the total policy loss, which is: ; in, This represents the policy function, whose parameters are... Used to determine the current potential state Output corresponding action The probability distribution is used to characterize the probability of choosing each candidate action in a given state.

[0059] Based on the alignment loss calculated above and strategy loss The parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory-level encoder to be trained are updated. Furthermore, the world model loss can be used as a basis for parameter updates. Update the model parameters. World model loss. A prior-posterior alignment and observation reconstruction based on a cyclic state-space model is employed to learn temporally consistent and predictable latent dynamics, using a world model loss. for:

[0060] in, and Let represent the prior latent distribution and the posterior latent distribution in the cyclic state-space model (RSSM), respectively.

[0061] Based on the above mechanism, this application constructs a unified joint optimization objective, according to the alignment loss... Strategy loss and world model loss The total loss is used to update the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory-level encoder to be trained. Total Loss: Through the synergistic optimization of the above three objectives, the model achieves a closed-loop learning process from semantic understanding and dynamic modeling to behavior generation within a unified framework. This enables semantic information not only to guide the generation of physical behaviors but also to be constrained by physical dynamics, thereby significantly improving the model's reasoning consistency and generalization ability in complex embodied environments.

[0062] Based on the above, such as Figure 2As shown, this application proposes a task-aware dynamic joint framework that achieves coupling between a multimodal large language model and a world model through a modular approach. Input Layer: Contains two main input sources: video observations and task instruction text, corresponding to embodied intelligence's environmental perception information and high-level task objectives (such as walking or kicking), respectively. Intermediate Encoding Layer: The multimodal large language model encoder (MLLM Enc) extracts semantic representations from the video observation sequences; the world model encoder (WM Enc) extracts dynamic latent states of the environment based on a cyclic state space model; and the task encoder extracts semantic representations (task embeddings) from the task instruction text. Core Fusion Layer: The task-aware dynamic coupling module (fusion module) is the core. Through a task-driven dynamic routing / gating mechanism, it maps semantic representations, dynamic states (latest states), and task embeddings to a shared latent space, adaptively adjusting the fusion weights of semantic and dynamic information to generate the current fusion representation z. t The fusion process is extended into a hierarchical structure, with both the encoder of the multimodal large language model and the encoder of the world model having M layers (e.g., 5 layers). Different tasks have different weights at different layers; for example, the task "walking" has higher weights at layers 1, 3, and 4, while the task "kicking" has higher weights at layers 2, 4, and 5. Counterfactual imagination planning layer: represented by fusion z. t Starting with the world model, multiple counterfactual candidate future trajectories (imagined future trajectories) are generated. These trajectories are then filtered and weighted using a task consistency scoring function to output the final future trajectory (target trajectory), thereby determining the optimal policy and action. The joint optimization layer comprises three loss functions: world model loss... Strategy loss Alignment loss This enables bidirectional closed-loop joint optimization of a multimodal large language model and a world model, with physical dynamic information used to correct semantic representations, forming a feedback loop. For example... Figure 3 The image shows examples of different types of intelligent agents walking in a simulation environment, using the dynamic joint framework of this application.

[0063] This application establishes a task-driven, bidirectional collaborative mechanism between a multimodal large language model and a world model, thereby achieving embodied intelligent reasoning and behavior generation capabilities that combine semantic consistency, dynamic rationality, and decision interpretability. Compared to related technologies, the technical solution of this application has the following advantages: (1) Most related technologies treat multimodal large language models as independent high-level semantic planners, guiding the world model to perform tasks only through unidirectional information flow, lacking a unified semantic-dynamic collaborative modeling mechanism. However, this application constructs a task-aware semantic-dynamic coupling framework, integrating semantic representation and environmental dynamic state in a unified latent space, enabling the model to adaptively adjust the weights between semantic understanding and physical modeling according to task requirements, thereby achieving more consistent cross-modal representation learning.

[0064] (2) Related technologies generally ignore the reverse constraint of environmental dynamics on semantic representation, which easily leads to reasoning results that do not conform to physical laws in complex interaction scenarios. However, this application introduces a feedback-based semantic correction mechanism and uses the future trajectory generated by the world model to reverse align the semantic representation, so that the semantic representation gradually approaches the physically realizable state distribution, thereby realizing closed-loop collaborative optimization between semantics and dynamics.

[0065] (3) This application achieves efficient collaboration between multimodal large language model and world model without relying on large-scale model structure modification. Through modular design and unified latent space docking, it has good scalability and cross-task generalization ability, and can adapt to different environments and task scenarios. Compared with related technologies that require complex architecture adjustment or large-scale training, it has higher practical value and deployment efficiency.

[0066] Based on the same inventive concept, one embodiment of this application provides a joint training device for a task-aware multimodal large language model and a world model, the device comprising: The video semantic extraction module is used to input the video observation sequence consisting of the first frame to the tth frame into the multimodal large language model to be trained, and to extract the semantic representation of the video observation sequence using the encoder of the multimodal large language model to be trained. The task semantic extraction module is used to extract the semantic representation of the task instruction text using the task encoder to be trained; The candidate trajectory generation module is used to generate K candidate future trajectories based on the semantic representation of the video observation sequence and the semantic representation of the task instruction text, using the world model to be trained. The trajectory fusion module is used to fuse the K candidate future trajectories based on the semantic matching degree between each of the K candidate future trajectories and the semantic representation of the task instruction text, so as to obtain the final future trajectory. The trajectory semantic extraction module is used to extract the trajectory-level semantic representation of the final future trajectory through the trajectory-level encoder to be trained; The alignment loss calculation module is used to obtain the alignment loss based on the difference between the trajectory-level semantic representation of the final future trajectory and the semantic representation of the video observation sequence; An update module is used to update the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory encoder to be trained, based at least on the alignment loss. The trained multimodal large language model, the trained world model, the trained task encoder, and the trained trajectory encoder constitute a task execution agent, which is used to execute tasks according to the target task instruction text.

[0067] In one optional implementation, the candidate trajectory generation module includes: The latent state generation submodule is used to input the current frame in the video observation sequence into the world model to be trained, and use the encoder of the world model to be trained to obtain the current latent state based on the current frame and the previous latent state and the previous action stored in the world model to be trained. The mapping submodule is used to map the semantic representation of the video observation sequence, the current latent state, and the semantic representation of the task instruction text to a shared latent space through the fusion module to be trained, so as to obtain the current fused representation; The candidate future trajectory generation submodule is used to generate the K candidate future trajectories by performing perturbation sampling in the shared latent space, starting from the current fusion representation and utilizing the world model to be trained.

[0068] In one optional implementation, the trajectory fusion module includes: The evaluation submodule is used to evaluate the semantic matching degree between the K candidate future trajectories and the semantic representation of the task instruction text, obtain the task consistency score of each of the K candidate future trajectories, and determine the weight of each of the K candidate future trajectories. The final future trajectory generation submodule is used to fuse the K candidate future trajectories according to their respective weights to obtain the final future trajectory.

[0069] In one alternative embodiment, the device further includes: The total policy loss calculation module is used to fuse the policy losses of the K candidate future trajectories according to the weights of their respective candidate trajectories to obtain the total policy loss. The update module also includes: The multiple update submodule is used to update the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory encoder to be trained, based at least on the alignment loss and the total policy loss.

[0070] In one optional implementation, the mapping submodule includes: The gating processing subunit is used to process the semantic representation of the task instruction text through a gating function to obtain the first fusion weight corresponding to the video observation sequence and the second fusion weight corresponding to the current potential state. An adaptive fusion subunit is used to fuse the semantic representation of the video observation sequence and the current potential state based on the first fusion weight and the second fusion weight to obtain the current fused representation.

[0071] In one optional implementation, the mapping submodule further includes: The task type determination subunit is used to determine the task type based on the semantic representation of the task instruction text; The first adjustment subunit is used to adjust the first fusion weight corresponding to the video observation sequence to be greater than the second fusion weight corresponding to the current potential state when the task type is a semantically dominant task type. The second adjustment subunit is used to adjust the second fusion weight corresponding to the current potential state to be greater than the first fusion weight corresponding to the video observation sequence when the task type is an environment dynamics-dominated task type.

[0072] In one optional implementation, the encoder of the multimodal large language model to be trained has M layers, and the encoder of the world model to be trained also has M layers, with each m layer having its own independent gating function; the adaptive fusion subunit is specifically used for: Based on the first fusion weight and the second fusion weight, the semantic representation of the video observation sequence output by the first layer of the encoder of the multimodal large language model to be trained and the current latent state output by the first layer of the encoder of the world model to be trained are fused to obtain the first layer fusion representation. Taking m sequentially from 1 to M, based on the semantic representation of the task instruction text and the independent gating function of each m layer, the first fusion weight and the second fusion weight of the m-th layer are determined. The semantic representation of the video observation sequence output by the encoder of the multimodal large language model to be trained, the current latent state output by the encoder of the m-th layer of the world model to be trained, and the fusion representation of the (m-1)-th layer are fused to obtain the fusion representation of the m-th layer. The M-th layer fusion representation is taken as the current fusion representation.

[0073] Based on the same concept, one embodiment of this application provides an electronic device, which includes a memory and a processor. The memory and the processor are connected via a bus for communication. The memory stores a program or instructions that can be executed on the processor to implement the steps in the joint training method of task-aware multimodal large language model and world model described in any of the above embodiments of this application.

[0074] Based on the same inventive concept, this disclosure also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps in the joint training method of the task-aware multimodal large language model and world model described in any of the above embodiments of this application.

[0075] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0076] Based on the same inventive concept, this disclosure also provides a computer program product, including a computer program that, when executed by a processor of a computer device, is capable of performing the steps in the joint training method of the task-aware multimodal large language model and world model described in any of the above embodiments of this application.

[0077] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0078] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0083] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0084] The foregoing has provided a detailed description of the joint training method, apparatus, device, and storage medium for a task-aware multimodal large language model and a world model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for jointly training a task-aware multimodal large language model and a world model, characterized in that, include: The video observation sequence consisting of the first frame to the tth frame is input into the multimodal large language model to be trained, and the semantic representation of the video observation sequence is extracted using the encoder of the multimodal large language model to be trained. Using the task encoder to be trained, the semantic representation of the task instruction text is extracted; Using the world model to be trained, and based on the semantic representation of the video observation sequence and the semantic representation of the task instruction text, K candidate future trajectories are generated; Based on the semantic matching degree between each of the K candidate future trajectories and the semantic representation of the task instruction text, the K candidate future trajectories are fused to obtain the final future trajectory; The trajectory-level semantic representation of the final future trajectory is extracted using the trajectory-level encoder to be trained. Alignment loss is obtained based on the difference between the trajectory-level semantic representation of the final future trajectory and the semantic representation of the video observation sequence; Based at least on the alignment loss, the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory encoder to be trained are updated. The trained multimodal large language model, the trained world model, the trained task encoder, and the trained trajectory encoder constitute a task execution agent, which is used to execute tasks according to the target task instruction text.

2. The joint training method for a task-aware multimodal large language model and a world model according to claim 1, characterized in that, Using the world model to be trained, and based on the semantic representation of the video observation sequence and the semantic representation of the task instruction text, K candidate future trajectories are generated, including: The current frame in the video observation sequence is input into the world model to be trained. The encoder of the world model to be trained is used to obtain the current potential state based on the current frame and the previous potential state and the previous action stored in the world model to be trained. The semantic representation of the video observation sequence, the current latent state, and the semantic representation of the task instruction text are mapped to a shared latent space through the fusion module to be trained, so as to obtain the current fusion representation. Using the world model to be trained, starting from the current fused representation, perturbation sampling is performed in the shared latent space to generate the K candidate future trajectories.

3. The joint training method for a task-aware multimodal large language model and a world model according to claim 1, characterized in that, Based on the semantic matching degree between each of the K candidate future trajectories and the semantic representation of the task instruction text, the K candidate future trajectories are fused to obtain the final future trajectory, including: The semantic matching degree between the K candidate future trajectories and the semantic representation of the task instruction text is evaluated to obtain the task consistency score of each of the K candidate future trajectories, and the weight of each candidate trajectory of the K candidate future trajectories is determined. Based on the weights of the K candidate future trajectories, the K candidate future trajectories are fused to obtain the final future trajectory.

4. The joint training method for the task-aware multimodal large language model and the world model according to claim 3, characterized in that, The method further includes: Based on the weights of the candidate trajectories of the K candidate future trajectories, the policy losses of the K candidate future trajectories are fused to obtain the total policy loss; Based at least on the alignment loss, parameter updates are performed on the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory-level encoder to be trained, including: Based at least on the alignment loss and the total policy loss, the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory encoder to be trained are updated.

5. The joint training method for a task-aware multimodal large language model and a world model according to claim 2, characterized in that, The semantic representation of the video observation sequence, the current latent state, and the semantic representation of the task instruction text are mapped to a shared latent space through the fusion module to be trained, resulting in the current fusion representation, including: The semantic representation of the task instruction text is processed by a gating function to obtain the first fusion weight corresponding to the video observation sequence and the second fusion weight corresponding to the current potential state. Based on the first fusion weight and the second fusion weight, the semantic representation of the video observation sequence and the current potential state are fused to obtain the current fused representation.

6. The joint training method for a task-aware multimodal large language model and a world model according to claim 5, characterized in that, After obtaining the first fusion weight corresponding to the video observation sequence and the second fusion weight corresponding to the current potential state, the method further includes: The task type is determined based on the semantic representation of the task instruction text; When the task type is a semantically dominant task type, the first fusion weight corresponding to the video observation sequence is adjusted to be greater than the second fusion weight corresponding to the current potential state; When the task type is an environment-dynamic-dominated task type, the second fusion weight corresponding to the current potential state is adjusted to be greater than the first fusion weight corresponding to the video observation sequence.

7. The joint training method for a task-aware multimodal large language model and a world model according to claim 5, characterized in that, The encoder of the multimodal large language model to be trained has M layers, and the encoder of the world model to be trained has M layers, with each m layer having its own independent gating function. Based on the first fusion weight and the second fusion weight, the semantic representation of the video observation sequence and the current latent state are fused to obtain the current fused representation, including: Based on the first fusion weight and the second fusion weight, the semantic representation of the video observation sequence output by the first layer of the encoder of the multimodal large language model to be trained and the current latent state output by the first layer of the encoder of the world model to be trained are fused to obtain the first layer fusion representation. Taking m sequentially from 1 to M, based on the semantic representation of the task instruction text and the independent gating function of each m layer, the first fusion weight and the second fusion weight of the m-th layer are determined. The semantic representation of the video observation sequence output by the encoder of the multimodal large language model to be trained, the current latent state output by the encoder of the m-th layer of the world model to be trained, and the fusion representation of the (m-1)-th layer are fused to obtain the fusion representation of the m-th layer. The M-th layer fusion representation is taken as the current fusion representation.

8. A joint training device for a task-aware multimodal large language model and a world model, characterized in that, The device includes: The video semantic extraction module is used to input the video observation sequence consisting of the first frame to the tth frame into the multimodal large language model to be trained, and to extract the semantic representation of the video observation sequence using the encoder of the multimodal large language model to be trained. The task semantic extraction module is used to extract the semantic representation of the task instruction text using the task encoder to be trained; The candidate trajectory generation module is used to generate K candidate future trajectories based on the semantic representation of the video observation sequence and the semantic representation of the task instruction text, using the world model to be trained. The trajectory fusion module is used to fuse the K candidate future trajectories based on the semantic matching degree between each of the K candidate future trajectories and the semantic representation of the task instruction text, so as to obtain the final future trajectory. The trajectory semantic extraction module is used to extract the trajectory-level semantic representation of the final future trajectory through the trajectory-level encoder to be trained; The alignment loss calculation module is used to obtain the alignment loss based on the difference between the trajectory-level semantic representation of the final future trajectory and the semantic representation of the video observation sequence; An update module is used to update the parameters of the multimodal large language model to be trained, the world model to be trained, the task encoder to be trained, and the trajectory encoder to be trained, based at least on the alignment loss. The trained multimodal large language model, the trained world model, the trained task encoder, and the trained trajectory encoder constitute a task execution agent, which is used to execute tasks according to the target task instruction text.

9. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the joint training method of the task-aware multimodal large language model and the world model as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by the processor, they implement the steps of the joint training method of the task-aware multimodal large language model and the world model as described in any one of claims 1-7.

Images