Large model and reasoning method for cross-space, cross-task and cross-ontology learning

Abstract

The invention relates to the field of artificial intelligence and self intelligence, and discloses a cross-space, cross-task and cross-ontology learning large model and reasoning method, the method comprises the following steps: a pre-training multi-modal large language model as a backbone network is used for processing visual and text input and generating a response; the intention bridging interface is used for compressing a hidden state output by the multi-mode large language model into semantic intention representation with a fixed length; the action strategy head is used for generating a continuous action sequence based on the semantic intention; the state encoder is used for encoding the body sensing state of the robot; and the motion encoder / decoder is used for embedding and reconstructing the motion. According to the method, cross-space migration, cross-task learning and cross-ontology generalization are realized in a single model, and the performance and generalization ability of digital space reasoning and physical space control are improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and embodied intelligence, specifically to a large model and reasoning method, system and device based on cross-space, cross-task and cross-ontology learning, which is particularly suitable for the seamless transfer of multimodal large language models in digital space and physical space, multi-task semantic alignment and unified control of multiple robot bodies. Background Technology

[0002] With the development of artificial intelligence, multimodal large language models (MLLMs) have made significant progress in visual-language reasoning and are increasingly being applied to embodied agents. However, existing technologies have obvious limitations: First, MLLMs have poor generalization ability between digital and physical spaces, lacking the ability to perceive the physical world and reason about space; second, visual-language-action models (VLAs) can output low-level actions, but usually sacrifice the original general reasoning and instruction following capabilities of MLLMs; third, most existing embodied large language models (ELLMs) are limited to abstract planning in digital space, making it difficult to generate reliable low-level control signals and generalize them to the physical world.

[0003] Current research has attempted to address these issues through multi-stage training or by connecting MLLMs to policy modules. However, these methods often lead to a degradation in the model's ability to follow instructions, or result in fragmented pipelines that are difficult to optimize end-to-end. A unified model capable of seamlessly operating in both digital and physical spaces, while generalizing to different tasks and robot bodies, remains lacking.

[0004] Therefore, there is an urgent need in this field for a unified basic model that can integrate high-level reasoning, instruction following, and low-level robot control, and has the ability to generalize across spaces, tasks, and ontology. Summary of the Invention

[0005] The main objective of this invention is to address the technical problems in existing technologies, such as the fragmentation of knowledge transfer between digital and physical spaces, insufficient cross-task semantic alignment and generalization, and difficulties in unified control of multiple robot ontology. These problems lead to difficulties in achieving stable and efficient continuous action generation without compromising the instruction following and reasoning capabilities of multimodal large language models. A large model and reasoning method for cross-space, cross-task, and cross-ontology learning includes the following steps: A pre-trained multimodal large language model serves as the backbone network for processing visual and text inputs and generating responses; The intent bridging interface is used to compress the hidden states output by a multimodal large language model into a fixed-length semantic intent representation; Action strategy head, used to generate a continuous sequence of actions based on semantic intent; A state encoder is used to encode the robot's body perception state. Action encoders / decoders are used to embed and reconstruct actions.

[0006] A second aspect of the present invention provides a large-scale model inference system based on cross-space, cross-task, and cross-ontology learning, comprising: Input reception and preprocessing steps: Receive multimodal input containing visual observations and text commands, and perform uniform frame sampling on the video data; Multimodal understanding and intent extraction steps: The input is processed through the multimodal large language model backbone, and a compressed high-level semantic intent is output via the intent bridging interface. ; Conditional action generation steps: based on the aforementioned high-level semantic intent Based on the current robot state, the motion strategy head is used to iteratively denoise and generate a smooth, continuous motion sequence. Closed-loop control and output steps: After denormalizing the generated motion sequence, output it to the robot controller to drive the physical body to complete the command task.

[0007] A third aspect of the present invention provides an electronic device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the electronic device to execute the large model and inference method for cross-space, cross-task and cross-ontology learning as described above.

[0008] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned large model and inference method for cross-space, cross-task, and cross-ontology learning.

[0009] The present invention has the following beneficial effects: 1. This invention effectively solves the problem of the imbalance between digital space reasoning and physical space control capabilities in a single model through a two-stage training paradigm. The first stage involves supervised fine-tuning on a large-scale digital space corpus, injecting embodied knowledge into the multimodal large language model while fully preserving its original instruction compliance and general reasoning capabilities. The second stage freezes the backbone of the large model and specifically trains the policy module, enabling the model to acquire precise low-level control capabilities while avoiding language capability degradation caused by robot data fine-tuning, thus achieving synergy and balance between high-order reasoning and low-level control within a single model.

[0010] 2. This invention innovatively designs an intent bridging interface, constructing a reliable path between high-level semantics and low-level control commands. This interface, through a perceptron-based compression module, refines the dense, high-dimensional intent tokens output by the multimodal large language model into a fixed-length, concise semantic representation. This not only prioritizes the retention of the most control-relevant guiding semantics but also significantly reduces computational overhead, thereby supporting high-frequency closed-loop control and fundamentally solving the problem of the disconnect between the semantic and control layers.

[0011] 3. This invention achieves true cross-robot generalization capability by establishing a unified policy representation through an architecture combining a shared policy backbone and a body-specific codec. The parameters of the action policy head are shared across all robot bodies to learn common behavioral patterns; while lightweight pose and action codecs are individually optimized for each body to adapt to its unique dynamic characteristics. This design allows a single model to directly control multiple robot bodies with different numbers of joints, workspaces, and morphologies without structural adjustments.

[0012] 4. This invention introduces future prediction as an auxiliary training objective, significantly enhancing the model's planning and robustness in long-term tasks. By having the policy module learn semantic representations of predicting future observations, the model is implicitly guided to execute actions that lead to the expected future state. This mechanism endows the model with a certain degree of forward-thinking ability, enabling it to self-correct and maintain the stability of its task trajectory when faced with uncertainties in action execution or slight changes in the environment, thereby improving its robustness in complex physical environments.

[0013] 5. This invention ultimately delivers a "versatile" single-model instance that achieves superior performance in both digital and physical spaces, eliminating the need for model switching in traditional solutions. This single model achieves an average performance improvement of approximately 6% in digital space reasoning tasks and approximately 3% in physical space control tasks compared to top-tier specialized models. It also possesses "unbounded" capabilities for cross-space task execution, cross-task knowledge transfer, and cross-ontology control generalization, providing a scalable technical path for constructing generalized embodied intelligent agents. Attached Figure Description

[0014] Figure 1 The large model and inference flowchart for cross-space, cross-task, and cross-ontology learning provided in the embodiments of the present invention. Detailed Implementation

[0015] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the large model and inference method for cross-space, cross-task, and cross-ontology learning in this invention includes: Model initialization steps: Construct the model architecture, which includes a pre-trained multimodal large language model as the backbone network, an intent bridging interface, and an action policy head; Data collection and preprocessing steps: acquire digital space corpus and physical space cross-ontology demonstration dataset, and standardize and weight sample video and motion data; Multimodal representation and intent embedding computation steps: process input data, generate visual, text and state embeddings, and extract high-level semantic intent through the multimodal large language model and intent bridging interface; The multi-objective loss calculation steps are as follows: In the first training phase, supervised fine-tuning loss is calculated based on the digital space corpus; in the second training phase, flow matching loss and future prediction loss are calculated based on the cross-ontology demonstration dataset. Parameter isolation and phased optimization steps: In the first phase, the visual encoder and word segmenter of the multimodal large language model are frozen, and only their language model parameters are optimized; in the second phase, the backbone network of the entire multimodal large language model is frozen, and the intent bridging interface, action strategy head, and ontology-specific encoder and decoder are optimized.

[0017] Specifically: Step 1: Construct the model architecture, which includes a pre-trained multimodal large language model as the backbone network, an intent bridging interface, and an action policy head.

[0018] The specific steps are as follows: A pre-trained multimodal large language model using the Qwen2.5-VL-7B-Instruct model as the backbone network is employed to process visual and text inputs and generate responses. A perceptron-based compression module is used as the intent bridging interface to compress the hidden states output by a multimodal large language model into a fixed-length semantic intent representation. This module adopts the Perceiver module from the following paper: Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Proceedings of the International Conference on Machine Learning, pages 4651–4664, 2021.

[0019] The DiT structure is used as the action policy head to generate continuous action sequences based on semantic intent; An MLP structure is used as a state encoder to encode the robot's body perception state; An MLP structure is used as the action encoder / decoder for embedding and reconstructing actions; The parameters of the motion strategy head are shared among different robot bodies, while the state encoder and motion encoder / decoder are unique to each robot body.

[0020] Specifically, the solution of the present invention is as follows: The Qwen2.5-VL-7B-Instruct model is used as the pre-trained multimodal large language model for the backbone network to process visual and text inputs and generate responses. Specifically, the backbone network receives multimodal input consisting of uniformly sampled video frame sequences and natural language instructions. Visual features and text features are extracted by a visual encoder and a word segmenter, respectively. After being fused by a lightweight multimodal projector, the extracted features are input to the language model to obtain a response token sequence and its corresponding multimodal hidden state sequence. Multimodal hidden states are extracted from the predetermined Transformer blocks of the multimodal large language model and used as high-level semantic intent features required for subsequent control.

[0021] A compression module based on a perceptron structure is used as the intent bridging interface to compress the hidden states output by the multimodal large language model into a fixed-length semantic intent representation. Specifically, the intent bridging interface constructs operation-related prompts based on a multi-view historical observation window. It takes a high-dimensional, multi-token intent hidden state sequence extracted from a predetermined layer of the backbone network as input. Through the cross-attention between the query latent and the input token set in the perceptron module, the variable-length high-dimensional intent token sequence is mapped to a fixed number of latent tokens. Subsequently, the latent tokens are re-featured and aggregated through several layers of self-attention and feedforward networks to output a fixed number of high-level semantic intent embeddings with fixed dimensions, which are used as conditional inputs to the policy module. This compression process prioritizes the preservation of guiding semantic information related to robot control and significantly reduces the computational overhead of the subsequent policy head, thereby supporting high-frequency closed-loop control.

[0022] A DiT (Diffusion Transformer) structure is used as the action policy head to generate continuous action sequences based on semantic intent. Specifically, the action policy head takes the semantic intent embedding output by the intent bridging interface as conditional input, and simultaneously receives the robot's current proprioceptive state and a predetermined action block within the prediction time domain: the current proprioceptive state is mapped to a state embedding by a state encoder, and the action block is mapped to an action embedding by an action encoder after adding noise. DiT performs alternating self-attention calculations on the state embedding and action embedding, and performs cross-attention calculations on the semantic intent embedding, and performs conditional denoising inference on the noisy action embedding to obtain the corresponding denoised action embedding. The denoised action embedding is reconstructed by the action decoder to output a continuous control action sequence within the prediction time domain. During training, DiT learns the conditional vector field by standardizing and adding noise to the action vectors, combined with flow matching loss, to achieve continuous trajectory generation from noisy actions to target actions.

[0023] An MLP structure is used as a state encoder to encode the robot's ontology perception state. Specifically, the ontology perception state of different robot bodies can include various low-level state variables such as joint position, joint velocity, end effector pose, and grasping state. The state encoder normalizes and linearly maps these ontology state inputs of different dimensions and scales, and uniformly maps them to a state embedding space that matches the DiT structure, so that the shared policy backbone can process state information from different bodies in the same embedding space.

[0024] An MLP structure is used as the motion encoder / decoder for embedding and reconstructing motions. Specifically, the motion encoder maps the original motion vectors in the form of continuous control quantities (such as joint space velocity commands or Cartesian space incremental poses) to the motion embedding space so that they can be used as inputs to DiT along with state embedding. The motion decoder then restores the denoised motion embeddings output by DiT to executable control commands in the actual motion space of each robot body, realizing the mapping from potential motion representation to physical motion.

[0025] The parameters of the action strategy head are shared among different robot bodies to learn common behavior patterns and a unified potential policy representation across bodies. The state encoder and action encoder / decoder are unique to each robot body and are used to adapt to the different body state spaces and action spaces of each robot body, thereby achieving cross-body generalization control among multiple heterogeneous robot bodies without changing the overall architecture.

[0026] In a preferred embodiment, the multimodal large language model backbone extracts multimodal hidden states from its 14th Transformer block as high-level semantic intent features. The perceptron-based compression module in the intent bridging interface adopts a 4-layer self-attention structure, with 32 attention heads per layer and a dimension of 48 for each attention head. A dropout of 0.2 is used in the attention layers, which balances stability and computational efficiency while preserving key information. The action policy head adopts a DiT structure with a hidden dimension of 1536 and 16 Transformer layers, with 32 attention heads per layer and a dimension of 48 for each attention head. A dropout of 0.2 is also used in the attention calculation. In this embodiment, the temporal length (action block length) of action prediction is 16 steps, the state embedding dimension output by the state encoder does not exceed 64, and the action embedding dimension output by the action encoder does not exceed 32. The total number of parameters for the intent bridging interface, the DiT action policy head, and the state encoders and action encoders / decoders dedicated to each ontology is approximately 7.6 × 10⁻⁶. 8 .

[0027] Step 2: Obtain the digital space corpus and the physical space cross-ontology demonstration dataset, and standardize and weight the video and motion data.

[0028] The specific steps are as follows: The multimodal data used in the first phase of training included RoboVQA, AgiBot, HoloAssist, BridgeDataV2, EgoPlan, and ShareRobot; the first four corpora were classified into the understanding and reasoning subsets, while EgoPlan and ShareRobot provided additional complex planning scenarios and multimodal embodied question answering tasks. The robot data used in the second phase of training was collected autonomously in the Maniskill simulation environment. The dataset is in lerobot format and covers four robot bodies: Franka Emika Panda, xArm-6, xArm-7, and WidowX AI. It includes six challenge tasks: PickCube, PullCube, StackCube, PushCube, PlaceSphere, and LiftPegUpright.

[0029] The dataset employs a uniform frame sampling strategy with a sampling interval of 0.5 seconds, and normalizes the number of frames in each video to 4 to 8 frames; a weighted sampling strategy is used during training, starting from the [previous frame]. Datasets Sampling one sample The probability is: in, This indicates the selection of samples under the sampling strategy. The probability of; This represents the total number of datasets. Indicates the first One dataset; This indicates the number of samples in the dataset; Indicates sample From dataset .

[0030] Step 3: Process the input data, generate visual, text and state embeddings, and extract high-level semantic intent through the multimodal large language model and intent bridging interface; The specific steps are as follows: Process video frame sequences using a visual encoder. Visual feature representation is obtained; Use a word segmenter to process natural language instructions. This yields the text feature representation; Visual and textual features are fused using a lightweight multimodal projector and then input into the multimodal large language model; From the first of the multimodal large language models Extract the multimodal hidden state using a Transformer block. ; Using the perceptron module Compress to a fixed quantity The intent of each token is embedded. , as a high-level semantic intent.

[0031] Step 4: In the first training phase, calculate the supervised fine-tuning loss based on the digital spatial corpus. Freeze the visual encoder and word segmenter of the multimodal large language model and optimize only its language model parameters.

[0032] The specific steps are as follows: The supervised fine-tuning loss function is used as the training objective function, specifically: in, This indicates the losses from the first phase of monitoring and fine-tuning; Indicates that it comes from the training corpus One sample, in which Input is a sequence of video frames. Natural language commands The target token sequence; For backbone model parameters The defined conditional probability; As a supervisory mask, used to indicate the first Does each token participate in the loss? This represents the total number of tokens.

[0033] Using the AdamW optimizer, with a peak learning rate of 1e-6, cosine decay scheduling, weight decay of 0.01, and a global batch size of 128, the training took approximately 18,046 steps, updating only the language model parameters of the multimodal large language model.

[0034] Step 5: In the second training phase, calculate the flow matching loss and future prediction loss based on the cross-ontology demonstration dataset. Freeze the backbone network of the entire multimodal large language model and optimize the intent bridging interface, action policy head, and ontology-specific encoder and decoder.

[0035] The specific steps are as follows: Training is performed using a flow matching method with corrected flow paths. Actions are determined using dataset statistics. and The standardization formula is as follows: in, This represents the standardized action vector; Represents the original action vector; This represents the average action value calculated by dimension. This represents the standard deviation of actions calculated by dimension. This indicates element-wise division; for and Perform sampling and define: in, Indicates continuous time The trajectory points of the lower edge correction flow path; This represents a noise vector sampled from a standard normal distribution; The standardized target action vector; For normalized time variables; Indicates the trajectory in time speed; Indicates by parameters The vector field (policy network) parameterizes the velocity; Represents the theoretically optimal vector field ( DiT pairs of vector fields Parameterization is performed, and the loss function is defined as follows: in, This represents the flow matching loss in the second stage; This represents the expectation of the sampled variable; This is the original action vector; Standardized action vectors; Standard normal noise; This represents the high-level semantic intent obtained from the backbone model via the intent bridging interface (the first...). (layers / fragments) This indicates linear interpolation on the corrected flow path; The conditional vector field being trained; This represents the square of the L2 norm.

[0036] The second stage uses the calculation of future prediction loss as an auxiliary objective, and its function is: in, This indicates the auxiliary loss for future predictions; A learnable sequence of future tokens; To extract from the strategic backbone Mapping of latent representations; For noisy action blocks; To map future observations to instructions; Indicates the future number Step observation, Index for the current time, To predict the length of the time domain; Natural language commands; Let be the cosine similarity.

[0037] In the later part of the second phase, the total loss was ,in These are the weighting coefficients.

[0038] Using the AdamW optimizer, with a peak learning rate of 1e-4, cosine decay scheduling, weight decay of 0.01, a global batch size of 128, and training for 200,000 steps, the intent bridge interface, action policy head, state encoder, and action encoder / decoder parameters are updated.

[0039] The above describes the large model method for cross-space, cross-task, and cross-ontology learning in the embodiments of the present invention. The following describes the large model inference method for cross-space, cross-task, and cross-ontology learning in the embodiments of the present invention: Input reception and preprocessing steps: Receive multimodal input containing visual observations and text commands, and perform frame sampling on the video data; Multimodal understanding and intent extraction steps: The input is processed through the multimodal large language model backbone, and a compressed high-level semantic intent is output via the intent bridging interface. ; Conditional action generation steps: based on the aforementioned high-level semantic intent Given the current robot state, the action strategy head performs iterative denoising to generate a smooth, continuous action sequence; this is achieved by solving ordinary differential equations. ,from Points to To obtain the standardized actions after noise reduction ; right Perform de-standardization: This yields the final executable action block. .

[0040] Closed-loop control and output steps: After denormalizing the generated motion sequence, output it to the robot controller to drive the physical body to complete the command task.

[0041] The following two scenario examples are provided to more intuitively demonstrate the effects and advantages of the present invention: First Scenario Case Study: Multi-Benchmark Quantitative Assessment in Digital Space Using the aforementioned approach, the model was systematically evaluated on six digital space benchmarks (RoboVQA, AgiBot, HoloAssist, RoboFail, EgoThink, and ShareRobot). As shown in Table 1, BLM1 (the model designed in this invention) achieved an overall average score of 64.88, significantly outperforming the best-performing closed-source model GPT-4o (59.86) and the best-performing open-source model Cosmos-7B (58.55). Among closed-source multimodal large language models, GPT-4o performs best on the EgoThink benchmark (72.42), demonstrating its advantage in first-person reasoning tasks. However, BLM1 maintains its lead on all other benchmarks, achieving improvements of 20.00%, 9.00%, and 8.00% on RoboVQA, AgiBot, and RoboFail, respectively. This indicates that while GPT-4o has advantages in specific reasoning scenarios, BLM1 exhibits stronger robustness in high-level planning, affordance reasoning, task validation, action generation, and cross-scene and cross-entity generalization. In comparison with open-source multimodal models, BLM1 improves performance by 7.37% compared to Qwen2.5-VL-7B-Instruct and by 6.33% compared to Cosmos-7B. It also leads HoloAssist and ShareRobot by 11.00% and 18.73% respectively, demonstrating its significant advantages in embodied multimodal reasoning, first-person understanding, and fine-grained action generation. Compared with the embodied large language model (ELLM) VeBrain-7B, BLM1 improves performance by 10.00%, 12.00% and 14.12% on RoboVQA, HoloAssist and ShareRobot respectively, further validating its superior ability in task transfer and cross-scenario generalization; Compared to the General Multimodal Large Model (GMLM) ChatVLA-2B and RoboBrain2-7B, BLM1 still leads, with an average improvement of over 11.00%, and the improvement is particularly significant in RoboVQA and ShareRobot, further validating its efficiency in embodied multimodal reasoning tasks. Overall, BLM1 outperforms all four architectures—closed-source and open-source MLLM, ELLM, and GMLM—with an average performance improvement exceeding 11.00%, particularly in the RoboVQA and ShareRobot tasks. These results demonstrate that BLM1 possesses a balanced design philosophy and excellent cross-task generalization capabilities, enabling stable and superior performance across various digital benchmarks.

[0042] Table 1: Comparison of the model with existing multimodal large language models, embodied large language models, and general multimodal large models on digital space benchmarks. Table 2: Comparison of the model with existing VLAs on physical space benchmarks Where † indicates that the model was trained independently on four different robots and evaluated on six tasks; This means that the model was trained independently on six tasks across four different robots, and evaluated on the corresponding tasks for each robot platform. Second scenario case: Quantitative evaluation of cross-ontology tasks in physical space Using the aforementioned approach, the model was systematically evaluated on four robot bodies (Franka Emika Panda, xArm-6, xArm-7, and WidowX AI) and six complex tasks (PickCube, PullCube, StackCube, PushCube, PlaceSphere, and LiftPegUpright). As shown in Table 2, BLM1 achieved an average task success rate of 75.83%, comprehensively outperforming all policy models trained from scratch and surpassing some pre-trained vision-language-action (VLA) models, fully demonstrating its cross-ontology generalization ability and overall stability under multi-task conditions. At the embodiment level, BLM1 consistently outperformed both pre-trained and zero-trained models across all four ontology levels. Compared to the pre-trained baseline, the overall performance of the GR00T series remained lower than BLM1, with the π0 model achieving only a 67.42% success rate, indicating that relying solely on large-scale semantic priors and streaming matching mechanisms cannot achieve robust cross-ontology generalization. While HPT and UniAct aimed for generalization, they only achieved success rates of 55.00% and 48.67% on Panda and xArm-6 platforms, respectively, while BLM1 surpassed them by 29.67% and 31.66%, demonstrating a significant advantage. BLM1 also performed better overall than the zero-trained Diffusion Policy; although Diffusion Policy had a slight advantage on xArm-7, BLM1 achieved higher success rates on Panda, xArm-6, and WidowX AI. Overall results show that BLM1 can maintain consistent performance under multi-aspect conditions with significant differences in the number of joints, workspace, and visual perception, demonstrating excellent cross-aspect generalization ability. In task-level analysis, BLM1 also demonstrated high task robustness and generalization ability. As shown in Table 2, in relatively simple tasks (PickCube, PushCube, and PullCube), the average success rate reached 83.5%, indicating its high stability in basic action prediction, perception, and execution. In more challenging tasks (PlaceSphere and LiftPegUpright), BLM1 still maintained an average success rate of 60.5%, outperforming all models trained from scratch and surpassing most pre-trained models. Its overall performance was comparable to GR00T-N1.5, demonstrating stable performance under increased task complexity.

[0043] In summary, BLM1 demonstrates significant advantages at both the embodiment and task levels. It consistently outperforms mainstream models across four different ontologies, showcasing strong cross-ontology generalization capabilities, while maintaining high success rates and stability across tasks of varying difficulty. These results validate the effectiveness of BLM1 as a unified policy model, enabling robust and superior performance under diverse task and ontology conditions.

[0044] This invention provides an electronic device including at least one processor (CPU) and a memory. The memory stores computer programs, and the processor executes instructions stored therein to implement the large-scale model and inference method for cross-space, cross-task, and cross-ontology learning described in this invention. The electronic device may also include a power supply, a network interface, input / output interfaces, and an operating system (such as Windows, Mac OS, Linux, etc.). The storage medium can be temporary or persistent storage, and program modules can execute specific instruction operations within it. Furthermore, the storage medium includes non-volatile or volatile computer-readable media, such as USB flash drives, hard disks, ROM, RAM, etc., to ensure the reliability of instruction execution.

[0045] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0046] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0047] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large model based on cross-space, cross-task and cross-ontology learning, characterized in that, The large model comprises: a pre-trained multi-modal large language model as a backbone network for processing visual and textual inputs and generating responses; an intent bridging interface for compressing hidden states output by the multi-modal large language model into fixed-length semantic intent representations; an action policy head for generating continuous action sequences based on the semantic intents; a state encoder for encoding the proprioceptive state of the robot; an action encoder / decoder for embedding and reconstructing actions.

2. The large model based on cross-space, cross-task and cross-ontology learning according to claim 1, wherein, The intent bridging interface is configured to perform the following steps: Processing a sequence of video frames using a visual encoder to obtain a visual feature representation; Processing natural language instructions using a tokenizer to obtain a textual feature representation; fusing visual and textual features through a multi-modal projector and inputting them into the multi-modal large language model; extracting a multi-modal hidden state from a first Transformer block of the multi-modal large language model ​ using the perceiver module to compress as a fixed number of token intent embeddings as high-level semantic intents.

3. The large model based on cross-space, cross-task and cross-ontology learning according to claim 1, wherein, The action policy head is configured to perform the following steps: with the high-level semantic intent and current robot state as conditions, iteratively denoise through the action policy head to generate smooth continuous action sequences.

4. The large model based on cross-space, cross-task and cross-ontology learning according to claim 1, wherein, The action sequence is obtained by the following steps: By solving ordinary differential equations , from integral to , get the normalized action after denoising ; To de-standardize: , resulting in the final action sequence .

5. A large model inference method based on cross-space, cross-task and cross-ontology learning, characterized in that, comprising the following steps: input receiving and preprocessing steps: receiving multi-modal inputs containing visual observations and textual instructions, and frame sampling on video data; Multimodal understanding and intent extraction step: processing the input by the multimodal large language model backbone and outputting compressed high-level semantic intent via the intent bridging interface ; Conditional action generation step: with the high-level semantic intent and the current robot state as conditions, iterate through the action policy head to denoise and generate a smooth continuous action sequence; closed-loop control and output steps: normalizing the generated action sequence and outputting it to a robot controller to drive the physical embodiment to complete the instruction task.

6. The inference method of claim 5, wherein, The conditional action generation step specifically comprises: By solving ordinary differential equations , from integral to , get the normalized action after denoising ; To de-standardize: , resulting in the final executable action block .

7. An electronic device comprising at least one processor and a memory, characterized in that, The memory stores computer instructions, and the at least one processor executes the instructions to implement the steps of the method of any one of claims 1-4.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps of the method of any one of claims 1-4.

Images