Multimodal body model updating method, system, device and storage medium

Abstract

This invention discloses a method, system, device, and storage medium for updating a multimodal embodied model. These are corresponding solutions. In these solutions: on the device side, interactive events are aggregated into spatial interaction prototypes and privacy protection processing is performed, limiting the information visible to the cloud; furthermore, the device side does not undertake annotation tasks, but the cloud completes the supervised completion and training, thereby avoiding the problems of unstable local training targets and excessive supervision noise on the device side; simultaneously, the local memory index of the device side retains user-private, short-cycle, and strongly scene-dependent knowledge information, and the spatial interaction prototypes are uploaded to the cloud. This avoids directly writing user-private facts into the cloud model and can utilize diverse data from real deployments to improve the inference capabilities of the cloud model; in addition, the device side uploads the spatial interaction prototypes, and the cloud distributes local update parameters, thereby reducing uplink bandwidth, downlink bandwidth, device-side storage, and version management overhead.

Description

Technical Field

[0001] This invention relates to the field of multimodal embodied model updating technology, and in particular to a multimodal embodied model updating method, system, device and storage medium. Background Technology

[0002] Multimodal embodied models are artificial intelligence models that deeply integrate multiple perceptual modalities such as vision, language, and touch with physical entities such as robots.

[0003] Currently, multimodal embodied model update technologies can be mainly categorized as follows.

[0004] The first category includes traditional federated learning and privacy-preserving federated learning. This type of technology typically allows multiple edge devices to train locally without uploading the original data, and then upload the model gradients, model parameters, statistics, or distilled signals to the server for aggregation and updates. Its advantage is that the original data does not directly leave the edge devices, but it usually assumes that the edge devices can form a stable training objective based on the local data and complete local training. Furthermore, existing research has shown that gradients, model updates, or intermediate representations may still leak sample information; therefore, simply ensuring "data does not leave the edge" is not equivalent to strong privacy protection.

[0005] The second category is knowledge distillation and data free distillation. This type of technique trains student models using teacher model output, generators, or aggregated knowledge to reduce reliance on the original training samples. It can alleviate the performance degradation caused by directly averaging model parameters in heterogeneous federated learning. However, existing methods are mostly geared towards classification models, language models, or general vision models, and have not yet developed structured empirical representations for spatial states, action execution, multimodal feedback, and task results in embodied systems.

[0006] The third category is spatial representation and 3D semantic mapping technologies. Technologies such as NeRF (Neural Radiance Fields), 3D Gaussian Splatting, Neural Implicit SLAM (Simultaneous Localization and Mapping), object-level mapping, open-vocabulary 3D semantic fields, and 3D scene graphs can represent environmental geometry, objects, regions, semantics, and relationships. These technologies are suitable for building edge-side scene memory and open-vocabulary query systems. However, directly uploading high-fidelity 3D geometry, semantic tags, trajectories, or scene graphs may still expose private information such as home layout, item distribution, and user habits.

[0007] The fourth category includes embodied multimodal models and visual-language-action models. These two types of models utilize internet data, robot data, language commands, visual observations, and motion trajectories for joint training to improve the robot's generalization ability in multiple tasks and environments. This illustrates the significant value of real-world robot deployment data in enhancing embodied intelligence models. However, current technologies typically rely on directly accessible or centrally processed robot training data, failing to adequately address the issue of the inability to upload raw scene data during large-scale terminal device deployments.

[0008] The fifth category includes cloud-based robot learning, semantic map crowdsourcing, and digital twin / synthetic data training technologies. Existing solutions involve cloud-edge-device robot learning, dynamic robot learning, edge-cloud collaborative image processing, crowdsourced semantic map updates, synthetic data generation, and simulation training. These solutions can supplement cloud capabilities or expand training data, but they often focus on skill sets, map updates, image processing, simulation training, or general privacy training, and have not formed a complete technical chain of "edge-side weakly supervised spatial interaction prototyping—cloud-based embodied experience distillation—small parameter update distribution."

[0009] The defects of the existing technology can be described as follows: (1) Traditional federated learning requires that the edge devices (i.e., multimodal embodied models) be trainable, but the edge devices often struggle to generate reliable supervision signals. Traditional federated learning typically assumes that edge devices can complete local training based on local data and upload gradients or model updates. However, edge devices such as home robots and mobile robots often collect continuous, multimodal, and strongly temporally coupled raw interaction data, including images, depth, laser, force sensing, motion trajectories, execution states, and task results. Although execution logs and sensor changes can be recorded, it is usually difficult to automatically complete high-quality annotations, such as determining whether the failure is due to visual recognition errors, language understanding errors, motion planning errors, or physical contact anomalies; it is also difficult to determine whether there is a transferable causal relationship between actions and state changes. If local training is performed directly, it is easy to cause large noise in the supervision signal, unstable local update direction, severe non-independent identically distributed offset, and decreased global aggregation performance.

[0010] (2) Directly uploading gradients, model parameters, or ordinary features still poses a risk of privacy leakage. Although existing federated learning does not upload raw data, gradients and intermediate representations may leak training samples or scene information. In embodied intelligence scenarios, spatial layout, object categories, user activity trajectories, and distribution of household items all have strong privacy attributes. Directly uploading model updates, visual features, point cloud features, or open-vocabulary semantic labels may still lead to scene reconstruction or sensitive semantic inference.

[0011] (3) Static semantic maps or 3D maps cannot fully capture embodied interaction experiences. Existing semantic maps, 3D scene maps, and neural fields mainly describe "what is in the space", "what are the relationships between objects", and "what is the scene geometry like", but multimodal embodied models need to learn "what sensory feedback and task results will be generated when performing a certain type of action in a certain type of spatial state". Uploading only static maps cannot fully express changes in action condition states, contact feedback, collision feedback, failure reasons, user corrections, and physical causal candidate relationships.

[0012] (4) Pure local knowledge bases cannot improve cross-device generalization capabilities. On-device local knowledge bases are suitable for storing private facts in a certain user scenario, such as the location of a cup, the layout of a piece of furniture, or the common state of a room. However, pure local knowledge bases cannot transform real interaction experiences under different families, different robots, and different sensor configurations into the general spatial-physical-action reasoning capabilities of cloud-based multimodal embodied models.

[0013] (5) Existing cloud-based training or digital twin methods may rely on reconstructable scene data. Uploading images, videos, point clouds, continuous trajectories, or complete 3D assets for cloud training may directly expose the user's private environment. Relying entirely on publicly available simulation or synthetic data makes it difficult to cover long-tail objects, realistic touch feedback, and complex user behaviors in real deployments.

[0014] Therefore, a new intermediate representation and training link is needed to transform real-world interaction experience on the device into a cloud-based training signal that is learnable, aggregable, low-bandwidth, and low-inversion-risk, without uploading the original scene data or requiring the device to complete full annotation training.

[0015] In view of this, the present invention is hereby proposed. Summary of the Invention

[0016] The purpose of this invention is to provide a method, system, device, and storage medium for updating multimodal embodied models. This method can update multimodal embodied models in the cloud with privacy protection, without uploading original images, videos, point clouds, continuous trajectories, or original multi-sensor time-series data, by utilizing spatial interaction experience formed in real deployments. It can be applied to home service robots, mobile operation robots, warehouse and inspection robots, delivery robots, intelligent companion devices, edge devices with vision and motion execution capabilities, and other embodied intelligent systems that need to protect user privacy on the edge while continuously improving general spatial-physical-motion reasoning capabilities through the cloud.

[0017] The objective of this invention is achieved through the following technical solution: A method for updating a multimodal embodied model includes: The edge device collects multimodal observation data and generates an edge spatial representation. Based on the edge spatial representation, the local multimodal embodied model combines local memory indexes to make decision-making actions, which are then executed by the actuator. The spatial interaction events are constructed using the multimodal observation data or its spatial state representation before the action is executed, the action information, and the multimodal observation data or state change information after the action is executed. Multiple spatial interaction events are integrated to form a spatial interaction prototype, which is then uploaded to the cloud server after privacy protection processing. The cloud server receives privacy-preserving spatial interaction prototypes uploaded by multiple edge devices and generates a cross-device aggregated prototype. Based on the cross-device aggregated prototype, the spatial interaction prototypes of the edge devices are supervised and completed to obtain completion supervision signals. The cross-device aggregated prototype and completion supervision signals are used to perform distillation training and action condition state change prediction training on the target multimodal embodied model deployed on the cloud server, or, may also include contrastive learning and / or failure attribution training. Based on the training results, local update parameters are generated and published preprocessing is performed to obtain the governed local update parameters and distribute them to the edge devices. The edge device updates the locally deployed multimodal embodied model using the local update parameters after governance.

[0018] A multimodal embodied model update system for implementing the aforementioned method includes: an edge device and a cloud server; wherein: End-side equipment includes: Multimodal sensor interface, used to acquire multimodal observation data; Spatial representation engine, used to generate edge spatial representations; Multimodal embodied models are used to combine local memory indexes with edge spatial representations to make decisions on actions; An actuator is used to perform actions; A spatial interactive event encoder is used to construct spatial interactive events using multimodal observation data or its spatial state representation before the action is executed, action information, and multimodal observation data or state change information after the action is executed. The spatial interaction prototype generation module is used to integrate multiple spatial interaction events to form a spatial interaction prototype. A prototype-level privacy processing module is used to process the spatial interaction prototype for privacy protection before uploading it to the cloud server; The local small parameter update module is used to update the locally deployed multimodal embodied model using the managed local update parameters; Cloud servers include: The security aggregation gateway is used to receive privacy-preserving space interaction prototypes uploaded by multiple end devices and generate cross-device aggregation prototypes. The cloud-based supervised completion module is used to supervise and complete the spatial interaction prototype of the end device based on the cross-device aggregated prototype, and obtain the completion supervision signal; The learning engine is used to perform distillation training and action condition state change prediction training on a target multimodal embodied model deployed on a cloud server by leveraging cross-device aggregated prototypes and completion supervision signals, or it may also include contrastive learning and / or failure attribution training. A parameter update manager that generates locally updated parameters based on training results; The preprocessing module is used to preprocess the local update parameters before publishing, obtain the remediated local update parameters, and send them to the edge devices.

[0019] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0020] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0021] As can be seen from the technical solutions provided by the present invention above: (1) Reduce the risk of leakage of original scene data: The present invention does not upload original images, videos, point clouds, continuous trajectories or original sensor time series data, but aggregates multiple interactive events into a spatial interaction prototype on the edge and performs privacy protection processing, limiting the information visible in the cloud, thus reducing the risk of sample inversion, scene reconstruction and semantic leakage. (2) Reduce the dependence on edge labeling and edge training capabilities: The present invention enables the edge to not undertake labeling tasks, but the cloud to complete the supervision completion and training, thereby avoiding the problems of unstable local training targets and excessive supervision noise on the edge. (3) Take into account both local personalization and cloud general capabilities: The local memory index of the edge device retains user private, short-cycle, strong scene-dependent knowledge and other information; the cross-device transferable space-action-feedback rules are uploaded to the cloud in the form of spatial interaction prototypes, so that user private facts are not directly written into the cloud model (multimodal embodied model deployed on the cloud server), and the general spatial physical reasoning capability of the cloud model can be improved by using the diverse data in the real deployment. (4) Reduce communication and update costs: The terminal device uploads the processed spatial interaction prototype, and the cloud sends out the local update parameters, thus reducing the overhead of uplink bandwidth, downlink bandwidth, terminal storage and version management. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the framework of a multimodal embodied model update method provided in an embodiment of the present invention.

[0024] Figure 2 This is a flowchart of a multimodal embodied model update method provided in an embodiment of the present invention.

[0025] Figure 3 This is a schematic diagram illustrating the spatial interaction prototype generation and privacy processing process provided in an embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of a cloud-based training target provided in an embodiment of the present invention.

[0027] Figure 5 This is a timing diagram of privacy processing and local update parameter distribution provided in an embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0030] First, the following explanations are provided for the terms that may be used in this article: The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0031] The terms “comprising,” “including,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “comprising a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0032] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0033] The following provides a detailed description of a multimodal embodied model update method, system, device, and storage medium provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Where the manufacturers of reagents or instruments used in the embodiments of this invention are not specified, they are all conventional products that can be purchased commercially.

[0034] Example 1 This invention provides a method for updating a multimodal embodied model, which mainly includes: (1) The edge device collects multimodal observation data and generates edge spatial representation. Based on the edge spatial representation, the local memory index is combined with the locally deployed multimodal embodied model to make decision on actions, and then the action is executed by the actuator. The multimodal observation data or its spatial state representation and action information before the action is executed, as well as the multimodal observation data or state change information after the action is executed, constitute spatial interaction events. Multiple spatial interaction events are integrated to form a spatial interaction prototype, and then uploaded to the cloud server after privacy protection processing.

[0035] In this embodiment of the invention, the multimodal observation data includes any one or a combination of visual observation, depth observation, laser ranging information, point cloud information, force information, tactile information, inertial measurement information, joint status, wheel speed information, action execution status, and user interaction signals.

[0036] In this embodiment of the invention, the generation of end-side spatial representation includes: generating scene-level, object-level, region-level, or relation-level spatial representations based on the multimodal observation data.

[0037] In this embodiment of the invention, the step of integrating multiple spatial interaction events to form a spatial interaction prototype, and then processing it for privacy protection, includes: (1.1) For multiple spatial interaction events under a time window, a task segment, a spatial region, or an action category, use any one or any combination of clustering, pooling, graph aggregation, vector quantization, and statistical summarization to process them and generate spatial interaction prototypes.

[0038] (1.2) Privacy protection processing is performed using the following formula: ; in, This represents the k-th spatial interaction prototype; Let R represent the k-th privacy-preserving space interaction prototype, and let R represent random projection, subspace pruning, feature selection, or hash mapping function. Represents the quantization function; This indicates privacy noise.

[0039] (2) The cloud server receives privacy-preserving spatial interaction prototypes uploaded by multiple end devices and generates cross-device aggregated prototypes; based on the cross-device aggregated prototypes, it performs supervised completion on the spatial interaction prototypes of the end devices to obtain completion supervision signals; it uses the cross-device aggregated prototypes and completion supervision signals to perform distillation training and action condition state change prediction training on the target multimodal embodied model deployed on the cloud server, or it may also include contrastive learning and / or failure attribution training; based on the training results, it generates local update parameters and performs release preprocessing to obtain the governed local update parameters and sends them to the end devices.

[0040] In this embodiment of the invention, the release preprocessing mainly includes: version signing, compatibility checking, audit logging, canary release, and rollback of exceptions.

[0041] In this embodiment of the invention, supervised completion of spatial interaction prototypes on end devices based on cross-device aggregated prototypes includes: clustering the cross-device aggregated prototypes, obtaining statistical information for each prototype cluster, and performing supervised completion using a multimodal embodied teacher model deployed on a cloud server in conjunction with the statistical information. This includes: distinguishing between local private facts and cross-scenario transferable rules based on the statistical information of the prototype clusters; determining the causal correlation between actions and state changes based on action information and spatial state change information after action execution; generating soft labels for action conditional state changes based on action information, spatial state representation before action execution, and spatial state change information after action execution, wherein the spatial state representation is obtained by mapping the corresponding multimodal observation data to the latent space; when the spatial interaction event also includes multimodal feedback information, task result information, and user correction information, estimating the failure cause category by combining the multimodal feedback information, task result information, and spatial state change information after action execution; and performing task-level attribution on user correction signals by combining user correction information, action information, spatial state change information after action execution, and task result information. The obtained information is collectively referred to as the completion supervision signal.

[0042] In this embodiment of the invention, the cluster management of cross-device aggregated prototypes includes: clustering the cross-device aggregated prototypes according to action type, coarse-grained scene category, object or region category, multimodal feedback type, failure type, device type, or time decay weight.

[0043] In this embodiment of the invention, when using distillation training, contrastive learning, action condition state change prediction training, and failure attribution training, the corresponding loss is calculated, and a training loss is constructed by combining regularization constraints. The training loss is then used to optimize the parameters of the target multimodal embodied model deployed on the cloud server. Specifically, distillation training mainly utilizes action condition state change soft labels, task-level attribution results, or other supplementary supervision signals generated by the cloud-based multimodal embodied teacher model; contrastive learning is responsible for learning the correspondence between the spatial state before action execution, action information, spatial state change information after action execution, and multimodal feedback information in the cross-device aggregated prototype, and uses causal correlation and local private facts and cross-scene transferable rules for supervision; action condition state change prediction training mainly utilizes the spatial state representation before action execution and action information to predict the spatial state change information after action execution, and uses action condition state change soft labels for supervision; failure attribution training mainly utilizes multimodal feedback information, task result information, user correction information, and spatial state change information after action execution to estimate the failure cause category, and uses the failure cause category for supervision.

[0044] The training loss is represented as: ; Where L is the training loss, This represents the loss from knowledge distillation. The contrastive learning loss represents the spatial state, action, and feedback. Indicates the loss predicted by changes in action condition state; Indicates failure attribution to loss; This indicates a privacy regularity or prototype reversibility penalty. This indicates version stability, catastrophic forgetting suppression, or cross-device fairness constraints. and Form regularization constraint terms; arrive The weights for each loss item.

[0045] (3) The end-side device updates the locally deployed multimodal embodied model using the local update parameters after the treatment.

[0046] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0047] I. Overall Overview of the Method.

[0048] This invention provides a method for updating a multimodal embodied model. This method uses a spatial interaction prototype as the intermediate data unit for edge-cloud collaborative embodied model updating, and constructs a complete technical chain around this data unit, including edge prototype generation, privacy processing, cloud-supervised completion, distillation training, and small parameter distribution. Based on this method, real-world edge interaction experience can be transformed into a learnable, aggregable, low-bandwidth, and low-inversion-risk training signal in the cloud without uploading original scene data or requiring complete annotation training on the edge.

[0049] The core technical points of the above method can be summarized as follows: 1. Define and use the spatial interaction prototype as the data unit for updating data between the end and the cloud.

[0050] The spatial interaction prototype defined in this invention differs from static semantic maps, single-frame visual features, complete point clouds, model gradients, or ordinary logs. It is formed through one or more of the aforementioned spatial interaction events, clustered (pooling or compression, etc.), and uploaded to a cloud server after privacy protection processing. This data unit can both carry transferable experiences from real embodied interactions and reduce the risk of directly exposing the real-world scenarios of individual users.

[0051] In this embodiment of the invention, a spatial interaction event refers to an atomic empirical unit that performs an action and generates sensor feedback and task results when the end device is in a certain spatial state.

[0052] A spatial interaction prototype refers to an empirical summary representation formed by processing one or more spatial interaction events generated by an edge device performing tasks in a real environment, through clustering, pooling, or compression. The spatial interaction prototype can be represented using vectors, tensors, graph structures, codebook indexes, hash buckets, cluster centers, probability distribution parameters, low-rank representations, or combinations thereof.

[0053] In this embodiment of the invention, spatial interaction events include at least multimodal observation data or its spatial state representation before action execution, action information, and multimodal observation data or state change information after action execution; preferably, they further include any one or a combination of multimodal feedback information, task result information, user correction information, reward information, failure markers, and safety trigger information. A spatial interaction prototype is formed by aggregating one or more of the above-mentioned spatial interaction events. The feedback information includes one or more of the following: force information, tactile information, laser information, depth information, collision, user correction, and task success or failure. This spatial interaction prototype differs from static semantic maps, single-frame visual features, complete 3D point clouds, or federated learning gradients. It is used to characterize the statistical correlation or causal candidate relationship between spatial state, action execution, multimodal feedback, and task result, and to support distillation training, contrastive learning, causal state change prediction, or efficient parameter updating of cloud-based models without uploading original images, videos, point clouds, continuous trajectories, or original sensor time-series data.

[0054] 2. A mechanism for prototyping with weak supervision on the edge and completing with strong supervision in the cloud.

[0055] This invention does not require edge devices to fully annotate the original multimodal data, nor does it require edge devices to complete local model training based on the original data. The edge device only performs lightweight processing (such as building spatial interaction prototypes and privacy protection processing); the cloud, on the other hand, utilizes a stronger multimodal embodied model, cross-device statistics, and prototype aggregation results to perform supervised completion and distillation learning on the action results, failure reasons, and causal relationships of state changes in the prototype. This mechanism solves the problem that traditional federated learning struggles to generate reliable supervision signals and stable local training targets on the edge device in embodied intelligence scenarios.

[0056] 3. Decoupling local scene fact memory from cloud-based general capability updates.

[0057] This invention stores short-cycle, highly privacy-conscious, and scenario-dependent local facts in an edge-side memory index. Specifically, the edge-side memory index refers to a searchable storage structure that stores scenario facts, object states, region states, user preferences, or short-term change information locally on the edge device, for online retrieval and real-time task execution. Cross-device transferable space-action-feedback / result patterns are uploaded to the cloud via a spatial interaction prototype to update the multimodal embodied model deployed on the edge device. This avoids writing high-frequency private facts into the target multimodal embodied model while ensuring that real-world deployment data continues to contribute to the improvement of general capabilities.

[0058] 4. Prototype-level privacy processing chain.

[0059] This invention performs privacy-preserving processing on the spatial interaction prototype layer, such as clustering, random projection, subspace pruning, low-bit quantization, differential privacy noise injection, secure aggregation, version signing, and audit rollback. Compared to directly uploading raw visual data, point clouds, trajectories, model gradients, or high-fidelity features, it can reduce the risks of sample inversion, scene reconstruction, and semantic leakage.

[0060] 5. Cloud-based training objectives oriented towards causal relationships in space physics.

[0061] The cloud-based training objective of this invention includes not only knowledge distillation loss, but also action conditional state change prediction loss, multimodal feedback alignment loss, failure cause classification or ranking loss, cross-device comparative learning loss, and privacy regularization term. This training objective enables cloud models to learn the patterns between space-action-feedback / outcome rules from real-world deployment experience.

[0062] 6. Only minor parameter updates will be issued.

[0063] This invention preferably updates local parameters, such as those of the visual projector, cross-modal bridging layer, action head, adapter, prompt, prefix, LoRA (low-rank adaptive), or other parameters for efficient fine-tuning modules, rather than frequently distributing all model parameters. This approach reduces cloud training costs, on-device download costs, and the risk of version rollback.

[0064] II. Detailed introduction of the method.

[0065] The method described above in this invention mainly involves two parts: an edge device and a cloud server. The edge device can be a home robot, a mobile robot, an operating robot, an inspection robot, or other embodied terminal; the cloud server can be a regular cloud server, a private cloud server, a trusted execution environment, a verifiable execution environment, or other secure computing platform.

[0066] 1. Framework introduction.

[0067] like Figure 2 As shown, the edge device mainly includes: a multimodal sensor interface 110, a spatial representation engine 120, a spatial interaction event encoder 130, a local memory index 140, a spatial interaction prototype generation module 150, a prototype-level privacy processing module 160, a local small parameter update module 170, a multimodal embodied model 180, and an actuator 190.

[0068] The multimodal sensor interface 110 is used to collect multimodal observation data such as visual information, depth information, laser information, force information, tactile information, inertial measurement information, joint status, wheel speed information, and action execution status of the end-side embodied device; the spatial representation engine 120 is used to generate scene-level, object-level, region-level, or relation-level spatial representations based on the multimodal observation data; the spatial interaction event encoder 130 is used to record multimodal observation data before action execution, action information, multimodal observation data after action execution, multimodal feedback information, and task result information during the execution of actions by the end-side device, forming spatial interaction events; the local memory index 140 is used to store user-private scene facts, object status, region status, or short-term change information; the spatial interaction prototype generation module 150 is used to aggregate, pool, or encode multiple spatial interaction events into a spatial interaction prototype; the prototype-level privacy processing module 160 is used to perform privacy protection processing on the spatial interaction prototype; and the local small parameter update module 170 is used to receive and load local update parameters sent from the cloud, so that the locally deployed multimodal embodied model 180 can call or take effect in subsequent embodied tasks.

[0069] The cloud server mainly includes: a security aggregation gateway 210, a prototype cluster manager 220, a cloud-based supervised completion module 230, a learning engine, an optional synthetic view or simulation enhancement module 250, a parameter efficient update manager 260, and a release preprocessing module. Since this invention mainly involves distillation, comparison, and causal learning, the learning engine is a distillation / comparison / causal learning engine 240. The release preprocessing mainly involves version signing, auditing, and rollback, therefore, the release preprocessing module is a version signing, auditing, and rollback module 270.

[0070] The secure aggregation gateway 210 is used to receive privacy-preserving spatial interaction prototypes or their secure aggregation shares uploaded by multiple end-device devices, and form cross-device aggregated prototypes; the prototype cluster manager 220 is used to perform cluster management of cross-device aggregated prototypes according to action type, scene category, object or region category, multimodal feedback type, failure type, device type, or time decay weight; the cloud-supervised completion module 230 is used to generate soft labels, failure attributions, action condition state change labels, or causal candidate relationships based on the aggregated spatial interaction prototypes and the multimodal embodied teacher model deployed on the cloud server; the distillation / comparison / causal learning engine 240 is used to use the cross-device aggregated prototypes and completion supervision signals to train the target multimodal embodied model deployed on the cloud. The system performs distillation training, contrastive learning, action condition state change prediction, or failure attribution training. Optional synthetic view or simulation enhancement modules 250 are used to generate synthetic views, simulation tasks, or enhanced training samples that do not correspond to a single user scenario based on aggregated prototype clusters, which are then used for distillation training, contrastive learning, action condition state change prediction training, and failure attribution training. A parameter efficiency update manager 260 is used to generate parameters for the visual projector, cross-modal bridging layer, action head, adapter, prompt, prefix, LoRA, or other parameter efficiency fine-tuning modules. A version signing, auditing, and rollback module 270 is used to perform version signing, compatibility checks, audit logging, canary deployment, and abnormal rollback on locally updated parameters.

[0071] In this embodiment of the invention, a multimodal embodied teacher model and a target multimodal embodied model are deployed on a cloud server. The former is used for supervised completion, and the latter is the model that is trained and updated. In some implementations, the two can be the same model or share some parameters.

[0072] 2. Process Introduction.

[0073] The method flow of this invention includes an edge processing stage, a cloud processing stage, and an update distribution stage. The edge processing stage is used to collect multimodal observations, construct spatial representations, record spatial interaction events, generate spatial interaction prototypes, and perform privacy processing. The cloud processing stage is used to securely aggregate the privacy-protected spatial interaction prototypes, manage prototype clusters, perform supervised completion, and conduct distillation / comparison / causal training. The update distribution stage is used to generate efficient update parameters and distribute them to the edge device after preprocessing. Figure 2 The diagram illustrates the main flow of the above method.

[0074] Step S101: Multimodal observation and acquisition.

[0075] The end-side device acquires multimodal observation data through the multimodal sensor interface 110. The multimodal observation data includes one or more of the following information: visual observation, depth observation, laser ranging information, point cloud information, force information, tactile information, inertial measurement information, joint status, wheel speed information, motion execution status, and user interaction signals.

[0076] This step processes the raw multimodal observations obtained by the edge device in a real deployment environment. The result of this step is a set of time-synchronized or nearly synchronized observation sequences, which are used for subsequent spatial characterization construction.

[0077] Step S102: Spatial representation construction.

[0078] The spatial representation engine 120 generates an edge-side spatial representation based on the multimodal observation data obtained in step S101. The spatial representation can be scene-level, object-level, region-level, or relation-level; in form, it can be: scene-level latent vectors, object-level latent vectors, region-level latent vectors, object relation graphs, region relation graphs, traversable region graphs, open-vocabulary semantic vectors, or temporal state vectors. The spatial representation can be generated from SLAM, neural implicit fields, 3D Gaussian representation, voxel maps, object-level maps, semantic maps, 3D scene graphs, open-vocabulary 3D maps, or combinations thereof.

[0079] The result of this step is a spatial state representation that can be called before and after the action is executed, which is used for subsequent spatial interaction event recording.

[0080] Step S103: Record spatial interaction events.

[0081] The multimodal embodied model 180 combines the local memory index 140 and the output of the spatial representation engine 120 to make action decisions, which are then executed by the actuator. The spatial interaction event encoder 130 records multimodal observation data before action execution, action information, multimodal observation data after action execution, multimodal feedback information, and task result information. Spatial interaction events can be constructed as needed. For example, spatial interaction events can be constructed using multimodal observation data before action execution or its spatial state representation, action information, and multimodal observation data or state change information after action execution. The spatial state representation can be obtained by mapping the corresponding multimodal observation data to the latent space. The latent space is a common term in this field and is the internal space of the encoder.

[0082] In addition, it may further include: multimodal feedback information, reward information, failure markers, task result information, user correction information, safety trigger information (such as collision warning, force exceeding threshold, emergency stop, entering a restricted area, fall risk or system safety policy triggering events, etc.), and any one or more combinations of optional metadata.

[0083] Taking all the above information as an example, a single spatial interaction event can be represented as: ; in, Indicates the first A spatial interaction event; The spatial state latent vector represents the multimodal observation data before the action is executed (i.e., the spatial state representation before the action is executed). Indicates action coding; This represents the spatial state latent vector of the multimodal observation data after the action is executed (i.e., the spatial state representation after the action is executed). This indicates information about changes in spatial state after an action is performed. Represents the multimodal feedback vector; It can represent one or more of the following: task result information, reward information, failure flag, security trigger information, and user correction information. This indicates optional metadata, including device type, sensor configuration, coarse-grained scene category, timestamp, or time decay weight.

[0084] Information on spatial state changes after action execution It can be calculated using the following formula: .

[0085] In another implementation, It can also be generated by an independent state change encoder, and is not limited to the vector difference form.

[0086] The technical role of this step is to convert the raw multimodal data from the edge into structured empirical units in the form of "space-action-feedback / result rules", providing input for subsequent prototyping.

[0087] Step S104: Generation of spatial interaction prototype.

[0088] The spatial interaction prototype generation module 150 performs clustering, pooling, graph aggregation, vector quantization, statistical summarization, or any combination thereof on multiple spatial interaction events under a time window, a task segment, a spatial region, or an action category to generate a spatial interaction prototype.

[0089] The k-th spatial interaction prototype can be represented as: ; in, This represents the k-th spatial interaction prototype; A set of indices representing spatial interaction events that are grouped into the same prototype cluster; Indicates the first Spatial Interaction Events The index i belongs to the index set Proto(⋅) represents the prototype generation function.

[0090] The prototype generation function can be K-means cluster center calculation, vector quantization codebook mapping, attention pooling, graph neural network pooling, statistical moment estimation, low-rank projection, hash bucket mapping, or a combination thereof. Preferably, the edge device generates 16 to 512 spatial interaction prototypes per round, more preferably 32 to 128 spatial interaction prototypes. The dimension of each spatial interaction prototype is preferably 256 to 4096, more preferably 512 to 1536.

[0091] The technical role of this step is to compress a large amount of frame-by-frame, multimodal, and strongly temporally coupled end-side interaction data into a smaller amount of semantically relevant experience summaries that can be used for cloud-based aggregation and training.

[0092] Step S105: Prototype-level privacy processing.

[0093] The prototype-level privacy processing module 160 performs privacy protection processing (privacy transformation) on the spatial interaction prototype obtained in step S104 to generate an uploadable privacy-protected spatial interaction prototype.

[0094] Privacy protection can be represented as: ; in, This represents the k-th spatial interaction prototype; R represents the prototype of privacy-preserving space interaction; R represents the random projection, subspace pruning, feature selection, or hash mapping function; Q(⋅) represents the quantization function; Representing privacy noise. Preferably, R reduces the spatial interaction prototype dimension to 1 / 2 to 1 / 8 of the original dimension. Preferably, the quantization function Q(⋅) employs 4-bit to 8-bit quantization, product quantization, or codebook quantization. Preferably, It can be Gaussian noise, Laplace noise, or other noise that satisfies differential privacy constraints.

[0095] In one implementation, a privacy-preserving space interaction prototype Instead of uploading in plaintext format for a single device, it is further encapsulated into a secure aggregate share so that the cloud can only obtain aggregated statistical results from multiple devices.

[0096] The technical purpose of this step is to reduce the risk of inferring a single user's real-world scene, object layout, behavioral trajectory, or original sensor sequence from uploaded content while retaining the statistical information required for cloud-based learning.

[0097] like Figure 3 As shown, this illustrates the process of generating a spatial interaction prototype and handling privacy.

[0098] Step S106, Secure Aggregation and Prototype Cluster Management.

[0099] The security aggregation gateway 210 receives privacy-preserving space interaction prototypes or their security aggregation shares uploaded by multiple end devices and generates cross-device aggregated prototypes. The prototype cluster manager 220 manages the cross-device aggregated prototypes by clustering them according to action type, coarse-grained scene category, object or region category, multimodal feedback type, failure type, device type, or time decay weight.

[0100] Each prototype cluster can store the following statistics: prototype mean, prototype variance, device coverage, timestamp, failure rate, success rate, feedback distribution, sensor type distribution, soft tags or combinations thereof.

[0101] The technical role of this step is to transform real deployment experience from different edge devices into a statistical experience library that spans devices, scenarios, and tasks, providing input for cloud-based supervision and model training.

[0102] Step S107: Cloud-based monitoring and completion.

[0103] The cloud-based supervised completion module 230, based on the cross-device aggregated prototype obtained in step S106 and the multimodal embodied teacher model or statistical information deployed on the cloud server, performs supervised completion on the spatial interaction prototype with weak supervision on the end side.

[0104] Based on the information content in spatial interaction events, supervised completion can include: determining the causal correlation between actions and state changes; estimating the category of failure causes; generating soft labels for action conditional state changes; distinguishing between local private facts and cross-scene transferable rules; performing task-level attribution of user correction signals; and generating soft labels, pseudo-labels, or task descriptions for distillation training, contrastive learning, or ranking learning.

[0105] The technical role of this step is to solve the problem that traditional federated learning requires edge devices to complete high-quality annotation and stable training locally. The edge devices only need to generate weakly supervised spatial interaction prototypes, while the cloud uses stronger models and cross-device statistics to complete the supervision signal completion.

[0106] Step S108: Cloud distillation, comparison, and causal training.

[0107] The Distillation / Contrastive / Causal Learning Engine 240 trains a target multimodal embodied teacher model deployed on a cloud server, based on aggregated spatial interaction prototypes, cloud-supervised completion signals, and a multimodal embodied teacher model. Training objectives include knowledge distillation, contrastive learning of space-action-feedback / outcome, prediction of action conditional state changes, failure cause modeling, and privacy and stability constraints. These objectives can be selected based on the information content of spatial interaction events; the following example includes all training objectives.

[0108] The training loss can be expressed as: ; Where L represents the total training loss; This represents the loss from knowledge distillation. The contrastive learning loss represents the spatial state, action, and feedback. Indicates the loss predicted by changes in action condition state; Indicates the loss for classifying, ranking, or calibrating the causes of failure (failure attribution loss); This indicates a privacy regularity or prototype reversibility penalty. This indicates constraints related to version stability, catastrophic forgetting suppression, or cross-device fairness. arrive The weights for each loss item.

[0109] The technical role of this step is to enable the cloud model to learn not only "what is in the scene", but also "what kind of multimodal feedback and task results will be produced when a certain type of action is performed in a certain spatial state".

[0110] like Figure 4 As shown, the cloud-based training objectives are illustrated.

[0111] S109, Small parameter update generation.

[0112] Based on the training results of step S108, the parameter efficiency update manager 260 generates local update parameters to be issued. The local update parameters include parameters of the visual projector, cross-modal bridging layer, action head, adapter, prompt, prefix, LoRA, or other parameter efficiency fine-tuning modules.

[0113] Small parameter updates can be represented as: ; in, The parameters of the target multimodal embodied model for a cloud deployment that is frozen or substantially frozen; This indicates a partial parameter update; This represents the updated model parameter combination formed by the target multimodal embodied model parameters and the local update parameters.

[0114] The technical purpose of this step is to avoid frequent training and distribution of the full large model, thereby reducing cloud training costs, device download costs, device storage costs, and version rollback costs.

[0115] Step S110: Version signing, distribution, and local activation.

[0116] The version signing, auditing, and rollback module 270 performs version signing, compatibility checks, audit logging, canary rollout, and abnormal rollback configuration on the partial update parameters obtained in step S109. After receiving the partial update parameters processed by the version signing, auditing, and rollback module 270, the end-side device verifies the signature and loads it into the multimodal embodied model 180 by the local small parameter update module 170. When the end-side device detects that the online task success rate, latency, security indicators, or number of user corrections does not meet preset conditions, it can trigger a version rollback to restore the previous stable version.

[0117] The technical purpose of this step is to enable cloud model updates to be applied to edge devices safely and controllably, without making edge devices dependent on frequent full model replacements.

[0118] Step S111: Local memory retrieval is used for online tasks.

[0119] The local memory index 140 stores user-private scene facts and short-term change information, such as common object locations, room layouts, user preferences, temporary obstacles, and local map changes. During online task execution, the edge device prioritizes using the local memory index to complete scene question answering, navigation, grasping, placement, obstacle avoidance, and task planning.

[0120] This step runs in parallel with the cloud update chain from S101 to S110. Its technical function is to retain the real-world scenario facts that are highly privacy-conscious, short-term, and highly personalized on the device side, rather than uploading them to the cloud for training.

[0121] like Figure 5 As shown, a sequence diagram illustrating privacy processing and the distribution of partial update parameters is presented.

[0122] The method provided in the embodiments of the present invention has the following main advantages: (1) Reduce the risk of leakage of original scene data. This invention does not upload original images, videos, point clouds, continuous trajectories, or original sensor time-series data. Instead, it aggregates multiple interaction events into a spatial interaction prototype on the edge, and then performs random projection, quantization, differential privacy (i.e., privacy protection processing), and secure aggregation. Since prototyping destroys the per-sample correspondence, random projection and quantization reduce reversibility, and differential privacy and secure aggregation limit the information visible in the cloud, it can reduce the risks of sample inversion, scene reconstruction, and semantic leakage.

[0123] (2) Reduce reliance on edge labeling and training capabilities. Traditional federated learning requires edge devices to train models based on local data, but embodied data is usually unlabeled, continuous, multimodal, and strongly temporally coupled interactive data. This invention enables the edge device to only undertake event summarization, state change encoding, and privacy processing tasks, while the cloud completes supervised completion, failure attribution, and distillation training, thereby avoiding the problems of unstable local training objectives and excessive supervision noise on the edge device.

[0124] (3) Balancing local personalization and cloud-based general capabilities. This invention stores user-private, short-cycle, and scenario-dependent knowledge in the device-side memory; and uploads cross-device transferable spatial-action-feedback / result patterns to the cloud in prototype form. Thus, user-private facts are not directly written into the cloud model, while the diverse data in real deployments can be used to improve the general spatial-physical reasoning capabilities of the cloud model.

[0125] (4) Enhance the model's ability to learn from real physical interactions. This invention not only uses visual semantic information, but also introduces feedback such as force, touch, laser, depth, collision, user correction, and task results, enabling the cloud model to learn "what consequences will occur when performing a certain type of action in a certain type of spatial state". This helps improve the model's generalization ability in tasks such as grasping, pushing, obstacle avoidance, navigation, contact judgment, and failure recovery.

[0126] (5) Reduce communication and update costs. The present invention uploads a low-dimensional, quantized, and aggregated spatial interaction prototype and sends out local update parameters, rather than the original data or full model parameters. Therefore, it can reduce uplink bandwidth, downlink bandwidth, end-side storage and version management overhead.

[0127] (6) It facilitates integration with synthetic data and simulation enhancement. When the aggregated prototype coverage is high, the cloud can generate synthetic views or simulation tasks that do not correspond to a single user scenario based on the prototype cluster, which can be used to further enhance distillation training, contrastive learning, action condition state change prediction training, or failure attribution training. This approach can expand the training distribution while reducing privacy risks.

[0128] III. Example Introduction.

[0129] Example 1: Home service robot grasping and placing tasks.

[0130] In this embodiment, the edge device is a home service robot, including an RGB-D (red-green-blue-depth) camera, a short-range laser sensor, a six-dimensional force / torque sensor, a gripper tactile sensor, and an IMU (inertial measurement unit). The robot performs tasks such as grasping, placing, pushing, opening and closing doors, and obstacle avoidance near desktops, counters, floors, and sofas. The edge spatial representation engine constructs an object-level semantic map, encoding desktops, cabinet doors, handles, cups, bottles, bowls, remote controls, and obstacles as region or object latent vectors. When the robot performs a grasping action, the spatial interaction event encoder records the object position before the action, the grasping posture, the gripper action, the object pose change after the action, the force-feedback sliding peak, the depth change, and the task success / failure result. For example, when the robot grasps a cylindrical cup near the edge of the desktop, the gripper force-feedback shows a sliding signal, the depth map shows the object rotating, the first grasping attempt fails, and it succeeds after adjusting the gripper angle. Multiple similar events are clustered to form a spatial interaction prototype, representing the empirical pattern that "cylindrical objects near the edge of a table are prone to rolling or slipping during grasping actions, and adjusting the gripper angle can improve the success rate." Before uploading, this prototype undergoes random projection, 8-bit quantization, differential privacy noise injection, and secure aggregation encapsulation. After receiving similar prototypes aggregated from multiple home robots, the cloud performs failure attribution, action condition state change prediction, and distillation training, updating only the visual bridging layer and the LoRA of the action head. The updated prototype is then distributed to the edge robot, enabling it to have better initial action selection capabilities in similar grasping scenarios.

[0131] Example 2: Navigation and obstacle avoidance tasks for mobile inspection robots.

[0132] In this embodiment, the edge device is a mobile inspection robot, including a forward-looking camera, a 2D LiDAR, an IMU, and a wheel speedometer. The robot performs inspection tasks in corridors, doorways, elevator lobbies, warehouse passages, and office areas. The edge-side spatial representation engine encodes passageways, doorways, dynamic obstacles, corners, turning areas, and narrow areas into region latent vectors and traversable relationship graphs. When the robot performs navigation actions, the spatial interaction event encoder records the local traversal status, speed commands, turning actions, LiDAR occlusion changes, collision warnings, number of local replanning attempts, and whether the passage was successful before the action is executed. Multiple similar events form a spatial interaction prototype, such as "narrow passage + dynamic pedestrian occlusion + left turn + sudden change in LiDAR distance + decreased success rate of local replanning". After aggregating such prototypes uploaded by multiple robots in the cloud, the relationship between dynamic obstacles and navigation action results is learned, and the risk assessment head of the multimodal embodied model is updated. Specifically, if the model is updated by the edge device using local update parameters, then the multimodal embodied model here is the multimodal embodied model deployed locally on the edge device; if the training object is in the cloud, then the multimodal embodied model here is the target multimodal embodied model deployed on the cloud server.

[0133] This embodiment illustrates that the present invention is not limited to robotic arm operation, but is also applicable to mobile robots based on spatial accessibility and sensor feedback.

[0134] Example 3: Verifiable cloud execution in a high-privacy home environment.

[0135] In this embodiment, the edge device is located in the user's home, where privacy requirements are high. The edge device only uploads the secure aggregation share and does not upload the plaintext prototype that can be parsed individually. The cloud-based secure aggregation gateway runs in a verifiable execution environment. This execution environment only retains the temporary data required for this round of aggregation, deletes intermediate states after training, and generates an audit summary. The cloud only trains on prototype clusters that have reached at least a preset number of device coverages. For example, supervised completion and distillation training only begins when the same prototype cluster comes from at least 100 devices and meets the differential privacy budget constraint. This embodiment reduces the direct impact of single-user scenarios on cloud model updates and also reduces the risk of inferring privacy information from a single home environment.

[0136] Example 4: Augmented training based on synthetic views.

[0137] In this embodiment, the cloud-based prototype cluster manager has aggregated a large number of spatial interaction prototypes related to "desktop edge grabbing," "cabinet door handle operation," and "obstacle avoidance in narrow passages." Optional composite views or simulation enhancement modules generate composite scenes that do not correspond to any single user's home based on a publicly available 3D asset library and aggregated prototype clusters. The cloud renders different perspectives, lighting, object sizes, and placement combinations within the composite scene, and uses the generated data for distillation training and motion condition state change prediction training. In this embodiment, the composite data is not reconstructed from the original scene of a single device, but rather generated based on prototype clusters aggregated across devices, thus expanding the training distribution while reducing the risk of leaking the spatial layout of individual users.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0139] Example 2 This invention also provides a multimodal embodied model update system, which is mainly used to implement the method provided in the foregoing embodiments. The system mainly includes: an edge device and a cloud server; wherein: End-side equipment includes: Multimodal sensor interface, used to acquire multimodal observation data; Spatial representation engine, used to generate edge spatial representations; Multimodal embodied models are used to combine local memory indexes with edge spatial representations to make decisions on actions; An actuator is used to perform actions; A spatial interactive event encoder is used to record multimodal observation data before action execution, action information, multimodal observation data after action execution, multimodal feedback information, and task result information, utilizing two or more of these types of information. The spatial interaction prototype generation module is used to integrate multiple spatial interaction events to form a spatial interaction prototype. A prototype-level privacy processing module is used to process the spatial interaction prototype for privacy protection before uploading it to the cloud server; The local small parameter update module is used to update the locally deployed multimodal embodied model using the managed local update parameters; Cloud servers include: The security aggregation gateway is used to receive privacy-preserving space interaction prototypes uploaded by multiple end devices and generate cross-device aggregation prototypes. The cloud-based supervised completion module is used to supervise and complete the spatial interaction prototype of the end device based on the cross-device aggregated prototype, and obtain the completion supervision signal; The learning engine is used to perform distillation training and action condition state change prediction training on a target multimodal embodied model deployed on a cloud server by leveraging cross-device aggregated prototypes and completion supervision signals, or it may also include contrastive learning and / or failure attribution training. A parameter update manager that generates locally updated parameters based on training results; The preprocessing module is used to preprocess the local update parameters before publishing, obtain the remediated local update parameters, and send them to the edge devices.

[0140] This system corresponds to the method framework in the aforementioned embodiments; therefore, it can also be referred to. Figure 1 The framework structure shown is illustrated. Furthermore, the main technical details involved in this system have been described in detail in previous embodiments, and therefore will not be repeated here.

[0141] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0142] Example 3 The present invention also provides a processing device, such as Figure 6 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0143] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0144] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0145] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0146] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0147] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for updating a multimodal embodied model, characterized in that, include: The edge device collects multimodal observation data and generates an edge spatial representation. Based on the edge spatial representation, the local multimodal embodied model combined with the local memory index makes a decision on the action, and then the action is executed by the actuator. The spatial interaction event is formed by using the multimodal observation data or its spatial state representation before the action is executed, the action information, and the multimodal observation data or state change information after the action is executed. Multiple spatial interaction events are combined to form a spatial interaction prototype, which is then processed for privacy protection and uploaded to the cloud server. The cloud server receives privacy-preserving spatial interaction prototypes uploaded by multiple edge devices and generates a cross-device aggregated prototype. Based on the cross-device aggregated prototype, the spatial interaction prototypes of the edge devices are supervised and completed to obtain completion supervision signals. The cross-device aggregated prototype and completion supervision signals are used to perform distillation training and action condition state change prediction training on the target multimodal embodied model deployed on the cloud server, or, may also include contrastive learning and / or failure attribution training. Based on the training results, local update parameters are generated and published preprocessing is performed to obtain the governed local update parameters and distribute them to the edge devices. The edge device updates the locally deployed multimodal embodied model using the local update parameters after governance.

2. The multimodal embodied model update method according to claim 1, characterized in that, The multimodal observation data includes any one or a combination of visual information, depth information, laser ranging information, point cloud information, force information, tactile information, inertial measurement information, joint status, wheel speed information, action execution status, and user interaction signals.

3. The multimodal embodied model update method according to claim 1, characterized in that, The generation of edge spatial representations includes: generating scene-level, object-level, region-level, or relation-level spatial representations based on the multimodal observation data.

4. The multimodal embodied model update method according to claim 1, characterized in that, The process of integrating multiple spatial interaction events into a spatial interaction prototype, followed by privacy protection processing, includes: For multiple spatial interaction events under a time window, a task segment, a spatial region, or an action category, use any one or any combination of clustering, pooling, graph aggregation, vector quantization, and statistical summarization to generate spatial interaction prototypes. Privacy protection is achieved using the following formula: ； in, This represents the k-th spatial interaction prototype; Let R represent the k-th privacy-preserving space interaction prototype, and let R represent random projection, subspace pruning, feature selection, or hash mapping function. Represents the quantization function; This indicates privacy noise.

5. The multimodal embodied model update method according to claim 1, characterized in that, The supervised completion of the spatial interaction prototype of the end-side device based on the cross-device aggregation prototype includes: Cluster management is implemented for cross-device aggregated prototypes, obtaining statistical information for each prototype cluster. Supervised completion is then performed using a multimodal embodied teacher model deployed on a cloud server, combined with this statistical information, including: By combining statistical information from prototype clusters, we distinguish between local private facts and cross-scenario transferable patterns; based on action information and spatial state change information after action execution, we determine the causal correlation between actions and state changes; based on action information, spatial state representation before action execution, and spatial state change information after action execution, we generate soft labels for action conditional state changes, where spatial state representation is obtained by mapping corresponding multimodal observation data to the latent space; when spatial interaction events also include multimodal feedback information, task result information, and user correction information, we estimate the failure cause category by combining multimodal feedback information, task result information, and spatial state change information after action execution; and by combining user correction information, action information, spatial state change information after action execution, and task result information, we perform task-level attribution of user correction signals.

6. The multimodal embodied model update method according to claim 5, characterized in that, The cluster management of cross-device aggregated prototypes includes: clustering cross-device aggregated prototypes according to action type, coarse-grained scene category, object or region category, multimodal feedback type, failure type, device type, or time decay weight.

7. The multimodal embodied model update method according to claim 5, characterized in that, The method of using cross-device aggregated prototypes and supplementary supervision signals to perform distillation training and action condition state change prediction training on the target multimodal embodied model deployed on the cloud server, or, further includes contrastive learning and / or failure attribution training, includes: When using distillation training, contrastive learning, action conditional state change prediction, and failure attribution training, the corresponding losses are calculated, and training losses are constructed by combining regularization constraints. The training losses are then used to optimize the parameters of the multimodal embodied model deployed on the cloud server. Specifically, distillation training utilizes action conditional state change soft labels, task-level attribution results, or other supplementary supervision signals generated by the cloud-based multimodal embodied teacher model; contrastive learning is responsible for learning the correspondence between the spatial state before action execution, action information, spatial state change information after action execution, and multimodal feedback information in the cross-device aggregated prototype, and is supervised using causal correlation and local private facts and cross-scene transferable rules; action conditional state change prediction training uses the spatial state representation before action execution and action information to predict the spatial state change information after action execution, and is supervised using action conditional state change soft labels; failure attribution training uses multimodal feedback information, task result information, user correction information, and spatial state change information after action execution to estimate the failure cause category, and is supervised using the failure cause category. The training loss is represented as: ； Where L is the training loss, This represents the loss from knowledge distillation. The contrastive learning loss represents the spatial state, action, and feedback. Indicates the loss predicted by changes in action condition state; Indicates failure attribution to loss; This indicates a privacy regularity or prototype reversibility penalty. This indicates version stability, catastrophic forgetting suppression, or cross-device fairness constraints. and Form regularization constraint terms; arrive The weights for each loss item.

8. A multimodal embodied model update system, characterized in that, To implement the method according to any one of claims 1 to 7, the method comprises: an end-side device and a cloud server; wherein: End-side equipment includes: Multimodal sensor interface, used to acquire multimodal observation data; Spatial representation engine, used to generate edge spatial representations; Multimodal embodied models are used to combine local memory indexes with edge spatial representations to make decisions on actions; An executive mechanism is used to perform actions; A spatial interactive event encoder is used to construct spatial interactive events using multimodal observation data or its spatial state representation before the action is executed, action information, and multimodal observation data or state change information after the action is executed. The spatial interaction prototype generation module is used to integrate multiple spatial interaction events to form a spatial interaction prototype. A prototype-level privacy processing module is used to process the spatial interaction prototype for privacy protection before uploading it to the cloud server; The local small parameter update module is used to update the locally deployed multimodal embodied model using the managed local update parameters; Cloud servers include: The security aggregation gateway is used to receive privacy-preserving space interaction prototypes uploaded by multiple end devices and generate cross-device aggregation prototypes. The cloud-based supervised completion module is used to supervise and complete the spatial interaction prototype of the end device based on the cross-device aggregated prototype, and obtain the completion supervision signal; The learning engine is used to perform distillation training and action condition state change prediction training on a target multimodal embodied model deployed on a cloud server by leveraging cross-device aggregated prototypes and completion supervision signals, or it may also include contrastive learning and / or failure attribution training. A parameter update manager that generates locally updated parameters based on training results; The preprocessing module is used to preprocess the local update parameters before publishing, obtain the remediated local update parameters, and send them to the end-side devices.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images