Robot operation and control methods, devices, electronic equipment and storage media

By aligning training with prior branch models and future visual-tactile evidence in robot operation control, a sequence of control actions is generated, which solves the problem of lag in robot contact state judgment in visually ambiguous scenarios and improves the safety and accuracy of operation.

CN123103620APending Publication Date: 2026-11-17BEIJING ZHIZAI WUJIE TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611076062.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In existing technologies, robots have difficulty accurately judging the contact state of objects in visually ambiguous scenarios such as occlusion, similar appearance, or limited field of view. Furthermore, fixed motion blocks have a lag in responding to rapid tactile changes, which affects the safety and accuracy of operation.

Method used

By acquiring current multimodal observation data and inputting it into a pre-trained prior branch model, a sequence of prior latent variables is generated. Then, the model is aligned with future visual and tactile evidence to generate a sequence of control actions, ensuring that the model can still respond to changes in contact state in a timely manner when future observations are unavailable.

Benefits of technology

This enables the robot to respond promptly to changes in contact status even when future observations are unavailable, thus improving the safety and accuracy of operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123103620A_ABST
    Figure CN123103620A_ABST
Patent Text Reader

Abstract

This invention provides a robot operation control method, device, electronic device, and storage medium, relating to the field of artificial intelligence technology. The method includes: acquiring current multimodal observation data, inputting it into a pre-trained prior branch model, and outputting a prior latent variable sequence; generating a control action sequence based on the prior latent variable sequence, and controlling the robot to perform operations based on the control action sequence; the prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, and inputting historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after executing the action corresponding to the historical multimodal observation data. Through the method provided by this invention, timely responses to changes in contact state can still be made under online inference conditions where future observations cannot be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a robot operation control method, device, electronic device, and storage medium. Background Technology

[0002] In the field of robot operation and control, visual language motion models typically use natural language commands, camera images, and robot body state as inputs to directly predict joint trajectories or end effector poses, while the utilization of tactile information relies on dedicated sensors installed on fingertips or palms.

[0003] In existing technologies, some methods introduce future frames or bidirectional attention mechanisms during training, enabling the model to learn the visual and tactile consequences of actions. However, these methods cannot acquire future observations during online inference, making it difficult for the deployed model to utilize this information containing contact consequences. Other methods, which only use current observations during training and inference, completely lack the ability to predict the contact state after action execution. These shortcomings make it difficult for robots to accurately determine the contact state of objects in visually ambiguous scenarios such as occlusion, similar appearances, or limited perspectives. Fixed action blocks also exhibit lag in their response to rapid tactile changes, affecting the safety and accuracy of operations.

[0004] Therefore, a solution is urgently needed to address the above problems. Summary of the Invention

[0005] This invention provides a robot operation control method, device, electronic device, and storage medium to overcome the deficiencies in the prior art.

[0006] This invention provides a robot operation control method, comprising: Acquire the current multimodal observation data and input the current multimodal observation data into the pre-trained prior branch model, and output the prior latent variable sequence; A control action sequence is generated based on the prior latent variable sequence, and the robot is controlled to perform operations based on the control action sequence; The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches for training; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after the action corresponding to the historical multimodal observation data is performed.

[0007] According to a robot operation control method provided by the present invention, the historical multimodal observation data includes tactile observation data; the process of acquiring the tactile observation data includes: A video of human operation is acquired, a hand mesh is reconstructed from the video frames of the human operation video, and the reconstructed hand mesh is mapped onto a preset standard hand surface to obtain multiple surface vertices. For each surface vertex, a continuous proximity value representing the degree of proximity is generated based on the distance between the surface vertex and the surface of the target object, and a contact state label is generated based on preset contact conditions. The tactile observation data is generated based on the continuous proximity value and contact state label of each surface vertex.

[0008] According to a robot operation control method provided by the present invention, after mapping the reconstructed hand mesh onto a preset standardized hand surface to obtain multiple surface vertices, the method further includes: For surface vertices that meet the preset conditions, an invalid flag is generated; The preset conditions include: the image region where the surface vertex is located is occluded, or the confidence level of the reconstructed hand mesh is lower than a preset threshold; the invalidation flag is used to exclude the supervision information of the corresponding surface vertex when calculating the loss of alignment training.

[0009] According to a robot operation control method provided by the present invention, before inputting historical multimodal observation data for training into the first branch and inputting the historical multimodal observation data and future visual-tactile evidence together into the second branch, the method further includes: Each tactile unit in the tactile observation data is represented as a tactile token; wherein, the tactile token includes the spatial position of the corresponding tactile unit, the continuous proximity value, the contact state label, and a mark indicating whether the supervision information is valid; Using a resampler capable of handling variable-length inputs, different numbers of the haptic tokens are converted into a fixed number of haptic latent variables, and the haptic latent variables are input to the first branch or the second branch for fusion.

[0010] According to a robot operation control method provided by the present invention, the training process of the prior branch model further includes: Obtain a pre-constructed canonical hand model; wherein the canonical hand model is used to define the semantic topology of wrist pose, static skeletal morphology, and joint motion variables; The joint states of different robot end effectors and the reconstructed joint states of the human hand are uniformly mapped to the state representation under the standard hand model; For different robots, an activity dimension mask is generated; wherein, the activity dimension mask is used to identify the validity of each joint activity variable in the standard hand model on the robot.

[0011] According to a robot operation control method provided by the present invention, the training objective of the alignment training includes a future tactile prediction loss; the calculation process of the future tactile prediction loss includes: The latent variable sequence output from the first branch is decoded to generate a prediction result for the future tactile state; The prediction result is compared with the future tactile observations in the future visual-tactile evidence to obtain the future tactile prediction loss.

[0012] According to a robot operation control method provided by the present invention, the step of generating a control action sequence based on the prior latent variable sequence and controlling the robot to perform operations based on the control action sequence includes: A first action block containing multiple control steps is generated based on the prior latent variable sequence, the robot is controlled to execute the first action block, and the context representation is cached during the generation of the first action block; When the first action block is executed to the preset anchor point, the latest body state and tactile observation data of the robot are obtained; Based on the robot's latest body state and tactile observations, as well as the cached context representation, the active action segments in the first action block that have not yet been executed are corrected to generate corrected control actions.

[0013] According to a robot operation control method provided by the present invention, the training process of the prior branch model adopts a phased training strategy, the phased training strategy including: In the first stage, the initial model was pre-trained using a hybrid dataset containing videos of human operation and data from multiple robots to obtain a pre-trained model. In the second stage, the pre-trained model is trained using operational demonstration data related to the target task to obtain an intermediate model. In the third stage, the intermediate model is post-trained using the synchronous visual, ontological state, action, and real tactile data collected by the target robot on the target task to obtain the prior branch model.

[0014] The present invention also provides a robot operation control device, comprising: The prior inference module is used to acquire the current multimodal observation data, input the current multimodal observation data into the pre-trained prior branch model, and output the prior latent variable sequence; An operation control module is used to generate a control action sequence based on the prior latent variable sequence, and to control the robot to perform operations based on the control action sequence; The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches for training; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after the action corresponding to the historical multimodal observation data is performed.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the robot operation control method described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the robot operation control method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the robot operation control method as described above.

[0018] This invention provides a robot operation control method, device, electronic device, and storage medium. It acquires current multimodal observation data and inputs this data into a pre-trained prior branch model, outputting a prior latent variable sequence. A control action sequence is generated based on the prior latent variable sequence, and the robot is controlled to perform operations based on this sequence. The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches. The future visual-tactile evidence consists of future visual observations and / or future tactile observations collected after the actions corresponding to the historical multimodal observation data are executed. Therefore, this invention, by using future visual-tactile evidence to align the latent variables of the two branches during the training phase, enables the prior branch model at deployment to generate control actions that implicitly contain information about future contact consequences, relying only on current multimodal observations. This allows for timely responses to changes in contact states even under online inference conditions where future observations are unavailable. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the robot operation control method provided by the present invention.

[0021] Figure 2 This is a diagram illustrating the overall model architecture of the robot operation and control method provided by this invention.

[0022] Figure 3 This is a schematic diagram of surface tactile annotation and multi-granularity tactile coding provided by the present invention.

[0023] Figure 4 This is a schematic diagram of the canonical hand model canonical topology and cross-mechanism mapping provided by the present invention.

[0024] Figure 5 This is a schematic diagram of the robot operation control device provided by the present invention.

[0025] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0027] The following description, in conjunction with the accompanying drawings, describes a robot operation control method, apparatus, electronic device, and storage medium of the present invention.

[0028] Figure 1 This is a flowchart illustrating the robot operation control method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 100: Obtain the current multimodal observation data and input the current multimodal observation data into the pre-trained prior branch model, and output the prior latent variable sequence.

[0029] It should be noted that the embodiments of the present invention achieve robust robot operation decisions by simulating the human cognitive mechanism of "thinking before verifying".

[0030] Specifically, observation data at the current moment is acquired through a multimodal perception module deployed on the robot body. This data includes, but is not limited to, natural language task instructions, two-dimensional or three-dimensional scene images captured by monocular or multi-view cameras, on-body state information such as robot joint angles and end-effector poses, and fingertip or palm tactile array readings when hardware is available. This heterogeneous raw data is then input into a pre-trained prior branch model. This prior branch model is essentially a deep neural network encoder, which internally performs cross-attention fusion on the above multimodal inputs through a set of learnable latent query tokens, and finally outputs a compact, fixed-length sequence of prior latent variables. This sequence implicitly encodes the visual semantics, physical state, and contact context information required to complete the task in the current scene.

[0031] Step 200: Generate a control action sequence based on the prior latent variable sequence, and control the robot to perform operations based on the control action sequence; The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches for training; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after the action corresponding to the historical multimodal observation data is performed.

[0032] Specifically, based on the prior latent variable sequence generated in step 100, a conditional action generation module, such as using conditional flow matching or diffusion model, decodes a continuous control action sequence over a predetermined time span. This action sequence can be represented as the desired pose trajectory of the robot's end effector in Cartesian space, the incremental changes in the angles of each joint, or the incremental joint motion in the topological space of the normalized hand model. Subsequently, the robot controller parses these action instructions into motor drive signals to drive the robot to perform corresponding physical operations.

[0033] Regarding the training mechanism of the prior branch model, this embodiment introduces a dual-branch architecture during the training phase: the first branch (i.e., the prior branch) only receives historical multimodal observation data that is currently available online from the training data, simulating the information conditions during actual deployment; the second branch (i.e., the posterior branch), as a supervised enhancement module, additionally receives future visual observations and / or future tactile observations collected after the action is performed within the same time window during training. This future evidence contains key information such as the contact consequences of the action, changes in the object's motion state, and the success or failure of the operation. During training, the system forces the distribution of the first and second branches to align in the corresponding latent variable slots, enabling the prior branch, which relies solely on current observations, to distill the contact and motion consequence information from the posterior branch that was originally only known by seeing future results. After such aligned training, the computationally expensive posterior branch and the collection of future data can be completely removed during the online inference phase. Only the lightweight prior branch needs to be run to generate control actions containing long-term contact prediction capabilities, thereby meeting the real-time requirements of robot online control while ensuring control performance.

[0034] Figure 2 This is a diagram illustrating the overall model architecture of the robot operation control method provided by this invention. It fully demonstrates the overall model architecture of Being-H0.8 and the prior-posterior latent variable alignment mechanism during the training period provided by the embodiments of this invention. Figure 2 As shown in (a), the overview section on the left first uses a Mixture of Transformers model to process Shared Context and Visuo-Tactile Query to fuse current multimodal observations. The system is designed with a Prior Branch and a Posterior Branch that is only enabled during the training phase. Both branches receive Noised Actions and generate corresponding latent variables through a Slow-Fast Action Expert with Shared Weights. The Posterior Branch additionally introduces Future Embedding to capture future visual-tactile evidence and computes it in the corresponding slot. L align (Latent variable alignment loss), thereby transferring future information to prior latent variables, while combining... L reg (Action generation regression loss) is used to supervise the action prediction of both branches; such as Figure 2As shown in (b), the right side details the timing processing of the Slow-Fast Action Expert. It divides the input in the sequence along the timeline into Shared SlowContext, Visuo-Tactile Query, and real-time updated Fast Token, Latent Tactile Token, and Latest State Token. Through block slicing and prompting guidance, combined with Causal Attention Mask, it achieves local correction of action block generation and haptic feedback, thereby ensuring that only the prior branch runs during the deployment phase, achieving an effective combination of slow global planning and fast haptic response.

[0035] The above describes the steps of the robot operation control method provided by this invention. As can be seen from the above description, according to the robot operation control method provided by this invention, current multimodal observation data is acquired and input into a pre-trained prior branch model to output a prior latent variable sequence; a control action sequence is generated based on the prior latent variable sequence, and the robot is controlled to perform operations based on the control action sequence; wherein, the prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence together into a second branch, and aligning and training the latent variables at corresponding positions in the latent variable sequences output by the first and second branches; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after executing the action corresponding to the historical multimodal observation data. Therefore, this invention, by using future visual-tactile evidence to align and train the latent variables of the two branches during the training phase, enables the prior branch model at deployment to generate control actions containing implicit future contact consequences information based solely on current multimodal observations, thus allowing for timely responses to changes in contact state even under online inference conditions where future observations cannot be obtained.

[0036] Based on the above embodiments, in this embodiment, the historical multimodal observation data includes tactile observation data; the process of acquiring the tactile observation data includes: A video of human operation is acquired, a hand mesh is reconstructed from the video frames of the human operation video, and the reconstructed hand mesh is mapped onto a preset standard hand surface to obtain multiple surface vertices. For each surface vertex, a continuous proximity value representing the degree of proximity is generated based on the distance between the surface vertex and the surface of the target object, and a contact state label is generated based on preset contact conditions. The tactile observation data is generated based on the continuous proximity value and contact state label of each surface vertex.

[0037] After mapping the reconstructed hand mesh onto a preset standardized hand surface to obtain multiple surface vertices, the method further includes: For surface vertices that meet the preset conditions, an invalid flag is generated; The preset conditions include: the image region where the surface vertex is located is occluded, or the confidence level of the reconstructed hand mesh is lower than a preset threshold; the invalidation flag is used to exclude the supervision information of the corresponding surface vertex when calculating the loss of alignment training.

[0038] Specifically, the hand mesh is reconstructed for each frame of the human operation video, the vertices of the hand surface in the camera coordinate system are determined, and the correspondence between the mesh and the preset standard hand surface vertices is established. Preferably, the standard surface adopts a MANO topology containing 778 vertices, but the number of vertices and the mesh model are not limited.

[0039] Temporal processing is performed on the vertices of the canonical hand surface and the surface of the target object across multiple consecutive frames, generating a continuous proximity value representing the degree of proximity for each surface vertex. In one implementation, a supervision value is constructed based on the non-negative distance d between the hand vertex and the surface of the target object. , Where q∈[0,1] represents the continuous proximity value, and τ is the distance decay coefficient. Simultaneously, based on preset contact conditions, such as whether the continuous proximity value exceeds a predetermined threshold, whether the geometric distance between the hand surface vertex and the object surface is less than a predetermined distance threshold, or whether there is a geometric intersection relationship between the two, a contact state label for the surface vertex is generated. This contact state label indicates whether the surface vertex is currently in contact with the object surface. Furthermore, for cases where the image region containing the surface vertex is occluded, hand mesh reconstruction fails, or reconstruction confidence is lower than a preset threshold, a corresponding invalid label is generated. This invalid label is used to exclude the supervision information of the corresponding surface vertex when calculating the loss for alignment training, avoiding unreliable pseudo-labels as valid training signals. For cases where there are no real pressure sensor readings in the human operation video, the corresponding pressure channel is also set to an invalid label, rather than being disguised as a pressure-free state with a zero value.

[0040] After the above processing, ordinary human operation videos are converted into tactile observation data that is aligned with the vertex positions of the standard hand surface and contains continuous proximity values, contact state labels, and invalid markers, so that it can be used together with real tactile sensor data during model training.

[0041] Furthermore, before inputting the historical multimodal observation data used for training into the first branch and inputting the historical multimodal observation data and future visual-tactile evidence together into the second branch, the method further includes: Each tactile unit in the tactile observation data is represented as a tactile token; wherein, the tactile token includes the spatial position of the corresponding tactile unit, the continuous proximity value, the contact state label, and a mark indicating whether the supervision information is valid; Using a resampler capable of handling variable-length inputs, different numbers of the haptic tokens are converted into a fixed number of haptic latent variables, and the haptic latent variables are input to the first branch or the second branch for fusion.

[0042] Specifically, each tactile unit is represented as a tactile token that includes at least spatial location, continuous proximity value, contact state label, and a mark indicating whether the monitoring information is valid; when the sensor can provide pressure, it may also include pressure value and pressure validity mark. Spatial location is preferably expressed in the camera coordinate system.

[0043] To adapt to different data sources, granularity-level identifiers are added to the haptic tokens. The granularity can be selected as complete surface vertices, sparse anchors, semantic regions, fingertip sets, fingertip-palm subsets, or global haptic tokens. For example, 778 surface vertices can be aggregated into 32 anchors, 17 semantic regions, or 5 fingertips. A resampler capable of handling variable-length inputs converts different numbers of haptic tokens into a fixed number of haptic latent variables, making the subsequent backbone network independent of specific sensor layouts. This fixed number of haptic latent variables is then input to the first or second branch for fusion.

[0044] In one embodiment, Figure 3 This is a schematic diagram of surface tactile annotation and multi-granularity tactile coding provided by the present invention, illustrating the data processing flow from human operation video to fixed-length tactile latent variables. For example... Figure 3As shown, the model extracts target-level data from the Coarse-to-Fine Tactile Pyramid (a tactile pyramid from a coarse-to-fine scale, i.e., from the global level of N=1 to the vertex level of N=778) through Random Selection, forming a sequence of tactile tokens with dimensions (B, T, N, D). These tokens specifically contain multi-source heterogeneous information such as Camera Space Position, Binary Contact, Proximity, Normalized Force, and Identity Embeddings. Subsequently, this sequence is feature-encoded using an MLP (Multilayer Perceptron). The encoded features are combined with a Learnable Query and input into a Query-base Perceiver for global aggregation and dimensionality compression. Finally, the model outputs a fixed number of tactile tokens with a uniform shape (B, N, D). Latents (tactile latent variables) are used to standardize the representation of tactile data with different granularities and hardware layouts, so that the subsequent backbone network of the model can uniformly process tactile information without relying on specific sensor types and layouts.

[0045] It is understood that the aggregation and resampling of haptic tokens can also be achieved by means of farthest point sampling, clustering, graph pooling, ensemble networks or graph neural networks, and this embodiment does not make any special limitations on this.

[0046] The robot operation control method provided in this embodiment converts human operation videos without tactile sensors into contact state labels, continuous proximity values, and invalid markers aligned with the vertices of the standard hand surface. Combined with granular level labeling and a resampler, it unifies tactile data of different densities and sources into a fixed-length tactile latent variable. This effectively expands the sources of tactile training data, avoids interference from unreliable pseudo-labels, and achieves cross-platform compatibility with different tactile sensor layouts and hardware configurations.

[0047] Based on the above embodiments, in this embodiment, the training process of the prior branch model further includes: Obtain a pre-constructed canonical hand model; wherein the canonical hand model is used to define the semantic topology of wrist pose, static skeletal morphology, and joint motion variables; The joint states of different robot end effectors and the reconstructed joint states of the human hand are uniformly mapped to the state representation under the standard hand model; For different robots, an activity dimension mask is generated; wherein, the activity dimension mask is used to identify the validity of each joint activity variable in the standard hand model on the robot.

[0048] Specifically, a canonical hand model with a fixed semantic topology is predefined, which includes semantic topological definitions of wrist pose, static skeleton morphology, and joint motion variables. The reconstructed human hand and the joint states of different robot end effectors are mapped to this canonical hand model. In one implementation, the complete state of a single hand includes 9-dimensional wrist pose, 60-dimensional static skeleton position, and 20-dimensional joint motion variables. A morphological encoder is used to compress the static skeleton into a 10-dimensional morphological vector, resulting in a 39-dimensional compact state for a single hand; the state for both hands is obtained by combining the compact states of the left and right hands. Actions may include wrist pose increments and joint motion variable increments, for example, 29 dimensions for a single hand and 58 dimensions for both hands. For grippers or end effectors with fewer degrees of freedom, an adapter is used to map them to the same semantic dimensions, and an activity dimension mask is used to mask non-existent degrees of freedom. This activity dimension mask is used to identify the validity of each joint motion variable in the canonical hand model on the robot. The specific dimensions mentioned above can vary depending on the canonical hand model.

[0049] Optionally, the standardized hand model can be MANO, other parametric hand models, custom meshes, point clouds, or implicit surfaces; haptic units can also be bound to the object surface or the robot link surface. Proximity can be determined using exponential decay, truncated linear functions, piecewise functions, signed distance functions, or direct regression from the network; contact can be determined by thresholds, geometric intersections, pressure thresholds, or probabilistic models. This embodiment does not impose any particular limitations on these aspects.

[0050] In one embodiment, Figure 4 This is a schematic diagram of the canonical topology and cross-mechanism mapping of the canonical hand model provided by the present invention, such as... Figure 4 As shown, it is divided into two parts: a unified motion space and a skeleton shape variational autoencoder (VAE); firstly, in Figure 4 In (a), the system extracts the Rest Pose from the Human Hand and Robot Hand, generates a Shape Vector using an MLP Encoder, and then maps the Wrist Pose, Joint Angles, and Gripper Open to the Action Space. Simultaneously, the Shape Vector is mapped to the unified Shape Vector space. Figure 4In (b), the system further utilizes VAE to model the skeleton shape, encodes the Posed Skeletons of different poses into μ (mean) and σ (standard deviation) through MLP Encoder, and obtains the latent variable z through Reparameterization formula, and then reconstructs the Rest Pose Skeleton by Decoder. The whole process is constrained by three Training Objectives, including Reconstruction Loss to minimize reconstruction error, KL Loss to push μ and σ to approach the standard normal distribution, and Latent Consistency Loss to ensure the stability of the latent variable structure under different poses, thereby achieving a unified and compact representation of hand shape and motion parameters across mechanisms.

[0051] The robot operation control method provided in this embodiment pre-constructs a standardized hand model that includes wrist pose, static skeleton morphology, and semantic topology of joint activity variables. It then uniformly maps the joint states of different robot end effectors and human hands to this standardized hand model and uses activity dimension masks to identify the validity of each joint activity variable on different robots. This achieves cross-platform unification and flexible compatibility of motion representation between heterogeneous end effectors.

[0052] Based on the above embodiments, in this embodiment, the training objective of the alignment training includes future tactile prediction loss; the calculation process of the future tactile prediction loss includes: The latent variable sequence output from the first branch is decoded to generate a prediction result for the future tactile state; The prediction result is compared with the future tactile observations in the future visual-tactile evidence to obtain the future tactile prediction loss.

[0053] Specifically, a first branch and a second branch are constructed, sharing parameters or partially sharing parameters. The first branch receives information that is currently available online, including language commands, current visual observations, robot body state, and current tactile observations when available. The second branch is only enabled during the training phase and, in addition to the above information, also receives future visual and / or future tactile evidence after the execution of actions.

[0054] The two branches fuse the multimodal input using a fixed number of latent query tokens and compute alignment loss at the corresponding latent locations in at least one network layer. This allows the latent variables of the first branch to learn information from the second branch related to future contact outcomes, object motion, and operational success. After training, the second branch is removed, and only the first branch is run during deployment, thus online inference does not require future observations.

[0055] The overall training objective can be expressed as: The first two terms are the action generation losses for the first and second branches, the third term is the latent variable alignment loss, the fourth term is the future tactile prediction loss (obtained by decoding the latent variable sequence output from the first branch to generate a prediction of the future tactile state, and comparing this prediction with future tactile observations in future visual-tactile evidence), and the fifth term is the fast action update loss. The action generation loss can employ conditional flow matching, diffusion, regression, or autoregressive objectives.

[0056] Alternatively, latent variable alignment can employ a structure with partially shared or non-shared parameters, and alternative losses such as KL divergence, contrastive learning, knowledge distillation, or optimal transfer. Future evidence can consist of only visual, only tactile, or both. The action generator can employ conditional flow matching, diffusion models, autoregressive models, hybrid density networks, or deterministic regression; actions can be represented as topological increments of a canonical hand model, or as joint, end-effector pose, velocity, force, or impedance parameters. This embodiment does not impose any particular limitations on these aspects.

[0057] The robot operation control method provided in this embodiment generates a prediction result of the future tactile state by decoding the latent variable sequence output of the first branch and comparing it with the actual future tactile observation to calculate the future tactile prediction loss. This enables the model to still have the ability to predict the tactile feedback result after the action is executed, even under deployment conditions that do not depend on actual future data input, thereby improving the predictability of contact perception and the foresight of control decisions.

[0058] Based on the above embodiments, in this embodiment, step 200 generates a control action sequence based on the prior latent variable sequence, and controls the robot to perform operations based on the control action sequence, including: Step 210: Generate a first action block containing multiple control steps based on the prior latent variable sequence, control the robot to execute the first action block, and cache the context representation in the process of generating the first action block.

[0059] Step 220: When the first action block is executed to the preset anchor point, obtain the robot's latest body state and tactile observation data.

[0060] Step 230: Based on the robot's latest body state and tactile observations, as well as the cached context representation, correct the active action segments in the first action block that have not yet been executed, and generate corrected control actions.

[0061] Specifically, in this embodiment, a slow-speed module generates a first action block containing multiple control steps, such as 32 control steps, based on the fusion features of language, vision, state, and tactile feedback, and caches the context representation during the generation of the first action block. When the first action block reaches a preset anchor point, such as steps 0, 8, 16, or 24, a fast-update module reads the robot's latest body state and the latest tactile observation data. Combining this with the cached context representation, it only updates the active action segments in the first action block that have not yet been executed, such as the subsequent 8 steps, without rerunning the image encoder and language model. The anchor point position, the length of the first action block, and the length of the active action segments can be predetermined according to the training configuration, or they can be changed to adaptive triggering based on contact or pressure events.

[0062] It should be noted that the conditional flow matching described above is only one possible implementation method. Actions can also be generated using diffusion models, autoregressive models, hybrid density networks, or deterministic regression. Actions can be represented as topological increments of the canonical hand model, or as joint angles, end-effector poses, velocities, force, or impedance parameters. The above method of updating the current active action segment with fixed anchor points and reusing the slow module cache context is only one specific implementation method. Adaptive triggering methods based on contact events or pressure thresholds can also be used to replace fixed anchor points, depending on actual needs.

[0063] The robot operation control method provided in this embodiment enables real-time adjustment of local actions without rerunning the image encoder and language model, balancing the stability of global semantic planning with the agility of tactile response.

[0064] Based on the above embodiments, in this embodiment, the training process of the prior branch model adopts a phased training strategy, which includes: In the first stage, the initial model was pre-trained using a hybrid dataset containing videos of human operation and data from multiple robots to obtain a pre-trained model. In the second stage, the pre-trained model is trained using operational demonstration data related to the target task to obtain an intermediate model. In the third stage, the intermediate model is post-trained using the synchronous visual, ontological state, action, and real tactile data collected by the target robot on the target task to obtain the prior branch model.

[0065] Specifically, the first stage uses a hybrid dataset containing human operation videos and multi-robot data to pre-train the initial model, resulting in a pre-trained model. The human operation videos are annotated with surface tactile data to obtain contact state labels and continuous proximity values ​​as pseudo-labels; the multi-robot data uses real sensor tactile data when available. The second stage uses operation demonstration data relevant to the target task to perform intermediate training on the pre-trained model, resulting in an intermediate model. This data may include real first-person perspective videos, selected synthetic first-person perspective hand-object interaction videos, and scene editing data. The third stage uses synchronous visual, proprioceptive state, motion, and real tactile data collected by the target robot on the target task to post-train the intermediate model, resulting in a priori branch model that adapts the model to specific mechanisms and control frequencies.

[0066] The robot operation control method provided in this embodiment enables the model to establish generalized operational priors on large-scale heterogeneous data, and then gradually migrate to the target task and specific physical platform, thereby effectively reducing the dependence on the amount of real tactile teaching data of the target robot.

[0067] The robot operation control method provided by the present invention will be described below in a complete embodiment.

[0068] (1) Training data construction.

[0069] Step S101: Acquire the human first-person perspective operation video and the robot teaching sequence. The robot sequence includes time-synchronized images, body state, control actions, and, when hardware is available, haptic array readings.

[0070] Step S102 involves performing hand detection, mesh reconstruction, and camera alignment frame by frame on the human video, cropping the local area centered on the hand, and mapping the reconstructed mesh onto the normalized hand surface.

[0071] Step S103: Predict the contact state labels and continuous proximity values ​​of each standard hand surface vertex using consecutive frames, and generate vertex-level invalidation markers. Save the results for the left and right hands separately. Set pressure invalidation markers for unobserved pressures.

[0072] Step S104: Map the human hand state and the joint states of each robot end effector to the standard hand model; convert tactile observations of different densities into tactile tokens with granular level labels. The fitting target, constraints, activity dimension mask generation rules for robot joints to 20-dimensional joint activity variables, as well as the adapter parameters for converting the actions of the standard hand model back into specific robot control commands, are determined by the calibration results of each robot.

[0073] Step S105: Organize the data according to the order of pre-training, intermediate training, and post-training of the target robot. Synthetic videos are screened before being included in the training set based on video quality, motion consistency, and hand-object interaction credibility scores.

[0074] (2) Model training.

[0075] Step S201: The current language instruction, current visual observation, current ontology state, and current tactile observation are sent to the first branch; the current tactile observation refers to the observation obtained by the real sensor at the current moment, excluding tactile labels inferred from future frames. For data without current real tactile feedback, an invalid flag is used to make the input default.

[0076] Step S202, only during the training phase, feeds future visual evidence and future tactile evidence that can be obtained in subsequent time steps into the second branch. The future window length and the time synchronization rules for vision, state, action, and touch are determined according to the training data configuration; different sampling rates can be selected by timestamp or interpolation alignment.

[0077] Step S203: Obtain the latent variables of the two branches in the corresponding slots through the shared fusion network, and calculate the mean square error, cosine distance, contrast loss, or a combination thereof as the latent variable alignment loss.

[0078] Step S204: The latent variables of the first branch and the second branch are used to supervise the generation of the first action block, and the future contact diagram, continuous proximity field or pressure change can be jointly supervised.

[0079] Step S205: Train the fast update module to correct the active action segment based on the latest ontology state and the latest tactile observation during the execution of the first action block; backpropagate the total loss and update the model parameters.

[0080] (3) Online control of robots.

[0081] As an optional exemplary implementation, taking the dual-arm ROKAE AR5 and dual-LinkerHand L25 as an example, a 12×6 pressure array is arranged at the fingertips, sampling at approximately 30 Hz; scene images are acquired by an external camera and a wrist camera. The system receives natural language task instructions and current multimodal observations. The first branch generates a sequence of latent variables, and the slow module generates the first action block. The robot begins executing the first action block, and at a preset anchor point, the fast update module reads the latest pressure observations and the latest body state, correcting the current active action segment. This hardware embodiment does not advocate that the pressure array can directly measure continuous proximity values. The second branch is not loaded or executed during deployment.

[0082] This embodiment can be used for tasks such as fruit picking and placing, pulling out tissues with both hands, and wiping. Alternative end effectors include the DexHand 021, LinkerHand L25, and grippers.

[0083] It is understood that the specific models and sensor parameters mentioned above are merely illustrative examples, and those skilled in the art can replace them with other models of robots and corresponding tactile sensors according to actual needs. This invention does not impose any specific limitations on this.

[0084] Furthermore, the technical solutions described in the above embodiments are not limited to specific robot forms or task scenarios. They can be extended to different end effectors such as single-arm or dual-arm configurations, single-hand or dual-hand configurations, dexterous hands or grippers, and are applicable to various operational tasks such as assembly, insertion, wiping, opening and closing, grasping and tool use.

[0085] The robot operation control device provided by the present invention is described below. The robot operation control device described below can be referred to in correspondence with the robot operation control method described above.

[0086] Figure 5 This is a schematic diagram of the robot operation control device provided by the present invention, as shown below. Figure 5 As shown, the robot operation control device provided by the present invention includes: The prior inference module 501 is used to acquire the current multimodal observation data, input the current multimodal observation data into the pre-trained prior branch model, and output the prior latent variable sequence; The operation control module 502 is used to generate a control action sequence based on the prior latent variable sequence, and control the robot to perform operations based on the control action sequence; The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches for training; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after the action corresponding to the historical multimodal observation data is performed.

[0087] The robot operation control device provided by this invention acquires current multimodal observation data and inputs it into a pre-trained prior branch model, outputting a prior latent variable sequence. It then generates a control action sequence based on the prior latent variable sequence and controls the robot to perform operations based on this sequence. The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches. The future visual-tactile evidence consists of future visual observations and / or future tactile observations collected after the actions corresponding to the historical multimodal observation data are executed. Therefore, this invention, by using future visual-tactile evidence to align and train the latent variables of the two branches during the training phase, enables the prior branch model at deployment to generate control actions that implicitly contain information about future contact consequences, relying only on current multimodal observations. This allows for timely responses to changes in contact states even under online inference conditions where future observations are unavailable.

[0088] Based on the above embodiments, in this embodiment, the historical multimodal observation data includes tactile observation data; the device further includes an acquisition module, specifically used for: A video of human operation is acquired, a hand mesh is reconstructed from the video frames of the human operation video, and the reconstructed hand mesh is mapped onto a preset standard hand surface to obtain multiple surface vertices. For each surface vertex, a continuous proximity value representing the degree of proximity is generated based on the distance between the surface vertex and the surface of the target object, and a contact state label is generated based on preset contact conditions. The tactile observation data is generated based on the continuous proximity value and contact state label of each surface vertex.

[0089] Based on the above embodiments, in this embodiment, the device further includes a generation module, specifically used for: After mapping the reconstructed hand mesh onto a preset standardized hand surface to obtain multiple surface vertices, For surface vertices that meet the preset conditions, an invalid flag is generated; The preset conditions include: the image region where the surface vertex is located is occluded, or the confidence level of the reconstructed hand mesh is lower than a preset threshold; the invalidation flag is used to exclude the supervision information of the corresponding surface vertex when calculating the loss of alignment training.

[0090] Based on the above embodiments, in this embodiment, the device further includes a resampling module, specifically used for: Before inputting the historical multimodal observation data used for training into the first branch and inputting the historical multimodal observation data together with future visual-tactile evidence into the second branch... Each tactile unit in the tactile observation data is represented as a tactile token; wherein, the tactile token includes the spatial position of the corresponding tactile unit, the continuous proximity value, the contact state label, and a mark indicating whether the supervision information is valid; Using a resampler capable of handling variable-length inputs, different numbers of the haptic tokens are converted into a fixed number of haptic latent variables, and the haptic latent variables are input to the first branch or the second branch for fusion.

[0091] Based on the above embodiments, in this embodiment, the device further includes a training module, specifically used for: Obtain a pre-constructed canonical hand model; wherein the canonical hand model is used to define the semantic topology of wrist pose, static skeletal morphology, and joint motion variables; The joint states of different robot end effectors and the reconstructed joint states of the human hand are uniformly mapped to the state representation under the standard hand model; For different robots, an activity dimension mask is generated; wherein, the activity dimension mask is used to identify the validity of each joint activity variable in the standard hand model on the robot.

[0092] Based on the above embodiments, in this embodiment, the training objective of the alignment training includes future tactile prediction loss; the device further includes a computing module, specifically used for: The latent variable sequence output from the first branch is decoded to generate a prediction result for the future tactile state; The prediction result is compared with the future tactile observations in the future visual-tactile evidence to obtain the future tactile prediction loss.

[0093] Based on the above embodiments, in this embodiment, the operation control module 502 is specifically used for: A first action block containing multiple control steps is generated based on the prior latent variable sequence, the robot is controlled to execute the first action block, and the context representation is cached during the generation of the first action block; When the first action block is executed to the preset anchor point, the latest body state and tactile observation data of the robot are obtained; Based on the robot's latest body state and tactile observations, as well as the cached context representation, the active action segments in the first action block that have not yet been executed are corrected to generate corrected control actions.

[0094] Based on the above embodiments, in this embodiment, the training process of the prior branch model adopts a phased training strategy, which includes: In the first stage, the initial model was pre-trained using a hybrid dataset containing videos of human operation and data from multiple robots to obtain a pre-trained model. In the second stage, the pre-trained model is trained using operational demonstration data related to the target task to obtain an intermediate model. In the third stage, the intermediate model is post-trained using the synchronous visual, ontological state, action, and real tactile data collected by the target robot on the target task to obtain the prior branch model.

[0095] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device can be a robot or other electronic device. This electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions from the memory 630 to execute robot operation control methods, including: Acquire the current multimodal observation data and input the current multimodal observation data into the pre-trained prior branch model, and output the prior latent variable sequence; A control action sequence is generated based on the prior latent variable sequence, and the robot is controlled to perform operations based on the control action sequence; The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches for training; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after the action corresponding to the historical multimodal observation data is performed.

[0096] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in at least one embodiment of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to execute the robot operation control methods provided by the above methods, including: Acquire the current multimodal observation data and input the current multimodal observation data into the pre-trained prior branch model, and output the prior latent variable sequence; A control action sequence is generated based on the prior latent variable sequence, and the robot is controlled to perform operations based on the control action sequence; The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches for training; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after the action corresponding to the historical multimodal observation data is performed.

[0098] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the robot operation control methods provided by the above methods, including: Acquire the current multimodal observation data and input the current multimodal observation data into the pre-trained prior branch model, and output the prior latent variable sequence; A control action sequence is generated based on the prior latent variable sequence, and the robot is controlled to perform operations based on the control action sequence; The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches for training; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after the action corresponding to the historical multimodal observation data is performed.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot operation control method, characterized in that, include: Acquire the current multimodal observation data and input the current multimodal observation data into the pre-trained prior branch model, and output the prior latent variable sequence; A control action sequence is generated based on the prior latent variable sequence, and the robot is controlled to perform operations based on the control action sequence; The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches for training; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after the action corresponding to the historical multimodal observation data is performed.

2. The robot operation control method according to claim 1, characterized in that, The historical multimodal observation data includes tactile observation data; The process of acquiring the tactile observation data includes: A video of human operation is acquired, a hand mesh is reconstructed from the video frames of the human operation video, and the reconstructed hand mesh is mapped onto a preset standard hand surface to obtain multiple surface vertices. For each surface vertex, a continuous proximity value representing the degree of proximity is generated based on the distance between the surface vertex and the surface of the target object, and a contact state label is generated based on preset contact conditions. The tactile observation data is generated based on the continuous proximity value and contact state label of each surface vertex.

3. The robot operation control method according to claim 2, characterized in that, After mapping the reconstructed hand mesh onto a preset standardized hand surface to obtain multiple surface vertices, the method further includes: For surface vertices that meet the preset conditions, an invalid flag is generated; The preset conditions include: the image region where the surface vertex is located is occluded, or the confidence level of the reconstructed hand mesh is lower than a preset threshold; the invalidation flag is used to exclude the supervision information of the corresponding surface vertex when calculating the loss of alignment training.

4. The robot operation control method according to claim 2, characterized in that, Before inputting the historical multimodal observation data used for training into the first branch and inputting the historical multimodal observation data and future visual-tactile evidence together into the second branch, the method further includes: Each tactile unit in the tactile observation data is represented as a tactile token; wherein, the tactile token includes the spatial position of the corresponding tactile unit, the continuous proximity value, the contact state label, and a mark indicating whether the supervision information is valid; Using a resampler capable of handling variable-length inputs, different numbers of the haptic tokens are converted into a fixed number of haptic latent variables, and the haptic latent variables are input to the first branch or the second branch for fusion.

5. The robot operation control method according to claim 1, characterized in that, The training process of the prior branch model also includes: Obtain a pre-constructed canonical hand model; wherein the canonical hand model is used to define the semantic topology of wrist pose, static skeletal morphology, and joint motion variables; The joint states of different robot end effectors and the reconstructed joint states of the human hand are uniformly mapped to the state representation under the standard hand model; For different robots, an activity dimension mask is generated; wherein, the activity dimension mask is used to identify the validity of each joint activity variable in the standard hand model on the robot.

6. The robot operation control method according to claim 1, characterized in that, The training objective of the alignment training includes future tactile prediction loss; the calculation process of the future tactile prediction loss includes: The latent variable sequence output from the first branch is decoded to generate a prediction result for the future tactile state; The prediction result is compared with the future tactile observations in the future visual-tactile evidence to obtain the future tactile prediction loss.

7. The robot operation control method according to claim 1, characterized in that, The step of generating a control action sequence based on the prior latent variable sequence and controlling the robot to perform operations based on the control action sequence includes: A first action block containing multiple control steps is generated based on the prior latent variable sequence, the robot is controlled to execute the first action block, and the context representation is cached during the generation of the first action block; When the first action block is executed to the preset anchor point, the latest body state and tactile observation data of the robot are obtained; Based on the robot's latest body state and tactile observations, as well as the cached context representation, the active action segments in the first action block that have not yet been executed are corrected to generate corrected control actions.

8. The robot operation control method according to claim 1, characterized in that, The training process of the prior branch model adopts a phased training strategy, which includes: In the first stage, the initial model was pre-trained using a hybrid dataset containing videos of human operation and data from multiple robots to obtain a pre-trained model. In the second stage, the pre-trained model is trained using operational demonstration data related to the target task to obtain an intermediate model. In the third stage, the intermediate model is post-trained using the synchronous visual, ontological state, action, and real tactile data collected by the target robot on the target task to obtain the prior branch model.

9. A robot operation control device, characterized in that, include: The prior inference module is used to acquire the current multimodal observation data, input the current multimodal observation data into the pre-trained prior branch model, and output the prior latent variable sequence; An operation control module is used to generate a control action sequence based on the prior latent variable sequence, and to control the robot to perform operations based on the control action sequence; The prior branch model is obtained by inputting historical multimodal observation data for training into a first branch, inputting the historical multimodal observation data and future visual-tactile evidence into a second branch, and aligning the latent variables at corresponding positions in the latent variable sequences output by the first and second branches for training; the future visual-tactile evidence is future visual observation and / or future tactile observation collected after the action corresponding to the historical multimodal observation data is performed.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the robot operation control method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot operation control method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot operation control method as described in any one of claims 1 to 8.