A multimodal information fusion embodied intelligent robot control method

Abstract

The application discloses a kind of multimodal information fusion embodied intelligent robot control method, it is related to industrial control technical field, including, the multiple sensor data of robot is collected in real time, and time-space alignment and feature extraction are carried out, generate multimodal feature set;Multimodal feature set is converted into feature vector and is fused, forms multimodal environment state representation;Multimodal environment state representation is combined with robot dynamics parameters, constructs and robot synchronous virtual digital twin;Based on multimodal environment state representation, generate multiple candidate motion trajectories;Using time series causal inference analysis each sensor data contribution to state bias value, adjust the fusion weight of each sensor data.The application realizes accurate fusion to multimodal sensor data in complex environment by generating multimodal environment state representation, improves the autonomous decision-making ability and efficiency of robot.

Description

Technical Field

[0001] This invention relates to the field of industrial control technology, and in particular to a control method for an embodied intelligent robot that integrates multimodal information. Background Technology

[0002] With the development of robotics technology, more and more robots are adopting multimodal sensors to improve their perception capabilities and decision-making efficiency in complex environments. Multimodal sensors integrate various types of data, such as vision, torque, and point cloud data, providing robots with more comprehensive environmental perception and enabling them to better perform tasks such as navigation, obstacle avoidance, and cleaning. In recent years, with the rise of technologies such as graph neural networks and virtual digital twins, robot control methods have gradually become more intelligent and adaptive. By combining multimodal data with robot dynamics models, more accurate motion prediction and decision-making can be achieved. In the field of industrial control, the development of robot intelligence has driven progress in production automation and precision control.

[0003] However, existing robot control systems often face challenges in spatiotemporal alignment and fusion of multi-sensor data. Efficiently extracting features and accurately fusing them during multimodal data processing to effectively drive the robot to make optimized decisions remains a technical challenge. This is especially true in complex dynamic environments, where existing technologies often fail to adjust control strategies promptly based on real-time sensor data changes, resulting in insufficient precision and stability in robot control. Therefore, improving the autonomous decision-making ability and adaptability of robots through multimodal data fusion and temporal causal inference is a significant current challenge. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a multimodal information fusion-based embodied intelligent robot control method to solve the problems of insufficient adaptive decision-making ability and accuracy in robot control.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] This invention provides a multimodal information fusion-based embodied intelligent robot control method, comprising: real-time acquisition of multiple sensor data from the robot, performing spatiotemporal alignment and feature extraction to generate a multimodal feature set; converting the multimodal feature set into feature vectors and fusing them to form a multimodal environmental state representation; combining the multimodal environmental state representation with robot dynamic parameters to construct a virtual digital twin synchronized with the robot; generating multiple candidate motion trajectories based on the multimodal environmental state representation; inputting the virtual digital twin with the multimodal environmental state representation as initial conditions, performing parallel probabilistic deduction on the multiple candidate motion trajectories respectively, and outputting the predicted probability distribution corresponding to each candidate motion trajectory; selecting the optimal control command according to the predicted probability distribution, and determining the predicted desired state representation from the predicted probability distribution corresponding to the optimal control command; sending the optimal control command to the robot for execution, generating the executed multimodal environmental state representation; calculating the state deviation value between the executed multimodal environmental state representation and the predicted desired state representation; and using temporal causal inference to analyze the contribution of each sensor data to the state deviation value and adjusting the fusion weight of each sensor data.

[0008] As a preferred embodiment of the embodied intelligent robot control method based on multimodal information fusion described in this invention, the steps for generating the multimodal feature set are as follows:

[0009] The various sensor data include visual data, spatial point cloud data, and torque data;

[0010] Spatiotemporal alignment and unified coordinate transformation are performed on data from multiple sensors to obtain spatiotemporally aligned multi-sensor data.

[0011] Normalize the spatiotemporally aligned multisensor data to generate standardized multimodal data;

[0012] Visual semantic feature vectors, environmental geometric feature vectors, and robot pose feature vectors are extracted from standardized multimodal data to generate a multimodal feature set.

[0013] As a preferred embodiment of the embodied intelligent robot control method based on multimodal information fusion described in this invention, the steps for forming a multimodal environmental state representation are as follows:

[0014] The visual semantic feature vector, environmental geometric structure feature vector, and robot pose feature vector in the multimodal feature set are used as nodes, and the spatiotemporal and semantic relationships are used as edges to construct an environmental spatiotemporal graph.

[0015] By applying a graph neural network to fuse nodes and edges, and using an attention mechanism to emphasize environmental elements that have a causal impact on environmental decisions, an environmental state vector is generated.

[0016] By imposing motion continuity constraints on the environmental state vector and performing joint optimization, a multimodal environmental state representation is formed.

[0017] As a preferred embodiment of the embodied intelligent robot control method based on multimodal information fusion described in this invention, the steps for constructing a virtual digital twin synchronized with the robot are as follows:

[0018] The virtual digital twin is initialized based on the state layer, task layer, and policy layer of the hierarchical active reasoning architecture;

[0019] The multimodal environmental state representation and robot body dynamics parameters are synchronously input into the state layer and task layer to infer the hidden state of the current environment and provide task context.

[0020] The hidden state and task context are input into the policy layer for policy reasoning and optimization, thereby constructing a virtual digital twin synchronized with the robot.

[0021] As a preferred embodiment of the multimodal information fusion-based intelligent robot control method of the present invention, the steps for generating multiple candidate motion trajectories are as follows:

[0022] In the strategy layer of the virtual digital twin, an evaluation function is defined with minimizing the expected free energy as the optimization objective;

[0023] Based on the multimodal environment state representation and evaluation function, the entropy regularization cross-entropy method is used for sampling and optimization in the policy space to generate multiple local optimal candidate policies.

[0024] Multiple locally optimal candidate strategies are decoded into corresponding robot control command sequences, forming multiple candidate motion trajectories.

[0025] As a preferred embodiment of the embodied intelligent robot control method based on multimodal information fusion described in this invention, the steps for outputting the predicted probability distributions corresponding to each candidate motion trajectory are as follows:

[0026] Within the virtual digital twin, using the multimodal environment state representation as the initial state and each candidate motion trajectory as the control input, parallel virtual physical deduction is initiated.

[0027] In each parallel virtual physical simulation, a preset random perturbation is injected into the virtual digital twin, and multiple Monte Carlo motion simulations are performed to generate a set of future state samples corresponding to each candidate motion trajectory, which serves as the predicted probability distribution.

[0028] As a preferred embodiment of the embodied intelligent robot control method based on multimodal information fusion described in this invention, the steps of selecting the optimal control command based on the predicted probability distribution and determining the predicted desired state representation from the predicted probability distribution corresponding to the optimal control command are as follows:

[0029] The predicted probability distributions corresponding to each candidate motion trajectory are evaluated, the optimal candidate motion trajectory is selected, and the robot control command at the first time step of the optimal candidate motion trajectory is taken as the optimal control command.

[0030] Calculate the expected values ​​of multiple future state trajectories in the predicted probability distribution corresponding to the optimal control command, and determine the expected values ​​as the predicted expected state representation.

[0031] As a preferred embodiment of the embodied intelligent robot control method based on multimodal information fusion described in this invention, the steps for generating the multimodal environment state representation after execution are as follows:

[0032] The optimal control command is sent to the robot for execution, and after confirming that the physical action has been completed, various sensor data are collected.

[0033] Spatiotemporal alignment and feature extraction are performed on the multi-sensor data after execution to generate a multimodal feature set after execution. The multimodal feature set after execution is converted into feature vectors and fused to form a multimodal environmental state representation after execution.

[0034] As a preferred embodiment of the embodied intelligent robot control method based on multimodal information fusion described in this invention, the steps for calculating the state deviation between the multimodal environment state representation after execution and the predicted desired state representation are as follows:

[0035] The multimodal environment state representation after execution and the expected state representation after prediction are mapped to the same Riemannian manifold space to obtain the corresponding manifold space representation.

[0036] Calculate the geodesic distance between the two manifold spatial representations and use the geodesic distance as the state deviation value.

[0037] As a preferred embodiment of the multimodal information fusion-based intelligent robot control method of the present invention, the steps for adjusting the fusion weights of each sensor data are as follows:

[0038] For the current control cycle, based on the multimodal sensor data and state deviation values ​​generated in this cycle, time-series causal inference analysis is used to determine the causal contribution of each sensor data to the state deviation values.

[0039] Based on the causal contribution of each sensor data, the fusion weights of each sensor data during the multimodal feature set generation process in the next control cycle are adjusted in real time.

[0040] The beneficial effects of this invention are as follows: by generating multimodal environmental state representations, it achieves accurate fusion of multimodal sensor data in complex environments, thereby improving the robot's autonomous decision-making ability and efficiency; in particular, by constructing a virtual digital twin synchronized with the robot, it can accurately simulate the robot's motion trajectory in various dynamic environments, which not only improves the prediction accuracy of the motion trajectory, but also adjusts the fusion weights of each sensor data according to real-time feedback data, thereby optimizing the robot's control strategy and improving navigation accuracy, motion efficiency and stability. Especially in industrial control environments, it exhibits stronger adaptability and responsiveness. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart for a control method for embodied intelligent robots based on multimodal information fusion.

[0043] Figure 2 A flowchart for generating a multimodal environmental state representation.

[0044] Figure 3 A flowchart for constructing a virtual digital twin and generating candidate motion trajectories.

[0045] Figure 4 This is a flowchart for probabilistic deduction, execution of control commands, and adjustment of fusion weights.

[0046] Figure 5 This is a graph showing the random changes in sensor data.

[0047] Figure 6 This is a comparison chart of the predicted trajectory and the actual error.

[0048] Figure 7 A graph showing the contribution of multimodal data fusion. Detailed Implementation

[0049] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0050] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0051] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0052] Reference Figures 1-7 As one embodiment of the present invention, this embodiment provides a multimodal information fusion-based embodied intelligent robot control method, comprising the following steps:

[0053] S1. Collect multiple sensor data of the robot in real time, perform spatiotemporal alignment and feature extraction to generate a multimodal feature set; convert the multimodal feature set into feature vectors and fuse them to form a multimodal environmental state representation.

[0054] S1.1: Data from various sensors, including visual data, spatial point cloud data, and torque data;

[0055] It should be noted that visual data refers to two-dimensional image information captured by optical sensors such as cameras, used to perceive color, texture, object category and semantic content in the environment;

[0056] Spatial point cloud data refers to a collection of discrete points in three-dimensional space acquired by LiDAR or depth cameras. Each discrete point contains its coordinates in space and is used to accurately describe the geometry of the environment and the shape of objects.

[0057] Torque data refers to the force and torque information measured by torque sensors at the robot's joints, reflecting the physical interaction between the robot and its environment, such as contact force, load, and motion resistance.

[0058] S1.2: Perform spatiotemporal alignment and unified coordinate transformation on data from multiple sensors to obtain spatiotemporally aligned multi-sensor data;

[0059] Specifically, the visual data, spatial point cloud data, and torque data are time-stamped and synchronized. Based on the hardware clock or unified time reference at the time of acquisition by each sensor, data with different frequencies and delays are aligned to the same time series. Based on the pre-calibrated multi-sensor extrinsic parameters, the pixel coordinates in the visual data, the three-dimensional point coordinates in the spatial point cloud data, and the joint coordinates corresponding to the torque data are transformed to the robot base coordinate system through a rigid body transformation matrix to achieve spatial unification. After the spatiotemporal alignment is completed, the visual data, spatial point cloud data, and torque data are all in a consistent time node and a common spatial reference frame, forming spatiotemporally aligned multi-sensor data.

[0060] It should be noted that the pre-calibrated multi-sensor extrinsic parameters refer to the fixed spatial position and attitude relationship of each sensor (including camera, lidar or depth camera, torque sensor) relative to the robot base coordinate system, determined by calibration methods before the deployment of multiple sensors.

[0061] Figure 5 The study demonstrates the changing trends of three different types of sensor data: visual data, spatial point cloud data, and torque data. Each type of sensor data is displayed using lines of different colors, with red, green, and blue representing the changes in these three types of data, respectively. These data reflect the robot's interaction with the environment at different points in time and can be used to evaluate the stability and consistency of sensor data. The study also describes how multiple sensor data (vision, point cloud, and torque) are spatiotemporally aligned and uniformly transformed.

[0062] S1.3: Normalize the spatiotemporally aligned multisensor data to generate standardized multimodal data;

[0063] Specifically, based on the physical dimensions and numerical range characteristics of various sensor data, linear scaling is used to map the pixel intensity values ​​of visual data to a unified numerical range, the three-dimensional coordinates of spatial point cloud data are converted into relative coordinates centered on the robot's current position and scaled to a consistent scale, and the raw readings of torque data are compressed proportionally according to the maximum working torque range after subtracting the static bias. According to the requirements of multimodal tasks, channel dimension alignment and numerical distribution balancing are performed on the linearly scaled visual data, spatial point cloud data, and torque data respectively, so that the data of different modalities are comparable in terms of numerical scale and statistical characteristics, and standardized multimodal data is generated.

[0064] S1.4: Extract visual semantic feature vectors, environmental geometric structure feature vectors, and robot pose feature vectors from standardized multimodal data to generate a multimodal feature set.

[0065] Specifically, visual data is fed into a pre-trained convolutional neural network, and visual semantic feature vectors representing object categories and scene semantics are obtained through activation responses in intermediate layers. Simultaneously, spatial point cloud data is input into a pre-trained point cloud feature extraction network, and environmental geometric structure feature vectors describing the shape of the environment surface and the distribution of obstacles are generated using local neighborhood aggregation operations. Robot joint angle and angular velocity information are directly extracted from standardized multimodal data, and the robot's own pose feature vector is formed based on the position and orientation in the robot's base coordinate system corresponding to this information. Together, they constitute a multimodal feature set, and the three types of feature vectors are consistent in dimensionality and temporal order.

[0066] It should be noted that the convolutional neural network (CNN) is trained in a supervised manner on a large-scale image dataset to learn general visual semantic representation capabilities. During training, image samples with category labels are fed into the CNN, and prediction results are generated through forward propagation. The difference between the prediction results and the true labels is calculated using a loss function commonly used in classification tasks. Based on this loss function, the gradient is calculated layer by layer using the backpropagation algorithm, and the weight parameters in the CNN are updated according to a set learning rate (e.g., 0.001) by an optimizer. The learning rate controls the step size of parameter updates, maintaining the highest value in the early stage of training to accelerate convergence, and gradually decreasing in the later stage to improve stability. The entire training process is iterated until the fluctuation of the validation index on the validation set is less than the preset fluctuation threshold (set based on the historical statistical characteristics of the validation index and the accuracy requirements of the task, which can be set to 0.002) and there is no upward or downward trend in the CNN over multiple consecutive training cycles, at which point the pre-training is complete.

[0067] The point cloud feature extraction network is trained in a supervised manner on a large-scale dataset containing 3D point clouds and their corresponding semantic or geometric annotations to acquire the ability to perceive the geometric structure of the environment. During training, the annotated spatial point cloud data is input into the point cloud feature extraction network, and after local neighborhood aggregation and hierarchical abstraction, it outputs point-level or global-level prediction results. Depending on the task type (such as point cloud segmentation, classification, or part detection), an appropriate loss function is selected to measure the inconsistency between the prediction results and the true annotations. Based on this loss function, the gradients of each learnable parameter in the network are calculated through backpropagation, and an optimizer is used to... The parameters are updated using a predetermined learning rate (set to 0.01). The learning rate is adjusted throughout the training process according to a predetermined strategy (such as decaying at a fixed number of epochs, cosine annealing, or adaptive adjustment based on validation performance, which are existing scheduling methods) to balance convergence speed and training accuracy. When the evaluation metric (such as classification accuracy or segmentation intersection-union ratio) no longer improves within a certain number of consecutive training epochs and the change is lower than the acceptable error range (set based on the numerical scale of the evaluation metric, the quality of the dataset annotation, and the requirements of downstream tasks for the stability of feature representation, which can be set to 0.005), it is considered to have reached saturation and pre-training is complete.

[0068] S1.5: Using the visual semantic feature vector, environmental geometric structure feature vector, and robot pose feature vector from the multimodal feature set as nodes, and the spatiotemporal and semantic relationships as edges, construct an environmental spatiotemporal graph;

[0069] Specifically, the visual semantic feature vectors, environmental geometric structure feature vectors, and robot pose feature vectors from the multimodal feature set are used as nodes in the graph structure. Connection edges are established between nodes based on the synchronization relationship of each feature vector in the time series and their spatial proximity. Semantic association edges are constructed based on the semantic consistency between the object category represented by the visual semantic feature vectors and the spatial structure described by the environmental geometric structure feature vectors. The weights of all edges are determined by a comprehensive measure of the spatiotemporal distance and semantic similarity of the corresponding nodes. The spatiotemporal distance reflects the proximity of the feature acquisition time to the spatial coordinates, while semantic similarity reflects the consistency between the visual category and the geometric structure at the scene understanding level. This forms an environmental spatiotemporal graph with multimodal feature vectors as nodes and spatiotemporal and semantic association relationships as edges.

[0070] S1.6: Apply graph neural networks to fuse nodes and edges, and use attention mechanisms to emphasize environmental elements that have a causal impact on environmental decisions, generating an environmental state vector;

[0071] Specifically, the nodes and edges in the spatiotemporal graph of the environment are fed into a graph neural network. The graph neural network uses a multi-layer message passing mechanism to enable each node to integrate the features of its neighboring nodes with the spatiotemporal or semantic relationships represented by the corresponding edges. In each layer of propagation, the attention mechanism assigns different emphasis weights to different nodes based on the degree of correlation between the node content and the target, so that environmental elements that have a causal impact on environmental decisions take the lead in feature updates. After multiple rounds of node feature updates, a global pooling operation is performed on the final representation of all nodes to form a single environmental state vector.

[0072] It should be noted that the attention mechanism, based on the degree of correlation between node content and the target, constructs a reference context by combining the robot's own pose feature vector (representing its current position and orientation) with the target position, and measures the similarity between this context and the visual semantic feature vector or environmental geometric feature vector corresponding to each node. This similarity measurement uses existing attention scoring methods in the form of dot product or weighted sum to obtain the original attention score for each node. The original attention scores of all nodes are normalized to obtain non-negative emphasis weights that sum to one. Nodes with larger emphasis weights have a stronger influence on neighboring nodes in the message aggregation of the graph neural network, thereby highlighting environmental elements that have a causal impact on navigation decisions.

[0073] It should be noted that the pre-training process of the graph neural network is carried out on a large-scale graph dataset containing node features, edge relationships, and corresponding task labels. Each graph sample in this dataset consists of nodes, edges, and global or node-level labels. During training, the graph samples are fed into the graph neural network, and neighborhood information is aggregated through multi-layer message passing to generate representations of nodes or graphs. Depending on the downstream task type (such as node classification, graph classification, or link prediction), an appropriate loss function is used to measure the difference between the prediction results and the true labels. Based on this loss function, the gradient of the learnable parameters in the graph neural network is calculated using the backpropagation algorithm, and the parameters are updated by the optimizer according to a predetermined learning rate (empirically determined based on the optimizer type and data size, for example, the initial learning rate can be set to 0.01). This process is repeated until the change in the evaluation metric of the graph neural network on the validation set is lower than the acceptable error range for multiple consecutive training cycles (set based on the historical fluctuation level of the validation set evaluation metric and the performance stability requirements of the task, for example, it can be set between 0.001 and 0.01), thus completing the pre-training.

[0074] S1.7: Apply motion continuity constraints to the environment state vector and perform joint optimization to form a multimodal environment state representation.

[0075] Specifically, the relationship between the environmental state vector and the robot's own pose feature vector changes over consecutive time steps is combined to apply a motion continuity constraint. This constraint requires that the environmental state vectors at adjacent time steps maintain a smooth transition in the representation space, avoiding abrupt changes. The joint optimization process uses the environmental state vector as the optimization variable, and on the basis of the graph neural network output, it simultaneously minimizes the multimodal feature consistency error and the regularization term defined by the motion continuity constraint. The multimodal feature consistency error reflects the degree of inherent coordination between the visual semantic feature vector, the environmental geometric structure feature vector, and the robot's own pose feature vector after fusion. The motion continuity constraint is reflected by the distance metric between environmental state vectors at adjacent time steps. By iteratively adjusting the target, a multimodal environmental state representation that satisfies spatiotemporal consistency and multimodal synergy is formed.

[0076] S2. Combine the multimodal environmental state representation with the robot's dynamic parameters to construct a virtual digital twin synchronized with the robot; generate multiple candidate motion trajectories based on the multimodal environmental state representation.

[0077] S2.1: Initialize the virtual digital twin based on the state layer, task layer, and policy layer of the hierarchical active reasoning architecture;

[0078] Specifically, the multimodal environment state representation is used as the initial basis to initialize the state layer of the hierarchical active reasoning architecture. The state layer contains a complete perceptual representation of the robot's current environment. Based on task instructions or target location information, combined with the content of the state layer, the task layer of the hierarchical active reasoning architecture is determined. The task layer represents the navigation or operation target that the robot needs to complete. Based on the state layer and the task layer, the strategy layer of the hierarchical active reasoning architecture is set according to the robot's dynamic parameters and motion capability constraints. The strategy layer contains feasible action space and control granularity. The three layers together constitute the initial structure of the virtual digital twin, enabling the virtual digital twin to maintain synchronization with the physical robot in terms of perception, target, and action.

[0079] S2.2: Synchronously input the multimodal environment state representation and robot body dynamics parameters into the state layer and task layer to infer the hidden state of the current environment and provide task context;

[0080] Specifically, the multimodal environmental state representation and robot body dynamics parameters are jointly provided to the state layer and task layer of the hierarchical active reasoning architecture. The state layer combines the visual semantic feature vector, environmental geometric feature vector, and robot pose feature vector contained in the multimodal environmental state representation to reflect the currently observable environmental information. The task layer forms a contextual description of the task execution conditions based on the task objective and the range of motion capabilities defined by the robot body dynamics parameters. Under the joint constraints of the state layer and the task layer, the hierarchical active reasoning architecture infers the hidden states in the current environment that are not directly observed but affect task execution, and provides the policy layer with a task context that includes the task objective, environmental context, and robot capability limitations.

[0081] S2.3: Input the hidden state and task context into the policy layer, perform policy reasoning and optimization, and construct a virtual digital twin synchronized with the robot.

[0082] Specifically, the hidden state and task context are provided to the policy layer of the hierarchical active reasoning architecture. Based on the unobserved environmental information described by the hidden state and the target requirements and motion constraints contained in the task context, the policy layer generates multiple candidate motion trajectories in the action space. Each candidate motion trajectory corresponds to a set of continuous pose sequences in the virtual digital twin, which are consistent with the robot's body dynamics parameters. Through policy reasoning within the hierarchical active reasoning architecture, the candidate motion trajectories are evaluated and adjusted. The evaluation of the candidate motion trajectory is based on the consistency between the environmental state changes caused by its execution in the virtual digital twin and the task objectives, whether the motion process violates the robot's body dynamics parameter constraints, and whether it effectively avoids potential risk areas revealed by the hidden state. The adjustment is achieved by eliminating candidate motion trajectories that do not meet the evaluation conditions and retaining trajectories that conform to the multi-objective optimization criteria, so that the state evolution of the virtual digital twin is synchronized with the perception, motion capabilities, and environmental interaction of the physical robot. The final virtual digital twin has the same environmental state, task objectives, and dynamic behavior as the physical robot.

[0083] It should be noted that the multi-objective optimization criterion refers to considering multiple performance indicators in trajectory evaluation, such as task completion accuracy (measured by endpoint deviation), motion safety (measured by minimum obstacle distance and dynamic constraint satisfaction), and energy consumption efficiency (measured by total joint torque amplitude as a proxy indicator), and selecting the candidate motion trajectory with the best overall performance by weighing these indicators.

[0084] S2.4: In the strategy layer of the virtual digital twin, with minimizing the expected free energy as the optimization objective, the evaluation function is defined as follows:

[0085] ;

[0086] In the formula, This represents the expected free energy, which serves as the evaluation function for strategy layer optimization. This represents the Kullback-Leibler divergence, used to measure the difference between two probability distributions; This represents the approximate posterior distribution of the hidden state sequence corresponding to the candidate motion trajectories generated by the policy layer; Indicates from time step 1 to time step 2. The robot's own pose feature vector sequence is derived from the multimodal environmental state representation; This represents the probability distribution of the robot's own pose feature vector sequence corresponding to the candidate motion trajectories generated by the policy layer; Indicates the location of a given navigation target. Under these conditions, the robot moves from time step 1 to time step 2. The probability distribution of the desired sequence of poses to be executed; this distribution is determined by the target information in the task context. The guidance reflects the reasonable movement trend that should be followed to complete the navigation task; express The mathematical expectation; Representation of state sequence The corresponding multimodal observation sequence includes the expected observations reconstructed from visual semantic feature vectors and environmental geometric feature vectors; Represents a given sequence of states Next observation sequence The negative log-likelihood reflects the consistency between the trajectory and environmental perception.

[0087] S2.5: Based on the multimodal environment state representation and evaluation function, the entropy regularization cross-entropy method is used for sampling and optimization in the policy space to generate multiple local optimal candidate policies;

[0088] Specifically, a set of initial candidate policies is randomly generated in the policy space, with each candidate policy corresponding to a sequence of robot pose feature vectors. The evaluation function values ​​of all candidate policies are sorted by evaluation function in ascending order, and a fixed number or proportion (e.g., the top 20 or top 15%) of candidate policies are selected to form an elite set. The local optimum region of the policy space is estimated based on the distribution of candidate policies in the elite set, and new candidate policies are generated in the vicinity of this region. At the same time, an entropy regularization term is added during the sampling process to maintain policy diversity and avoid premature convergence. The sampling, evaluation, and updating are repeated to obtain multiple locally optimum candidate policies that meet the evaluation function requirements and are significantly different from each other.

[0089] It should be noted that the policy space refers to the set of all possible sequences of robot pose feature vectors, each sequence corresponding to a feasible motion trajectory that satisfies the constraints of the robot's body dynamics parameters.

[0090] S2.6: Decode multiple locally optimal candidate strategies into corresponding robot control command sequences to form multiple candidate motion trajectories.

[0091] Specifically, each sequence of robot pose feature vectors from multiple locally optimal candidate strategies is converted into corresponding joint angle, angular velocity, and torque command sequences based on the motion constraints specified by the robot's body dynamics parameters using inverse kinematics, forming an executable robot control command sequence. Each robot control command sequence drives the physical robot to move along a specific path, corresponding to a candidate motion trajectory that is continuous in the policy space and satisfies dynamic feasibility. All candidate motion trajectories maintain spatiotemporal consistency with the multimodal environment state representation and reflect the behavior selection under different locally optimal strategies.

[0092] S3. Input the virtual digital twin with the multimodal environment state representation as the initial condition, perform parallel probabilistic deduction on multiple candidate motion trajectories, and output the predicted probability distribution corresponding to each candidate motion trajectory; based on the predicted probability distribution, select the optimal control command, and determine the predicted expected state representation from the predicted probability distribution corresponding to the optimal control command.

[0093] S3.1: Within the virtual digital twin, using the multimodal environment state representation as the initial state, and each candidate motion trajectory as the control input, parallel virtual physical deduction is initiated.

[0094] Specifically, within the virtual digital twin, using the multimodal environment state representation as the initial state, the robot control command sequence corresponding to each candidate motion trajectory is applied to the virtual digital twin. Based on the mass, inertia, and joint motion constraints described by the robot's body dynamics parameters, the temporal evolution of each candidate motion trajectory is advanced in a physical simulation environment that supports rigid body dynamics simulation. Each candidate motion trajectory drives an independent deduction process, simulating the robot's pose changes and environmental interaction results in the current environment. All deduction processes share the same multimodal environment state representation as the starting point and run synchronously at a unified time step, forming multiple parallel virtual physical deductions.

[0095] S3.2: In each parallel virtual physical simulation, a preset random perturbation is injected into the virtual digital twin, and multiple Monte Carlo motion simulations are performed to generate a set of future state samples corresponding to each candidate motion trajectory, which serves as the predicted probability distribution.

[0096] Specifically, in each parallel virtual physical simulation, a preset random perturbation is applied to the virtual digital twin. This perturbation is used in the robot control command sequence or environmental interaction to simulate actual execution deviations or environmental uncertainties. Based on the same candidate motion trajectory, Monte Carlo motion simulations are repeatedly run multiple times with different preset random perturbations injected. Each Monte Carlo motion simulation starts from the same multimodal environmental state representation and follows the motion constraints specified by the robot's body dynamics parameters to generate a future state sequence. The future state sequences obtained from all Monte Carlo motion simulations are collected to form a future state sample set corresponding to the candidate motion trajectory. This future state sample set represents the range of state evolution that the candidate motion trajectory may trigger in the future, serving as a predicted probability distribution.

[0097] It should be noted that random perturbation refers to a small perturbation quantity that conforms to a specific probability distribution and is introduced into a virtual digital twin to simulate sensor noise, execution error, or external interference in a real environment.

[0098] Figure 6The diagram shows a comparison between predicted trajectory error and actual trajectory error. The main plot at the top displays the overall trend of the two curves over time. The solid orange line represents the actual trajectory error, and the dashed blue line represents the predicted trajectory error. By observation, we can see the changing trends of the two at most time points. The magnified view at the bottom highlights the error differences in certain specific time periods, especially at the points where the differences between the two curves are greatest. The magnified area is marked by a red dashed box in the diagram. The diagram discusses the predicted probability distribution of each candidate motion trajectory generated by parallel virtual physics simulation, as well as how to select the optimal control command and evaluate the prediction results.

[0099] S3.3: Evaluate the predicted probability distribution corresponding to each candidate motion trajectory, select the optimal candidate motion trajectory, and take the robot control command of the first time step in the optimal candidate motion trajectory as the optimal control command.

[0100] Specifically, the predicted probability distributions corresponding to each candidate motion trajectory are evaluated based on the constraints imposed by the target and robot body dynamics parameters. The evaluation criteria include the proximity of the future state sequence in the predicted probability distribution to the target position, the minimum distance guarantee to obstacles, and the amplitude level of joint torque data. Candidate motion trajectories that meet the requirements of mission safety and feasibility are sorted according to the evaluation criteria, and the one with the best performance is selected as the optimal candidate motion trajectory. In the robot control command sequence corresponding to the optimal candidate motion trajectory, the pose change and torque command of the first time step constitute the robot control command of the time step. The robot control command of the first time step is used to drive the physical robot to execute.

[0101] It should be noted that the candidate motion trajectories that meet the requirements of mission safety and feasibility are sorted according to the evaluation criteria as follows: Candidate motion trajectories whose predicted probability distribution shows any future state sample with a distance less than a safety threshold (the minimum obstacle avoidance margin determined by the robot's geometric dimensions, environmental obstacle positioning accuracy, and motion control response delay; an exemplary value range of 0.2 meters to 0.5 meters, representing the physical safety boundary required for both passage efficiency and collision avoidance in common mobile or maneuvering robots in indoor environments) are excluded. Among the remaining trajectories, they are sorted in ascending order of the average distance between the endpoint pose and the navigation target position. If the distances are similar, they are further sorted in ascending order of the total amplitude of the joint torque data. The candidate motion trajectory ranked highest is selected as the optimal one.

[0102] S3.4: Calculate the expected value of multiple future state trajectories in the predicted probability distribution corresponding to the optimal control command, and determine the expected value as the representation of the predicted expected state. The expression is:

[0103] ;

[0104] In the formula, This represents the predicted expected state, and the predicted expected state corresponding to the optimal control command. The expected value is the average of the trajectories of multiple future state samples, representing the value at time step [missing information]. The desired state at any given moment; This represents the number of future state sample trajectories generated by Monte Carlo motion simulation using the optimal control command. The reciprocal of the number of future state sample trajectories; Indicates the index of the trajectory of a future state sample; Indicates the first The future state sample trajectory at time step The state at time step, that is, the trajectory of each future state sample at time step. The state after multimodal data fusion, including visual, spatial point cloud and torque data.

[0105] S4. Send the optimal control command to the robot for execution and generate a multimodal environmental state representation after execution; calculate the state deviation value between the multimodal environmental state representation after execution and the predicted expected state representation.

[0106] S4.1: Send the optimal control command to the robot for execution, and collect various sensor data after confirming that the physical action has been completed;

[0107] Specifically, the robot control command in the first time step of the optimal control command, which includes joint angle, angular velocity, and torque commands, is sent to the robot actuator to drive the robot to complete the corresponding action. During the robot's execution, the built-in state feedback mechanism monitors whether the joint movement has reached the target position and speed set by the command. When the deviation between the actual pose of all joints and the command requirements is less than the preset tolerance and the speed approaches zero, the physical action is determined to be completed. Simultaneously, multiple sensors are triggered to collect data, including image data acquired by a vision sensor, spatial point cloud data acquired by a LiDAR or depth camera, and joint angle and angular velocity information returned by the joint encoder.

[0108] It should be noted that the built-in state feedback mechanism refers to a closed-loop monitoring method in which the robot obtains its own pose and motion state in real time through body sensors such as joint encoders and inertial measurement, and uses this data to determine whether the current action meets the requirements of the command.

[0109] The preset tolerance is set based on the robot's joint positioning accuracy, control cycle response delay, and the task's requirements for pose accuracy. The specific steps are as follows: obtain the joint encoder resolution and repeatability error calibrated by the robot at the factory; then, combine the measured steady-state error of the control algorithm under typical loads, and use the product of the sum of the two and a safety factor (usually 1.2 to 1.5) as the preset tolerance; the exemplary value range is 0.005 to 0.02, and the value is based on engineering experience in balancing execution efficiency and motion reliability in precision operation or navigation tasks of industrial and service robots.

[0110] S4.2: Perform spatiotemporal alignment and feature extraction on the multi-sensor data after execution to generate a multimodal feature set after execution. Convert the multimodal feature set after execution into feature vectors and fuse them to form a multimodal environmental state representation after execution.

[0111] Specifically, after execution, the data from multiple sensors are synchronized in time based on a unified timestamp. Based on pre-calibrated multi-sensor extrinsic parameters, the pixel coordinates in the visual data, the 3D point coordinates in the spatial point cloud data, and the joint coordinates corresponding to the torque data are transformed into the robot base coordinate system through a rigid body transformation matrix, forming spatiotemporally aligned multi-sensor data. The spatiotemporally aligned multi-sensor data is then normalized, mapping the pixel intensity values ​​of the visual data to a unified numerical range, converting the spatial point cloud data into relative coordinates centered on the robot's current position and scaling it to a consistent scale, and compressing the torque data proportionally according to the maximum working torque range to generate standardized multimodal data. Standardized multimodal data is fed into a pre-trained convolutional neural network and a pre-trained point cloud feature extraction network to extract visual semantic feature vectors and environmental geometric structure feature vectors, respectively. At the same time, the robot's own pose feature vector is extracted from the standardized multimodal data to form a multimodal feature set. The three types of feature vectors in the multimodal feature set are used as nodes and spatiotemporal and semantic relationships are used as edges to construct an environmental spatiotemporal graph. A graph neural network is applied to fuse nodes and edges and an attention mechanism is used to emphasize environmental elements that have a causal impact on environmental decisions to generate an environmental state vector. Motion continuity constraints are applied to the environmental state vector and joint optimization is performed to form a multimodal environmental state representation after execution.

[0112] S4.3: Map the multimodal environment state representation after execution and the predicted expected state representation to the same Riemannian manifold space to obtain the corresponding manifold space representation;

[0113] Specifically, the executed multimodal environment state representation and the predicted expected state representation are fed into a pre-trained nonlinear embedding mapper. The nonlinear embedding mapper consists of three fully connected neural networks, each followed by a ReLU activation function. The output layer is normalized to the unit norm to fit the spherical manifold, which can project high-dimensional state vectors onto a Riemannian manifold space with geometric constraints. The Riemannian manifold space is a unit spherical manifold, where the geodesic distance between any two points is determined by the radian value of the angle between the corresponding vectors. In the unit spherical manifold, the geodesic distance is equal to the inverse cosine function of the angle between the two vectors. In this space, the distance metric between states follows the geodesic criterion of the manifold, reflecting semantic and dynamic similarity. The executed multimodal environment state representation is mapped to the corresponding executed manifold space representation, and the predicted expected state representation is mapped to the corresponding expected manifold space representation. The two manifold space representations are located in the same Riemannian manifold space, possessing comparability and geometric consistency.

[0114] It should be noted that the pre-training process of the nonlinear embedding mapper is carried out on a dataset containing a large number of multimodal environmental state representation samples. This dataset is generated by fusing visual semantic feature vectors, environmental geometric structure feature vectors, and robot pose feature vectors collected during historical operations. During training, a self-supervised contrastive learning strategy is adopted, constructing positive sample pairs for state representations from different sensor perspectives or adjacent time steps within the same time window, and constructing negative sample pairs for state representations from irrelevant time steps or different scenes. The loss function adopts the normalized temperature-scaled cross-entropy form, with the cosine similarity of positive sample pairs as the numerator. The sum of the similarities of all negative sample pairs is used as the denominator. Minimizing this loss function achieves the aggregation of positive samples on the unit spherical manifold and the separation of negative samples. The objective is to minimize the geodesic distance of positive sample pairs in the Riemannian manifold space and maximize the geodesic distance of negative sample pairs. During training, backpropagation is used to adjust the parameters of the nonlinear embedding mapper, learning to map the original state representation to the geometrically structured Riemannian manifold space. Pre-training is completed when the average distance difference between positive and negative sample pairs on the validation set changes less than the acceptable error range (which can be set to 0.005) for several consecutive periods. S4.4: Calculate the geodesic distance between the two manifold space representations and use the geodesic distance as the state bias value. The expression is:

[0115] ;

[0116] In the formula, This represents the distance to the geodesic line, which is also the state deviation value. Represents the inverse cosine function; The manifold space representation obtained after the execution of the multimodal environment state representation is mapped by a pre-trained nonlinear embedding mapper. The manifold space representation is obtained by mapping the predicted expected state representation to the same pre-trained nonlinear embedding mapper. This represents the transpose operation of a vector; express and The inner product of.

[0117] S5. Use time-series causal inference to analyze the contribution of each sensor's data to the state deviation value and adjust the fusion weight of each sensor's data.

[0118] S5.1: For the current control cycle, based on the multimodal sensor data and state deviation values ​​generated in this cycle, use time-series causal inference to analyze the causal contribution of each sensor data to the state deviation values;

[0119] Specifically, for the current control cycle, based on the various sensor data and state deviation values ​​generated in this cycle, the Granger causality test method is used to evaluate the causal contribution of each sensor data to the state deviation value. The historical observations of visual data, spatial point cloud data, and torque data in their respective time series are used as candidate causal variables, and the time series of state deviation values ​​is used as the outcome variable. On the basis of unified time alignment, it is tested whether adding the historical values ​​of a certain type of sensor data improves the predictive ability of future state deviation values ​​given the knowledge of past state deviation values. If the predictive ability is improved, it is determined that this type of sensor data has a causal contribution to the state deviation value. By comparing the degree of improvement in predictive ability corresponding to various types of sensor data, the magnitude of their causal contribution is determined.

[0120] It should be noted that the Granger causality test is a time-series-based statistical test used to determine whether historical observations of one variable help improve the prediction accuracy of future values ​​of another variable. If they do, the former is said to have Granger causality towards the latter.

[0121] For example, given the state deviation value sequence of the past five control cycles, adding only historical values ​​of visual data (such as the visual semantic feature vector of each step) yields a residual variance of 0.018 for predicting future state deviation values. However, without adding visual data, the residual variance of the baseline model constructed solely using the historical state deviation values ​​is 0.025. Since the residual variance decreases after adding visual data, it indicates that visual data has a causal contribution to the state deviation values. Similarly, adding spatial point cloud data reduces the residual variance to 0.020, and adding torque data reduces it to 0.023. Comparing the three, visual data brings the largest decrease in variance (0.007), thus its causal contribution is the highest, followed by spatial point cloud data, and then torque data.

[0122] Figure 7Scatter plots were used to illustrate the relationships between visual data and spatial point cloud data, and between visual data and torque data. Blue and red dots represent the relationships between these two types of data, respectively, helping to analyze the contribution of different sensor data to the overall robot control. Through these data points, the roles of visual data, point cloud data, and torque data in robot decision-making can be observed. The discussion focuses on how to fuse multimodal data into the spatiotemporal plot of the environment, and how to use temporal causal inference to analyze the contribution of each sensor data to the state deviation value.

[0123] S5.2: Based on the causal contribution of each sensor data, adjust the fusion weight of each sensor data in real time during the multimodal feature set generation process of the next control cycle.

[0124] Specifically, based on the causal contribution of each sensor data, the fusion weights in the multimodal feature set generation process in the next control cycle are adjusted; the causal contribution of visual data, spatial point cloud data, and torque data are used as the initial weights for each in the feature fusion stage; the higher the causal contribution, the greater the proportion of the corresponding sensor data in the generation of visual semantic feature vectors, environmental geometric structure feature vectors, and robot pose feature vectors; before constructing the environmental spatiotemporal graph, the three types of feature vectors are normalized according to their causal contribution to form a weighted fusion coefficient, and this weight tendency is maintained in the message passing and attention mechanism of the graph neural network, so that the mode with high causal contribution dominates in the state representation.

[0125] In summary, this invention achieves precise fusion of multimodal sensor data in complex environments by generating multimodal environmental state representations, thereby improving the robot's autonomous decision-making capabilities and efficiency. In particular, by constructing a virtual digital twin synchronized with the robot, the robot's motion trajectory in various dynamic environments can be accurately simulated. This not only improves the prediction accuracy of the motion trajectory but also allows for adjustments to the fusion weights of various sensor data based on real-time feedback data, thereby optimizing the robot's control strategy and improving navigation accuracy, motion efficiency, and stability. Especially in industrial control environments, it exhibits stronger adaptability and responsiveness.

[0126] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A control method for an embodied intelligent robot based on multimodal information fusion, characterized in that: include, The robot collects data from multiple sensors in real time, performs spatiotemporal alignment and feature extraction to generate a multimodal feature set; the multimodal feature set is converted into feature vectors and fused to form a multimodal environmental state representation. By combining multimodal environmental state representation with robot dynamic parameters, a virtual digital twin synchronized with the robot is constructed; based on the multimodal environmental state representation, multiple candidate motion trajectories are generated. The virtual digital twin is input with the multimodal environment state representation as the initial condition. Parallel probabilistic deductions are performed on multiple candidate motion trajectories, and the predicted probability distributions corresponding to each candidate motion trajectory are output. Based on the predicted probability distributions, the optimal control command is selected, and the predicted expected state representation is determined from the predicted probability distributions corresponding to the optimal control command. The optimal control command is sent to the robot for execution, generating a multimodal environmental state representation after execution; the state deviation value between the multimodal environmental state representation after execution and the predicted expected state representation is calculated. The contribution of each sensor's data to the state deviation value is analyzed using time-series causal inference, and the fusion weights of each sensor's data are adjusted accordingly.

2. The embodied intelligent robot control method based on multimodal information fusion as described in claim 1, characterized in that: The steps for generating the multimodal feature set are as follows: The various sensor data include visual data, spatial point cloud data, and torque data; Spatiotemporal alignment and unified coordinate transformation are performed on data from multiple sensors to obtain spatiotemporally aligned multi-sensor data. Normalize the spatiotemporally aligned multisensor data to generate standardized multimodal data; Visual semantic feature vectors, environmental geometric feature vectors, and robot pose feature vectors are extracted from standardized multimodal data to generate a multimodal feature set.

3. The embodied intelligent robot control method based on multimodal information fusion as described in claim 2, characterized in that: The steps for forming a multimodal environmental state representation are as follows: The visual semantic feature vector, environmental geometric structure feature vector, and robot pose feature vector in the multimodal feature set are used as nodes, and the spatiotemporal and semantic relationships are used as edges to construct an environmental spatiotemporal graph. By applying a graph neural network to fuse nodes and edges, and using an attention mechanism to emphasize environmental elements that have a causal impact on environmental decisions, an environmental state vector is generated. By imposing motion continuity constraints on the environmental state vector and performing joint optimization, a multimodal environmental state representation is formed.

4. The embodied intelligent robot control method based on multimodal information fusion as described in claim 1, characterized in that: The steps for constructing a virtual digital twin synchronized with the robot are as follows: The virtual digital twin is initialized based on the state layer, task layer, and policy layer of the hierarchical active reasoning architecture; The multimodal environmental state representation and robot body dynamics parameters are synchronously input into the state layer and task layer to infer the hidden state of the current environment and provide task context. The hidden state and task context are input into the policy layer for policy reasoning and optimization, thereby constructing a virtual digital twin synchronized with the robot.

5. The embodied intelligent robot control method based on multimodal information fusion as described in claim 4, characterized in that: The steps for generating multiple candidate motion trajectories are as follows: In the strategy layer of the virtual digital twin, an evaluation function is defined with minimizing the expected free energy as the optimization objective; Based on the multimodal environment state representation and evaluation function, the entropy regularization cross-entropy method is used for sampling and optimization in the policy space to generate multiple local optimal candidate policies. Multiple locally optimal candidate strategies are decoded into corresponding robot control command sequences, forming multiple candidate motion trajectories.

6. The embodied intelligent robot control method based on multimodal information fusion as described in claim 1, characterized in that: The steps for outputting the predicted probability distribution for each candidate motion trajectory are as follows: Within the virtual digital twin, using the multimodal environment state representation as the initial state and each candidate motion trajectory as the control input, parallel virtual physical deduction is initiated. In each parallel virtual physical simulation, a preset random perturbation is injected into the virtual digital twin, and multiple Monte Carlo motion simulations are performed to generate a set of future state samples corresponding to each candidate motion trajectory, which serves as the predicted probability distribution.

7. The embodied intelligent robot control method based on multimodal information fusion as described in claim 6, characterized in that: The steps for selecting the optimal control command based on the predicted probability distribution and determining the predicted desired state representation from the predicted probability distribution corresponding to the optimal control command are as follows: The predicted probability distributions corresponding to each candidate motion trajectory are evaluated, the optimal candidate motion trajectory is selected, and the robot control command at the first time step of the optimal candidate motion trajectory is taken as the optimal control command. Calculate the expected values ​​of multiple future state trajectories in the predicted probability distribution corresponding to the optimal control command, and determine the expected values ​​as the predicted expected state representation.

8. The embodied intelligent robot control method based on multimodal information fusion as described in claim 1, characterized in that: The steps for generating the post-execution multimodal environment state representation are as follows: The optimal control command is sent to the robot for execution, and after confirming that the physical action has been completed, various sensor data are collected. Spatiotemporal alignment and feature extraction are performed on the multi-sensor data after execution to generate a multimodal feature set after execution. The multimodal feature set after execution is converted into feature vectors and fused to form a multimodal environmental state representation after execution.

9. The embodied intelligent robot control method based on multimodal information fusion as described in claim 8, characterized in that: The steps for calculating the state deviation between the multimodal environment state representation after execution and the predicted expected state representation are as follows: The multimodal environment state representation after execution and the expected state representation after prediction are mapped to the same Riemannian manifold space to obtain the corresponding manifold space representation. Calculate the geodesic distance between the two manifold spatial representations and use the geodesic distance as the state deviation value.

10. The embodied intelligent robot control method based on multimodal information fusion as described in claim 1, characterized in that: The steps for adjusting the fusion weights of the data from each sensor are as follows: For the current control cycle, based on the multimodal sensor data and state deviation values ​​generated in this cycle, time-series causal inference analysis is used to determine the causal contribution of each sensor data to the state deviation values. Based on the causal contribution of each sensor data, the fusion weights of each sensor data during the multimodal feature set generation process in the next control cycle are adjusted in real time.

Images