A Robot End-to-End Closed-Loop Control Method and System Based on Multimodal Perception Fusion
By employing an end-to-end closed-loop control method based on multimodal perception fusion, the problem of multimodal sensor data access and fusion was solved, enabling the robot to operate efficiently and stably in dynamic environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 重庆中科汽车软件创新中心
- Filing Date
- 2026-04-16
- Publication Date
- 2026-06-26
AI Technical Summary
In existing robot control systems, multimodal sensor data cannot be uniformly accessed and fused, resulting in large perception errors, high decision delays, and difficulty in stable operation in dynamic environments.
An end-to-end closed-loop control method using multimodal sensing fusion is adopted. The sensing access layer realizes standardized access of multi-type sensor data, the fusion processing layer performs spatiotemporal alignment and multimodal fusion, the decision control layer generates execution commands, and the execution feedback layer forms closed-loop control.
It achieves real-time and accurate fusion of multimodal data, reduces perception errors and decision delays, and improves the robot's adaptability and robustness in complex environments.
Smart Images
Figure CN122274973A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot control technology, specifically to a robot end-to-end closed-loop control method and system based on multimodal perception fusion. Background Technology
[0002] Robotics technology is rapidly developing in fields such as autonomous navigation, industrial grasping, and service robots, placing extremely high demands on the perception accuracy, decision-making speed, and system robustness of robots in dynamic and complex environments. To achieve a comprehensive understanding and precise control of the environment, multimodal perception fusion has become the core development direction of current robot control systems. An ideal robot control system should possess end-to-end closed-loop control capabilities, from multi-source sensor data acquisition to the final actuator actions, to achieve highly integrated and intelligent autonomous operation.
[0003] However, in the existing robot control technology system, there are still many problems in achieving efficient end-to-end closed-loop control.
[0004] Currently, core components of robots (sensors, actuators, controllers) generally come from different suppliers, and there are significant differences in hardware interfaces and driving protocols, making it difficult to form a unified control logic at the operating system level. For example, LiDAR and vision cameras usually require independent driving modules for management. This hardware-level fragmentation makes it impossible for the operating system to achieve coordinated scheduling of multimodal data at the underlying level, making it difficult to build a unified perception base.
[0005] Under the existing technical architecture, various sensors employ different data transmission protocols, such as ROS LaserScan messages, PROFINET industrial Ethernet, and USB 3.0 Vision. Due to incompatible communication standards, data sources of different modes cannot directly communicate with each other. Current robot control solutions often employ a combination of large-scale system coordination and independent decision-making by multiple modules. For example, referencing NASA's NASREM layered architecture, there is significant hierarchical delay in command transmission between the organization, coordination, and execution levels. More critically, each perception module, such as vision and radar, typically outputs its judgment independently, lacking a mechanism for unified arbitration of multimodal fusion results. When perceptual conflicts arise between different modal sensors (e.g., vision identifies an "obstacle" while lidar determines it as "interference"), existing systems struggle to perform effective logical arbitration and decision fusion, leading to misjudgments or system freezes in complex scenarios.
[0006] Because the perception, decision-making, and execution processes are loosely coupled in the system architecture, the state information of the actuators generally cannot be fed back to the perception fusion layer in real time and without loss. This open-loop or semi-closed-loop control mode prevents the robot from dynamically adjusting the perception strategy and fusion weights based on the execution results, resulting in poor adaptability and robustness of the system in dynamic environments. Summary of the Invention
[0007] The purpose of this invention is to propose an end-to-end closed-loop control method and system for robots based on multimodal perception fusion. This technical solution can solve the problem of standardized access and real-time fusion of multi-source heterogeneous data, and build a closed-loop control architecture with high fault tolerance and low latency.
[0008] To achieve the above objectives, in a first aspect, the present invention proposes an end-to-end closed-loop control method for robots based on multimodal perception fusion, comprising: The sensing access layer is used to enable standardized access to data from multiple types of sensors; The fusion processing layer is used to perform spatiotemporal alignment and multimodal fusion processing on standardized access data; The decision control layer is used to generate execution instructions based on the fusion processing results. An execution feedback layer is used to feed back the execution status to the fusion processing layer to form closed-loop control.
[0009] Beneficial effects of the basic solution: This technical solution achieves standardized access to data from multiple types of sensors through the perception access layer, breaking the limitations of single-sensor perception. Then, through the fusion processing layer, it completes spatiotemporal alignment and multimodal information fusion, which can complement the perception defects of different sensors and eliminate perception errors caused by environmental interference, occlusion, noise, etc., making the robot's recognition of the environment, target and its own state more accurate and comprehensive.
[0010] By adopting an end-to-end control architecture, the multi-stage data conversion and redundant processing procedures in traditional segmented control are eliminated. The integrated and processed information can be directly transmitted to the decision control layer to generate instructions, reducing data transmission and calculation latency, effectively improving the robot's decision-making speed, and meeting the operational requirements of rapid response and real-time control in dynamic scenarios.
[0011] Multimodal fusion can avoid control failure caused by the failure or anomaly of a single sensor. At the same time, the execution feedback layer transmits the execution status back to the fusion processing layer in real time, forming a closed-loop correction mechanism that can dynamically correct perception deviations and decision-making errors. This allows the robot to operate stably in complex environments with unstructured, strong interference, and changing working conditions, reducing the risk of loss of control and erroneous actions.
[0012] From perception and access, fusion processing to decision execution and feedback correction, the entire chain is connected, and data at all levels flows seamlessly and works in synergy. This solves the problems of disconnect between perception and control and poor module coordination in traditional robots. While simplifying the system architecture, it improves the overall control fluency and operational reliability.
[0013] As a feasible preferred embodiment, the sensing access layer includes: The sensing interface module is equipped with a general hardware interface matrix, which supports the simultaneous access of multiple types of sensors. A routing conversion module is used to convert the communication protocol data of the access device into a system-unified perception data frame format, and to route the standardized data to the corresponding processing module of the fusion processing layer according to the data type; and The signal conditioning module is used to amplify, filter, and perform analog-to-digital conversion on analog signals.
[0014] As a feasible preferred solution, the routing conversion module automatically identifies the communication protocol of the access device by matching data packet features. The perceived data frame format includes device ID, timestamp, data type, and payload field.
[0015] As a feasible preferred embodiment, the fusion processing layer includes: The spatiotemporal alignment module is used to unify the timestamps of each sensor to the system reference clock, and to map the spatial data of each sensor to the robot base coordinate system based on the preset sensor extrinsic matrix. The multimodal fusion module is used to extract features from data of each modality, dynamically allocate fusion weights for each feature through an attention mechanism, and perform target recognition and state estimation based on the fused feature map. The data caching module is used to cache fused data and provide data input to the decision control layer.
[0016] As a feasible preferred solution, the spatiotemporal alignment module uses a precise clock synchronization protocol to achieve timestamp unification and uses linear interpolation to compensate for data with delay differences.
[0017] As a feasible preferred embodiment, the multimodal fusion module fuses the confidence data from each sensor to generate a comprehensive confidence score.
[0018] As a feasible preferred embodiment, the decision control layer includes: The intent parsing module is used to receive user commands and parse task intents; The end-to-end control module takes the fused feature map and task intent as input and directly outputs the sequence of execution instructions. The security verification module is used to pre-verify the generated execution instructions and trigger instruction correction when there are security risks.
[0019] As a feasible preferred solution, the security verification module uses a collision detection algorithm to determine whether there are obstacles on the instruction path. If there are, it calls a path planning algorithm to replan the path.
[0020] As a feasible preferred embodiment, the execution feedback layer includes: An actuator drive module is used to convert the instructions output by the decision control layer into control signals that can be recognized by the actuator. The perception feedback module is used to collect execution status data in real time and feed it back to the fusion processing layer for comparison and verification; The closed-loop adjustment module is used to generate compensation commands and add them to the control commands of the next cycle when there is a deviation between the feedback data and the command target.
[0021] Secondly, this application proposes a robot end-to-end closed-loop control system based on multimodal perception fusion, which utilizes the aforementioned robot end-to-end closed-loop control method based on multimodal perception fusion, including the following steps: Multiple types of sensors simultaneously collect environmental data, and the data from different protocols are uniformly converted into standardized sensing data frames and routed to the fusion processing layer; Multimodal data is unified to the robot's base coordinate system, features of each modality are extracted, and fusion weights are dynamically allocated through an attention mechanism to output a fused feature map and identify the target state. The task intent is parsed, and the execution instruction sequence is directly output based on the fusion result. The execution instructions are then subjected to security verification. The execution instructions are converted into control signals to drive the actuator, the execution status data is collected in real time and fed back, and when a deviation is detected, a compensation instruction is generated for dynamic correction to complete the closed-loop control. Attached Figure Description
[0022] Figure 1 This is a schematic diagram of the architecture of an end-to-end closed-loop control system for a robot based on multimodal perception fusion. Detailed Implementation
[0023] To make the technical solution and advantages of this application clearer, the technical solution of the present invention will be further described in detail below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only some embodiments of the present invention, and are only used to explain this application, not to limit it. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered isolated; they can be combined with each other to achieve better technical effects. The same reference numerals appearing in the accompanying drawings of the following embodiments represent the same features or components, and can be applied to different embodiments.
[0024] Furthermore, unless otherwise defined, the technical or scientific terms used in this invention description shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains.
[0025] The present invention will now be described in further detail with reference to the accompanying drawings.
[0026] Reference Figure 1 This disclosure provides an end-to-end closed-loop control system for a robot with multimodal perception fusion, including a perception access layer, a fusion processing layer, a decision control layer, and an execution feedback layer.
[0027] The sensing access layer is used to achieve standardized access to data from multiple types of sensors, including a sensing interface module, a routing conversion module, and a signal conditioning module.
[0028] The sensing interface module is equipped with a general-purpose hardware interface matrix, covering mainstream industrial interfaces such as Ethernet (Gigabit / 10 Gigabit), USB 3.2, PCIe 4.0, and CAN FD. It supports simultaneous access for various types of devices, including LiDAR (such as Velodyne 16-line), vision sensors (RGB cameras, ToF cameras), IMUs, and force sensors. The number of interfaces can be flexibly expanded through expansion slots, supporting up to 16 parallel connections.
[0029] The routing conversion module is used to automatically identify the communication protocol of the access device through packet feature matching, such as ROS 1 / 2, PROFINET, EtherCAT, etc. It is also used to convert non-standard protocol data into a system-wide unified perception data frame format. The frame structure includes device ID, timestamp, data type, and payload fields, with a conversion latency of ≤1ms. Furthermore, it is used to route standardized data to the corresponding processing module in the fusion processing layer based on the data type; for example, visual data to the image preprocessing module, and point cloud data to the point cloud processing module.
[0030] The signal conditioning module provides signal amplification, filtering (using Kalman filtering to suppress noise), and A / D conversion functions for analog signal sensors (such as some force sensors). It converts analog signals into digital signals before they are connected to the protocol routing module, ensuring signal integrity. For example, it amplifies the mV-level signal output by a force sensor by 1000 times and filters out 50Hz power frequency interference, achieving a conversion accuracy of 16 bits. It employs a configurable protocol rule base, supporting online updates of new protocols (adding protocol parsing scripts via OTA), allowing adaptation to future new sensor types without hardware reconstruction.
[0031] The fusion processing layer is used to realize data fusion and solve the problems of data spatiotemporal asynchrony and low fusion degree in existing technologies. It includes a spatiotemporal alignment module, a multimodal fusion module, and a data caching module.
[0032] The spatiotemporal alignment module is used to unify the timestamps of all connected sensors to the system reference clock, with a clock deviation ≤50μs. In this embodiment, the IEEE 1588 PTP precise clock synchronization protocol is adopted. Linear interpolation is used for time compensation of data with delay differences to ensure consistency of data from different modalities in the time dimension, as shown in the following formula:
[0033] in, The timestamp of the point to be interpolated. and For the timestamps of known data points, and Given the values of the data points, This is the interpolation result.
[0034] Based on a pre-defined sensor extrinsic parameter matrix (obtained through hand-eye calibration, such as camera and robotic arm calibration in an eye-in-hand architecture), the spatial data of each sensor is mapped to the robot's base coordinate system. For example, the depth data from the ToF camera and the point cloud data from the LiDAR are fused to eliminate spatial offset errors, achieving a registration accuracy of ≤±3mm.
[0035] The multimodal fusion module is used to extract features from the preprocessed modal data. Specifically, visual features are extracted from image data using ResNet-50, 3D features from point cloud data using PointNet, and pose features from IMU data. An attention mechanism dynamically allocates the fusion weights for each feature; for example, the weights of LiDAR features are increased in bright light conditions, while the weights of ToF camera features are increased in low-light conditions, resulting in a fused feature map.
[0036]
[0037] in, Let N be the attention score for the i-th feature, and N be the total number of features. is the fusion weight for the i-th feature.
[0038] The multimodal fusion module also uses an improved YOLOv8 algorithm based on the fused feature map to achieve target recognition and state estimation, such as recognizing a blue part and outputting its spatial coordinates and size information. At the same time, it integrates the confidence data of each sensor. For example, if the confidence score of visual recognition is 0.85 and the confidence score of LiDAR recognition is 0.92, the overall confidence score is taken as 0.89, reducing the risk of misjudgment by a single sensor.
[0039] The data caching module is configured with a high-speed cache pool and uses a FIFO scheduling strategy to cache fused data, providing stable data input for the decision control layer. It also supports abnormal data backtracking (caching the raw data and fusion results of the most recent 10 minutes) to facilitate troubleshooting.
[0040] The decision control layer is used to realize the direct mapping from the fusion perception results to the execution instructions, including the intent parsing module, the end-to-end control module, and the security verification module.
[0041] The intent parsing module receives user commands (supporting natural language, API interface, and teach pendant input methods) and parses the task intent using the BERT language model. For example, placing the red box from shelf A onto conveyor belt B is parsed as target recognition (red box), path planning (shelf A → conveyor belt B), and action execution (grab-move-place).
[0042] The end-to-end control module employs an improved VLA (Vision-Language-Action) multimodal learning model, taking fused feature maps and task intent as input and directly outputting a sequence of execution commands. The command formats include chassis control commands and robotic arm control commands.
[0043] The security verification module has a built-in security rule base to pre-verify the generated execution instructions. For example, it uses a collision detection algorithm (based on OctoMap environment modeling) to determine whether there are obstacles in the instruction path. If obstacles are found, it triggers instruction correction (calling an improved RRT algorithm to replan the path) to ensure execution safety.
[0044] The execution feedback layer realizes real-time feedback of the execution status and dynamic adjustment of control commands, forming a closed-loop control link, including the actuator drive module, the perception feedback module, and the closed-loop adjustment module.
[0045] The actuator drive module converts commands output from the decision control layer into control signals recognizable by the actuators. For example, it converts joint angle commands into servo motor PWM signals and speed commands into motor pulse signals. It supports various actuator types, including servo motors, stepper motors, and pneumatic grippers, with a drive latency of ≤5ms.
[0046] The perception feedback module collects execution status data (such as the actual position of the robotic arm and the gripping force) in real time through encoders (position accuracy ±0.01mm) and force sensors installed on the actuator, and feeds the data back to the fusion processing layer for comparison and verification with the perception data.
[0047] The closed-loop adjustment module is used to activate the PID adjustment mechanism when there is a deviation between the feedback data and the target command. Specifically, this includes calculating the deviation value (current position - target position); generating a compensation command using the PID algorithm; and superimposing the compensation command into the control command of the next cycle to achieve dynamic correction and ensure control accuracy. The PID algorithm formula is as follows:
[0048] in, To control the output, This is the deviation value. , and These are the proportional, integral, and differential coefficients, respectively.
[0049] This disclosure also provides a multimodal sensing fusion end-to-end closed-loop control method, including the following:
[0050] Perception access phase (T0-T1, T1-T0≤2ms): LiDAR, RGB camera, and ToF camera simultaneously collect environmental data. The protocol routing module converts the EtherCAT protocol data of the LiDAR and the USB3.0 Vision protocol data of the camera into standardized perception data frames and routes them to the fusion processing layer.
[0051] Fusion processing stage (T1-T2, T2-T1≤50ms): The spatiotemporal alignment unit unifies the multimodal data to the robot base coordinate system (time synchronization deviation <50μs, spatial registration accuracy ±2mm). The fusion engine extracts features and dynamically allocates weights through the attention mechanism (due to sufficient lighting, the weight of visual features is set to 0.6, and the weight of LiDAR features is set to 0.4). The fused feature map is output and the spatial coordinates of the red part are identified (x=5.2m, y=3.1m, z=0.8m).
[0052] Decision control phase (T2-T3, T3-T2≤30ms): The intent parsing module parses the user command into a "grab-move-place" sub-task. The end-to-end control model outputs the command sequence based on the fusion result: "Chassis moves to (5.2m, 3.1m) → robotic arm extends to z=0.8m → gripping force 60N → move to target position (2.0m, 1.5m) → release". The safety verification module confirms that there are no obstacles on the path.
[0053] Execution feedback phase (T3-T4, T4-T3 is the execution cycle, feedback once every 10ms): The actuator drive module controls the chassis and robotic arm to execute commands, and the status feedback module collects the position data of the robotic arm in real time. When the offset of the grasped part is detected (deviation 1.5mm), the closed-loop adjustment module generates compensation commands through PID algorithm to correct the posture of the robotic arm and ensure placement accuracy.
[0054] Closed-loop completion (T4): The part is successfully placed in the target position, the system outputs a task completion signal, and at the same time stores the raw data, fusion results and execution logs in the local database, completing one closed-loop control cycle.
[0055] This disclosure also provides a multimodal perception fusion robot end-to-end closed-loop control method, which utilizes the aforementioned multimodal perception fusion robot end-to-end closed-loop control system.
[0056] The above content is merely an embodiment of the present invention. Commonly known structures and characteristics of the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can improve and implement this solution based on the guidance provided in this application and their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.
Claims
1. An end-to-end closed-loop control system for a robot based on multimodal perception fusion, characterized in that, include: The sensing access layer is used to enable standardized access to data from multiple types of sensors; The fusion processing layer is used to perform spatiotemporal alignment and multimodal fusion processing on standardized access data; The decision control layer is used to generate execution instructions based on the fusion processing results. An execution feedback layer is used to feed back the execution status to the fusion processing layer to form closed-loop control.
2. The multimodal perception fusion based robot end-to-end closed loop control system according to claim 1, wherein, The sensing access layer includes: The sensing interface module is equipped with a general hardware interface matrix, which supports the simultaneous access of multiple types of sensors. A routing conversion module is used to convert the communication protocol data of the access device into a system-unified perception data frame format, and to route the standardized data to the corresponding processing module of the fusion processing layer according to the data type; and The signal conditioning module is used to amplify, filter, and perform analog-to-digital conversion on analog signals.
3. The robot end-to-end closed-loop control system based on multimodal perception fusion according to claim 2, characterized in that, The routing conversion module automatically identifies the communication protocol of the access device by matching data packet features. The perceived data frame format includes device ID, timestamp, data type, and payload field.
4. The robot end-to-end closed-loop control system based on multimodal perception fusion according to claim 1, characterized in that, The fusion processing layer includes: The spatiotemporal alignment module is used to unify the timestamps of each sensor to the system reference clock, and to map the spatial data of each sensor to the robot base coordinate system based on the preset sensor extrinsic matrix. The multimodal fusion module is used to extract features from data of each modality, dynamically allocate fusion weights for each feature through an attention mechanism, and perform target recognition and state estimation based on the fused feature map. The data caching module is used to cache fused data and provide data input to the decision control layer.
5. The robot end-to-end closed-loop control system based on multimodal perception fusion according to claim 4, characterized in that, The spatiotemporal alignment module uses a precise clock synchronization protocol to unify timestamps and employs linear interpolation to compensate for data with delay differences.
6. The robot end-to-end closed-loop control system based on multimodal perception fusion according to claim 5, characterized in that, The multimodal fusion module integrates the confidence data from each sensor to generate a comprehensive confidence score.
7. The robot end-to-end closed-loop control system based on multimodal perception fusion according to claim 1, characterized in that, The decision control layer includes: The intent parsing module is used to receive user commands and parse task intents; The end-to-end control module takes the fused feature map and task intent as input and directly outputs the sequence of execution instructions. The security verification module is used to pre-verify the generated execution instructions and trigger instruction correction when there are security risks.
8. The robot end-to-end closed-loop control system based on multimodal perception fusion according to claim 1, characterized in that, The security verification module uses a collision detection algorithm to determine whether there are obstacles on the command path. If there are, it calls a path planning algorithm to replan the path.
9. The robot end-to-end closed-loop control system based on multimodal perception fusion according to claim 1, characterized in that, The execution feedback layer includes: An actuator drive module is used to convert the instructions output by the decision control layer into control signals that the actuator can recognize; The perception feedback module is used to collect execution status data in real time and feed it back to the fusion processing layer for comparison and verification; The closed-loop adjustment module is used to generate compensation commands and add them to the control commands of the next cycle when there is a deviation between the feedback data and the command target.
10. An end-to-end closed-loop control system for a robot based on multimodal perception fusion, characterized in that, The robot end-to-end closed-loop control method based on multimodal perception fusion as described in any one of claims 1-9 includes the following steps: Multiple types of sensors simultaneously collect environmental data, and data from different protocols are uniformly converted into standardized sensing data frames and routed to the fusion processing layer; Multimodal data is unified to the robot's base coordinate system, features of each modality are extracted, and fusion weights are dynamically allocated through an attention mechanism to output a fused feature map and identify the target state. The task intent is parsed, and the execution instruction sequence is directly output based on the fusion result. The execution instructions are then subjected to security verification. The execution instructions are converted into control signals to drive the actuator, the execution status data is collected in real time and fed back, and when a deviation is detected, a compensation instruction is generated for dynamic correction to complete the closed-loop control.