Multi-sensor fusion robotic teleoperation platform

Abstract

The application belongs to the technical field of robot remote control, and particularly relates to a robot remote control platform with multi-sensor fusion, which comprises four modules of a robot end, edge fusion, cloud cooperation and a remote control end, and each module is connected through high-speed double-link redundancy; the robot end module is provided with a hardware synchronous triggering unit to realize synchronous acquisition of multi-source sensor data; the edge fusion module adopts a two-stage fusion algorithm of a data layer and a feature layer to complete low-latency fusion of multi-source data; the cloud cooperation module undertakes data backup, instruction forwarding and state monitoring functions; and the remote control end module completes control instruction input and safety check. The application improves the real-time performance, precision and stability of robot remote control by improving data synchronization precision through hardware synchronization and reducing latency through edge fusion, and is suitable for high-precision operation scenes such as industrial inspection, and has reasonable structure and strong versatility.

Description

Technical Field

[0001] This invention belongs to the field of robot remote control technology, specifically relating to a robot remote control platform based on multi-sensor fusion. Background Technology

[0002] In the field of robot remote control, multi-sensor fusion is a core technology for improving the accuracy of robot environmental perception and expanding operational scenarios. Currently, mainstream remote control platforms are equipped with various heterogeneous sensors such as vision, LiDAR, IMU (Inertial Measurement Unit), and force sensors, achieving precise positioning and environmental recognition through multi-source data complementarity. However, all existing multi-sensor fusion robot remote control platforms suffer from a core and unresolved technical problem: high latency and poor spatiotemporal synchronization in multi-source sensor data fusion. Specifically, this manifests as follows: Existing platforms mostly adopt a traditional architecture of sensor acquisition-cloud transmission-cloud fusion-command issuance. Multi-source data (images, point clouds, attitude data, etc.) are transmitted to the cloud and then fused. Affected by network fluctuations, the fusion latency is generally higher than 200ms, which cannot meet the real-time control requirements of high-precision remote operations. Different types of sensors have different acquisition frequencies, timestamp references, and spatial coordinate systems. Existing platforms only use simple software timestamp alignment methods and lack high-precision synchronization mechanisms, which leads to deviations in the fused data (such as misalignment between visual images and LiDAR point clouds, and asynchrony between attitude data and position data), which in turn causes distortion of control commands and reduces the accuracy of remote operation. Existing fusion algorithms are mostly designed for single-type sensor data and do not consider the temporal differences of multi-source heterogeneous data. Data redundancy or missing data is prone to occur during the fusion process, further aggravating fusion latency and synchronization errors.

[0003] The improvements in existing technologies are limited to conventional methods such as software synchronization and cloud integration, and cannot break out of the inherent framework. They still rely on traditional synchronization and integration modes, resulting in persistently high synchronization errors and integration latency. They cannot meet the needs of high-precision remote operations and have not formed a technical solution that balances synchronization accuracy and integration efficiency. Summary of the Invention

[0004] In view of the above-mentioned shortcomings in the existing technology, the present invention provides a multi-sensor fusion robot remote control platform to solve the problems in the background technology.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A multi-sensor fusion robot remote control platform includes a robot-end module, an edge fusion module, a cloud collaboration module, and a remote control module. Each module is connected via a high-speed dual-link redundant connection, with a transmission latency ≤30ms and a link switching latency ≤10ms. The robot-end module includes a multi-source sensor unit, a hardware synchronization triggering unit, and an actuator. The edge fusion module includes a data preprocessing unit, a high-precision spatiotemporal registration unit, and a hierarchical fusion unit. The cloud collaboration module includes a data backup unit, a command forwarding unit, and a status monitoring unit. The remote control module includes a command input unit, a fused data display unit, and a security verification unit. The hierarchical fusion unit adopts a two-level fusion algorithm of data layer and feature layer. The data layer adopts the weighted Kalman filter algorithm, and the feature layer adopts the improved YOLOv5 algorithm. The fusion latency is controlled within 50ms. The security verification unit combines the environmental feature map output by the edge fusion module to perform security verification on the control commands.

[0006] The data layer of the hierarchical fusion unit adopts the weighted Kalman filter algorithm, and the formula for calculating the weight coefficients is as follows: in, For LiDAR positioning weights, For IMU pose weights, To improve the positioning accuracy of lidar, For IMU attitude accuracy; The state equation of the Kalman filter algorithm is: The observation equation is: in, for The system state vector at any given time. for The system state vector at any given time. Here is the state transition matrix. To control the input matrix, To control the input vector, for Time-process noise vector, for Observation vector at time, For the observation matrix, for Observe the noise vector at all times.

[0007] The security verification unit employs a collision detection algorithm based on AABB axis-aligned bounding boxes, with the intersection determination formula being: when It was determined that there is an intersection, where , , , , , Let x, y, and z be the minimum and maximum x, y, and z coordinates of the bounding box of the robot's expected position, respectively. , , , , , These are the minimum and maximum x, y, and z coordinates of the bounding box of the obstacle, respectively.

[0008] The hardware synchronization triggering unit integrates a high-precision temperature-controlled crystal oscillator synchronizer and a trigger signal generator. The frequency stability of the high-precision temperature-controlled crystal oscillator synchronizer is... The trigger frequency of the trigger signal generator is adjustable; the data synchronization error of the multi-source sensor unit... .

[0009] The multi-source sensor unit includes a binocular depth camera, a lidar, an IMU inertial measurement unit, and a force sensor.

[0010] The high-precision spatiotemporal registration unit adopts a hand-eye joint calibration method to unify the spatial coordinate system of multi-source sensors, and uses a timestamp interpolation alignment algorithm based on the time reference of the hardware synchronous triggering unit to complete the accurate timestamp alignment of multi-source data.

[0011] The improved YOLOv5 algorithm introduces a CBAM attention module, which includes a channel attention module and a spatial attention module. The feature layer fusion latency of the improved YOLOv5 algorithm satisfies the requirement that the total fusion latency of the two levels is ≤50ms.

[0012] The high-speed dual-link redundant connection is a 5G plus fiber optic dual-link redundant connection, a 6G plus fiber optic dual-link redundant connection, or other high-speed dual-link combinations that meet the requirements of transmission latency ≤30ms and link switching latency ≤10ms; wherein the 5G link adopts SA standalone networking mode, the 6G link adopts the terahertz frequency band, and the fiber optic link adopts single-mode fiber with a transmission rate ≥1Gbps.

[0013] The status monitoring unit monitors the working status of the multi-source sensors, edge fusion module, and actuator in real time. It sets the sensor fault judgment threshold as data loss or abnormal duration ≥ preset threshold. When an abnormality occurs, it sends an alarm signal and triggers a redundancy switching mechanism. After switching, a simplified weighted fusion algorithm is used for temporary fusion.

[0014] The data preprocessing unit employs median filtering, histogram equalization, and Kalman filtering algorithms to remove noise from lidar point clouds, calibrate binocular camera image distortion, and eliminate zero-bias errors in IMU attitude data, respectively. The multi-source sensor unit is flexibly replaceable, and the actuator includes a robot joint actuator and a mobile chassis.

[0015] Compared with the prior art, the present invention has the following advantages: 1. This invention features a high-precision hardware synchronization triggering unit with a built-in high-frequency stable temperature-controlled crystal oscillator synchronizer and trigger signal generator. It generates a unified and adjustable hardware trigger signal to synchronously control the acquisition timing of multiple source sensors. By replacing traditional software synchronization with hardware-level synchronization, it effectively solves the problem of low synchronization accuracy in existing technologies, controls the synchronization error of multi-source data to an extremely low range, lays a reliable foundation for subsequent accurate fusion, and has outstanding substantive features. 2. This invention constructs an edge-local fusion architecture, migrating the multi-source data fusion process to edge nodes. It combines a weighted Kalman filter algorithm with an improved attention mechanism YOLOv5 algorithm to adapt to the perception requirements of complex ground environments, controlling the fusion latency to within 50ms. This completely solves the pain point that existing platforms cannot meet the real-time control requirements of high-precision remote operations, significantly improves the smoothness and accuracy of robot remote control, and provides technical support for various high-precision operation scenarios. It has outstanding substantive features and significant progress. 3. This invention adopts high-speed dual-link redundant transmission technology and optimizes the link switching mechanism to adapt to the bidirectional data transmission requirements and optimizes the command forwarding protocol. It avoids the problem of easy interruption of single-link transmission, improves the stability of platform operation, and realizes long-term continuous uninterrupted operation. At the same time, it adopts a modular design, and multi-source sensor units can be flexibly replaced to adapt to a variety of remote operation scenarios, making it highly versatile and practical. Attached Figure Description

[0016] Fig. 1 This is a platform architecture diagram of an embodiment of the multi-sensor fusion robot remote control platform of the present invention; Fig. 2 This is a schematic diagram of collision detection for the safety verification unit. The reference numerals in the accompanying drawings include: 1. Robot's intended location bounding box; 2. Obstacle bounding box; 3. Robot; 4. Obstacle; Detailed Implementation To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0017] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual images. They should not be construed as limiting the scope of this application. To better illustrate the embodiments of the present invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0018] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present application. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0019] In the description of this invention, unless otherwise explicitly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0020] Example 1: like Figs. 1-2 As shown, this invention discloses a multi-sensor fusion robot remote control platform, characterized by comprising a robot end module, an edge fusion module, a cloud collaboration module, and a remote control end module. These four modules are connected via a high-speed transmission link to achieve low-latency synchronous fusion and remote control of multi-source sensor data. The robot end module, serving as a data acquisition and command execution terminal, includes a multi-source sensor unit, a hardware synchronization triggering unit, and an execution mechanism. The high-speed dual-link redundant connection employs a heartbeat mechanism to monitor link status in real time. The cloud collaboration module and the edge fusion module send heartbeat packets to each other at fixed intervals to determine link connectivity and transmission quality. If any link fails to receive a heartbeat packet within a preset time or if the transmission delay exceeds a preset threshold, the link is immediately identified as faulty and a link switch is triggered. The system automatically switches to another normal link within 10ms, maintaining data transmission continuity during the switchover process without packet loss or interruption of control commands and fused data forwarding, ensuring uninterrupted remote control of the robot.

[0021] The multi-source sensor unit includes a binocular depth camera, a lidar, an inertial measurement unit (IMU), and a force sensor. The binocular depth camera is used to acquire color and depth image information of the robot's surrounding environment; the lidar is used to scan and acquire three-dimensional spatial point cloud data of the robot's surrounding environment; the IMU is used to measure the robot's three-axis attitude angles and acceleration, and the force sensor, usually installed on the robot's end effector, is used to collect contact force data during operation.

[0022] The hardware synchronization trigger unit integrates a high-stability temperature-controlled crystal oscillator synchronizer and a trigger signal generator, its core function being the generation of a unified hardware trigger signal. The frequency of this trigger signal can be adjusted within the range of 100Hz to 500Hz according to actual operational requirements. This signal is transmitted in parallel to the binocular depth camera, LiDAR, IMU, and force sensor, strictly controlling the start-up time of all sensor acquisition. This ensures that all sensors begin data acquisition based on the same high-precision time reference provided by the high-stability crystal oscillator, thereby guaranteeing strict synchronization of multi-source data in the time dimension from the source. The actuators include robot joint actuators and a mobile chassis, used to receive fused control commands and complete corresponding posture adjustments, movements, and maneuvers.

[0023] The edge fusion module is deployed on an edge computing node near the robot and is connected to the robot module via a wired high-speed link. It introduces edge local fusion technology to achieve low-latency preprocessing and fusion of multi-source sensor data. Specifically, it includes a data preprocessing unit, a high-precision spatiotemporal registration unit, and a hierarchical fusion unit.

[0024] The data preprocessing unit performs noise reduction and distortion correction on the raw data collected by multiple sensors: it uses a median filtering algorithm to remove noise from the lidar point cloud, a histogram equalization algorithm to correct image distortion from the binocular camera, and a Kalman filtering algorithm to eliminate the zero bias error of the IMU attitude data, ensuring the validity of the raw data.

[0025] A high-precision spatiotemporal registration unit, combined with spatial coordinate calibration technology, uses a hand-eye joint calibration method to unify the spatial coordinate system of multi-source sensors (using LiDAR coordinates as a reference to transform the coordinates of the camera, IMU, and force sensor); simultaneously, based on the time reference of the hardware synchronization trigger unit, it precisely aligns the timestamps of the preprocessed multi-source data, controlling the synchronization error within a specified range. Within.

[0026] The hierarchical fusion unit employs a two-level fusion algorithm: a data layer and a feature layer. The data layer uses a weighted Kalman filter algorithm to fuse IMU attitude data and LiDAR positioning data, outputting accurate robot pose information. The feature layer uses an improved YOLOv5 algorithm to fuse binocular camera image features and LiDAR point cloud features, outputting a unified environmental feature map (including obstacle position, size, and distance information). Both fusion processes are completed locally at the edge nodes, avoiding latency caused by data transmission to the cloud, with fusion latency controlled within 50ms.

[0027] The cloud collaboration module adopts high-speed redundant transmission technology and is connected to the edge fusion module through 5G+fiber dual-link redundancy. It is used to realize the backup of edge fusion data, the forwarding of remote control commands and status monitoring. Specifically, it includes a data backup unit, a command forwarding unit and a status monitoring unit.

[0028] The data backup unit performs real-time backup of the fused data (pose information, environmental feature map) output by the edge fusion module. It adopts a distributed storage architecture to ensure that the data is not lost, and also supports historical data backtracking to facilitate troubleshooting.

[0029] The instruction forwarding unit uses a low-latency forwarding protocol for data transmission to achieve efficient and low-latency forwarding of control instructions and fused data, ensuring the real-time performance of remote control. The control instructions issued by the remote control module are forwarded to the edge fusion module after security verification, with the forwarding latency controlled within 30ms. At the same time, the robot status data (pose, environment, sensor working status) output by the edge fusion module is forwarded to the remote control module to achieve bidirectional data synchronization.

[0030] The status monitoring unit monitors the working status of multi-source sensors, edge fusion module, and robot actuator in real time. The sensor fault determination threshold is defined as data loss or abnormal duration ≥ a preset threshold. The preset threshold can be flexibly adjusted according to the operation scenario and real-time requirements. In this embodiment, the preset threshold is set to 50ms. When any sensor experiences data loss or abnormality and the duration reaches the preset threshold, it is immediately determined as a sensor fault and a redundancy switching mechanism is triggered. After switching, a simplified weighted fusion algorithm is used to perform temporary fusion. Specifically, the simplified weighted fusion algorithm selects the sensor with the highest positioning and attitude perception accuracy among the currently working sensors as the main sensor, and the remaining normal sensors as auxiliary sensors. The weight of the main sensor is set to 0.6, and the total weight of the auxiliary sensors is set to 0.4. The temporary fusion output is completed based on the remaining valid data. When the faulty sensor recovers and the data is continuous and stable, it automatically reverts to the original hierarchical fusion algorithm to ensure operation continuity and control accuracy. When abnormal situations such as sensor faults or data transmission interruptions occur, an alarm signal is immediately sent to the remote control module, and a redundancy switching mechanism is triggered (e.g., in the case of a single sensor fault, the data from the remaining sensors is automatically used for temporary fusion) to ensure uninterrupted control.

[0031] The remote control module serves as a human-machine interaction terminal, including an instruction input unit, a fusion data display unit, and a security verification unit. The instruction input unit supports three control modes: joystick, gesture, and voice. Users can select the appropriate control method according to their operational needs and input control instructions such as robot movement, posture adjustment, and end effector operation. The fusion data display unit uses high-definition display technology to display the environmental feature map, robot pose information and sensor working status output by the edge fusion module in real time. It also supports multi-view switching (global view and local close-up view) to facilitate users to accurately judge the working environment. The safety verification unit performs safety verification on the control commands input by the user. Combined with the environmental feature map output by the edge fusion module, it determines whether the command has a collision risk (such as the command pointing to an obstacle area). If a risk exists, the command is intercepted and the user is notified to avoid equipment damage or operational accidents caused by misoperation.

[0032] Furthermore, the temperature-controlled crystal synchronizer of the hardware synchronization trigger unit has high frequency stability. This ensures the high precision and stability of the trigger signal, providing a reliable time reference for synchronous acquisition by multiple sensors.

[0033] Furthermore, the hierarchical fusion unit of the edge fusion module, with its improved YOLOv5 algorithm, introduces an attention mechanism to extract key feature information of obstacles in the environment, improving the fusion speed by 30% compared to the traditional YOLOv5 algorithm and further reducing fusion latency.

[0034] Furthermore, the high-speed dual-link redundant connection of the cloud collaboration module can adopt 5G+fiber, 6G+fiber or other high-speed link combinations that meet the requirements of transmission latency ≤30ms and link switching latency ≤10ms. When one link is interrupted, it can automatically switch to the other link within 10ms to ensure the continuity of data transmission, avoid control failure caused by link interruption, and adapt to the requirements of high reliability operation.

[0035] Example 2: Taking an industrial robot as an example, a multi-source sensor unit is deployed on the robot's chassis and robotic arm: a binocular depth camera with a resolution of 1920×1080 and a sampling frequency of 30fps, mounted on the robot's head, used to collect image information of the inspection area; a lidar with a detection range of 0.5-50m and a sampling frequency of 100Hz, mounted on the top of the robot, used to construct a 3D point cloud of the inspection area; an inertial measurement unit (IMU) with a sampling frequency of 200Hz, mounted at the center of the robot's chassis, used to collect the robot's attitude data, such as pitch angle, roll angle, and yaw angle; and a force sensor with a measurement range of 0-500N, mounted at the end of the robotic arm. It is used to collect contact force data with equipment during inspection; the raw data of the force sensor is processed by the data preprocessing unit to remove high-frequency noise and retain effective force characteristics; the high-precision spatiotemporal registration unit completes the timestamp alignment based on the unified time reference of the hardware synchronization triggering unit, and the force sensor coordinates are unified to the robot end effector coordinate system through hand-eye joint calibration to achieve spatial synchronization; the hierarchical fusion unit combines the force data with the robot pose information, compares the real-time contact force value with the preset safety force threshold (set to 300N in this embodiment), and when the contact force exceeds the threshold, outputs the force control constraint signal to the control command link to achieve flexible limit and overload protection.

[0036] The hardware synchronization trigger unit uses a temperature-controlled crystal oscillator synchronizer, with a trigger frequency of 200Hz, generating a uniform hardware trigger signal with an amplitude of 5V and a pulse width of 10. The timing of data acquisition from all the aforementioned sensors is synchronized to ensure that all sensors start data acquisition at the same time, avoiding data asynchrony caused by differences in acquisition timing. The actuator uses a servo joint driver (response frequency 1kHz, positioning accuracy ±0.01mm) and a tracked mobile chassis to receive control commands from the edge fusion module and complete actions such as movement, turning, and extension / retraction of the robotic arm of the inspection robot.

[0037] The edge fusion module is deployed inside the inspection robot's control box and connected to the robot's end module via a gigabit wired link, receiving raw data from multiple source sensors in real time. The data preprocessing unit performs targeted processing on the raw data. Specifically: In this embodiment, the median filtering algorithm removes isolated noise points and salt-and-pepper noise from the LiDAR point cloud, ensuring the integrity of the point cloud data. The core principle of the median filtering algorithm is to select the median of pixels (or point cloud data) within a sliding window and replace the center pixel (or point cloud data) of the window, thereby suppressing isolated noise points and salt-and-pepper noise and avoiding the impact of noise on the subsequent fusion accuracy. The parameters of 5×5 window size and 3 iterations in this scheme are based on the density (acquisition frequency 100Hz, detection distance 0.5-50m) and noise distribution characteristics of the LiDAR point cloud in industrial inspection scenarios. The specific steps are as follows; Step 1: Read the raw point cloud data collected by the LiDAR. This data is 3D point cloud data collected in an industrial inspection scenario, containing environmental obstacles, equipment outlines, and isolated noise points, mainly generated by dust and vibration. Step 2: Set a 5×5 sliding window and traverse the entire point cloud data. Sort the point cloud data in each window and select the median of the point cloud data in the window. Step 3: Replace the point cloud data in the center of the window with the selected median to complete the denoising process for a single window. Repeat Step 3 for 3 iterations to ensure that isolated noise points and salt-and-pepper noise in the point cloud are completely removed, while avoiding point cloud data distortion and ensuring the integrity of the point cloud data, providing high-quality data for subsequent environmental feature fusion.

[0038] In this embodiment, the histogram equalization algorithm, with a grayscale range of 0-255 and a contrast gain coefficient of 1.2, corrects the distortion of binocular camera images, eliminates image shift caused by lens distortion, and improves image clarity. The core principle of the histogram equalization algorithm is to adjust the grayscale histogram distribution of the image, stretch the grayscale range, improve image contrast, and simultaneously correct image shift caused by lens optical distortion, making image details clearer. In this solution, the grayscale range of 0-255 is the standard setting for this algorithm, adapted to the 1920×1080 resolution of the binocular camera. The contrast gain coefficient of 1.2 is adapted to the complex lighting conditions (uneven lighting in the factory) in industrial inspection scenarios, effectively improving the image recognition of equipment defects and obstacles in inspection scenarios. The specific steps are as follows; Step 1: Read the original environmental image captured by the binocular camera. This image is of an industrial plant and suffers from image shift due to lens optical distortion and insufficient clarity due to uneven lighting. Step 2: Calculate the grayscale histogram of the original image, count the number of pixels at each grayscale level (0-255), and determine the distribution range of the grayscale histogram. Step 3: Based on the grayscale range of 0-255, adjust the grayscale histogram distribution by stretching the grayscale range with a contrast gain coefficient of 1.2 to improve image contrast and correct image shift caused by lens optical distortion. Step 4: Output the corrected image. This image clearly presents details of equipment defects and obstacles, providing high-quality image data for feature extraction in the subsequent improved YOLOv5 algorithm.

[0039] In this embodiment, the Kalman filter algorithm is used. The state equation of the Kalman filter algorithm is: The observation equation is: in, for The system state vector at any given time. for The system state vector at any given time. Here is the state transition matrix. To control the input matrix, To control the input vector, for Time-process noise vector, for Observation vector at time, For the observation matrix, for The noise vector is observed at any given time. Specifically: Step 1: Read the raw attitude data collected by the IMU inertial measurement unit, including the robot's pitch angle, roll angle, and yaw angle data. This data contains vibration interference caused by the robot's movement and zero bias error. Step 2: Set the state equation of the Kalman filter algorithm as follows: The observation equation is: Step 3: Set the adaptation parameters Q-value, R-value, and initial covariance matrix. Substitute the parameters into the formula to clarify the correspondence and dimensional matching between each parameter and the vector in the formula, ensuring that the substitution logic is rigorous. Step 4 executes the state prediction and observation update iteration process of Kalman filtering to filter out the zero bias error and vibration interference in the original attitude data; Step 5 outputs the filtered attitude data, the error of which is ≤0.1°, to ensure the accuracy and stability of the IMU attitude data and provide reliable data for the fusion of the weighted Kalman filter algorithm.

[0040] in, for The system state vector at time step corresponds to the data acquired by the IMU. The robot's attitude vector at any given moment is specifically [pitch angle, roll angle, yaw angle]^T; for The system state vector at time step corresponds to the data acquired by the IMU. The robot's attitude vector at any given moment is specifically [pitch angle, roll angle, yaw angle]^T; The state transition matrix is ​​3×3 in dimension, adapted to the temporal continuity of IMU pose data. Since the robot's pose changes smoothly over a short period of time, it is set as an identity matrix. , characterization At any moment, attitude and The attitude at any given time is linearly correlated, meaning there are no abrupt changes in the attitude data; To control the input matrix, it is set to a zero matrix B=[0,0,0]^T, since the IMU is only used to collect attitude data and no additional control signal is required; The control input vector corresponds to the control input quantity. Since there is no additional control signal, therefore... ; for The time-matter noise vector corresponds to the process noise generated by vibrations in industrial inspection scenarios during the acquisition and transmission of IMU attitude data. Specifically, it is [pitch angle process noise, roll angle process noise, yaw angle process noise]^T, and its covariance matrix is ​​the Q value. Substituting the adaptation parameter Q=0.01, we get Q= This is used to quantify the intensity of the process noise, adapting to the characteristics of weak vibration interference in industrial inspection scenarios, and ensuring that the quantification of process noise matches the scenario. for The time-time observation vector corresponds to the actual observations of the IMU. The attitude vector at any given time, specifically [observed pitch angle, observed roll angle, observed yaw angle]^T, is related to the system state vector. One-to-one correspondence; H is the observation matrix, which is set as the identity matrix. Characterizing the observation vector With system state vector There is a linear correspondence, meaning that the observed data can directly reflect the actual state of the system; for The noise vector observed at any given time corresponds to the inherent noise of the IMU observation data itself, which is determined by the sensor's own accuracy. Its covariance matrix is ​​the R value. Substituting the adaptation parameter R=0.1, we get R= This is used to quantify the intensity of the observation noise, adapting to the characteristic of IMU measured attitude accuracy ≤0.1°, and ensuring that the observation noise quantization matches the sensor accuracy; Initial covariance matrix Set as , which are the standard initial values ​​for Kalman filtering, corresponding to the initial time ( System state vector The covariance is adapted to the uncertainty of the IMU's initial attitude, providing a reasonable initial benchmark for subsequent filtering iterations.

[0041] Specifically, Q value represents the process noise covariance, characterizing the intensity of process noise generated during the transmission and acquisition of IMU attitude data. In industrial inspection scenarios, IMUs are mainly affected by vibration interference generated by robot movement, which is relatively weak. Referring to the conventional Q value range (0.001-0.05) for IMU filtering in the existing robotics field, through multiple tests and calibrations, it was found that when Q value = 0.01, it can effectively filter process noise caused by vibration and avoid attitude data lag caused by over-filtering, thus adapting to the inspection robot's 1m / s moving speed and attitude adjustment requirements. R value represents the observation noise covariance, characterizing the noise intensity of the IMU observation data itself. Combining the measured attitude accuracy of the IMU in this embodiment (0.1°), and referring to the filtering R value range (0.05-0.2) of existing IMUs with similar accuracy, through testing and verification, it was found that when R value = 0.1, it can accurately match the observation noise characteristics of the IMU, ensuring that the error of the filtered attitude data meets the attitude accuracy requirements of industrial inspection. The initial covariance matrix is ​​set to... , which is the standard initial value for Kalman filtering. It will then automatically converge to the optimal value through algorithm iteration, adapting to the dynamic changes in IMU attitude data and ensuring the stability of the filtering effect.

[0042] In this embodiment, the high-precision spatiotemporal registration unit uses a hand-eye joint calibration method (adopting Zhang's calibration method, with a 12×9 checkerboard calibration board, each square being 10mm×10mm, 10 calibration iterations, and a reprojection error ≤0.3 pixels) to transform the coordinates of the binocular camera, IMU, and force sensor to the lidar coordinate system, with the calibration error controlled within 0.5mm. Simultaneously, based on the time reference of the hardware synchronization trigger unit, a timestamp interpolation alignment algorithm is used to accurately align the timestamps of the preprocessed multi-source data, with the synchronization error controlled within 10μs, ensuring the spatiotemporal synchronization of image, point cloud, and attitude data, and providing a reliable data foundation for subsequent fusion.

[0043] The hierarchical fusion unit performs two-level fusion. The data layer uses a weighted Kalman filter algorithm to fuse IMU attitude data and LiDAR positioning data. The weighting coefficients are dynamically adjusted according to the sensor accuracy, and the specific adjustment rules are as follows: in, For LiDAR positioning weights, For IMU pose weights, To improve the positioning accuracy of lidar, For IMU attitude accuracy, the accuracy parameters of the two sensors are acquired in real time and their weights are calculated. Initial lidar positioning accuracy in this embodiment IMU attitude accuracy According to the weight formula defined in the claim, the initial weight is calculated and conforms to the algorithm design logic. , The core logic of this initial weight setting is as follows: First, it strictly adheres to the calculation results of the weight formula, conforming to the algorithm design specifications defined in the claims; second, it adapts to the core needs of industrial inspection scenarios. In this embodiment, the focus of the inspection operation is to achieve precise robot positioning. LiDAR has higher positioning accuracy (1mm), therefore it is given a higher weight to ensure positioning accuracy. The core advantage of IMU lies in its fast attitude response speed, mainly used to assist in correcting robot pose, therefore it is given a lower weight to achieve a balance between positioning accuracy and attitude responsiveness. Subsequently, the weight allocation will be dynamically fine-tuned based on the accuracy parameters collected by the sensors in real time to ensure that the fusion accuracy always adapts to the operational requirements. In the Kalman filter parameter initialization stage, the core parameters, including the state vector, are initialized according to the parameter specifications defined in this specific embodiment. State transition matrix Control input matrix Observation matrix Process noise vector variance matrix Q, observation noise vector variance matrix R, initial covariance matrix All parameters are adapted to the actual needs of robot pose fusion, ensuring that the filtering process is stable and reliable.

[0044] Weighting and data fusion core steps by time series Iterative execution ensures that each time step fully completes the process of weight substitution, filter update, and pose output, guaranteeing the real-time performance of the fusion process and adapting to the low latency requirements of the edge fusion module. The specific steps are as follows: Step 1: Acquire data through a multi-source sensor unit. Real-time lidar positioning data location coordinates With IMU attitude data After the data preprocessing unit completes denoising, zero bias elimination and other processing, clean and effective observation data is obtained, which serves as the input basis for subsequent fusion. Step 2 involves substituting weights and constructing a comprehensive observation vector, using the real-time calculated... , Substitute the observation data into the fusion process and construct Time-based integrated observation vector The fusion formula is in, As a weighted and fused comprehensive observation vector, it retains the positioning accuracy advantage of lidar while taking into account the attitude response speed advantage of IMU. Step 3, Kalman filter prediction based on state equation Combination The optimal state vector at time t, prediction State vector at time step Simultaneously update the predicted covariance matrix. The prediction formula is: In practical engineering, process noise compensation can be achieved by combining the Q matrix; however, process noise is temporarily ignored in this embodiment. Predicting the covariance matrix The formula is: in, The control input vector sent from the remote control terminal. for The optimal covariance matrix at time t. State transition matrix transpose; Step 4, Kalman filter update: First, calculate the Kalman gain. in, Observation matrix The transpose of the Kalman gain is then used in conjunction with the weighted integrated observation vector obtained in step 2. ,renew The optimal state vector at time: Should This refers to the fused, precise robot pose information, which is then updated last. The optimal covariance matrix at time: in, The identity matrix is ​​given, and the optimal covariance matrix is ​​given at the next time step. Prepare for filtering prediction; Step 5: Output precise pose information. Optimal state vector at any given time It includes the robot's position coordinates (x, y, z) and attitude angles (pitch angle, roll angle, yaw angle), with a positioning error ≤1mm and an attitude error ≤0.1°. This pose information will be synchronously transmitted to the feature layer for further fusion with subsequent environmental feature maps, and forwarded to the cloud collaboration module and remote control module to provide accurate data support for remote robot control.

[0045] Furthermore, the core logic of weight substitution is through... , Real-time dynamic adjustment , To ensure pose fusion accuracy, sensors with higher precision are given higher weights during the fusion process. The entire fusion process is executed in real-time within the hierarchical fusion unit. The iteration cycle of each time sequence k is matched with the trigger frequency (100Hz-500Hz) of the hardware synchronization trigger unit, ensuring that the total fusion latency of the two levels meets the design requirement of ≤50ms. When the status monitoring unit determines that a sensor has lost or is abnormal, and the duration is ≥ a preset threshold, the weight coefficient will automatically switch to the weight allocation mode of the simplified weighted fusion algorithm, i.e., the main sensor has a weight of 0.6, and the auxiliary sensors have a combined weight of 0.4. Specifically, the core requirement of industrial inspection is precise robot positioning. LiDAR has higher positioning accuracy and needs to be given a higher weight of 0.6 to ensure positioning accuracy. IMU focuses on attitude response speed and assists in correcting pose, so it is given a lower weight of 0.4 to balance accuracy and responsiveness. This ensures that the fusion process is uninterrupted and guarantees the continuity of remote robot control.

[0046] After completing the data layer pose fusion, the hierarchical fusion unit enters the feature layer fusion stage, employing an improved YOLOv5 algorithm. Based on the YOLOv5s model, the improved YOLOv5 algorithm embeds a CBAM attention module sequentially after the C3 module in the network neck structure. This module performs channel attention weighting and spatial attention weighting on the fused feature maps, enhancing key target features such as device defects and obstacles while suppressing redundant background features. The CBAM attention module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule extracts the channels of the feature map through global average pooling and global max pooling. The features are processed by a fully connected layer, and the output channel weights are used to weight the channels of the feature map. The spatial attention submodule obtains two two-dimensional feature maps through global average pooling and global max pooling. After concatenation, the spatial weights are output to weight the spatial position of the feature map. Through the synergistic effect of channel attention and spatial attention, the features of equipment defects in the image and obstacles in the point cloud are extracted and fused to generate a unified environmental feature map. The location of equipment defects and the distance of obstacles in the inspection area are marked. The feature layer fusion latency is controlled within 45ms, and the fusion with the data layer meets the requirement that the total latency of the two-level fusion is ≤50ms.

[0047] The specific training process of the improved YOLOv5 algorithm is as follows: Using YOLOv5s as the base model, the input image size is set to 640×640, the batch size to 16, and the initial learning rate to 0.001. A cosine annealing learning rate decay strategy is adopted, with a decay coefficient of 0.1 and a learning rate decay period of 50 epochs. The momentum is set to 0.937, the weight decay coefficient to 0.0005, the IoU (Intersection over Union) threshold to 0.5, and the confidence threshold to 0.25, used to filter valid detection boxes. The total number of training epochs is set to 100, with the first 10 epochs used for warm-up training to avoid model overfitting. All the above parameters are adapted to the edge fusion module's total fusion latency requirement of ≤50ms, and are consistent with the equipment defect and obstacle recognition requirements of industrial inspection scenarios. Combined with conventional YOLOv5 algorithm training experience, these parameters ensure that training efficiency and model performance meet the requirements of this invention's platform.

[0048] Step 1: Collect images of equipment defects and obstacles in industrial inspection scenarios, along with corresponding LiDAR point cloud data, to construct a training dataset containing 5,000 sample images. Divide the dataset into training, validation, and test sets in an 8:1:1 ratio. Label the samples, including categories such as equipment defects and obstacles, and generate a tag file conforming to the YOLO format, consistent with the environmental feature extraction requirements of the platform of this invention. Step 2 performs data augmentation operations on the training set images, such as random flipping, translation, scaling, and color gamut transformation, to improve the model's generalization ability. Simultaneously, the images are normalized using the following formula: in, The original pixel values ​​of the image. The minimum value of the image pixels. The maximum value of the image pixels is represented by this formula. This formula normalizes the pixel value to the [0,1] interval, eliminating the influence of dimensions and laying the foundation for accurate model training. Step 3: Load YOLOv5s pre-trained weights, embed the CBAM attention module in the neck structure of the model, freeze the first 10 layers of the backbone network of the pre-trained weights, and train only the neck and head structures and the CBAM attention module to speed up the training while ensuring the accuracy of feature extraction. Step 4: Input the preprocessed training set into the improved YOLOv5 model, and use the CIoU loss function as the training loss. The formula for the CIoU loss function is: in, The intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box. The squared Euclidean distance between the center of the predicted bounding box and the center of the ground truth bounding box is given. The length of the diagonal of the minimum bounding rectangle that encloses the predicted bounding box and the ground truth bounding box. For balance coefficient, The aspect ratio consistency parameter between the predicted bounding box and the ground truth bounding box; the model parameters are updated through backpropagation, and the parameter update formula during backpropagation is: in, For the updated parameters, For the current parameter, For learning rate, This represents the partial derivative of the loss function with respect to the current parameters; model performance is validated every 5 epochs, and the mean accuracy is calculated using the following formula: in, To indicate the number of categories, For the first The average precision of each category is used to calculate the recall rate, which is calculated using the following formula: in, For the actual number of cases, This represents the number of false negatives. If the average precision of the validation set does not improve over 10 consecutive epochs, training is stopped, the optimal model weights are saved, and the model performance is ensured to be optimal. Step 5: Apply the optimal model weights to the test set to test the detection accuracy and inference speed of the model. If the detection accuracy is lower than 90%, adjust the learning rate and batch size and retrain. If the inference speed does not meet the feature layer fusion latency requirements, optimize the convolution kernel size of the CBAM attention module and simplify the model structure until it meets the requirements of the platform of this invention and ensures that it is compatible with the latency requirements of the edge fusion module.

[0049] After training, the optimal improved YOLOv5 model is deployed to the hierarchical fusion unit of the edge fusion module. It receives the binocular camera image features and LiDAR point cloud features after data preprocessing and spatiotemporal registration. The CBAM attention module extracts the features of equipment defects and obstacles. The model inference outputs the feature fusion result, which works in conjunction with the pose information fused from the data layer to generate a unified environmental feature map. This ensures that the total fusion latency is ≤50ms, which meets the real-time requirements of remote robot inspection and is fully consistent with the core design goals of the platform of this invention.

[0050] In this embodiment, the cloud collaboration module is deployed on a server cluster in the industrial control center. It is connected to the edge fusion module via a 5G+fiber dual-link redundant connection. The dual-link redundant connection uses a heartbeat mechanism to monitor the physical link status. The heartbeat packet sending period is 50ms, and the timeout threshold is 10ms. If either the 5G link or the fiber link does not receive a heartbeat packet within 10ms, a link switch is immediately triggered, and the normal link switch is completed within 10ms. The switchover process is without packet loss or interruption of data transmission. The 5G link adopts SA standalone networking mode, with a frequency band of 2.6GHz, a transmission rate of ≥1Gbps, and a latency of ≤20ms. The fiber link uses single-mode fiber, with a preferred transmission rate of ≥10Gbps and a latency of ≤10ms. The data backup unit adopts a distributed storage architecture (number of nodes ≥3, number of replicas 3, storage latency ≤10ms) to back up the pose information, environmental feature maps, and raw sensor data output by the edge fusion module in real time. The storage capacity can be expanded as needed (maximum expandable to 100TB), supports historical data back-up, and facilitates fault diagnosis and operation review.

[0051] The command forwarding unit employs the UDP low-latency forwarding protocol, port number 5000, data packet size ≤ 1024 bytes, 3 timeout retransmissions, and a retransmission interval of 5ms. It forwards control commands such as movement speed and robotic arm movement angles issued by the remote control module to the edge fusion module after security verification, with forwarding latency controlled within 25ms. Simultaneously, the command forwarding unit uses an end-to-end heartbeat mechanism to keep the business link alive, with a heartbeat interval of 100ms and a timeout threshold of 500ms. It forwards robot status data (pose, environment, and sensor operating status) output by the edge fusion module to the remote control module in real time, achieving bidirectional data synchronization and ensuring the remote end has real-time control over the robot's operating status. Specifically, the low-latency forwarding protocol is a lightweight data transmission protocol designed to meet the real-time requirements of remote robot control. By simplifying the protocol message header, shortening the handshake interaction process, and optimizing data scheduling strategies, it reduces data processing latency and forwarding waiting time in the transmission link, enabling fast and stable transmission of control commands and environmental fusion data, further improving the platform's overall response speed and control real-time performance.

[0052] The status monitoring unit monitors the working status of multi-source sensors, edge fusion module, and actuators in real time. The redundancy switching mechanism is implemented as follows: A fault judgment threshold is set as sensor data loss or abnormality exceeding 50ms. When the data loss or abnormality of a certain sensor (such as LiDAR) is detected for 50ms, redundancy switching is immediately triggered. The temporary fusion algorithm after switching adopts a simplified weighted fusion algorithm, selecting the sensor with the highest accuracy among the remaining normal sensors as the main sensor. In this embodiment, the IMU inertial measurement unit is selected first, and the remaining normal sensors are used as auxiliary sensors. The weight of the main sensor is set to 0.6, and the total weight of the auxiliary sensors is 0.4. The data of each sensor is weighted and fused, and the robot pose and environmental perception information are temporarily output to ensure that the control of the inspection robot is not interrupted and to ensure the continuity of the inspection operation. When the faulty sensor returns to normal and the data is normal for 100ms, the original fusion algorithm is automatically switched back.

[0053] The remote control module is deployed at the monitoring terminal of the industrial control center. The command input unit adopts a joystick and gesture control mode. Users control the movement direction and speed of the inspection robot with the joystick and control the extension and rotation of the robotic arm with gestures. The fusion data display unit adopts a 4K high-definition display screen to display the environmental feature map, robot pose information and sensor working status output by the edge fusion module in real time. It supports switching between global view and local close-up view (displaying the details of equipment defects), which makes it easy for users to accurately judge the inspection situation.

[0054] The safety verification unit performs safety verification on the user-input control commands. The specific interception logic employs a collision detection algorithm based on AABB (Axis-Aligned Bounding Box). First, based on the environmental feature map output by the edge fusion module, the coordinate information of all obstacles is extracted, and an AABB bounding box is constructed for each obstacle, determining the minimum and maximum x, y, and z coordinates of the bounding box. Second, the user-input control command is parsed, the expected coordinates of the end effector or robot body after the robot executes the command are calculated, and a corresponding AABB bounding box is constructed for it. Finally, it is determined whether two bounding boxes intersect. If an intersection exists (i.e., the expected position overlaps with an obstacle), or the shortest straight-line distance between the robot's expected position and an obstacle is ≤0.5m, a collision risk is identified, the command is immediately intercepted, and a collision risk warning is displayed on the screen, requesting adjustments to the command to avoid accidental collisions between the inspection robot and equipment, thus ensuring operational safety.

[0055] The above are merely embodiments of the present invention. The circuits, electronic components, and modules involved are all prior art, fully achievable by those skilled in the art, and require no further explanation. Commonly known structures and characteristics in the solutions are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all prior art in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A robot remote control platform based on multi-sensor fusion, characterized in that: The system includes a robot-end module, an edge fusion module, a cloud collaboration module, and a remote control module. These modules are connected via a high-speed dual-link redundant connection, enabling low-latency and rapid data switching transmission. The robot-end module includes a multi-source sensor unit, a hardware synchronization triggering unit, and an actuator. The edge fusion module includes a data preprocessing unit, a high-precision spatiotemporal registration unit, and a hierarchical fusion unit. The cloud collaboration module includes a data backup unit, a command forwarding unit, and a status monitoring unit. The remote control module includes a command input unit, a fused data display unit, and a security verification unit. The hierarchical fusion unit adopts a two-level fusion algorithm of data layer and feature layer. The data layer adopts the weighted Kalman filter algorithm, and the feature layer adopts the improved YOLOv5 algorithm to ensure the real-time performance of robot remote control. The security verification unit performs security verification on control commands by combining the environmental feature map output by the edge fusion module.

2. The multi-sensor fusion robot remote control platform as described in claim 1, characterized in that: The data layer of the hierarchical fusion unit adopts the weighted Kalman filter algorithm, and the formula for calculating the weight coefficients is as follows: in, For LiDAR positioning weights, For IMU pose weights, To improve the positioning accuracy of lidar, For IMU attitude accuracy; The state equation of the Kalman filter algorithm is: The observation equation is: in, for The system state vector at any given time. for The system state vector at any given time. Here is the state transition matrix. To control the input matrix, To control the input vector, for Time-process noise vector, for Observation vector at time, For the observation matrix, for Observe the noise vector at all times.

3. The multi-sensor fusion robot remote control platform as described in claim 1, characterized in that: The security verification unit employs a collision detection algorithm based on AABB axis-aligned bounding boxes, with the intersection determination formula being: when It was determined that there is an intersection, where , , , , , Let x, y, and z be the minimum and maximum x, y, and z coordinates of the bounding box of the robot's expected position, respectively. , , , , , These are the minimum and maximum x, y, and z coordinates of the bounding box of the obstacle, respectively.

4. The multi-sensor fusion robot remote control platform as described in claim 1, characterized in that: The hardware synchronization triggering unit is equipped with a high-precision constant-temperature crystal oscillator synchronizer and a trigger signal generator. The high-precision constant-temperature crystal oscillator synchronizer has high frequency stability, and the trigger frequency of the trigger signal generator is adjustable, which is used to control the multi-source sensor unit to achieve synchronous data acquisition.

5. The multi-sensor fusion robot remote control platform as described in claim 1, characterized in that: The multi-source sensor unit includes a binocular depth camera, a lidar, an IMU inertial measurement unit, and a force sensor.

6. The multi-sensor fusion robot remote control platform as described in claim 1, characterized in that: The high-precision spatiotemporal registration unit adopts a hand-eye joint calibration method to unify the spatial coordinate system of multi-source sensors, and uses a timestamp interpolation alignment algorithm based on the time reference of the hardware synchronous triggering unit to complete the accurate timestamp alignment of multi-source data.

7. The multi-sensor fusion robot remote control platform as described in claim 1, characterized in that: The improved YOLOv5 algorithm introduces a CBAM attention module, which includes a channel attention module and a spatial attention module. The feature layer fusion latency of the improved YOLOv5 algorithm satisfies the requirement that the total fusion latency of the two levels is ≤50ms.

8. The multi-sensor fusion robot remote control platform as described in claim 1, characterized in that: The high-speed dual-link redundant connection is a 5G plus fiber optic dual-link redundant connection, a 6G plus fiber optic dual-link redundant connection, or other high-speed dual-link combinations that meet the requirements of high-speed transmission; wherein the 5G link adopts SA standalone networking mode, the 6G link adopts the terahertz frequency band, and the fiber optic link adopts single-mode fiber and has high-speed transmission capability.

9. The multi-sensor fusion robot remote control platform as described in claim 1, characterized in that: The status monitoring unit is used to monitor the working status of multi-source sensors, edge fusion module, and actuator in real time; the sensor fault judgment threshold is set as data loss or abnormal duration ≥ preset threshold. When an abnormality occurs, an alarm signal is sent and a redundancy switching mechanism is triggered. After switching, a simplified weighted fusion algorithm is used for temporary fusion; after the faulty sensor recovers to normal and the data is continuous and stable, it automatically reverts to the original hierarchical fusion algorithm.

10. The multi-sensor fusion robot remote control platform as described in claim 1, characterized in that: The data preprocessing unit employs median filtering, histogram equalization, and Kalman filtering algorithms to remove noise from lidar point clouds, calibrate binocular camera image distortion, and eliminate zero-bias errors in IMU attitude data, respectively. The multi-source sensor unit is flexibly replaceable, and the actuator includes a robot joint actuator and a mobile chassis.

Images