Robot environment recognition system based on multi-mode perception

Abstract

The invention discloses a robot environment recognition system based on multi-modal sensing, and aims to solve the problems of environment recognition precision reduction and insufficient robustness caused by single-modal sensing failure, insufficient multi-modal information fusion mechanism and dynamic motion interference in an extreme environment. The system comprises a multi-modal sensing module, a feature depth fusion module, an environment dynamic modeling module and an identification decision module, and finally outputs an environment identification result and an operation decision by synchronously acquiring multi-modal data, performing cross-modal feature alignment and adaptive weight fusion, constructing a dynamic three-dimensional semantic model and compensating motion distortion. According to the invention, the identification precision, the adaptive capability and the overall reliability of the system in extreme and dynamic environments are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of robot environmental perception technology, and in particular to a robot environmental recognition system based on multimodal perception. Background Technology

[0002] Robot environmental recognition technology, as a key component of autonomous systems interacting with their environment, has been widely applied in fields such as industrial inspection, special operations, and intelligent services. Among these, multimodal perception technology, by fusing data from heterogeneous sensors such as vision, laser, and inertial sensors, aims to improve the recognition accuracy and system robustness of robots in complex and unstructured environments.

[0003] In specialized operational scenarios such as ash removal and coke removal in incinerators, existing robotic environmental recognition systems face severe challenges. Current technologies typically rely on a single vision sensor or simple multi-sensor data stacking, making it difficult to adapt to the severe image quality degradation caused by harsh conditions inside the furnace, such as high temperatures, pervasive dust, and uneven lighting. This leads to feature extraction failures or target recognition errors. Furthermore, robots must enter furnace spaces up to twenty meters high through narrow entrances. Traditional recognition methods lack deep fusion and collaborative optimization mechanisms for multimodal sensory information under extreme geometric constraints, making them prone to recognition failures due to sensor occlusion or data conflicts.

[0004] Furthermore, existing systems, when operating in high-altitude spaces, struggle to achieve a balance between environmental recognition accuracy and robot operational stability due to insufficient consideration of the interference of the robot's extended arm's dynamic motion on sensor observations and a lack of effective modeling of the complex three-dimensional structure within the furnace. In delicate operations such as ash removal and coke removal, these issues can easily lead to robot misidentification of cleaning targets or loss of operational control, seriously threatening operational safety and efficiency. Therefore, there is an urgent need for a robotic environmental recognition system capable of adapting to extreme environments and achieving efficient multimodal perception and collaboration. Summary of the Invention

[0005] The purpose of this invention is to provide a robot environment recognition system based on multimodal perception, which solves the problems mentioned in the background art.

[0006] This invention is implemented as follows: a robot environment recognition system based on multimodal perception, comprising a multimodal perception module, a feature deep fusion module, an environment dynamic modeling module, and a recognition decision module. The multimodal perception module simultaneously acquires visual image data, 3D laser point cloud data, and inertial measurement data of the robot's operating environment. The feature deep fusion module, connected to the multimodal perception module, performs cross-modal feature alignment and adaptive weight fusion on the heterogeneous perception data. The environment dynamic modeling module, connected to both the multimodal perception module and the feature deep fusion module, constructs a dynamic 3D semantic model of the scene and compensates for observation distortions caused by robot motion. The recognition decision module integrates the feature deep fusion module and the environment dynamic modeling module, outputting environment recognition results and generating robot operation decision commands based on the fused features and semantic model.

[0007] Furthermore, the multimodal perception module includes a high dynamic range (HDR) vision unit, a solid-state LiDAR unit, and a high-precision inertial measurement unit (IMU). The HDR vision unit employs two global shutter cameras with a resolution of 2448 x 2048, acquiring wide dynamic range scene images at a rate of 20 frames per second, achieving a dynamic range of 120 dB. The solid-state LiDAR unit uses a 128-channel area array scanning architecture, acquiring high-density environmental point cloud data at a frequency of 15 Hz, achieving a point cloud density of 10,000 points per square meter. The high-precision IMU integrates a fiber optic gyroscope and a quartz flexural accelerometer; the gyroscope's zero-bias stability reaches 0.01 degrees per hour, and the accelerometer's zero-bias stability reaches 50 micrograms, with a data output frequency of 500 Hz. The multimodal perception module also features a hardware-level synchronization triggering unit, ensuring that the timestamp deviation of all sensor data is less than 0.5 milliseconds through a precise clock protocol.

[0008] Furthermore, the feature deep fusion module adopts a three-level cross-modal fusion architecture. The first level is the data preprocessing and alignment unit, which performs spatiotemporal registration on the raw perceptual data, including radiometric distortion correction and dehazing enhancement of visual images, motion distortion compensation and downsampling of laser point clouds, and attitude calculation and coordinate system transformation of inertial data. The second level is the feature extraction and encoding unit, which extracts complementary features from the aligned data, including dense semantic feature maps of vision, geometric structural feature vectors of lasers, and motion trajectory feature sequences of inertial data. The third level is the adaptive fusion inference unit, which constructs a cross-modal feature interaction network based on a multi-head attention mechanism. This network contains four attention heads, each of which calculates the correlation weights between features of different modalities, and achieves dynamic weighted summation and dimensionality fusion of feature vectors through a gated recurrent unit.

[0009] Furthermore, the environmental dynamic modeling module includes a scene semantic reconstruction unit and a motion distortion compensation unit. The scene semantic reconstruction unit, based on the fused features output by the feature deep fusion module, employs an incremental 3D semantic occupancy grid map construction method. This method discretizes the environmental space into cubic grids with sides of 0.05 meters. Each grid stores its occupancy probability value and the object category confidence represented by an 8-dimensional semantic vector. The semantic vectors are normalized using a softmax function and updated at a frequency of 10 Hz. The motion distortion compensation unit, based on the robot pose change rate provided by the high-precision inertial measurement unit, uses a Bayesian filtering method to predict and compensate for feature drift in the sensor observation point cloud and image caused by the robot's own motion, achieving sub-pixel level compensation accuracy.

[0010] Furthermore, the identification and decision-making module includes a scene understanding unit and a task planning unit. The scene understanding unit analyzes the 3D semantic occupancy grid map output by the environmental dynamic modeling module in real time, identifying key operational targets, obstacle areas, and passable spaces. This unit uses a combination of a rule-based inference engine and a deep learning-based classifier to perform multi-label classification on each semantic grid, with a classification confidence threshold set to 0.9. The task planning unit generates the robot's operation sequence based on the scene understanding results, including waypoints, end-effector posture, and operational force parameters. The planning cycle is 100 milliseconds, and a model predictive control algorithm is used to ensure the smoothness and safety of the operation.

[0011] Furthermore, the system also includes an online adaptive learning module. This module continuously monitors the accuracy and consistency of the environmental recognition results. When it detects that the recognition accuracy drops by more than 10 percentage points continuously under specific environmental conditions, it automatically triggers an online fine-tuning mechanism for the model parameters. This mechanism utilizes newly collected small batches of data to update the gradient of the cross-modal feature interaction network in the feature deep fusion module through a backpropagation algorithm. The learning rate is set to 0.0001, and a single fine-tuning takes less than 200 milliseconds.

[0012] Furthermore, the system adopts a heterogeneous computing platform architecture. Sensor data stream processing, deep learning model inference, and real-time motion planning tasks are offloaded to three different computing units: a graphics processing unit (GPU), a neural network processor (NNPM), and a field-programmable gate array (FPGA). The computing units exchange data via a high-speed on-chip interconnect bus, achieving a peak bandwidth of 32 gigabits per second and an end-to-end processing latency of less than 30 milliseconds.

[0013] Furthermore, the feature deep fusion module also integrates a perception quality assessment unit. This unit dynamically adjusts the initial weights of different modalities in the fusion architecture by analyzing the signal-to-noise ratio, contrast, and integrity indices of the raw data for each modality. In environments where the dust concentration instantaneously exceeds 150 milligrams per cubic meter, the initial weight of the visual modality is automatically lowered to 0.2, while the weight of the laser modality is increased to 0.8. When severe mechanical vibration is detected causing inertial data anomalies, the weight of the inertial modality is temporarily set to zero, and the system switches to a pure vision-laser fusion mode.

[0014] Furthermore, the hardware-level synchronization triggering unit of the multimodal sensing module adopts a global triggering scheme based on white pulse signals. The main control unit generates a precise synchronization pulse with a period of 50 milliseconds. This pulse is simultaneously distributed via coaxial cable to the exposure control pin of the high dynamic range vision unit, the scanning trigger interface of the solid-state LiDAR unit, and the data sampling clock input of the high-precision inertial measurement unit, achieving absolute time synchronization at the physical level.

[0015] Furthermore, the cross-modal feature interaction network of the feature deep fusion module is pre-trained using a contrastive learning strategy during the training phase. The network learning objective is to minimize the distance between feature vectors from different modalities in the same scene within the embedding space, while maximizing the distance between feature vectors from different scenes. Pre-training uses a multimodal dataset containing 10,000 labeled scenes, with a training cycle of 200 rounds.

[0016] Furthermore, the incremental 3D semantic occupancy raster map of the environmental dynamic modeling module uses an octree data structure for efficient storage and updating. Each non-leaf node contains pointers to 8 child nodes, while the leaf nodes store the occupancy probability and semantic vector of the raster. When new fused observation data arrives, the system updates the state of all rasteres along the path from the sensor origin to the observation point using a ray casting algorithm, with the update operation having a time complexity on the logarithmic level.

[0017] Furthermore, the model predictive control algorithm for the identification decision module employs a rolling time-domain optimization framework. Within each control cycle, the algorithm solves an optimal control problem within a finite time domain. The objective function includes a tracking error term, a control smoothness term, and an obstacle distance penalty term. The optimization problem is solved using a sequential quadratic programming method, with a prediction time domain length of 2 seconds and a control time domain length of 0.5 seconds.

[0018] Furthermore, the online adaptive learning module's online model parameter fine-tuning mechanism incorporates a catastrophic forgetting prevention strategy. This strategy determines the degree of knowledge conflict by calculating the inner product of the gradient directions of new data and historical data. When the inner product is negative, it indicates that new knowledge may conflict with old knowledge. In this case, the fine-tuning process introduces elastic weights to reinforce the regularization term, protecting important old network parameters from excessive modification.

[0019] Furthermore, the heterogeneous computing platform architecture includes a dynamic task scheduler. This scheduler monitors the load rate and task queue length of each computing unit in real time. When the graphics processor load consistently exceeds 90%, it automatically migrates some feature extraction computation tasks to the neural network processor. Scheduling decisions are based on an improved shortest job first algorithm, evaluating and performing task reallocation every 100 milliseconds.

[0020] Furthermore, the perception quality assessment unit also includes a sensor fault diagnosis and recovery submodule. This submodule diagnoses whether a sensor has experienced a hard or soft fault by analyzing the differences between the historical statistical characteristics of each sensor's data stream and the current values. For diagnosed soft faults, such as temporary lens contamination, the system automatically initiates a cleaning mechanism or switches to a backup algorithm model; for hard faults, it sends an alarm to the monitoring system and records the fault log.

[0021] Compared with existing technologies, the advantages and positive effects of this invention are as follows: This invention achieves high-fidelity, high-synchronization multimodal data acquisition in extremely harsh environments through a high-performance sensor array and hardware-level synchronization mechanism in the multimodal perception module, providing a high-quality data foundation for subsequent processing. The three-level cross-modal fusion architecture and attention-based feature interaction network of the feature deep fusion module overcome the limitations of traditional simple data stacking or post-decision fusion, achieving deep collaboration at the data, feature, and decision levels, significantly improving the robustness and discriminative power of feature representation in the event of single-modal failure or degradation. The incremental three-dimensional semantic occupancy grid map constructed by the environmental dynamic modeling module not only provides an accurate geometric model of the environment but also endows it with rich semantic information and can compensate for observation errors caused by robot movement in real time, ensuring that the environmental model remains accurate and consistent. The recognition and decision module, combining scene understanding and model predictive control, realizes closed-loop intelligent decision-making from environmental perception to robot operation, improving the autonomy and safety of operations. The online adaptive learning module enables the system to continuously optimize performance during operation and proactively adapt to environmental changes. The heterogeneous computing platform architecture and dynamic task scheduling ensure the efficient and stable operation of complex algorithms under stringent real-time requirements. Perception quality assessment and sensor fault-tolerance mechanisms further enhance the overall reliability of the system. Through the tight coupling and collaborative optimization of its modules, the overall system fundamentally solves the core challenge of environmental recognition in extreme, dynamic, and unstructured environments, providing crucial technical support for reliable autonomous robot operation in complex industrial scenarios. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall technical architecture of a robot environment recognition system based on multimodal perception proposed in this invention; Figure 2This is a structural block diagram of the multimodal perception module of a robot environment recognition system based on multimodal perception proposed in this invention; Figure 3 This is a structural block diagram of a feature deep fusion module for a robot environment recognition system based on multimodal perception proposed in this invention; Figure 4 This is a structural block diagram of the environmental dynamic modeling module of a robot environmental recognition system based on multimodal perception proposed in this invention; Figure 5 This is a structural block diagram of the recognition decision module of a robot environment recognition system based on multimodal perception proposed in this invention; Figure 6 This is a structural block diagram of an online adaptive learning module for a robot environment recognition system based on multimodal perception proposed in this invention; Figure 7 This is a structural block diagram of a heterogeneous computer platform for a robot environment recognition system based on multimodal perception proposed in this invention. Figure 8 This is a structural block diagram of the perception quality assessment unit of a robot environment recognition system based on multimodal perception proposed in this invention; Figure 9 This is a structural block diagram of a catastrophic forgetting prevention strategy for a robot environment recognition system based on multimodal perception proposed in this invention. Detailed Implementation

[0023] Example 1 This embodiment details the specific technical implementation of a robot environment recognition system based on multimodal perception. The system consists of a multimodal perception module, a feature deep fusion module, an environmental dynamic modeling module, a recognition decision module, an online adaptive learning module, a heterogeneous computing platform architecture, and a perception quality assessment unit. Each module interacts with shared memory via a high-speed on-chip interconnect bus, ensuring high-precision environmental recognition and intelligent decision-making even in extreme dynamic operating environments.

[0024] The multimodal perception module is responsible for simultaneously acquiring visual image data, 3D laser point cloud data, and inertial measurement data of the robot's operating environment. This module includes a high dynamic range (HDR) vision unit, a solid-state LiDAR unit, and a high-precision inertial measurement unit. The HDR vision unit uses two global shutter cameras with a resolution of 2448 x 2048 pixels, acquiring wide dynamic range scene images at a rate of 20 frames per second, achieving a dynamic range of 120 dB. The cameras achieve wide dynamic range imaging through double exposure technology: a short exposure time of 0.1 ms to capture highlight details and a long exposure time of 10 ms to capture shadow details; the two images are fused in an image signal processor. The solid-state LiDAR unit uses a 128-channel area array scanning architecture, acquiring high-density environmental point cloud data at a frequency of 15 Hz, achieving a point cloud density of 10,000 points per square meter. This unit emits laser pulses with a wavelength of 905 nm, achieving two-dimensional scanning through a microelectromechanical system (MEMS) galvanometer; each channel has a field of view of 0.1 degrees x 0.1 degrees. The high-precision inertial measurement unit integrates a fiber optic gyroscope and a quartz flexure accelerometer. The gyroscope achieves a zero-bias stability of 0.01 degrees per hour, and the accelerometer achieves a zero-bias stability of 50 micrograms per second. The data output frequency is 500 Hz. The multimodal sensing module also features a hardware-level synchronization trigger unit, which ensures that the timestamp deviation of all sensor data is less than 0.5 milliseconds through a precise clock protocol. The hardware-level synchronization trigger unit adopts a global triggering scheme based on white pulse signals. The main control unit generates a precise synchronization pulse with a period of 50 milliseconds. This pulse is simultaneously distributed through a 50-ohm coaxial cable to the exposure control pin of the high dynamic range vision unit, the scan trigger interface of the solid-state LiDAR unit, and the data sampling clock input of the high-precision inertial measurement unit, achieving absolute time synchronization at the physical level.

[0025] The feature deep fusion module receives heterogeneous sensing data collected by the multimodal perception module and performs cross-modal feature alignment and adaptive weight fusion. This module adopts a three-level cross-modal fusion architecture, including a first-level data preprocessing and alignment unit, a second-level feature extraction and encoding unit, and a third-level adaptive fusion inference unit. The first-level data preprocessing and alignment unit performs spatiotemporal registration on the raw sensing data: For images acquired by the high dynamic range vision unit, it performs radiometric distortion correction and dehazing enhancement. Radiometric distortion correction corrects the nonlinear response of each pixel using a lookup table, while dehazing enhancement uses a dark channel prior algorithm to restore image clarity. For point cloud data acquired by the solid-state lidar unit, it performs motion distortion compensation and downsampling. Motion distortion compensation is based on the instantaneous angular velocity and linear velocity provided by the high-precision inertial measurement unit, and linear interpolation corrects the true position of each laser point at its emission moment. Downsampling uses voxel grid filtering to merge all point clouds within a 0.01-meter cube space into its centroid. For high-precision inertial measurement unit data, it performs attitude calculation and coordinate system transformation. Attitude calculation uses a quaternion complementary filtering algorithm, fusing gyroscope angular velocity and accelerometer gravity vector. Coordinate system transformation converts the inertial data from the sensor coordinate system to the robot base coordinate system. The second-level feature extraction and encoding unit extracts complementary features from the aligned data: it extracts dense semantic feature maps from the visual image using a convolutional neural network with an encoder-decoder structure. The input image size is 2448 x 2048, and the output is the original... Figure 4 A 256-channel feature map with a resolution of 1 / 1000 is used, with each pixel corresponding to a 64-dimensional semantic feature vector. Geometric structure feature vectors are extracted from the laser point cloud using a point cloud deep learning network. The input is the downsampled point cloud, and the network contains four feature extraction layers. Each layer aggregates local features through farthest point sampling and sphere query, ultimately outputting a 1024-dimensional global feature vector. Motion trajectory feature sequences are extracted from inertial data using a one-dimensional convolutional network to process the attitude quaternions and acceleration sequences for 100 consecutive time steps, outputting a 128-dimensional temporal feature vector. The third-level adaptive... The fusion inference unit constructs a cross-modal feature interaction network based on a multi-head attention mechanism. This network contains four attention heads, each calculating the correlation weights between features from different modalities. For visual feature maps, they are flattened into sequences; for laser and inertial feature vectors, they are mapped to the same hidden dimension as the visual features through fully connected layers. The attention weight calculation process is as follows: Where Q is the query matrix, derived from features of one modality, K and V are key-value matrices, derived from features of another modality, and d_k is the dimension of the key vector, set to 64. After parallel computation by four attention heads, their outputs are concatenated along the feature dimension and fused through a feedforward neural network. Finally, the fused features are dynamically weighted and dimensionally fused through a gated recurrent unit, outputting a unified 1024-dimensional environment representation vector. This cross-modal feature interaction network is pre-trained using a contrastive learning strategy during the training phase. The network learning objective is to minimize the distance between feature vectors from different modalities in the same scene within the embedding space, while maximizing the distance between feature vectors from different scenes. Pre-training uses a multimodal dataset containing 10,000 labeled scenes, with 200 training cycles, a batch size of 16, and the AdamW optimizer with an initial learning rate of 0.001.

[0026] The environmental dynamic modeling module constructs a dynamic 3D semantic model of the scene and compensates for observation distortions caused by robot motion, based on the fused features output by the feature deep fusion module and the raw observation data provided by the multimodal perception module. This module includes a scene semantic reconstruction unit and a motion distortion compensation unit. The scene semantic reconstruction unit employs an incremental 3D semantic occupancy grid map construction method. This method discretizes the environmental space into cubic grids with a side length of 0.05 meters. Each grid stores its occupancy probability and the object category confidence represented by an 8-dimensional semantic vector. The semantic vector is normalized using a softmax function to ensure that the sum of all category probabilities is 1. The map uses an octree data structure for efficient storage and updating. Each non-leaf node contains pointers to 8 child nodes, and the leaf nodes store the occupancy probability and semantic vector of the grid. When new fused observation data arrives, the system updates the state of all grids along the path from the sensor origin to the observation point using a ray casting algorithm. For grids traversed by rays, their occupancy probability is updated logarithmically based on the inverse sensor model. For grids at the endpoints of rays, their semantic vectors are fused with the currently observed semantic features using a Bayesian update rule. The map update frequency is 10 Hz, and the time complexity of the update operation is logarithmic. The motion distortion compensation unit, based on the robot pose change rate provided by the high-precision inertial measurement unit, uses an extended Kalman filter to predict and compensate for feature drift in the sensor-observed point cloud and image caused by the robot's own motion. The filter state vector contains 13 dimensions, including robot position, velocity, attitude, and sensor time offset. The observation model combines laser ranging values ​​with visual feature matching results. The compensation process first predicts the robot pose at each data acquisition moment, and then corrects the observation data to a unified reference moment through rigid body transformation, achieving sub-pixel level compensation accuracy.

[0027] The identification and decision-making module, based on the environmental representation vector output by the feature deep fusion module and the 3D semantic map constructed by the environmental dynamic modeling module, outputs environmental identification results and generates robot operation decision commands. This module includes a scene understanding unit and a task planning unit. The scene understanding unit analyzes the 3D semantic occupancy grid map output by the environmental dynamic modeling module in real time, identifying key operational targets, obstacle areas, and passable spaces. This unit combines a rule-based inference engine with a deep learning-based classifier. The inference engine makes a preliminary judgment based on the maximum value category and its confidence level of the semantic vector; for example, when the confidence level of the "clearing target" category in a grid exceeds 0.9, it is marked as a potential operational point. The deep learning classifier is a 3-layer fully connected network; the input is a local 3D semantic grid block centered on the target point, and the output is a fine-grained classification label and spatial bounding box for that region. The classification confidence threshold is set to 0.9. The task planning unit generates the robot's operation sequence based on the scene understanding results. The planning cycle is 100 milliseconds, and a model predictive control algorithm is used to ensure the smoothness and safety of the operation. The model predictive control algorithm employs a rolling time-domain optimization framework. Within each control cycle, the algorithm solves an optimal control problem within a finite time domain. The objective function comprises three terms: a tracking error term, penalizing the deviation between the end effector position and the target position; a control smoothness term, penalizing drastic changes in joint acceleration; and an obstacle distance penalty term, which calculates the minimum distance between the robotic arm link and obstacles based on obstacle information in the semantic map, applying a secondary penalty when the distance is less than 0.2 meters. The optimization problem is solved using a sequential quadratic programming method, with a prediction time domain length of 2 seconds and a control time domain length of 0.5 seconds. The first element of the resulting control sequence is sent to the robot's underlying controller for execution.

[0028] The online adaptive learning module continuously monitors the performance of the environment recognition system. This module includes a performance monitoring unit and a model fine-tuning unit. The performance monitoring unit calculates the accuracy and consistency metrics of the scene understanding unit's output in real time. Accuracy is determined by comparing the online recognition results with a small number of manually labeled or high-confidence historical results; the consistency metric is calculated by checking the spatial stability of recognition results across multiple consecutive frames. When it is detected that under specific environmental conditions, such as a sudden increase in dust concentration, the recognition accuracy drops by more than 10 percentage points in 10 consecutive calculations, the online model parameter fine-tuning mechanism is automatically triggered. The model fine-tuning unit uses a small batch of newly collected data, typically the most recent 100 frames of multimodal data and their automatically generated pseudo-labels, to update the gradient of the cross-modal feature interaction network in the feature deep fusion module using the backpropagation algorithm. The learning rate is set to 0.0001, and the time for a single fine-tuning iteration is less than 200 milliseconds. To prevent catastrophic forgetting, the fine-tuning mechanism includes a protection strategy. This strategy determines the degree of knowledge conflict by calculating the inner product of the gradient of the new data and the gradient direction of the old data randomly sampled from the historical data buffer. When the inner product is negative, it indicates that new knowledge may conflict with old knowledge. In this case, the fine-tuning process will introduce elastic weights to reinforce the regularization term. The regularization term takes the form of: Here, `i` iterates through all the weight parameters of the network, `θ_i` is the current parameter value, `θ_i*` is the important parameter value in the old task, `F_i` is the importance estimate of the corresponding parameter on the diagonal of the Fisher information matrix, and `λ` is the regularization strength coefficient, set to 1000. This strategy protects important old network parameters from being excessively modified.

[0029] The system employs a heterogeneous computing platform architecture to ensure the real-time performance of complex algorithms. This architecture offloads sensor data stream processing, deep learning model inference, and real-time motion planning tasks to three different computing units: a graphics processing unit (GPU), a neural network processor (NNP), and a field-programmable gate array (FPGA). The GPU runs the convolutional neural network and attention network in the feature deep fusion module, utilizing CUDA cores for parallel computation. The NLP is dedicated to running the deep learning classifier in the scene understanding unit, leveraging its tensor cores for low-power, high-efficiency inference. The FPGA implements the sequential quadratic programming solver in the model predictive control algorithm in hardware, compressing the solution time to less than 5 milliseconds through pipelined design. Data exchange between computing units occurs via a high-speed on-chip interconnect bus using the AXI4 streaming protocol, achieving a peak bandwidth of 32 gigabits per second. The total end-to-end processing latency from sensor data acquisition to control command output is less than 30 milliseconds. The architecture includes a dynamic task scheduler that monitors the load rate and task queue length of each computing unit in real time, with a sampling period of 10 milliseconds. When the graphics processing unit (GPU) load exceeds 90% for three consecutive cycles, some visual feature extraction computation tasks are automatically migrated to the neural network processor. The scheduling decision is based on an improved shortest job first algorithm, which comprehensively considers the estimated computational load of the task, data dependencies, and the current load of the target computing unit, and evaluates and performs task reallocation every 100 milliseconds.

[0030] The feature deep fusion module also integrates a perception quality assessment unit. This unit dynamically adjusts the initial weights of different modalities in the fusion architecture by analyzing the signal-to-noise ratio, contrast, and integrity indices of the raw data for each modality. For visual images, the signal-to-noise ratio is obtained by calculating the ratio of the standard deviation to the mean of the smoothed region of the image; the contrast is obtained by calculating the entropy value of the image's gray-level histogram; and the integrity is obtained by detecting the proportion of invalid pixels in the image. For laser point clouds, the signal-to-noise ratio is obtained by analyzing the distribution of point cloud reflection intensity; and the integrity is obtained by statistically analyzing the deviation between the number of point clouds in a single frame and the historical mean. In environments where the dust concentration instantaneously exceeds 150 milligrams per cubic meter, the contrast and integrity indices of visual images will significantly decrease. At this time, the initial weight of the visual modality is automatically lowered to 0.2, and the weight of the laser modality is increased to 0.8. When severe mechanical vibration is detected causing the amplitude of the inertial data angular velocity to continuously exceed the limit, the inertial data is determined to be abnormal, the weight of the inertial modality is temporarily set to zero, and the system switches to a pure visual-laser fusion mode until the inertial data returns to normal for more than 2 seconds. The perception quality assessment unit also includes a sensor fault diagnosis and recovery submodule. This submodule analyzes the historical statistical characteristics of each sensor's data stream, such as mean, variance, and autocorrelation, and performs hypothesis testing against the statistical values ​​within the current sliding window. For diagnosed soft faults, such as minor temporary lens smudges caused by fly ash resulting in localized image blurring, the system automatically activates a smudge detection algorithm based on image gradients and reduces the weight of features in that region during feature fusion, or switches to a pre-trained backup visual feature extraction model robust to smudges. For hard faults, such as no data output from the LiDAR, a level-three alarm is sent to the central monitoring system, recording the fault time, sensor identifier, and possible fault codes, and an attempt is made to reset the sensor via bus commands.

[0031] In practical implementation, the sensors of the multimodal perception module require rigorous joint calibration. The high dynamic range vision unit needs binocular stereo calibration to obtain the intrinsic, distortion coefficients, and extrinsic parameters of the left and right cameras. The solid-state LiDAR unit and the vision unit need joint laser vision calibration, obtaining the transformation matrix from the laser coordinate system to the camera coordinate system using a calibration board. The installation position and attitude of all sensors relative to the robot's base coordinate system need to be accurately measured using a total station. The cross-modal feature interaction network of the feature deep fusion module needs end-to-end training on a large-scale simulation and experimental mixed dataset. The octree map resolution and update probability model parameters of the environmental dynamic modeling module need to be configured according to the spatial scale of the operating environment and the sensor accuracy. The weight matrix of the model predictive control algorithm in the recognition decision module needs to be determined through simulation and physical debugging to balance tracking accuracy, motion smoothness, and safety. The historical data buffer size of the online adaptive learning module is set to 10,000 samples, managed using a first-in, first-out (FIFO) strategy. The firmware and drivers for heterogeneous computing platform architectures need to be deeply optimized for specific graphics processors, neural network processors, and field-programmable gate arrays to fully utilize hardware performance.

[0032] Upon system startup, a complete self-test and initialization process is executed: the multimodal perception module checks all sensor connections and loads calibration parameters; the feature deep fusion module loads pre-trained network weights; the environmental dynamic modeling module initializes an empty octree map, setting the origin to the robot's initial position; the recognition and decision-making module loads the task configuration file; and the heterogeneous computing platform initializes each computing unit and establishes communication links. After the self-test passes, the system enters a real-time running loop.

[0033] This embodiment constructs a robot environment recognition system capable of adapting to extremely harsh and highly dynamic environments through the deep integration and collaborative work of the aforementioned modules. The multimodal perception module provides highly synchronized, high-quality multi-source data; the feature deep fusion module achieves intelligent complementarity and enhancement of cross-modal information; the environmental dynamic modeling module generates accurate and semantically rich real-time environmental models; the recognition decision module generates safe and efficient operation instructions based on the model; the online adaptive learning module endows the system with the ability to continuously evolve and adapt to new environments; the heterogeneous computing platform architecture ensures the smooth operation of the entire system under stringent real-time requirements; and the perception quality assessment and fault-tolerance mechanism guarantees the long-term reliability of the system. This system is particularly suitable for complex industrial scenarios such as incinerator ash removal and coke removal, effectively improving the autonomy, accuracy, and safety of robot operations.

[0034] Example 2 This embodiment, based on Embodiment 1, enhances and expands the functionality and architecture of the robot environment recognition system based on multimodal perception to meet the needs of higher accuracy and more complex dynamic scenarios. The main enhancements include super-resolution and event camera integration in the multimodal perception module, spatiotemporal graph neural network fusion in the feature deep fusion module, neural radiation field representation in the environmental dynamic modeling module, and a reinforcement learning decision-maker in the recognition decision module.

[0035] The multimodal sensing module integrates an event camera and a super-resolution LiDAR on top of the existing sensor array. The event camera uses a dynamic vision sensor with a resolution of 1280x720, responding only to changes in pixel brightness and outputting an asynchronous event stream. Its time resolution reaches the microsecond level, and its dynamic range exceeds 120 dB, enabling it to capture high-speed motion and subtle changes in lighting. The super-resolution LiDAR, based on a solid-state LiDAR, utilizes optical diffraction elements and sub-pixel scanning technology to increase the effective point cloud density to 50,000 points per square meter. It also integrates spectral analysis capabilities, acquiring near-infrared spectral characteristics of each point in addition to distance and reflection intensity, for preliminary material identification. The hardware-level synchronization trigger unit has been upgraded to a master clock supporting multi-protocol synchronization. In addition to the original white pulse, a new precise time protocol network synchronization interface has been added, ensuring that the synchronization accuracy between the event camera's timestamp and the global clock is within 10 microseconds. The high dynamic range vision unit adds polarization imaging functionality, acquiring polarized images in four directions (0°, 45°, 90°, and 135°) by rotating a polarizer. This is used to suppress glare on highly reflective surfaces and enhance surface texture information.

[0036] The feature deep fusion module's architecture evolved into a spatiotemporal graph neural network fusion architecture. This architecture models observation data from different modalities and at different times as a dynamic heterogeneous graph. Nodes in the graph are divided into three categories: visual feature nodes, derived from keypoints on the dense feature map of each frame; laser feature nodes, derived from feature points in the point cloud of each frame; and inertial nodes, derived from the attitude and velocity at each timestamp. Edges are divided into two categories: spatial edges, connecting nodes from different modalities at the same time, with their weights calculated by a cross-modal attention mechanism; and temporal edges, connecting nodes from adjacent times within the same modality, with their weights calculated by temporal correlation. The graph neural network contains five graph convolutional layers, each aggregating neighbor node information and updating node features. Finally, a global graph feature is obtained through graph pooling, which simultaneously contains multimodal information and spatiotemporal evolution information, with a dimension of 2048. This architecture is particularly adept at handling motion prediction and interaction relationship reasoning for targets in dynamic scenes.

[0037] The scene representation method of the environmental dynamic modeling module is upgraded from discrete semantic occupancy grids to continuous neural radiation fields. The neural radiation field uses a multilayer perceptron to map 3D spatial coordinates and viewing direction to voxel density and color values. In this system, the input to the multilayer perceptron includes not only coordinates and direction but also multimodal feature embeddings from the feature deep fusion module. During neural radiation field training, multi-frame, multi-view visual images and laser point clouds are used as supervision signals, and differentiable volume rendering technology is used to optimize network parameters. After training, the neural radiation field can generate high-fidelity color images and depth maps from any viewpoint, and the semantic attributes of any point can be obtained by querying the feature embedding of any point in space. The motion distortion compensation unit is directly performed in this continuous field representation, achieved by applying an inverse transformation based on inertial data to the sampling points on the observation ray, avoiding errors caused by discretization.

[0038] The task planning unit of the decision-making module introduces a deep reinforcement learning-based decision-maker as the high-level policy. This decision-maker takes the neural radiation field characteristics output by the environmental dynamic modeling module, the current robot state, and the task objective as input, and outputs an abstract action command, such as "approach the target," "obstacle avoidance," or "fine alignment." The decision-maker is trained using a proximal policy optimization algorithm in a simulation environment containing various dynamic obstacles, target movement, and partial sensor failures. The model predictive control algorithm acts as the low-level actuator, receiving the high-level abstract command and converting it into specific joint trajectories. This hierarchical decision-making architecture ensures both the intelligence and adaptability of high-level decision-making while preserving the high precision and safety of low-level control.

[0039] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robot environment recognition system based on multimodal perception, characterized in that, include: The multimodal perception module is used to simultaneously acquire visual image data, 3D laser point cloud data, and inertial measurement data of the robot's working environment; The feature deep fusion module, connected to the multimodal perception module, is used to perform cross-modal feature alignment and adaptive weight fusion on heterogeneous perception data; the environment dynamic modeling module, which connects both the multimodal perception module and the feature deep fusion module, is used to construct a dynamic 3D semantic model of the scene and compensate for observation distortion caused by robot motion. The recognition and decision-making module integrates a deep feature fusion module and an environmental dynamic modeling module. It outputs environmental recognition results and generates robot operation decision commands based on the fused features and semantic model. The multimodal perception module includes a high dynamic range vision unit, a solid-state LiDAR unit, and a high-precision inertial measurement unit. The high dynamic range vision unit uses two global shutter cameras with a resolution of 2448 x 2048, acquiring wide dynamic range scene images at a rate of 20 frames per second, achieving a dynamic range of 120 dB. The solid-state LiDAR unit uses a 128-channel area array scanning architecture, acquiring high-density environmental point cloud data at a frequency of 15 Hz, achieving a point cloud density of 10,000 points per square meter. The high-precision inertial measurement unit integrates a fiber optic gyroscope and a quartz flexural accelerometer. The gyroscope's zero-bias stability reaches 0.01 degrees per hour, and the accelerometer's zero-bias stability reaches 50 micrograms, with a data output frequency of 500 Hz. The multimodal perception module also features a hardware-level synchronization triggering unit, ensuring that the timestamp deviation of all sensor data is less than 0.5 milliseconds through a precise clock protocol.

2. The robot environment recognition system based on multimodal perception according to claim 1, characterized in that, The feature deep fusion module adopts a three-level cross-modal fusion architecture; the first-level data preprocessing and alignment unit performs spatiotemporal registration on the original sensing data, including radiation distortion correction and dehazing enhancement of visual images, motion distortion compensation and downsampling of laser point clouds, and attitude calculation and coordinate system transformation of inertial data; The second-level feature extraction and encoding unit extracts complementary features from the aligned data, including dense semantic feature maps of vision, geometric feature vectors of lasers, and inertial motion trajectory feature sequences. The third-level adaptive fusion inference unit constructs a cross-modal feature interaction network based on a multi-head attention mechanism. This network contains four attention heads, each of which calculates the correlation weights between features of different modalities. The feature vectors are dynamically weighted and dimensionally fused through a gated recurrent unit.

3. The robot environment recognition system based on multimodal perception according to claim 1, characterized in that, The environmental dynamic modeling module includes a scene semantic reconstruction unit and a motion distortion compensation unit. The scene semantic reconstruction unit is based on the fusion features output by the feature deep fusion module and adopts an incremental three-dimensional semantic occupancy grid map construction method. This method discretizes the environmental space into a cubic grid with a side length of 0.05 meters. Each grid stores its probability value of being occupied and the object category confidence represented by an 8-dimensional semantic vector. The semantic vector is normalized using the softmax function and updated at a frequency of 10 Hz. The motion distortion compensation unit is based on the robot pose change rate provided by the high-precision inertial measurement unit. It uses Bayesian filtering to predict and compensate for the feature drift of the sensor observation point cloud and image caused by the robot's own motion, achieving a compensation accuracy of sub-pixel level.

4. The robot environment recognition system based on multimodal perception according to claim 1, characterized in that, The identification and decision-making module includes a scene understanding unit and a task planning unit. The scene understanding unit analyzes the 3D semantic occupancy grid map output by the environmental dynamic modeling module in real time to identify key operational targets, obstacle areas, and passable spaces. This unit uses a combination of a rule-based inference engine and a deep learning-based classifier to perform multi-label classification on each semantic grid, with a classification confidence threshold set to 0.

9. The task planning unit generates the robot's operation sequence based on the scene understanding results, including waypoints, end-effector postures, and operational force parameters. The planning period is 100 milliseconds, and a model predictive control algorithm is used to ensure the smoothness and safety of the operation.

5. A robot environment recognition system based on multimodal perception according to claim 1, characterized in that, The system also features an online adaptive learning module. This module continuously monitors the accuracy and consistency of the environmental recognition results. When it detects that the recognition accuracy drops by more than 10 percentage points under specific environmental conditions, it automatically triggers an online fine-tuning mechanism for the model parameters. This mechanism uses a small batch of newly collected data to update the gradient of the cross-modal feature interaction network in the feature deep fusion module through a backpropagation algorithm. The learning rate is set to 0.0001, and the time for a single fine-tuning is less than 200 milliseconds.

6. The robot environment recognition system based on multimodal perception according to claim 1, characterized in that, The system adopts a heterogeneous computing platform architecture; sensor data stream processing, deep learning model inference, and real-time motion planning tasks are offloaded to three different computing units: graphics processor, neural network processor, and field-programmable gate array, respectively. The computing units exchange data via a high-speed on-chip interconnect bus, with a peak bandwidth of 32 gigabits per second and an end-to-end processing latency of less than 30 milliseconds.

7. A robot environment recognition system based on multimodal perception according to claim 2, characterized in that, The feature deep fusion module also integrates a perception quality assessment unit. The perception quality assessment unit dynamically adjusts the initial weights of different modes in the fusion architecture by analyzing the signal-to-noise ratio, contrast, and integrity indicators of the original data of each mode. In an environment where the dust concentration instantaneously exceeds 150 milligrams per cubic meter, the initial weight of the visual mode is automatically reduced to 0.2, and the weight of the laser mode is increased to 0.

8. When severe mechanical vibration is detected, causing abnormal inertial data, the weight of the inertial mode is temporarily set to zero, and the system switches to a pure visual-laser fusion mode.

8. A robot environment recognition system based on multimodal perception according to claim 5, characterized in that, The online adaptive learning module's online model parameter fine-tuning mechanism includes a catastrophic forgetting protection strategy. This strategy determines the degree of knowledge conflict by calculating the inner product of the gradient directions of new data and historical data. When the inner product is negative, it indicates that new knowledge may conflict with old knowledge. In this case, the fine-tuning process will introduce elastic weights to consolidate the regularization term to protect important old network parameters from being over-modified.

Images