Real-time processing method and system for robot dog environment perception and decision

Abstract

The invention discloses a real-time processing method and system for robot dog environment perception and decision, and relates to the field of robot dogs. Carrying out multi-source data fusion and semantic segmentation based on deep learning; carrying out semantic enhanced 3D point cloud construction and target detection; real-time path planning and obstacle avoidance decision making in a dynamic environment; and carrying out adaptive motion control and system fault-tolerant processing. According to the invention, through four core technologies of multi-modal fusion, semantic enhancement, real-time calculation optimization and intelligent decision making, the robustness and real-time performance of multi-modal sensor fusion are significantly improved, and the precision of environmental perception is improved; calculation delay is greatly reduced, edge computing power distribution is optimized, and high-real-time decision making is achieved; the intelligent obstacle avoidance and path planning capability in the dynamic environment is enhanced, the intelligent decision-making capability in the dynamic environment is improved, the reliability and practicability of the system are improved, and an efficient, stable and low-cost solution is provided for large-scale application of the system in the fields of industry, security and logistics.

Description

Technical Field

[0001] This invention relates to the field of robot dog technology, specifically to a real-time processing method and system for robot dog environmental perception and decision-making. Background Technology

[0002] With the continuous advancement of robotics technology, robots are increasingly being applied in numerous fields, encompassing intelligent logistics, industrial automation, smart homes, medical assistance, and autonomous driving. In these applications, robots must accurately perceive their surroundings to make correct decisions and take appropriate actions. In particular, quadrupedal robotic dogs (bionic robots) are finding increasingly widespread use in complex scenarios such as inspection, security, rescue, and logistics. These robotic dogs need to frequently interact with people, equipment, and other obstacles in unstructured environments, achieving autonomous navigation and task execution, which places extremely high demands on environmental perception and real-time decision-making capabilities. To ensure safe movement and efficient task execution in complex environments, robots must promptly perceive environmental changes, avoid collisions with obstacles, and reduce the occurrence of accidents. Therefore, they must possess excellent environmental perception and decision-making capabilities, able to make decisions and adjust actions based on perceived information to adapt to diverse work scenarios and task requirements.

[0003] Existing methods and systems for real-time environmental perception and decision-making in robotic dogs (especially quadruped robots) exhibit strong adaptability in complex scenarios, but still suffer from the following problems: 1. Multimodal sensor fusion has shortcomings in terms of real-time performance and robustness; Data synchronization and calibration issues: Robot dogs typically rely on multiple sensors such as LiDAR, vision cameras, and IMUs for data fusion. However, the sampling frequencies of different sensors (e.g., 10Hz for LiDAR and 30Hz for cameras) and their delay differences may lead to spatiotemporal synchronization errors, which in turn affect real-time decision-making.

[0004] Insufficient environmental adaptability: In extreme weather conditions (such as rain, fog, and strong light) or when encountering dynamic obstacles (such as pedestrians and vehicles), a single sensor (e.g., LiDAR is susceptible to reflectivity, and visual cameras are susceptible to lighting interference) is prone to failure. Existing technologies lack adaptive weight adjustment mechanisms and cannot dynamically optimize sensor fusion strategies.

[0005] Insufficient semantic information fusion: Most existing technologies are based on geometric features (such as point cloud clustering) for analysis, but lack a deep understanding of the semantic information of objects (such as distinguishing between "pedestrians" and "static obstacles"), resulting in overly simplistic obstacle avoidance strategies that cannot effectively predict pedestrians' intentions.

[0006] 2. Computation latency and computing power bottleneck; High computing power requirements: Real-time semantic segmentation (such as YOLOv7), 3D point cloud processing (such as PointNet++) and SLAM mapping place extremely high demands on the computing power of embedded platforms (such as Jetson AGX Orin), which may lead to decision delays (>100ms), thereby affecting obstacle avoidance response under high-speed motion.

[0007] Insufficient algorithm optimization: Most existing technologies use fixed parameter filtering (such as EKF) and fail to combine incremental optimization or lightweight models (such as MobileNetV3), resulting in a waste of computing resources.

[0008] 3. Limitations of decision-making logic in dynamic environments; Insufficient trajectory prediction capability: Traditional obstacle avoidance technology relies on static maps or short-term dynamic detection, but lacks long-term trajectory prediction capability (such as Markov models or LSTM), making it difficult to cope with sudden moving obstacles (such as vehicles that suddenly turn).

[0009] Weak multi-task generalization ability: Most existing technologies are optimized for single tasks (such as inspection or obstacle avoidance) and cannot flexibly switch strategies in complex scenarios (such as rescue and logistics), resulting in the phenomenon of "pseudo-intelligence" and the preset actions cannot adapt to new environments.

[0010] 4. System reliability and cost issues; Insufficient sensor redundancy design: Existing technologies rely excessively on LiDAR (such as multi-line radar, which is expensive), and the system will completely collapse in the event of a failure. In contrast, low-cost vision solutions (such as INDEMIND) can reduce costs by 60-80%, but are greatly limited by lighting conditions.

[0011] Energy and battery life limitations: The high power consumption of real-time perception and decision-making processes (such as 275 TOPS computing power) significantly reduces the robot dog's battery life (only 1-2 hours), severely restricting its ability to perform long-term tasks. Summary of the Invention

[0012] To address the core problems of current robot dogs, especially quadruped robots, in environmental perception and real-time decision-making, such as insufficient robustness of multi-sensor fusion, high computational latency, poor adaptability to dynamic environments, and low system reliability, this invention provides a real-time processing method and system for robot dog environmental perception and decision-making.

[0013] The perception-decision closed-loop latency of this invention is <50ms (@Jetson Orin + FPGA), which significantly improves real-time performance; at the same time, the obstacle avoidance success rate is >95% in rain, fog and dynamic obstacle scenarios, demonstrating strong robustness; and the battery life of this invention is improved by 40%, with high energy efficiency, enabling the robot dog to operate autonomously with high reliability in complex environments.

[0014] The technical solution adopted by this invention to solve the technical problem is as follows: On the one hand, the present invention provides a real-time processing method for environmental perception and decision-making in a robot dog, which specifically includes the following steps: Step S1: Synchronous acquisition and preprocessing of multimodal sensor data; Step S2: Multi-source data fusion and semantic segmentation based on deep learning; Step S3: Semantically enhanced 3D point cloud construction and object detection; Step S4: Real-time path planning and obstacle avoidance decision-making in dynamic environments; Step S5: Adaptive motion control and system fault tolerance.

[0015] Furthermore, the specific implementation process of step S1 is as follows: S1.1: Hardware synchronization trigger; Hardware synchronization signals are used to align the sampling times of the LiDAR, binocular RGB-D camera, infrared thermal imager, and IMU sensor to ensure that the timestamp error is <1ms; if hardware synchronization is unavailable, software timestamp interpolation is used to compensate for the delay. S1.2: Data preprocessing; 1) Image processing: Radiometric correction and noise reduction for visible light and infrared images; 2) Point cloud distortion correction: Motion compensation of LiDAR point clouds based on IMU pose data; 3) Temperature-humidity compensation: Corrects the radiation value of infrared thermal imaging based on ambient temperature and humidity data; 4) Output: Time-aligned visible light images, infrared images, 3D point clouds, and IMU pose data.

[0016] Furthermore, the hardware synchronization signal adopts the PTP protocol or an external trigger line, the LiDAR adopts Velodyne VLP-16, the binocular RGB-D camera adopts Intel RealSense D455, the infrared thermal imager adopts FLIR A655SC, and the IMU sensor adopts BMI088.

[0017] Furthermore, the radiation correction employs a histogram equalization method, the denoising uses the BM3D algorithm, and the motion compensation uses linear interpolation in LOAM.

[0018] Furthermore, the specific implementation process of step S2 is as follows: S2.1: Image-level fusion; The input visible light and infrared images are first registered, and then bilinear interpolation is used to achieve image alignment. The visible light image, infrared image, 3D point cloud, and IMU pose data are extracted and fused through a cross-modal feature fusion network, and the fused multispectral image is output. S2.2: Semantic segmentation; The fused multispectral image is input into a trained lightweight semantic segmentation model, which outputs a 2D fused image with pixel-level semantic labels. When training the model, we used the COCO dataset and a self-built infrared dataset for joint training, and adopted DiceLoss + Focal Loss as the loss function.

[0019] Furthermore, the structure of the cross-modal feature fusion network includes a dual-branch encoder and a cross-modal attention module; the lightweight semantic segmentation model adopts BiSeNetV3.

[0020] Furthermore, the specific implementation process of step S3 is as follows: S3.1: Semantic projection; Using the camera-LiDAR extrinsic calibration matrix, the 2D fused image with pixel-level semantic labels is projected pixel by pixel onto the 3D semantic point cloud using perspective projection transformation, and projection outliers are removed. S3.2: Semantic point cloud clustering; Using the improved DBSCAN algorithm: points within the same cluster must satisfy the condition that the geometric distance is <0.2m and the semantic labels are consistent, and the clustering results are output; S3.3: Target Detection and Tracking; PointNet+++Kalman filtering is used to implement multi-target tracking for dynamic clusters, and the position, velocity and semantic category of obstacles are output. The final result is a 3D point cloud map with semantic labels and a list of dynamic obstacle states.

[0021] Furthermore, the specific implementation process of step S4 is as follows: S4.1: Global Path Planning; At the global level, a topological navigation map is constructed based on a semantic point cloud map, and the A* algorithm is used to calculate the initial path to avoid impassable areas. S4.2: Local obstacle avoidance and trajectory prediction; Dynamic obstacle prediction: For targets such as pedestrians / vehicles, input historical trajectories into an LSTM+Markov hybrid model to predict the probability distribution of movement within the next 3 seconds; Risk field modeling: A dynamic potential field is generated based on the prediction results of dynamic obstacles and superimposed on the traditional DWA algorithm for DWA sampling; S4.3: Real-time replanning; If a sudden obstacle is detected, such as a newly added cluster, local replanning is triggered. MPC is used to optimize the robot dog's gait and path, and real-time motion control commands are output.

[0022] Furthermore, the specific implementation process of step S5 is as follows: S5.1: Motion control; The QP-optimized controller, based on the MIT Cheetah open-source framework, converts real-time motion control commands into joint motor torque outputs and achieves dynamic balance by combining IMU feedback. S5.2: Fault Detection and Recovery; Sensor fault diagnosis: Real-time monitoring of the validity of data from each sensor and automatic switching to backup sensors; Computational resource scheduling: ROS2's real-time priority scheduling ensures that critical tasks have priority access to computing power; S5.3: Energy Management; Dynamic voltage and frequency adjustment technology is adopted to adjust the power consumption of FPGA / GPU according to the task load, and output stable robot dog movement behavior and system status logs.

[0023] On the other hand, the present invention provides a real-time processing system for environmental perception and decision-making in a robot dog, comprising: The perception layer consists of multimodal components, namely LiDAR, binocular RGB-D camera, infrared thermal imager and IMU sensor. The LiDAR acquires 3D point cloud, the binocular RGB-D camera acquires visible light images, the infrared thermal imager acquires infrared images, and the IMU sensor acquires IMU pose data. The computing layer employs a heterogeneous computing approach using FPGA+GPU to process sensor data from various sensors in the multimodal components and perform complex calculations. The decision layer is used to implement global / local planning, deploy multi-sensor SLAM engines and dynamic obstacle avoidance controllers to achieve incremental SLAM and dynamic obstacle avoidance. The control layer, through the deployment of control systems, enables motion execution and output commands.

[0024] The beneficial effects of this invention are: This invention provides a real-time processing method and system for environmental perception and decision-making in robotic dogs. Through innovative multimodal data fusion, semantically enhanced decision-making, lightweight computing architecture, and adaptive fault-tolerant mechanisms, it significantly improves the perception accuracy, decision-making speed, and system reliability of robotic dogs in complex environments. Compared with existing technologies, it has the following advantages: 1. Significantly improves the robustness and real-time performance of multimodal sensor fusion, and enhances the accuracy of environmental perception; Multimodal data adaptive fusion: Through dynamic weight adjustment mechanisms (such as optimizing sensor confidence based on ambient light, temperature, and motion state) and cross-modal fusion of visible light + infrared thermal imaging + LiDAR, it solves the problems of data synchronization and calibration of heterogeneous sensors such as visible light cameras, infrared thermal imaging, LiDAR, and millimeter-wave radar, as well as the failure of single sensors in extreme environments (such as strong light, rain and fog, darkness, and dynamic obstacles), ensuring stable perception in extreme environments (such as rain and fog, strong light, and dynamic obstacles), and improving the success rate of environmental perception by more than 30%.

[0025] Semantic-enhanced environment modeling: Combining deep learning techniques (such as Transformer-based multimodal fusion networks) to achieve pixel-level semantic segmentation and accurately map semantic information (such as "pedestrians" and "high-temperature equipment") to 3D point clouds, improving obstacle classification and intent understanding capabilities, enabling the robot dog to accurately distinguish obstacle types (such as pedestrians, vehicles, and glass doors), and reducing the misidentification rate to <2%.

[0026] Dynamic obstacle tracking: Combining LSTM+Markov model for trajectory prediction, the accuracy of trajectory prediction for moving targets reaches 90% (within 3 seconds), which is better than the traditional pure geometric obstacle avoidance method (about 70%).

[0027] 2. Significantly reduce computation latency, optimize edge computing power allocation, and achieve highly real-time decision-making; Lightweight hybrid computing architecture: It adopts a heterogeneous acceleration solution of "FPGA+GPU" (such as Xilinx Zynq UltraScale+ working together with NVIDIA Jetson Orin) to perform block optimization for tasks such as SLAM, object detection, and path planning, and compresses the end-to-end latency of tasks such as SLAM mapping and object detection to <50ms (existing technologies are usually >100ms), significantly reducing end-to-end latency.

[0028] Incremental Semantic SLAM: By optimizing local point clouds and reusing semantic features, the computational overhead of global reconstruction is reduced, increasing the SLAM update frequency to 15Hz (compared to approximately 5-10Hz in existing technologies). This invention effectively avoids the high computational cost required for global reconstruction and significantly improves mapping efficiency in dynamic environments through incremental point cloud registration and semantic feature extraction techniques (such as SemanticLOAM).

[0029] Lightweight model design: Using pruned and quantized BiSeNetV3 and PointNet++, the computational cost of the model is reduced by 60% and the memory usage is reduced by 50%.

[0030] 3. Enhance intelligent obstacle avoidance and path planning capabilities in dynamic environments, and improve intelligent decision-making capabilities in dynamic environments; The risk field enhanced DWA algorithm combines dynamic obstacle prediction results to generate a probabilistic risk field, making obstacle avoidance paths safer and reducing the collision rate by 80% in complex scenarios (such as crowds).

[0031] Multi-scale path planning: Combining global topology navigation with local MPC optimization improves path smoothness by 40%, making it suitable for unstructured environments such as narrow passages and irregular terrain.

[0032] Multi-scale trajectory prediction: Integrating short-term (LSTM / GRU) and long-term (Markov model) trajectory prediction technologies, it accurately predicts the movement trend of dynamic obstacles (such as pedestrians and vehicles) and generates probabilistic obstacle avoidance paths.

[0033] Adaptive Policy Switching: By incorporating interpretable reinforcement learning (XRL) and introducing interpretable modules (such as visualization of attention mechanisms), the training of reinforcement learning policies is aided, enabling the robot dog to dynamically switch policies in complex tasks (such as rescue and logistics). The robot dog can automatically adjust its decision-making strategy according to task requirements (such as inspection and rescue), improving the task success rate by 25%.

[0034] 4. Improve system reliability and usability; Fault self-recovery mechanism: Design a multi-level sensor redundancy scheme (e.g., automatically switch to pure vision SLAM when LiDAR fails) and hardware fault tolerance strategy (e.g., FPGA dynamic reconfiguration) to ensure that the system can continue to operate when some hardware fails.

[0035] Multi-level fault tolerance mechanism: When the LiDAR or camera fails, the system can switch to a backup sensor (such as pure vision or IMU + wheel odometry) within 200ms to ensure continuous operation.

[0036] Low power consumption optimization: By employing event-driven perception technology (such as dynamic region interest detection) and adaptive computing power scheduling mechanisms (such as DPM dynamic power management), the battery life of the robot dog in applications such as inspection and security is effectively extended. In this invention, through event-driven perception and dynamic power management (DVFS), the robot dog's battery life is extended to more than 4 hours (compared to about 1-2 hours in the prior art).

[0037] Modular hardware and software design: Supports flexible replacement of sensors and computing units to meet the cost and performance requirements of different application scenarios (such as industrial inspection and home services), and can reduce costs by 30-50% (such as replacing LiDAR with RGB-D cameras).

[0038] 5. Promote the implementation and application of technologies in multiple scenarios; Industrial inspection: Stable detection of equipment status in high temperature, high humidity or low light factory environments, with a false negative rate of <1%.

[0039] Security patrol: At night, intruders are identified through infrared and visible light fusion, with a detection range of up to 50 meters.

[0040] Logistics and distribution: Autonomous obstacle avoidance in dynamic warehouses improves handling efficiency by 20%.

[0041] Rescue search: Rapid mapping in unstructured terrain such as ruins reduces search and rescue response time by 40%.

[0042] In summary, this invention solves the bottleneck problems of existing robot dogs in terms of perception accuracy, real-time performance, dynamic obstacle avoidance, and system reliability through four core technologies: multimodal fusion, semantic enhancement, real-time computing optimization, and intelligent decision-making. It provides an efficient, stable, and low-cost solution for their large-scale application in industries such as industry, security, and logistics. Attached Figure Description

[0043] Figure 1 The present invention provides a structural block diagram of a real-time processing system for environmental perception and decision-making in a robotic dog.

[0044] Figure 2 The flowchart illustrates a real-time processing method for environmental perception and decision-making in a robotic dog, as provided by this invention.

[0045] Figure 3 This is a flowchart of step S2 of the present invention, which involves multi-source data fusion and semantic segmentation based on deep learning.

[0046] Figure 4 This is a flowchart of the semantically enhanced 3D point cloud construction and object detection in step S3 of the present invention.

[0047] Figure 5 This is a flowchart of real-time path planning and obstacle avoidance decision-making in a dynamic environment in step S4 of the present invention. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the accompanying drawings.

[0049] In a first aspect, the present invention provides a real-time processing system for environmental perception and decision-making in robotic dogs.

[0050] like Figure 1 As shown, the present invention provides a real-time processing system for environmental perception and decision-making in a robotic dog, specifically including the following modules: The perception layer consists of multimodal components, namely a LiDAR, a binocular RGB-D camera, an infrared thermal imager, and an IMU sensor. The LiDAR acquires 3D point clouds, the binocular RGB-D camera acquires visible light images, the infrared thermal imager acquires infrared images, and the IMU sensor acquires IMU pose data.

[0051] The computing layer mainly adopts the FPGA+GPU heterogeneous computing approach to process sensor data from various sensors in the multimodal components and perform complex calculations.

[0052] The decision-making layer is mainly used to implement global / local planning, deploy multi-sensor SLAM engines and dynamic obstacle avoidance controllers to achieve incremental SLAM and dynamic obstacle avoidance.

[0053] The control layer, through the deployment of control systems, enables motion execution and output commands.

[0054] Secondly, the present invention provides a real-time processing method for environmental perception and decision-making in a robot dog, which is implemented using the real-time processing system for environmental perception and decision-making in a robot dog provided in the first aspect.

[0055] The present invention provides a real-time processing method for environmental perception and decision-making in robotic dogs, comprising the following core steps: (1) Synchronous acquisition and preprocessing of multimodal sensor data; (2) Multi-source data fusion and semantic segmentation based on deep learning; (3) Semantically enhanced 3D point cloud construction and target detection; (4) Real-time path planning and obstacle avoidance decision-making in dynamic environments; (5) Adaptive motion control and system fault tolerance.

[0056] The following detailed description of each step ensures the operability and repeatability of the method proposed in this invention.

[0057] like Figure 2 As shown, the real-time processing method for environmental perception and decision-making in a robot dog provided by this invention has the following specific implementation process:

[0058] By synchronously acquiring and preprocessing multimodal sensor data, the spatiotemporal synchronization problem of multimodal sensor data can be solved, ensuring the consistency of input data. The specific implementation process is as follows: S1.1: Hardware synchronization trigger; The hardware synchronization signal emitted by the PSP signal generator is used to align the sampling times of the LiDAR, binocular RGB-D camera, infrared thermal imager, and IMU sensor, ensuring that the timestamp error is <1ms.

[0059] If hardware synchronization is unavailable, software timestamp interpolation is used to compensate for the delay.

[0060] Specifically, the hardware synchronization signal can use the PTP protocol or an external trigger line; the LiDAR can use the Velodyne VLP-16; the binocular RGB-D camera can use the Intel RealSense D455; the infrared thermal imager can use the FLIR A655SC; and the IMU sensor can use the BMI088, but is not limited to these.

[0061] S1.2: Data preprocessing; 1) Image processing: Radiometric correction and denoising are performed on visible light and infrared images. Specifically, radiometric correction can be performed using histogram equalization, and denoising can be performed using the BM3D algorithm.

[0062] 2) Point cloud distortion correction: Motion compensation is performed on the LiDAR point cloud based on IMU pose data. Specifically, the motion compensation can be achieved using the linear interpolation method in LOAM.

[0063] 3) Temperature-humidity compensation: Use a temperature and humidity sensor (such as the SHT31 sensor) to collect ambient temperature and humidity data, and correct the radiation value of the infrared thermal image based on the ambient temperature and humidity data.

[0064] 4) Output: Time-aligned visible light images, infrared images, 3D point clouds, and IMU pose data.

[0065] By employing deep learning-based multi-source data fusion and semantic segmentation steps, multimodal data is integrated and semantic information is extracted to enhance environmental understanding capabilities. For example... Figure 3 As shown, the specific implementation process is as follows: S2.1: Image-level fusion; The input visible light (RGB) and infrared (IR) images are first registered, and then bilinear interpolation is used to achieve image alignment. The cross-modal feature fusion network (CMF-Net) is used to extract and fuse features from the visible light image, infrared image, 3D point cloud, and IMU pose data, and outputs the fused multispectral image.

[0066] The cross-modal feature fusion network consists of a dual-branch Encoder (ResNet-18) and a cross-modal attention module.

[0067] S2.2: Semantic segmentation; The fused multispectral image is input into a trained lightweight semantic segmentation model, which outputs a 2D fused image with pixel-level semantic labels (such as "pedestrian", "vehicle", "passable area").

[0068] The lightweight semantic segmentation model can specifically use BiSeNetV3, but is not limited to it. The training data used is the COCO dataset and a self-built infrared dataset. Joint training is performed using the COCO dataset and the self-built infrared dataset, with the loss function being Dice Loss (logarithmic loss) + Focal Loss (focal loss).

[0069] The self-built infrared dataset used in this invention to train a lightweight semantic segmentation model is constructed around typical application scenarios of robotic dogs (industrial inspection, security patrol, logistics delivery, and rescue search). It aims to compensate for the lack of samples in publicly available infrared datasets in complex, unstructured environments and dynamic obstacle interaction scenarios, thereby improving the model's semantic segmentation accuracy and environmental adaptability to infrared images. Its construction process mainly includes four core stages: data acquisition, data preprocessing, annotation and validation, and dataset partitioning, as detailed below: I. Data Acquisition Phase; (a) Selection of data acquisition equipment; The infrared thermal imager (640×512 resolution, thermal sensitivity <50mK) was used in conjunction with an Intel RealSense D455 binocular RGB-D camera for synchronous acquisition. At the same time, the device pose was recorded by a BMI088 IMU sensor to ensure the spatiotemporal alignment of the infrared image with the visible light image and the pose data (timestamp error <1ms).

[0070] (ii) Data collection scenario design; The dataset covers four core scenarios for practical applications of robotic dogs, with different sub-scenarios and environmental conditions selected for each scenario to ensure the diversity and representativeness of the dataset: Industrial inspection scenarios: Collect data from sub-scenarios such as factory workshops (high-temperature equipment, pipelines, robotic arms), warehouses (shelves, forklifts, conveyor belts), and outdoor industrial parks (power distribution cabinets, iron towers, rain / fog / strong light environments), focusing on collecting infrared features of high-temperature targets, metal reflective surfaces, and industrial equipment.

[0071] Security patrol scenarios: Data collection is conducted in sub-scenarios such as residential areas, commercial buildings, and parking lots, covering situations such as low light at night, dynamic movement of pedestrians / vehicles, and obstruction by fences / obstacles. The focus is on collecting infrared thermal features of people, vehicles, and security facilities.

[0072] Logistics and distribution scenarios: Collect infrared information from sub-scenarios such as warehouse interiors (shelves, express parcels, AGV carts), park roads (dynamic pedestrians, transport vehicles), and narrow passages, with a focus on collecting infrared information from passable areas, logistics equipment, and dynamic obstacles.

[0073] Rescue and search scenarios: Collect sub-scenarios such as simulated ruins, forest / mountainous terrain, and indoor collapse scenarios, covering extreme conditions such as smoke, obstruction, and complex terrain, with a focus on collecting infrared features of human thermal targets and dangerous obstacles.

[0074] (III) Data Collection Parameters and Scale; Acquisition parameters: The infrared image sampling frequency is set to 30fps, and the visible light image is acquired synchronously as a reference. During the acquisition process, auxiliary information such as ambient temperature and humidity (via SHT31 sensor) and device motion status (IMU pose) are recorded.

[0075] Data scale: A total of 100,000 raw infrared images were collected, covering different environmental conditions such as sunny / rainy / foggy, day / night, static / dynamic. Among them, dynamic scene samples accounted for 40%, and extreme environment (rain, fog, strong light, low light) samples accounted for 30%.

[0076] II. Data Preprocessing Stage; The acquired raw infrared images are preprocessed to eliminate noise, correct biases, and improve data quality. The specific steps are as follows: Radiometric correction: Histogram equalization is used to correct the radiometric deviation of infrared images caused by differences in ambient temperature and humidity and equipment response, so as to make the gray-scale distribution of the image more uniform and highlight the thermal characteristics of the target.

[0077] Denoising: The BM3D algorithm is used to denoise the infrared image, eliminating the inherent noise of the thermal imager and environmental electromagnetic interference, while preserving the target edge and detail information.

[0078] Geometric correction: Based on the camera-LiDAR extrinsic calibration matrix, the geometric distortion of the infrared image is corrected to ensure pixel-level alignment with the visible light image; for motion-blurred images, motion compensation is performed by combining IMU pose data.

[0079] Data augmentation: To improve the model's generalization ability, online augmentation is performed on the preprocessed images, including: Geometric transformations: random cropping (preserving the target area), rotation (±15°), scaling (0.8-1.2 times); Pixel enhancement: Randomly adjust the contrast and brightness of the infrared image (±20%) to simulate changes in thermal characteristics under different temperature environments; Scene fusion: Infrared images are combined with different backgrounds (such as industrial equipment and natural terrain) to expand the sample of complex scenes.

[0080] III. Labeling and Verification Stage; (a) Labeling system design; Referring to the semantic annotation specifications of the COCO dataset and considering the environmental perception needs of robot dogs, 18 core semantic labels were developed, divided into three main categories: Dynamic obstacles: pedestrians, vehicles, animals; Static obstacles: building structures, industrial equipment, shelves, fences, trees, ruins; Accessible areas: ground, roads, steps, platforms.

[0081] (II) Annotation tools and procedures; Annotation tools: LabelMe and LabelStudio are used for joint annotation, which supports pixel-level semantic segmentation annotation and target detection box annotation, and records the semantic category, location and temperature features (infrared image) of the target.

[0082] Labeling process: Initial annotation: Three professional annotators independently annotated the preprocessed image, with annotation accuracy required at the pixel level; Cross-validation: The annotation results are cross-checked, and discrepancies in samples (such as blurred targets or occluded targets) are collectively reviewed to unify the annotation standards; Correction and optimization: Use a semi-automatic annotation tool (based on a pre-trained PointNet++ model) to batch annotate similar samples, and then manually correct them to improve annotation efficiency and consistency.

[0083] (iii) Data validation; 5% of the labeled samples were randomly selected and verified by technical experts in the field of robot dog perception to ensure that the labeling accuracy rate was ≥98%; the incorrectly labeled samples were corrected, and finally 80,000 high-quality labeled infrared images were obtained.

[0084] IV. Dataset partitioning stage; The labeled dataset is divided into a training set: validation set: test set ratio of 7:2:1 to meet the needs of model training, tuning, and evaluation. Training set: 56,000 images, covering various scenes and targets, used for the main training of the model; Validation set: 16,000 images, used to adjust model hyperparameters (such as learning rate and loss function weights) and monitor model overfitting; Test set: 8,000 images, containing a large number of unseen new scene samples, used to objectively evaluate the semantic segmentation performance of the model.

[0085] Meanwhile, to ensure the fairness of the dataset, the scenario independence principle is followed when dividing the dataset, meaning that the scenarios of the test set do not overlap with those of the training set, ensuring that the model evaluation results can reflect its generalization ability in real unknown environments.

[0086] The self-built infrared dataset constructed through the above process, after joint training with the COCO dataset, can effectively improve the semantic understanding ability of the BiSeNetV3 lightweight semantic segmentation model for infrared images, reducing the obstacle misidentification rate of the robot dog in extreme environments such as low light, rain and fog to <2%.

[0087] By constructing semantically enhanced 3D point clouds and detecting objects, 2D semantic information is mapped to 3D space to build a semantic point cloud map. For example... Figure 4 As shown, the specific implementation process is as follows: S3.1: Semantic projection; Using a camera-LiDAR extrinsic calibration matrix, the 2D fused image with pixel-level semantic labels is projected pixel by pixel onto a 3D semantic point cloud using perspective projection transformation, and projection anomalies, such as depth discontinuities, are removed.

[0088] S3.2: Semantic point cloud clustering; Using the improved DBSCAN algorithm (combined with semantic constraints): points within the same cluster must satisfy the condition that the geometric distance is <0.2m and the semantic labels are consistent, and the clustering results are output, such as "dynamic pedestrian cluster" and "static obstacle cluster".

[0089] This invention improves the DBSCAN algorithm, and its core improvements are as follows: I. Core Improvement 1: Add semantic label consistency constraints to achieve "semantic + geometric" two-dimensional clustering; Traditional DBSCAN determines whether point clouds belong to the same cluster solely based on spatial distance (ε-neighborhood) and density threshold (MinPts), failing to distinguish between spatially adjacent but semantically different targets (such as adjacent high-temperature pipelines and inspection personnel in an industrial setting). In this invention, the improved DBSCAN algorithm adds semantic label consistency constraints to the clustering determination, with the specific rules as follows: Unified semantics of points within a cluster: When determining whether a point belongs to the current cluster, in addition to satisfying that the spatial distance from the core point within the cluster is <0.2m (geometric constraint, which can be adjusted according to the scenario), the semantic label of the point must also be consistent with the semantic label of the core point within the cluster. Semantic conflict handling: If the spatial location of a point meets the density requirements of the cluster, but its semantic label conflicts with most points in the cluster (e.g., 80% of the labels in the cluster are pedestrian labels, and this point is a device label), it is determined to be an isolated point and is not included in the current cluster to avoid mis-clustering across semantic categories; Semantic priority setting: Semantic priority is set for dynamic semantic categories (such as pedestrians and vehicles) and static semantic categories (such as shelves and ground) (dynamic categories have higher priority than static categories). When the two types of point cloud space overlap, the clustering results of dynamic categories are retained first, which is adapted to the robot dog's key perception needs for dynamic obstacles.

[0090] 2. Core Improvement 2: Dynamically adjust the density threshold (MinPts) to adapt to the point cloud density differences of different semantic categories; Traditional DBSCAN uses a fixed MinPts threshold (e.g., globally set to 5). However, in the 3D point clouds collected by the robot dog, there are natural differences in point cloud density for different semantic categories (e.g., dense ground point clouds and sparse point clouds for tall, thin equipment). A fixed threshold easily leads to over-segmentation of dense categories and missed clustering of sparse categories. In this invention, the improved DBSCAN algorithm dynamically adjusts MinPts based on the point cloud density features of semantic labels: Semantic-Density Mapping Table Construction: By statistically analyzing the point cloud density (number of points per unit volume) of different semantic categories in the training set, a semantic label-recommendation MinPts mapping table is constructed, for example: High-density categories (such as ground and walls): MinPts is set to 10 to avoid cluster splitting caused by excessive point cloud density; Medium-density categories (e.g., pedestrians, vehicles): MinPts is set to 5 to balance cluster integrity and accuracy; Low-density categories (such as thin tubes and fences): MinPts is set to 3 to avoid missed clustering due to sparse point clouds; Real-time dynamic adjustment: During the clustering process, for each new semantic category identified, the corresponding MinPts threshold is automatically retrieved from the mapping table without manual intervention, adapting to the clustering needs of multiple semantic categories in complex scenarios.

[0091] 3. Core Improvement 3: Optimize the ε-neighborhood calculation method and correct the spatial distance weight by combining point cloud semantic features; Traditional DBSCAN uses Euclidean distance to calculate the spatial proximity between point clouds, without considering the influence of the semantic attributes of the point clouds on the effective distance (for example, point clouds of a person's limbs may be spatially distant but semantically related; point clouds with different semantics may be close but semantically related). In this invention, the improved DBSCAN algorithm optimizes the distance calculation for the ε-neighborhood: Semantic weighted distance calculation: Combining Euclidean distance with semantic similarity, the formula is as follows: Weighted distance = Euclidean distance × (1 - semantic similarity coefficient); where the semantic similarity coefficient is set according to the relevance of the two semantic labels (the coefficient is 0.8 for the same category, 0.5 for different sub-items of the same category (such as cars and trucks), and 0 for completely different categories (such as pedestrians and the ground). ε-threshold dynamic scaling: Based on the spatial distribution characteristics of point clouds for different semantic categories, the basic ε-threshold (e.g., 0.2m) is scaled. For example, for dynamic flexible targets (e.g., pedestrians): the ε-scaling factor is set to 1.2 to adapt to the point cloud discreteness caused by human movement. For static rigid targets (such as equipment and shelves): set the ε scaling factor to 0.8 to avoid cluster expansion caused by the discrepancy of the point cloud at the target edge; Anomaly filtering optimization: Combine semantic labels to determine the attributes of isolated points. If the semantic label of an isolated point is a dynamic obstacle (such as a fast-moving small object), it is not directly judged as noise, but its position information is retained for subsequent target tracking (such as Kalman filtering) to reduce the loss of clustering of dynamic targets.

[0092] IV. Core Improvement 4: Introduce temporal semantic relevance to adapt to clustering continuity in dynamic environments; Traditional DBSCAN is a static clustering algorithm that only processes point clouds in a single frame. In the dynamic environment of a moving robot dog, the same target may be classified into different clusters in adjacent frames (e.g., pedestrian movement causes changes in point cloud positions), affecting the stability of multi-target tracking. In this invention, the improved DBSCAN algorithm introduces semantic correlation constraints in the time dimension: Cross-frame semantic cluster matching: The clustering results of the current frame are matched with the clustering results of the previous frame (or the previous N frames). If two clusters in the two frames have the same semantic label and the distance between their centers is <0.5m (a larger distance is allowed for dynamic targets), they are determined to be clusters of the same target and inherit the cluster ID of the previous frame to ensure clustering continuity. Dynamic cluster lifecycle management: Set a lifecycle (e.g., 5 frames) for dynamic semantic clusters (such as pedestrians and vehicles). If a dynamic cluster is not matched for N consecutive frames, it is determined that the target has disappeared. If the number of point clouds of a static cluster (such as a device) decreases suddenly (e.g., due to occlusion), it is marked as temporarily hidden instead of being directly deleted, thus improving the clustering robustness in complex dynamic environments.

[0093] The core advantages of the improved DBSCAN algorithm in this invention are as follows: Compared to the traditional DBSCAN, the improved algorithm incorporating semantic constraints demonstrates significant advantages in robot dog environmental perception scenarios: Improved clustering accuracy: The cross-semantic category misclustering rate is reduced to <3% (compared to about 15% for traditional algorithms), enabling accurate differentiation of spatially adjacent targets; Enhanced environmental adaptability: In unstructured scenarios (such as ruins and factory workshops), cluster integrity is improved by 25%, avoiding missed clustering of sparse targets; Adapted for dynamic tracking: Cross-frame cluster matching accuracy reaches 92%, providing stable input for subsequent multi-target tracking with PointNet+++Kalman filtering, and reducing the dynamic obstacle state recognition latency to <10ms.

[0094] S3.3: Target Detection and Tracking; Multi-object tracking (MOT) is implemented using PointNet+++Kalman filtering on dynamic clusters, outputting the position, velocity, and semantic category of obstacles.

[0095] The final result is a 3D point cloud map with semantic labels and a list of dynamic obstacle states.

[0096] By combining real-time path planning and obstacle avoidance decisions in dynamic environments with semantic information and trajectory prediction, safe movement paths are generated. Figure 5 As shown, the specific implementation process is as follows: S4.1: Global Path Planning; At the global level, a topological navigation map is constructed based on the semantic point cloud map, and the A* algorithm is used to calculate the initial path to avoid "impassable" areas.

[0097] S4.2: Local obstacle avoidance and trajectory prediction; Dynamic obstacle prediction: For targets such as pedestrians / vehicles, input historical trajectories into the LSTM+Markov hybrid model to predict the motion probability distribution within the next 3 seconds, i.e., the Markov probability distribution. Risk field modeling: A dynamic potential field is generated based on the dynamic obstacle prediction results. For example, the risk value around a pedestrian decays exponentially with distance and is superimposed on the traditional DWA algorithm for DWA sampling.

[0098] S4.3: Real-time replanning; If a sudden obstacle is detected, such as a newly added cluster, local replanning is triggered (frequency ≥10Hz). MPC is used to optimize the robot dog's gait and path, and real-time motion control commands (linear velocity, angular velocity, gait parameters) are output.

[0099] Adaptive motion control and system fault tolerance ensure stable system operation under abnormal conditions. The specific implementation process is as follows: S5.1: Motion control; The QP-optimized controller, based on the MIT Cheetah open-source framework, converts real-time motion control commands into joint motor torque outputs and achieves dynamic balance by combining IMU feedback.

[0100] S5.2: Fault Detection and Recovery; Sensor fault diagnosis: Real-time monitoring of the validity of data from each sensor (e.g., determining failure when LiDAR point cloud density drops sharply), and automatic switching to backup sensors (e.g., pure vision SLAM). Computational resource scheduling: Real-time priority scheduling through ROS2 (Robot Operating System 2) ensures that critical tasks (such as obstacle avoidance tasks) have priority access to computing power.

[0101] S5.3: Energy Management; Dynamic Voltage Frequency Scaling (DVFS) technology is used to adjust the FPGA / GPU power consumption according to the task load and output stable robot dog movement behavior and system status logs.

[0102] The present invention will be further described in detail below with reference to the embodiments. System Implementation Example 1

[0103] The real-time processing system for environmental perception and decision-making of a robot dog in this embodiment mainly includes the following core components in its hardware platform: (1) Perception module; Visual unit: Intel RealSense D455 depth camera (RGB resolution 1280×720@30fps, depth resolution 848×480@90fps); Infrared unit: FLIR Boson 640 thermal imaging camera (resolution 640×512, thermal sensitivity <50mK); LiDAR: RoboSense M1 solid-state LiDAR (detection range 150m, horizontal FOV 120°, vertical FOV 25°); Inertial measurement: TDK ICM-42688 six-axis IMU (accelerometer ±16g, gyroscope ±2000dps).

[0104] (2) Calculation module; Main processor: NVIDIA Jetson AGX Orin (64GB RAM, 275 TOPS computing power); Coprocessor: Xilinx Zynq UltraScale+ MPSoC (for point cloud acceleration processing); Communication interface: Dual Gigabit Ethernet + Wi-Fi 6 wireless module.

[0105] (3) Motion control module; Joint motor: Maxon EC90 brushless motor (peak torque 15Nm); Driver: Elmo Gold Twitter Digital Server Driver; Motion control board: A custom control board based on STM32H7.

[0106] The real-time processing method for environmental perception and decision-making in this embodiment is implemented as follows: (1) Data acquisition and synchronization; Multi-sensor synchronization is achieved through hardware triggering, as follows: The lidar and IMU are synchronized using PPS pulse signals; camera exposure is triggered via GPIO (synchronization error <100μs); software timestamps are calibrated using ROS2's clock mechanism.

[0107] (2) Multimodal data fusion; Visible-infrared image registration: SIFT feature points are extracted (threshold set to 0.03), the homography matrix H is calculated using the RANSAC algorithm, and bilinear interpolation is used to achieve image alignment.

[0108] Point cloud-image fusion: Using the calibrated extrinsic matrix T_cam_lidar (accuracy <0.5mm), semantic labels are mapped to 3D point clouds using perspective projection, and voxel filtering (leaf size=0.05m) is applied for downsampling.

[0109] (3) Semantic segmentation; The network training parameters are set as follows: Input size: 640×480; Optimizer: AdamW (lr=3e-4); Loss function: DiceLoss + FocalLoss (γ=2); Training data: Self-built dataset (100,000 labeled images); Deployment optimization: TensorRT acceleration (FP16 accuracy), model pruning rate 30%.

[0110] (4) Real-time path planning implementation; The overall plan is as follows: Construct an octree map (resolution 0.1m) using the ARA* algorithm (heuristic weight = 1.5). Local obstacle avoidance is as follows: Dynamic window parameters: v_max=1.5m / s, ω_max=1.2rad / s; Evaluation function weights: distance=0.6, velocity=0.3, orientation=0.1.

[0111] (5) Implementation of motion control; Gait generation is as follows: The MIT Cheetah algorithm is used, with a control period of 1kHz; Torque control is as follows: Impedance control stiffness coefficient: Kp = 2000 N / m; Damping coefficient: Kd = 100 Ns / m.

[0112] In this embodiment, the configuration of key parameters is shown in Table 1.

[0113] Table 1

[0114] Taking a warehousing and logistics scenario as an example, the system of Example 1 and the method of Example 2 are used to perform real-time processing of the robot dog's environmental perception and decision-making. The specific implementation process is as follows: (1) Initialization; Load the preset map (format: .pgm) and calibrate the sensors (time < 2 minutes).

[0115] (2) Operation process; Real-time acquisition of environmental data (10Hz update), detection of shelves (confidence > 0.9), path planning (A*+DWA), control of mechanical leg movement (step length 0.4m), and triggering of emergency stop when encountering dynamic obstacles (response time < 50ms).

[0116] (3) Performance indicators; Positioning error <0.1m; obstacle avoidance success rate >98%; battery life: 3.5 hours (load capacity 10kg).

[0117] (4) Special scene handling; 1) Handling sensor failures; LiDAR failure: Switch to Visual Inertial Odometry (VIO). Camera malfunction: Enable pure laser SLAM; IMU malfunction: Compensation is achieved using a wheeled odometer.

[0118] 2) Adaptation to extreme environments; Strong light environment: Reduce visible light weight (<0.3); Rain and fog environment: Increase infrared weight (>0.7); Electromagnetic interference: Enable redundant communication (dual-frequency switching).

[0119] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A real-time processing method for environmental perception and decision-making in robotic dogs, characterized in that, Includes the following steps: Step S1: Synchronous acquisition and preprocessing of multimodal sensor data; Step S2: Multi-source data fusion and semantic segmentation based on deep learning; Step S3: Semantically enhanced 3D point cloud construction and object detection; Step S4: Real-time path planning and obstacle avoidance decision-making in dynamic environments; Step S5: Adaptive motion control and system fault tolerance.

2. The real-time processing method for environmental perception and decision-making in a robot dog according to claim 1, characterized in that, The specific implementation process of step S1 is as follows: S1.1: Hardware synchronization trigger; Hardware synchronization signals are used to align the sampling times of LiDAR, binocular RGB-D cameras, infrared thermal imagers, and IMU sensors to ensure timestamp errors are <1ms. If hardware synchronization is unavailable, software timestamp interpolation is used to compensate for the delay. S1.2: Data preprocessing; 1) Image processing: Radiometric correction and noise reduction for visible light and infrared images; 2) Point cloud distortion correction: Motion compensation of LiDAR point clouds based on IMU pose data; 3) Temperature-humidity compensation: Corrects the radiation value of infrared thermal imaging based on ambient temperature and humidity data; 4) Output: Time-aligned visible light images, infrared images, 3D point clouds, and IMU pose data.

3. The real-time processing method for environmental perception and decision-making in a robot dog according to claim 2, characterized in that, The hardware synchronization signal uses the PTP protocol or an external trigger line. The LiDAR uses Velodyne VLP-16, the binocular RGB-D camera uses Intel RealSense D455, the infrared thermal imager uses FLIR A655SC, and the IMU sensor uses BMI088.

4. The real-time processing method for environmental perception and decision-making in a robot dog according to claim 2, characterized in that, The radiation correction uses histogram equalization, the denoising uses the BM3D algorithm, and the motion compensation uses linear interpolation in LOAM.

5. The real-time processing method for environmental perception and decision-making in a robot dog according to claim 1, characterized in that, The specific implementation process of step S2 is as follows: S2.1: Image-level fusion; The input visible light and infrared images are first registered, and then bilinear interpolation is used to achieve image alignment. The visible light image, infrared image, 3D point cloud, and IMU pose data are extracted and fused through a cross-modal feature fusion network, and the fused multispectral image is output. S2.2: Semantic segmentation; The fused multispectral image is input into a trained lightweight semantic segmentation model, which outputs a 2D fused image with pixel-level semantic labels. When training the model, we used the COCO dataset and a self-built infrared dataset for joint training, and adopted DiceLoss + Focal Loss as the loss function.

6. The real-time processing method for environmental perception and decision-making in a robot dog according to claim 5, characterized in that, The structure of the cross-modal feature fusion network includes a dual-branch encoder and a cross-modal attention module; the lightweight semantic segmentation model adopts BiSeNetV3.

7. The real-time processing method for environmental perception and decision-making in a robot dog according to claim 1, characterized in that, The specific implementation process of step S3 is as follows: S3.1: Semantic projection; Using the camera-LiDAR extrinsic calibration matrix, the 2D fused image with pixel-level semantic labels is projected pixel by pixel onto the 3D semantic point cloud using perspective projection transformation, and projection outliers are removed. S3.2: Semantic point cloud clustering; Using the improved DBSCAN algorithm: points within the same cluster must satisfy the condition that the geometric distance is <0.2m and the semantic labels are consistent, and the clustering results are output; S3.3: Target Detection and Tracking; PointNet+++Kalman filtering is used to implement multi-target tracking for dynamic clusters, and the position, velocity and semantic category of obstacles are output. The final result is a 3D point cloud map with semantic labels and a list of dynamic obstacle states.

8. The real-time processing method for environmental perception and decision-making in a robot dog according to claim 1, characterized in that, The specific implementation process of step S4 is as follows: S4.1: Global Path Planning; At the global level, a topological navigation map is constructed based on a semantic point cloud map, and the A* algorithm is used to calculate the initial path to avoid impassable areas. S4.2: Local obstacle avoidance and trajectory prediction; Dynamic obstacle prediction: For targets such as pedestrians / vehicles, input historical trajectories into an LSTM+Markov hybrid model to predict the probability distribution of movement within the next 3 seconds; Risk field modeling: A dynamic potential field is generated based on the prediction results of dynamic obstacles and superimposed on the traditional DWA algorithm for DWA sampling; S4.3: Real-time replanning; If a sudden obstacle is detected, such as a newly added cluster, local replanning is triggered. MPC is used to optimize the robot dog's gait and path, and real-time motion control commands are output.

9. The real-time processing method for environmental perception and decision-making in a robot dog according to claim 1, characterized in that, The specific implementation process of step S5 is as follows: S5.1: Motion control; The QP-optimized controller, based on the MIT Cheetah open-source framework, converts real-time motion control commands into joint motor torque outputs and achieves dynamic balance by combining IMU feedback. S5.2: Fault Detection and Recovery; Sensor fault diagnosis: Real-time monitoring of the validity of data from each sensor and automatic switching to backup sensors; Computational resource scheduling: ROS2's real-time priority scheduling ensures that critical tasks have priority access to computing power; S5.3: Energy Management; Dynamic voltage and frequency adjustment technology is adopted to adjust the power consumption of FPGA / GPU according to the task load, and output stable robot dog movement behavior and system status logs.

10. A real-time processing system for robot dog environmental perception and decision-making, applied to the real-time processing method for robot dog environmental perception and decision-making according to any one of claims 1-9, characterized in that, include: The perception layer consists of multimodal components, namely LiDAR, binocular RGB-D camera, infrared thermal imager and IMU sensor. The LiDAR acquires 3D point cloud, the binocular RGB-D camera acquires visible light images, the infrared thermal imager acquires infrared images, and the IMU sensor acquires IMU pose data. The computing layer employs a heterogeneous computing approach using FPGA+GPU to process sensor data from various sensors in the multimodal components and perform complex calculations. The decision layer is used to implement global / local planning, deploy multi-sensor SLAM engines and dynamic obstacle avoidance controllers to achieve incremental SLAM and dynamic obstacle avoidance. The control layer, through the deployment of control systems, enables motion execution and output commands.

Images