A multi-source perception fusion underground garage unmanned inspection method and system

Abstract

The application discloses a multi-source perception fusion underground garage unmanned inspection method and system, comprising a mobile chassis, a multi-modal sensor group, a fire extinguishing device, a main control unit and a synchronization plate are arranged on the mobile chassis, the main control unit is in communication connection with the mobile chassis, the multi-modal sensor group and the fire extinguishing device respectively, and the multi-modal sensor group comprises a laser radar, multiple RGB cameras and a thermal infrared camera. The application realizes full-process unmanned safety inspection of the underground garage, improves positioning accuracy and inspection reliability, and is suitable for complex operation and maintenance scenes of the underground garage.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection and safety maintenance technology, specifically to a method and system for unmanned inspection of underground parking garages using multi-source sensing fusion. Background Technology

[0002] With the rapid development of the new energy vehicle industry, underground parking garages have become the core scenario for parking and charging. Safe operation and maintenance are directly related to the safety of personnel and property, placing stringent demands on the timeliness, accuracy, and comprehensiveness of inspections. Currently, underground parking garage inspections are still mainly based on traditional manual inspections. Although widely used, these methods have many insurmountable technical shortcomings and application limitations, making them unsuitable for today's intelligent operation and maintenance needs.

[0003] Traditional manual inspections have significant drawbacks: low efficiency, limited by human strength and energy, making 24-hour continuous inspection impossible, and prone to omissions in large parking garages and remote areas; inconsistent inspection quality, with accuracy depending on the staff's sense of responsibility and professional competence, easily leading to missed or misjudged safety hazards such as abnormal charging vehicle temperatures; high maintenance costs and poor safety, requiring a large investment of manpower, and underground parking garages with insufficient lighting and limited visibility, making manual inspections prone to safety accidents.

[0004] While existing unmanned inspection equipment attempts to address the aforementioned issues, significant technical bottlenecks remain: insufficient positioning and navigation accuracy, reliance on a single sensor leading to drift in environments without GPS or with weak textures; unreliable environmental perception, with a single sensor struggling to comprehensively capture complex environmental information; lack of temperature anomaly warning and automatic fire suppression linkage mechanisms, resulting in untimely handling of potential hazards; low synchronization accuracy of multi-source data, affecting inspection effectiveness, and an imperfect automatic recharging mechanism, preventing the realization of fully unmanned operation and maintenance. Technological improvements are urgently needed to fill these industry gaps. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for unmanned inspection of underground parking garages based on multi-source sensing fusion, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for unmanned inspection of underground parking garages using multi-source sensing fusion, characterized by the following steps: S1, the system is powered on, and the data collection time of the LiDAR, multiple RGB cameras and thermal infrared cameras is synchronized through hardware triggering of the synchronization board; S2, acquire synchronized laser point cloud and visual image data, and use a tightly coupled multi-source fusion SLAM algorithm to construct a high-precision 2D map of the underground parking garage; S3: Preset inspection work points in the constructed map, and solve the preset inspection work points sequentially through the global inspection path optimization model to obtain a global reference path covering all inspection work points; and generate a local collision-free obstacle avoidance path based on the real-time updated local dynamic cost map. When the local path conflicts with the global reference path and the original path is impassable, global dynamic replanning is triggered to form a passable inspection route covering all inspection work points. S4 controls the mobile chassis to perform inspections along the planned route, and simultaneously collects temperature data of charging vehicles and charging piles through thermal infrared cameras. S5 analyzes temperature data to identify abnormal temperature locations, and completes map marking and early warning of abnormal locations through semantic target spatial coordinate transformation and map anchoring algorithms; S6, control the mobile chassis and fire extinguishing device to align with the abnormal point, execute the spray fire extinguishing operation, and after the fire is extinguished, perform temperature verification. After confirming that the hidden danger has been eliminated, continue the inspection. S7, during the inspection process, identifies obstacles based on synchronized multi-source data and performs autonomous obstacle avoidance and dynamic path replanning; S8: When the inspection task is completed or the chassis battery level is lower than the preset threshold, control the mobile chassis to automatically return to the charging station.

[0007] Furthermore, in step S1, the synchronization board satisfies the multi-sensor time synchronization accuracy constraint formula through hardware triggering: , In the formula: t LIDAR t is the timestamp for the data collected by the lidar. RGBi Let t be the timestamp for the data collected by the i-th RGB camera. thermal Timestamps for data collected by thermal infrared cameras.

[0008] Furthermore, the map construction process in S2 includes the following steps: S2.1, Voxelization downsampling processing is performed on the synchronized laser point cloud; S2.2, perform feature extraction and cross-view temporal matching on images acquired by multiple RGB cameras to obtain visual observation information; S2.3, the geometric constraints, visual feature constraints and inertial prediction information of laser point cloud are modeled in a unified manner to construct a tightly coupled multi-source fusion state estimation model; S2.4, The error state iterative Kalman filter method is used to recursively optimize the pose of the mobile chassis to obtain the real-time pose of the lidar in the world coordinate system; S2.5 Projects the 3D environmental information onto a 2D grid map based on the optimized pose results, constructs a high-precision 2D map of the underground parking garage, and achieves autonomous positioning.

[0009] Furthermore, in S2.1, the formula for calculating representative points of the laser point cloud after voxelization is as follows: , In the formula, V v Let v be the set of points within the v-th individual element. This represents the number of points within the voxel. p represents the representative point corresponding to this voxel. j This indicates that the vth v is located in the vth v. v The coordinates of the j-th point cloud within; In S2.4, the state update formula for the error state iterative Kalman filter method is: , , In the formula, x (m) x (m+1) The system state estimates, δx, are for the m-th and (m+1)-th iterations, respectively. (m) H represents the error state increment. (m) The observation Jacobian matrix is ​​given by the superscript T, which denotes transpose, R is the observation noise covariance matrix, P is the prior covariance matrix, and r is the number of covariance matrices. (m) To observe the residuals.

[0010] Furthermore, in S3, the multi-objective optimization objective function of the global inspection path is: , , In the formula: For the optimal global inspection path, arg min() represents the arg min function, p k p k+1 These are the k-th and (k+1)-th path points in the sequence, representing the location nodes on the global inspection path of the mobile chassis. For path point p k In a global static 2D raster map The corresponding obstacle cost function, For a global static 2D raster map, λ smoothness This is the path smoothness weighting coefficient. p is the Euclidean distance between adjacent path points. robot (t) represents the current position of the moving chassis at time t, corresponding to the starting point of the global inspection path; p goal The target inspection endpoint location; For global static 2D raster map A defined set of walkable free space; st represents the constraint condition, and N is the index of the last path point.

[0011] Furthermore, in step S3, an obstacle avoidance path is generated using a locally dynamic cost map that is updated in real time. The formulas for generating the locally dynamic cost map and the obstacle avoidance path in a dynamic environment are as follows: , , In the formula: Let Ψ() be the local dynamic cost map at time t, and Ψ() be the fusion mapping function between the global map and the real-time laser point cloud. LIDAR (t) represents the dynamic obstacle occupancy information collected by the lidar at time t; P local (t) represents the collision-free local obstacle avoidance path at time t, Planner local For time-based local path planners, This is the optimal global reference path.

[0012] Furthermore, in the semantic target spatial coordinate transformation and map anchoring algorithm of S5, the two-level pose transformation and map anchoring formula for the semantic target is as follows: , , , In the formula: x represents the 3D coordinates of the semantically detected target in the camera coordinate system. plate Let x be the position coordinates of the target along the x-direction in the world coordinate system, and y be the position coordinates of the target along the x-direction. plate Let be the position coordinates of the target along the y-direction in the world coordinate system. Let be the rotation matrix, representing the attitude relationship between the camera coordinate system and the lidar coordinate system. The translation vector represents the offset of the camera coordinate system origin relative to the lidar coordinate system origin. The target's 3D coordinates in the lidar coordinate system. , To determine the real-time pose of the laser radar in the world coordinate system. and Let represent the rotation matrix and translation vector from the lidar coordinate system to the world coordinate system, respectively. u represents the absolute 3D coordinates of the target in the world coordinate system. map Let x be the anchor index of the target in the global raster map, and Π() be the projection transformation function from the world coordinate system to the raster map. min ,y min ) represents the coordinates of the map origin, Δ x Δ y represents the raster resolution, and represents the actual physical size of each raster in the x and y directions, respectively.

[0013] Furthermore, the field of view of multiple RGB cameras is stitched together to form a 360° panoramic visual perception range.

[0014] Furthermore, in S7, when the original planned path is impassable, global dynamic path replanning is immediately triggered to regenerate a passable inspection route. In S8, the absolute spatial coordinates and relative pose of the semantic target are calculated through a two-level pose transformation formula, which is used for pose fine-tuning control for abnormal point alignment and automatic recharging.

[0015] This invention also provides a multi-source sensing fusion-based unmanned inspection system for underground parking garages, used to realize the unmanned inspection method for underground parking garages as described above. The system includes a mobile chassis, on which a multimodal sensor group, a fire extinguishing device, a main control unit, and a synchronization board are mounted. The main control unit is communicatively connected to the mobile chassis, the multimodal sensor group, and the fire extinguishing device. The multimodal sensor group includes a lidar, multiple RGB cameras, and a thermal infrared camera. The synchronization board is electrically connected to the lidar, RGB cameras, and thermal infrared cameras. The synchronization board achieves time synchronization of data collected by multiple sensors through hardware triggering. The synchronization board is communicatively connected to the main control unit.

[0016] Compared with the prior art, the beneficial effects of the present invention are:

[0017] Employing multimodal sensor fusion technology and combining it with a synchronization board for hardware-level time synchronization, the timestamp error is controlled within 10μs through a clear accuracy constraint formula. Combined with a tightly coupled multi-source fusion SLAM algorithm, this effectively solves the positioning drift problem in underground parking garages without GPS and in environments with weak textures, keeping the positioning error within 15cm and improving the reliability of environmental perception and map building. Map building adopts a dual initialization method of autonomous and manual guidance, coupled with an adaptive update mechanism, which can adapt to the dynamic changes in the underground parking garage environment, ensuring the accuracy and timeliness of mapping. An innovative two-level coupled path planning architecture of "global multi-objective optimization path planning + local dynamic grid obstacle avoidance" is designed. Through a constrained multi-objective optimization objective function, it achieves the shortest and most complete inspection path planning for the entire underground parking garage area, solving the pain points of redundancy and inspection omissions in traditional point-to-point planning. Simultaneously, through a real-time updated local dynamic cost map, it adapts to scenarios with many dynamic obstacles and narrow passages in underground parking garages, significantly improving inspection efficiency and traffic safety. A semantic target end-to-end processing architecture of "pixel-level detection - 3D spatial calculation - global map anchoring" is constructed. Through two-level coordinate system transformation formulas, relative pose calculation formulas, and map anchoring formulas, accurate spatial positioning of semantic targets and persistent global map marking are achieved. This solves the industry pain point of traditional visual detection being "able to recognize but not accurate in positioning," providing a high-precision pose reference for fire extinguishing aiming and automatic recharging, and significantly improving the reliability of anomaly handling and automatic recharging. The overall system realizes unmanned operation of the entire process of underground parking garages from mapping, planning, inspection, anomaly handling to recharging. It is not a simple patchwork of existing functions, but rather solves multiple technical bottlenecks in specific scenarios of underground parking garages through targeted algorithmic innovation, effectively improving inspection efficiency, reducing operation and maintenance costs, and avoiding the safety risks of manual inspection. It is adapted to the actual safety operation and maintenance needs of underground parking garages for new energy vehicle parking and charging. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the overall structure of the multi-source sensing fusion unmanned inspection system for underground parking garages provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the layout structure of the multimodal sensor group provided in an embodiment of the present invention.

[0020] Figure 3 This is a partial structural schematic diagram of the fire extinguishing device provided in an embodiment of the present invention.

[0021] Figure 4 This is a partial structural diagram of the control body provided in an embodiment of the present invention.

[0022] Figure 5 This is a partial structural diagram of a mobile chassis provided in an embodiment of the present invention.

[0023] Figure 6 This is a flowchart illustrating the multi-source sensing fusion-based unmanned inspection method for underground parking garages provided in an embodiment of the present invention.

[0024] The reference numerals in the figure are as follows:

[0025] 10-Multimodal sensor; 11-Thermal infrared camera; 12-Rear RGB camera; 13-Left RGB camera; 14-LiDAR; 15-Right RGB camera; 16-Front-end RGB camera; 20-Fire extinguishing device; 21-Fire extinguisher; 22-Fire extinguisher control device; 30-Control unit; 31-Network switch; 32-Voltage regulator board; 33-Synchronization board; 34-Main control unit; 40-Mobile chassis; 41-Right charging electrode; 42-Left charging electrode. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Please see Figures 1-5 An unmanned inspection system for underground parking garages based on multi-source sensing fusion includes a mobile chassis 40, a multimodal sensor group 10, a control unit 30, and a fire extinguishing device 20. The multimodal sensor group 10, the control unit 30, and the fire extinguishing device 20 are all fixedly mounted on the mobile chassis 40. The bottom of the mobile chassis 40 integrates a right-side charging electrode 41 and a left-side charging electrode 42, which together form a charging electrode structure. The main control unit 34 in the control unit 30 is connected to the mobile chassis 40, the multimodal sensor group 10, and the fire extinguishing device 20 via a CAN bus or an industrial Ethernet communication system to achieve coordinated control of the entire system.

[0028] Continue to refer to Figure 5The mobile chassis 40 adopts an Ackermann front-wheel steering and rear-wheel drive structure, with a rated load of no less than 50kg, a maximum travel speed of 1.5m / s, and a minimum turning radius of 1.3m, adapting to the travel and turning requirements of flat ground and narrow passages in underground parking garages. The chassis has a built-in 48V / 20Ah battery pack, with a full-charge range of no less than 3 hours, which can meet the range requirements of full-area inspection in large underground parking garages. The mobile chassis 40 has a built-in motor drive controller, which communicates with the main control unit 34 to receive travel, steering, braking, and speed adjustment commands issued by the main control unit 34, and to complete actions such as path following, in-situ fine-tuning of steering, and precise alignment. The right charging electrode 41 and the left charging electrode 42 both use elastic copper electrode plates, which correspond and match with the electrode contacts of the charging compartment fixedly installed in the underground parking garage for power connection during automatic recharging. As a preferred embodiment, the mobile chassis 40 can be a Yuhesen MK_mini front-wheel steering chassis. The robot's shell is equipped with easy-to-use buttons, including a power-on / power-off button and an emergency stop button. The buttons are electrically connected to the main control unit 34 via I / O ports, allowing operators to quickly control the equipment.

[0029] Reference Figure 2 The multimodal sensor group 10 is fixedly mounted on the top center of the mobile chassis 40 via an aluminum alloy bracket. It includes a LiDAR 14, a front-end RGB camera 16, a rear-end RGB camera 12, a left-end RGB camera 13, a right-end RGB camera 15, and a thermal infrared camera 11, used to comprehensively collect environmental perception data from the underground parking garage. The LiDAR 14 is mounted at the very top of the sensor bracket, with a mounting height of no less than 75cm from the upper surface of the mobile chassis 40 to avoid obstruction of the scanning field of view by the vehicle structure. Its core parameters are: detection range 0.1m-40m, horizontal field of view 360°, vertical field of view ±57.5°, point cloud output rate no less than 400,000 points / second, ranging accuracy ±2cm, support for external hardware triggering synchronization, and the ability to output dense 3D environmental point cloud data, providing high-precision geometric contour information for mapping, positioning, and obstacle avoidance. It also has a built-in six-axis IMU.

[0030] The front-end RGB camera 16, rear-end RGB camera 12, left-end RGB camera 13, and right-end RGB camera 15 are all global shutter industrial cameras, evenly distributed around the bottom of the LiDAR 14. The installation angle between two adjacent RGB cameras is 90°, facing the front, rear, left, and right directions of the mobile chassis 40, respectively. The core parameters of each camera are: resolution 1920×1080, frame rate 30fps, horizontal field of view not less than 120°, and support for external hardware triggering synchronization. The field of view of the four cameras is stitched together to form a 360° panoramic visual perception range without blind spots, used to collect visual image data of the underground parking garage. It works with the LiDAR 14 to complete multi-source fusion mapping, localization, obstacle recognition and classification, and at the same time provides visual positioning support for automatic recharging. The thermal infrared camera 11 is fixedly mounted on the front bracket of the mobile chassis 40, facing the same direction as the travel of the mobile chassis 40. Its core parameters are: temperature measurement range -20℃ to 150℃, temperature measurement accuracy ±2℃, resolution not less than 384×288, frame rate 25fps, and support for external hardware triggering synchronization. It is used to collect surface temperature data of charging vehicles and charging piles in underground parking garages in real time to identify potential temperature anomalies. As a preferred embodiment, the thermal infrared camera 11 can be a TM16P series network-based industrial-grade thermal infrared camera.

[0031] Reference Figure 4 The control unit 30 integrates a network switch 31, a voltage regulator board 32, a synchronization board 33, and a main control unit 34. All components are integrated and installed in a waterproof electrical box, fixedly mounted in the middle of the mobile chassis 40. The voltage regulator board 32 is electrically connected to the lithium battery pack of the mobile chassis 40, providing a stable DC power supply to the entire control unit 30, the multi-modal sensor group 10, and the fire extinguishing device 20, with an output voltage adapted to the operating requirements of each hardware component. The network switch 31 is an industrial-grade 10 Gigabit switch, enabling communication between the main control unit 34 and the various sensors and the mobile chassis 40. High-speed data interaction between drive controllers; the synchronization board 33 adopts a multi-sensor hardware synchronization trigger board based on the Raspberry Pi core chip, and is configured with 7 hardware trigger channels, of which 4 channels are connected to the front-end RGB camera 16, the back-end RGB camera 12, the left-end RGB camera 13, and the right-end RGB camera 15 respectively, 2 channels are used to connect to the LiDAR 14, and 1 channel is a reserved interface for the GNSS module; the synchronization board 33 is electrically connected to each component of the multi-modal sensor group 10 through hardware trigger lines, and is also connected to the main control unit 34 through a serial port.

[0032] The synchronization board 33 adopts a hardware-triggered synchronization method, and outputs a trigger signal to the lidar 14, each RGB camera and thermal infrared camera 11 at a frequency of 10Hz, controlling all sensors to start data acquisition at the same time. The synchronization board 33 can be an Infinite Sensing series multi-sensor synchronization board.

[0033] The main control unit 34 adopts an x86 architecture embedded industrial control host, equipped with a quad-core or higher processor, at least 8GB of memory, a 256GB solid-state drive, and pre-installed with the Ubuntu 20.04 operating system and the ROSNoetic robot operating system. It possesses multi-threaded data processing and algorithm computation capabilities, meeting the real-time requirements of multi-source data fusion, SLAM mapping, path planning, and anomaly recognition. The main control unit 34 integrates a multi-source data synchronization module, a tightly coupled SLAM mapping and localization module, a global and local path planning module, a thermal infrared temperature anomaly recognition module, a fire extinguishing control module, an autonomous obstacle avoidance module, an automatic recharging control module, and a semantic target spatial coordinate transformation and map anchoring module.

[0034] The multi-source data synchronization module is used to synchronize the hardware trigger time of each sensor through the synchronization board 33, and to complete the time alignment and cache management of multi-source data. The tightly coupled SLAM mapping and localization module adopts a tightly coupled multi-source fusion SLAM algorithm, combining the multi-view observation advantages of four RGB cameras and the 3D point cloud geometric information of LiDAR 14. It adopts a cross-view temporal migration strategy to ensure photometric consistency, and performs fusion optimization through voxelization downsampling processing and error state iterative Kalman filtering. Based on the 3D point cloud dimensionality reduction, it constructs a high-precision 2D grid map of the underground garage, and at the same time realizes the real-time autonomous localization of the mobile chassis 40. The localization error is controlled within 15cm in static environment.

[0035] The global and local path planning modules are implemented based on the ROS navigation framework, with built-in global inspection path planning and dynamic obstacle avoidance algorithms. They adopt a two-level coupled architecture of "global multi-objective optimization path planning + local dynamic grid obstacle avoidance". The global path planning uses a multi-objective optimization objective function to generate the optimal inspection route covering the entire area based on preset inspection work points. The local path planning uses a time-bound local path planning algorithm adapted to the Ackerman steering model, combined with the real-time generated local dynamic cost map and obstacle avoidance path formula, and combined with the fusion perception data of LiDAR 14 and RGB camera to achieve dynamic obstacle avoidance and real-time path optimization.

[0036] The thermal infrared temperature anomaly identification module has a built-in preset basic temperature threshold for charging equipment, and also has built-in ambient temperature adaptive calibration logic. It dynamically calibrates the temperature threshold according to the real-time ambient temperature of the underground garage to avoid false alarms caused by ambient temperature fluctuations. When the temperature of the point collected by the thermal infrared camera 11 exceeds the calibrated threshold, it is determined to be a temperature anomaly. At the same time, through the semantic target spatial coordinate transformation and map anchoring module, the global coordinate marking of the abnormal point in the 2D map is completed, and an audible and visual warning signal is triggered.

[0037] The fire extinguishing control module is communicatively connected to the fire extinguisher control device 22 of the fire extinguishing device 20. It is used to output control commands to control the fire extinguisher control device 22 to perform pressing and resetting actions. At the same time, combined with the relative posture calculation results of the abnormal temperature points, it controls the mobile chassis 40 to complete precise steering and positioning, so as to achieve precise control of fire extinguishing spray. The main control unit 34 has a reserved fire extinguishing device control interface to support control commands such as starting spray and stopping spray.

[0038] The semantic target spatial coordinate transformation and map anchoring module has built-in two-level pose transformation formulas, relative pose calculation formulas, and map anchoring index calculation formulas. It is used to complete pixel-level detection, three-dimensional spatial calculation, and global map anchoring of semantic targets such as license plates, charging piles, abnormal temperature points, and charging compartment positioning marks, providing a high-precision pose reference for fire extinguishing alignment and automatic recharging.

[0039] The automatic recharge control module has a built-in low battery threshold. When the remaining battery power is detected to be below the threshold or the inspection task is completed, the automatic recharge process is started. Combining global path planning and visual positioning mark recognition of the backend RGB camera 12, the chassis posture is adjusted in real time through the relative pose calculation formula to control the mobile chassis 40 to complete the charging docking. At the same time, a retry mechanism for docking failure is built-in.

[0040] Reference Figure 3 The fire extinguishing device 20 is fixedly installed at the front end of the mobile chassis 40, below the thermal infrared camera 11. It includes a fire extinguisher 21 and a fire extinguisher control device 22. The fire extinguisher control device 22 is a clamping mechanism driven by a high-torque digital servo motor. The fire extinguisher 21 is a 3kg portable ABC dry powder fire extinguisher, suitable for extinguishing electrical fires in new energy vehicles. The fire extinguisher control device 22 includes a digital servo motor, a metal swing arm, and a pressing clamp. The servo motor adopts a metal gear design, with an operating voltage of 6-8.4V and a torque of not less than 30kg·cm. One end of the swing arm is fixedly connected to the output shaft of the servo motor, and the other end is fixed to the pressing clamp, which is directly opposite the pressing nozzle of the fire extinguisher 21. The nozzle of the fire extinguisher 21 is sealed with a spray guide tube. The other end of the guide tube extends horizontally forward along the front end of the mobile chassis 40, and the outlet of the guide tube is in the same direction of travel as the mobile chassis 40.

[0041] During firefighting operations, the main control unit 34 sends control commands to the servo motor of the fire extinguisher control device 22. The servo motor rotates, causing the swing arm to swing downwards. By pressing the clamp plate, the nozzle of the fire extinguisher 21 is pressed, realizing the automatic spraying of the extinguishing agent. After the spraying is completed, the servo motor rotates in the opposite direction to reset, releasing the nozzle and stopping the spraying. At the same time, the spraying dosage and spraying duration can be controlled as needed by controlling the rotation angle and holding time of the servo motor.

[0042] Please see Figure 6 A method for unmanned inspection of underground parking garages using multi-source sensing fusion includes the following steps:

[0043] S1, the system is powered on and started. Through the synchronization board 33, the data time synchronization of the lidar 14, four RGB cameras and thermal infrared camera 11 is achieved by hardware triggering. This ensures that the timestamps of the collected data are consistent, providing a basis for multi-source data fusion. The field of view of multiple RGB cameras is stitched together to form a 360° panoramic visual perception range.

[0044] The synchronization board satisfies the multi-sensor time synchronization accuracy constraint formula through hardware triggering:

[0045]

[0046] In the formula: t LIDAR This refers to the timestamp of the data collected by the lidar via hardware triggering through the synchronization board; t RGBi The timestamp of the data acquired by the i-th RGB industrial camera (i=1~4 corresponding to the front, rear, left, and right cameras) uniformly arranged at a 90° circumference below the lidar, triggered by the synchronization board hardware; t thermal This provides a timestamp for the data acquired by the front-mounted thermal infrared camera via hardware triggering through a synchronization board. It addresses issues such as mapping distortion, positioning drift, and insufficient fusion accuracy caused by time asynchrony in multi-source data, providing a reliable time reference for subsequent multi-source data fusion. As a preferred implementation, it ensures time consistency in multi-source data fusion. Hardware triggering ensures that the timestamp error of all sensor data acquisitions is less than 10 μs.

[0047] S2: Acquire synchronized laser point cloud and visual image data, and construct a 2D map of the underground parking garage using a tightly coupled multi-source fusion SLAM algorithm, providing a foundation for subsequent positioning and navigation. The map construction process includes the following steps:

[0048] S2.1, Voxelization downsampling is performed on the synchronized laser point cloud to reduce point cloud redundancy and preserve environmental geometric features. The formula for calculating representative points of the laser point cloud after voxelization is:

[0049]

[0050] In the formula, V v Let v be the set of points within the v-th individual element. This represents the number of points within the voxel. p represents the representative point corresponding to this voxel. j This indicates that the vth v is located in the vth v. v The coordinates of the j-th point cloud within the range can be represented as p. j = (x j ,y j ,z j A voxelization process, i.e., a three-dimensional coordinate point, reduces the amount of point cloud computing while preserving environmental contour features.

[0051] S2.2 performs feature extraction and cross-view temporal matching on images acquired by multiple RGB cameras to obtain visual observation information.

[0052] S2.3 unifies the modeling of laser point cloud geometric constraints, visual feature constraints, and inertial prediction information to construct a tightly coupled multi-source fusion state estimation model.

[0053] S2.4 The error state iterative Kalman filter method is used to recursively optimize the pose of the mobile chassis to obtain the real-time pose of the lidar in the world coordinate system.

[0054] The state update formula for the error state iterative Kalman filter method is:

[0055]

[0056]

[0057] In the formula, x (m) x (m+1) The system state estimates, δx, are for the m-th and (m+1)-th iterations, respectively. (m) H represents the error state increment. (m) The observation Jacobian matrix is ​​given by the superscript T, which denotes transpose, R is the observation noise covariance matrix, P is the prior covariance matrix, and r is the number of covariance matrices. (m) To observe the residuals, an iterative Kalman filter based on the error state is used to achieve tight coupling and fusion optimization of laser point cloud, visual features, and inertial information.

[0058] S2.5 Projects the 3D environmental information onto a 2D grid map based on the optimized pose results, constructs a high-precision 2D map of the underground parking garage, and achieves autonomous positioning.

[0059] S3 involves pre-setting inspection work points in the constructed map and sequentially solving for these points using a global inspection path optimization model to obtain a global reference path covering all inspection work points. Based on a real-time updated local dynamic cost map, a local collision-free obstacle avoidance path is generated. When a local path conflicts with the global reference path and the original path becomes impassable, a global dynamic replanning is triggered to form a passable inspection route covering all inspection work points. The global inspection path optimization model is the A* global path optimization algorithm, a well-known technique.

[0060] In the two-level coupled path planning algorithm, the multi-objective optimization objective function of the global inspection path is:

[0061]

[0062]

[0063] In the formula: For the optimal global inspection path, arg min() represents the arg min function, p k p k+1 These are the k-th and (k+1)-th path points in the sequence, representing the location nodes on the global inspection path of the mobile chassis. For path point p k In a global static 2D raster map The corresponding obstacle cost function, λ smoothness This is the path smoothness weighting coefficient. p is the Euclidean distance between adjacent path points. robot (t) represents the current position of the moving chassis at time t, corresponding to the starting point of the global inspection path; p goal The target inspection endpoint location; For global static 2D raster map A defined set of traversable free space; st represents the constraint condition, and N is the index of the last path point. By solving the above multi-objective optimization objective function, a globally optimal inspection path covering the inspection work points is generated under the conditions of satisfying the start point, end point, and traversability constraints.

[0064] In the two-stage coupled path planning algorithm, obstacle avoidance paths are generated through a real-time updated local dynamic cost map. The formulas for generating the local dynamic cost map and the obstacle avoidance path in a dynamic environment are as follows:

[0065]

[0066]

[0067] In the formula: Let Ψ() be the local dynamic cost map at time t, and Ψ() be the fusion mapping function between the global map and the real-time laser point cloud. LIDAR (t) represents the dynamic obstacle occupancy information collected by the lidar at time t; P local (t) represents the collision-free local obstacle avoidance path at time t, Planner local For time-based local path planners, This is the optimal global reference path.

[0068] S4 controls the mobile chassis 40 to perform inspections along the planned path, and simultaneously collects temperature data of the charging vehicle and charging pile through the thermal infrared camera 11, which is then transmitted to the main control unit 34 for analysis in real time.

[0069] S5, the main control unit 34 analyzes thermal infrared temperature data, combines the preset equipment basic temperature threshold and the ambient temperature adaptive calibration mechanism to identify abnormal temperature points, and completes the map marking and early warning of abnormal points through semantic target spatial coordinate transformation and map anchoring algorithm, and simultaneously records the coordinates of abnormal points in the 2D map, effectively avoiding false alarms and missed alarms.

[0070] In the semantic target spatial coordinate transformation and map anchoring algorithm, the two-level pose transformation and map anchoring formula for the semantic target is as follows:

[0071]

[0072]

[0073]

[0074] In the formula: x represents the 3D coordinates of the semantically detected target in the camera coordinate system. plate Let x be the position coordinates of the target along the x-direction in the world coordinate system, and y be the position coordinates of the target along the x-direction. plate Let be the position coordinates of the target along the y-direction in the world coordinate system. Let be the rotation matrix, representing the attitude relationship between the camera coordinate system and the lidar coordinate system. The translation vector represents the offset of the camera coordinate system origin relative to the lidar coordinate system origin. The target's 3D coordinates in the lidar coordinate system. and Let represent the rotation matrix and translation vector from the lidar coordinate system to the world coordinate system, respectively, used to describe the real-time pose of the lidar in the world coordinate system. u represents the absolute 3D coordinates of the target in the world coordinate system. map Let x be the anchor index of the target in the global raster map, and Π() be the projection transformation function from the world coordinate system to the raster map. min ,y min ) represents the coordinates of the map origin, Δ x Δ y represents the raster resolution, and represents the actual physical size of each raster in the x and y directions, respectively.

[0075] S6 controls the mobile chassis and fire extinguishing device to align with the abnormal location and execute the spraying fire extinguishing operation. After the fire is extinguished, the temperature is checked, and the inspection continues after confirming that the hazard has been eliminated. Specifically, this includes: driving the fire extinguishing device 20, based on the coordinates of the abnormal location marked on the 2D map and the real-time location information fed back by the thermal infrared camera 11, and automatically turning through the mobile chassis 40 to align with the abnormal temperature location, and then controlling the fire extinguisher control device 22 to press the nozzle of the fire extinguisher 21 to spray fire extinguishing; after the fire is extinguished, the temperature of the location is immediately checked through the thermal infrared camera 11, and the inspection task continues after confirming that the hazard has been eliminated, forming a complete safety closed loop of "perception-early warning-response-check".

[0076] S7, during the inspection process, identifies obstacles based on synchronized multi-source data and performs autonomous obstacle avoidance and dynamic path replanning. Specifically, during the inspection, fixed and temporary obstacles in the underground parking garage are captured in real time through the fusion data of LiDAR 14 and RGB camera to achieve autonomous obstacle avoidance; if an obstacle makes the original planned path impassable, the dynamic path replanning logic in S3 is immediately triggered to quickly plan a detour route, ensuring that no inspection work points are missed and the inspection process is not interrupted.

[0077] S8, when the inspection task is completed or the chassis battery level is lower than a preset threshold, controls the mobile chassis to automatically return to charging. Specifically, when the inspection is completed or the mobile chassis 40 detects that the remaining battery level has reached the preset low battery threshold, the automatic recharging process is initiated: first, the mobile chassis 40 navigates to the preset location of the charging compartment through global path planning, then identifies the visual positioning mark (such as a QR code or feature code) at the charging compartment through an RGB camera, and adjusts the position of the mobile chassis 40 in real time to ensure that the right charging electrode 41 and the left charging electrode 42 are aligned with the charging compartment. Then, the mobile chassis 40 connects to the charging compartment through the charging electrode to complete the automatic recharging. If the first connection fails, a retry mechanism of a preset number of times is initiated. If it still fails, an alarm signal is triggered, and manual intervention is required to ensure the reliability of the fully unmanned operation and maintenance process.

[0078] In S8 above, the absolute spatial coordinates and relative pose of the semantic target are calculated through a two-level pose transformation formula, which is used for pose fine-tuning control for abnormal point alignment and automatic recharging.

[0079] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for unmanned inspection of underground parking garages using multi-source sensing fusion, characterized in that, Includes the following steps: S1, the system is powered on, and the data collection time of the LiDAR, multiple RGB cameras and thermal infrared cameras is synchronized through hardware triggering of the synchronization board; S2, acquire synchronized laser point cloud and visual image data, and use a tightly coupled multi-source fusion SLAM algorithm to construct a high-precision 2D map of the underground parking garage; S3: Preset inspection work points in the constructed map, and solve the preset inspection work points sequentially through the global inspection path optimization model to obtain a global reference path covering all inspection work points; and generate a local collision-free obstacle avoidance path based on the real-time updated local dynamic cost map. When the local path conflicts with the global reference path and the original path is impassable, global dynamic replanning is triggered to form a passable inspection route covering all inspection work points. S4 controls the mobile chassis to perform inspections along the planned route, and simultaneously collects temperature data of charging vehicles and charging piles through thermal infrared cameras. S5 analyzes temperature data to identify abnormal temperature locations, and completes map marking and early warning of abnormal locations through semantic target spatial coordinate transformation and map anchoring algorithms; S6, control the mobile chassis and fire extinguishing device to align with the abnormal point, execute the spray fire extinguishing operation, and after the fire is extinguished, perform temperature verification. After confirming that the hidden danger has been eliminated, continue the inspection. S7, during the inspection process, identifies obstacles based on synchronized multi-source data and performs autonomous obstacle avoidance and dynamic path replanning; S8: When the inspection task is completed or the chassis battery level is lower than the preset threshold, control the mobile chassis to automatically return to the charging station.

2. The method for unmanned inspection of underground parking garages based on multi-source sensing fusion according to claim 1, characterized in that, In step S1, the synchronization board satisfies the multi-sensor time synchronization accuracy constraint formula through hardware triggering: ， In the formula: t LIDAR t is the timestamp for the data collected by the lidar. RGBi Let t be the timestamp for the data collected by the i-th RGB camera. thermal Timestamps for data collected by thermal infrared cameras.

3. The method for unmanned inspection of underground parking garages based on multi-source sensing fusion according to claim 1, characterized in that, The map construction process in S2 includes the following steps: S2.1, Voxelization downsampling processing is performed on the synchronized laser point cloud; S2.2, perform feature extraction and cross-view temporal matching on images acquired by multiple RGB cameras to obtain visual observation information; S2.3, the geometric constraints, visual feature constraints and inertial prediction information of laser point cloud are modeled in a unified manner to construct a tightly coupled multi-source fusion state estimation model; S2.4, The error state iterative Kalman filter method is used to recursively optimize the pose of the mobile chassis to obtain the real-time pose of the lidar in the world coordinate system; S2.5 Projects the 3D environmental information onto a 2D grid map based on the optimized pose results, constructs a high-precision 2D map of the underground parking garage, and achieves autonomous positioning.

4. The method for unmanned inspection of underground parking garages based on multi-source sensing fusion according to claim 1, characterized in that, In step S2.1, the formula for calculating representative points of the laser point cloud after voxelization is as follows: ， In the formula, V v Let v be the set of points within the v-th individual element. This represents the number of points within the voxel. p represents the representative point corresponding to this voxel. j This indicates that the vth v is located in the vth v. v The coordinates of the j-th point cloud within; In S2.4, the state update formula for the error state iterative Kalman filter method is: ， ， In the formula, x (m) x (m+1) The system state estimates, δx, are for the m-th and (m+1)-th iterations, respectively. (m) H represents the error state increment. (m) The observation Jacobian matrix is ​​given by the superscript T, which denotes transpose, R is the observation noise covariance matrix, P is the prior covariance matrix, and r is the number of covariance matrices. (m) To observe the residuals.

5. The method for unmanned inspection of underground parking garages based on multi-source sensing fusion according to claim 1, characterized in that, In S3, the multi-objective optimization objective function of the global inspection path is: ， ， In the formula: For the optimal global inspection path, arg min() represents the arg min function, p k p k+1 These are the k-th and (k+1)-th path points in the sequence, representing the location nodes on the global inspection path of the mobile chassis. For path point p k In a global static 2D raster map The corresponding obstacle cost function, For a global static 2D raster map, λ smoothness This is the path smoothness weighting coefficient. p is the Euclidean distance between adjacent path points. robot (t) represents the current position of the moving chassis at time t, corresponding to the starting point of the global inspection path; p goal The target inspection endpoint location; For global static 2D raster map A defined set of accessible free space; st represents the constraint condition, and N is the index of the last path point.

6. The method for unmanned inspection of underground parking garages based on multi-source sensing fusion according to claim 5, characterized in that, In step S3, an obstacle avoidance path is generated using a locally dynamic cost map that is updated in real time. The formulas for generating the locally cost map and the obstacle avoidance path in a dynamic environment are as follows: ， ， In the formula: Let Ψ() be the local dynamic cost map at time t, and Ψ() be the fusion mapping function between the global map and the real-time laser point cloud. LIDAR (t) represents the dynamic obstacle occupancy information collected by the lidar at time t; P local (t) represents the collision-free local obstacle avoidance path at time t, Planner local For time-based local path planners, This is the optimal global reference path.

7. The method for unmanned inspection of underground parking garages based on multi-source sensing fusion according to claim 4, characterized in that, In the semantic target spatial coordinate transformation and map anchoring algorithm of S5, the two-level pose transformation and map anchoring formula for the semantic target is as follows: ， ， ， In the formula: x represents the 3D coordinates of the semantically detected target in the camera coordinate system. plate Let x be the position coordinates of the target along the x-direction in the world coordinate system, and y be the position coordinates of the target along the x-direction. plate Let be the position coordinates of the target along the y-direction in the world coordinate system. Let be the rotation matrix, representing the attitude relationship between the camera coordinate system and the lidar coordinate system. The translation vector represents the offset of the camera coordinate system origin relative to the lidar coordinate system origin. The target's 3D coordinates in the lidar coordinate system. , To determine the real-time pose of the laser radar in the world coordinate system. and Let represent the rotation matrix and translation vector from the lidar coordinate system to the world coordinate system, respectively. u represents the absolute 3D coordinates of the target in the world coordinate system. map Let x be the anchor index of the target in the global raster map, and Π() be the projection transformation function from the world coordinate system to the raster map. min ,y min ) represents the coordinates of the map origin, Δ x Δ y represents the raster resolution, and represents the actual physical size of each raster in the x and y directions, respectively.

8. The method for unmanned inspection of underground parking garages based on multi-source sensing fusion according to claim 1, characterized in that, The field of view of multiple RGB cameras is stitched together to form a 360° panoramic visual perception range.

9. The multi-source sensing fusion method for unmanned inspection of underground parking garages according to claim 7, characterized in that, In S7, when the original planned route is impassable, global dynamic route replanning is immediately triggered to regenerate a passable inspection route. In S8, the absolute spatial coordinates and relative pose of the semantic target are calculated through a two-level pose transformation formula, which is used for pose fine-tuning control for abnormal point alignment and automatic recharging.

10. A multi-source sensing fusion unmanned inspection system for underground parking garages, used to implement the unmanned inspection method for underground parking garages as described in any one of claims 1-9, characterized in that, The system includes a mobile chassis on which a multimodal sensor array, a fire extinguishing device, a main control unit, and a synchronization board are mounted. The main control unit is communicatively connected to the mobile chassis, the multimodal sensor array, and the fire extinguishing device. The multimodal sensor array includes a lidar, multiple RGB cameras, and a thermal infrared camera. The synchronization board is electrically connected to the lidar, RGB cameras, and thermal infrared cameras. The synchronization board achieves time synchronization of data collected by the multiple sensors through hardware triggering. The synchronization board is communicatively connected to the main control unit.

Images