A method and system for multi-target identification and localization for firefighting robots
By combining lidar point cloud mapping, fire scene visual enhancement, and an improved CornerNet model, the firefighting robot has achieved stability and consistency in multi-target recognition and localization in environments with dense smoke and heat radiation, solving the problem of unstable target detection in existing technologies and generating high-precision operation control data.
Patent Information
- Application Number
- CN202611251216.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-18
- Publication Date
- 2026-11-10
AI Technical Summary
Existing firefighting robots suffer from unstable target detection and localization under conditions of dense smoke, thermal radiation interference, and missing point clouds. Map-level position calculations show large deviations, and the localization results of consecutive frames exhibit significant jumps.
By employing lidar point cloud mapping, fire scene visual enhancement, and an improved CornerNet model, combined with LeGO-LOAM continuous frame registration and point cloud layering, a visual corner point-point cloud structure anchoring map is constructed to achieve stability and consistency in target recognition and localization.
Despite interference from dense smoke and heat radiation, the firefighting robot can obtain stable real-time pose and multi-target localization results, improving the stability and anti-interference ability of target boundary extraction and generating reliable operation control data.
Smart Images

Figure CN123071965A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a method and system for multi-target recognition and localization of firefighting robots. Background Technology
[0002] With the increasing application of firefighting robots in scenarios involving high temperatures, dense smoke, toxic gases, and collapse risks, fire scene environmental perception and target localization technologies are gradually becoming crucial foundations for the autonomous operation of firefighting robots. Existing firefighting robots typically carry sensors such as visible light cameras, infrared thermal imagers, lidar, inertial measurement units, and odometry. They acquire information about the surrounding environment through image recognition, heat source detection, point cloud mapping, and pose estimation, and provide identified objects such as obstacles, personnel, fire sources, or passage boundaries to path planning, obstacle avoidance control, and rescue decision-making. In existing technologies, visual images are suitable for representing the appearance and thermal characteristics of targets, while lidar point clouds are suitable for representing spatial distance and structural morphology. These two types of data are often used for target detection and map building, respectively.
[0003] In basic technical solutions, simultaneous localization and mapping (SLAM) using lidar is commonly employed to acquire robot pose and local point cloud maps. A target recognition model based on corner detection is then used to extract target boundary regions from the images. The image detection results are then correlated with the point cloud spatial locations to estimate the target's position on the map. While this approach can achieve target detection and localization in fire situations to a certain extent, in cases of dense smoke obscuring the view, thermal radiation interference, missing point clouds, and blurred image boundaries, inconsistencies between visual detection results and point cloud structure results can easily arise. This leads to unstable target boundary localization, significant deviations in map-level position calculations, and jumps in localization results across consecutive frames.
[0004] Therefore, how to provide a multi-target identification and positioning method and system for firefighting robots is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] One objective of this invention is to propose a multi-target recognition and localization method and system for firefighting robots. This invention fully utilizes lidar point cloud mapping, fire scene visual enhancement, an improved CornerNet model, and point cloud structure anchoring technology. It details the data processing process for firefighting robots to achieve multi-target recognition, spatial localization, and operational control in environments with dense smoke, obstruction, and thermal radiation interference. This invention preprocesses multi-source sensing data from the fire scene, performs LeGO-LOAM continuous frame registration and point cloud layering, obtaining the firefighting robot's real-time pose, a local point cloud map of the fire scene, a static structure layer, and a dynamic target layer. It further enhances the fire scene visual image by combining point cloud structure projection constraint information, and extracts the target visual corner point set through an improved CornerNet model to construct a visual corner point-point cloud structure anchoring map. This achieves stable calculation of target map coordinates, target distance, target orientation, and target source markers. It possesses advantages such as strong adaptability to fire scenes, stable target localization, high consistency between recognition results and point cloud structure, and reliable firefighting robot control data.
[0006] A multi-target identification and localization method for firefighting robots according to an embodiment of the present invention includes: Collect multi-source sensing data of the fire scene during the operation of the fire-fighting robot, and preprocess the multi-source sensing data of the fire scene to generate standardized fire scene sensing data; Based on standardized fire scene perception data, the LeGO-LOAM algorithm is executed to extract point cloud features and layer the point cloud data of lidar point cloud data, thereby obtaining the real-time pose of the fire-fighting robot, the local point cloud map of the fire scene, the static structure layer and the dynamic target layer. Ground support extraction, vertical boundary extraction and occlusion structure analysis are performed based on the static structure layer, and point cloud cluster center extraction and spatial location association are performed based on the dynamic target layer to obtain the fire scene point cloud structure prior. Based on the prior knowledge of the fire point cloud structure and standardized fire perception data, back projection processing is performed to obtain the point cloud structure projection constraint information, and fire visual enhancement processing is performed to generate fire enhanced visual images. An improved CornerNet model was constructed to perform corner detection and category recognition on enhanced visual images of fire scenes. Structural constraint processing was performed by combining point cloud structure projection constraint information to obtain the target visual corner set. Based on the target visual corner point set, corner points of the same category are paired and target boundaries are generated to obtain the target boundary area. The target visual positioning points are extracted and mapped to the local point cloud map of the fire scene. They are then associated with the static structure layer and the dynamic target layer to construct a visual corner point-point cloud structure anchoring map. Based on the visual corner point-point cloud structure anchoring map, multi-relationship anchoring verification, spatial positioning calculation and continuous stabilization processing are performed to obtain the stabilized map-level multi-target positioning results and generate fire robot operation control data.
[0007] Optionally, the multi-source sensing data of the fire scene specifically includes visible light image data, infrared thermal imaging data, lidar point cloud data, lidar reflection intensity data, inertial measurement data, wheel speed and mileage data, gimbal attitude data, robot chassis pose data, smoke concentration data, and acquisition timestamp data.
[0008] Optionally, the preprocessing of multi-source sensing data at the fire scene specifically includes image distortion correction, infrared registration, time synchronization, point cloud cleaning, pose correction, data standardization, and multi-source data association processing.
[0009] Optionally, obtaining the real-time pose of the firefighting robot, the local point cloud map of the fire scene, the static structure layer, and the dynamic target layer includes: Based on standardized fire scene perception data, the LeGO-LOAM algorithm is executed to perform range filtering, time alignment and reflection intensity normalization on lidar point cloud data. Based on reflection intensity fluctuation, point cloud density change, echo persistence and smoke concentration, a fire scene smoke degradation field is generated to obtain a degradation weighted effective point cloud. Based on the local curvature of the degradation weighted effective point cloud, edge feature points and planar feature points are extracted by combining the degradation field of fire smoke and dust. The residual weights are assigned according to the degradation intensity of smoke and dust and the persistence of echoes to obtain the weighted point cloud feature set and the point cloud registration residual set. Based on the weighted point cloud feature set, point cloud registration residual set, inertial measurement data and wheel speed mileage data, LeGO-LOAM continuous frame registration is performed. Starting from the second acquisition time, visual corner point anchorage residual is generated by combining the visual corner point-point cloud structure anchorage map of the previous acquisition time, and joint pose optimization and accumulation are performed to obtain the real-time pose of the fire-fighting robot. A local point cloud map of the fire scene is constructed based on the real-time pose of the firefighting robot and the weighted point cloud feature set. The update status is determined based on the point cloud registration residual, the point cloud density change and the fire smoke degradation field. Stable areas are written into the local point cloud map of the fire scene, degraded areas are frozen, and point clouds in the degraded areas that meet the dynamic change conditions are temporarily stored as a dynamic candidate point cloud set to obtain the updated local point cloud map of the fire scene. Based on the real-time pose of the firefighting robot, the gimbal attitude, the updated local point cloud map of the fire scene, and the dynamic candidate point cloud set, visual observation rays are generated. The visual observation rays are cross-judged with the fire scene structure to generate an occlusion ray topology map. The point cloud is then layered according to the spatial changes of the point cloud, the changes in the outer range of the point cloud, the registration residual of the point cloud, and the occlusion ray topology map to obtain a static structure layer and a dynamic target layer.
[0010] Optionally, obtaining the prior structure of the fire point cloud includes: Ground support point cloud is extracted based on point cloud height, normal direction and spatial stability in static structural layer, and support surface boundary is fitted to obtain ground support structure; Vertical boundary point cloud is extracted based on edge feature points, planar feature points and reflection intensity changes in the static structure layer, and occlusion structure analysis is performed by combining the real-time pose of the fire-fighting robot and the gimbal attitude to obtain the vertical boundary structure and occlusion structure. The point cloud cluster centers are extracted based on the dynamic target layer, and the spatial location of the point cloud cluster centers is associated with the ground support structure, vertical boundary structure and occlusion structure to obtain the fire scene point cloud structure prior.
[0011] Optionally, generating the enhanced visual image of the fire scene includes: Based on the prior knowledge of the fire scene point cloud structure, the real-time pose of the fire-fighting robot, the gimbal attitude data and the robot chassis pose data, a point cloud visual mapping relationship is established between the local point cloud map of the fire scene and the visible light image and infrared thermal imaging. Based on the visual mapping relationship of point cloud, the ground support structure, vertical boundary structure, occlusion structure and point cloud cluster center are back-projected to obtain the point cloud structure projection constraint information. Based on point cloud structure projection constraint information, visible light images, infrared thermal imaging, and smoke concentration data, boundary preservation, contour enhancement, and thermal feature fusion are performed to generate enhanced visual images of the fire scene.
[0012] Optionally, obtaining the target visual corner point set includes: An improved CornerNet model is constructed, which consists of a visual feature extraction module, a point cloud structure constraint module, a corner detection and recognition module, and a corner matching output module. The visual feature extraction module performs multi-scale feature extraction and hierarchical feature fusion on the enhanced visual image of the fire scene, extracting infrared thermal boundary features, visible light contour features and target boundary texture features to obtain a set of visual features of the fire scene. The point cloud structure constraint module encodes the constraint features of the point cloud structure projection constraint information, generates a corner constraint weight map based on the vertical boundary projection line, the occlusion projection area, the ground support projection area and the dynamic target projection center, and fuses it with the fire scene visual feature set to obtain the structural constraint visual feature set. The corner detection and recognition module performs response prediction of the top left corner key point, response prediction of the bottom right corner key point, and target category prediction on the set of structural constraint visual features, respectively, to obtain the response map of the top left corner key point, the response map of the bottom right corner key point, and the target category recognition result; The response maps of the top left and bottom right key points are corrected based on the corner constraint weight map. Missing corner points are filled in based on the occlusion projection area, vertical boundary projection line and the detected corner point position. The response of candidate corner points is resampled along the point cloud projection boundary direction to obtain the candidate target corner point set. The corner matching output module performs same-category corner matching on the candidate target corner set and the target category recognition result. It performs ternary consistency verification based on the candidate corner, the dynamic target projection center and the ground support projection area. It performs categorization correction based on the conflict area between the infrared thermal boundary and the vertical boundary of the point cloud. It also performs corner offset correction and confidence screening to obtain the target visual corner set. The improved CornerNet model was trained using corner detection error, target category recognition error, corner structure consistency error, and corner matching error as joint optimization objectives. The parameters of the visual feature extraction module, point cloud structure constraint module, corner detection and recognition module, and corner matching output module were continuously optimized. When the relative decrease rate of the joint error in 5 consecutive training rounds was less than 0.001, or when the number of training rounds reached 200, the improved CornerNet model was deemed to have converged and training was stopped.
[0013] Optionally, the construction of the visual corner point-point cloud structure anchoring map includes: Based on the corner point category, corner point direction, and corner point confidence of the target visual corner point set, corner points of the same category are paired to generate the target boundary region; Based on the target boundary area, ground support projection area and dynamic target projection center, the target visual positioning points are extracted, and combined with the point cloud structure projection constraint information, the real-time pose of the fire robot, the gimbal attitude and the robot chassis pose data, they are mapped to the local point cloud map of the fire scene to obtain target map candidate points. Based on the spatial correspondence between the candidate points of the target map and the static structure layer and dynamic target layer, corner support association, boundary anchoring association and dynamic center association are generated, and a visual corner point-point cloud structure anchoring map is constructed.
[0014] Optionally, the generation of fire-fighting robot operation control data includes: Anchoring verification is performed based on the visual corner point-point cloud structure anchoring map, including corner point support association, boundary anchoring association, and dynamic center association. Candidate target map points with inconsistent structures are eliminated to obtain effective target anchoring points. Spatial positioning calculations are performed based on effective target anchor points, real-time pose of the firefighting robot, and local point cloud map of the fire scene to obtain target map coordinates, target distance, target orientation, and target source marker; Based on the collected timestamp data, the target map coordinates, target distance, target orientation and target source markers in the continuous collection time are correlated and stabilized. Short-term sudden change in positioning results are removed to obtain stable map-level multi-target positioning results, and fire robot operation control data is generated.
[0015] A multi-target identification and positioning system for firefighting robots according to an embodiment of the present invention includes the following modules: The data preprocessing module is used to collect and preprocess multi-source fire scene sensing data to generate standardized fire scene sensing data. The point cloud mapping module is used to execute the LeGO-LOAM algorithm to extract features and layer the LiDAR point cloud, and generate the real-time pose of the fire-fighting robot, the local point cloud map of the fire scene, the static structure layer and the dynamic target layer. The structural prior module is used to extract ground supports, vertical boundaries, occlusion structures, and point cloud cluster centers to generate fire scene point cloud structural priors. The projection enhancement module is used to perform back projection and visual enhancement to generate point cloud structure projection constraint information and enhanced visual images of the fire scene; The corner detection module is used to build an improved CornerNet model, perform corner detection, category recognition, and structural constraints, and generate a set of target visual corners; The anchoring map module is used to generate the target boundary region, extract and map the target visual positioning points, and construct a visual corner point-point cloud structure anchoring map; The positioning control module is used to perform anchoring verification, spatial positioning and continuous stabilization, and generate map-level multi-target positioning results and fire robot operation control data.
[0016] The beneficial effects of this invention are: This invention presents a multi-target recognition and localization method and system for firefighting robots. By collecting multi-source sensing data from the fire scene and combining LeGO-LOAM continuous frame registration, point cloud layering, prior knowledge of the fire scene point cloud structure, point cloud visual mapping, and an improved CornerNet model, the firefighting robot can simultaneously obtain stable real-time pose, a local point cloud map of the fire scene, a static structure layer, and a dynamic target layer under conditions of dense smoke obstruction, thermal radiation interference, and point cloud degradation. Compared to methods that rely solely on image recognition or individual point cloud mapping, this invention inversely constrains the ground support structure, vertical boundary structure, occlusion structure, and dynamic target layer information from the point cloud into the visual image. This ensures that target corner detection no longer relies solely on blurred image boundaries, thereby improving the stability and anti-interference capability of target boundary extraction in the fire scene.
[0017] This invention further constructs a visual corner point-point cloud structure anchoring map by establishing spatial correspondences between the target visual corner point set and the static structure layer and dynamic target layer. Anchoring verification is performed through corner point support association, boundary anchoring association, and dynamic center association. This process can eliminate candidate target map points with inconsistent structures, reducing the impact of falsely detected corner points, occluded corner points, and point cloud drift on map-level positioning results. Through continuous association stabilization processing at each acquisition time, this invention can suppress short-term abrupt changes in positioning results, obtaining stable map-level multi-target positioning results. Based on these results, fire robot operation control data is generated, offering advantages such as high positioning accuracy, good continuity, strong fire scene adaptability, and high operational control reliability. Attached Figure Description
[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a multi-target identification and localization method for firefighting robots proposed in this invention; Figure 2 This is a schematic diagram of the structure of an improved CornerNet model for a multi-target recognition and localization method for firefighting robots proposed in this invention; Figure 3 This is a schematic diagram of the structure of a multi-target recognition and positioning system for firefighting robots proposed in this invention. Detailed Implementation
[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0020] refer to Figure 1 and Figure 2 A multi-target recognition and localization method for firefighting robots, comprising: Collect multi-source sensing data of the fire scene during the operation of the fire-fighting robot, and preprocess the multi-source sensing data of the fire scene to generate standardized fire scene sensing data; Based on standardized fire scene perception data, the LeGO-LOAM algorithm is executed to extract point cloud features and layer the point cloud data of lidar point cloud data, thereby obtaining the real-time pose of the fire-fighting robot, the local point cloud map of the fire scene, the static structure layer and the dynamic target layer. Ground support extraction, vertical boundary extraction and occlusion structure analysis are performed based on the static structure layer, and point cloud cluster center extraction and spatial location association are performed based on the dynamic target layer to obtain the fire scene point cloud structure prior. Based on the prior knowledge of the fire point cloud structure and standardized fire perception data, back projection processing is performed to obtain the point cloud structure projection constraint information, and fire visual enhancement processing is performed to generate fire enhanced visual images. An improved CornerNet model was constructed to perform corner detection and category recognition on enhanced visual images of fire scenes. Structural constraint processing was performed by combining point cloud structure projection constraint information to obtain the target visual corner set. Based on the target visual corner point set, corner points of the same category are paired and target boundaries are generated to obtain the target boundary area. The target visual positioning points are extracted and mapped to the local point cloud map of the fire scene. They are then associated with the static structure layer and the dynamic target layer to construct a visual corner point-point cloud structure anchoring map. Based on the visual corner point-point cloud structure anchoring map, multi-relationship anchoring verification, spatial positioning calculation and continuous stabilization processing are performed to obtain the stabilized map-level multi-target positioning results and generate fire robot operation control data.
[0021] In this embodiment, the multi-source sensing data of the fire scene specifically includes visible light image data, infrared thermal imaging data, lidar point cloud data, lidar reflection intensity data, inertial measurement data, wheel speed and mileage data, gimbal attitude data, robot chassis pose data, smoke concentration data, and acquisition timestamp data.
[0022] In this embodiment, the preprocessing of multi-source sensing data at the fire scene specifically includes image distortion correction, infrared registration, time synchronization, point cloud cleaning, pose correction, data standardization, and multi-source data association processing, wherein: Point cloud cleaning refers to reading lidar point cloud data and lidar reflection intensity data, calculating the distance from each lidar point to the fire-fighting robot, deleting lidar points with missing coordinates, reflection intensity less than 0.05, distance less than 0.30 meters, and distance greater than 30 meters, counting the number of neighboring points with a radius of 0.20 meters, deleting lidar points with fewer than 5 neighboring points, dividing the point cloud space with a voxel side length of 0.05 meters, and retaining the lidar point with the highest reflection intensity within the same voxel; Pose correction involves converting the inertial attitude angle to the robot chassis coordinate system, multiplying the left and right wheel velocities by the sampling time interval to obtain the left and right wheel displacements, and using the average of the left and right wheel displacements as the forward displacement of the chassis. Subtracting the left wheel displacement from the right wheel displacement and dividing by the wheelbase yields the change in heading angle. When the attitude angle change between two adjacent synchronized data sets exceeds 15 degrees or the chassis displacement change exceeds 0.50 meters, the corresponding pose sampling value is deleted. The two valid pose sampling values before and after the deletion position are read. The timestamp of the deletion position is subtracted from the previous valid timestamp, and then divided by the difference between the previous and subsequent valid timestamps to obtain the weight of the next pose. The weight of the next pose is subtracted from 1 to obtain the weight of the previous pose. The two pose sampling values before and after are multiplied by their corresponding weights and then added together to obtain the completed pose sampling value.
[0023] In this embodiment, obtaining the real-time pose of the firefighting robot, the local point cloud map of the fire scene, the static structure layer, and the dynamic target layer includes: Based on standardized fire scene perception data, the LeGO-LOAM algorithm is executed to perform range filtering, time alignment, and reflection intensity normalization on the lidar point cloud data. A fire scene smoke degradation field is generated based on reflection intensity fluctuations, point cloud density changes, echo persistence, and smoke concentration, resulting in a degradation-weighted effective point cloud. The lidar point cloud data is subjected to range filtering, time alignment, and reflection intensity normalization, specifically as follows: Read the lidar point cloud data, lidar reflection intensity data, and acquisition timestamp data. Sum the squares of the three-dimensional coordinates of each lidar point and then take the square root to obtain the distance from the lidar point to the fire-fighting robot. Keep lidar points within a distance of 0.30 meters to 30 meters. Based on the visible light image acquisition timestamp, keep point cloud frames with a time difference of less than 20 milliseconds. Divide the reflection intensity value of the lidar point by the maximum reflection intensity value of the lidar to obtain the normalized reflection intensity value. The fire scene smoke degradation field is generated based on reflection intensity fluctuations, point cloud density changes, echo persistence, and smoke concentration, specifically as follows: Point cloud spatial units are divided with a side length of 0.20 meters. The difference between the average normalized reflection intensity of the same spatial unit in the current frame and the previous frame is taken as the absolute value to obtain the reflection intensity fluctuation. The difference between the number of laser points in the same spatial unit in the current frame and the previous frame is taken as the absolute value, and then divided by the number of laser points in the previous frame. When the number of laser points in the previous frame is 0, the denominator is 1 to obtain the point cloud density change. The number of point frames in the same spatial unit in 3 consecutive frames is divided by 3 to obtain the echo duration ratio. The smoke concentration value is divided by the upper limit of the sensor range to obtain the smoke concentration normalization value. The reflection intensity fluctuation, point cloud density change, the value after 1 minus the echo duration ratio and the smoke concentration normalization value are summed according to the weights of 0.30, 0.25, 0.25 and 0.20 respectively to obtain the smoke degradation intensity. The smoke degradation field of the fire scene is composed of the smoke degradation intensity of all spatial units. The degenerate weighted effective point cloud is obtained as follows: The smoke degradation intensity of the corresponding spatial unit in the smoke degradation field of the fire site is written into the laser point. Laser points with smoke degradation intensity greater than 0.85 and echo duration ratio less than 0.34 are deleted. The normalized reflection intensity value of the remaining laser points is multiplied by 1 minus the smoke degradation intensity to obtain the degradation weighted reflection intensity value. The degradation weighted effective point cloud is composed of the coordinates of the remaining laser points, the degradation weighted reflection intensity value and the smoke degradation intensity. Based on the local curvature of the degradation-weighted effective point cloud, edge feature points and planar feature points are extracted by combining the degradation field of fire smoke. Residual weights are then assigned according to the smoke degradation intensity and echo persistence to obtain a weighted point cloud feature set and a point cloud registration residual set, where: Local curvature in cloud computing based on degenerate weighted effective points is specifically as follows: Read the current laser point, the left adjacent laser point, and the right adjacent laser point from the degenerate weighted effective point cloud according to the LiDAR scan line. Obtain the coordinates and degenerate weighted reflection intensity values of the three laser points respectively. Subtract twice the coordinates of the current laser point from the sum of the coordinates of the left and right adjacent laser points to obtain the coordinate change. Divide the average degenerate weighted reflection intensity of the left and right adjacent laser points by the degenerate weighted reflection intensity value of the current laser point to obtain the reflection intensity correction coefficient. When the degenerate weighted reflection intensity value of the current laser point is 0, take 0.01 for division. Sum the squares of the three axis values of the coordinate change, take the square root, and then multiply by the reflection intensity correction coefficient to obtain the local curvature value of the current laser point. By combining the degradation field of fire smoke and dust, edge feature points and planar feature points are extracted, specifically: Read the local curvature value, degraded weighted reflection intensity value, smoke and dust degradation intensity, and the number of laser points in a 0.20-meter neighborhood for each laser point in the degraded weighted effective point cloud. Retain laser points with smoke and dust degradation intensity less than 0.70 and degraded weighted reflection intensity greater than 0.10. Laser points with local curvature values greater than 0.15 are designated as edge feature points, and laser points with local curvature values less than 0.03 and a neighborhood number of laser points greater than or equal to 8 are designated as planar feature points, thus obtaining the set of edge feature points and the set of planar feature points. The residual weights are assigned based on the intensity of smoke degradation and the persistence of echoes, specifically as follows: Read the smoke degradation intensity, echo duration ratio, and degradation weighted reflection intensity value of edge feature points and planar feature points. Multiply the value obtained by subtracting the smoke degradation intensity from 1 by 0.60, the echo duration ratio by 0.25, and the degradation weighted reflection intensity value by 0.15. Then add the three products to obtain the residual weight. When the residual weight is less than 0.20, take 0.20; when the residual weight is greater than 1, take 1. Write the residual weight into the corresponding feature point to obtain the weighted point cloud feature set. The point cloud registration residual set is composed of the point-to-line residual term of the edge feature point, the point-to-area residual term of the planar feature point, and the residual weight. Based on the weighted point cloud feature set, point cloud registration residual set, inertial measurement data, and wheel speed-mileage data, LeGO-LOAM continuous frame registration is performed. Starting from the second acquisition time, visual corner point anchorage residuals are generated by combining the visual corner point-point cloud structure anchorage map from the previous acquisition time. Joint pose optimization and accumulation are then performed to obtain the real-time pose of the fire-fighting robot, where: The execution of LeGO-LOAM continuous frame registration is as follows: Read the weighted point cloud feature set, point cloud registration residual set, inertial measurement data and wheel speed mileage data, use the attitude angle change as the initial value of rotation and the wheel speed displacement change as the initial value of translation, and combine them to form the initial value of pose for the current frame. Based on the initial pose value of the current frame, the edge feature points and planar feature points of the current frame are transformed to the point cloud coordinate system of the previous frame. For edge feature points, the two nearest edge feature points from the previous frame that are less than 0.50 meters apart are matched, and the distance from the point to the line is calculated. For planar feature points, the three nearest planar feature points from the previous frame that are less than 0.50 meters apart are matched, and the distance from the point to the surface is calculated. The smoke degradation intensity, echo duration ratio, and degradation-weighted reflection intensity value of the feature points are read, and the value obtained by subtracting the smoke degradation intensity from 1 is multiplied by 0.60. Multiply the echo duration ratio by 0.25, multiply the degradation weighted reflection intensity value by 0.15, and then add the three products to obtain the residual weight. When the residual weight is less than 0.20, take 0.20, and when the residual weight is greater than 1, take 1. Multiply the point-to-line distance and the point-to-area distance by the corresponding residual weight to obtain the single-point weighted residual. The continuous frame registration residual set is composed of all the single-point weighted residuals. After adding all the single-point weighted residuals, divide by the number of feature points participating in the registration to obtain the continuous frame registration residual value. Starting from the second acquisition time, the visual corner point anchoring residual is generated by combining the visual corner point-point cloud structure anchoring map from the previous acquisition time. Specifically: Read the visual corner point-point cloud structure anchoring map from the previous acquisition time, extract the target map candidate points that have passed the anchoring check and have a corner point comprehensive confidence score greater than 0.75 as visual anchoring points, transform the visual anchoring points to the current frame point cloud coordinate system based on the initial pose value of the current frame, match the nearest point cloud structure position with a distance of less than 0.50 meters in the current frame point cloud structure, calculate the three-dimensional distance between the transformed position of the visual anchoring point and the nearest point cloud structure position, and obtain a single visual anchoring residual. The visual corner point anchoring residual set is composed of all the single visual anchoring residuals, and the visual corner point anchoring residual value is obtained by adding all the single visual anchoring residuals and dividing by the number of visual anchoring points. Joint pose optimization and accumulation are performed, specifically as follows: Read the registration residual value of consecutive frames and the visual corner point anchoring residual value. Multiply the registration residual value of consecutive frames by 0.80 and the visual corner point anchoring residual value by 0.20, and then add them together to obtain the joint pose error value. Using the translation and rotation in the initial pose value of the current frame as the adjustment object, reduce the joint pose error value round by round. When the difference between the joint pose error values of two adjacent rounds is less than 0.001 or the iteration reaches 30 rounds, stop the adjustment. Use the translation and rotation at the time of stopping the adjustment as the pose transformation of the current frame. Multiply the pose of the fire robot at the previous acquisition moment with the pose transformation of the current frame to obtain the real-time pose of the fire robot. A local point cloud map of the fire scene is constructed based on the real-time pose of the firefighting robot and a weighted point cloud feature set. The update status is determined according to the point cloud registration residual, point cloud density change, and fire smoke degradation field. Stable regions are written into the local point cloud map, degraded regions are frozen, and point clouds in the degraded regions that meet the dynamic change conditions are temporarily stored as a dynamic candidate point cloud set, resulting in the updated local point cloud map of the fire scene. The construction of a local point cloud map of the fire scene is as follows: The real-time pose and weighted point cloud feature set of the fire-fighting robot are read. Based on the real-time pose of the fire-fighting robot, the coordinates of the laser points are transformed into the coordinate system of the local fire scene map. The map space is divided with a voxel side length of 0.05 meters. The smoke degradation intensity, echo duration ratio and degradation weighted reflection intensity value of each laser point in the same voxel are read. The residual weights are obtained by weighted summation according to weights of 0.60, 0.25 and 0.15. The laser point with the largest residual weight in the same voxel is retained. The map coordinates, degradation weighted reflection intensity value and residual weight of the retained laser points are used as map point content. The initial local fire scene point cloud map is composed of all map points according to the map coordinates. The update status is determined based on the point cloud registration residual, point cloud density change, and fire smoke degradation field, specifically as follows: The initial local point cloud map of the fire scene is divided into map regions with a side length of 0.20 meters. The weighted residuals of single points falling into the same map region in the registration residual set of consecutive frames are added together and then divided by the number of feature points participating in the registration in the map region to obtain the average weighted residual of single points. The number of laser points falling into the same map region in the current frame and the number of laser points falling into the same map region in the previous frame are counted. The two numbers are subtracted and the absolute value is taken. Then, it is divided by the number of laser points in the previous frame to obtain the percentage change in the number of points. When the number of laser points in the previous frame is 0, the denominator is 1. The smoke degradation intensity of each spatial unit in the same map region in the smoke degradation field of the fire scene is read. The total smoke degradation intensity is added together and then divided by the number of spatial units to obtain the average smoke degradation intensity. When the average weighted residual of a single point is less than 0.10 meters, the percentage change in the number of points is less than 20%, and the average smoke and dust degradation intensity is less than 0.40, the corresponding map area will be marked as a stable area, and the remaining map areas will be marked as degraded areas. The dynamic change condition refers to the criteria for determining whether the point cloud region undergoes positional or morphological changes during continuous acquisition. Specifically, it involves calculating the three-dimensional distance between the center coordinates of the current frame point cloud and the center coordinates of the previous frame point cloud as the change in the spatial position of the point cloud. It also involves calculating the absolute values of the differences between the outer length, width, and height of the current frame point cloud and the outer length, width, and height of the previous frame point cloud, and taking the maximum value as the change in the outer range of the point cloud. The dynamic change condition is deemed to be met when any one of the following conditions is met: the change in the spatial position of the point cloud is not less than 0.20 meters, the change in the outer range of the point cloud is not less than 0.15 meters, or the average weighted residual of a single point is not less than 0.10 meters. The updated local point cloud map of the fire area is obtained as follows: Read the stable region, the degraded region, the initial local point cloud map of the fire scene, and the dynamic change conditions. Keep the map points in the stable region in the local point cloud map of the fire scene. Stop writing new static map points in the degraded region. Write the coordinates, circumscribed range, single-point weighted residual average value, and acquisition timestamp of the laser points in the degraded region that meet the dynamic change conditions into the dynamic candidate point cloud set. The updated local point cloud map of the fire scene is composed of the map points after keeping the stable region and freezing the degraded region. Based on the real-time pose of the firefighting robot, the gimbal attitude, the updated local point cloud map of the fire scene, and the dynamic candidate point cloud set, visual observation rays are generated. Cross-judgment is performed between the visual observation rays and the fire scene structure to generate an occlusion ray topology map. Point cloud layers are then created based on point cloud spatial changes, changes in the point cloud circumscribed range, point cloud registration residuals, and the occlusion ray topology map, resulting in a static structure layer and a dynamic target layer. The generation of visually observed rays is as follows: The system reads the real-time pose of the firefighting robot, the attitude of the gimbal, the updated local point cloud map of the fire scene, and the dynamic candidate point cloud set. The position of the firefighting robot is used as the starting point of the ray. The gimbal pitch angle, gimbal yaw angle and firefighting robot heading angle are added to obtain the observation direction. The center point of the dynamic candidate point cloud and the boundary point of the fire scene structure are used as the endpoints of the ray. When the ray length exceeds 30 meters, it is truncated to 30 meters. The visual observation ray is composed of the ray starting point, the ray endpoint, the observation direction and the target marker. The analysis involves cross-referencing visual observation rays with fire scene structures, specifically: Read the walls, door frames, fixed obstacles, and passage boundaries in the visual observation rays and the updated local point cloud map of the fire scene. Calculate the shortest distance from each visual observation ray to the fire scene structure point cloud. When the shortest distance is less than 0.05 meters and the fire scene structure point cloud is located between the ray origin and the center point of the dynamic candidate point cloud, connect the visual observation ray with the corresponding occlusion structure. The connection relationship between all visual observation rays, occlusion structures, occlusion positions, and occluded targets constitutes the occlusion ray topology map. The static structure layer and dynamic target layer are obtained as follows: Read the updated local fire point cloud map, dynamic candidate point cloud set, continuous frame registration residual set, and occlusion ray topology map. Write the point clouds in the updated local fire point cloud map whose spatial position change is less than 0.20 meters, whose point cloud boundary change is less than 0.15 meters, whose single-point weighted residual average is less than 0.10 meters, and which are not marked by the dynamic candidate point cloud set into the static structure layer to form the static structure layer. Write the point clouds in the dynamic candidate point cloud set that meet the dynamic change conditions and are not marked as fixed occlusion by the occlusion ray topology map into the dynamic target layer to form the dynamic target layer.
[0024] In this embodiment, obtaining the prior structure of the fire point cloud includes: Ground support point cloud is extracted based on point cloud height, normal direction, and spatial stability in the static structural layer, and support surface boundary fitting is performed to obtain the ground support structure, wherein: Ground support point cloud is extracted based on point cloud height, normal direction, and spatial stability in the static structural layer, specifically as follows: Read the point cloud coordinates in the static structure layer at the current acquisition time, take the vertical coordinate value of the point cloud as the point cloud height, sort the point cloud heights from smallest to largest, take the average of the top 5% of point cloud heights to obtain the ground reference height, subtract the ground reference height from the point cloud height to obtain the relative height, select two non-overlapping adjacent point clouds within a 0.20-meter neighborhood, subtract the current point cloud coordinates from the coordinates of the two adjacent point clouds to obtain two direction vectors, perform a cross product on the two direction vectors to obtain the normal direction, take the difference between the current point cloud coordinates and the nearest point cloud coordinates at the previous acquisition time, sum the squares of the differences across the three axes and take the square root to obtain the spatial stability value, retain the point clouds with a relative height less than 0.15 meters, an angle between the normal direction and the vertical direction less than 15 degrees, and a spatial stability value less than 0.05 meters as ground support point clouds; The support surface boundary is fitted, specifically as follows: Read the ground support point cloud, retain the horizontal coordinates of the ground support point cloud and remove the vertical coordinates to obtain the horizontal projection points. Divide the horizontal grid at intervals of 0.10 meters, retain the grid containing the horizontal projection points, and connect the center points of the outermost grid according to the rule that the adjacent distance is less than 0.15 meters to form the boundary line. The ground support structure is composed of the ground support point cloud, the ground reference height and the boundary line. Vertical boundary point clouds are extracted based on edge feature points, planar feature points, and reflection intensity changes in the static structure layer. Occlusion structure analysis is then performed using the real-time pose of the firefighting robot and the gimbal's attitude, yielding the vertical boundary structure and occlusion structure. Among these: Vertical boundary point clouds are extracted based on edge feature points, planar feature points, and reflection intensity variations in the static structure layer. Specifically: Read the point cloud coordinates and degraded weighted reflection intensity values retained by the updated local point cloud map of the fire scene in the static structure layer. Within a 0.20-meter neighborhood, subtract twice the current point cloud coordinates from the sum of the coordinates of the left and right adjacent point clouds. Take the square root of the sum of the squares of the differences in the three axes to obtain the local curvature value. Point clouds with local curvature values greater than 0.15 are used as edge feature points. Point clouds with local curvature values less than 0.03 and a number of neighborhood points greater than or equal to 8 are used as planar feature points. Subtract the minimum value from the maximum value of the degraded weighted reflection intensity in the same neighborhood to obtain the reflection intensity change. Subtract the minimum value from the maximum value of the point cloud height in the same neighborhood to obtain the height span. Retain edge feature points and planar feature points with a height span greater than 0.50 meters, an angle between the normal direction and the horizontal plane less than 20 degrees, and a reflection intensity change greater than 0.18 as vertical boundary point clouds. The occlusion structure is analyzed by combining the real-time pose of the firefighting robot and the attitude of the gimbal. Specifically: The system reads the real-time pose of the firefighting robot, the attitude of the gimbal, and the point cloud of the vertical boundary. Taking the position of the firefighting robot as the observation starting point, the system adds the pitch angle, yaw angle of the gimbal, and the heading angle of the robot to obtain the observation direction. An observation line segment is formed from the observation starting point toward the point cloud of the vertical boundary. The system calculates the shortest distance from the point cloud in the static structure layer to the observation line segment and the distance from the point cloud to the observation starting point. When the shortest distance is less than 0.05 meters and the distance from the point cloud to the observation starting point is more than 0.10 meters less than the distance from the vertical boundary point cloud to the observation starting point, the corresponding point cloud is taken as the occlusion point cloud. The coordinates of the occlusion point cloud are projected onto the observation line segment to obtain the coordinates of the projection point of the line segment. The coordinates of the projection point of the line segment are taken as the occlusion position. The vertical boundary structure is formed by the vertical boundary point cloud, and the occlusion structure is formed by the occlusion point cloud and the occlusion position. Point cloud cluster centers are extracted based on the dynamic target layer, and spatial relationships are established between these cluster centers and ground support structures, vertical boundary structures, and occlusion structures to obtain the fire scene point cloud structure prior, where: Point cloud cluster centers are extracted based on a dynamic target layer, specifically as follows: Read the point cloud coordinates in the dynamic target layer. Taking any point cloud as the starting point, group the point clouds that are less than 0.30 meters away from the starting point into the same target point cloud block. Continue to merge point clouds that are less than 0.30 meters away from any point cloud in the target point cloud block. Delete target point cloud blocks with less than 10 points. Add the three-axis coordinates of all points in the retained target point cloud block and divide by the number of points to obtain the point cloud cluster center. The prior knowledge of the fire point cloud structure is as follows: Read the point cloud cluster center, ground support structure, vertical boundary structure, and occlusion structure. Subtract the ground reference height from the vertical coordinates of the point cloud cluster center to obtain the vertical distance. Calculate the horizontal distance from the point cloud cluster center to the nearest boundary point in the vertical boundary structure. Calculate the three-dimensional distance from the point cloud cluster center to the nearest occlusion point in the occlusion structure to obtain the occlusion distance. When the vertical distance is less than 2.50 meters, the horizontal distance is less than 1.00 meters, and the occlusion distance is greater than 0.10 meters, establish a spatial relationship between the point cloud cluster center and the corresponding ground support structure and vertical boundary structure. The fire scene point cloud structure prior is composed of the point cloud cluster center, ground support structure, vertical boundary structure, occlusion structure, and spatial relationship.
[0025] In this embodiment, generating enhanced visual images of the fire scene includes: Based on prior knowledge of the fire scene point cloud structure, real-time pose of the firefighting robot, gimbal attitude data, and robot chassis pose data, a point cloud visual mapping relationship is established between the local point cloud map of the fire scene and visible light images and infrared thermal images, wherein: Establish a visual mapping relationship between the local point cloud map of the fire scene and visible light images and infrared thermal images, specifically as follows: The system reads the local point cloud map of the fire scene, the prior structure of the fire scene point cloud, the real-time pose of the firefighting robot, the gimbal attitude data, and the robot chassis pose data. It subtracts the firefighting robot's position coordinates from the point cloud coordinates to obtain the robot's relative point cloud coordinates. The coordinates are then corrected sequentially using the chassis heading angle, gimbal yaw angle, and gimbal pitch angle. For each rotation, the first axis coordinate is multiplied by the cosine of the angle, and then the second axis coordinate is multiplied by the sine of the angle to obtain the rotated first axis coordinate. The first axis coordinate is then multiplied by the sine of the angle and then the second axis coordinate is multiplied by the cosine of the angle to obtain the rotated second axis coordinate. The coordinates are obtained by taking the camera observation coordinates, deleting point cloud points with a depth value less than or equal to 0.10 meters, dividing the horizontal coordinate by the depth value, multiplying by the visible light focal length, and adding the horizontal coordinate of the visible light principal point to obtain the visible light horizontal pixel coordinates, dividing the vertical coordinate by the depth value, multiplying by the visible light focal length, and adding the vertical coordinate of the visible light principal point to obtain the visible light vertical pixel coordinates. The registered infrared thermal image adopts the visible light image coordinate system, using the visible light pixel coordinates as the infrared pixel coordinates. The point cloud visual mapping relationship is composed of point cloud coordinates, visible light pixel coordinates, infrared pixel coordinates, and depth values. Based on the visual mapping relationship of point clouds, back projection is performed on the ground support structure, vertical boundary structure, occlusion structure, and point cloud cluster centers to obtain the point cloud structure projection constraint information, where: Back projection is performed on the ground support structure, vertical boundary structure, occlusion structure, and point cloud cluster centers, specifically as follows: Read the visual mapping relationship of the point cloud, the ground support structure, the vertical boundary structure, the occlusion structure, and the point cloud cluster center. Find the pixel coordinates corresponding to the ground support point cloud in the visual mapping relationship, connect the outermost pixels and fill the pixels within the connection range to obtain the ground support projection area. Find the pixel coordinates corresponding to the vertical boundary point cloud, connect the pixels with an adjacent distance of less than 3 pixels to obtain the vertical boundary projection line. Find the pixel coordinates corresponding to the occlusion point cloud and the occlusion position, and expand outward by 5 pixels with the corresponding pixel as the center to obtain the occlusion projection area. Find the pixel coordinates corresponding to the point cloud cluster center to obtain the dynamic target projection center. The point cloud structure projection constraint information is composed of the four types of projection results. Based on point cloud structure projection constraint information, visible light images, infrared thermal imaging, and smoke concentration data, boundary preservation, contour enhancement, and thermal feature fusion are performed to generate enhanced visual images of the fire scene, including: The boundary preservation, contour enhancement, and thermal feature fusion are performed as follows: The system reads point cloud structure projection constraint information, visible light image, infrared thermal imaging, and smoke concentration data. It copies the visible light image, retaining the original pixel values within a 3-pixel range on both sides of the vertical boundary projection line. Within the occluded projection area, the original pixel values are multiplied by 0.70. Pixels with a grayscale difference greater than 25 between adjacent pixels are taken as contour pixels, and their values are multiplied by 1.20 (or 255 if the difference exceeds 255). The infrared temperature value is subtracted by 20 degrees Celsius and divided by 580 to obtain the infrared normalized value. The smoke concentration value is divided by the sensor's upper limit to obtain the smoke normalized value. When the smoke normalized value is less than 0.50, the processed visible light pixel value is multiplied by 0.70, the infrared normalized value by 255, and then by 0.30, and the results are added. When the smoke normalized value is greater than or equal to 0.50, the processed visible light pixel value is multiplied by 0.45, the infrared normalized value by 255, and then by 0.55, and the results are added. The sum of each pixel's values is written to the corresponding pixel position to form an enhanced visual image of the fire scene.
[0026] In this embodiment, obtaining the target visual corner point set includes: An improved CornerNet model is constructed, which consists of a visual feature extraction module, a point cloud structure constraint module, a corner detection and recognition module, and a corner matching output module, wherein: The CornerNet model is improved as follows: Based on the traditional CornerNet model's basic image feature extraction structure and corner pooling structure, a joint extraction process for visible light contours of the fire scene, infrared thermal boundaries, and target boundary textures is added to obtain a visual feature extraction module. Before the corner prediction process of the traditional CornerNet model, a point cloud structure constraint process is added, which converts the point cloud structure projection constraint information into a corner constraint weight map and fuses it with the fire scene visual feature set to obtain a point cloud structure constraint module. Based on the traditional CornerNet model's top-left corner keypoint prediction, bottom-right corner keypoint prediction, and category prediction processes, a corner constraint weight map correction process is added to obtain a corner detection and recognition module. Based on the traditional CornerNet model's corner embedding matching and corner offset correction processes, same-category corner matching, ternary consistency check, and conflict correction processes are added to obtain a corner matching output module, resulting in an improved CornerNet model. The visual feature extraction module performs multi-scale feature extraction and hierarchical feature fusion on the enhanced visual images of the fire scene, extracting infrared thermal boundary features, visible light contour features, and target boundary texture features to obtain a set of visual features of the fire scene, including: The visual feature extraction module includes: Multi-scale sampler: Receives enhanced visual images of the fire scene and extracts pixel grayscale, local mean, and local variance at the original size, half size, and quarter size. Hierarchical fusion: Fuse original size features, half-size features, and quarter-size features with weights of 0.50, 0.30, and 0.20 respectively; Boundary marker: Marks the location of infrared thermal boundary features, visible light contour features, and target boundary texture features; Visual Feature Pool: Stores a set of visual features of the fire scene; The multi-scale feature extraction and hierarchical feature fusion are performed as follows: Read the enhanced visual image of the fire scene, calculate the grayscale difference between adjacent pixels, the mean of 3×3 pixels, and the variance of 3×3 pixels at the original size. The variance of 3×3 pixels is the average of the squares of the nine pixel values minus the mean of nine pixels. The half-size feature is obtained by averaging the values of each 2×2 pixel block and the quarter-size feature is obtained by averaging the values of each 4×4 pixel block. The half-size feature value is written back to the corresponding 2×2 pixel block and the quarter-size feature value is written back to the corresponding 4×4 pixel block. The original size feature is multiplied by 0.50, the half-size feature is multiplied by 0.30, and the quarter-size feature is multiplied by 0.20 and then added together to obtain the fused feature value of each pixel position. The fused feature values of all pixel positions are used to form the hierarchical fused feature. Extracting infrared thermal boundary features, visible light contour features, and target boundary texture features, specifically: Read the hierarchical fusion features and the fire scene enhanced visual image, subtract the fusion feature values of adjacent pixels and take the absolute value. The position with a difference greater than 30 and a continuous length greater than 5 pixels is regarded as the infrared thermal boundary feature. The position with a gray level difference greater than 25 between adjacent pixels in the fire scene enhanced visual image is regarded as the visible light contour feature. The position within 5 pixels around the visible light contour feature and with a 3×3 pixel variance greater than 40 is regarded as the target boundary texture feature. The fire scene visual feature set is composed of 3 types of features. The point cloud structure constraint module encodes constraint features of the point cloud structure projection constraint information. Based on the vertical boundary projection line, occlusion projection area, ground support projection area, and dynamic target projection center, it generates a corner constraint weight map, which is then fused with the fire scene visual feature set to obtain a structural constraint visual feature set, where: The point cloud structure constraint module includes: Projection constraint reader: Reads vertical boundary projection lines, occluded projection areas, ground support projection areas, and dynamic target projection centers; Constraint encoder: performs pixel-level encoding of vertical boundaries, occluded areas, ground support areas, and the center of dynamic targets respectively; Corner weight generator: Calculates corner constraint weight values based on constraint encoding values and generates a corner constraint weight map; Structural Fusioner: Fuses the set of visual features of the fire scene with the corner constraint weight map, and outputs a set of visual features constrained by structure. The constraint feature encoding process is as follows: Read the vertical boundary projection line, the occluded projection area, the ground support projection area, and the dynamic target projection center. Set the constraint coding value of the vertical boundary projection line position to 1.00, the occluded projection area position to 0.30, the ground support projection area position to 0.60, the position within a 10-pixel radius of the dynamic target projection center to 0.90, and the remaining positions to 0.50. The constraint coding values of all pixel positions form the point cloud structure constraint coding. The generation of the corner constraint weight map is as follows: Read the point cloud structure constraint code, and use the constraint code value of each pixel position as the base value. Add 0.20 to the position within 30 pixels of the vertical boundary projection line and the dynamic target projection center, add 0.10 to the position within 50 pixels above the ground support projection area, and subtract 0.20 from the position within the occluded projection area. If the value is less than 0.30, take 0.30, and if it is greater than 1.20, take 1.20 to obtain the corner constraint weight value of each pixel position. Write the corner constraint weight value into the image position of the same size according to the corresponding pixel coordinates to obtain the corner constraint weight map. The set of structurally constrained visual features is obtained as follows: Read the fire scene visual feature set and corner constraint weight map, and multiply the infrared thermal boundary feature, visible light contour feature and target boundary texture feature of each pixel position by the corner constraint weight value at the same position. The weighted three types of features form a structural constraint visual feature set. The corner detection and recognition module performs top-left corner keypoint response prediction, bottom-right corner keypoint response prediction, and target category prediction on the structural constraint visual feature set, respectively, to obtain the top-left corner keypoint response map, the bottom-right corner keypoint response map, and the target category recognition result, where: The corner detection and recognition module includes: Response graph register: Stores the response graphs of the top left and bottom right key points; Base Response Calculator: Calculates the base response value of pixel corners based on infrared thermal boundary, visible light profile, and target boundary texture; Corner source determiner: Compares the three types of feature results at the same pixel location to determine the source of the corner; Missing Corner Completer: Completes missing corners based on the occluded projection area, vertical boundary projection line, and the detected corner positions; Candidate corner pool: Stores the location, attributes, response value, and source of candidate corners; The execution of keypoint response prediction in the top left corner, keypoint response prediction in the bottom right corner, and target category prediction is as follows: Read the structural constraint visual feature set, which includes the infrared thermal boundary feature location, the visible light contour feature location, and the target boundary texture feature location. Establish a top-left corner keypoint response map and a bottom-right corner keypoint response map with the same size as the fire-enhanced visual image. Initially, all pixel values are 0. At each pixel location, determine whether it belongs to one of the three feature types. If it does, the corresponding label value is set to 1; otherwise, it is set to 0. Multiply the infrared thermal boundary label value by 0.30, the visible light contour label value by 0.40, and the target boundary texture label value by 0.30, and then add them together to obtain the pixel corner base response value. Calculate the average value of the pixel corner base response value within a 3×3 area to the top left of the pixel and write it into the top-left corner keypoint response map. Calculate the average value of the pixel corner base response value within a 3×3 area to the bottom right of the pixel and write it into the bottom-right corner keypoint response map. Compare the three label values at the same pixel location. The feature name with the highest weight and a value of 1 is taken as the corner source. Write the pixel location and corner source into the corner source recognition result as the predicted target category. Based on the corner constraint weight map, the response maps of the top left and bottom right key points are corrected. Missing corner points are filled in based on the occlusion projection area, vertical boundary projection line, and the positions of detected corner points. The responses of candidate corner points are resampled along the point cloud projection boundary direction to obtain a set of candidate target corner points, where: Based on the corner constraint weight map, the response maps of the top left and bottom right key points are corrected, specifically as follows: Read the top-left corner keypoint response map, the bottom-right corner keypoint response map, and the corner constraint weight map. Multiply each pixel value in the top-left corner keypoint response map by the corner constraint weight value at the same position to obtain the corrected top-left corner response value. Multiply each pixel value in the bottom-right corner keypoint response map by the corner constraint weight value at the same position to obtain the corrected bottom-right corner response value. Positions with corrected top-left corner response values greater than 0.60 are taken as the detected top-left corner positions, and positions with corrected bottom-right corner response values greater than 0.60 are taken as the detected bottom-right corner positions. The detected corner positions are composed of the detected top-left corner positions and the detected bottom-right corner positions. Missing corner points are filled in based on the occluded projection area, the vertical boundary projection line, and the positions of the detected corner points. Specifically: Read the occluded projection area, vertical boundary projection line, detected top-left corner position, detected bottom-right corner position, corrected top-left corner response value, and corrected bottom-right corner response value; count the number of detected corner points within a 30-pixel range of the edge of the occluded projection area. When the number is 0, no missing corner point completion is performed. When the number is 1, based on the top-left or bottom-right attribute of the detected corner point, search along the nearest vertical boundary projection line for the pixel position that is more than 20 pixels away from the detected corner point and has the largest correction response value of another type. When the maximum correction response value is greater than 0.45, the corresponding pixel position is filled as the missing corner point. When the number is greater than or equal to 2, the existing corner point position is directly retained and no completion is performed. The candidate corner point responses are resampled along the direction of the point cloud projection boundary, specifically as follows: Read the vertical boundary projection line, the ground support projection area boundary, the corrected upper left corner response value, and the corrected lower right corner response value. Take two adjacent boundary pixels, subtract the horizontal coordinates to get the horizontal difference, and subtract the vertical coordinates to get the vertical difference. Divide the horizontal difference and the vertical difference by the square root of the sum of their squares to get the unit direction. Read the corrected upper left corner response value and the corrected lower right corner response value every 2 pixels along the unit direction. Read them 5 times and calculate the average. Positions with an average value greater than 0.55 are used as candidate corner positions. The candidate target corner position is composed of the candidate corner position, the upper left or lower right corner attribute, the resampled average value, and the corner source. The corner matching output module performs same-category corner matching on the candidate target corner set and the target category recognition results. It performs a ternary consistency check based on the candidate corners, the dynamic target projection center, and the ground support projection area. It also performs categorical correction based on the conflict area between the infrared thermal boundary and the point cloud vertical boundary, and performs corner offset correction and confidence screening to obtain the target visual corner set, where: The corner matching output module includes: Same-origin matcher: Matches top-left and bottom-right candidate corners that have the same origin; Three-element verifier: verifies the positional consistency between candidate corner points, the dynamic target projection center, and the ground support projection area; Conflict corrector: Corrects corner origins based on the deviation relationship between the infrared thermal boundary and the vertical boundary projection line; Offset corrector: Selects the pixel with the largest response value within the corner's neighborhood to correct the corner coordinates; Target visual corner point pool: Stores a set of target visual corner points that have been filtered by confidence level; The matching of corner points of the same category is performed as follows: Read the candidate target corner point set and the corner point source identification results, and pair the top left corner point and the bottom right corner point with the same corner point source. When the bottom right corner point is located to the right of the top left corner point and the horizontal distance is greater than 15 pixels and the vertical distance is greater than 15 pixels, it is retained as a corner point pair of the same category. The ternary consistency check is performed as follows: Read the corner point pairs of the same category, the dynamic target projection center, and the ground support projection area. Add the x-coordinates of the candidate corner points in the upper left corner and the lower right corner, and divide by 2 to get the x-coordinate of the corner point pair center. Add the y-coordinates of the two candidate corner points and divide by 2 to get the y-coordinate of the corner point pair center. Sum the squares of the differences between the x-coordinates of the corner point pair center and the dynamic target projection center, and take the square root to get the center distance. Calculate the absolute value of the difference between the y-coordinate of the lower right corner candidate corner point and the y-coordinate of the upper boundary of the ground support projection area to get the support distance. If the center distance is less than 30 pixels and the support distance is less than 20 pixels, record the ternary consistency pass result as 1; otherwise, record it as 0, and write the ternary consistency pass result into the corresponding corner point pair of the same category. The categorization correction is performed as follows: Read the infrared thermal boundary feature position, vertical boundary projection line, corner source identification result and corner point pairs of the same category. Calculate the pixel distance from the infrared thermal boundary feature position to the nearest vertical boundary projection line. When the pixel distance is greater than 8 pixels and is located within the area enclosed by corner point pairs of the same category, the corresponding area is regarded as a conflict area. The corner source in the conflict area is adjusted to the target boundary texture feature. Corner source that is less than or equal to 8 pixels from the vertical boundary projection line remains unchanged, and the corrected corner source is obtained. The corner offset correction and confidence screening are performed as follows: Read the categorically corrected corner point pairs, the corrected top-left corner response value, the corrected bottom-right corner response value, and the ternary consistency pass result. Find the pixel position with the largest corrected top-left corner response value within a 3×3 range around the candidate top-left corner point, and move the top-left corner coordinate to the corresponding pixel position. Find the pixel position with the largest corrected bottom-right corner response value within a 3×3 range around the candidate bottom-right corner point, and move the bottom-right corner coordinate to the corresponding pixel position. Multiply the corner point response value by 0.60 and the ternary consistency pass result by 0.40, and add them together to obtain the corner point comprehensive confidence score. Keep the corrected corner points with a comprehensive confidence score greater than 0.70 to obtain the target visual corner point set. The improved CornerNet model is trained using corner detection error, target category recognition error, corner structure consistency error, and corner matching error as joint optimization objectives. The parameters of the visual feature extraction module, point cloud structure constraint module, corner detection and recognition module, and corner matching output module are continuously optimized. When the relative decrease rate of the joint error is less than 0.001 for five consecutive training rounds, or when the training rounds reach 200, the improved CornerNet model is considered converged and training is stopped. The improved CornerNet model is trained as follows: Read the enhanced visual images of the fire scene, the projection constraint information of the point cloud structure, the annotation of the top left corner point, the annotation of the bottom right corner point, the annotation of the corner point source, and the annotation of the corner point pairing relationship from the training samples. Input the enhanced visual images of the fire scene and the projection constraint information of the point cloud structure into the improved CornerNet model, and output the response map of the top left corner key point, the response map of the bottom right corner key point, the corner point source identification result, the predicted corner point pairing result, and the target visual corner point set; Set the standard response value of the pixel containing the marked corner point to 1, and set the other pixels to 0. Read the response values of the corresponding pixels in the response maps of the upper left and lower right key points, calculate the difference between each value and the standard response value, and take the absolute value. Then, average the results over all pixels to obtain the corner point detection error. Compare the predicted corner point source with the marked corner point source; if they match, record it as 0; if they don't match, record it as 1. Average the results to obtain the corner point source identification error. Calculate the distance from the center of the predicted corner point to the projection center of the dynamic target; if the distance is less than 30 pixels, the center error is recorded as 0; otherwise, it is recorded as 1. Calculate the predicted corner point source identification error. The distance from the bottom edge of the corner point to the upper boundary of the ground support projection area is measured. If the distance is less than 20 pixels, the support error is recorded as 0; otherwise, it is recorded as 1. The center error and the support error are added together and then divided by 2 to obtain the corner point structure consistency error. The predicted corner point pairing relationship is compared with the labeled corner point pairing relationship. If they are inconsistent, it is recorded as 1; if they are consistent, it is recorded as 0. The average is calculated to obtain the corner point pairing error. The corner point detection error is multiplied by 0.40, the corner point source identification error is multiplied by 0.20, the corner point structure consistency error is multiplied by 0.20, and the corner point pairing error is multiplied by 0.20, and then added together to obtain the joint error. Based on the joint error, the parameters of the visual feature extraction module, point cloud structure constraint module, corner detection and recognition module, and corner matching output module are adjusted in reverse. When the relative decrease rate of the joint error in 5 consecutive training rounds is less than 0.001, or the number of training rounds reaches 200, the improved CornerNet model is determined to have converged and training is stopped.
[0027] In this embodiment, the construction of the visual corner point-point cloud structure anchoring map includes: Based on the corner point category, direction, and confidence score of the target visual corner point set, corner points of the same category are paired to generate the target boundary region, where: The process of pairing corner points of the same category is as follows: Read the corrected corner coordinates, corner category, corner direction, and comprehensive confidence score of the target visual corner point set. Select two corner points with the same corner category and corner directions of upper left and lower right respectively as corner points to be paired. Calculate the horizontal and vertical distance between the two corner points. When the horizontal distance is greater than 15 pixels, the vertical distance is greater than 15 pixels, and the comprehensive confidence score of the two corner points is greater than 0.70, they are retained as corner point pairs of the same category. The image area enclosed by the coordinates of the upper left corner point and the coordinates of the lower right corner point is taken as the target boundary area. Based on the target boundary region, ground support projection region, and dynamic target projection center, target visual positioning points are extracted. These points are then combined with point cloud structure projection constraint information, real-time pose of the firefighting robot, gimbal attitude, and robot chassis pose data, and mapped onto a local point cloud map of the fire scene to obtain candidate target map points. Among these: The extraction of target visual positioning points is as follows: Read the target boundary area, the ground support projection area, and the dynamic target projection center. Add the left and right horizontal coordinates of the target boundary area and divide by 2 to obtain the horizontal coordinate of the positioning point. When the distance between the lower boundary of the target boundary area and the upper boundary of the ground support projection area is less than 20 pixels, use the vertical coordinate of the lower boundary of the target boundary area as the vertical coordinate of the positioning point. Otherwise, add the vertical coordinates of the upper and lower boundaries of the target boundary area and divide by 2 to obtain the vertical coordinate of the positioning point. When the dynamic target projection center is located within the target boundary area, add the horizontal coordinate of the positioning point to the horizontal coordinate of the dynamic target projection center and divide by 2 for correction to obtain the target visual positioning point. The target map candidate points are obtained as follows: Read the target visual positioning point, point cloud structure projection constraint information, real-time pose of the fire-fighting robot, gimbal attitude, and robot chassis pose data. Search for point cloud projection positions less than 5 pixels away from the target visual positioning point in the point cloud structure projection constraint information. If there is one point cloud projection position, take the corresponding point cloud coordinates as the target candidate point cloud coordinates. If there are two or more point cloud projection positions, take the point cloud coordinates with the smallest depth value as the target candidate point cloud coordinates. If there is no point cloud projection position, take the point cloud coordinates corresponding to the dynamic target projection center as the target candidate point cloud coordinates. Based on the real-time pose of the fire-fighting robot, gimbal attitude, and robot chassis pose data, convert the target candidate point cloud coordinates to the local point cloud map coordinate system of the fire scene to obtain the target map candidate points. Based on the spatial correspondence between candidate points on the target map and the static structure layer and dynamic target layer, corner support association, boundary anchoring association, and dynamic center association are generated, and a visual corner-point cloud structure anchoring map is constructed, wherein: The spatial correspondence is performed as follows: Read the target map candidate points, static structure layer, and dynamic target layer. Calculate the vertical distance from the target map candidate points to the ground support point cloud in the static structure layer. If the distance is less than 0.20 meters, establish corner support association. Calculate the horizontal distance from the target map candidate points to the vertical boundary point cloud in the static structure layer. If the distance is less than 0.30 meters, establish boundary anchoring association. Calculate the 3D distance from the target map candidate points to the point cloud cluster center in the dynamic target layer. If the distance is less than 0.50 meters, establish dynamic center association. Constructing a visual corner point-point cloud structure anchoring map, specifically: The target visual corner point, target map candidate point, ground support point cloud, vertical boundary point cloud and dynamic target layer point cloud cluster center are used as anchoring objects. The corner support association, boundary anchoring association and dynamic center association are written between the corresponding anchoring objects. The visual corner point-point cloud structure anchoring map is composed of the relationship between the anchoring objects and the objects.
[0028] In this embodiment, generating the firefighting robot operation control data includes: Anchoring verification is performed based on the visual corner point-point cloud structure anchoring map, including corner point support association, boundary anchoring association, and dynamic center association. Candidate target points with inconsistent structures are removed, resulting in valid target anchoring points. The anchorage verification of corner support association, boundary anchorage association, and dynamic center association is carried out as follows: Read the visual corner point-point cloud structure anchoring map. When the candidate point of the target map has corner support association, boundary anchoring association and dynamic center association at the same time, calculate the three association distances. If the corner support association distance is less than 0.20 meters, the boundary anchoring association distance is less than 0.30 meters, and the dynamic center association distance is less than 0.50 meters, the anchoring check is deemed to pass. If any distance does not meet the corresponding threshold, the structure is deemed to be inconsistent. The effective target anchor points are obtained as follows: Read the anchoring verification results and target map candidate points, delete target map candidate points with inconsistent structures, retain target map candidate points that pass the anchoring verification, and write the corresponding corner point category, corner point direction, corner point comprehensive confidence and target source mark to obtain the effective target anchoring points; Spatial positioning calculations are performed based on effective target anchor points, the real-time pose of the firefighting robot, and a local point cloud map of the fire scene to obtain the target map coordinates, target distance, target orientation, and target source marker, among which: The calculation for spatial positioning is as follows: Read the effective target anchor points, the real-time pose of the fire robot, and the local point cloud map of the fire scene. Add the three-dimensional coordinates of the effective target anchor points corresponding to the same target and divide by the number of anchor points to obtain the target map coordinates. Subtract the fire robot position coordinates from the target map coordinates, sum the squares of the differences in the three axes, and take the square root to obtain the target distance. Convert the horizontal and vertical differences between the target map coordinates and the fire robot position coordinates into horizontal angles, and then subtract the fire robot heading angle to obtain the target bearing. Read the target source marker in the effective target anchor points to obtain the target map coordinates, target distance, target bearing, and target source marker. Based on the collected timestamp data, the target map coordinates, target distance, target orientation, and target source markers from consecutive collection times are correlated and stabilized. Short-term abrupt changes in positioning results are removed, resulting in stabilized map-level multi-target positioning results. This data is then used to generate fire robot operation control data, including: The correlation stabilization process is carried out as follows: Read the target map coordinates, target distance, target azimuth, target source marker, and collection timestamp data at consecutive collection times. For adjacent collection times, the target map coordinate distance is less than 0.50 meters, the target azimuth difference is less than 15 degrees, and the target source marker is the same, and the positioning results are associated as the same target. If the same target has positioning results in 3 consecutive collection times, it is retained. If the target distance changes by more than 1.00 meter or the target azimuth changes by more than 30 degrees between two adjacent collection times, the corresponding positioning results are regarded as short-term mutation positioning results and are removed. The retained positioning results are arranged in the order of collection timestamps to obtain the stable map-level multi-target positioning results. Generate the fire-fighting robot operation control data, specifically: After reading the stabilized map-level multi-target localization results, deceleration control data is generated when the target distance is less than 1.00 meters. When the target is located to the left of the fire robot and the absolute azimuth angle is greater than 20 degrees, left turn avoidance control data is generated. When the target is located to the right of the fire robot and the absolute azimuth angle is greater than 20 degrees, right turn avoidance control data is generated. When the target source is marked as an infrared thermal boundary and the target distance is less than 3.00 meters, heat source approach warning data is generated. The fire robot operation control data consists of deceleration control data, turning avoidance control data, and heat source approach warning data.
[0029] refer to Figure 3 A multi-target recognition and positioning system for firefighting robots includes the following modules: The data preprocessing module is used to collect and preprocess multi-source fire scene sensing data to generate standardized fire scene sensing data. The point cloud mapping module is used to execute the LeGO-LOAM algorithm to extract features and layer the LiDAR point cloud, and generate the real-time pose of the fire-fighting robot, the local point cloud map of the fire scene, the static structure layer and the dynamic target layer. The structural prior module is used to extract ground supports, vertical boundaries, occlusion structures, and point cloud cluster centers to generate fire scene point cloud structural priors. The projection enhancement module is used to perform back projection and visual enhancement to generate point cloud structure projection constraint information and enhanced visual images of the fire scene; The corner detection module is used to build an improved CornerNet model, perform corner detection, category recognition, and structural constraints, and generate a set of target visual corners; The anchoring map module is used to generate the target boundary region, extract and map the target visual positioning points, and construct a visual corner point-point cloud structure anchoring map; The positioning control module is used to perform anchoring verification, spatial positioning and continuous stabilization, and generate map-level multi-target positioning results and fire robot operation control data.
[0030] Example 1: To verify the feasibility of this invention in practice, it was applied to a simulated operation scenario in a closed indoor fire training space. The space included an entrance passage, a corner corridor, the exterior of an equipment room, and a storage area, and was equipped with walls, door frames, metal cabinets, scattered boxes, movable obstructions, and a heat source simulation device. During the simulation, the smoke concentration normalization value increased from 0.38 to 0.76, the infrared thermal area temperature remained between 86°C and 296°C, visible light images showed gray-white obstruction and blurred boundaries, and the lidar point cloud showed weakened echoes near the heat source and dense smoke. During a continuous 180-second operation, the fire robot collected 2700 frames of visible light images, 2700 frames of infrared thermal images, 1800 frames of lidar point cloud data, 10800 sets of inertial measurement data, 5400 sets of wheel speed and mileage data, and 1800 sets of smoke concentration data. After preprocessing, approximately 116,000 point clouds with distances less than 0.30 meters and greater than 30 meters were deleted, 38 image frames with timestamp deviations greater than 20 milliseconds were removed, and 27 sets of missing wheel speed and mileage sampling values were supplemented to obtain standardized fire scene perception data.
[0031] In contrast, the basic technology uses conventional LiDAR simultaneous localization and mapping methods to build a local point cloud map, and employs ordinary image target detection methods to directly identify target boundaries from visible light images. It then estimates the target map coordinates based on the nearest point cloud projection position near the target boundary center. The basic technology only performs image target detection and nearest neighbor matching of point cloud positions; it does not utilize the fire scene point cloud structure as a priori to correct the visual image, does not generate corner constraint weight maps, and does not perform anchoring checks between visual corner points and the point cloud structure. In the same simulated scenario, when the smoke concentration normalization value reaches 0.70 or higher, the basic technology easily identifies the door frame edge, the edge of bright smoke spots, and the edge of heat sources as the same target boundary, leading to target boundary offset and the selection of incorrect point cloud positions during map localization.
[0032] In the operation of the method of this invention, the fire-fighting robot performs point cloud mapping processing after entering the entrance passage. Each frame of the original point cloud contains an average of about 62,300 points. After distance filtering, reflection intensity normalization, and smoke degradation weighting, an average of about 48,700 effective points are retained, and about 3,120 edge feature points and about 8,560 planar feature points are extracted. The average registration residual of consecutive frames is 0.047 meters, while the average registration residual of consecutive frames without smoke degradation weighting in the basic technology is 0.083 meters. After the fire-fighting robot passes through the corner corridor, the method of this invention forms a local point cloud map of the fire scene, writing the walls, door frames, and ground into the static structure layer, and the mobile heat source device and the pushed box into the dynamic target layer. The cumulative pose offset of the method of this invention is 0.19 meters, while the cumulative pose offset of the basic technology is 0.46 meters.
[0033] As the robot approaches the outside of the equipment room, the edge of the front enclosure is obscured by smoke in the visible light image, while a high-temperature area appears behind the enclosure in the infrared thermal image. The system extracts approximately 12,400 ground support point clouds and approximately 3,760 vertical boundary point clouds from the static structure layer, and identifies two occlusion structures. It also extracts three point cloud cluster centers from the dynamic target layer, located approximately 3.4 meters, 4.1 meters, and 5.2 meters in front of the robot, respectively. The system projects the ground support structure, vertical boundary structure, occlusion structure, and dynamic target projection centers back into the image, resulting in one ground support projection area, seven vertical boundary projection lines, two occlusion projection areas, and three dynamic target projection centers, generating an enhanced visual image of the fire scene. The basic technology generates five target boundaries in this area, two of which are incorrect boundaries; the method of this invention generates four target boundaries, with only one being deleted in subsequent anchoring verification.
[0034] The improved CornerNet model uses 12,000 frames of simulated samples during the training phase, with 8,400 frames used for training, 1,800 frames for validation, and 1,800 frames for testing. The samples cover light smoke, moderate smoke, heavy smoke, and strong infrared interference scenes. Annotations include the top-left corner, bottom-right corner, corner source, corner pairing relationships, point cloud projection position, ground support projection area, vertical boundary projection line, and dynamic target projection center. When training stopped, the corner detection error decreased from 0.392 to 0.071, the corner source identification error decreased from 0.284 to 0.058, the corner structure consistency error decreased from 0.317 to 0.066, the corner pairing error decreased from 0.269 to 0.052, and the joint error decreased from 0.346 to 0.066.
[0035] In actual simulation recognition, the method of this invention detected only one upper left corner point at the edge of an occluded projection area, while the lower right corner point was missing. The system searched for the pixel position with the largest correction response value along the nearest vertical boundary projection line. The maximum correction response value was 0.53, which is greater than 0.45, so the corresponding lower right corner point was added. The pixel coordinates of the upper left corner point of a certain box-shaped target are (318, 214), and the pixel coordinates of the lower right corner point are (462, 386). The comprehensive confidence scores of the corner points are 0.86 and 0.82, respectively. The horizontal distance is 144 pixels, and the vertical distance is 172 pixels. The system retains this corner point pair and generates the target boundary area. The system combines the ground support projection area and the dynamic target projection center to obtain the target visual positioning point (390, 386). Within a 5-pixel range of this positioning point, three point cloud projection positions are found, with corresponding depth values of 3.62 meters, 3.74 meters, and 4.18 meters, respectively. The system selects the point cloud coordinates with the smallest depth value and converts them to the local point cloud map of the fire scene to obtain the target map candidate point (3.51, 1.08, 0.42).
[0036] The system then spatially correlates the target map candidate points with the static structure layer and the dynamic target layer. The vertical distance from the aforementioned target map candidate point to the ground support point cloud is 0.13 meters, the horizontal distance to the vertical boundary point cloud is 0.24 meters, and the 3D distance to the cluster center of the dynamic target layer point cloud is 0.37 meters. All of these meet the anchoring check thresholds of 0.20 meters, 0.30 meters, and 0.50 meters, respectively, and are therefore retained as valid target anchoring points. Another candidate point formed by a bright smoke spot has coordinates (4.22, 1.91, 1.37). Its vertical distance to the ground support point cloud is 0.68 meters, and its 3D distance to the cluster center of the dynamic target layer point cloud is 0.92 meters. The system determines that the structure is inconsistent and deletes it. The underlying technology retains this erroneous candidate point in the same segment, causing the target distance to abruptly change from 4.21 meters to 2.96 meters in consecutive frames.
[0037] After continuous stabilization processing, the method of this invention outputs three stable target positioning results. The first target has map coordinates of (3.51, 1.08, 0.42), a target distance of 3.68 meters, a target orientation of 18 degrees to the right, and its source is marked as a visible light outline. The second target has map coordinates of (4.06, 0.74, 0.88), a target distance of 4.13 meters, a target orientation of 11 degrees to the right, and its source is marked as an infrared thermal boundary. The third target has map coordinates of (2.74, -0.62, 0.36), a target distance of 2.81 meters, a target orientation of 24 degrees to the left, and its source is marked as a target boundary texture. As the robot continues to approach the storage stacking area, the system detects that the target distance to the left has decreased to 0.92 meters and the orientation is 23 degrees to the left, generating deceleration control data and right-turn avoidance control data. When the target distance from the infrared thermal boundary source decreases to 2.64 meters, the system generates heat source approach warning data.
[0038] On the same test sample set, the basic technical target recognition accuracy was 78.7%, the corner point matching accuracy was 74.2%, the average map positioning error was 0.47 meters, the maximum positioning error was 1.36 meters, and short-term abrupt positioning results occurred 42 times. The method of this invention achieved a target recognition accuracy of 93.6%, a corner point matching accuracy of 91.8%, an average map positioning error of 0.19 meters, a maximum positioning error of 0.54 meters, and the number of short-term abrupt positioning results was reduced to 9. In samples with a smoke concentration normalization value greater than 0.70 and an obscuration ratio greater than 40%, the basic technical target recognition accuracy dropped to 68.7%, while the method of this invention still achieved 89.5%. The above simulation results demonstrate that this invention can improve the multi-target recognition accuracy of firefighting robots, reduce map-level positioning errors, and enhance the reliability of operational control data under conditions of dense smoke obscuration, thermal radiation interference, and partial point cloud defects.
[0039] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A multi-target identification and localization method for firefighting robots, characterized in that, include: Collect multi-source sensing data of the fire scene during the operation of the fire-fighting robot, and preprocess the multi-source sensing data of the fire scene to generate standardized fire scene sensing data; Based on standardized fire scene perception data, the LeGO-LOAM algorithm is executed to extract point cloud features and layer the point cloud data of lidar point cloud data, thereby obtaining the real-time pose of the fire-fighting robot, the local point cloud map of the fire scene, the static structure layer and the dynamic target layer. Ground support extraction, vertical boundary extraction and occlusion structure analysis are performed based on the static structure layer, and point cloud cluster center extraction and spatial location association are performed based on the dynamic target layer to obtain the fire scene point cloud structure prior. Based on the prior knowledge of the fire point cloud structure and standardized fire perception data, back projection processing is performed to obtain the point cloud structure projection constraint information, and fire visual enhancement processing is performed to generate fire enhanced visual images. An improved CornerNet model was constructed to perform corner detection and category recognition on enhanced visual images of fire scenes. Structural constraint processing was performed by combining point cloud structure projection constraint information to obtain the target visual corner set. Based on the target visual corner point set, corner points of the same category are paired and target boundaries are generated to obtain the target boundary area. The target visual positioning points are extracted and mapped to the local point cloud map of the fire scene. They are then associated with the static structure layer and the dynamic target layer to construct a visual corner point-point cloud structure anchoring map. Based on the visual corner point-point cloud structure anchoring map, multi-relationship anchoring verification, spatial positioning calculation and continuous stabilization processing are performed to obtain the stabilized map-level multi-target positioning results and generate fire robot operation control data.
2. The multi-target identification and positioning method for firefighting robots according to claim 1, characterized in that, The multi-source sensing data of the fire scene specifically includes visible light image data, infrared thermal imaging data, lidar point cloud data, lidar reflection intensity data, inertial measurement data, wheel speed and mileage data, gimbal attitude data, robot chassis pose data, smoke concentration data, and acquisition timestamp data.
3. The multi-target identification and positioning method for firefighting robots according to claim 1, characterized in that, The preprocessing of multi-source sensing data at the fire scene specifically includes image distortion correction, infrared registration, time synchronization, point cloud cleaning, pose correction, data standardization, and multi-source data association processing.
4. The multi-target identification and positioning method for firefighting robots according to claim 1, characterized in that, The obtained real-time pose of the firefighting robot, local point cloud map of the fire scene, static structure layer, and dynamic target layer include: Based on standardized fire scene perception data, the LeGO-LOAM algorithm is executed to perform range filtering, time alignment and reflection intensity normalization on lidar point cloud data. Based on reflection intensity fluctuation, point cloud density change, echo persistence and smoke concentration, a fire scene smoke degradation field is generated to obtain a degradation weighted effective point cloud. Based on the local curvature of the degradation weighted effective point cloud, edge feature points and planar feature points are extracted by combining the degradation field of fire smoke and dust. The residual weights are assigned according to the degradation intensity of smoke and dust and the persistence of echoes to obtain the weighted point cloud feature set and the point cloud registration residual set. Based on the weighted point cloud feature set, point cloud registration residual set, inertial measurement data and wheel speed mileage data, LeGO-LOAM continuous frame registration is performed. Starting from the second acquisition time, visual corner point anchorage residual is generated by combining the visual corner point-point cloud structure anchorage map of the previous acquisition time, and joint pose optimization and accumulation are performed to obtain the real-time pose of the fire-fighting robot. A local point cloud map of the fire scene is constructed based on the real-time pose of the firefighting robot and the weighted point cloud feature set. The update status is determined based on the point cloud registration residual, the point cloud density change and the fire smoke degradation field. Stable areas are written into the local point cloud map of the fire scene, degraded areas are frozen, and point clouds in the degraded areas that meet the dynamic change conditions are temporarily stored as a dynamic candidate point cloud set to obtain the updated local point cloud map of the fire scene. Based on the real-time pose of the firefighting robot, the gimbal attitude, the updated local point cloud map of the fire scene, and the dynamic candidate point cloud set, visual observation rays are generated. The visual observation rays are cross-judged with the fire scene structure to generate an occlusion ray topology map. The point cloud is then layered according to the spatial changes of the point cloud, the changes in the outer range of the point cloud, the registration residual of the point cloud, and the occlusion ray topology map to obtain a static structure layer and a dynamic target layer.
5. A multi-target identification and positioning method for firefighting robots according to claim 1, characterized in that, The prior knowledge of the fire point cloud structure includes: Ground support point cloud is extracted based on point cloud height, normal direction and spatial stability in static structural layer, and support surface boundary is fitted to obtain ground support structure; Vertical boundary point cloud is extracted based on edge feature points, planar feature points and reflection intensity changes in the static structure layer, and occlusion structure analysis is performed by combining the real-time pose of the fire-fighting robot and the gimbal attitude to obtain the vertical boundary structure and occlusion structure. The point cloud cluster centers are extracted based on the dynamic target layer, and the spatial location of the point cloud cluster centers is associated with the ground support structure, vertical boundary structure and occlusion structure to obtain the fire scene point cloud structure prior.
6. A multi-target identification and positioning method for firefighting robots according to claim 1, characterized in that, The generation of enhanced visual images of the fire scene includes: Based on the prior knowledge of the fire scene point cloud structure, the real-time pose of the fire-fighting robot, the gimbal attitude data and the robot chassis pose data, a point cloud visual mapping relationship is established between the local point cloud map of the fire scene and the visible light image and infrared thermal imaging. Based on the visual mapping relationship of point cloud, the ground support structure, vertical boundary structure, occlusion structure and point cloud cluster center are back-projected to obtain the point cloud structure projection constraint information. Based on point cloud structure projection constraint information, visible light images, infrared thermal imaging, and smoke concentration data, boundary preservation, contour enhancement, and thermal feature fusion are performed to generate enhanced visual images of the fire scene.
7. A multi-target identification and positioning method for firefighting robots according to claim 1, characterized in that, The obtained target visual corner point set includes: An improved CornerNet model is constructed, which consists of a visual feature extraction module, a point cloud structure constraint module, a corner detection and recognition module, and a corner matching output module. The visual feature extraction module performs multi-scale feature extraction and hierarchical feature fusion on the enhanced visual image of the fire scene, extracting infrared thermal boundary features, visible light contour features and target boundary texture features to obtain a set of visual features of the fire scene. The point cloud structure constraint module encodes the constraint features of the point cloud structure projection constraint information, generates a corner constraint weight map based on the vertical boundary projection line, the occlusion projection area, the ground support projection area and the dynamic target projection center, and fuses it with the fire scene visual feature set to obtain the structural constraint visual feature set. The corner detection and recognition module performs response prediction of the top left corner key point, response prediction of the bottom right corner key point, and target category prediction on the set of structural constraint visual features, respectively, to obtain the response map of the top left corner key point, the response map of the bottom right corner key point, and the target category recognition result; The response maps of the top left and bottom right key points are corrected based on the corner constraint weight map. Missing corner points are filled in based on the occlusion projection area, vertical boundary projection line and the detected corner point position. The response of candidate corner points is resampled along the point cloud projection boundary direction to obtain the candidate target corner point set. The corner matching output module performs same-category corner matching on the candidate target corner set and the target category recognition result. It performs ternary consistency verification based on the candidate corner, the dynamic target projection center and the ground support projection area. It performs categorization correction based on the conflict area between the infrared thermal boundary and the vertical boundary of the point cloud. It also performs corner offset correction and confidence screening to obtain the target visual corner set. The improved CornerNet model was trained using corner detection error, target category recognition error, corner structure consistency error, and corner matching error as joint optimization objectives. The parameters of the visual feature extraction module, point cloud structure constraint module, corner detection and recognition module, and corner matching output module were continuously optimized. When the relative decrease rate of the joint error in 5 consecutive training rounds was less than 0.001, or when the number of training rounds reached 200, the improved CornerNet model was deemed to have converged and training was stopped.
8. A multi-target identification and positioning method for firefighting robots according to claim 1, characterized in that, The construction of the visual corner point-point cloud structure anchoring map includes: Based on the corner point category, corner point direction, and corner point confidence of the target visual corner point set, corner points of the same category are paired to generate the target boundary region; Based on the target boundary area, ground support projection area and dynamic target projection center, the target visual positioning points are extracted, and combined with the point cloud structure projection constraint information, the real-time pose of the fire robot, the gimbal attitude and the robot chassis pose data, they are mapped to the local point cloud map of the fire scene to obtain target map candidate points. Based on the spatial correspondence between the candidate points of the target map and the static structure layer and dynamic target layer, corner support association, boundary anchoring association and dynamic center association are generated, and a visual corner point-point cloud structure anchoring map is constructed.
9. A multi-target identification and positioning method for firefighting robots according to claim 1, characterized in that, The generated fire-fighting robot operation control data includes: Anchoring verification is performed based on the visual corner point-point cloud structure anchoring map, including corner point support association, boundary anchoring association, and dynamic center association. Candidate target map points with inconsistent structures are eliminated to obtain effective target anchoring points. Spatial positioning calculations are performed based on effective target anchor points, real-time pose of the firefighting robot, and local point cloud map of the fire scene to obtain target map coordinates, target distance, target orientation, and target source marker; Based on the collected timestamp data, the target map coordinates, target distance, target orientation and target source markers in the continuous collection time are correlated and stabilized. Short-term sudden change in positioning results are removed to obtain stable map-level multi-target positioning results, and fire robot operation control data is generated.
10. A multi-target recognition and positioning system for firefighting robots, comprising the multi-target recognition and positioning method for firefighting robots as described in any one of claims 1 to 9, characterized in that, Includes the following modules: The data preprocessing module is used to collect and preprocess multi-source fire scene sensing data to generate standardized fire scene sensing data. The point cloud mapping module is used to execute the LeGO-LOAM algorithm to extract features and layer the LiDAR point cloud, and generate the real-time pose of the fire-fighting robot, the local point cloud map of the fire scene, the static structure layer and the dynamic target layer. The structural prior module is used to extract ground supports, vertical boundaries, occlusion structures, and point cloud cluster centers to generate fire scene point cloud structural priors. The projection enhancement module is used to perform back projection and visual enhancement to generate point cloud structure projection constraint information and enhanced visual images of the fire scene; The corner detection module is used to build an improved CornerNet model, perform corner detection, category recognition, and structural constraints, and generate a set of target visual corners; The anchoring map module is used to generate the target boundary region, extract and map the target visual positioning points, and construct a visual corner point-point cloud structure anchoring map; The positioning control module is used to perform anchoring verification, spatial positioning and continuous stabilization, and generate map-level multi-target positioning results and fire robot operation control data.
