Fire-fighting robot dog fire source positioning system based on image detection

Abstract

This invention relates to the field of image recognition and fixed-scene intelligent inspection technology, and discloses a fire source localization system for a fire-fighting robot dog based on image detection. The system includes: acquiring a fire scene constraint map bound to preset inspection points; acquiring visible light images and infrared thermal imaging images, and performing artificial intelligence image recognition to obtain initial recognition results; mapping the initial recognition results to the fire scene constraint map to generate scene conflict events; filtering observable verification poses based on scene conflict events and generating a verification pose sequence; controlling the fire-fighting robot dog to perform multi-pose verification image acquisition and image recognition, generating cross-pose consistency results, and generating inspection anomaly records and inspection handling instructions accordingly.

Description

Technical Field

[0001] This invention relates to the field of image recognition and intelligent inspection technology for fixed scenes, and particularly to a fire source localization system for a fire-fighting robot dog based on image detection. Specifically, it is applicable to a fire-fighting robot dog acquiring visible light images and infrared thermal images in fixed fire inspection scenes, and performing image recognition, scene constraint verification, and multi-pose verification of the status of fire-fighting objects, thermal anomaly areas, fixed thermal interference areas, fixed reflective areas, and occlusion boundary areas. The fire-fighting robot dog described in this invention is essentially a four-legged fire-fighting inspection robot equipped with image acquisition, infrared thermal imaging, pose detection, motion control, and computing devices. Background Technology

[0002] In fire inspection scenarios, fire-fighting robots or fire-fighting robot dogs can automatically detect the inspection area using visible light images, infrared thermal images, and environmental detection data, and use image recognition to identify flames, high-temperature areas, smoke, or other abnormal targets. This type of technology is typically applicable to fixed or semi-fixed fire inspection scenarios such as factories, warehouses, power distribution areas, underground spaces, and building corridors.

[0003] In existing technologies, firefighting robots are typically equipped with components such as visible light image detection modules, infrared thermal imaging detection modules, laser rangefinders, and rotating gimbals. They are controlled by a built-in motion control system to move along a predetermined patrol route. During the patrol process, the firefighting robot collects visible light detection data, infrared thermal imaging detection data, and ranging data, and then combines this with image recognition or automatic tracking methods to identify and locate abnormally high-temperature points or early-stage fires. For example, CN115591165A discloses a system and method for identifying very early-stage fires in a firefighting robot. The identification system includes a visible light image detection module, a laser rangefinder, an infrared thermal imaging detection module, a supplementary light, a rotating gimbal, and a controller. This identification system is installed on the robot body and linked with the fire hose. The identification method, based on visible light detection data, infrared thermal imaging detection data, laser ranging detection data, and automatic tracking technology, identifies the temperature and location of abnormally high-temperature points and very early-stage open flames before ignition.

[0004] The aforementioned technologies can automatically identify fire targets or high-temperature hazard points during fire robot inspections, and can adjust the orientation of photoelectric detection equipment by rotating the pan-tilt unit, aligning the visible light image detection module, infrared thermal imaging detection module, or laser rangefinder with abnormal locations. However, in fixed fire inspection scenarios, the area to be inspected typically contains fire extinguishers, fire hydrants, fire doors, evacuation signs, electrical cabinets, smoke detectors, sprinkler heads, and evacuation routes; simultaneously, there may be fixed heat sources, fixed reflective objects, permanent obstructions, and localized obstruction boundaries formed by equipment placement. These factors can affect the identification results of target areas, thermal anomaly areas, or obstructed areas in visible light and infrared thermal images.

[0005] Especially when fire-fighting robot dogs perform fixed-point inspections, image acquisition results are affected by factors such as the robot's height, orientation, pan-tilt angle, and parking position. If anomaly detection is based solely on visible light or infrared thermal images from a single perspective, fixed heat sources and their associated thermal interference areas may affect the identification of thermal anomaly areas; fixed reflective objects and their associated reflective areas may affect visible light image recognition results; and permanent obstructions and their associated boundary areas may lead to errors in identifying missing or abnormally positioned fire-fighting targets. While existing technologies can perform image recognition, thermal imaging recognition, and pan-tilt alignment of fire targets, it remains difficult to specifically verify and confirm anomaly detection results caused by fixed thermal interference areas, fixed reflective areas, and obstruction boundary areas in fixed fire-fighting inspection scenarios, taking into account the characteristics of the fixed inspection scenario.

[0006] Therefore, how to perform scenario-based verification and confirmation of abnormal recognition results caused by fixed thermal interference areas, fixed reflective areas, and occlusion boundary areas during the image recognition process of fire robot dogs inspecting fixed locations is a technical problem that needs to be solved in this field. Summary of the Invention

[0007] The technical problem this invention aims to solve is: how to perform scene-based verification and confirmation of abnormal recognition results caused by fixed thermal interference areas, fixed reflective areas, and occlusion boundary areas during image recognition of fire-fighting robot dogs at fixed locations. To solve this problem, this invention provides a fire source localization system for fire-fighting robot dogs based on image detection, comprising: acquiring a fire scene constraint map bound to a preset inspection point, wherein the fire scene constraint map includes a reference scene image, a fire-fighting object area, an object reference state, fixed thermal interference areas, fixed reflective areas, occlusion boundary areas, a map version number, a set of candidate verification poses, and a mapping relationship between image coordinates and robot dog poses; controlling the fire-fighting robot dog to collect visible light images and infrared thermal imaging images of a fixed fire inspection scene at the preset inspection point, and acquiring the current pose data of the fire-fighting robot dog; and performing artificial intelligence image recognition on the visible light images and the infrared thermal imaging images. The initial recognition result is obtained; the initial recognition result is mapped to the fire scene constraint map, and scene conflict events are generated based on the fire object area, the object reference state, the fixed thermal interference area, the fixed reflective area, and the occlusion boundary area; based on the scene conflict events, at least two observable verification poses are selected from the candidate verification pose set, and a verification pose sequence is generated according to the mapping relationship between the image coordinates and the robot dog pose; the fire robot dog is controlled to perform multi-pose verification image acquisition and image recognition according to the verification pose sequence to generate cross-pose consistency results; and an inspection anomaly record and inspection handling instructions are generated based on the cross-pose consistency results.

[0008] Through the above technical solution, the present invention establishes a fire scene constraint map bound to a preset inspection point, maps the initial recognition result obtained by image recognition to the fire scene constraint map to generate scene conflict events, and filters the verification pose sequence based on the scene conflict events to perform multi-pose verification and cross-pose consistency confirmation, thereby solving the problem that abnormal recognition results caused by fixed thermal interference area, fixed reflective area and occlusion boundary area are difficult to verify and confirm in a scene-based manner.

[0009] This invention discloses a fire source localization system for a fire-fighting robot dog based on image detection, comprising:

[0010] Obtain the fire scene constraint map bound to the preset inspection points. The fire scene constraint map includes the reference scene image, fire object area, object reference state, fixed thermal interference area, fixed reflective area, occlusion boundary area, map version number, candidate verification pose set, and the mapping relationship from image coordinates to robot dog pose.

[0011] The fire-fighting robot dog is controlled to collect visible light and infrared thermal images of a fixed fire inspection scene at a preset inspection point, and the current pose data of the fire-fighting robot dog is obtained.

[0012] Artificial intelligence image recognition is performed on visible light images and infrared thermal imaging images to obtain initial recognition results, which include object category, object region, object state, thermal anomaly region, and occlusion relationship.

[0013] The initial recognition results are mapped to the fire scene constraint map, and scene conflict events are generated based on the fire object area, object baseline state, fixed thermal interference area, fixed reflective area and occlusion boundary area.

[0014] Based on scene conflict events, at least two observable core poses are selected from the candidate core pose set, and a core pose sequence is generated according to the mapping relationship between image coordinates and robot dog pose.

[0015] The fire-fighting robot dog is controlled to perform multi-pose verification image acquisition and image recognition according to the verification pose sequence, and generate cross-pose consistency results.

[0016] Based on the cross-pose consistency results, generate inspection anomaly records and inspection handling instructions.

[0017] Preferably, the fire scene constraint diagram is established in the following way:

[0018] At the preset inspection points, multiple sets of reference scene images are collected at at least two fuselage heights and at least two gimbal pitch angles;

[0019] Mark fire inspection targets, fixed heat sources, fixed reflective objects, permanent obstructions, and passable areas in multiple sets of baseline scene images;

[0020] Write the image area corresponding to the fire inspection object into the fire object area;

[0021] Write the normal status of the fire inspection object into the object's baseline status;

[0022] Write the image region corresponding to the fixed heat source into the fixed thermal interference region.

[0023] Write the image area corresponding to the fixed reflective object into the fixed reflective area;

[0024] Write the image boundary corresponding to the permanent occlusion into the occlusion boundary area.

[0025] The correspondence between the passable area, the aircraft height, the gimbal pitch angle, and the reference scene image is written into the candidate verification pose set.

[0026] Preferably,

[0027] Each fire inspection target corresponds to at least two candidate verification positions;

[0028] At least two candidate verification positions corresponding to the same fire inspection object have different fuselage heights, different gimbal pitch angles, or different verification parking positions;

[0029] The candidate verification pose record shows the unobstructed visible area, image area coordinates, fuselage orientation, and gimbal yaw angle of the fire inspection object in the corresponding reference scene image.

[0030] Preferably,

[0031] Before acquiring visible light images and infrared thermal imaging images, control the fire-fighting robot dog to acquire the current parking image and current pose data;

[0032] The current parking image is registered with the reference scene image in the fire scene constraint diagram, and the current pose data is matched with the candidate verification pose in the candidate verification pose set.

[0033] When the image area corresponding to the fire-fighting object area in the current paused image does not fall within the preset pixel range, or the current pose data does not fall within the preset pose range, a pause compensation action is generated.

[0034] Parking compensation actions include at least one of the following: fuselage yaw adjustment, lateral step adjustment, forward and backward step adjustment, fuselage altitude adjustment, and gimbal pitch adjustment.

[0035] Preferably,

[0036] Artificial intelligence image recognition is performed using an artificial intelligence image recognition model;

[0037] The artificial intelligence image recognition model includes branches for fire-fighting object recognition, status recognition, thermal anomaly recognition, and occlusion boundary recognition.

[0038] The fire object identification branch outputs the object category and object area;

[0039] The state recognition branch outputs the object state, which includes at least one of the following: in-place state, missing state, open state, closed state, occluded state, channel occupied state, and thermal anomaly-accompanied state.

[0040] The thermal anomaly identification branch outputs the thermal anomaly region.

[0041] Occlusion boundary recognition branches output occlusion relationships.

[0042] Preferably,

[0043] The generated scene conflict events include:

[0044] When an object area does not fall into the corresponding fire-fighting object area, an area mismatch conflict event is generated;

[0045] When the state of an object is inconsistent with the corresponding baseline state of the object, a state deviation conflict event is generated.

[0046] A thermal anomaly conflict event is generated when the thermal anomaly region is outside the fixed thermal interference region and the object is in a thermal anomaly-accompanying state.

[0047] When the object area overlaps with the fixed reflective area and the object is in a state accompanied by thermal anomaly, a reflective interference conflict event is generated.

[0048] When the fire-fighting object area overlaps with the occlusion boundary area and no corresponding object category is identified, an occlusion missing conflict event is generated.

[0049] Among them, scene conflict events include conflict category, conflict area, associated object category, associated object status, and associated graph version number.

[0050] Preferably,

[0051] The generation of the verification pose sequence includes:

[0052] Select the initial verification pose corresponding to the conflict area from the candidate verification pose set;

[0053] Remove the initial verification pose that overlaps with the imaging field of view and the fixed thermal interference area;

[0054] Remove the initial verification pose that overlaps with the imaging field of view and the fixed reflective area;

[0055] Remove the initial verification poses that are obscured by the boundary areas of the conflict area;

[0056] The remaining initial verification poses are determined as observable verification poses;

[0057] Calculate the action switching cost of each observable core pose relative to the current pose data, and arrange at least two observable core poses in ascending order of action switching cost to generate a core pose sequence.

[0058] Preferably,

[0059] The verification pose sequence includes the first observable verification pose and the second observable verification pose;

[0060] The first observable verification pose and the second observable verification pose have different fuselage heights, different gimbal pitch angles, or different verification parking positions;

[0061] The fire-fighting robot dog acquires a first-core visible light image, a first-core infrared image, a second-core visible light image, and a second-core infrared image in the first observable core pose and the second observable core pose, respectively.

[0062] The first verification visible light image, the first verification infrared image, the second verification visible light image, and the second verification infrared image are respectively mapped to the fire scene constraint map to generate cross-pose consistency results.

[0063] Preferably,

[0064] Generating cross-pose consistency results includes:

[0065] When the first observable verification pose and the second observable verification pose identify the same object category and the same object state for the same conflict region, and the conflict region does not fall into the fixed thermal interference region and the fixed reflective region, an anomaly confirmation result is generated.

[0066] When the first observable verification pose and the second observable verification pose identify different object categories or different object states for the same conflict region, a verification inconsistency result is generated.

[0067] When the conflict area falls into a fixed thermal interference area or a fixed reflective area, an interference elimination result is generated.

[0068] Preferably,

[0069] When generating inconsistent verification results, a supplementary verification pose is called from the candidate verification pose set;

[0070] The supplementary verification pose has different fuselage height, different gimbal pitch angle, or different verification parking position compared to the first and second observable verification poses.

[0071] The fire-fighting robot dog is controlled to acquire supplementary and verification visible light images and supplementary and verification infrared images in a positional verification context.

[0072] The cross-pose consistency results are updated based on supplementary verified visible light images and supplementary verified infrared images.

[0073] Preferably,

[0074] An inspection cycle is defined as the completion of one inspection of a preset inspection point.

[0075] When generating interference elimination results, the conflict region, conflict category, correlation map version number, first observable verification pose and second observable verification pose are written into the interference observation record.

[0076] When the same conflict area generates interference elimination results in two consecutive inspection cycles, a map update request is generated.

[0077] Map update requests include the region to be updated, the type of the region to be updated, the trigger inspection cycle, and the version number of the map to be updated.

[0078] Preferably,

[0079] Update the fire scene constraint graph based on the graph update request;

[0080] When the type of the area to be updated is thermal interference, the area to be updated is written into the fixed thermal interference area.

[0081] When the area to be updated is of the reflective interference type, the area to be updated is written into the fixed reflective area;

[0082] When the type of the area to be updated is occlusion, the area to be updated is written to the occlusion boundary area;

[0083] The updated fixed thermal interference area, fixed reflective area, and occlusion boundary area participate in the generation of scene conflict events and the generation of verification pose sequences in the next inspection cycle.

[0084] Preferably,

[0085] After the parking compensation action is executed, the parking image after compensation is re-acquired;

[0086] Map the compensated parking image onto the fire scene constraint diagram;

[0087] When the image area corresponding to the fire-fighting target area in the compensated parking image falls within the preset pixel range, a visible light image and an infrared thermal imaging image are acquired.

[0088] When the image area corresponding to the fire-fighting object area in the compensated parking image does not fall within the preset pixel range, a backup acquisition pose corresponding to the current pose data is selected from the candidate verification pose set, and the fire-fighting robot dog is controlled to move to the backup acquisition pose.

[0089] Preferably,

[0090] When the cross-pose consistency result is an abnormal confirmation result, control the fire-fighting robot dog to collect the current environmental detection data;

[0091] The inspection anomaly record includes preset inspection points, associated map version number, conflict category, conflict area, cross-pose consistency results, environmental detection data, and verification pose sequence;

[0092] Send inspection anomaly records to the edge device;

[0093] Based on the conflict area and verification pose sequence in the inspection anomaly record, the edge performs secondary image recognition on the first verification visible light image, the first verification infrared image, the second verification visible light image, and the second verification infrared image, and returns the secondary recognition result;

[0094] The secondary identification results include secondary anomaly confirmation results or secondary anomaly exclusion results.

[0095] Preferably,

[0096] The generated inspection and handling instructions include:

[0097] When the secondary identification result is a secondary anomaly confirmation result, and the conflict category is a thermal anomaly conflict event, a thermal anomaly handling instruction is generated.

[0098] When the secondary identification result is a secondary anomaly confirmation result, and the conflict category is a state deviation conflict event, an object state re-inspection instruction is generated.

[0099] When the secondary identification result is a secondary anomaly confirmation result, and the conflict category is an occlusion / missing conflict event, an occlusion clearance instruction is generated.

[0100] When the secondary identification result is a secondary anomaly exclusion result, a re-verification instruction is generated.

[0101] Compared with existing technologies, the above technical solution has the following advantages:

[0102] I. This invention establishes a fire scene constraint diagram bound to preset inspection points, which associates the fire object area, object baseline state, fixed thermal interference area, fixed reflective area, occlusion boundary area, and candidate verification pose set in a fixed fire inspection scene. This enables the fire robot dog to no longer rely solely on the recognition results of a single visible light image or infrared thermal imaging image when performing image recognition. Instead, it can perform scene-based verification by combining the regional constraints and object states of the fixed fire inspection scene, thereby reducing the interference caused by fixed heat sources, fixed reflective objects, and permanent occlusions on the initial recognition results.

[0103] Second, this invention maps the initial recognition results to a fire scene constraint diagram and generates scene conflict events based on the fire object area, object baseline state, fixed thermal interference area, fixed reflective area, and occlusion boundary area. This enables different types of anomalies, such as area mismatch, state deviation, thermal anomaly, reflective interference, and occlusion loss, to be distinguished and processed. It avoids uniformly outputting image recognition deviations from different sources as ordinary alarm results, thereby improving the pertinence of fire inspection anomaly judgment.

[0104] Third, this invention selects at least two observable core poses from the candidate core pose set based on scene conflict events, and excludes candidate core poses that overlap with fixed thermal interference areas or fixed reflective areas or whose conflict areas are occluded by occluded boundary areas during the selection process, so that the core image acquisition position can avoid known fixed interference sources and occlusion boundaries, reducing the situation where the same interference factor still affects ordinary multi-angle shooting.

[0105] Fourth, this invention controls a fire robot dog to perform multi-pose verification image acquisition by verifying pose sequence, and generates cross-pose consistency results based on object category and object state under different verification poses. This enables anomaly confirmation results, verification inconsistency results, and interference elimination results to be formed based on multiple observation poses, thereby improving the verifiability and judgment stability of image recognition results in fixed fire inspection scenarios.

[0106] Fifth, after generating interference elimination results, this invention writes the conflict area, conflict type, associated map version number, and verification pose into the interference observation record. When interference elimination results are generated in two consecutive inspection cycles in the same conflict area, a map update request is generated, so that the fixed thermal interference area, fixed reflective area, and occlusion boundary area can be updated during the inspection process, avoiding the fire scene constraint map from remaining in the initial calibration state for a long time.

[0107] VI. This invention enables the updated fixed thermal interference area, fixed reflective area, and occlusion boundary area to continue participating in the generation of scene conflict events and the generation of verification pose sequences in the next inspection cycle. This allows the updated results of the fire scene constraint map to directly affect the subsequent image recognition and verification control process, rather than being saved as ordinary historical records. This helps to reduce repeated false alarms in continuous inspection cycles. Attached Figure Description

[0108] Figure 1 This is a flowchart illustrating a fire source location system for a fire-fighting robot dog based on image detection.

[0109] Figure 2 A curve showing the comparison of false alarm rates under different inspection cycles;

[0110] Figure 3 This is a schematic diagram illustrating the composition of a fire safety scenario constraint diagram;

[0111] Figure 4 A flowchart illustrating the process for verifying pose selection and cross-pose consistency judgment;

[0112] Figure 5 This is a flowchart illustrating the process of edge-end secondary recognition and fire scene constraint graph update. Detailed Implementation

[0113] The following is combined Figures 1 to 5 Specific embodiments of the present invention will be described below. It should be understood that the following embodiments are used to clearly and completely describe the technical solution of the present invention, and are not intended to limit the scope of protection. Without departing from the technical concept of the present invention, those skilled in the art can make adaptive adjustments to some implementation parameters according to specific fire inspection scenarios, fire robot dog models, image acquisition equipment types, infrared thermal imaging device resolution, and the layout of fire inspection objects.

[0114] This embodiment provides a fire source localization system for a fire-fighting robot dog based on image detection. This method is applied to image recognition and intelligent inspection in fixed fire inspection scenarios. In this embodiment, the fire-fighting robot dog is a quadrupedal fire inspection robot equipped with image acquisition, infrared thermal imaging, pose detection, motion control, and computing devices. The core processing objects of this embodiment are visible light images, infrared thermal imaging images, and verification image recognition results obtained based on the above images in the fixed fire inspection scenario. The focus is on image recognition, scene constraint verification, multi-pose verification, and cross-pose consistency confirmation of the fixed fire inspection scenario images. The fixed fire inspection scenario can be a factory fire lane, storage space, power distribution room, underground space, building corridor, equipment room, hazardous chemical temporary storage area, or data center corridor, etc., which require periodic inspection.

[0115] The fire-fighting robot dog is equipped with a visible light image acquisition device, an infrared thermal imaging device, an environmental detection device, a pan-tilt unit, a pose detection device, and a motion control unit. The visible light image acquisition device is used to acquire visible light images of fixed fire inspection scenarios; the infrared thermal imaging device is used to acquire infrared thermal images; the environmental detection device is used to acquire environmental detection data; the pose detection device is used to acquire the current pose data of the fire-fighting robot dog; the pan-tilt unit is used to adjust the image acquisition direction; and the motion control unit is used to control the fire-fighting robot dog to perform actions such as parking, lateral strides, forward and backward step adjustments, body height adjustments, and verification parking.

[0116] like Figure 1 As shown, in one embodiment, before performing inspection tasks, the fire-fighting robot dog first acquires a fire scene constraint map bound to preset inspection points. The preset inspection points refer to the stopping positions of the fire-fighting robot dog during image recognition and verification collection in a fixed fire inspection scene. These points can be pre-marked manually on a map or automatically generated by the fire-fighting robot dog during the initial mapping process, taking into account the location of fire-fighting objects. Each preset inspection point corresponds to a fire scene constraint map, which records prior scene information related to image recognition, anomaly detection, and verification collection at that preset inspection point.

[0117] like Figure 3As shown, the fire scene constraint diagram includes a reference scene image, a fire-fighting object area, an object reference state, a fixed thermal interference area, a fixed reflective area, an occlusion boundary area, a map version number, a candidate verification pose set, and a mapping relationship between image coordinates and the robot dog pose. The reference scene image is a standard reference image collected at preset inspection points, and may include visible light reference images and infrared reference images. The fire-fighting object area is the image area corresponding to the fire inspection object in the reference scene image. The fire inspection object includes at least one of the following: fire extinguisher, fire hydrant, fire door, evacuation sign, electrical cabinet, smoke detector, sprinkler head, and evacuation route. The object reference state is used to record the status information of the fire inspection object under normal inspection conditions, such as fire extinguisher in place, fire door closed, evacuation route unobstructed, evacuation sign lit, electrical cabinet closed, or sprinkler head unobstructed. The fixed thermal interference area refers to the area of ​​a fixed heat source in the reference scene image or infrared reference image, such as the normal heating area of ​​a power distribution cabinet, HVAC vents, the hot area near lighting fixtures, or the heat dissipation area of ​​equipment. The fixed reflective area refers to the area of ​​a fixed reflective object in the visible light image, such as metal cabinet doors, glass doors, reflective signs, or glossy floor areas. The occlusion boundary area refers to the boundary area formed by a permanent occlusion or spatial structure in the image, such as the edge of a shelf, the edge of a column, the edge of an equipment casing, or the edge of a fixed partition. The map version number is used to identify the current version of the fire scene constraint map and is used for traceability when generating scene conflict events, interference observation records, and map update requests. The candidate verification pose set includes multiple candidate verification poses, each corresponding to the fuselage pose and gimbal attitude that the fire robot dog can reach and acquire verification images. The gimbal attitude includes the gimbal pitch angle and the gimbal yaw angle. The mapping relationship between the image coordinates and the robot dog's pose is used to establish the correspondence between the image area position and the fire robot dog's verification parking position, fuselage height, fuselage orientation, gimbal pitch angle, and gimbal yaw angle.

[0118] When establishing fire scene constraint diagrams, the fire-fighting robot dog can collect multiple sets of reference scene images at preset inspection points with at least two fuselage heights and at least two gimbal tilt angles. For example, the fire-fighting robot dog can park at a first fuselage height and a second fuselage height, and at each fuselage height, set a first gimbal tilt angle and a second gimbal tilt angle to collect images, thereby obtaining multiple sets of reference scene images. The first fuselage height can correspond to the normal inspection height, and the second fuselage height can correspond to the low-position observation height after lowering the fuselage; the first gimbal tilt angle can correspond to the forward observation angle, and the second gimbal tilt angle can correspond to the upward or downward observation angle. The specific values ​​of the above heights and angles can be set according to the mechanical structure of the fire-fighting robot dog, the gimbal rotation range, and the height of the fire-fighting object on site.

[0119] After obtaining multiple sets of baseline scene images, fire inspection objects, fixed heat sources, fixed reflective objects, permanent obstructions, and passable areas can be labeled in these images through manual annotation, semi-automatic annotation, or image recognition-assisted annotation. After annotation, the image region corresponding to the fire inspection object is written into the fire object region, the normal state of the fire inspection object is written into the object baseline state, the image region corresponding to the fixed heat source is written into the fixed thermal interference region, the image region corresponding to the fixed reflective object is written into the fixed reflective region, and the image boundary corresponding to the permanent obstruction is written into the obstruction boundary region. The correspondence between the passable area, fuselage height, gimbal pitch angle, and baseline scene images is written into the candidate verification pose set. The resulting fire scene constraint map not only contains baseline information for image recognition comparison but also includes fixed thermal interference regions, fixed reflective regions, obstruction boundary regions, and pose mapping information for subsequent verification pose selection.

[0120] In one implementation, each fire inspection target corresponds to at least two candidate verification poses. The at least two candidate verification poses for the same fire inspection target have different fuselage heights, different gimbal pitch angles, or different verification parking positions. Each candidate verification pose records the unobstructed visible area of ​​the fire inspection target in the corresponding reference scene image, image region coordinates, fuselage orientation, and gimbal yaw angle. Therefore, when a fire inspection target exhibits missing, occluded, deviated state, or thermal anomalies in the initial identification results, a verification pose that allows observation of the fire inspection target and is less susceptible to fixed thermal interference areas, fixed reflective areas, and occluded boundary areas can be selected from the candidate verification pose set.

[0121] When the fire-fighting robot dog performs inspections, it first moves to a preset inspection point. Before acquiring visible light and infrared thermal images, the robot dog can acquire a current parking image and obtain current pose data. This current pose data may include the current verification parking position, fuselage height, fuselage orientation, fuselage lateral offset, fuselage longitudinal offset, gimbal pitch angle, and gimbal yaw angle. Subsequently, the current parking image is registered with the reference scene image in the fire scene constraint diagram, and the current pose data is matched with candidate verification poses in the candidate verification pose set. Registration can be performed using feature point matching, edge matching, region contour matching, or image semantic region matching. The matching determines whether the current pose data falls within a preset pose range.

[0122] In one specific implementation, the image region center offset distance is used to determine whether the currently paused image meets the acquisition conditions. The image region center offset distance is calculated according to the following formula:

[0123] ;

[0124] in, This represents the offset distance from the center of the image region, in pixels. This represents the x-coordinate of the center point of the image region corresponding to the fire-fighting target area in the current stationary image; This represents the ordinate of the center point of the image region corresponding to the fire-fighting target area in the current stationary image; The x-coordinate of the center point of the corresponding fire-fighting object area in the baseline scene image; This represents the ordinate of the center point of the corresponding fire-fighting object area in the baseline scene image.

[0125] For example, at a predetermined inspection point, the center point of the fire extinguisher's corresponding fire-fighting object area in the baseline scene image is (640, 360), and the center point of the fire extinguisher's corresponding image area in the currently parked image is (628, 365). Then, the image area center offset distance is:

[0126] ;

[0127] In this example, if the preset pixel range is an image area center offset distance of no more than 25 pixels, then 13.00 pixels meet the acquisition conditions, and visible light images and infrared thermal imaging images can continue to be acquired. The preset pixel range can be determined based on the calibration error of the reference scene image, camera resolution, fire-fighting object area size, and on-site test data. If the image area center offset distance is greater than 25 pixels, a pause compensation action is generated.

[0128] In another specific implementation, a pose deviation score is used to determine whether the current pose data meets a preset pose range. The pose deviation score is calculated according to the following formula:

[0129] ;

[0130] in, Indicates the pose deviation score; This indicates the fuselage facing angle in the current pose data; Indicates the reference fuselage orientation angle in the candidate verification pose; This indicates the permissible deviation in fuselage orientation angle; This indicates the fuselage height in the current pose data; Indicates the reference fuselage height in the candidate verification pose; This indicates the permissible deviation in fuselage height; This represents the lateral offset in the current pose data; Indicates the reference lateral position in the candidate verification pose; Indicates the allowed lateral offset; This represents the forward and backward offset in the current pose data; Indicates the reference front and back positions in the candidate verification pose; This indicates the allowed forward and backward offsets.

[0131] For example, if the fuselage facing angle in the current pose data is 2.0 degrees, the reference fuselage facing angle in the candidate verification pose is 0 degrees, and the allowable fuselage facing angle deviation is 5.0 degrees; if the current fuselage height is 0.53 meters, the reference fuselage height is 0.50 meters, and the allowable fuselage height deviation is 0.08 meters; if the current lateral offset is 0.04 meters, the reference lateral position is 0 meters, and the allowable lateral offset is 0.10 meters; and if the current forward / backward offset is 0.05 meters, the reference forward / backward position is 0 meters, and the allowable forward / backward offset is 0.12 meters, then the pose deviation score is:

[0132] ;

[0133] ;

[0134] In this example, if the preset pose range is the pose deviation score If the value is not greater than 0.50, the current pose data meets the preset pose range, and image acquisition can continue. The preset pose range can be determined based on the candidate verification pose calibration error, the fire robot dog's positioning error, the gimbal angle control error, and historical inspection stability data. If the pose deviation score... If the value is greater than 0.50, a parking compensation action is generated. This parking compensation action includes at least one of the following: fuselage yaw adjustment, lateral step adjustment, forward / backward step adjustment, fuselage altitude adjustment, and gimbal pitch adjustment. This method enables subsequent image recognition to be based on a field of view that matches the constraints of the fire scene.

[0135] After the parking compensation action is executed, the fire-fighting robot dog re-acquires the compensated parking image and maps it to the fire scene constraint map. When the image area corresponding to the fire-fighting object area in the compensated parking image falls within a preset pixel range, the fire-fighting robot dog acquires visible light and infrared thermal images. When the image area corresponding to the fire-fighting object area in the compensated parking image still does not fall within the preset pixel range, a backup acquisition pose corresponding to the current pose data is selected from the candidate verification pose set, and the fire-fighting robot dog is controlled to move to the backup acquisition pose. The backup acquisition pose is preferentially selected from candidate verification poses that have a shorter action switching distance from the current pose data and can cover the fire-fighting object area.

[0136] After the fire-fighting robot dog completes its parking or standby acquisition posture adjustment, it acquires visible light and infrared thermal images of a fixed fire inspection scene at preset inspection points and obtains its current posture data. Subsequently, it performs artificial intelligence image recognition on the visible light and infrared thermal images to obtain initial recognition results. The initial recognition results include object category, object region, object status, thermal anomaly area, and occlusion relationship.

[0137] In one implementation, the artificial intelligence image recognition is performed using an artificial intelligence image recognition model. The model includes a fire object recognition branch, a state recognition branch, a thermal anomaly recognition branch, and an occlusion boundary recognition branch. The fire object recognition branch outputs the object category and object region; the state recognition branch outputs the object state; the thermal anomaly recognition branch outputs the thermal anomaly region; and the occlusion boundary recognition branch outputs the occlusion relationship. The outputs of each branch collectively form the initial recognition result and serve as input for generating scene conflict events. The object state includes at least one of the following: in-place state, missing state, on state, off state, occlusion state, channel occupancy state, and thermal anomaly-accompanying state. The object region can be a detection box region, segmentation mask region, or semantic region in a visible light image; the thermal anomaly region can be a region in an infrared thermal imaging image whose temperature distribution is higher than the baseline temperature distribution of the same scene; the occlusion relationship can indicate whether there is occlusion, partial occlusion, or no occlusion between the fire object region and the occlusion boundary region.

[0138] In one specific implementation, the average temperature difference between regions is used to determine thermal anomaly regions in an infrared thermal imaging image. The average temperature difference between regions is calculated according to the following formula:

[0139] ;

[0140] in, Indicates the average temperature difference in the region; This represents the average temperature of the region to be determined in the infrared thermal imaging image. This indicates the scene reference temperature of the corresponding area in the fire scene constraint diagram.

[0141] For example, in a power distribution room inspection scenario, the average temperature of an area to be assessed next to the power distribution cabinet in the infrared thermal imaging image is 46.8 degrees Celsius, while the baseline temperature of the corresponding area in the fire scene constraint diagram is 34.5 degrees Celsius. Therefore, the average temperature difference between the areas is:

[0142] ;

[0143] In this example, if the preset thermal anomaly condition is that the average temperature difference of the area is not less than 10.0 degrees Celsius, then the area to be judged can be identified as a thermal anomaly area. The preset thermal anomaly condition can be determined based on the scene reference temperature in the fire scene constraint diagram, the historical temperature fluctuation range, the temperature measurement error of the infrared thermal imaging device, and the on-site fire safety requirements. If the thermal anomaly area falls within a fixed thermal interference area, then the interference elimination result can be generated by combining the cross-pose consistency result; if the thermal anomaly area is outside the fixed thermal interference area, and the object state is a thermal anomaly-accompanying state, then a thermal anomaly conflict event can be generated.

[0144] Artificial intelligence image recognition models can be implemented using one or more combinations of object detection networks, semantic segmentation networks, instance segmentation networks, infrared thermal anomaly detection networks, and multi-task recognition networks. Model training data can include visible light image samples, infrared thermal imaging image samples, fire-fighting object area annotation samples, object state annotation samples, fixed thermal interference area samples, fixed reflective area samples, and occlusion boundary area samples from fixed fire inspection scenarios. To adapt the image recognition results to fixed fire inspection scenarios, the training samples can include image samples with different aircraft heights, different gimbal pitch angles, different inspection and parking positions, different lighting conditions, and different occlusion states.

[0145] After obtaining the initial identification results, these results are mapped to a fire scene constraint map, and scene conflict events are generated based on the fire object area, object baseline state, fixed thermal interference area, fixed reflective area, and occlusion boundary area. Specifically, when an object area does not fall into the corresponding fire object area, an area mismatch conflict event is generated; when the object state is inconsistent with the corresponding object baseline state, a state deviation conflict event is generated; when a thermal anomaly area is outside the fixed thermal interference area and the object state is accompanied by a thermal anomaly, a thermal anomaly conflict event is generated; when an object area overlaps with a fixed reflective area and the object state is accompanied by a thermal anomaly, a reflective interference conflict event is generated; when a fire object area overlaps with an occlusion boundary area and no corresponding object category is identified, an occlusion missing conflict event is generated. Each scene conflict event includes a conflict category, conflict area, associated object category, associated object state, and association map version number. The association map version number indicates the version of the fire scene constraint map called when generating the scene conflict event, facilitating subsequent tracking and updates.

[0146] like Figure 4As shown, after generating scene conflict events, at least two observable kernel poses are selected from the candidate kernel pose set based on the scene conflict events, and a kernel pose sequence is generated according to the mapping relationship from image coordinates to robot dog poses. Specifically, firstly, preliminary kernel poses corresponding to the conflict area are selected from the candidate kernel pose set; then, preliminary kernel poses that overlap with the imaging field of view and the fixed thermal interference area, preliminary kernel poses that overlap with the fixed reflective area, and preliminary kernel poses that are occluded by the occluded boundary area of ​​the conflict area are deleted; finally, the remaining preliminary kernel poses are determined as observable kernel poses, and at least two observable kernel poses are arranged in ascending order of action switching cost to generate a kernel pose sequence. In other words, the preliminary kernel poses are determined as observable kernel poses only after being filtered by the exclusion rules corresponding to the fixed thermal interference area, the fixed reflective area, and the occluded boundary area.

[0147] In one specific implementation, the execution order of the observable verification pose is determined using action switching cost. The action switching cost is calculated according to the following formula:

[0148] ;

[0149] in, Indicates the cost of switching actions; This represents the distance traveled from the position corresponding to the current pose data to the stopping position corresponding to the candidate verification pose; This indicates the height difference between the current fuselage height and the fuselage height corresponding to the candidate verification pose; This indicates the angular difference between the current fuselage orientation and the fuselage orientation corresponding to the candidate verification pose; This represents the angle difference between the current gimbal pitch angle and the gimbal pitch angle corresponding to the candidate verification pose; This represents the angle difference between the current gimbal yaw angle and the gimbal yaw angle corresponding to the candidate verification pose. Before calculation, all the above items can be normalized according to their respective allowable ranges, ensuring that each value is between 0 and 1. The weights of each item can be calibrated based on the fire-fighting robot dog's motion energy consumption, movement stability, inspection time requirements, and on-site access space.

[0150] For example, a certain conflict region corresponds to 3 initial verification poses. After eliminating the fixed thermal interference region, fixed reflective region, and occlusion boundary region, the remaining initial verification poses are the first, second, and third initial verification poses. The normalized parameters are shown in Table 1 below.

[0151] Table 1 Preliminary Selection and Verification Pose Data Table ;

[0152] The action switching cost for the first initial verification pose is:

[0153] ;

[0154] Based on the order of action switching costs from smallest to largest, the first and second preliminary verification poses are determined as the first observable verification pose and the second observable verification pose, respectively. The third preliminary verification pose can be used as a supplementary verification pose for supplementary acquisition when verification is inconsistent.

[0155] In one implementation, the verification pose sequence includes a first observable verification pose and a second observable verification pose. The first and second observable verification poses have different fuselage heights, different gimbal pitch angles, or different verification parking positions. The firefighting robot dog acquires a first verification visible light image, a first verification infrared image, a second verification visible light image, and a second verification infrared image from the first and second observable verification poses, respectively. Subsequently, the first verification visible light image, the first verification infrared image, the second verification visible light image, and the second verification infrared image are mapped to the fire scene constraint graph to generate cross-pose consistency results.

[0156] When generating cross-pose consistency results, an anomaly confirmation result is generated when the first and second observable verification poses identify the same object category and state for the same conflict region, and the conflict region does not fall within a fixed thermal interference region or a fixed reflective region. An inconsistency result is generated when the first and second observable verification poses identify different object categories or states for the same conflict region. An interference elimination result is generated when the conflict region falls within a fixed thermal interference region or a fixed reflective region.

[0157] In one specific implementation, object category consistency markers and object state consistency markers can be used for determination. The object category consistency marker is determined according to the following formula:

[0158] ;

[0159] in, Indicates a consistent object category marker; This indicates the object category identified under the first observable verification pose; This indicates the object category identified under the second observable verification pose.

[0160] The object state consistency flag is determined according to the following formula:

[0161] ;

[0162] in, Indicates a consistent state flag for the object; This represents the object state identified under the first observable verification pose; This represents the object state identified under the second observable verification pose.

[0163] An anomaly confirmation result is generated when both the object category and object status are consistently marked as 1, and the conflict area does not fall within the fixed thermal interference area or the fixed reflective area. For example, if the first observable verification pose identification determines that the object category is a power distribution cabinet and the object status is accompanied by thermal anomaly; and the second observable verification pose identification also determines that the object category is a power distribution cabinet and the object status is also accompanied by thermal anomaly, and the conflict area does not belong to the fixed thermal interference area or the fixed reflective area, then the object category is consistently marked as 1, the object status is consistently marked as 1, and an anomaly confirmation result is generated.

[0164] When an inconsistency in the verification result is generated, a supplementary verification pose is called from the candidate verification pose set. This supplementary verification pose has a different fuselage height, a different gimbal pitch angle, or a different verification parking position compared to the first and second observable verification poses. The fire-fighting robot dog acquires supplementary verification visible light and infrared images using the supplementary verification poses and updates the cross-pose consistency result based on these images. For example, if the first observable verification pose identifies the fire-fighting object as occluded, while the second observable verification pose identifies it as in place, supplementary verification poses with different fuselage heights or gimbal pitch angles can be called to determine if the difference is caused by an occlusion boundary region.

[0165] When interference elimination results are generated, an inspection cycle is defined as one inspection of a preset inspection point. The conflict area, conflict type, associated map version number, first observable verification pose, and second observable verification pose are written into the interference observation record. When interference elimination results are generated for the same conflict area in two consecutive inspection cycles, a map update request is generated. Using two consecutive inspection cycles as the trigger condition avoids directly updating the fire scene constraint map due to a single accidental interference. The map update request includes the area to be updated, the type of the area to be updated, the trigger inspection cycle, and the map version number to be updated. The type of the area to be updated can include thermal interference, reflective interference, and occlusion. When updating the fire scene constraint map based on the map update request, if the area to be updated is a thermal interference type, it is written into a fixed thermal interference area; if the area to be updated is a reflective interference type, it is written into a fixed reflective area; and if the area to be updated is an occlusion type, it is written into an occlusion boundary area. The updated fixed thermal interference area, fixed reflective area, and occlusion boundary area participate in the generation of scene conflict events and the generation of verification pose sequences in the next inspection cycle. Therefore, the fire scene constraint map can be updated with the long-term changes of the fixed fire inspection scene, directly affecting subsequent image recognition, scene conflict event generation, and verification pose selection.

[0166] like Figure 5 As shown, in one embodiment, when the cross-pose consistency result is an anomaly confirmation result, the fire-fighting robot dog collects current environmental detection data. This environmental detection data includes at least one of the following: ambient temperature data, smoke concentration data, carbon monoxide concentration data, combustible gas concentration data, humidity data, and sound anomaly data. Subsequently, an inspection anomaly record is generated. The inspection anomaly record includes preset inspection points, associated map version number, conflict category, conflict area, cross-pose consistency result, environmental detection data, and verification pose sequence. The inspection anomaly record can be written to a local inspection database or other data storage unit used to store inspection anomaly records and sent to the edge device. Based on the conflict area and verification pose sequence in the inspection anomaly record, the edge device performs secondary image recognition on the first verification visible light image, the first verification infrared image, the second verification visible light image, and the second verification infrared image, and returns the secondary recognition result. The secondary recognition result includes a secondary anomaly confirmation result or a secondary anomaly exclusion result. The edge device can deploy a more computationally powerful image recognition model to re-identify the verification images selected by the fire-fighting robot dog's local end.

[0167] When generating inspection and handling instructions, a thermal anomaly handling instruction is generated when the secondary identification result is a secondary anomaly confirmation result and the conflict type is a thermal anomaly conflict event; an object status re-inspection instruction is generated when the secondary identification result is a secondary anomaly confirmation result and the conflict type is a state deviation conflict event; an occlusion clearance instruction is generated when the secondary identification result is a secondary anomaly confirmation result and the conflict type is an occlusion missing conflict event; and a re-verification instruction is generated when the secondary identification result is a secondary anomaly exclusion result. The thermal anomaly handling instruction can be used to trigger the fire robot dog to maintain a safe distance, send thermal anomaly image packets to the edge, or link with the fire monitoring platform; the object status re-inspection instruction can be used to increase the re-inspection priority of the corresponding preset inspection point in the next inspection cycle; the occlusion clearance instruction can be used to prompt manual inspection of whether there are temporary obstructions in front of the fire-fighting object; and the re-verification instruction can be used to call a supplementary verification pose or a backup acquisition pose to acquire images again. The aforementioned inspection and handling instructions are all associated with image recognition, scene conflict events, and cross-pose consistency results, and are used to adjust the fire robot dog's review collection, image uploading, or collection strategy for the next inspection cycle.

[0168] To illustrate the implementation effect of this embodiment, a comparative test was conducted in a combined scenario of a power distribution room and a storage passage. The test scenario included six preset inspection points, each equipped with at least three types of fire inspection objects: fire extinguishers, fire hydrants, power distribution cabinets, evacuation signs, and passageways. During the test, fixed heat sources, fixed reflective objects, and permanent obstructions were set up, simulating thermal anomalies, deviations in the status of fire-fighting objects, and temporary obstructions. The comparative example used common fire-fighting robot methods such as visible light image recognition, infrared thermal imaging recognition, and gimbal alignment for anomaly detection; this embodiment uses fire scene constraint diagrams, scene conflict events, verified pose sequences, and cross-pose consistency results for anomaly detection. The number of test images was 600 sets for each method, and the number of false alarms was counted based on manual verification conclusions. The test data shown in Table 2 below are exemplary data used to illustrate the technical effects.

[0169] Table 2 Comparison of test results between the comparative example and this embodiment ;

[0170] The average number of images reviewed for a single anomaly event is calculated using the following formula:

[0171] ;

[0172] in, This indicates the average number of images reviewed for a single anomaly event; Indicates the total number of images reviewed; This indicates the number of exceptions that were reviewed.

[0173] For example, in this embodiment, if the total number of reviewed images is 240 sets and the number of reviewed anomalies is 100, then the average number of reviewed images per anomaly is:

[0174] ;

[0175] The anomaly detection accuracy is calculated using the following formula:

[0176] ;

[0177] in, Indicates the accuracy rate of anomaly confirmation; Indicates the number of correctly confirmed exceptions; This indicates the total number of actual abnormal events.

[0178] For example, in this embodiment, if the number of correctly identified abnormal events is 110, and the total number of actual abnormal events is 116, then the anomaly identification accuracy rate is:

[0179] ;

[0180] To further illustrate the impact of fixed thermal interference areas, fixed reflective areas, and obstruction boundary areas on the false alarm rate under different inspection cycles, false alarm rate statistics were performed using seven consecutive inspection cycles. The results are shown below. Figure 2 The false alarm rate is calculated using the following formula:

[0181] ;

[0182] in, Indicates the false alarm rate; Indicates the number of false alarms; This indicates the total number of anomaly detections.

[0183] In the first inspection cycle, the comparative sample had 21 false alarms, and the total number of anomaly identifications was 100. Therefore, the false alarm rate of the comparative sample was:

[0184] ;

[0185] In this embodiment, the number of false alarms in the first inspection cycle was 13, and the total number of anomaly identifications was 102. Therefore, the false alarm rate of this embodiment is:

[0186] ;

[0187] Figure 2The false alarm rate comparison curves shown indicate that, as the inspection cycle increases, this embodiment, by interfering with observation records and map update requests, writes continuously occurring and excluded fixed thermal interference areas, fixed reflective areas, and occlusion boundary areas into the fire scene constraint map. This allows the updated fire scene constraint map to continue participating in the generation of scene conflict events and the generation of verification pose sequences in the next inspection cycle. Therefore, the false alarm rate of this embodiment shows a decreasing trend in continuous inspection cycles, while the comparative embodiment, which mainly relies on single image recognition and gimbal alignment, shows a smaller decrease in false alarm rate, as shown in Table 3.

[0188] Table 3 Comparison of False Alarm Rates During Inspection Cycles ;

[0189] The test results above show that in fixed fire inspection scenarios with fixed thermal interference areas, fixed reflective areas, and obstruction boundary areas, this embodiment performs scenario-based verification of the initial recognition results through fire scene constraint diagrams, obtains multi-pose verification images by verifying pose sequences, and confirms anomalies or eliminates interference through cross-pose consistency results. This reduces false alarms caused by fixed heat sources, fixed reflective objects, and permanent obstructions. Compared with methods that rely solely on visible light image recognition, infrared thermal imaging recognition, and gimbal alignment, this embodiment combines image recognition results with regional constraints, object baseline states, and robot dog verification poses in fixed fire inspection scenarios, thereby improving the verifiability of anomaly recognition results and the accuracy of handling criteria.

[0190] In another implementation, the fire scene constraint map can be created locally by the fire robot dog, or it can be created by the edge terminal or a server connected to the edge terminal based on historical inspection data and then sent to the fire robot dog. The local terminal can store the fire scene constraint map related to the current inspection area, while the edge terminal can store the fire scene constraint maps corresponding to multiple inspection areas and their historical versions. When the fire robot dog detects a map update request locally, it can first generate a record to be updated locally, and then the edge terminal verifies the review images and interference observation records in the continuous inspection cycle. After the verification is passed, the updated fire scene constraint map is sent to the fire robot dog. This method is suitable for scenarios where multiple fire robot dogs conduct collaborative inspections.

[0191] In another implementation, environmental monitoring data can be used to expand the judgment of inspection anomaly records. For example, when the thermal anomaly area is outside the fixed thermal interference area and the smoke concentration data in the environmental monitoring data meets the preset smoke concentration conditions, the review priority of the inspection anomaly record corresponding to the thermal anomaly conflict event can be increased; when the object status is a channel occupancy status and the environmental monitoring data does not show thermal or smoke anomalies, an object status re-inspection instruction can be generated first; when occlusion missing conflict events occur continuously but the environmental monitoring data is not abnormal, an occlusion clearance instruction can be generated first. The above auxiliary judgment is still based on image recognition results, fire scene constraint diagrams, and cross-pose consistency results, and does not change the core processing flow of this embodiment.

[0192] In another implementation, the verification pose sequence can be adjusted based on the firefighting robot dog's mobility capabilities. For firefighting robot dogs with adjustable fuselage height, different viewing angles can be obtained primarily through changes in fuselage height; for firefighting robot dogs with a large gimbal pitch angle range, different viewing angles can be obtained primarily through changes in gimbal pitch angle; for narrow passage scenarios, candidate verification poses with minimal changes in the verification parking position can be prioritized; for scenarios with obstructions from pillars, shelves, or large equipment, candidate verification poses avoiding the obstruction boundary areas can be prioritized. All of the above alternative methods are specific implementations of filtering observable verification poses from a set of candidate verification poses and generating a verification pose sequence.

[0193] Through the above implementation method, image recognition can be performed on visible light images and infrared thermal images in fixed fire inspection scenarios. Combined with fire scene constraint diagrams, scene conflict events, verification pose sequences, and cross-pose consistency results, the initial recognition result deviations caused by fixed thermal interference areas, fixed reflective areas, and occlusion boundary areas can be verified and confirmed in a scenario-based manner. This implementation method provides a complete data processing and control process for anomaly identification, verification data collection, fire scene constraint diagram updates, and inspection handling in autonomous inspections of fixed locations by fire-fighting robot dogs.

[0194] It should be noted that the embodiments of the present invention have better implementability and are not intended to limit the present invention in any way. Any person skilled in the art may use the above-disclosed technical content to change or modify it into equivalent effective embodiments. However, any modifications or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A fire source location system for a fire-fighting robot dog based on image detection, characterized in that, include: Obtain the fire scene constraint map bound to the preset inspection point. The fire scene constraint map includes the reference scene image, fire object area, object reference state, fixed thermal interference area, fixed reflective area, occlusion boundary area, map version number, candidate verification pose set, and the mapping relationship from image coordinates to robot dog pose. The fire-fighting robot dog is controlled to collect visible light images and infrared thermal images of a fixed fire inspection scene at the preset inspection points, and to obtain the current pose data of the fire-fighting robot dog. Artificial intelligence image recognition is performed on the visible light image and the infrared thermal imaging image to obtain initial recognition results, which include object category, object region, object state, thermal anomaly region and occlusion relationship; The initial recognition results are mapped to the fire scene constraint map, and scene conflict events are generated based on the fire object area, the object reference state, the fixed thermal interference area, the fixed reflective area and the occlusion boundary area. Based on the scene conflict event, at least two observable core poses are selected from the candidate core pose set, and a core pose sequence is generated according to the mapping relationship between the image coordinates and the robot dog pose. The fire-fighting robot dog is controlled to perform multi-pose verification image acquisition and image recognition according to the verification pose sequence, and generate cross-pose consistency results. Based on the cross-pose consistency results, an inspection anomaly record and inspection handling instructions are generated.

2. The fire source location system for a fire-fighting robot dog based on image detection according to claim 1, characterized in that, The fire scene constraint diagram is established in the following way: At the preset inspection point, multiple sets of reference scene images are acquired at at least two fuselage heights and at least two gimbal pitch angles; In the multiple sets of reference scene images, fire inspection objects, fixed heat sources, fixed reflective objects, permanent obstructions, and passable areas are marked; Write the image area corresponding to the fire inspection object into the fire object area; Write the normal state of the fire inspection object into the object's baseline state; Write the image region corresponding to the fixed heat source into the fixed thermal interference region; Write the image area corresponding to the fixed reflective object into the fixed reflective area; Write the image boundary corresponding to the permanent occlusion into the occlusion boundary region; The correspondence between the passable area, the fuselage height, the gimbal pitch angle, and the reference scene image is written into the candidate verification pose set.

3. The fire source location system for a fire-fighting robot dog based on image detection according to claim 2, characterized in that, Each of the aforementioned fire inspection targets corresponds to at least two candidate verification poses; At least two candidate verification poses corresponding to the same fire inspection object have different fuselage heights, different gimbal pitch angles, or different verification parking positions; The candidate verification pose records the unobstructed visible area, image area coordinates, fuselage orientation, and gimbal yaw angle of the fire inspection object in the corresponding reference scene image.

4. The fire source location system for a fire-fighting robot dog based on image detection according to claim 1, characterized in that, Before acquiring the visible light image and the infrared thermal imaging image, the fire-fighting robot dog is controlled to acquire the current parking image and the current pose data; The current parking image is registered with the reference scene image in the fire scene constraint diagram, and the current pose data is matched with the candidate verification poses in the candidate verification pose set. When the image area corresponding to the fire-fighting object area in the current paused image does not fall within the preset pixel range, or when the current pose data does not fall within the preset pose range, a pause compensation action is generated. The parking compensation action includes at least one of the following: fuselage yaw adjustment, lateral step adjustment, forward and backward step adjustment, fuselage altitude adjustment, and gimbal pitch adjustment.

5. The fire source location system for a fire-fighting robot dog based on image detection according to claim 1, characterized in that, The artificial intelligence image recognition is performed using an artificial intelligence image recognition model; The artificial intelligence image recognition model includes fire object recognition branch, status recognition branch, thermal anomaly recognition branch, and occlusion boundary recognition branch; The fire-fighting object identification branch outputs the object category and the object area; The state recognition branch outputs the object state, which includes at least one of the following: in-place state, missing state, open state, closed state, occluded state, channel occupied state, and thermal anomaly accompanied state. The thermal anomaly identification branch outputs the thermal anomaly region; The occlusion boundary recognition branch outputs the occlusion relationship.

6. The fire source location system for a fire-fighting robot dog based on image detection according to claim 5, characterized in that, The generation of the scene conflict event includes: When the object area does not fall into the corresponding fire-fighting object area, an area mismatch conflict event is generated. When the state of the object is inconsistent with the corresponding baseline state of the object, a state deviation conflict event is generated; A thermal anomaly conflict event is generated when the thermal anomaly region is outside the fixed thermal interference region and the object state is the thermal anomaly accompanying state. When the object area overlaps with the fixed reflective area, and the object state is the accompanying thermal anomaly state, a reflective interference conflict event is generated. When the fire-fighting object area overlaps with the occlusion boundary area and no corresponding object category is identified, an occlusion missing conflict event is generated. The scenario conflict events include conflict category, conflict area, associated object category, associated object status, and associated graph version number.

7. The fire source location system for a fire-fighting robot dog based on image detection according to claim 6, characterized in that, Generating the verification pose sequence includes: Select the initial verification pose corresponding to the conflict region from the set of candidate verification poses; Delete the initial verification pose that overlaps with the fixed thermal interference region; Delete the initial verification pose that overlaps with the fixed reflective area; Delete the initial verification pose of the conflict area that is occluded by the occlusion boundary area; The remaining initial verification poses are determined as the observable verification poses; Calculate the action switching cost of each observable core pose relative to the current pose data, and arrange at least two observable core poses in ascending order of action switching cost to generate the core pose sequence.

8. The fire source location system for a fire-fighting robot dog based on image detection according to claim 7, characterized in that, The kernel pose sequence includes a first observable kernel pose and a second observable kernel pose; The first observable verification pose and the second observable verification pose have different fuselage heights, different gimbal pitch angles, or different verification parking positions; The fire-fighting robot dog acquires a first verification visible light image, a first verification infrared image, a second verification visible light image, and a second verification infrared image in the first observable verification pose and the second observable verification pose, respectively. The first verified visible light image, the first verified infrared image, the second verified visible light image, and the second verified infrared image are respectively mapped to the fire scene constraint map to generate the cross-pose consistency result.

9. The fire source location system for a fire-fighting robot dog based on image detection according to claim 8, characterized in that, Generating the cross-pose consistency result includes: When the first observable verification pose and the second observable verification pose identify the same object category and the same object state for the same conflict region, and the conflict region does not fall into the fixed thermal interference region and the fixed reflective region, an anomaly confirmation result is generated. When the first observable verification pose and the second observable verification pose identify different object categories or different object states for the same conflict region, a verification inconsistency result is generated. When the conflict area falls into the fixed thermal interference area or the fixed reflective area, an interference elimination result is generated.

10. The fire source location system for a fire-fighting robot dog based on image detection according to claim 9, characterized in that, When the aforementioned inconsistency in the verification result is generated, a supplementary verification pose is called from the candidate verification pose set; The supplementary verification pose has a different fuselage height, a different gimbal pitch angle, or a different verification parking position than the first observable verification pose and the second observable verification pose. The fire-fighting robot dog is controlled to acquire supplementary verification visible light images and supplementary verification infrared images in the supplementary verification pose. The cross-pose consistency result is updated based on the supplementary verified visible light image and the supplementary verified infrared image.

Images