Arc and spark recognition and early warning system and method based on visual-infrared image data fusion processing for mechanical dog

Abstract

This invention discloses an arc spark recognition and early warning system and method for mechanical dogs based on visual-infrared image data fusion processing, comprising: a multimodal sensing module for acquiring RGB images and thermal infrared image sequences from a power site; a time-space alignment module for timestamping, synchronizing exposure signals, and spatially registering the two types of images to generate a temporally and spatially consistent dual-modal image pair; a visual feature extraction module for extracting brightness abrupt changes, regional flicker, and frequency domain flicker features from the aligned RGB image; an infrared feature extraction module for extracting local hot spots and temperature gradient abrupt changes features from the aligned thermal infrared image; a fusion judgment and false alarm suppression module for performing feature-level / decision-level fusion of the dual-modal features, and outputting a danger level signal after cross-modal mutual verification and multi-frame judgment; and a graded early warning and response module for receiving the signal and executing local audible and visual alarms, mechanical dog motion control, sending alarms and on-site image data to a remote monitoring center according to the danger level.

Description

Technical Field

[0001] This application relates to the field of electrical digital data processing technology, specifically to an arc spark recognition and early warning system and method for mechanical dogs based on visual-infrared image data fusion processing, which realizes collaborative analysis, feature extraction and risk decision-making of multimodal sensing data through electrical digital data processing. Background Technology

[0002] With the deepening of intelligent and unmanned operations in the power industry, mobile robot platforms such as robotic dogs are gradually being introduced into power facilities such as substations and distribution rooms. Safe operation of power facilities relies on the accurate analysis of abnormal equipment conditions using electrical digital data processing technology. Electric arcs and sparks, as typical manifestations of power equipment faults, require real-time analysis of multi-source information through electrical digital data processing for early identification. However, existing solutions, when adapted to robotic dog platforms, have exposed multiple deficiencies in electrical digital data processing.

[0003] Traditional arc spark detection and early warning schemes rely solely on the processing of single electrical parameters such as current and voltage waveforms using digital data, lacking sensitivity to sparks with minute current fluctuations. Schemes primarily based on visual image recognition are suitable for identifying and warning of such sparks. However, image data from visual / infrared / ultraviolet camera modules requires digital data processing to reconstruct spatial pose. Existing digital data processing algorithms are insufficiently resistant to interference from strong light, dust, and other environmental factors, making it difficult to maintain detection stability. Digital processing schemes using only infrared thermal image data are prone to missing transient, small-scale arc sparks due to the lag in temporal data parsing caused by thermal diffusion effects and insufficient feature extraction accuracy at spatial resolution.

[0004] Furthermore, the existing system's electro-digital data processing architecture is not adapted to the resource constraints of the robotic dog: under the limitation of battery life, the power consumption optimization of electro-digital data processing is insufficient; the electro-digital data processing performance of the embedded computing unit is limited, making it difficult to support real-time correction of posture jitter data during movement; at the same time, there is a lack of a risk level quantification model based on electro-digital data processing, which makes it impossible to link the robotic dog's motion control module to dynamically adjust the response strategy, resulting in limited warning timeliness and reliability.

[0005] Although existing technologies have attempted to achieve arc detection from a single sensor through electro-digital data processing, a multi-modal data electro-digital collaborative processing framework adapted to the mobile scenarios of robotic dogs has not been constructed, nor have optimizations been made in power consumption management, dynamic feature extraction, and hierarchical decision-making at the electro-digital data processing level. As a result, the detection accuracy, environmental adaptability, and platform compatibility cannot meet the requirements. There is an urgent need to build an arc and spark detection and early warning solution adapted to robotic dogs based on electro-digital data processing technology. Summary of the Invention

[0006] The purpose of this application is to provide an arc spark recognition and early warning system and method for mechanical dogs based on visual-infrared image data fusion processing. It has the advantages of improving the accuracy and reliability of arc spark detection, reducing false alarms, adapting to complex power operation environments, and realizing intelligent response based on hazard level, thereby improving the safety of mechanical dog operation.

[0007] This application provides an arc spark recognition and early warning system for a robotic dog based on visual-infrared image data fusion processing. The system is integrated into the robotic dog itself and includes:

[0008] A multimodal sensing module is used to simultaneously acquire RGB image sequences and thermal infrared image sequences;

[0009] The time-space alignment module is used to perform timestamp synchronization, exposure signal alignment, and spatial registration based on calibration parameters on the received RGB image sequence and thermal infrared image sequence to generate a time-space consistent bimodal image pair.

[0010] A visual feature extraction module is used to extract brightness abrupt changes, regional flicker, and frequency domain flicker features from the RGB images of the dual-modal image pair;

[0011] The infrared feature extraction module is used to extract local hotspot regions and temperature gradient abrupt change features from the thermal infrared images of the dual-modal image pair;

[0012] The fusion judgment and false alarm suppression module is used to obtain dual-modal features from the visual feature extraction module and the infrared feature extraction module, and to perform feature-level or decision-level fusion calculation on the dual-modal features through the built-in weighted fusion unit to generate a preliminary danger probability; then, through the built-in multi-frame confirmation logic unit, the module applies cross-modal mutual verification rules to jointly judge the preliminary danger probability and dual-modal features of multiple consecutive frames, and outputs a danger level signal.

[0013] The graded early warning and response module is used to execute corresponding early warning and response actions according to different hazard levels; wherein, the early warning and response actions include local audible and visual alarms, limiting or emergency stopping the movement speed of the mechanical dog, and sending alarm information and on-site image data to the remote monitoring center.

[0014] The danger level signal of the graded early warning and response module is determined by the fusion confidence level, dual-modal confirmation status, and danger event duration output by the fusion judgment and false alarm suppression module, combined with the distance judgment result between the robot dog and the danger source.

[0015] Furthermore, the multimodal sensing module includes an RGB camera, an infrared thermal imager, and a timer; the RGB camera is configured as a global shutter RGB camera, and the infrared thermal imager is configured as an uncooled microbolometer infrared thermal imager; the RGB camera and the infrared thermal imager are fixedly connected by a rigid structure, and their optical axes remain parallel; the timer is used to send periodic hardware trigger signals to the RGB camera and the infrared thermal imager.

[0016] Furthermore, the time-space alignment module includes a time alignment submodule and a space alignment submodule;

[0017] The time alignment submodule is configured to receive the timestamp of the hardware trigger signal from the timer and the image exposure signal from the RGB camera and the infrared thermal imager, and to compensate for the differences in image acquisition time caused by different exposure durations or data transmission delays based on the timestamp of the hardware trigger signal and the image exposure signal, so as to align the time of the dual-modal image pair.

[0018] The spatial alignment submodule has built-in intrinsic and extrinsic parameters and distortion coefficients of the RGB camera and the infrared thermal imager obtained by the dual-target alignment method. The intrinsic and extrinsic parameters include the intrinsic parameters of the RGB camera and the infrared thermal imager, as well as the extrinsic parameters between the two sensors. Based on the intrinsic and extrinsic parameters and distortion coefficients, a perspective transformation matrix is ​​calculated. The thermal infrared image is resampled to the image coordinate system of the RGB camera or the RGB image is resampled to the image coordinate system of the infrared thermal imager, so that the same physical scene point is located at the same pixel coordinate in the RGB image and the thermal infrared image.

[0019] Furthermore, the visual feature extraction module employs a combination of image processing algorithms and neural networks, wherein:

[0020] The image processing algorithm is configured to perform image enhancement based on histogram equalization and image dehazing preprocessing based on dark channel prior on the input RGB image; convert the preprocessed image to the HSV color space, extract the V component as the luminance channel for temporal domain analysis, and detect luminance abrupt change regions by calculating the absolute value of the luminance difference between the current frame and the previous frame and setting a dynamic threshold; perform multi-frame frequency domain analysis on the luminance abrupt change regions, and detect the presence of high-frequency flicker features unique to electric arcs by fast Fourier transform, thereby filtering out candidate regions;

[0021] The neural network is configured to perform secondary discrimination on the candidate region through a pre-trained convolutional neural network and output the probability value that the candidate region is a real electric arc spark.

[0022] Furthermore, the infrared feature extraction module is used to perform non-uniformity correction and temperature calibration on the thermal infrared images of the dual-modal image pair, convert grayscale values ​​into accurate temperature values ​​and generate a temperature distribution matrix, and then use a threshold segmentation algorithm to extract connected regions above a set threshold of ambient temperature from the temperature distribution matrix and mark them as suspected hot spots. The highest temperature, average temperature, area and temperature gradient of each suspected hot spot are calculated, and temperature change events are detected by comparing the rate of temperature change at the same spatial location between consecutive frames. Finally, the temperature statistics, geometric features and change indicators of the hot spot area are combined to form an infrared feature vector and output to the fusion judgment and false alarm suppression module.

[0023] Furthermore, the weighted fusion unit is configured to execute at least one of a feature-level fusion strategy and a decision-level fusion strategy;

[0024] The feature-level fusion strategy is configured to concatenate the feature vectors output by the visual feature extraction module and the infrared feature extraction module before inputting them into the classifier, assign learnable weights to different modal features, and then input them into a lightweight fully connected neural network for joint classification.

[0025] The decision-level fusion strategy is configured to independently classify the feature vectors output by the visual feature extraction module and the infrared feature extraction module to obtain a visual confidence score and an infrared confidence score, and then perform a weighted average of the two scores to obtain a fusion score.

[0026] The multi-frame confirmation logic unit is configured to apply cross-modal mutual verification rules based on the fusion result output by the weighted fusion unit: when a single modality detects a suspected event, another modality needs to confirm it in the same spatial area within the next 1 to 3 frames, or the single modality itself needs to continuously alarm for N consecutive frames before it is determined to be a real dangerous event. The value of N can be dynamically set according to the false alarm tolerance.

[0027] Furthermore, the value of N ranges from 1 to 5.

[0028] Furthermore, the hazard level signal of the graded early warning and response module is jointly determined by the fusion confidence level, dual-modal confirmation status, and hazard event duration output by the fusion judgment and false alarm suppression module, combined with the distance determination result between the robot dog and the hazard source. The hazard level signal corresponds one-to-one with the response mechanism, including:

[0029] Level 1 Warning: When the detection result is low confidence, or when a suspected event is detected in only a single modality and the mutual verification conditions of the cross-modal mutual verification rules are not met, it is determined to be a Level 1 warning signal; the response mechanism is to trigger the LED indicator light of the robot dog to give a yellow warning, without interfering with the normal operation of the robot dog, and to remind on-site personnel to pay attention to potential risks;

[0030] Level 2 Warning: When the detection result meets the mutual verification conditions of the cross-modal mutual verification rules, and the degree of danger is moderate or the distance between the danger source and the robot dog is less than the first distance threshold, it is determined to be a Level 2 warning signal; the response mechanism is that the LED indicator of the robot dog body turns red and the buzzer is activated to emit a warning sound; the robot dog automatically limits its maximum movement speed and maintains the current working direction; and sends regular priority alarm information to the remote monitoring center, including the warning level, real-time location and on-site image data;

[0031] Level 3 Response: When the detection result is of high confidence, the dangerous event persists, or the distance between the dangerous source and the robot dog is less than the second distance threshold, it is determined to be a Level 3 response signal; the response mechanism is that the robot dog immediately executes the emergency stop procedure and locks the motion system; the LED indicator light is constantly red, and the buzzer continuously emits a warning sound; the highest priority alarm signal is sent to the remote monitoring center, and the alarm-related dual-modal image data and feature extraction logs are transmitted back, maintaining real-time data transmission with the remote monitoring center until a manual reset command is received.

[0032] Furthermore, it also includes a lazy dual-modal wake-up strategy management unit for controlling whether the system operates in the first operating mode or the second operating mode, wherein:

[0033] In the default first working mode, only the visual feature extraction module is controlled to be in working state, while the infrared feature extraction module and the fusion judgment and false alarm suppression module are controlled to be in sleep state; when the visual feature extraction module detects a brightness change event for multiple consecutive frames and the features of the event match the preset preliminary model of electric arc spark, the lazy dual-modal wake-up strategy management unit switches the system to the second working mode.

[0034] In the second working mode, the infrared feature extraction module and the fusion judgment and false alarm suppression module are awakened to work together; if the danger level signal does not reach the preset condition within a preset time, the lazy dual-modal wake-up strategy management unit will switch the system back to the first working mode.

[0035] Furthermore, the robotic dog body includes a navigation and positioning system, and the lazy bimodal wake-up strategy management unit is connected to the navigation and positioning system to obtain the real-time location and task information of the robotic dog, and accordingly determine whether the robotic dog is in a preset high-risk area or a known safe area, and whether the robotic dog is performing a non-detection task; when the robotic dog enters the preset high-risk area, the lazy bimodal wake-up strategy management unit controls the system to operate in the second working mode; when the robotic dog is in a known safe area or performing a non-detection task, the lazy bimodal wake-up strategy management unit controls the system to operate in the first working mode.

[0036] Furthermore, an arc spark recognition and early warning method for a robotic dog based on visual-infrared image data fusion processing, based on the above system, includes at least the following steps:

[0037] S1. The RGB image sequence and thermal infrared image sequence of the power operation site are simultaneously acquired through the multimodal sensing module;

[0038] S2. The RGB image sequence and the thermal infrared image sequence are time-stamped, exposure signal aligned, and spatially registered by the time-space alignment module to generate the temporally and spatially consistent dual-modal image pair.

[0039] S3. Extract brightness abrupt changes, regional flicker, and frequency domain flicker features from the RGB image of the dual-modal image pair using the visual feature extraction module;

[0040] S4. Extract local hotspot regions and temperature gradient abruptness features from the thermal infrared image of the dual-modal image pair using the infrared feature extraction module;

[0041] S5. Through the fusion judgment and false alarm suppression module, dual-modal features are obtained from the visual feature extraction module and the infrared feature extraction module. Feature-level or decision-level fusion calculations are performed on the dual-modal features to generate a preliminary danger probability. The cross-modal mutual verification rules are applied to jointly judge the preliminary danger probability and the dual-modal features of multiple consecutive frames and output a danger level signal.

[0042] S6. The graded warning and response module executes corresponding warning and response actions according to different hazard levels; wherein, the warning and response actions include local audible and visual alarms, limiting or stopping the mechanical dog's movement speed, and sending alarm information and on-site image data to the remote monitoring center.

[0043] The danger level signal of the graded early warning and response module is determined by the fusion confidence level, dual-modal confirmation status, and danger event duration output by the fusion judgment and false alarm suppression module, combined with the distance judgment result between the robot dog and the danger source.

[0044] Furthermore, in step S1, a periodic hardware trigger signal is sent to both the RGB camera and the infrared thermal imager simultaneously via a timer to ensure that the two sensors begin exposure within the same microsecond-level time window, thereby achieving hardware-level synchronous acquisition of the RGB image sequence and the thermal infrared image sequence.

[0045] Furthermore, in step S4, if a feature-level fusion strategy is adopted for fusion judgment, the dual-modal feature vectors are concatenated and assigned learnable weights, and then input into a lightweight fully connected neural network for joint classification; if a decision-level fusion strategy is adopted, the confidence scores of the two channels are weighted and the weights are dynamically adjusted according to the ambient lighting conditions; the joint judgment follows the multi-frame confirmation rule: after a single modality detects a suspected event, another modality must confirm it in the same spatial area within the next 1 to 3 frames, or a single modality must continuously alarm for 1 to 5 frames, before it is determined to be a real dangerous event.

[0046] Compared with the prior art, this application has the following beneficial effects:

[0047] This application provides an arc spark recognition and early warning system and method for mechanical dogs based on visual-infrared image data fusion processing. It integrates a multimodal sensing module for image acquisition, a time-space alignment module to ensure data consistency, a visual and infrared feature extraction module to extract key features, a fusion judgment and false alarm suppression module for cross-modal joint analysis, and a hierarchical early warning and response module to execute intelligent action control. This effectively solves the problems of low accuracy and high false alarm rate in arc spark detection in power operation environments, improves the accuracy and reliability of arc spark detection, reduces false alarms, adapts to complex power operation environments, and achieves intelligent response based on hazard level, thereby enhancing the safety of mechanical dog operations. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of one of the structures of the arc spark recognition and early warning system for mechanical dogs based on visual-infrared image data fusion processing provided in the embodiments of this application;

[0049] Figure 2 This is another structural schematic diagram of the arc spark recognition and early warning system for mechanical dogs based on visual-infrared image data fusion processing provided in the embodiments of this application;

[0050] Figure 3 This is a flowchart illustrating the arc spark recognition and early warning method for mechanical dogs based on visual-infrared image data fusion processing provided in this application embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as being processed sequentially, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. A process can be terminated when its operation is completed, but it may also have additional steps not included in the drawings. A process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0052] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0053] Traditional existing arc and spark detection technologies have several drawbacks when applied in power operation sites. These include insufficient sensitivity to detect sparks with small current fluctuations, inability to accurately locate fault points, high false alarm rates due to interference from changes in ambient light, easy misses of small-scale instantaneous events by single infrared thermal imagers, and the fact that most systems are fixed and lack intelligent response mechanisms for collaboration with mobile platforms. As a result, these technologies suffer from insufficient detection accuracy, environmental adaptability, and platform compatibility.

[0054] In this regard, such as Figure 1 and Figure 2As shown, this embodiment proposes an arc spark recognition and early warning system for a robotic dog based on visual-infrared image data fusion processing. The system is integrated into the robotic dog itself and mainly comprises: a multimodal sensing module for simultaneously acquiring RGB image sequences and thermal infrared image sequences from the power operation site; a temporal-spatial alignment module for performing timestamp synchronization, exposure signal alignment, and spatial registration based on calibration parameters on the received RGB and thermal infrared images to generate temporally and spatially consistent dual-modal image pairs; a visual feature extraction module for extracting brightness abrupt changes, regional flicker, and frequency domain flicker features from the aligned RGB images; and an infrared feature extraction module for extracting local hotspot regions and temperature gradient abrupt changes from the aligned thermal infrared images. Features include: a fusion judgment and false alarm suppression module, which has a built-in weighted fusion unit and a multi-frame confirmation logic unit, used to perform feature-level or decision-level fusion calculations on the input dual-modal features, and apply cross-modal mutual verification rules to perform joint judgments on multiple consecutive frames to output the final danger level signal; and a graded early warning and response module, whose input end receives the danger level signal and whose output end is connected to the robot dog's body control unit and remote communication unit, used to perform local audible and visual alarms, limit the robot dog's movement speed or emergency stop, and send alarm information and on-site image data to the remote monitoring center according to different danger levels.

[0055] It should be noted that the multimodal sensing module is configured to integrate multiple types of sensors to simultaneously acquire data from different modalities. For example, the module may include a camera for acquiring visible light images and an infrared imager for acquiring thermal infrared images. These sensors are fixedly mounted on the robot dog's head or top pan-tilt unit to ensure stable acquisition of on-site data during inspections.

[0056] The temporal-spatial alignment module processes data from different sensors, eliminating data inconsistencies caused by differences in acquisition time, exposure, and sensor location. This module uses timestamp synchronization, exposure signal alignment, and spatial registration to precisely match image data from different modalities in time and space, generating temporally and spatially consistent bimodal image pairs for subsequent processing.

[0057] The visual feature extraction module is designed to identify visual cues related to electric arcs and sparks from visible light images. These cues may include sudden changes in brightness, rapid flickering in specific areas, and frequency-domain flicker characteristics. This module quantizes and extracts these features using image processing algorithms.

[0058] The infrared feature extraction module focuses on identifying thermal cues related to electric arcs and sparks from thermal infrared images. This includes detecting localized hotspots in the image where the temperature is significantly higher than the surrounding environment, as well as abrupt temperature gradient changes occurring within a short period. This module identifies potential hazards by analyzing temperature distribution and variations.

[0059] The fusion judgment and false alarm suppression module is the core decision-making unit of the system. Its built-in weighted fusion unit is responsible for comprehensively processing the features or preliminary judgment results from the visual and infrared modalities. The multi-frame confirmation logic unit analyzes data from multiple consecutive frames and applies cross-modal mutual verification rules to reduce the possibility of false alarms from a single modality, thereby outputting a more reliable hazard level signal.

[0060] The graded early warning and response module executes corresponding early warning and response actions based on the hazard level signal output by the fusion judgment and false alarm suppression module. The output of this module is connected to the robot dog's main control unit and remote communication unit, enabling the robot dog to automatically adjust its behavior according to the degree of danger, such as issuing local alarms, limiting movement speed, or making emergency stops, and sending alarm information and on-site image data to the remote monitoring center.

[0061] In this embodiment, the system is designed to be directly mounted on the robot dog's body. For example, the various hardware components of the system, including sensors, processing units, and communication modules, can be installed in the robot dog's external structure or internal space using methods such as bolt fixing, snap-fit ​​connection, or integrated housing. This integration method allows the system to move with the robot dog, enabling mobile detection and early warning functions.

[0062] A multimodal sensing module is used to simultaneously acquire RGB image sequences and thermal infrared image sequences at power operation sites. For example, this module can consist of a standard visible light camera and an infrared thermal imager. These two sensors are mounted on the head or top pan-tilt unit of a robotic camera, acquiring images through their respective independent triggering mechanisms. The visible light camera captures image information within the visible spectrum, while the infrared thermal imager captures thermal radiation emitted from the object's surface. The acquired image data is transmitted to the subsequent processing module via their respective data lines. The multimodal sensing module includes an RGB camera, an infrared thermal imager, and a timer; the RGB camera is configured as a global shutter RGB camera, and the infrared thermal imager is configured as an uncooled microbolometer infrared thermal imager; the RGB camera and the infrared thermal imager are rigidly connected, and their optical axes remain parallel; the timer sends periodic hardware trigger signals to the RGB camera and the infrared thermal imager, ensuring that both sensors begin exposure within the same microsecond-level time window, achieving hardware-level synchronous acquisition of RGB image sequences and thermal infrared image sequences.

[0063] The temporal-spatial alignment module is used to synchronize timestamps, align exposure signals, and perform spatial registration based on calibration parameters on received RGB and thermal infrared images, generating temporally and spatially consistent bimodal image pairs. This module includes a temporal alignment submodule and a spatial alignment submodule: the temporal alignment submodule is configured to receive the timestamp of the hardware trigger signal from the timer and the image exposure signals from the RGB camera and the infrared thermal imager, and compensate for differences in image acquisition time caused by different exposure durations or data transmission delays based on the timestamp of the hardware trigger signal and the image exposure signals, thereby aligning the time of the bimodal image pairs; the spatial alignment submodule has built-in calibration parameters between the RGB camera and the infrared thermal imager obtained through a bi-target calibration method, and calculates a perspective transformation matrix based on the intrinsic and extrinsic parameters and distortion coefficients. It then resamples the thermal infrared image to the image coordinate system of the RGB camera, or resamples the RGB image to the image coordinate system of the infrared thermal imager, ensuring that the same physical scene point is located at the same pixel coordinate in both the RGB and thermal infrared images. For example, this module can receive image data from visible light cameras and infrared thermal imagers, and read the timestamp information inherent in each image. By comparing these timestamps and combining them with the compensation mechanism of the time alignment submodule, precise time synchronization is achieved; pixel-level spatial alignment is achieved through the perspective transformation matrix of the spatial alignment submodule.

[0064] The visual feature extraction module is used to extract brightness abrupt changes, regional flicker, and frequency domain flicker features from the aligned RGB image. For example, this module can convert the RGB image to grayscale and then detect brightness abrupt change regions by calculating the difference in pixel brightness values ​​between adjacent frames. Regional flicker features can be identified by statistically analyzing the changes in pixel values ​​of specific regions in the image across consecutive frames. Frequency domain flicker features can be identified by performing a Fourier transform on the image region and analyzing its frequency components. After these features are extracted, they can be combined to form a visual feature vector. Specifically, the visual feature extraction module employs a combination of image processing algorithms and neural networks. The image processing algorithm is configured to perform image enhancement based on histogram equalization and image dehazing preprocessing based on dark channel prior on the input RGB image. The preprocessed image is then converted to the HSV color space, and the V component is extracted as the luminance channel for temporal analysis. By calculating the absolute value of the luminance difference between the current frame and the previous frame and setting a dynamic threshold, regions of sudden luminance changes are detected. Multi-frame frequency domain analysis is performed on these luminance change regions, and a fast Fourier transform is used to detect the presence of high-frequency flickering features unique to electric arcs, thereby selecting candidate regions. The neural network is configured to perform secondary discrimination on the candidate regions using a pre-trained convolutional neural network and output the probability value that the candidate region is a real electric arc spark.

[0065] The infrared feature extraction module is used to extract local hotspot regions and temperature gradient abrupt change features from the thermal infrared images of the dual-modal image pair. For example, this module can perform non-uniformity correction and temperature calibration on the thermal infrared images of the dual-modal image pair, convert grayscale values ​​into accurate temperature values ​​and generate a temperature distribution matrix. Then, a threshold segmentation algorithm is used to extract connected regions above a set threshold of ambient temperature from the temperature distribution matrix and mark them as suspected hotspots. The highest temperature, average temperature, area, and temperature gradient of each suspected hotspot are calculated. Temperature abrupt change events are detected by comparing the rate of temperature change at the same spatial location between consecutive frames. Finally, the temperature statistics, geometric features, and abrupt change markers of the hotspot regions are combined to form an infrared feature vector, which is output to the fusion judgment and false alarm suppression module.

[0066] The fusion judgment and false alarm suppression module is used to obtain dual-modal features from the visual feature extraction module and the infrared feature extraction module, and to perform feature-level or decision-level fusion calculation on the dual-modal features through the built-in weighted fusion unit to generate a preliminary danger probability; then, through the built-in multi-frame confirmation logic unit, cross-modal mutual verification rules are applied to jointly judge the preliminary danger probability and dual-modal features of multiple consecutive frames, and output a danger level signal. Specifically, the weighted fusion unit is configured to execute at least one of a feature-level fusion strategy and a decision-level fusion strategy. The feature-level fusion strategy is configured to concatenate the feature vectors output by the visual feature extraction module and the infrared feature extraction module before inputting them into the classifier, assign learnable weights to different modal features, and then input them into a lightweight fully connected neural network for joint classification. The decision-level fusion strategy is configured to independently classify the feature vectors output by the visual feature extraction module and the infrared feature extraction module to obtain a visual confidence score and an infrared confidence score, and then perform a weighted average of the two scores to obtain a fusion score. The multi-frame confirmation logic unit is configured to apply cross-modal mutual verification rules based on the fusion result output by the weighted fusion unit: when a single modality detects a suspected event, another modality must confirm it in the same spatial region within the next 1 to 3 frames, or the single modality itself must continuously alarm for N frames to determine it as a real dangerous event. The value of N can be dynamically set according to the false alarm tolerance, and the value of N ranges from 1 to 5. For example, the weighted fusion unit can select a decision-level fusion strategy to adjust the confidence weights of the visual and infrared channels according to the current ambient lighting conditions; the multi-frame confirmation logic unit is set to N=3, and when a suspected event is detected in three consecutive frames in a single mode, it is determined to be a real danger.

[0067] The graded early warning and response module is used to execute corresponding early warning and response actions based on different hazard levels. The hazard level signal of the graded early warning and response module is determined by the fusion confidence level, bimodal confirmation status, and hazard event duration output by the fusion judgment and false alarm suppression module, combined with the distance determination result between the robot dog and the hazard source. The hazard level signal and the response mechanism correspond one-to-one, including:

[0068] Level 1 Warning: When the detection result is low confidence, or when a suspected event is detected in only a single modality and the mutual verification conditions of the cross-modal mutual verification rules are not met, it is determined to be a Level 1 warning signal; the response mechanism is to trigger the LED indicator light of the robot dog to give a yellow warning, without interfering with the normal operation of the robot dog, and to remind on-site personnel to pay attention to potential risks;

[0069] Level 2 Warning: When the detection result meets the mutual verification conditions of the cross-modal mutual verification rules, and the degree of danger is moderate or the distance between the danger source and the robot dog is less than the first distance threshold, it is determined to be a Level 2 warning signal; the response mechanism is that the LED indicator of the robot dog body turns red and the buzzer is activated to emit a warning sound; the robot dog automatically limits its maximum movement speed and maintains the current working direction; and sends regular priority alarm information to the remote monitoring center, including the warning level, real-time location and on-site image data;

[0070] Level 3 Response: When the detection result is of high confidence, the dangerous event persists, or the distance between the hazard source and the robot dog is less than the second distance threshold, a Level 3 response signal is generated. The response mechanism is that the robot dog immediately executes an emergency stop procedure, locks the motion system, the LED indicator light remains constantly red, and the buzzer continuously emits an alarm sound. A highest priority alarm signal is sent to the remote monitoring center, and the alarm-related dual-modal image data and feature extraction logs are transmitted back, maintaining real-time data transmission with the remote monitoring center until a manual reset command is received. The first and second distance thresholds can be set based on empirical formulas or experimental test data.

[0071] The distance determination result between the robotic dog and the hazard source can be obtained through visual-infrared fusion ranging. For example, using visual-infrared fusion ranging, based on spatially registered dual-modal image pairs, the pixel width / height of the arc spark hazard source in the RGB image is obtained. Combined with the calibrated intrinsic focal length f of the RGB camera, the distance is calculated using the monocular ranging formula D=(W×f) / w, where D is the actual distance between the robotic dog and the hazard source, W is the actual characteristic size of the arc spark hazard source in the power scene (preset to be the conventional size of arc sparks in power equipment, 0.1-0.5 meters), and w is the pixel width of the hazard source in the RGB image. At the same time, the hot spot pixel coordinates of the infrared thermal imager are matched with the characteristic pixel coordinates of the RGB image to verify the accuracy of the ranging result.

[0072] The graded early warning and response module receives hazard level signals at its input and connects to the robot's main control unit and remote communication unit at its output. It is used to execute local audible and visual alarms, limit the robot's movement speed or cause an emergency stop, and send alarm information and on-site image data to a remote monitoring center based on different hazard levels. For example, based on the received hazard level signal, the module can control the LEDs on the robot to emit different colors or flashing frequencies, and emit alarm sounds of varying volumes via a buzzer. For the robot's movement control, commands can be sent to the main control unit to reduce its movement speed or stop it completely. Simultaneously, the module can send alarm information and on-site image data to a remote monitoring center via the remote communication unit, such as through a wireless network.

[0073] This system also includes an inertial dual-modal wake-up strategy management unit, used to control whether the system operates in a first working mode or a second working mode. In the default first working mode, only the visual feature extraction module is controlled to be active, while the infrared feature extraction module and the fusion judgment and false alarm suppression module are controlled to be in a dormant state. When the visual feature extraction module detects a brightness mutation event for multiple consecutive frames, and the characteristics of the event match a preset preliminary arc spark model, the inertial dual-modal wake-up strategy management unit switches the system to the second working mode. In the second working mode, the infrared feature extraction module and the fusion judgment and false alarm suppression module are awakened to work together. If the danger level signal does not reach the preset condition within a preset time, the inertial dual-modal wake-up strategy management unit switches the system back to the first working mode. Furthermore, the robotic dog body includes a navigation and positioning system, and the lazy dual-modal wake-up strategy management unit is connected to the navigation and positioning system to obtain the real-time location and task information of the robotic dog, and accordingly determine whether the robotic dog is in a preset high-risk area or a known safe area, and whether the robotic dog is performing a non-detection task. When the robotic dog enters the preset high-risk area, the lazy dual-modal wake-up strategy management unit controls the system to operate in the second working mode; when the robotic dog is in a known safe area or performing a non-detection task, the lazy dual-modal wake-up strategy management unit controls the system to operate in the first working mode. For example, when the robotic dog enters a high-risk area near a substation switchgear, the lazy dual-modal wake-up strategy management unit automatically switches to the second working mode and initiates dual-modal joint detection; if no danger level signal is detected within 10 seconds, it switches back to the first working mode.

[0074] This system effectively overcomes the limitations of single sensors, such as susceptibility to environmental interference, insufficient detection sensitivity, or easy missed detections, by integrating visual and infrared multimodal sensing and performing precise temporal-spatial alignment. Through the fusion and judgment of dual-modal features and multi-frame cross-verification, the false alarm rate is significantly reduced, improving the detection accuracy and robustness of electric arcs and sparks at power operation sites. Furthermore, the tiered early warning and intelligent response mechanism enables the robotic dog to dynamically adjust its behavior according to the level of danger and report information promptly, ensuring the safety and efficiency of mobile operations.

[0075] like Figure 3 As shown, this embodiment also proposes an arc spark recognition and early warning method for mechanical dogs based on visual-infrared image data fusion processing. This method is based on the aforementioned system, and its specific implementation steps include:

[0076] S1. The RGB image sequence and thermal infrared image sequence of the power operation site are synchronously acquired through the multimodal sensing module; the periodic hardware trigger signal is sent to the RGB camera and the infrared thermal imager at the same time through the timer to ensure that the two sensors start exposure within the same microsecond time window, so as to realize the hardware-level synchronous acquisition of the RGB image sequence and the thermal infrared image sequence.

[0077] S2. The RGB image sequence and the thermal infrared image sequence are time-stamped, exposure signal aligned, and spatially registered by the time-space alignment module to generate the temporally and spatially consistent dual-modal image pair.

[0078] S3. Extract brightness abrupt changes, regional flicker, and frequency domain flicker features from the RGB image of the dual-modal image pair using the visual feature extraction module;

[0079] S4. Extract local hotspot regions and temperature gradient abruptness features from the thermal infrared image of the dual-modal image pair using the infrared feature extraction module;

[0080] S5. Through the fusion judgment and false alarm suppression module, dual-modal features are obtained from the visual feature extraction module and the infrared feature extraction module. Feature-level or decision-level fusion calculations are performed on the dual-modal features to generate a preliminary danger probability. The cross-modal mutual verification rule is applied to jointly judge the preliminary danger probability and the dual-modal features in multiple consecutive frames, and output a danger level signal. If a feature-level fusion strategy is adopted during fusion judgment, the dual-modal feature vectors are concatenated and assigned learnable weights, and then input into a lightweight fully connected neural network for joint classification. If a decision-level fusion strategy is adopted, the confidence scores of the two channels are weighted and the weights are dynamically adjusted according to the ambient lighting conditions. The joint judgment follows a multi-frame confirmation rule: after a single modality detects a suspected event, another modality must confirm it in the same spatial area within 1 to 3 subsequent frames, or a single modality must continuously alarm for 1 to 5 frames, before it is determined to be a real dangerous event.

[0081] S6. The graded warning and response module executes corresponding warning and response actions according to different hazard levels; wherein, the warning and response actions include local audible and visual alarms, limiting or stopping the mechanical dog's movement speed, and sending alarm information and on-site image data to the remote monitoring center.

[0082] Through the above technical solution, this method achieves efficient arc and spark detection on the mobile platform of the robotic dog, significantly reducing the false alarm rate, improving the detection rate of instantaneous small-scale events, and dynamically adjusting the response strategy according to the hazard level to ensure operational safety. Furthermore, the system, combined with an inertial dual-modal wake-up strategy management unit, optimizes the power consumption management of the robotic dog platform and extends its battery life.

[0083] When acquiring multimodal images at power work sites, although the system aims to simultaneously acquire RGB image sequences and thermal infrared image sequences, relying solely on software-level timestamps or coarse triggering mechanisms may result in slight time discrepancies between different sensors. Step S1, through hardware-level synchronous acquisition, ensures the temporal consistency of the dual-modal images, laying the foundation for subsequent feature extraction and fusion analysis.

[0084] In the fusion judgment stage, the flexible fusion strategy selection and dynamic weight adjustment in step S5 enable the system to adapt to different ambient lighting conditions; the strict multi-frame confirmation rules effectively filter out instantaneous noise and occasional interference, further improving the reliability of the detection results.

[0085] The above embodiments significantly improve the detection accuracy and robustness of the arc spark recognition and early warning system and method based on visual-infrared image data fusion processing for robotic dogs. Feature-level fusion automatically mines complex correlations between modes through deep learning, giving the system a stronger pattern recognition capability when identifying arc sparks. The dynamic weight adjustment mechanism of decision-level fusion enables the system to intelligently adapt to changes in ambient lighting, maintaining optimal detection performance under different operating conditions and effectively avoiding missed or false alarms caused by limitations of a single mode. In addition, the refined multi-frame confirmation rules and hierarchical early warning response mechanism greatly enhance the system's ability to identify real dangerous events and its response flexibility, ensuring the timeliness and accuracy of early warnings and providing more reliable technical support for safety inspections at power operation sites.

[0086] In some embodiments, the multimodal sensing module includes a global shutter RGB camera and an uncooled microbolometer infrared thermal imager, which are fixed by a rigid structure and keep their optical axes parallel; the multimodal sensing module also includes a timer that simultaneously sends periodic hardware trigger signals to the RGB camera and the infrared thermal imager.

[0087] The global shutter RGB camera is an image sensor whose pixels begin and end exposure simultaneously, capturing a fleeting image of the entire scene within a single exposure cycle. This characteristic effectively avoids image distortion caused by the rolling shutter effect when shooting high-speed moving objects, ensuring that the acquired RGB image accurately reflects the instantaneous state of the scene during moments such as mechanical dog movement or electric arcs, providing high-quality raw data for visual feature extraction. The uncooled microbolometer infrared thermal imager is an infrared detector that operates without cryogenic cooling, imaging by sensing minute temperature changes caused by infrared radiation. It can acquire the surface temperature distribution of an object in real time and convert it into a thermal infrared image. This type of thermal imager has advantages such as small size, low power consumption, and fast start-up, making it suitable for integration into mobile platforms such as mechanical dogs for detecting infrared features such as localized overheating and abnormal discharge effects in electrical equipment. It can operate effectively even in low-light or smoky environments. Rigid structure fixing refers to firmly connecting the global shutter RGB camera and the uncooled microbolometer infrared thermal imager together using robust mechanical components to prevent relative displacement or vibration between them. Maintaining parallel optical axes means that the optical central axes of the two sensors are parallel to each other, ensuring that they ideally observe approximately the same scene area and have similar perspectives. This physical configuration greatly simplifies subsequent spatial registration, reduces registration errors introduced by changes in the relative positions of the sensors, and improves the spatial consistency of the dual-modal image pair. A timer is an electronic device capable of generating precise and stable clock signals, with output signals exhibiting extremely high frequency stability and timing accuracy. In this system, it serves as the master clock source, generating periodic hardware trigger signals to ensure that all connected sensors can operate synchronously at preset precise time intervals. The periodic hardware trigger signal is an electrical signal generated by the timer and directly sent to the global shutter RGB camera and the uncooled microbolometer infrared thermal imager. When a sensor receives this trigger signal, it immediately begins its image acquisition process. This hardware-level triggering mechanism bypasses the uncertainties and delays at the operating system and software levels, ensuring that the two sensors synchronously start exposure within the same microsecond-level time window, thereby achieving high-precision image acquisition time synchronization.

[0088] In the arc spark recognition and early warning system used by mechanical dogs based on visual-infrared image data fusion processing, the RGB image sequence and thermal infrared image sequence acquired synchronously by the multimodal sensing module may have slight deviations in time and geometric distortions and field of view differences in space due to the inherent differences in the working principles of different sensors, such as exposure time, data transmission rate and optical characteristics.

[0089] In this embodiment, the time-space alignment module specifically includes a time alignment submodule and a spatial alignment submodule. The time alignment submodule receives the timestamp from the hardware trigger signal of the timer and the image exposure signals from the RGB camera and the infrared thermal imager to compensate for differences in image acquisition time caused by different sensor exposure durations or data transmission delays, thus forming a dual-modal image pair. The spatial alignment submodule has built-in intrinsic and extrinsic parameters and distortion coefficients between the RGB camera and the infrared thermal imager obtained through a dual-target alignment method. It resamples the thermal infrared image to the image coordinate system of the RGB camera by applying a perspective transformation matrix, or vice versa, so that the same physical scene point is located at the same pixel coordinate in both images.

[0090] The time alignment submodule is a key component of the time-space alignment module. Its main function is to eliminate or compensate for differences in image acquisition time caused by different sensor characteristics. Specifically, this submodule receives the timestamp of the hardware trigger signal from the timer and the image exposure signals from the RGB camera and the infrared thermal imager. The hardware trigger timestamp is typically issued by the timer, ensuring that the RGB camera and the infrared thermal imager begin exposure at the same moment, thus achieving initial time synchronization at the hardware level. The image exposure signal provides information on the actual exposure duration of the sensors. By analyzing this information, the time alignment submodule performs fine-grained compensation for differences in image acquisition time caused by different sensor exposure durations or data transmission delays. For example, it can resample the images on the timeline using interpolation algorithms, or precisely match the nearest image frames based on the timestamps, to ensure that the RGB image and the thermal infrared image that ultimately constitute the dual-modal image pair are highly consistent in time, representing the scene state at the same instant.

[0091] The spatial alignment submodule is another core component of the temporal-spatial alignment module, responsible for resolving geometric inconsistencies between images of different modalities. This submodule incorporates the intrinsic and extrinsic parameters and distortion coefficients between the RGB camera and the infrared thermal imager, obtained through a bi-target calibration method. The bi-target calibration method typically involves taking images at different positions and angles using a specific calibration board (such as a checkerboard). By analyzing the projection of the calibration board into the two images, the intrinsic parameters (focal length, principal point, distortion coefficients) of each camera and the extrinsic parameters (rotation matrix and translation vector) between the two cameras are calculated. Based on these precise calibration parameters, the spatial alignment submodule can construct a perspective transformation matrix. By applying this perspective transformation matrix, the thermal infrared image can be resampled to the image coordinate system of the RGB camera, or vice versa, the RGB image can be resampled to the image coordinate system of the infrared thermal imager. The resampling process typically employs bilinear interpolation or bicubic interpolation to ensure a smooth transition of image pixel values. Ultimately, this process ensures that points of the same physical scene lie at the same pixel coordinates in both images, achieving pixel-level spatial alignment.

[0092] In this embodiment, the visual feature extraction module combines image processing algorithms with neural networks. It first performs image enhancement based on histogram equalization and image dehazing preprocessing based on dark channel prior on the input RGB image. Then, it converts the image to the HSV color space and extracts the V component as the luminance channel for temporal domain analysis. It detects regions with abrupt changes in luminance by calculating the absolute value of the luminance difference between the current frame and the previous frame and setting a dynamic threshold. At the same time, it performs multi-frame frequency domain analysis on suspected regions and detects the high-frequency flicker features unique to electric arcs through fast Fourier transform. Finally, it uses a pre-trained lightweight convolutional neural network to perform secondary discrimination on candidate regions and outputs the probability value of them being real electric arc sparks.

[0093] Specifically, the combination of image processing algorithms and neural networks aims to reduce computational complexity and resource consumption while ensuring detection accuracy, enabling efficient operation on the embedded processing platform of the robotic dog. Image processing algorithms typically refer to traditional image processing methods with low computational cost and easy parallel processing, such as filtering and thresholding. Neural networks, on the other hand, refer to deep learning models with fewer parameters and layers and a compact structure. Their design goal is to achieve specific recognition or classification tasks with limited computational resources, such as lightweight convolutional neural networks like MobileNet and ShuffleNet. Combining the two can fully utilize the efficiency of traditional algorithms and the generalization ability of neural networks.

[0094] The image enhancement based on histogram equalization and the image dehazing preprocessing based on dark channel prior are described above. Histogram equalization is a commonly used image enhancement technique that redistributes the pixel gray levels of an image, making the gray-level histogram flatter and increasing image contrast. This is particularly effective in low-light or overexposed scenes, significantly improving visual quality and detail. Dark channel prior-based image dehazing is an effective dehazing algorithm. Its basic principle is that in most non-sky areas of the image, at least one color channel has a low pixel value. By estimating and removing the haze component in the image, image clarity and contrast can be significantly improved, eliminating the interference of haze on visual feature extraction.

[0095] The process involves converting the image to the HSV color space and extracting the V component as the luminance channel for temporal analysis. The HSV (Hue, Saturation, Value) color space is a color model that separates color and luminance information. The V component (Value) represents luminance, which is independent of color and saturation, and can more directly reflect the brightness changes of the image. In temporal analysis, processing only the V component effectively avoids interference from color changes in the detection of abrupt luminance changes, simplifies calculations, and improves sensitivity to luminance features such as flicker.

[0096] The method involves calculating the absolute value of the brightness difference between the current frame and the previous frame and setting a dynamic threshold to detect regions of sudden brightness changes. Brightness difference is an effective method for detecting rapid brightness changes in an image. By calculating the difference between the V components of corresponding pixels in two consecutive frames and taking their absolute value, areas with drastic brightness changes can be highlighted. The dynamic threshold is not fixed but adjusted in real time based on factors such as ambient lighting conditions, overall image brightness distribution, or historical data to adapt to brightness changes in different scenarios, thereby more accurately identifying the instantaneous intense light generated by electric arcs or sparks.

[0097] The process involves multi-frame frequency domain analysis of suspected areas and detection of high-frequency flicker characteristics unique to electric arcs using Fast Fourier Transform (FFT). Electric arcs and sparks produce rapid and irregular flickering during combustion, which manifests as high-frequency changes in brightness in the time domain. By performing multi-frame frequency domain analysis on suspected areas detected by brightness abrupt changes—specifically, by applying a Fast Fourier Transform (FFT) to the brightness signal of that area over time—the time domain can be transformed into the frequency domain. In the frequency domain, the high-frequency flicker characteristics unique to electric arcs will manifest as specific frequency components. By detecting the presence and intensity of these frequency components, it can be further confirmed whether the suspected area is a genuine electric arc or spark.

[0098] The process employs a pre-trained lightweight convolutional neural network (CNN) to perform secondary discrimination on candidate regions, outputting a probability value indicating whether it is a real electric arc spark. CNNs possess powerful feature learning and classification capabilities in image recognition. Here, a pre-trained lightweight CNN is used, with an optimized model structure, small parameter count, and fast inference speed, making it suitable for deployment on resource-constrained robotic systems. This CNN receives the initially detected candidate regions as input, performing deeper feature extraction and pattern recognition to distinguish real electric arc sparks from background noise or interference of similar brightness. Finally, the CNN outputs a probability value representing the confidence level that the candidate region is a real electric arc spark, providing a quantitative basis for subsequent fusion judgment.

[0099] In some implementations, the infrared feature extraction module is used to perform non-uniformity correction and temperature calibration on the thermal infrared images of the dual-modal image pair, convert grayscale values ​​into accurate temperature values ​​and generate a temperature distribution matrix, and then use a threshold segmentation algorithm to extract connected regions above a set threshold of ambient temperature from the temperature distribution matrix and mark them as suspected hot spots. The highest temperature, average temperature, area and temperature gradient of each suspected hot spot are calculated, and temperature change events are detected by comparing the rate of temperature change at the same spatial location between consecutive frames. Finally, the temperature statistics, geometric features and change indicators of the hot spot area are combined to form an infrared feature vector and output to the fusion judgment and false alarm suppression module.

[0100] Specifically, after receiving the aligned thermal infrared image, the infrared feature extraction module first performs non-uniformity correction and temperature calibration on the thermal infrared image of the dual-modal image pair. Non-uniformity correction aims to eliminate the differences in the response characteristics of each detection unit in the infrared detector array, ensuring that objects of the same temperature in the image present the same grayscale value at different locations, thereby improving the uniformity of the image and the accuracy of subsequent temperature measurements. Common correction methods include single-point or two-point correction based on a blackbody radiation source. Temperature calibration converts the raw grayscale values ​​output by the infrared thermal imager into accurately measured temperature values ​​with physical meaning, such as degrees Celsius or Kelvin, using a pre-established mathematical model or lookup table.

[0101] Subsequently, the infrared feature extraction module employs a threshold segmentation algorithm to extract connected regions from the temperature distribution matrix that exceed a set threshold for ambient temperature, and marks them as potential hotspots. The threshold segmentation algorithm divides pixels in the temperature distribution matrix into different regions by setting one or more temperature thresholds. Its main purpose is to separate pixels with temperatures above a specific threshold from the background. Considering that electric arcs are often accompanied by a sharp increase in local temperature, this threshold is typically set to a fixed or dynamic value higher than the current ambient or background temperature to filter out normal temperature areas in the environment and focus on potentially abnormally high-temperature areas. A connected region refers to a set of pixels in an image whose pixel values ​​satisfy specific conditions and are adjacent to each other. Extracting connected regions helps cluster discrete, potentially noise-induced, individual high-temperature pixels into meaningful "hotspot" entities and eliminates isolated noise points.

[0102] For each connected region marked as a suspected hotspot, the infrared feature extraction module calculates its maximum temperature, average temperature, area, and temperature gradient. The maximum temperature reflects the most intense thermal effect within the hotspot area and is a crucial indicator of hazard level. The average temperature provides the overall temperature level of the hotspot area, helping to assess its persistence and scale. The area represents the spatial extent of the hotspot in the image and reflects the size of the electric arc. The temperature gradient describes the intensity and direction of temperature changes within the hotspot area; a high temperature gradient typically indicates concentrated heat sources and rapid energy release, a key characteristic of electric arcs. These statistical and geometric features together provide a multi-dimensional quantitative description of each suspected hotspot, helping to distinguish real hazardous events from ordinary heat sources.

[0103] Furthermore, the infrared feature extraction module detects temperature abrupt events by comparing the rate of temperature change at the same spatial location across consecutive frames. Electric arc sparks are instantaneous and dynamic physical phenomena, characterized by rapid temperature increases and potential fluctuations. The rate of temperature change is obtained by calculating the difference between the temperature of a pixel in the current frame and the corresponding pixel in the previous frame, divided by the time interval. When this rate of change exceeds a preset threshold, it is identified as a temperature abrupt event. This time-series analysis method effectively distinguishes between static high-temperature objects (such as normally heating equipment) and dynamic electric arc sparks, significantly reducing false alarms.

[0104] Finally, the temperature statistics data such as the highest temperature, average temperature, area, and temperature gradient obtained from the above calculations, as well as the mutation flag indicating whether a temperature mutation event exists, are integrated into a unified infrared feature vector and output to the fusion judgment and false alarm suppression module.

[0105] In some implementations, the fusion judgment and false alarm suppression module has a built-in weighted fusion unit and a multi-frame confirmation logic unit, which are used to perform feature-level or decision-level fusion calculation on the input dual-modal features, and apply the cross-modal mutual verification rules to perform joint judgment on multiple consecutive frames to output the final danger level signal.

[0106] To enhance the system's adaptability to different scenarios and detection requirements, the fusion judgment and false alarm suppression module provides two flexibly selectable fusion strategies. This allows the system to choose the most suitable fusion method based on the characteristics of the actual application scenario, such as different emphases on detection speed, accuracy, robustness, or computational resource consumption, thereby optimizing overall performance.

[0107] Feature-level fusion is a deep fusion method that involves concatenating the original feature vectors output by the visual feature extraction module and the infrared feature extraction module before feeding them into the classifier for judgment. During the concatenation process, learnable weights can be assigned to features from different modalities. These weights can be automatically adjusted during model training using optimization algorithms such as backpropagation to reflect the contribution of different modal features to the final classification result. The concatenated high-dimensional feature vector is then input into a lightweight fully connected neural network for joint classification. This fusion method allows the neural network to learn deeper and more complex correlations between features from different modalities, thereby achieving information complementarity and enhancement at the feature level and improving the accuracy and robustness of arc and spark event recognition.

[0108] Decision-level fusion is a more flexible approach. It involves the visual and infrared channels independently making preliminary judgments about their respective features and outputting their confidence scores for the occurrence of an event. For example, the visual channel might output a probability value indicating the presence of an electric arc, while the infrared channel might output a probability value indicating the presence of a hot spot. Subsequently, the fusion judgment and false alarm suppression module performs a weighted average of these two independent confidence scores to obtain the final fused confidence score. It's worth noting that the weights of these weighted averages can be dynamically adjusted based on ambient lighting conditions. For example, in well-lit environments, the weight of the visual channel can be appropriately increased; while in low-light or smoky environments, the weight of the infrared channel can be increased to ensure the system maintains high detection performance in various complex environments.

[0109] Furthermore, the multi-frame confirmation logic unit aims to significantly reduce the false alarm rate of the system, its core being the introduction of a cross-verification mechanism between time and modality. This unit sets strict judgment rules: when a single modality (e.g., vision or infrared) initially detects a suspected arc or spark event, the system will not immediately alarm, but will require one of the following two conditions to be met before ultimately determining it as a real dangerous event. The first condition is that another modality must be within the same or overlapping spatial area and confirm the event in subsequent 1 to 3 frames of image data. This cross-modal mutual verification mechanism utilizes the complementarity of visual and infrared modalities; for example, vision may detect a flash, while infrared confirms the existence of a local hotspot, thus effectively eliminating transient interference from a single modality. The second condition is that if a single modality itself continuously emits alarm signals for N consecutive frames, it is also determined to be a real dangerous event. This is mainly used to confirm persistent events or to provide redundant confirmation when another modality is temporarily disabled or limited. The N value in the aforementioned multi-frame confirmation logic unit, i.e., the frame threshold for continuous alarms by a single modality, is not fixed but can be dynamically set according to the tolerance for false alarms based on the actual application scenario. For example, in scenarios with extremely high security requirements and low tolerance for false alarms, the N value can be set larger to further reduce false alarms; while in scenarios with high requirements for response speed and where a small number of false alarms is acceptable, the N value can be set smaller to speed up the response. This dynamic setting mechanism allows the system to flexibly adapt to different operating environments and security policies, achieving the best balance between false alarm suppression and response speed.

[0110] In some implementations, the value of N ranges from 1 to 5. This N value, representing the number of consecutive alarm frames in a single mode, is limited to a range of 1 to 5. This range is designed to provide the system with a proven, optimized interval that balances detection sensitivity and false alarm suppression. When N is set to 1, the system responds immediately to any suspected event detected in a single frame. This results in extremely high sensitivity and the fastest response speed for transient, sudden arc or spark events, making it suitable for applications with extremely high real-time requirements. As the N value increases, for example, to 3 or 5, the system requires the single-mode alarm signal to persist for a longer period before being confirmed as a genuine hazardous event. This mechanism effectively enhances the system's robustness to occasional noise, brief interference, or atypical flickering, significantly reducing the false alarm rate and thus improving system stability. By limiting the N value to the range of 1-5, the system can flexibly select appropriate confirmation strategies under different power operating environments and risk preferences to adapt to different needs, from high sensitivity and rapid response to high robustness and low false alarms.

[0111] In some implementations, the hazard level signal of the graded early warning and response module is jointly determined by the fusion confidence level, dual-modal confirmation status, and hazard event duration output by the fusion judgment and false alarm suppression module, combined with the distance judgment result between the robot and the hazard source. The hazard level signal and response mechanism correspond one-to-one, including: Level 1 warning: When the detection result is low confidence, or only a single modality detects a suspected event and the mutual verification conditions of the cross-modal mutual verification rules are not met, it is determined to be a Level 1 warning signal; the response mechanism is to trigger the LED indicator of the robot body to flash a yellow warning, without interfering with the normal operation of the robot, prompting on-site personnel to pay attention to potential risks; Level 2 warning: When the detection result meets the mutual verification conditions of the cross-modal mutual verification rules, and the hazard level is moderate or the distance between the hazard source and the robot is less than a first distance threshold, it is determined to be a Level 2 warning. Level 2 warning signal: The response mechanism is that the robot dog's LED indicator light turns red, and the buzzer sounds an alarm; the robot dog automatically limits its maximum movement speed and maintains its current working direction; it sends regular priority alarm information to the remote monitoring center, including the warning level, real-time location, and on-site image data; Level 3 response: When the detection result is of high confidence, the dangerous event persists, or the distance between the dangerous source and the robot dog is less than the second distance threshold, it is determined to be a Level 3 response signal; The response mechanism is that the robot dog immediately executes an emergency stop procedure and locks the motion system; the LED indicator light remains constantly red, and the buzzer continuously sounds an alarm; it sends the highest priority alarm signal to the remote monitoring center, transmits back the alarm-related dual-modal image data and feature extraction logs, and maintains real-time data transmission with the remote monitoring center until a manual reset command is received.

[0112] This tiered early warning and response module aims to classify potential hazards into different levels based on the severity, confidence level, and persistence of detected arc or spark events. This tiered mechanism enables the system to take differentiated and more precise response measures to different threat levels, avoiding a "one-size-fits-all" approach, thereby optimizing the allocation of early warning resources and improving the overall system's intelligence and response efficiency.

[0113] The Level 1 warning is the lowest level of alert, primarily targeting suspected events with low confidence detected by the system in their initial stages, or anomalies detected only briefly by a single modality (such as vision or infrared). At this point, the system's assessment of the degree of danger is unclear or the risk is low, so it only provides a warning by flashing a yellow LED indicator on the robot dog itself. This lightweight warning method aims to attract the initial attention of on-site personnel while avoiding excessive interference with the robot dog's normal operation due to false alarms, allowing time for further confirmation or manual intervention.

[0114] A Level 2 warning indicates that the system has confirmed the presence of an electric arc or spark event through cross-verification using both visual and infrared sensors. However, based on its characteristics (such as intensity, duration, and distance from the robot dog), the system assesses the hazard level as moderate, or that the hazard source is still far from the robot dog. In this case, in addition to turning the LED indicator red to show a higher level of warning, the system will automatically limit the robot dog's maximum movement speed to reduce potential risks and provide operators with more reaction time. Simultaneously, an audible alarm will further enhance the alertness of on-site personnel, prompting them to verify the situation.

[0115] Level 3 response is the highest level of emergency handling, addressing a high-confidence, persistent, and severe arc or spark hazard detected by the system and located very close to the robot. In this situation, the system determines the danger is imminent and requires immediate and decisive action. The robot will immediately execute an emergency stop procedure to prevent further approach to the hazard or damage to itself. Simultaneously, the system will send a high-priority alarm signal to the remote monitoring center, ensuring that monitoring personnel receive notification and intervene immediately. To facilitate post-event analysis and manual verification, the system will also transmit multi-frame dual-modal image data and feature extraction logs before and after the alarm trigger, providing comprehensive on-site information.

[0116] In some implementations, the inertial dual-modal wake-up strategy management unit is used to control whether the system operates in a first working mode or a second working mode, wherein: in the default first working mode, only the visual feature extraction module is controlled to be in working state, while the infrared feature extraction module and the fusion judgment and false alarm suppression module are controlled to be in sleep state; when the visual feature extraction module detects a brightness change event for multiple consecutive frames, and the features of the event match the preset preliminary model of electric arc spark, the inertial dual-modal wake-up strategy management unit switches the system to the second working mode; in the second working mode, the infrared feature extraction module and the fusion judgment and false alarm suppression module are woken up to work together; if the danger level signal does not reach the preset condition within a preset time, the inertial dual-modal wake-up strategy management unit switches the system back to the first working mode.

[0117] The lazy dual-modal wake-up strategy management unit is a core control component. Its main function is to dynamically manage the operating status of different detection modules in the system based on environmental conditions and preliminary detection results, thereby optimizing energy consumption. Specifically, this unit can be an independent microcontroller, a specific software module running on the mechanical dog's main control unit, or a combination of hardware and software. It is responsible for monitoring the output of the vision module and controlling the power supply or the start / stop of the processing threads of the infrared feature extraction module and the fusion judgment and false alarm suppression module according to a preset strategy.

[0118] When the system is in default or non-alarm state, the lazy bimodal wake-up strategy management unit only activates the visual feature extraction module and its associated RGB camera. The visual feature extraction module typically has relatively low power consumption and can continuously perform preliminary analysis of RGB images at the power operation site to detect potential anomalies. At this time, the infrared feature extraction module and the fusion judgment and false alarm suppression module are in sleep or low-power mode and do not perform data processing, thereby significantly reducing the overall energy consumption of the system.

[0119] This is the key trigger condition for the system to switch from low-power mode to high-precision detection mode. During continuous monitoring, the visual feature extraction module analyzes the RGB image sequence in real time. When it detects areas with abrupt brightness changes in the image for multiple consecutive frames (e.g., 3 to 5 consecutive frames), this indicates the possible presence of an electric arc or spark. Simultaneously, this brightness abrupt event undergoes preliminary feature matching, for example, using a lightweight classifier or rule-based algorithm to determine whether it conforms to a preset model of the spatial morphology, temporal duration, or flicker frequency characteristic of an electric arc or spark. Only when both conditions are met simultaneously will the lazy dual-modal wake-up strategy management unit send an interrupt signal to the system controller, requesting the system to enter a higher-level detection state.

[0120] Once the system controller receives an interrupt signal from the inertial dual-modal wake-up strategy management unit, it will immediately activate the infrared feature extraction module and the fusion judgment and false alarm suppression module, which are in a dormant state. At this time, the infrared thermal imager begins acquiring thermal infrared images, and the infrared feature extraction module begins processing these images to extract hotspot and temperature gradient information. Subsequently, the fusion judgment and false alarm suppression module fuses the features from the visual and infrared modalities for a more comprehensive and accurate joint diagnosis to determine whether a real arc or spark hazard exists and outputs the corresponding hazard level signal.

[0121] To avoid prolonged high-power states due to brief false alarms or non-continuous events, the lazy dual-modal wake-up strategy management unit sets a preset confirmation time. After waking up the infrared and fusion modules, if the fusion judgment and false alarm suppression module fails to confirm the existence of a real dangerous event (i.e., does not output a continuous danger level signal) within this preset time (e.g., 5 to 10 seconds), the lazy dual-modal wake-up strategy management unit will automatically instruct the system master controller to cause the infrared feature extraction module and the fusion judgment and false alarm suppression module to re-enter a sleep or low-power state, and the system will return to a low-power mode where only the visual feature extraction module performs monitoring.

[0122] In some implementations, the lazy bimodal wake-up strategy management unit is communicatively connected to the robot dog's navigation and positioning system, enabling it to acquire the robot dog's real-time location and task information, and thereby determine whether the robot dog is in a preset high-risk area or a known safe area, and whether the robot dog is performing a non-detection task. When the robot dog enters the preset high-risk area, the lazy bimodal wake-up strategy management unit controls the system to operate in the second working mode; when the robot dog is in a known safe area or performing a non-detection task, the lazy bimodal wake-up strategy management unit controls the system to operate in the first working mode.

[0123] Specifically, the lazy dual-modal wake-up strategy management unit is the core component responsible for managing system power consumption and operating modes. By establishing a communication connection with the robot dog's navigation and positioning system, this unit can acquire the robot dog's precise location information in the operating environment in real time. This connection can be achieved through wired or wireless means, such as data exchange via communication protocols like CAN bus, Ethernet, Wi-Fi, or Bluetooth. The navigation and positioning system typically includes sensors such as a GPS / GNSS module (GPS stands for Global Positioning System, GNSS stands for Global Navigation Satellite System), an Inertial Measurement Unit (IMU), and lidar or visual odometry, providing data such as the robot dog's global coordinates, local attitude, and motion trajectory. Real-time location information refers to the robot dog's geographic coordinates or precise position relative to a reference point at the current moment. Task information refers to the specific task type the robot dog is currently performing, such as inspection, equipment maintenance, or troubleshooting, as well as the waypoints, target area, and task duration included in the task plan. This information can be provided directly by the navigation and positioning system or parsed and forwarded by the robot dog's central control unit according to the task scheduling plan. Acquiring this information allows the inertial bimodal wake-up strategy management unit to have a more comprehensive understanding of the environmental risk level and task requirements of the robot dog. Pre-defined high-risk areas refer to specific areas in the power operation environment that, based on experience, historical data, or safety regulations, are identified as having a high probability of occurrence of dangerous events such as electric arcs and sparks. Examples include areas near substation switchgear, high-voltage line joints, and areas with easily aging equipment. The geographical coordinates or ranges of these areas can be pre-stored in the system's map database. When the inertial bimodal wake-up strategy management unit detects that the robot dog is about to enter or has already entered these high-risk areas through the navigation and positioning system, it actively sends commands to control the system to operate in the second working mode, ensuring that the system is fully prepared before potential hazards occur. Known safe areas refer to areas where, according to environmental assessment, the probability of occurrence of dangerous events such as electric arcs and sparks is extremely low, such as the robot dog's charging station, rest area, or non-power equipment area. Non-detection tasks refer to tasks currently performed by the robot dog that are unrelated to electric arc and spark detection, such as simple path planning, material transportation, and environmental cleanup. In this scenario, the lazy dual-modal wake-up strategy management unit instructs the system to operate in the first working mode, thereby maximizing energy savings and extending the robot dog's battery life.

[0124] like Figure 3As shown, this embodiment also proposes an arc spark recognition and early warning method for mechanical dogs based on visual-infrared image data fusion processing. Based on the aforementioned system, the specific implementation steps include: S1, synchronously acquiring RGB image sequences and thermal infrared image sequences from the power operation site through the multimodal sensing module; S2, performing timestamp synchronization, exposure signal alignment, and spatial registration on the RGB and thermal infrared images through the time-space alignment module to generate a temporally and spatially consistent dual-modal image pair; S3, extracting brightness abrupt changes, regional flicker, and frequency domain flicker features from the aligned RGB images through the visual feature extraction module; S4, extracting local hotspot regions and temperature gradient abrupt changes from the aligned thermal infrared images through the infrared feature extraction module; S5, performing feature-level or decision-level fusion calculations on the dual-modal features through the fusion judgment and false alarm suppression module, and applying the cross-modal mutual verification rules to perform joint judgment across multiple consecutive frames, outputting the final danger level signal; S6, executing corresponding local audible and visual alarms, mechanical dog motion state adjustments, or remote alarm information transmission actions through the graded early warning and response module based on the danger level signal.

[0125] The multimodal sensing module includes a global shutter RGB camera and an uncooled microbolometer infrared thermal imager. Both are rigidly fixed and their optical axes are kept parallel. A timer simultaneously sends periodic hardware trigger signals to both the RGB camera and the infrared imager, ensuring exposure begins within the same microsecond time window, achieving hardware-level synchronous acquisition. The temporal-spatial alignment module performs timestamp synchronization, exposure signal alignment, and spatial registration on the received RGB and thermal infrared images, generating a temporally and spatially consistent dual-modal image pair. Specifically, the temporal alignment submodule compensates for differences in image acquisition time caused by different sensor exposure durations or data transmission delays by receiving the timestamps from the timer's hardware trigger signals and the image exposure signals. The spatial alignment submodule incorporates the intrinsic and extrinsic parameters and distortion coefficients between the RGB camera and the infrared imager obtained through a dual-target alignment method. By applying a perspective transformation matrix, it resamples the thermal infrared image to the image coordinate system of the RGB camera, ensuring that the same physical scene point is located at the same pixel coordinates in both images.

[0126] The visual feature extraction module combines image processing algorithms with neural networks. First, it performs image enhancement based on histogram equalization and image dehazing preprocessing based on dark channel priors on the RGB image. Then, it converts the image to the HSV color space and extracts the V component as the luminance channel for temporal analysis. By calculating the absolute value of the luminance difference between the current frame and the previous frame and setting a dynamic threshold, it detects regions with abrupt luminance changes. Simultaneously, it performs multi-frame frequency domain analysis on suspected regions and detects the high-frequency flicker characteristics unique to electric arcs using Fast Fourier Transform. Finally, it uses a pre-trained lightweight convolutional neural network to perform secondary discrimination on candidate regions, outputting the probability value that it is a real electric arc spark. The infrared feature extraction module extracts local hotspot regions and temperature gradient abrupt change features from the aligned thermal infrared image. In specific implementation, this module performs non-uniformity correction and temperature calibration on the thermal infrared images of the dual-modal image pair, converts grayscale values ​​into accurate temperature values ​​and generates a temperature distribution matrix, and then uses a threshold segmentation algorithm to extract connected regions above the set threshold of ambient temperature from the temperature distribution matrix and marks them as suspected hot spots. It calculates the highest temperature, average temperature, area and temperature gradient of each suspected hot spot, detects temperature change events by comparing the rate of temperature change at the same spatial location between consecutive frames, and finally outputs the infrared feature vector composed of temperature statistics, geometric features and change indicators of the hot spot area to the fusion judgment and false alarm suppression module.

[0127] The fusion judgment and false alarm suppression module performs feature-level or decision-level fusion calculations on the dual-modal features and applies the cross-modal mutual verification rules to perform joint judgments across multiple consecutive frames, outputting the final danger level signal. In implementation, this module includes two configurable fusion strategies: feature-level fusion concatenates the feature vectors output by the visual feature extraction module and the infrared feature extraction module before inputting them into a classifier, assigns learnable weights to different modal features, and then inputs them into a lightweight fully connected neural network for joint classification; decision-level fusion allows the visual and infrared channels to independently perform preliminary judgments and obtain their respective confidence scores, then performs a weighted average of the two scores, with the weights dynamically adjusted according to ambient lighting conditions. The multi-frame confirmation logic unit is configured as follows: when a single modality detects a suspected event, another modality must confirm it in the same spatial area within the next 1 to 3 frames, or the single modality itself must continuously alarm for 1 to 5 frames, before it is determined to be a real dangerous event. The graded warning and response module executes corresponding local audible and visual alarms, mechanical dog movement status adjustments, or remote alarm information transmission actions based on the danger level signal. During implementation, the danger level signal of this module is determined based on the fusion confidence, dual-modal confirmation status, duration of dangerous events, and distance judgment results between the robot dog and the dangerous source output by the fusion judgment and false alarm suppression module. The danger level signal corresponds one-to-one with the response mechanism, specifically including the response actions of Level 1 warning, Level 2 warning, and Level 3 response (such as locking the motion system and transmitting feature extraction logs back).

[0128] Through the above technical solution, this method achieves efficient arc and spark detection on the mobile platform of the robotic dog, significantly reducing the false alarm rate, improving the detection rate of instantaneous small-scale events, and dynamically adjusting the response strategy according to the hazard level to ensure operational safety. Furthermore, the system can be combined with the inertial dual-modal wake-up strategy management unit to control the system to operate in the first working mode by default. Only when the visual feature extraction module detects significant brightness abrupt changes in multiple consecutive frames and matches a preset model will it switch to the second working mode, waking up the infrared feature extraction module and the fusion judgment and false alarm suppression module for joint diagnosis, further optimizing the power consumption management of the robotic dog platform.

[0129] When acquiring multimodal images at power work sites, although the system aims to simultaneously acquire RGB image sequences and thermal infrared image sequences, relying solely on software-level timestamps or coarse triggering mechanisms may result in slight time discrepancies between different sensors. To address this, in this embodiment, in step S1, a timer simultaneously sends periodic hardware trigger signals to both the RGB camera and the infrared thermal imager, ensuring that both sensors begin exposure within the same microsecond-level time window, thus achieving hardware-level synchronous acquisition of the RGB image sequences and thermal infrared image sequences.

[0130] Specifically, the timer can be a dedicated hardware timing chip, such as a high-precision programmable timer / counter, or a timer module integrated into a Field-Programmable Gate Array (FPGA) or Microcontroller Unit (MCU). Its core function is to generate stable and precise pulse signals at extremely high frequencies (e.g., megahertz levels), ensuring microsecond or even nanosecond-level accuracy in time intervals. The periodic hardware trigger signal refers to an electrical signal generated by the timer and transmitted at a fixed frequency. These signals are directly connected to the external trigger input ports of the RGB camera and infrared thermal imager, rather than being scheduled through software instructions or the operating system. When the camera receives this hardware trigger signal, it immediately starts its exposure process, thus avoiding software-level uncertainties such as operating system latency, USB transmission latency, or internal processing latency. Ensuring that both sensors begin exposure within the same microsecond-level time window means that the exposure start time difference between the RGB camera and the infrared thermal imager is strictly controlled within microseconds. This is typically achieved through precise circuit design and signal routing to minimize delay differences along the signal transmission path. Achieving hardware-level synchronous acquisition of RGB image sequences and thermal infrared image sequences means that the synchronicity of the entire image acquisition process is guaranteed at the hardware level, rather than relying on subsequent software correction. This means that the physical process from the sensor receiving light to the start of exposure occurs almost simultaneously for both modalities of image acquisition.

[0131] In some implementations, during fusion judgment, if a feature-level fusion strategy is adopted, the dual-modal feature vectors are concatenated and assigned learnable weights, which are then input into a lightweight fully connected neural network for joint classification; if a decision-level fusion strategy is adopted, the confidence scores of the two channels are weighted and the weights are dynamically adjusted according to the ambient lighting conditions; during joint judgment, a multi-frame confirmation rule is followed: after a single modality detects a suspected event, another modality must confirm it within the same spatial area in the following 1 to 3 frames, or a single modality must continuously alarm for 1 to 5 frames, before it is determined to be a real dangerous event.

[0132] Specifically, when the system employs a feature-level fusion strategy, the original or pre-processed feature vectors from the visual feature extraction module and the infrared feature extraction module are directly concatenated into a longer composite feature vector. This concatenation operation deeply integrates information from different modalities before it is input into the classifier, enabling the subsequent classifier to learn more complex correlations and complementary information between modalities. To optimize this integration, the concatenated feature vector is assigned learnable weights, which are automatically adjusted during the training of the neural network using a backpropagation algorithm to maximize classification accuracy. This allows the system to intelligently identify the relative importance of different modal features for arc and spark determination. Subsequently, this weighted composite feature vector is input into a lightweight fully connected neural network for joint classification. This pre-trained neural network can identify arc or spark patterns from high-dimensional fused features and output the final classification result, such as the probability of the event occurring. The lightweight design ensures that the network can run efficiently on the embedded computing platform of the robotic dog, meeting real-time requirements.

[0133] When the system employs a decision-level fusion strategy, the visual and infrared channels first independently analyze the data of their respective modalities and derive confidence scores for the occurrence of the event. For example, the visual channel might output a confidence score based on brightness abrupt changes and flicker features, while the infrared channel outputs a confidence score based on local hotspots and temperature gradient abrupt changes. These two independent confidence scores are then weighted and averaged to obtain a comprehensive confidence score. It is noteworthy that the weights of these weighted averages are not fixed but dynamically adjusted according to the current ambient lighting conditions. For instance, in well-lit environments, the weight of the visual modality may be increased because it provides richer detail information; while in dimly lit or smoky environments, the weight of the infrared modality will be increased accordingly because it is less sensitive to these conditions and provides more reliable temperature information. This dynamic adjustment mechanism allows the system to flexibly select and emphasize information from different modalities based on the advantages and disadvantages of the actual environment, thereby improving detection accuracy and robustness in various complex environments.

[0134] During the joint judgment phase, to further suppress false alarms and ensure detection reliability, the system follows a strict multi-frame confirmation rule. This rule includes two main scenarios: The first is cross-modal mutual verification, where when a single modality (e.g., visual modality) detects a suspected event, the system does not immediately alarm but requires another modality (e.g., infrared modality) to confirm within the same spatial area in the subsequent 1 to 3 frames of image data. This mechanism utilizes the complementarity of visual and infrared modalities, avoiding false alarms caused by noise or transient interference from a single sensor. The 1 to 3-frame time window design considers both the transience of the event and allows for minor delays in data processing and transmission from different sensors. The second scenario is continuous confirmation by a single modality, where if a suspected event is detected only by a single modality, but that modality continuously issues alarm signals for 1 to 5 frames, the system will also classify it as a genuine dangerous event. This mechanism aims to capture real-world events that may only be significant in a specific modality but have a certain duration, such as weak electric arcs or persistent localized overheating. Simultaneously, by requiring continuous alarms across multiple frames, it effectively distinguishes between transient noise and real events. The continuous alarm range of 1 to 5 frames provides the system with the possibility of flexible configuration based on specific application scenarios and false alarm tolerance.

[0135] The above embodiments significantly improve the detection accuracy and robustness of the arc spark recognition and early warning system and method based on visual-infrared image data fusion processing for robotic dogs. Feature-level fusion automatically mines complex correlations between modes through deep learning, giving the system a stronger pattern recognition capability when identifying arc sparks. The dynamic weight adjustment mechanism of decision-level fusion enables the system to intelligently adapt to changes in ambient lighting, maintaining optimal detection performance under different operating conditions and effectively avoiding missed or false alarms caused by limitations of a single mode. In addition, the refined multi-frame confirmation rules and hierarchical early warning response mechanism greatly enhance the system's ability to identify real dangerous events and its response flexibility, ensuring the timeliness and accuracy of early warnings and providing more reliable technical support for safety inspections at power operation sites.

[0136] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application. The scope of this application is determined by the scope of the claims.

Claims

1. An arc spark recognition and early warning system for a mechanical dog based on visual-infrared image data fusion processing, characterized in that, The system is integrated into the mechanical dog body and includes: A multimodal sensing module is used to simultaneously acquire RGB image sequences and thermal infrared image sequences; The time-space alignment module is used to perform timestamp synchronization, exposure signal alignment, and spatial registration based on calibration parameters on the received RGB image sequence and thermal infrared image sequence to generate a time-space consistent bimodal image pair. A visual feature extraction module is used to extract brightness abrupt changes, regional flicker, and frequency domain flicker features from the RGB images of the dual-modal image pair; The infrared feature extraction module is used to extract local hotspot regions and temperature gradient abrupt change features from the thermal infrared images of the dual-modal image pair; The fusion judgment and false alarm suppression module is used to obtain dual-modal features from the visual feature extraction module and the infrared feature extraction module, and to perform feature-level or decision-level fusion calculation on the dual-modal features through the built-in weighted fusion unit to generate a preliminary danger probability; then, through the built-in multi-frame confirmation logic unit, the module applies cross-modal mutual verification rules to jointly judge the preliminary danger probability and dual-modal features of multiple consecutive frames, and outputs a danger level signal. The graded early warning and response module is used to execute corresponding early warning and response actions according to different hazard levels; wherein, the early warning and response actions include local audible and visual alarms, limiting or emergency stopping the movement speed of the mechanical dog, and sending alarm information and on-site image data to the remote monitoring center. The danger level signal of the graded early warning and response module is determined by the fusion confidence, dual-modal confirmation status, and duration of the dangerous event output by the fusion judgment and false alarm suppression module, combined with the distance judgment result between the mechanical dog and the dangerous source. The weighted fusion unit is configured to execute at least one of a feature-level fusion strategy and a decision-level fusion strategy; The feature-level fusion strategy is configured to concatenate the feature vectors output by the visual feature extraction module and the infrared feature extraction module before inputting them into the classifier, assign learnable weights to different modal features, and then input them into a lightweight fully connected neural network for joint classification. The decision-level fusion strategy is configured to independently classify the feature vectors output by the visual feature extraction module and the infrared feature extraction module to obtain a visual confidence score and an infrared confidence score, and then perform a weighted average of the two scores to obtain a fusion score. The multi-frame confirmation logic unit is configured to apply cross-modal mutual verification rules based on the fusion result output by the weighted fusion unit: when a single modality detects a suspected event, another modality must confirm it in the same spatial area within the next 1 to 3 frames, or the single modality itself must continuously alarm for N consecutive frames before it is determined to be a real dangerous event, where N is dynamically set according to the false alarm tolerance.

2. The arc spark recognition and early warning system for mechanical dogs based on visual-infrared image data fusion processing as described in claim 1, characterized in that, The multimodal sensing module includes an RGB camera, an infrared thermal imager, and a timer; the RGB camera is configured as a global shutter RGB camera, and the infrared thermal imager is configured as an uncooled microbolometer infrared thermal imager; the RGB camera and the infrared thermal imager are fixedly connected by a rigid structure, and their optical axes remain parallel; the timer is used to send periodic hardware trigger signals to the RGB camera and the infrared thermal imager.

3. The arc spark recognition and early warning system for a mechanical dog based on visual-infrared image data fusion processing according to claim 2, characterized in that, The time-space alignment module includes a time alignment submodule and a space alignment submodule; The time alignment submodule is configured to receive the timestamp of the hardware trigger signal from the timer and the image exposure signal from the RGB camera and the infrared thermal imager, and to compensate for the differences in image acquisition time caused by different exposure durations or data transmission delays based on the timestamp of the hardware trigger signal and the image exposure signal, so as to align the time of the dual-modal image pair. The spatial alignment submodule has built-in intrinsic and extrinsic parameters and distortion coefficients of the RGB camera and the infrared thermal imager obtained by the dual-target alignment method. The intrinsic and extrinsic parameters include the intrinsic parameters of the RGB camera and the infrared thermal imager, as well as the extrinsic parameters between the two sensors. Based on the intrinsic and extrinsic parameters and distortion coefficients, a perspective transformation matrix is ​​calculated. The thermal infrared image is resampled to the image coordinate system of the RGB camera or the RGB image is resampled to the image coordinate system of the infrared thermal imager, so that the same physical scene point is located at the same pixel coordinate in the RGB image and the thermal infrared image.

4. The arc spark recognition and early warning system for a mechanical dog based on visual-infrared image data fusion processing according to claim 1, characterized in that, The visual feature extraction module employs a combination of image processing algorithms and neural networks, wherein: The image processing algorithm is configured to perform image enhancement based on histogram equalization and image dehazing preprocessing based on dark channel prior on the input RGB image; convert the preprocessed image to the HSV color space, extract the V component as the luminance channel for temporal domain analysis, and detect luminance abrupt change regions by calculating the absolute value of the luminance difference between the current frame and the previous frame and setting a dynamic threshold; perform multi-frame frequency domain analysis on the luminance abrupt change regions, and detect the presence of high-frequency flicker features unique to electric arcs by fast Fourier transform, thereby filtering out candidate regions; The neural network is configured to perform secondary discrimination on the candidate region through a pre-trained convolutional neural network and output the probability value that the candidate region is a real electric arc spark.

5. The arc spark recognition and early warning system for a mechanical dog based on visual-infrared image data fusion processing according to claim 1, characterized in that, The infrared feature extraction module is used to perform non-uniformity correction and temperature calibration on the thermal infrared images of the dual-modal image pair, convert grayscale values ​​into accurate temperature values ​​and generate a temperature distribution matrix. Then, a threshold segmentation algorithm is used to extract connected regions above a set threshold of ambient temperature from the temperature distribution matrix and mark them as suspected hot spots. The highest temperature, average temperature, area and temperature gradient of each suspected hot spot are calculated. Temperature abrupt events are detected by comparing the rate of temperature change at the same spatial location between consecutive frames. Finally, the temperature statistics, geometric features and abrupt event markers of the hot spot area are combined to form an infrared feature vector and output to the fusion judgment and false alarm suppression module.

6. The arc spark recognition and early warning system for a mechanical dog based on visual-infrared image data fusion processing according to claim 1, characterized in that, The hazard level signal of the graded early warning and response module is determined by the fusion confidence level, dual-modal confirmation status, and hazard event duration output by the fusion judgment and false alarm suppression module, combined with the distance determination result between the robot dog and the hazard source. The hazard level signal and the response mechanism correspond one-to-one, including: Level 1 Warning: When the detection result is low confidence, or when a suspected event is detected in only a single modality and the mutual verification conditions of the cross-modal mutual verification rules are not met, it is determined to be a Level 1 warning signal; the response mechanism is to trigger the LED indicator light of the robot dog to give a yellow warning, without interfering with the normal operation of the robot dog, and to remind on-site personnel to pay attention to potential risks; Level 2 Warning: When the detection result meets the mutual verification conditions of the cross-modal mutual verification rules, and the degree of danger is moderate or the distance between the danger source and the robot dog is less than the first distance threshold, it is determined to be a Level 2 warning signal; the response mechanism is that the LED indicator of the robot dog body turns red and the buzzer is activated to emit a warning sound; the robot dog automatically limits its maximum movement speed and maintains the current working direction; and sends regular priority alarm information to the remote monitoring center, including the warning level, real-time location and on-site image data; Level 3 Response: When the detection result is of high confidence, the dangerous event persists, or the distance between the dangerous source and the robot dog is less than the second distance threshold, it is determined to be a Level 3 response signal; the response mechanism is that the robot dog immediately executes the emergency stop procedure and locks the motion system; the LED indicator light is constantly red, and the buzzer continuously emits a warning sound; the highest priority alarm signal is sent to the remote monitoring center, and the alarm-related dual-modal image pair and feature extraction log are transmitted back, maintaining real-time data transmission with the remote monitoring center until a manual reset command is received.

7. The arc spark recognition and early warning system for a mechanical dog based on visual-infrared image data fusion processing according to claim 1, characterized in that, It also includes a lazy dual-modal wake-up strategy management unit for controlling the system to operate in either the first or second operating mode, wherein: In the default first working mode, only the visual feature extraction module is controlled to be in working state, while the infrared feature extraction module and the fusion judgment and false alarm suppression module are controlled to be in sleep state; when the visual feature extraction module detects brightness change events for multiple consecutive frames, and the features of the brightness change events match the preset preliminary model of electric arc spark, the lazy dual-modal wake-up strategy management unit switches the system to the second working mode. In the second working mode, the infrared feature extraction module and the fusion judgment and false alarm suppression module are awakened to work together; if the danger level signal does not reach the preset condition within a preset time, the lazy dual-modal wake-up strategy management unit will switch the system back to the first working mode.

8. The arc spark recognition and early warning system for a mechanical dog based on visual-infrared image data fusion processing according to claim 7, characterized in that, The robotic dog includes a navigation and positioning system. The lazy bimodal wake-up strategy management unit is connected to the navigation and positioning system to obtain the real-time location and task information of the robotic dog, and determines whether the robotic dog is in a preset high-risk area or a known safe area, and whether the robotic dog is performing a non-detection task. When the robotic dog enters the preset high-risk area, the lazy bimodal wake-up strategy management unit controls the system to operate in the second working mode. When the robotic dog is in a known safe area or performing a non-detection task, the lazy bimodal wake-up strategy management unit controls the system to operate in the first working mode.

9. An arc spark recognition and early warning method based on visual-infrared image data fusion processing for a mechanical dog, applied to the system described in any one of claims 1 to 8, characterized in that, It should include at least the following steps: S1. The RGB image sequence and thermal infrared image sequence of the power operation site are simultaneously acquired through the multimodal sensing module; S2. The RGB image sequence and the thermal infrared image sequence are time-stamped, exposure signal aligned, and spatially registered by the time-space alignment module to generate the temporally and spatially consistent dual-modal image pair. S3. Extract brightness abrupt changes, regional flicker, and frequency domain flicker features from the RGB image of the dual-modal image pair using the visual feature extraction module; S4. Extract local hotspot regions and temperature gradient abruptness features from the thermal infrared image of the dual-modal image pair using the infrared feature extraction module; S5. Through the fusion judgment and false alarm suppression module, dual-modal features are obtained from the visual feature extraction module and the infrared feature extraction module. Feature-level or decision-level fusion calculations are performed on the dual-modal features to generate a preliminary danger probability. The cross-modal mutual verification rules are applied to jointly judge the preliminary danger probability and the dual-modal features of multiple consecutive frames and output a danger level signal. S6. The graded warning and response module executes corresponding warning and response actions according to different hazard levels; wherein, the warning and response actions include local audible and visual alarms, limiting or stopping the mechanical dog's movement speed, and sending alarm information and on-site image data to the remote monitoring center. The danger level signal of the graded early warning and response module is determined by the fusion confidence level, dual-modal confirmation status, and danger event duration output by the fusion judgment and false alarm suppression module, combined with the distance judgment result between the robot dog and the danger source.

Images