A quadruped robot multi-modal perception anomaly early warning method and system

Abstract

The application discloses a kind of four-legged robot multi-modal perception anomaly early warning method and system, the method includes: through the multiple sensors of four-legged robot carrying synchronous acquisition multi-modal perception data;After spatial alignment to multi-modal perception data is respectively carried out feature extraction, into multidimensional feature vector;Each dimension feature vector is input into causal inference model, carries out multi-source data cross validation, outputs causal inference result and equipment health state vector, causal inference model at least supports sound-vibration-temperature joint diagnosis and / or gas-temperature-view linkage leakage tracing;According to the confidence score of causal inference result calculation, according to the confidence score grading early warning is carried out, and early warning signal is returned to remote monitoring platform;The equipment health state vector of each inspection is stored in time series database, establishes equipment state time sequence evolution model, fits equipment deterioration trend curve, predicts equipment remaining useful life, and the prediction result is returned to remote monitoring platform.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection robot technology, specifically to a method and system for multimodal perception and early warning of anomalies in quadruped robots. Background Technology

[0002] The application of quadruped robots in industrial inspection has been increasing. Existing quadruped robots are usually equipped with a variety of detection devices such as visible light cameras, infrared thermal imagers, acoustic sensors, and gas sensors, and move autonomously along a preset route to collect equipment status data.

[0003] However, existing quadruped robots still have the following shortcomings in multi-sensor data processing: First, existing solutions typically process data from each sensor independently, lacking an effective mechanism for multi-source data fusion and cross-validation. When a sensor detects an abnormal signal, the system cannot automatically link data from other sensors for comprehensive judgment, resulting in a high false alarm rate. Inspection personnel still need to manually review a large number of alarm messages, which is insufficient to meet the actual needs of the field for alarm accuracy.

[0004] Second, existing solutions mostly adopt a single-point threshold alarm method, that is, each sensor is set with a fixed alarm threshold, and an alarm is triggered when the detected value exceeds the threshold. This method cannot distinguish between the gradual degradation process of equipment failure and transient interference from the external environment, and is prone to false alarms or missed alarms under complex operating conditions such as dust, drastic changes in temperature and humidity.

[0005] Third, the existing solutions lack the ability to predict equipment status trends. They can only provide alarms for abnormalities in the current state and cannot predict the remaining service life of the equipment based on historical inspection data. Equipment maintenance is still mainly based on post-event repairs or periodic inspections, making it difficult to achieve predictive maintenance. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a multimodal perception anomaly early warning method and system for quadruped robots. It aims to achieve comprehensive judgment and hierarchical early warning of equipment anomalies through spatial alignment and causal reasoning of multi-source data, and to predict the remaining lifespan of the equipment based on time-series data.

[0007] To achieve the above objectives, this invention provides a method for early warning of multimodal perception anomalies in a quadruped robot, comprising the following steps: synchronously collecting multimodal perception data using multiple sensors mounted on the quadruped robot, wherein the multimodal perception data includes at least: high-definition video images, infrared thermal imaging, acoustic signature signals, visual vibration images, gas concentration data, meter readings, and temperature and humidity data; spatially aligning the multimodal perception data; extracting features from the spatially aligned multimodal perception data and converting them into multidimensional feature vectors; inputting the feature vectors of each dimension into a causal inference model, performing multi-source data cross-validation, and outputting causal inference results and equipment health status vectors, wherein the causal inference model at least supports joint diagnosis of acoustic-vibration-temperature and / or gas-temperature-visual linkage for leak tracing; calculating a confidence score based on the causal inference results, performing graded early warning based on the confidence score, and transmitting the early warning signal back to a remote monitoring platform; storing the equipment health status vector from each inspection into a time-series database, establishing a time-series evolution model of equipment status, fitting an equipment degradation trend curve, predicting the remaining service life of the equipment, and transmitting the prediction result back to the remote monitoring platform.

[0008] The spatial alignment includes: acquiring robot posture data using an inertial measurement unit installed on the quadruped robot, obtaining motion data from the encoders of each joint of the robot, and performing motion compensation on the high-definition video image, the infrared thermal image, and the visual vibration image based on the posture data and the motion data.

[0009] The combined sound-vibration-temperature diagnosis is as follows: when both visual vibration characteristics and acoustic characteristics are abnormal, infrared thermal imaging characteristics are invoked in conjunction. If the temperature of the corresponding part rises abnormally at the same time, it is confirmed as a mechanical wear fault; if the temperature is normal, it is determined to be an external interference. The gas-temperature-visual linkage leak tracing method is as follows: when the gas concentration characteristics are abnormal, the infrared thermal imaging and the high-definition video images at a first preset time before and after the trigger time are automatically traced back. The abnormal temperature difference is identified by the infrared thermal imaging and the leak source status is identified by combining the high-definition video images to locate the leak source.

[0010] The causal reasoning model supports environmental adaptive compensation: using temperature and humidity characteristics as an environmental benchmark, it dynamically corrects the alarm threshold of the infrared thermal imaging and the alarm threshold of the gas concentration data.

[0011] The tiered early warning system includes: triggering a level one early warning when the confidence score is lower than a first preset threshold, controlling the quadruped robot to perform secondary data collection on the abnormal area; triggering a level two early warning when the confidence score is between the first and second preset thresholds, transmitting the abnormal data and diagnostic report back to the remote monitoring platform; and triggering a level three early warning when the confidence score is higher than the second preset threshold, controlling the quadruped robot to initiate an emergency response, continuously tracking the source of the anomaly and linking the on-site audible and visual alarm device. The confidence score is calculated comprehensively based on the clarity of the data from each sensor, the current environmental interference, and the consistency of the multimodal cross-validation.

[0012] The quadruped robot multimodal perception anomaly early warning method further includes, after feature extraction of the voiceprint signal, executing the following three-modal fusion early warning logic: threshold early warning: when the voiceprint feature value exceeds a preset absolute threshold in real time, a threshold alarm is triggered; trend increase early warning: a sliding window algorithm is used to calculate the cumulative change of the current voiceprint feature value relative to the dynamic health baseline, and when the cumulative change exceeds a preset tolerance range, an increase alarm is triggered and it is determined to be an abnormal trend increase; trend growth rate early warning: the rate of change of the voiceprint feature value over time is calculated, and when the rate of change exceeds a preset critical value, a growth rate alarm is triggered and it is determined to be an abnormal trend growth rate; the threshold early warning, the trend increase early warning, and the trend growth rate early warning are fused and decided through logic gates and a weight matrix to form an adaptive early warning strategy, including: NOT logic: When the voiceprint feature value instantaneously exceeds the preset absolute threshold but quickly falls back within a second preset time and there is no continuous cumulative change, it is determined to be an external transient interference and filtered out, and the threshold alarm is not triggered. AND logic: When the trend increase is determined to be abnormal and the trend growth rate continues to be positive, the warning level is increased; OR logic: When the threshold warning is triggered, the level 3 warning is triggered.

[0013] The feature extraction includes meter recognition of the high-definition video image, comprising: preprocessing the high-definition video image, including at least strong light reflection suppression, low light enhancement, and perspective distortion correction; locating the region of interest (ROI) of the dial in the image using a target detection network to distinguish between digital and analog meters; for digital meters, segmenting and recognizing numbers using a character recognition network; for analog meters, locating the center of the dial, zero mark, full scale, and pointer tip coordinates using key point detection, calculating the angle between the pointer and the zero mark, and converting the reading according to the range ratio; outputting the recognition confidence score, determining recognition failure when the confidence score is lower than a preset threshold, controlling the quadruped robot to automatically retake the image, performing time-series verification on the recognition result, and uploading the recognition data and the original image together to the remote monitoring platform.

[0014] In another aspect, the present invention provides a multimodal perception anomaly early warning system for a quadruped robot, comprising: a data acquisition module, used to synchronously acquire multimodal perception data through multiple sensors mounted on the quadruped robot, wherein the multimodal perception data includes at least: high-definition video images, infrared thermal imaging, acoustic signals, visual vibration images, gas concentration data, meter readings, and temperature and humidity data; A spatial alignment module is used to spatially align the multimodal sensing data; The system includes a feature extraction module for extracting features from the spatially aligned multimodal sensing data and converting them into multidimensional feature vectors; a causal inference module for inputting the feature vectors of each dimension into a causal inference model, performing multi-source data cross-validation, and outputting causal inference results and equipment health status vectors, wherein the causal inference model at least supports joint diagnosis of sound-vibration-temperature and / or gas-temperature-visual linkage for leak tracing; a graded early warning module for calculating confidence scores based on the causal inference results, performing graded early warnings based on the confidence scores, and transmitting the early warning signals back to the remote monitoring platform; and a time-series prediction module for storing the equipment health status vectors from each inspection into a time-series database, establishing a time-series evolution model of the equipment status, fitting the equipment degradation trend curve, predicting the remaining service life of the equipment, and transmitting the prediction results back to the remote monitoring platform.

[0015] The quadruped robot is also equipped with a millimeter-wave phased array radar. The millimeter-wave phased array radar adopts a frequency-modulated continuous wave system and integrates a phased array antenna array. It achieves beam spatial scanning by electronically controlling the phase of each radiating element. The millimeter-wave phased array radar and lidar form a redundant sensing combination. When the lidar fails in dusty or foggy environments, the detection data of the millimeter-wave phased array radar is switched as the main data source for navigation, positioning, and map building. It is combined with a fusion SLAM algorithm to achieve localization and dynamic path planning. The millimeter-wave phased array radar is also used to cover the robot's blind spot and detect obstacles in the blind spot to trigger emergency stop or detour decisions.

[0016] The quadruped robot communicates with the remote monitoring platform using a dual-mode communication method combining 5G public network and wireless local area network; the wireless local area network is used for batch data transmission after inspection, and the 5G public network is used for remote control takeover in abnormal situations.

[0017] As can be seen from the above technical solutions, the advantages of the present invention are: This invention utilizes a quadruped robot equipped with multiple sensors to synchronously perceive multimodal data. After spatial alignment, multidimensional feature vectors are extracted from each sensor and input into a causal inference model for multi-source data cross-validation. This effectively solves the technical problems of high false alarm rates and lack of comprehensive judgment capabilities caused by isolated processing of sensor data in existing solutions. Furthermore, by establishing a temporal evolution model of equipment status, this invention can predict the remaining service life of equipment based on historical inspection data, achieving a leap from passive alarm to proactive prediction and providing reliable technical support for predictive maintenance in industrial scenarios.

[0018] This invention utilizes inertial measurement unit and joint encoder data for motion compensation, eliminating image spatial offset caused by quadrupedal gait and ensuring accurate alignment of multi-source data. Through joint acoustic-vibration-temperature diagnosis, gas-temperature-visual linkage leak tracing, and environmental adaptive compensation mechanisms, it achieves multimodal cross-validation and accurate leak source location, effectively eliminating environmental interference and reducing false alarms. By combining voiceprint three-mode fusion early warning logic (threshold early warning, trend increase early warning, trend speed increase early warning) with AND / OR / NOT logic gate decision-making, it can filter out transient interference, identify gradual degradation, and capture early signs of sudden deterioration, achieving full-cycle dynamic monitoring. By using high-definition video images for meter identification, it achieves high-precision automatic meter reading for various old meters, replacing manual meter reading operations in high-risk areas.

[0019] This invention equips a quadruped robot with a millimeter-wave phased array radar, forming a redundant sensing combination with a lidar. In dusty or foggy environments, it automatically switches to the primary data source, eliminating the blind spots of a single lidar. At the same time, by covering the robot's blind spots, it detects potential collision risks and triggers emergency stop or detour decisions, thereby improving operational safety in complex environments. Attached Figure Description

[0020] Figure 1 A flowchart of a multimodal perception anomaly early warning method for quadruped robots provided in an embodiment of the present invention; Figure 2 A flowchart for the inspection process of a quadruped robot; Figure 3A A schematic diagram of a pipe gallery crack image captured by a quadruped robot using a high-definition camera; Figure 3B This is a schematic diagram of a quadruped robot capturing images of water accumulation on the ground using a high-definition camera. Figure 4 A schematic diagram of a pipe image captured by a quadruped robot using infrared thermography; Figure 5 This is a flowchart illustrating the meter identification process for high-definition video images according to the present invention; Figure 6A This is a schematic diagram of the threshold warning function applied to the voiceprint signal according to the present invention; Figure 6B This is a schematic diagram of the trend increase warning of the voiceprint signal according to the present invention; Figure 6C This is a schematic diagram illustrating the trend growth warning of voiceprint signals according to the present invention; Figure 7 This is a schematic diagram of the structure of a quadruped robot multimodal perception anomaly early warning system provided in another embodiment of the present invention; Figure 8 This is an architecture diagram of a multimodal perception anomaly early warning system for a quadruped robot provided in another embodiment of the present invention; Figure 9 This is a schematic diagram showing the detection coverage of the whole-body sensors of the quadruped robot of the present invention; Figure 10 This is a scanning schematic diagram of the auxiliary lidar navigation and positioning technology of the present invention; Figure 11 This is a diagram of the IoT system architecture of the present invention; Figure 12 This is a schematic diagram of the swing gate type automatic rodent barrier structure of the present invention; In the attached figures, the following labels are used: 100-Multimodal perception anomaly early warning system for quadruped robots; 110-Data acquisition module; 111-Quadruped robot; 120-Spatial alignment module; 130-Feature extraction module; 140-Causal reasoning module; 150-Hierarchical early warning module; 160-Temporal prediction module; 200 - Tunnel; 201 - Auxiliary road signs; 300 - Swing gate type automatic rodent barrier; 301 - Support column; 302 - Baffle; 303 - Rotating shaft; 400 - Ground; 500 - Internet of Things (IoT) System; 510 - Cloud and Remote Management Layer; 511 - Cloud Server; 512 - Monitoring Computer; 513 - Internet; 520 - Field Network Hub Layer; 521 - Station Monitoring Computer; 522 - Gateway; 530 - Field Equipment Layer; 531 - Wireless Communication Network; 532 - Weather Station; 533 - Automatic Door; 534 - Outdoor Charging Room; Steps S1~S6, S30~S34. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0022] like Figure 1 The diagram shown is a flowchart of a multimodal perception anomaly early warning method for a quadruped robot according to an embodiment of the present invention, including the following steps: S1: Multimodal perception data is collected synchronously through multiple sensors mounted on the quadruped robot.

[0023] By integrating various sensing hardware components—including high-definition visible light cameras, infrared thermal imagers, acoustic microphone arrays, visual vibration acquisition cameras, gas sensors, and temperature and humidity sensors—on the quadruped robot's body, robotic arm, and foot measuring points, a hardware-triggered synchronization clock scheme is employed to unify the sampling timing of all sensors, achieving millisecond-level synchronous acquisition of multimodal sensing data. This multimodal sensing data includes at least seven categories: high-definition video images, infrared thermal imaging, equipment operation acoustic signals, visual vibration images, on-site gas concentration data, meter readings, and raw environmental temperature and humidity sensor data.

[0024] The quadruped robot aims to achieve equipment status monitoring and anomaly early warning, as well as online safety detection in utility tunnels. Utilizing multi-sensor data fusion technology and a comprehensive application of multiple methods, it replaces human workers in performing daily inspections under complex working conditions. This not only improves employee efficiency but also enhances the technological content of industrial settings, maximizing personnel safety and minimizing property and personnel losses.

[0025] like Figure 2 As shown, the quadruped robot inspection process includes: setting inspection parameters, issuing inspection tasks, executing inspection tasks, collecting and displaying inspection data, and returning to charging. That is, the quadruped robot can automatically perform inspection work according to pre-set routes, inspection content, alarm rules, and inspection schedules, and can also execute various temporarily issued inspection tasks during the inspection process. The industrial-grade intelligent inspection system based on quadruped robots covers basic physical movement and data acquisition throughout the entire operation process, deeply integrating multimodal perception, autonomous decision-making, edge computing, and cloud collaboration. The specific workflow is as follows: (1) Set inspection parameters. This step includes planning robot inspection points, planning the work tasks for each inspection point, planning the alarm rules for the inspection points, planning the robot inspection route, and compiling the robot inspection schedule. Before the task is issued, system-level initialization from environmental modeling to policy configuration needs to be completed.

[0026] Environmental mapping and route planning: High-precision 3D mapping of complex terrain is carried out using multi-sensor fusion technology (such as phased array radar and 32-line lidar). Through the "marking-simulation-ready-use" closed loop, inspection routes, key points and restricted areas are set in 2D grid or 3D digital twin maps. Task and Strategy Configuration: Dynamically adjust inspection strategies based on equipment health status, configuring inspection frequency, dwell time at points, and operating modes of data acquisition payloads (such as dual-light pan-tilt units, acoustic sensors, and gas probes). Simultaneously, set safety thresholds (such as terrain risk thresholds and network latency thresholds) and abnormal triggering conditions. System self-check and calibration: After the robot starts, it loads a high-precision map and performs relocalization to calibrate the initial pose. It also performs a status self-check on the battery, communication link, and sensors to ensure that operating conditions are met.

[0027] (2) Assigning inspection tasks. This step includes assigning inspection schedules to the robot, the robot automatically starting inspections when the inspection time arrives, and manually assigning temporary tasks to the robot. Relying on the intelligent scheduling platform, accurate task distribution and multi-machine collaboration are achieved.

[0028] Task parsing and distribution: The system converts the inspection plan into a machine-readable task sequence and distributes it to the robot's edge computing unit through industrial-grade communication networks such as 4G / 5G / Wi-Fi.

[0029] Dynamic scheduling and takeover mechanism: The scheduling platform allocates the optimal task based on the robot's current battery level, location, and working status. If the robot encounters positioning failure, path blockage, or network disconnection during execution, the system can automatically trigger the "manual takeover" mechanism, allowing a remote operator to intervene via a controller or console through low-latency video feedback. Autonomous operation resumes after the anomaly is resolved.

[0030] (3) Performing inspection tasks and dynamic obstacle avoidance. This step includes the robot collecting and analyzing data at inspection point 1, with the built-in sensor at point 1 transmitting the data to the robot; the robot then proceeds to point 2 until the inspection task is completed, with the built-in sensors at other points transmitting the data to the robot. The robot performs autonomously in an unstructured environment, exhibiting high adaptability and high safety.

[0031] All-terrain autonomous navigation: Based on model predictive control and deep reinforcement learning, the robot travels along a planned path. It is capable of crossing steps, climbing steep slopes, and wading through water, and can perceive unknown obstacles in real time while moving, performing dynamic obstacle avoidance and path replanning.

[0032] Safety control closed loop: During execution, the system continuously monitors video status, network quality, and chassis attitude. If excessive video latency or control command timeout occurs, the system will proactively limit speed or trigger a safety stop to prevent blind execution from causing a collision.

[0033] Precise berthing and attitude adjustment: After arriving at the designated inspection point, the robot automatically adjusts its body attitude and the angle of the three-axis anti-vibration gimbal to ensure that the sensors are facing the target equipment and eliminate image distortion and temperature measurement deviation caused by motion.

[0034] (4) Collect and display inspection data. This step involves storing the collected inspection data into the database in real time to form a comprehensive equipment inspection platform for the plant area. This step is the core of the inspection, realizing a closed loop from data acquisition to value extraction.

[0035] Multi-dimensional data acquisition: Simultaneously acquires high-definition video images, infrared thermal imaging, acoustic signals, visual vibration images, gas concentration data, temperature and humidity data, and meter readings. For complex operating conditions, dynamic and stable acquisition technology is used to ensure data reproducibility.

[0036] Edge AI Recognition and Early Warning: AI algorithms run in real time on the robot's local machine or at edge nodes to automatically complete meter reading recognition, equipment appearance defect detection, temperature anomaly determination, gas concentration data anomaly recognition, and visual vibration status analysis. If an anomaly is detected, an alarm event is immediately generated, video clips before and after the anomaly are retained as evidence, and the alarm is pushed to the monitoring backend in real time according to the level of the anomaly.

[0037] Data visualization and digital twin: The backend platform uses a 3D visualization interface to reproduce the robot's inspection trajectory in real time, simultaneously displaying multi-source detection perspectives and equipment status parameters. It supports remote video review and secondary confirmation of the recognition results.

[0038] (5) Task completion, report generation, and autonomous recharging. This step involves the robot returning to the charging station to recharge and stand by. This achieves a complete closed loop in the work process and the accumulation of data assets.

[0039] Autonomous return and precise docking: Upon completion of all tasks or when the battery level falls below a preset threshold, the robot autonomously plans its return route. Upon arrival at the charging station, it precisely docks using infrared ranging or visual guidance, initiating automatic wireless or contact charging without human intervention.

[0040] Intelligent Report Generation: The system automatically summarizes the visible light / infrared images, meter readings, temperature anomaly detection, gas concentration anomaly identification, visual vibration status, and abnormal alarms from this inspection, generating a structured inspection report. It supports historical data comparison, trend curve plotting, and fault risk probability assessment.

[0041] Work order linkage and closed-loop management: For confirmed equipment defects, the system supports the automatic generation of maintenance work orders and linkage with third-party management systems such as SCADA, DCS or EAM of enterprises to realize closed-loop management of the entire life cycle of "monitoring-early warning-handling-review".

[0042] In the above step (4), when performing the inspection task, the robot arrives at the designated inspection point according to the inspection route, and uses its own high-definition camera (for visible light imaging and meter reading recognition), infrared thermal imager, acoustic sensor, visual vibration analyzer, gas sensor, temperature and humidity sensor, etc. to synchronously collect data on the object to be detected and obtain multimodal perception data.

[0043] Visible light recognition: Quadruped robot inspection can include various monitoring functions such as visible light meter recognition. From real-time images, real-time data, and remote control, it provides comprehensive auxiliary monitoring of pipe gallery cracks, seepage, and pipe leaks, providing comprehensive and real-time decision-making basis for the production process. For example... Figure 3A and Figure 3B The images shown are schematic diagrams of pipe cracks, oil pipe joint leaks, and ground water accumulation (running, leaking, and dripping) captured by a quadruped robot using a high-definition camera.

[0044] Meter recognition: Digital meter recognition: electricity meters, water meters, gas meters, digital display instruments, automatically extracting integers and decimals; Pointer meter recognition: pressure gauges, temperature gauges, oil pressure gauges, voltmeters, calculating pointer angles to convert to accurate values; Compatible with all types: mechanical roller meters, digital tubes, LCDs, round / square irregular dials, no need to modify existing old meters.

[0045] Infrared thermal imaging recognition: Temperature monitoring of specific equipment has always been a key focus of inspections in industries such as manufacturing and energy. Traditional detection methods have consistently faced three major challenges: detecting hidden defects, operating at night and in harsh environments, and the need for manual operation in high-risk scenarios. Quadruped robots, through infrared imaging equipment, can eliminate the need for manual entry into high-risk environments, eliminating the risks of electric shock, explosions, and falls from heights. They can also convert temperature anomalies at power cable joints into quantifiable images, providing managers with more intuitive decision-making support. Furthermore, they overcome environmental limitations such as light and smoke, enabling unmanned operation in key scenarios. Figure 4 The image shown is a schematic diagram of a pipe image acquired by a quadruped robot using infrared temperature measurement.

[0046] It should be noted that, Figures 3A to 4 These are example images collected by the quadruped robot of the present invention during an inspection. They are only used to visually demonstrate the recognition effect of visible light and infrared thermal imaging and do not constitute a limitation on the technical solutions claimed in this invention.

[0047] Monitoring objects: Temperature monitoring and early warning of metal surfaces on pipes after steam pipeline shut-off valves, surfaces of electrical equipment (switch cabinets, transformers, etc.), and non-contact equipment.

[0048] Technical logic: By using infrared cameras to capture thermal images of workshops or equipment, local temperature anomalies can be monitored.

[0049] Achieved effect: Monitor the operating temperature status of the equipment, read the temperature distribution data of the equipment in real time, perform long-distance thermal imaging of the target and analyze the temperature distribution data, and issue an alarm in real time if the temperature exceeds the limit.

[0050] Visual vibration analysis: A non-contact measurement method based on optical imaging and computer vision. It captures minute vibration displacement fields on the surface of monitored equipment using a high frame rate industrial camera (≥1000fps) or a laser Doppler vibrometer (LDV). Vibration modal parameters are extracted using image processing algorithms (such as phase optical flow and 3D digital image correlation). Visual vibration analysis can simultaneously monitor the full-domain vibration deformation (e.g., amplitude, frequency, and phase distribution) of the equipment surface, allowing for more precise analysis of structure, motion, deformation, and phase. It is suitable for monitoring large structural components, such as generator bodies, coal mills, and boilers.

[0051] In visual vibration analysis, phase optical flow and three-dimensional digital image correlation are two core non-contact measurement techniques. Both algorithms are based on computer vision, but they differ significantly in their underlying logic, preconditions, and applicable scenarios. Detailed logic analysis:

[0052] 1) Phase optical flow method The core logic of optical flow is "apparent motion estimation." It calculates the velocity and displacement fields of a structural surface by analyzing the spatiotemporal changes in pixel brightness within a video sequence. Optical flow is based on the fundamental assumption that the grayscale value (brightness) of a target pixel in an image remains constant within extremely short intervals. By calculating the rate of grayscale change and spatial grayscale gradient of each pixel in the image sequence, the algorithm can solve for the pixel's motion vector (i.e., optical flow) on the image plane.

[0053] No surface pretreatment required: Optical flow method directly utilizes the natural texture, edge or pattern information of the surface of the object being tested, without the need for special treatment or spraying of speckles on the structural surface, which makes it highly advantageous in field testing of large structures (such as bridges and buildings).

[0054] Algorithm Advancement and Noise Resistance: Traditional intensity optical flow methods are susceptible to changes in illumination and noise interference. To improve accuracy, this quadruped robot system employs phase optical flow (such as combining it with the texture analysis tool Gabor filter), which estimates displacement by extracting phase information from the image, resulting in better noise resistance and robustness.

[0055] Modal parameter identification: After extracting the dense displacement time history, the system sets up "virtual sensors" and uses system identification algorithms such as the random subspace method to identify modal parameters such as the structure's natural frequency, damping ratio, and mode shape distribution from the apparent motion response.

[0056] 2) Three-dimensional digital image correlation method The core logic of the 3D digital image correlation method is "stereo vision matching and 3D reconstruction". It accurately calculates the full-field 3D displacement and strain by tracking the positional changes of artificially created speckle patterns on the surface of an object in 3D space.

[0057] Surface speckle preparation: Unlike phase optical flow method, three-dimensional digital image correlation method creates a high-contrast random speckle pattern (such as black and white spray paint) on the surface to be measured as a "target" for algorithm tracking.

[0058] Binocular stereo vision mechanism: The 3D digital image correlation method uses two or more cameras to simultaneously capture images from different angles. Before measurement, the intrinsic and extrinsic parameters of the cameras need to be accurately calculated using a calibration board. Based on epipolar geometric constraints, the algorithm matches corresponding speckles in the left and right views, accurately reconstructing the 2D pixel coordinates into 3D physical world coordinates.

[0059] Local sub-region gray-level correlation matching: The algorithm defines sub-regions containing speckle patterns in the reference image and performs gray-level correlation matching in the deformed image sequence using shape functions and sub-pixel interpolation. By calculating the correlation peaks of sub-regions between adjacent frames, the pixel-level displacement time history of each measurement point in the entire field is accurately calculated.

[0060] Vibration mode extraction: After obtaining the three-dimensional dynamic displacement of the entire field, it is usually used as the output of a virtual sensor. The power spectral density matrix is ​​constructed by fast Fourier transform (FFT), and then the operating mode analysis techniques such as frequency domain decomposition are used to identify the natural frequencies and reconstruct the three-dimensional vibration modes.

[0061] In summary, specifically, optical flow is used to track the temporal displacement trajectories of tiny pixels on the equipment surface, extracting the time-domain sequence of macroscopic vibration displacement. Three-dimensional digital image correlation is used to perform sub-region matching on the speckle area of ​​the equipment surface, calculating the micrometer-level micro-deformation field and acquiring local high-frequency micro-vibration data. The two types of displacement time-series signals are then converted to the frequency domain using Fast Fourier Transform (FFT) to separate vibration modal parameters such as natural frequencies, damping ratios, and mode shapes, accurately capturing the vibration characteristics corresponding to early micro-wear, loosening, and cracks in components such as bearings, gears, and housings.

[0062] After extracting the vibration modal parameters, it is necessary to further establish the mapping relationship between the vibration modal parameters and the fault. Mechanical faults alter the physical structural characteristics of equipment (such as local stiffness, mass distribution, and damping characteristics), leading to characteristic shifts in its vibration modal parameters. Simply relying on the natural frequency and damping ratio cannot directly and uniquely determine the fault type; for example, the natural frequency reflects more of the structure's inherent dynamic properties. The following uses a core component fault as an example to illustrate the mapping judgment logic and change law between vibration modal parameters and faults: First, bearing wear (or increased clearance). Changes in natural frequency: Bearing wear leads to increased clearance and reduced support stiffness. As the wear on the bearing bush or bearing increases, the natural frequency of the rotor or shaft system decreases overall, and the critical speed also drops. Changes in damping ratio: Wear and increased clearance exacerbate friction and energy dissipation between components, typically leading to an increase in the system's damping ratio. Overall characteristics: In the frequency domain, in addition to the decrease in natural frequency, large-amplitude vibrations may occur at the harmonics of the rotational frequency. Furthermore, minor imbalances or misalignments can be amplified due to excessive clearance, resulting in a significant increase in resonant amplitude.

[0063] Second, gear tooth breakage or cracks. Natural frequency changes: Cracks or broken teeth weaken the local stiffness of the gear teeth. The larger the crack size, the more significant the drop in natural frequencies (especially lower-order natural frequencies); if the crack is located at the tooth root, the frequency drop is more significant than if it is located at the tooth tip. Mode shape changes: Gears with localized faults will exhibit significantly different mode shapes compared to normal gears. Obvious local resonance characteristics will appear in the vicinity of the crack or defect, with a significantly increased vibration amplitude. Time-frequency domain characteristics: Broken teeth or cracks will generate periodic impact pulses in the time domain, which manifests in the frequency domain as a significant increase in the amplitude of the gear meshing frequency, accompanied by a rich group of sidebands spaced at intervals of the faulty gear's rotational frequency.

[0064] Third, loosening of the shell / structure. Mode shape changes: Structural loosening (such as loose anchor bolts or deformation of the base foundation) has a strong directionality. In the direction of loosening, the amplitude of the mode shape will be much larger than in other directions. For example, the amplitude of the mode shape in the vertical direction may be much larger than that in the horizontal direction. Natural frequencies and modal characteristics: Loosening introduces nonlinear stiffness, which may lead to fractional harmonic resonances in the system (such as subharmonic resonances at 1 / 2 or 3 / 2 times the rotational speed). In terms of mode shape distribution, the local modes of the loosened parts will change, and even new local modes may emerge.

[0065] In this system, the mapping pattern from "modal parameters" to "fault type" is typically a combination of "mechanism model (pre-defined)" and "data-driven (machine learning)": Pre-defined physical mechanisms (prior knowledge): Based on dynamic theory and finite element simulation, engineers can pre-calculate the baseline modal parameters of equipment under healthy conditions, as well as the theoretical impact of various typical faults (such as stiffness reduction and mass loss) on frequencies and mode shapes. For example, "frequency reduction + abrupt change in local mode shape" corresponds to gear cracks, and "frequency reduction + anisotropic mode shape" corresponds to bearing wear. This constitutes the physical boundary for fault diagnosis.

[0066] The system employs data-driven feature mapping obtained through learning: Due to the complexity of actual working conditions, theoretical formulas alone are insufficient for accurate classification. Therefore, this system utilizes AI algorithms (such as support vector machines, neural networks, and extreme learning machines). Feature extraction: The massive time-series signals extracted using optical flow and 3D digital image correlation methods are processed using algorithms such as variational mode decomposition to extract feature vectors containing fault information. Pattern recognition training: The classifier is trained using historical fault samples or simulated experimental data. The system model can automatically learn the nonlinear mapping relationships between high-dimensional features such as natural frequency offset, damping ratio change rate, and mode shape spatial distribution under normal and various fault conditions.

[0067] Modal parameters extracted from quadruped robots serve as diagnostic "clues." Bearing wear primarily manifests as a decrease in natural frequency and an increase in damping due to reduced support stiffness; gear tooth breakage / cracks manifest as a decrease in specific-order frequencies, local distortion of mode shapes, and a surge in sidebands caused by local stiffness weakening; and shell loosening manifests as abnormal mode shape directionality and the appearance of nonlinear harmonics. These patterns of change rely not only on pre-defined physical mechanisms but also on classification models "learned" from a large amount of field data, thereby achieving accurate fault mapping.

[0068] Voiceprint recognition: Mechanical sound fault diagnosis analyzes the sound signals emitted by equipment to determine its health status and detect potential faults. This method utilizes physical phenomena such as vibration and noise in mechanical equipment, combined with signal processing and analysis techniques, to identify different types of fault modes. Common fault modes include friction, collision, wear, loosening, imbalance, cracks, cavitation, and whistling.

[0069] The core value of voiceprint recognition lies in its non-contact sensing and high-frequency signal capture capabilities. It can identify different types of fault modes. ① Mechanical structural failures. Such as bearing wear, abnormal gear meshing, minor wear of the gearbox, and broken teeth.

[0070] ② Electrical and insulation faults. For example, insulation deterioration or partial discharge inside transformers, switchgear, or overhead lines can generate high-frequency transient sound waves (ultrasonic band) and corona discharge sounds.

[0071] ③ Fluid and gas-related faults. Gas / liquid leaks, such as leaks in positioning pipes, valves, flanges, or electrolyte leaks in energy storage cabinets; internal leaks in valves and abnormal flow fields, such as increased high-frequency noise caused by aging cooling fans and worn bearings, as well as abnormal fan vibrations.

[0072] ④ Abnormal oil flow and terminals. In equipment such as transformers, oil flow noise characteristics can be detected; in energy storage cabinets, high-risk hazards such as loose battery terminals can be identified.

[0073] Gas detection: In complex environments such as industrial and energy sectors, specific equipment areas are prone to gas generation due to equipment malfunctions or material aging. Decomposition products, combustible gases ( ), toxic gases ( (And the risk of oxygen deficiency.) To ensure operational safety, the quadruped robot is equipped with a multi-gas composite detection system, which adopts advanced circuit design and mature core algorithm processing. It is suitable for detecting gas concentrations in pipelines, hazardous environments, and atmospheric environments, and can quickly and accurately detect the concentrations of multiple gases.

[0074] Temperature and humidity monitoring: Traditional methods for monitoring temperature and humidity in production scenarios involve installing temperature and humidity sensors at fixed locations. This approach cannot detect the temperature and humidity in all equipment areas, resulting in significant data inaccuracies. Quadruped robots can be equipped with high-precision temperature and humidity sensors. By dynamically scanning the equipment area, they can collect environmental temperature and humidity data in real time, enabling closed-loop management of equipment health and environmental safety. This ensures data accuracy and provides comprehensive data support for preventative maintenance, energy efficiency optimization, and fault tracing in various production scenarios.

[0075] S2: Spatial alignment of multimodal sensing data.

[0076] The quadruped robot generates severe vibrations when walking, causing spatial offsets in visual and infrared data. The first step is to perform dynamic spatiotemporal calibration: Spatial alignment: The robot's posture data is collected using the IMU (Inertial Measurement Unit) of the quadruped robot, and motion data of the encoders of each joint of the robot is obtained. Based on the posture data and motion data, motion compensation (EIS) is performed on the images from high-definition cameras, infrared thermometry, and visual vibration analysis to eliminate image jitter and distortion caused by quadruped gait and ensure that multi-source data are accurately aligned at the pixel level.

[0077] Specifically, the quadruped robot's body incorporates an IMU (Inertial Measurement Unit) to collect real-time, high-frequency data on three-axis acceleration, three-axis angular velocity, and body pitch / roll / yaw attitude. Each servo joint of the robot has a built-in high-precision encoder that transmits real-time data on joint rotation angle, speed, and displacement. This real-time attitude data and joint motion data are input into a motion compensation algorithm module to calculate camera jitter displacement and lens distortion offset during the robot's walking, climbing, obstacle crossing, and fixed-point detection processes. A global motion compensation matrix is ​​generated, and electronic image stabilization (EIS) is applied frame-by-frame to high-definition camera footage, infrared thermal imaging footage, and visual vibration acquisition images. This corrects image shifts and stretching distortions caused by gait impacts and body shaking pixel-by-pixel, ultimately achieving pixel-level precise alignment of visible light, infrared, and visual vibration images. This ensures that the coordinates of the same measurement point are perfectly matched in different imaging images, avoiding misalignment and distortion of vibration and temperature characteristics.

[0078] The quadrupedal inspection robot uses inertial measurement unit (IMU) and joint encoder data for motion compensation, with the core being multi-sensor fusion state estimation technology. By combining the high-frequency attitude perception of the IMU with the leg geometric constraints of the joint encoder, the robot can correct accumulated errors in real time, achieving high-precision pose and velocity estimation.

[0079] The motion compensation algorithm module is divided into five processing stages: multi-source data preprocessing, robot camera pose calculation, gait disturbance compensation modeling, global transformation matrix solving, and image geometric correction (multi-image pixel alignment). It integrates inertial navigation attitude calculation, robot forward kinematics, Kalman multi-sensor data fusion, homography matrix geometric transformation, and distortion correction models, belonging to a forward motion compensation scheme that combines robot motion perception and image geometric correction. This system is mainly based on three technical paths: The first method is multi-sensor fusion based on Extended Kalman Filter (EKF). EKF constructs state vectors to complementarily fuse IMU and joint encoder data.

[0080] IMU data is used for state propagation: High-frequency data from an IMU (containing an accelerometer and gyroscope) is used to calculate the robot's attitude, velocity, and position changes in real time through integration. Due to the integral characteristics, IMU data contains drift errors that accumulate over time.

[0081] Encoder data is used for state updates: using joint encoder data, the position of the foot in the body coordinate system is calculated through a forward kinematics model. Combined with the "plantar odometer" model (i.e., assuming the supporting foot does not slip on the ground and the foot velocity is 0 in the world frame), the velocity and position of the body in the world coordinate system are calculated to correct IMU drift.

[0082] The second method is zero-bias online estimation based on Error State Kalman Filtering (ESKF). To cope with complex terrain and long-term operation, ESKF is used in the body perception of quadruped robots.

[0083] Online IMU Bias Correction: Changes in operating temperature can cause IMU bias drift, which is a major cause of attitude error. ESKF estimates and corrects accelerometer and gyroscope biases in real time as part of the state vector, thereby ensuring attitude stability during long-term motion.

[0084] Foot-to-ground contact probability weighting: When fusing leg odometry information, ESKF introduces a foot-to-ground contact detection mechanism. By outputting a reliable "foot-to-ground contact probability" as a weight, the confidence level of the encoder data is dynamically adjusted, thereby effectively reducing the body velocity estimation error caused by foot slippage or suspension.

[0085] The third approach is kinematic trajectory optimization based on joint angle compensation. In addition to macroscopic state estimation, this system employs joint angle compensation at the microscopic motion control level to improve trajectory accuracy.

[0086] Kinematic error model compensation: Joint errors (such as mechanical clearance and assembly tolerances) directly affect the trajectory accuracy of robot motion. By establishing a leg kinematic error model, the system can calculate the compensation amount for the joint output angle, thereby improving the positional accuracy of the leg foot and optimizing the motion trajectory of the torso center.

[0087] The quadruped inspection robot uses an IMU and joint encoders for motion compensation. Essentially, it achieves the complementary advantages of "high-frequency short-time accuracy of the IMU" and "low-frequency long-time stability of the encoder" through a Kalman filter framework (EKF / ESKF). It also combines foot odometry and zero-bias online estimation to eliminate accumulated errors and slippage effects. At the same time, joint angle compensation is used at the underlying control level to ensure the robot's motion accuracy and stability in complex environments.

[0088] Spatial alignment of quadruped inspection robots is crucial for ensuring the accurate fusion of multimodal sensor data (such as vision, LiDAR, and inertial measurement units). Due to the violent shaking of the quadruped robot's torso during movement, and the differences in physical installation positions and time synchronization among different sensors, spatial alignment typically requires a combination of multi-sensor joint calibration and feature-level fusion algorithms. The specific methods employed in this system are mainly divided into the following levels:

[0089] (1) Basic physical space alignment (joint calibration) This is the cornerstone of spatial alignment, designed to unify data from different sensors into a global or body coordinate system.

[0090] Intrinsic and extrinsic parameter calibration: Intrinsic parameter calibration eliminates the inherent distortion of optical sensors such as cameras; extrinsic parameter calibration calculates the precise rotation and translation relationships (i.e., spatial positioning) of each sensor (camera, lidar, inertial measurement unit) relative to the robot body or high-precision map.

[0091] Based on trajectory and motion consistency alignment: Addressing the complex motion excitation characteristics of quadruped robots, a Lie algebra based on B-splines is employed to fit and align the sensor trajectory. By matching the IMU pre-integrated trajectory with the visual / laser odometry trajectory, rotational extrinsic parameters and time offset are jointly optimized to achieve high-precision spatiotemporal calibration.

[0092] Targetless Joint Calibration: In complex inspection environments, it is difficult to frequently place calibration boards. A targetless framework is adopted, utilizing the visual optical flow during robot movement and the edge of the LiDAR to perform spherical cross-modal correlation, jointly optimizing and estimating extrinsic parameters to achieve centimeter-level and sub-degree-level alignment accuracy.

[0093] (2) Feature-level spatial search and dynamic compensation alignment During the dynamic movement of a robot, strict physical alignment may fail due to calibration residuals or motion deformation, requiring dynamic compensation after feature extraction.

[0094] Feature-level spatial search alignment: Point cloud features are mapped onto a multi-scale feature map of the image to obtain keypoint coordinates. These coordinates are then used as pointers, and a learnable region offset is added around them for searching to obtain the corresponding image region features. This method effectively compensates for spatial biases introduced after feature extraction.

[0095] Weak Alignment Adaptive Correction: To address weak alignment issues at scale or location caused by sensor differences, a distribution alignment module based on optimal transport theory or an attention-guided deformable convolutional alignment module is employed. Accurate pixel-level or region-level alignment is achieved by learning feature offsets, avoiding the performance loss caused by forced alignment.

[0096] Spatial Transformation-Aware Geometric Encoding: A geometric position encoding mechanism based on complex rotation representation is designed to enhance the model's perceptual adaptability to rigid spatial transformations and viewpoint rotations. Spatial relative position feature similarity is introduced into the attention calculation module to improve orientation perception and geometric consistency during cross-modal fusion.

[0097] (3) Multimodal and multi-level matching and semantic alignment To achieve high-precision mapping and localization of quadruped robots in complex scenarios, data association needs to be implemented at a higher level.

[0098] Multimodal multi-level matching algorithm: Inspired by human spatial cognition, it uses observation data acquired by cameras and lidar to perform multi-level matching, achieving accurate and robust data association, and then constructing a globally consistent 3D point cloud map through factor graph optimization.

[0099] Implicit semantic alignment: Automatically learns a shared latent space through neural networks. Dynamically allocates cross-modal weights using attention mechanisms, or constructs a unified semantic embedding space based on contrastive learning (such as the CLIP multimodal pre-trained model), enabling different modalities to achieve consistency in deep semantics.

[0100] Decoupling and Hierarchical Cross-Modal Alignment: To address the heterogeneity between modalities (such as differences in image and point cloud distributions), a hierarchical alignment framework (such as a multimodal decoupling strategy) is adopted. Features are decoupled into modality-specific (heterogeneous) and modality-shared (homogeneous) parts, and alignment is performed using optimal transmission and maximum mean difference regularization, respectively, to enhance cross-modal semantic consistency.

[0101] Spatial alignment of quadrupedal inspection robots is a progressive process from static physical calibration to dynamic feature compensation, and then to deep semantic fusion. This system first obtains high-precision initial extrinsic parameters through leg-eye calibration or trajectory alignment, and then introduces deformable convolution or multi-layer matching algorithms at the algorithm level to cope with the dynamic deviations caused by the violent movements of quadrupedal robots, ultimately achieving accurate spatial alignment of multimodal data.

[0102] Based on the robot's 3D LiDAR, a global point cloud coordinate system is constructed on-site. The factory calibration intrinsic parameters and on-site extrinsic parameters of each sensor are imported into a pre-configured coordinate transformation matrix. All data, including high-definition video images, infrared thermal imaging, acoustic signals, gas concentration data, and temperature and humidity data, are uniformly mapped to the global point cloud coordinate system. This completes the spatial registration of multimodal sensing data, eliminates spatial misalignment problems caused by the installation position deviation and orientation deviation of multiple sensors, and provides a unified spatial benchmark for subsequent multimodal fusion analysis.

[0103] Among them, the camera intrinsic parameters (focal length, principal point coordinates, distortion coefficients) are obtained from the manufacturer's laboratory calibration using factory calibration results; acoustic, gas, and temperature / humidity parameters are for non-imaging sensors and have no optical intrinsic parameters, only defining their own installation reference coordinate system. In the spatial registration of multimodal perception data of the inspection robot, the transformation relationship (i.e., extrinsic parameters) between the sensor coordinate system and the global point cloud coordinate system is calculated through a calibration algorithm. The core method for determining the transformation relationship between the sensor coordinate system and the point cloud coordinate system is a data-driven extrinsic parameter calibration algorithm. Depending on the application scenario and sensor type, the following calibration methods are used: The first method is a joint calibration method based on cooperative targets: used to determine the relative pose of sensors such as cameras and LiDAR. The core logic is as follows: Place calibration objects with known geometric features (such as checkerboard patterns, 2D markers, or custom-designed perforated plates) in the scene. The camera extracts the 2D pixel coordinates (corner points or marker centers) of the calibration plate through image processing, while the LiDAR extracts the 3D point cloud features of the calibration plate through scanning (such as fitting plane equations or extracting hole centers). The algorithm solves by establishing 2D-3D or 3D-3D correspondences using these matched feature points. An n-point perspective localization algorithm or a point cloud registration algorithm (based on SVD singular value decomposition) is used to solve for the initial rotation matrix and translation vector. Finally, nonlinear optimization (such as bundle adjustment) is used to minimize the reprojection error, thereby obtaining a high-precision extrinsic parameter matrix.

[0104] The second method is targetless / natural feature calibration: For situations where the inspection robot is already deployed and frequent placement of calibration boards is inconvenient, the system utilizes natural features in the environment for online or offline calibration. Core logic: The system automatically extracts features such as edges and lines from camera images, and simultaneously extracts corresponding 3D structural features (such as building corners, ground, and equipment bodies) from the LiDAR point cloud. Algorithm solution: By maximizing the gradient alignment between image edges and point cloud projection edges, or minimizing the distance from points to surfaces, an iterative optimization algorithm continuously fine-tunes the extrinsic parameter matrix. This method requires no manual intervention and can continuously monitor and automatically calibrate minor sensor offsets caused by vibration during robot operation.

[0105] The third method is a joint calibration method based on motion trajectory: For the joint calibration of inertial measurement unit (IMU) and lidar, a motion trajectory-based method is typically used. The core logic is to control the inspection robot to perform movements within the site that include sufficient excitation (such as translation, rotation, and turning). The pre-integration algorithms of the lidar odometry and IMU are run separately to obtain their respective motion trajectories. Algorithm solution: Constraints are constructed using the turning area or a large-scale motion trajectory. The two trajectories are aligned, and the rotation, translation, and time offset parameters between them are solved through joint optimization.

[0106] The fourth method is direct point cloud registration: This method is often used when it is necessary to directly calibrate the relationship between two lidar sensors (or multiple stations). Core logic: Two sensors simultaneously scan a scene with overlapping areas, extracting common geometric features from their respective point clouds. Algorithm solution: Using the direct point cloud registration algorithm or its improved versions (such as iterative nearest point, normal distribution transformation), the algorithm continuously searches for nearest point pairs and calculates the optimal rigid body transformation, gradually minimizing the matching error between the overlapping point clouds, thereby solving for the accurate coordinate transformation matrix.

[0107] The coordinate transformation matrix pre-configured in the inspection robot system can be calculated using the aforementioned calibration algorithm combined with a high-precision optimization model during deployment or system initialization. These algorithms eliminate installation deviations through mathematical methods (such as SVD decomposition, least squares method, and nonlinear optimization), ensuring that heterogeneous data such as high-definition video and infrared thermal imaging can be accurately mapped to the global three-dimensional point cloud coordinate system.

[0108] The spatial alignment includes: when the lidar is effective, constructing a global point cloud coordinate system of the field environment based on the 3D lidar, and mapping the data of each sensor to the global point cloud coordinate system; when the lidar fails in a dusty or foggy environment, switching to a temporary spatial coordinate system constructed with millimeter-wave phased array radar detection data as the spatial reference to complete the spatial registration of multi-sensor data.

[0109] The two spatial alignment logics described above address spatial misalignment issues at different levels: motion compensation based on IMU and joint encoders eliminates inter-frame jitter and distortion caused by the quadruped robot's gait, ensuring the continuity and stability of multi-source image data at pixel-level accuracy; global point cloud coordinate mapping based on 3D LiDAR or millimeter-wave radar unifies the spatial reference between sensors, eliminating data misalignment caused by differences in sensor installation positions and orientations. These two logics work together to achieve complete spatial alignment of multimodal sensing data.

[0110] S3: Extract features from the spatially aligned multimodal sensing data and convert them into multidimensional feature vectors.

[0111] Multimodal sensing data is transformed into standardized feature vectors. For example, high-definition video and meter recognition are transformed into "state semantic features" (such as switch on / off status and reading); infrared thermometry and temperature and humidity detection are transformed into "thermodynamic features"; and voiceprint recognition and visual vibration analysis are transformed into "dynamic features" (such as bearing wear frequency and abnormal resonance).

[0112] Specifically, for high-definition video images, semantic features of equipment status are extracted using a target detection network, including equipment on / off status and instrument readings, and status identification is performed for pipe gallery cracks, seepage, and pipe leaks; for infrared thermal imaging, thermodynamic features are extracted, including equipment surface temperature distribution, highest temperature point, and temperature value; for acoustic signals, dynamic acoustic features are extracted, including loudness, sharpness, center of gravity frequency, and spectral centroid; for visual vibration images, dynamic vibration features are extracted using optical flow or digital image correlation methods, including vibration amplitude, vibration frequency, and phase distribution; for gas concentration data, the current gas concentration value is extracted; and for temperature and humidity data, the current ambient temperature and humidity values ​​are extracted.

[0113] The aforementioned feature values ​​are concatenated according to preset dimensions to form a one-dimensional or multi-dimensional feature vector, which serves as the input for subsequent causal reasoning models.

[0114] Among them, meter recognition is performed on high-definition video images, such as... Figure 5 As shown, it includes: S30: Preprocess high-definition video images, including at least strong light reflection suppression, low-light brightness enhancement, and perspective distortion correction.

[0115] Unified restoration of real-shot images: Gaussian noise reduction, strong light reflection suppression, low light brightening, perspective distortion correction (can straighten the dial even when the meter is tilted 30~60°), binarization to separate the digits / pointers from the background.

[0116] S31: Locate the region of interest (ROI) of the dial in the image using an object detection network to distinguish between digital and analog meters.

[0117] A lightweight model (adapted to low-computing-power onboard chips for robots) is adopted: all meters are selected in complex backgrounds (distribution cabinets, pipes, walls); digital meters and pointer meters are distinguished simultaneously and distributed to different recognition branches; cropped pure meter face images are output, and irrelevant backgrounds are removed to reduce computation.

[0118] S32: For digital meters, a character recognition network is used to segment and recognize the numbers.

[0119] For digital meters (segment display / LCD / roller water meters), a CRNN / OCR character recognition network is used to segment each digit and decimal point; special training samples are used for seven-segment displays to solve the recognition problems of "half-character skipping and missing strokes due to glare"; and the value is completed according to the meter's range rules (e.g., 6-digit integer + 2-digit decimal meter).

[0120] S33: For pointer-type meters, the coordinates of the center of the dial, zero scale, full scale and pointer tip are located by key point detection, the angle between the pointer and the zero scale is calculated and the reading is converted according to the range ratio.

[0121] For pointer-type instruments (pressure gauges, temperature gauges), two main technical approaches are adopted: Key point detection: Locate the center of the dial, zero mark, full scale, and pointer tip coordinates; calculate the angle between the pointer and the zero mark using the arctangent function, and convert the reading according to the range ratio; Semantic segmentation + Hough transform: Segmenting scale lines and pointer outlines, fitting pointer angles at the sub-pixel level, supporting dual-pointer and multi-range switching. Specifically, this system employs a four-level recognition process: "coarse positioning → fine segmentation → geometric analysis → range adaptation," with layered constraints to ensure no confusion between dual pointers and automatic adaptation to multiple ranges: Lightweight target detection (coarse positioning): Quickly detects the instrument dial area from complex inspection screens, distinguishes instrument types, and initially identifies the number of pointers; High-precision semantic segmentation (fine segmentation): Performs pixel-level segmentation on the dial ROI area, accurately distinguishing four types of pixels: dial surface, scale area, long pointer, and short pointer, solving the problems of dual pointer adhesion and edge blurring; Hough transform geometric analysis (angle solution): Performs line detection on the segmented pointer mask, accurately extracting the deflection angles of each pointer; Multi-range adaptive matching algorithm: Combines the dial scale segmentation results with pointer angles to automatically determine the current instrument range specifications, dynamically switching reading conversion formulas to achieve multi-range compatible readings.

[0122] S34: Output recognition confidence level. When the confidence level is lower than the preset threshold, the recognition is determined to be unsuccessful. The quadruped robot is controlled to automatically retake the image and the recognition result is time-series verified. The recognition data and the original image are uploaded to the remote monitoring platform together.

[0123] Post-processing verification and confidence output: Output recognition confidence (0~100%). If it is lower than 85%, it is considered a recognition failure and the robot will automatically retake the picture; Time sequence verification: Compare with the previous round of inspection readings and filter out abnormal values ​​that do not conform to physical logic; Structured output: Values, units, table numbers, fault labels, and original images are uploaded to the remote monitoring platform.

[0124] Furthermore, the meter identification also supports condition defect identification (core value-added function of inspection): dial abnormalities: glass fogging, water ingress, oil stains, cracks, reflection obstruction, incomplete numbers; equipment failures: pointer jamming, abnormal zeroing, dial damage, missing cover; status identification: switch opening and closing, indicator light on and off, equipment leakage, dial corrosion; jump warning: compare with historical readings, sudden increase / decrease in value, automatic alarm for exceeding threshold.

[0125] Specifically, in quadruped robot inspection, meter recognition goes beyond simply "automatic reading" to identify operational defects. Essentially, it's a comprehensive application of "visual feature extraction + temporal logic verification + multimodal cross-validation." By integrating computer vision, deep learning, and multimodal perception, the meter recognition module can achieve a comprehensive diagnosis of equipment appearance, operating status, and potential faults.

[0126] In this system, the specific logic for meter identification to achieve operational defect identification is as follows: (1) Dashboard anomaly identification: based on image quality assessment and visual feature extraction In response to abnormal physical conditions of the dial itself, the algorithm mainly relies on detailed analysis of image clarity, texture, and optical features.

[0127] Fogging, water ingress, and oil stains: These phenomena manifest in images as localized blurring, water droplet-like highlights, or large areas of semi-transparent obstruction. AI algorithms can accurately determine whether the watch face is contaminated or the seal has failed by comparing image edge gradients, identifying abnormal reflective areas, and detecting abrupt texture changes.

[0128] Cracks and missing numbers: Cracks exhibit obvious linear irregularities, while missing numbers cause a sharp drop in character recognition confidence. By locating the dial area using object detection algorithms and combining this with integrity verification through character recognition (OCR), the system can automatically mark dial damage or worn-out scales / numbers.

[0129] Reflective glare: Strong light reflections can obscure the true information on the dial. The system can identify the location of reflections by analyzing the image histogram or localized overexposed areas, and then coordinate with the robotic gimbal to adjust the shooting angle for secondary data acquisition.

[0130] (2) Equipment fault identification: based on geometric topology and kinematic analysis Equipment failures are often accompanied by deformation or abnormal movement of the mechanical structure, which requires visual algorithms to have accurate perception of the target's geometry and spatial position.

[0131] Pointer jamming and zero-point drift: The system accurately extracts the pixel-level mask of the pointer through semantic segmentation and calculates its deflection angle. If the equipment is in a stopped state but the pointer does not return to zero, or if the pointer angle shows a significant deviation under the same operating conditions in multiple consecutive inspections, it can be determined that there is mechanical jamming or zero-point drift.

[0132] Dial damage and missing watch cover: Using image classification or object detection techniques, the algorithm can learn the visual features of a normal watch cover. When the watch cover is missing, the internal structure of the dial is exposed, and the image features change significantly. The system can identify such serious appearance defects in seconds.

[0133] (3) State recognition: based on multimodal perception and pattern matching Status recognition is a core value-added function of inspection, covering a comprehensive assessment from machine position to environmental safety.

[0134] Switch opening and closing and indicator light on / off: Through high-precision target detection and color / shape recognition, the algorithm can accurately determine the mechanical position of the knife switch and knob, as well as the color and on / off status of various status indicator lights, replacing manual visual verification.

[0135] Equipment leaks and dial corrosion: Leaks (such as oil stains and water marks) appear visually as dark patches or abnormal reflections; corrosion presents as specific reddish-brown textures. Combining infrared thermal imaging (such as detecting localized temperature differences caused by leaks) can achieve high-confidence identification of potential leaks.

[0136] (4) Jump warning: based on time sequence logic and comparison with historical data Alternating warning is the key to meter identification moving from "perception" to "cognition," as it combines the results of a single image recognition with the temporal patterns of equipment operation.

[0137] Sudden increase / decrease in value: The system not only reads the current value, but also compares it with historical inspection data and data from adjacent measuring points. If the rate of change (slope) of the reading exceeds the physical threshold set based on thermodynamic or fluid dynamic laws, the system can determine that there is an identification error or that the equipment has actually experienced a drastic change in operating conditions.

[0138] Automatic alarm for exceeding threshold: The system verifies the identification results in real time by combining the preset safe operating range (such as 1 / 3 to 2 / 3 of the range). Once the reading exceeds the limit, an audible and visual alarm will be triggered, and video clips and thermal images before and after the abnormality will be automatically captured to generate a structured alarm work order containing the fault location, abnormal value, and on-site environment, which will be pushed to the centralized control backend.

[0139] Furthermore, the meter identification also supports data closed-loop management functions: real-time edge identification: robot-mounted edge computing, results are generated in milliseconds after taking a picture, without the need to send it back to the cloud for further processing; image archiving: each set of readings is bound to the original photo for auditing and dispute tracing; automatic platform upload: readings, confidence level, equipment number, inspection time, and location coordinates are synchronized to the monitoring system; multi-source verification: fault-tolerant verification is performed by combining time sequence logic and range rules (for example, the daily electricity consumption of an electricity meter is unlikely to be thousands of kWh), reducing false alarms; unmanned replacement of manual labor: fully automatic meter reading in high-risk areas (high-voltage power distribution rooms, underground pipe corridors, toxic chemical areas), reducing the need for personnel to enter the site.

[0140] In the process of meter recognition (such as pointer-type meters and digital meters) by quadruped inspection robots, errors are easily generated in a single recognition due to the severe shaking of the robot body, changes in lighting in complex environments, and glare interference from the dial glass. Combining range rules (spatial physical boundaries) and temporal logic (time evolution constraints) for fault-tolerant verification is the core means to ensure high confidence in readings and prevent false alarms.

[0141] The specific implementation plan can be developed from the following three dimensions: First, physical rationality verification based on range rules (spatial defense). Range rules are used to determine whether a single reading violates physical common sense or equipment safety boundaries, and are mainly used to filter out "outliers" caused by visual algorithm failure.

[0142] Absolute boundary and dead zone determination: The system first checks whether the identified reading exceeds the instrument's physical range (for example, if a pressure gauge has a range of 0-10MPa, and the visual algorithm identifies 12MPa or -1MPa), then it is directly determined as invalid data and a sensor or visual recognition fault alarm is triggered. Simultaneously, according to the instrument specification, if the reading remains below 10% or above 90% of the upper limit of the range for an extended period (accuracy typically decreases in these ranges), the system should reduce the confidence level of the data.

[0143] Dynamic operating range verification: Based on the current operating conditions of the equipment, verify whether the reading is within a reasonable operating range. For example, for stable pressure measurement, the normal operating pressure should be between 1 / 3 and 2 / 3 of the instrument's range. If the reading is consistently outside this range, the system can determine that the range selection is incompatible or the equipment is in an abnormal overload / no-load state.

[0144] Change rate (slope) limit: Abrupt changes in physical quantities (such as temperature and pressure) are limited by thermodynamic or fluid dynamic laws. The system can set a maximum change rate threshold. If the change rate of readings in two adjacent inspections (or continuous sampling within a very short period of time) exceeds this threshold, it is judged as a visual recognition error (such as misidentification of pointer reflection) and is discarded or processed by moving average filtering.

[0145] Second, state consistency verification based on time-series logic (time defense). Timing logic is used to verify whether the evolution of readings over time conforms to the physical laws of equipment operation, preventing false alarms or data jumps caused by momentary interference.

[0146] Persistence and Time Filtering: Before triggering an anomaly alarm, the system introduces a time-dimensional AND logic. For example, it not only requires "reading > 90℃" but also requires "this abnormal state must be maintained continuously for T seconds (e.g., 500ms to 3s)". This effectively filters out instantaneous anomalies caused by violent movements of the quadruped robot or brief external obstructions.

[0147] Hysteresis Comparison and Dead Zone: To prevent the system from frequently switching between "normal" and "alarm" states (i.e., "alarm jitter") due to fluctuations in the reading around the threshold boundary, a hysteresis mechanism needs to be introduced into the timing logic. An alarm is triggered when the reading exceeds the threshold or dead zone, and the alarm is only deactivated when the reading falls back to the threshold or dead zone.

[0148] State machines and causal timing constraints: Using timing logic specifications, the operation of the equipment is written as a "logic instruction manual". For example, a timing rule can be set: "If the motor speed > 0, then the bearing temperature must show an upward trend within 5 seconds". If the system detects an increase in speed but a continuous decrease in temperature reading, it is determined that the temperature gauge has failed to recognize the data or that the data is abnormal.

[0149] Third, visual loop closure and multi-source cross-validation (system-level fault tolerance). When the reading of a single meter appears reasonable in terms of range and timing, but there is still uncertainty, a system-level fault tolerance and error correction mechanism needs to be introduced.

[0150] Viewpoint correction and secondary acquisition closed loop: For target areas such as dashboards, the system performs dual invalidation by analyzing the center position offset and geometric deformation factor. If the current image is determined to be invalid (e.g., severe reflection, excessive tilt angle), the quadruped robot is driven to perform viewpoint correction and adjustment according to the type of deviation, and a second acquisition is performed for re-identification until a valid image is obtained.

[0151] Redundant voting mechanism: For critical safety parameters, a two-out-of-three redundancy design can be adopted. The system calculates the absolute difference between three independent sensor readings or multiple independent identification readings in real time. If the difference exceeds a preset threshold and persists for a certain period of time, a signal out-of-synchronization alarm is triggered. The majority of consistent values ​​are selected as the valid output through a voting algorithm to mask single-point identification failures.

[0152] Cross-validation and adaptive thresholds: Cross-validation is performed using the multimodal perception of the quadruped robot. For example, when the system visually detects an abnormal temperature at a certain node, it needs to combine infrared thermal imaging data to confirm whether there is actually a thermal defect at that location; or in gas / pressure monitoring, the warning threshold is dynamically adjusted based on the variance of recent sampled concentrations to reduce misjudgments caused by environmental fluctuations.

[0153] In summary, in meter recognition by quadruped inspection robots, range rules act as "physical boundaries in space," used to eliminate outrageous visual recognition errors; temporal logic acts as "evolutionary constraints in time," used to filter transient interference and prevent alarm jitter. Combining these two, supplemented by robot perspective correction and multimodal cross-validation, allows for the construction of a highly robust fault-tolerant verification system, ensuring the authenticity and validity of inspection data.

[0154] Furthermore, the meter recognition also supports environmental adaptive capabilities: automatically processing images blurred by backlight, low light, dust, multi-angle tilt, and vibration; and coordinating with robot gimbal zoom and fill light to focus the dial at close range.

[0155] Specifically, the environmental adaptability of the lightweight meter recognition algorithm is a core supporting capability that ensures stable readings by the robot in complex industrial environments and under non-ideal shooting conditions. Unlike standard laboratory scene recognition, it can fully adapt to interference from real inspection environments such as lighting, angle, image quality, dirt, working condition disturbances, and equipment differences. Specifically, it includes the following six categories of implementable supporting adaptive capabilities: The first category is the ability to adapt to different lighting environments. To address issues such as strong light reflection, low light at night, uneven brightness, and localized light spots in industrial settings, a complementary software and hardware co-adaptive optimization system is implemented to resolve pointer segmentation failure, Hough line misdetection, and reading deviation caused by light interference.

[0156] Adaptive Image Enhancement: Automatically performs contrast-limited adaptive histogram equalization and image enhancement technology illumination compensation to correct issues such as darkness in low light and uneven grayscale in the image, improve the contrast of the dial scale and hands, and ensure the accuracy of semantic segmentation pixel classification.

[0157] Adaptive reflection suppression: For strong light reflection and local overexposure scenes on instrument glass surfaces, the system filters out reflective noise by repairing pixels in the mask area and reconstructing gradient features, preserving the complete pointer and scale outlines and avoiding false detection of reflective edges by Hough transform.

[0158] Day and night dual-mode adaptive: Linked with hardware infrared supplementary lighting strategy, the algorithm automatically recognizes the ambient light level, adapts to night vision imaging characteristics in low light scenes, requires no manual parameter adjustment, and provides stable recognition in all weather conditions.

[0159] The second category is the ability to adapt to shooting posture and deformation. It adapts to slight pose deviations in fixed-point shooting for quadruped robots, and solves angle calculation errors caused by tilt, perspective distortion, and shooting distance deviation, making it suitable for regular inspection shooting scenarios where the device is not directly facing the dial.

[0160] Adaptive perspective distortion correction: Through homography matrix transformation, adaptive correction is performed on tilted and perspective-distorted dial areas to restore the standard round dial shape and ensure accurate calculation of pointer deflection angle.

[0161] Adaptive shooting distance: The algorithm is compatible with conventional inspection shooting distances of 0.5–5 meters. It relies on target detection adaptive scaling and semantic segmentation multi-scale feature extraction to adapt to differences in dial size and will not fail to recognize due to focusing deviations at different distances.

[0162] Slight jitter anti-interference: For slight jitter in the image caused by robot walking and equipment operation, the instantaneous angle jump error is suppressed by inter-frame smoothing filtering and pointer angle timing calibration.

[0163] The third category is the ability to adapt to dial damage and image quality degradation. It is suitable for aging scenarios such as dust, stains, water mist, minor scratches, and fading that occur during long-term operation of industrial field instruments, and solves the problems of traditional Hough transform being susceptible to noise interference and breakage at the segmentation edge.

[0164] Adaptive noise filtering: For salt-and-pepper noise and dust in the image, it automatically matches morphological filtering parameters and filters out invalid interference pixels while preserving the core features of the pointer and scale.

[0165] Adaptive repair of missing features: For wear on the edge of the pointer, local fading of the scale, and smudges on the dial, the missing pointer outline is completed by relying on the semantic prior of the semantic segmentation pixels, ensuring the complete mask output of the two pointers and supporting the accurate fitting of straight lines by Hough transform.

[0166] Adaptive background removal for complex backgrounds: Automatically removes complex background interference such as equipment cabinets, pipelines, and debris outside the dial, accurately locks the effective area of ​​the dial, and avoids misidentification caused by background clutter.

[0167] The fourth category is dynamic adaptive capability under operating conditions. By combining the operating conditions of the equipment with the inspection conditions of the robot, the algorithm strategy can be dynamically adapted to avoid invalid identification and erroneous readings caused by operating condition disturbances.

[0168] Vibration adaptive adaptation: Linking with equipment vibration and stress wave operating data, the inter-frame mean smoothing strategy is automatically activated during periods of strong equipment vibration to suppress reading fluctuations caused by pointer dynamic jitter.

[0169] Start-stop variable operating condition shielding adaptation: Under the conditions of equipment start-stop and speed fluctuation, the operating condition is automatically determined, instantaneous jump readings are filtered and de-jittered, and a steady-state effective reading is output.

[0170] Multi-range dynamic adaptive: Automatically identifies the scale characteristics of different dial specifications, dynamically switches the range conversion formula, and adapts to different models and ranges of meters.

[0171] The fifth category is the generalization and adaptive capability of multiple meter types. It breaks free from the limitations of single template matching, adapts to the different forms of various pointer instruments in industrial sites, and achieves compatibility of multiple types of meters with a single algorithm framework.

[0172] Single / Dual Pointer Adaptive Recognition: Automatically determines the number of instrument pointers, adapts to single-pointer conventional instruments and long / short dual-pointer instruments, and intelligently distinguishes between reference pointers and real-time measurement pointers.

[0173] Dial style generalization adaptation: compatible with pressure gauges, temperature gauges, ammeters, and voltmeters with different color schemes, scale layouts, and dial sizes, without the need for separate modeling for each device.

[0174] The sixth category is the ability to adapt to and adapt to the edge environment. It adapts to the embedded computing power and network environment of robots, ensures the stability of complex field deployments, and achieves the ability to adapt to the operating environment.

[0175] Computational adaptive inference: The lightweight model dynamically adapts to the computing resources on the edge, automatically adjusting the inference accuracy and frame rate, ensuring real-time inspection speed on low-computing-power devices without lag or frame drops.

[0176] Network adaptive caching: Supports local image and recognition result caching in weak network or network outage environments, and automatically retransmits data after the network is restored to ensure that the inspection data is complete and not lost.

[0177] The system's environmental adaptability is all pre-served in the four-stage pipeline of detection-segmentation-Hough transform-range conversion: through environmental preprocessing, dynamic parameter adaptation, working condition anti-shake, and feature repair, it solves the problems of pointer omission, false detection, angle deviation, and range adaptation failure in complex environments from the source, and ultimately ensures the stable implementation of dual pointer recognition and multi-range switching functions in industrial scenarios.

[0178] S4: Input the feature vectors of each dimension into the causal reasoning model, perform multi-source data cross-validation, and output the causal reasoning results and equipment health status vector. The causal reasoning model at least supports joint diagnosis of sound-vibration-temperature and / or gas-temperature-visual linkage for leak tracing.

[0179] This is the "AI brain" of the method of the present invention. It abandons simple data superposition and adopts a causal graph reasoning mechanism to eliminate false alarms through cross-validation of data from different modalities.

[0180] Specifically, a causal reasoning graph model for equipment failure is built. The graph nodes include equipment components, failure types, and multimodal characteristic indicators (temperature peak, vibration frequency domain amplitude, abnormal noise spectrum, and harmful gas concentration). The graph edges are pre-labeled with the causal relationships and threshold correspondences between failures and multimodal abnormal features. Multimodal features such as infrared temperature rise, vibration mode, acoustic peak value, and gas concentration are simultaneously input into the causal graph inference engine. The engine traverses the graph correlation to complete the cross-validation of multimodal data: when a single mode exceeds the threshold but other modes are not abnormal, it is determined to be environmental interference or sensor noise, and false alarms are automatically eliminated; when multiple modes simultaneously show corresponding abnormal features and match the causal path of the graph, a real equipment fault warning is output, which greatly reduces the false alarm rate of on-site inspection.

[0181] The combined sound-vibration-temperature diagnosis is as follows: when both visual vibration characteristics and acoustic characteristics are abnormal, infrared thermal imaging characteristics are invoked in conjunction. If the temperature of the corresponding part rises abnormally at the same time, it is confirmed as a mechanical wear-related fault; if the temperature is normal, it is determined to be an external interference. Specifically, when "visual vibration analysis" detects high-frequency vibration of the motor and "voiceprint recognition" captures bearing friction sound in a specific frequency band, the algorithm will not immediately trigger an alarm. Instead, it will link up and call up "infrared temperature measurement" data. If the temperature of that part shows an abnormal rise, it is confirmed as a mechanical wear fault; if the temperature is normal, it may be due to external resonance interference, thus significantly reducing the false alarm rate.

[0182] The gas-temperature-visual linkage leak tracing method is as follows: when the gas concentration characteristics are abnormal, the infrared thermal imaging and high-definition video images at the first preset time before and after the trigger time are automatically retrieved. The abnormal temperature difference is identified by infrared thermal imaging and the leak source status is identified by combining the high-definition video images to locate the leak source.

[0183] Specifically, when the "gas detection" detects that the concentration of combustible gas exceeds the standard, the algorithm automatically triggers "high-definition video recording" and "infrared thermometry" to perform a backtracking scan of the area. By using thermal imaging to find abnormal temperature differences in pipeline leaks, and combining this with visual recognition of valve / flange status, the source of the leak can be accurately located.

[0184] The causal reasoning model supports environmental adaptive compensation: using temperature and humidity characteristics as environmental benchmarks, it dynamically corrects the alarm thresholds of infrared thermal imaging and gas concentration data.

[0185] For example, in high temperature and high humidity environments, the algorithm automatically raises the alarm baseline of thermal imaging to avoid false alarms caused by environmental heat radiation.

[0186] S5: Calculate the confidence score based on the causal reasoning results, issue graded warnings based on the confidence scores, and send the warning signals back to the remote monitoring platform.

[0187] This step S5, based on the results of causal reasoning, performs confidence assessment and tiered early warning decisions for equipment anomalies. Simultaneously, in the voiceprint signal dimension, it implements tiered early warning through three-mode fusion early warning logic.

[0188] Specifically, based on the results of causal reasoning, the algorithm introduces a dynamic confidence assessment model to provide tiered early warnings for abnormal events: Confidence score calculation: The confidence score (0-100%) of the current anomaly is calculated by combining the clarity of data from each sensor (such as the percentage of pixels identified by the meter), the degree of environmental interference (such as the impact of dust on vision), and the consistency of multimodal cross-validation.

[0189] In the joint diagnosis of "sound-vibration-temperature" and the linkage of "gas-temperature-visual" leak tracing, the causal reasoning graph model transforms scattered monitoring data into structured fault knowledge. The confidence level calculation and dynamic evaluation mechanism are the core of ensuring that the model outputs reliable conclusions in complex industrial environments.

[0190] (1) Confidence calculation in causal reasoning graph model In causal reasoning graphs, confidence calculation aims to quantify the reliability of the reasoning chain from "initial evidence (sensor data)" to "final conclusion (root cause of failure)." Its core logic is typically based on uncertainty reasoning (such as confidence methods) and the fusion of multi-source evidence. 1) Static strength modeling of evidence and rules The reasoning process consists of two elements: "facts (evidence)" and "knowledge (rules)". The system assigns confidence levels to each of these elements (typically in the range of [-1, 1]).

[0191] Uncertainty of evidence: The initial signals collected by the sensor (such as abnormal vibrations or high-frequency ultrasonic leakage) inherently possess a degree of credibility. For example, if the voiceprint feature matching degree is 80%, then the initial confidence level of the evidence is 0.8.

[0192] Static strength of a rule: The degree to which domain experts or AI models trust a causal rule. For example, if the rule "high-frequency ultrasound + pressure drop → pipeline leakage" has a 90% effectiveness rate in historical cases, then the rule confidence level is 0.9.

[0193] 2) Evidence fusion and chain-transfer computation When multiple parameters (such as sound, vibration, and temperature) are cross-validated, the system calculates the overall confidence level through a specific combination function.

[0194] Multi-source evidence synthesis: When multiple independent pieces of evidence point to the same fault, a bounded linear superposition model is used. For example, if evidence A and evidence B simultaneously support a certain fault, the synthesis formula will introduce a normalization term to suppress excessive trust accumulation, reflecting the simulation of cognitive overload and preventing artificially high confidence levels.

[0195] Uncertainty propagation: Rule knowledge is only activated to participate in reasoning when the evidence itself is credible. The confidence level of a conclusion is equal to the product of the "static strength of the rule" and the "credibility of the evidence." If the chain of reasoning is too long or the evidence in a certain link is insufficient, the confidence level of the entire causal chain will be truncated or significantly weakened, thereby enhancing the system's fault tolerance.

[0196] (2) The logic of the dynamic confidence assessment model Industrial environments are complex, and static thresholds can easily lead to false alarms or missed alarms. The core logic of the dynamic confidence assessment model is to automatically adjust the judgment boundaries based on real-time feedback, historical performance, and business risks, thereby achieving tiered response and a safety net.

[0197] 1) Adaptive adjustment based on historical performance and statistical characteristics The model is not static but possesses self-evolution capabilities. The system uses a sliding window to calculate the mean and standard deviation of recent data and dynamically updates the threshold range using a smoothing factor. For example, when the system detects a recent decline in historical accuracy or data drift, it automatically raises the confidence threshold to reduce false alarms; conversely, when accuracy is stable, it appropriately lowers the threshold to improve automation efficiency.

[0198] 2) Multi-dimensional comprehensive calibration and prevention of overconfidence The raw probabilities output by models often suffer from "overconfidence." Dynamic evaluation models integrate multiple dimensions of features, such as intent recognition confidence, slot completeness, and ambiguity, and use calibration techniques like temperature scaling or Pratt scaling to ensure that the predicted probabilities accurately reflect the frequency of events. Furthermore, they distinguish between inherent data noise (random uncertainty) and model knowledge limitations (cognitive uncertainty) to provide a more realistic confidence assessment.

[0199] 3) Triggering safety safeguards and manual intervention In high-risk business scenarios (such as high-pressure gas leaks and critical equipment shutdowns), the dynamic confidence level mechanism acts as a safety net. When sensitive operations are involved or the confidence level remains below the threshold, the system automatically triggers a manual intervention mechanism. This mechanism of "automatic processing for high confidence levels and manual intervention for low confidence levels" ensures diagnostic efficiency while maximizing the safety of industrial systems.

[0200] The tiered early warning system includes: triggering a level 1 warning when the confidence score is below a first preset threshold (e.g., 50%), controlling the quadruped robot to perform secondary data collection on the abnormal area; triggering a level 2 warning when the confidence score is between the first preset threshold and a second preset threshold (90%), transmitting the abnormal data and diagnostic report back to the remote monitoring platform; and triggering a level 3 warning when the confidence score is above the second preset threshold, controlling the quadruped robot to initiate an emergency response, continuously tracking the source of the anomaly and triggering on-site audible and visual alarm devices. The confidence score is calculated based on the clarity of the data from each sensor, the current level of environmental interference, and the consistency of multimodal cross-validation.

[0201] For example, a tiered early warning mechanism: Low confidence (suspected): Triggers the "attention" level, and the robot automatically adjusts its posture or zooms in to perform a second test (such as re-identifying the meter or re-collecting the voiceprint).

[0202] Medium confidence level (confirmed): Triggers the "alarm" level, transmits on-site images and data to the central control room in real time, and generates a preliminary diagnostic report.

[0203] High confidence level (critical): Triggering the "emergency intervention" level, the robot immediately initiates an emergency response, triggers on-site audible and visual alarms, and continuously tracks the source of the anomaly.

[0204] In this embodiment, after feature extraction of the voiceprint signal, the following three-mode fusion early warning logic is also executed: Threshold warning: When the voiceprint feature value exceeds the preset absolute threshold in real time, a threshold alarm is triggered; Trend increase warning: The sliding window algorithm is used to calculate the cumulative change of the current voiceprint feature value relative to the dynamic health baseline. When the cumulative change exceeds the preset tolerance range, the increase alarm is triggered and the trend increase is judged to be abnormal. Trend growth warning: Calculate the rate of change of voiceprint feature value over time. When the rate of change exceeds a preset threshold, trigger a growth alarm and determine that the trend growth is abnormal. The threshold warning, the trend increase warning, and the trend growth rate warning are fused together using logic gates and a weight matrix to form an adaptive warning strategy, which includes: NOT logic: When the voiceprint feature value instantaneously exceeds the preset absolute threshold but quickly falls back within a second preset time and there is no continuous cumulative change, it is determined to be an external transient interference and filtered out, and the threshold alarm is not triggered. AND logic: When the trend increase is determined to be abnormal and the trend growth rate continues to be positive, the warning level is increased; OR logic: When the threshold warning is triggered, a level 3 warning is triggered.

[0205] Voiceprint recognition combines threshold-based early warning, trend increase warning, and trend speed-up warning to achieve high robustness and a low false alarm rate. These three modes are not isolated but correspond to different stages of equipment failure evolution (steady-state anomaly, gradual degradation, and sudden deterioration). The technical logic behind the combination of these three warning modes is explained below (where...). Figures 6A to 6C (The units for the ordinate of the curves are all mm / s). (1) such as Figure 6A Threshold warning: the "absolute bottom line" for steady-state anomalies. Technical logic: Based on historical data from normal equipment operation, static or dynamic upper and lower limits are set for sound signature characteristics (such as sound pressure level in a specific frequency band and total sound level). When the real-time monitored value exceeds this threshold, an early warning is triggered directly.

[0206] Adaptive Mechanism: Traditional thresholds are fixed and easily affected by ambient noise. Adaptive thresholds require dynamic compensation based on the aforementioned "ambient background noise." For example, in a high-noise environment where the fan is running at full speed, the threshold is automatically increased; in a low-noise environment at night, the threshold is automatically decreased.

[0207] Application positioning: As a safety red line, it is mainly used to capture sudden, high-intensity abnormal voiceprints caused by severe physical damage (such as component breakage or severe friction).

[0208] (2) For example Figure 6B Warning of increasing trend: A "relative benchmark" for gradual deterioration. Technical Logic: Focuses on the cumulative change (ΔX) of voiceprint features over a period of time. The algorithm calculates the difference between the current feature value and the set "healthy baseline". Even if the absolute value does not exceed the threshold, as long as the increase exceeds the allowable tolerance range, it is judged as deterioration.

[0209] Adaptive mechanism: Employs a sliding window algorithm or exponentially weighted moving average (EWMA) to dynamically update the health baseline, eliminating baseline drift caused by seasonal temperature differences or normal equipment break-in period.

[0210] Application Positioning: As a degradation indicator, it is mainly used to detect early, slowly developing mechanical faults such as bearing wear, gear pitting, and loose fasteners. At this time, the equipment may still be operating normally, but the "underlying characteristics" of the sound signature have already deteriorated.

[0211] Figure 6B In the text, "high increase" and "very high increase" are alarm status descriptions, corresponding to a significant increase in the peak acceleration and a significant exceedance of the peak acceleration limit, respectively.

[0212] (3) such as Figure 6C Warning of rapid growth trend: An "accelerator" for sudden deterioration. Technical Logic: Focusing on the derivative (dX / dt) of voiceprint feature changes, i.e., the degradation rate. The algorithm calculates the slope of the increase in voiceprint anomalies over time through first-order difference or linear regression fitting. When the rate of increase exceeds a critical value, it indicates that the equipment condition is deteriorating rapidly.

[0213] Adaptive mechanism: Normalization is performed based on the equipment's operating conditions (such as load rate and speed). Normal growth rate under high load and abnormal growth rate under low load need to be treated differently.

[0214] Application Positioning: As a circuit breaker mechanism, it is mainly used to detect early signs of impending catastrophic failures (such as sudden lubricant failure, or rapid deterioration of temperature rise and acoustic vibration due to cooling system blockage). It can buy valuable intervention opportunities before the equipment completely shuts down.

[0215] Figure 6C In the text, "High Growth Rate Report" and "Extremely High Growth Rate Report" describe alarm states, corresponding to a significant increase in the envelope peak value and a substantial exceedance of the envelope peak value, respectively. The aforementioned acceleration peak value and envelope peak value are peak feature parameters extracted from the acoustic signature signal after time-domain analysis, used to characterize the transient impact intensity of the acoustic signature signal in the time domain.

[0216] (4) Adaptive early warning decision engine with three-mode fusion In actual algorithm implementation, these three modes are fused together using logic gates and a weight matrix to form the final adaptive early warning strategy: AND logic (high confidence confirmation): When the "trend increase" exceeds the limit and the "trend growth rate" remains positive (accelerating deterioration), the warning level should be raised even if the absolute threshold is not reached, because this indicates that the equipment is accelerating towards failure.

[0217] OR logic (safety fallback): As soon as the "threshold warning" is triggered, regardless of the trend, the highest level alarm will be triggered immediately because physical damage has already occurred.

[0218] NOT logic (filtering out transient interference): If the voiceprint momentarily exceeds the threshold, but the "trend growth rate" shows that the peak value drops rapidly within milliseconds (in a pulse-like manner) and there is no continuous increase, the algorithm automatically determines it as external transient interference (such as knocking or collision) and filters it out to avoid false alarms.

[0219] Voiceprint recognition, by combining threshold warnings, trend increase warnings, and trend growth warnings, constructs a comprehensive defense network. Threshold warnings safeguard the bottom line and prevent major problems; trend increase warnings provide keen insight and nip problems in the bud; and trend growth warnings anticipate crises and allow for timely intervention. The combination of these three features elevates voiceprint recognition from a simple "alarm" to a self-adaptive "device health stethoscope."

[0220] A complete sound information analysis and diagnosis process should include at least the following modules: sound data acquisition, sound signal preprocessing, sound signal feature extraction, recognition model establishment, sound signal recognition processing, and output of recognition results.

[0221] Common sound characteristic indicators include: Loudness: Loudness is an attribute used to reflect the auditory judgment of the strength of a sound. It is related not only to the magnitude of the sound pressure, but also to factors such as frequency and waveform.

[0222] Sharpness: describes the sensation related to the frequency components of a sound but not its loudness. It reflects the contrast between the high-frequency energy and the total energy of the sound.

[0223] Center of gravity frequency: When the bearing is functioning normally, the center of gravity frequency is relatively low, with its frequency components mainly concentrated in the low frequencies. When the bearing has localized damage, the value increases.

[0224] Spectral centroid: One of the important physical parameters describing timbre attributes, it is the center of gravity of frequency components, and is the frequency weighted by energy within a certain frequency range.

[0225] The quadruped inspection robot is equipped with an acoustic acquisition module to collect radiated noise from industrial equipment. Faults identified include: uniform bearing wear, localized pitting / cracks in bearings, gear tooth root cracks / broken teeth, uniform gear meshing wear, and loose housing bolts or foundations. In this system, acoustic signature indicators provide fault tendency, but lack spatial localization capabilities. Final fault location requires a combination of high-precision visual vibration measurement and spatial deformation information from the robot dog for joint judgment; relative changes compared to a healthy baseline are used, rather than globally fixed thresholds.

[0226] The mapping relationship between each voiceprint indicator and the fault is as follows:

[0227] 1) Loudness Elevated indicators correspond to the following faults: uniform bearing wear, bearing pitting / cracks; gear cracks, broken teeth, tooth surface wear; loose housing bolts or foundation. Mechanism: Wear friction, localized impact, and impact from loose gaps increase the total acoustic energy radiated outwards by the equipment.

[0228] Limitations of judgment: Increased loudness only indicates an acoustic abnormality of the equipment and cannot distinguish the type of fault; increased speed and load can also cause an increase in loudness and cannot be used as a fault criterion alone.

[0229] 2) Sharpness Significantly elevated indicators correspond to faults including: localized spalling and roller / raceway cracks in bearings; root cracks, localized spalling, and broken teeth in gears. Mechanism: Localized damage generates periodic narrow-pulse impacts, exciting a large number of high-frequency noise components, resulting in a substantial increase in the proportion of high-frequency energy.

[0230] A slight increase in the indicators corresponds to faults including: uniform bearing wear and uniform gear meshing wear. The mechanism is progressive wear with a weak impact effect and limited increase in high-frequency components.

[0231] Faults that remain largely unchanged or fluctuate slightly include: loose housing bolts or foundation. Mechanism: loosening is mainly caused by low-frequency gap impacts, with no increase in the proportion of high-frequency energy.

[0232] Judgment value: It is a key indicator to distinguish between housing looseness and local damage to bearings / gears; increased loudness but low sharpness suggests a loose connection as the primary cause of the fault.

[0233] 3) Center of gravity frequency Significantly increased indicators (shift towards higher frequencies) correspond to faults including: localized pitting, spalling, and cracks in bearings; and root cracks, localized spalling, and broken teeth in gears. The mechanism is that localized pulse impacts excite broadband high-frequency components, causing the overall acoustic energy to shift towards higher frequencies.

[0234] A slight increase in the index corresponds to faults including: uniform bearing wear and uniform gear meshing wear. Mechanism: gradual degradation, without strong impact pulses, and limited frequency center of gravity shift.

[0235] Small changes in indicators indicate a fault: loose housing bolts or foundation. Mechanism: the fault energy is concentrated in the low to medium frequency range and will not cause a significant increase in the center of gravity frequency.

[0236] 4) Spectral centroid Significantly elevated indicators correspond to faults including: bearing pitting, raceway / roller cracks; gear tooth root cracks, localized spalling, and tooth breakage. Mechanism: Localized impact introduces a large amount of high-frequency energy, raising the energy-weighted average frequency.

[0237] A slight increase in the indicators corresponds to faults including: uniform bearing wear and uniform gear meshing wear. Mechanism: slow wear, limited high-frequency energy increment, and slight early-stage fault shift.

[0238] The absence of a significant increase in the index corresponds to the following fault: loose housing bolts or foundation. Mechanism: The fault is mainly characterized by low-frequency impact noise, with no increase in the proportion of high-frequency energy.

[0239] The joint fault determination rule for the voiceprint signal recognition device in this system is illustrated in the following logical example: ① Increased loudness + significantly increased sharpness + obvious increase in center of gravity frequency and spectral centroid → tends to local pitting and cracking faults in bearings / gears, and further distinguishes components by combining the deformation field hotspot location with the three-dimensional digital image correlation method. ② Increased loudness + slight increase in sharpness + slight increase in center of gravity frequency and spectral centroid → tends to cause uniform and gradual wear of bearings or gears; ③ Loudness increases, but sharpness, center of gravity frequency, and spectral centroid do not increase significantly → tends to indicate loose housing bolts or assembly gap faults.

[0240] The acoustic signature features of this system lack spatial positioning capabilities and can only determine acoustic signal characteristics. When fused with high-precision measurement of visual vibration modes and deformation field spatial information, fault component location can be completed. The acoustic signature features of very early micro-wear and micro-cracks have weak offsets and their recognition ability is weaker than that of visual vibration measurement. The system uses the equipment's own health baseline for relative offset judgment and does not use a globally fixed threshold. Rotation speed, load, and environmental noise can interfere with acoustic signature indicators, so the algorithm performs working condition normalization processing.

[0241] The diagnostic principle is as follows: By collecting sound data from the equipment over a period of time, we can analyze the historical trends to determine if the equipment is malfunctioning. Alternatively, we can analyze specific metrics at a particular moment, such as RMS value, loudness, and frequency centroid, to determine the specific type of equipment malfunction.

[0242] Unlike vibration data, sound acquisition can capture problems and signal characteristics that conventional frequency band mechanical vibrations cannot detect, such as electrical faults, partial discharge, pipeline leaks, flow channel abnormalities, howling, aerodynamic noise, and ultrasonic bands. It can serve as a good supplement to vibration monitoring, and the integration of the feature layer and the decision layer will provide more comprehensive protection for equipment monitoring and diagnosis.

[0243] S6: Store the equipment health status vector of each inspection into the time series database, establish a time series evolution model of equipment status, fit the equipment deterioration trend curve, predict the remaining service life of the equipment, and send the prediction results back to the remote monitoring platform.

[0244] The "equipment health status vector" from each inspection is stored in a time-series database, and machine learning algorithms are used to fit the equipment degradation curve.

[0245] The algorithm not only focuses on the current state, but also establishes a temporal evolution model of the device state.

[0246] The equipment status time-series evolution model is the core algorithm model for quadruped inspection robots to achieve intelligent diagnosis and predictive maintenance, which differs from the traditional fixed single-point threshold alarm mechanism. The model relies on seven types of multi-dimensional, multi-modal time-series perception data collected by the robot, including high-definition video images, infrared thermal imaging, acoustic signals, visual vibration images, gas concentration data, meter readings, and environmental temperature and humidity data. It abandons the discrete judgment logic of "instantaneous data determining status" and focuses on continuous time evolution laws, multi-modal index correlation and coupling, and equipment physical degradation mechanisms. It accurately quantifies the complete time-series evolution process of equipment from a healthy steady state, early latent degradation, gradual anomaly to failure, achieving continuous quantitative assessment of equipment status, early warning of potential hazards, fault source analysis, and health trend prediction. This effectively solves the industry pain points of traditional single-point monitoring, such as high false alarms, many missed alarms, only post-event alarms, and no predictive capabilities.

[0247] I. Core essence of the model: The vast majority of industrial equipment failures are gradual deterioration rather than sudden, instantaneous failures. Under the combined effects of factors such as changes in environmental temperature and humidity, long-term load operation, mechanical wear, media aging, and seal degradation, the equipment condition strictly follows a time-series degradation pattern of "healthy steady-state operation → early latent deterioration → mild performance abnormalities → moderate failure evolution → severe failure and critical failure".

[0248] The core logic of the equipment status time-series evolution model is: instead of relying on single-frame or single-point instantaneous data to determine the health status of equipment, it is based on long-cycle continuous time-series datasets to deeply capture the continuous offset, cumulative degradation, trend drift, and cross-modal correlation anomalies of various sensing indicators. It quantifies the progressive failure process of equipment such as mechanical fatigue, component wear, thermal aging, sealing leakage, and unstable operating conditions, and realizes the technical upgrade of equipment operation and maintenance from "post-fault alarm" to "pre-problem prediction".

[0249] II. Full Multimodal Input Data for the Model (Adapted to the Perception Capabilities of Quadruped Robots): The model uniformly integrates seven types of native sensory time-series data from the robot, constructing a time-series feature matrix that is spatiotemporally synchronized, dimensionally complementary, and fully covers all scenarios. This matrix comprehensively covers seven monitoring dimensions: equipment mechanical operation, thermal state, acoustic characteristics, visual appearance, instrument condition, safety leakage, and environmental disturbance. High-definition video image temporal features: continuously extract the temporal change features of visual defects such as equipment appearance deformation, bolt loosening, component displacement, shell corrosion, media leakage, foreign object obstruction, and structural damage, and accurately capture the gradual evolution process of equipment appearance failures.

[0250] Infrared thermal imaging time sequence characteristics: Collect the average temperature of key parts of the equipment, regional temperature difference, temperature gradient, short-term temperature rise rate, and thermal field distribution shift time sequence to effectively identify thermal degradation trends such as local overheating, heat dissipation failure, aging of line insulation, and poor contact.

[0251] Acoustic signal timing characteristics: Continuously analyze the timing fluctuations and drift characteristics of abnormal noise energy, acoustic feature similarity, spectral distortion degree, and friction and impact noise during equipment operation, accurately capturing early hidden faults that are difficult to identify by traditional monitoring, such as slight mechanical friction, loose parts, and stuck shafts.

[0252] Temporal characteristics of visual vibration images: Based on visual imaging technology, the temporal characteristics of equipment micro-vibration amplitude, vibration frequency, vibration area distribution, and vibration stability are extracted to identify the dynamic evolution of equipment structural loosening, operational resonance, and component fatigue damage.

[0253] Gas concentration time series characteristics: Collect time series data such as real-time concentration, cumulative concentration, and concentration rise slope of flammable and explosive gases and leaked media gases to continuously monitor the evolution trend of safety hazards such as equipment sealing failure, media leakage, and gas accumulation.

[0254] Meter reading timing characteristics: The system integrates various pointer and digital meter timing readings that are intelligently recognized by the robot, covering process parameters such as pressure, temperature, current, voltage, and flow rate. It captures slow drifts, steady-state deviations, and periodic fluctuations in the equipment's operating process parameters, reflecting the deterioration trend of the equipment's internal operating conditions.

[0255] Temporal characteristics of ambient temperature and humidity: Collect the continuous change sequence of ambient temperature and humidity in the inspection area as a dynamic working condition correction factor to offset the interference of environmental disturbances, seasonal temperature differences and humidity changes on equipment status judgment and ensure the accuracy of diagnostic results.

[0256] All multimodal raw data undergo unified timestamp alignment, environmental interference filtering, steady-state condition screening, and feature standardization and normalization to form a high-quality, time-continuous, and dimensionally unified time series sample set, providing a standardized data base for model evolution analysis.

[0257] III. The core operational logic of the four-layer progressive model: (1) Dynamic time-series baseline modeling to construct personalized health benchmarks for equipment The model uses full-scale, multimodal time-series data from long-term, normal operation of deep learning devices to build a device-specific adaptive dynamic health baseline, replacing the traditional fixed threshold determination method. Simultaneously, it incorporates real-time environmental temperature and humidity, process conditions, and device runtime to dynamically adjust the baseline, adapting to the device's operating status under different environments and loads.

[0258] The dynamic baseline fully covers the normal fluctuation range of each indicator, the steady-state change rate, the normal correlation and coupling relationship of multimodal indicators, and the environmental adaptation benchmark threshold. It can accurately distinguish between normal environmental fluctuations, temporary disturbances in operating conditions, and actual equipment degradation, thereby significantly reducing invalid alarms and false alarms from the source.

[0259] (2) Multimodal temporal evolution quantification to capture early latent degradation features The model uses a sliding time window mechanism to perform comprehensive evolutionary quantification calculations on seven types of multimodal time-series features, accurately capturing early-stage faults where individual data points do not exceed limits but the long-term trend continues to deteriorate, thus overcoming the limitations of traditional threshold monitoring. Trend slope analysis: Quantify the continuous rise and fall rate of equipment temperature, gas concentration, abnormal sound energy, vibration amplitude, and meter process parameters to identify hidden problems such as slow equipment aging, gradual media leakage, and slight component wear. Cumulative Deviation Analysis: This analysis statistically measures the cumulative deviation of various indicators from the healthy baseline over a long period, quantifying the overall deterioration of equipment due to long-term fatigue damage, performance degradation, and operational drift. Time-series volatility analysis: Monitor changes in operational stability based on acoustic signature characteristics, visual vibration, infrared thermal field, and meter readings; accurately identify abnormal trends such as loosening of equipment components, increased friction, and operational instability. Cross-modal coupling anomaly analysis: uncovering the correlation patterns of multiple indicators, such as "vibration anomaly + temperature drift", "acoustic distortion + increased micro-vibration", "meter parameter deviation + local temperature rise", "temperature rise + gas micro-leakage" and other composite anomalies, to identify hidden coupling faults that cannot be detected by a single signal.

[0260] By employing the above multiple strategies, early signs of equipment degradation can be detected several days to several weeks in advance, enabling early warning of potential hazards and reserving sufficient window period for preventive maintenance.

[0261] (3) Timing state migration matching to accurately divide equipment degradation stages The model encompasses the temporal evolution mechanism of equipment, combining the duration of abnormal evolution, degradation magnitude, and degree of cross-index correlation anomalies of multimodal features to automatically iteratively determine the real-time health status of equipment, achieving continuous and gradual state tracking. This differs from the discrete on / off determination of traditional systems, supporting the following progression: Healthy steady state → Early latent degradation (only slight trend shift, no obvious anomalies) → Mild anomaly (single-type index continuously deviates from baseline) → Moderate fault evolution (multimodal index coupling anomalies) → Critical severe fault (indicators significantly exceed limits, equipment condition drastically deteriorates). (4) Degradation source tracing and time-series trend prediction to support predictive operation and maintenance. Based on the long-term multimodal time-series evolution patterns of equipment and the characteristics of deterioration samples, the model completes intelligent analysis, source tracing, and prediction output, enabling fault traceability, trend prediction, and practical operation and maintenance. Quantitative Health Score: Outputs a continuous quantitative health score of 0-100 for the equipment, providing an intuitive and accurate reflection of the overall health level of the equipment; Multi-dimensional degradation source tracing: accurately locate the root cause of abnormalities and distinguish different sources of degradation such as appearance and structural defects, thermal aging, mechanical vibration and wear, acoustic abnormalities, media leakage, and process parameter drift; Trend prediction and analysis: Fit multimodal time-series evolution curves to predict the future health trend of equipment and the window period for failure, and output the optimal maintenance time and operation and maintenance suggestions.

[0262] IV. The core differences between time-series evolution models and traditional threshold monitoring: Traditional threshold monitoring: It only determines whether the limit is exceeded based on instantaneous single-point data, and cannot distinguish between environmental fluctuations, operating condition disturbances and actual equipment degradation. It has a high false alarm and false alarm rate, and can only trigger alarms after the fault has formed, without early prediction capability. Multimodal temporal evolution model: Based on the continuous time dimension, the correlation and coupling of multiple indicators, and the equipment degradation mechanism, it focuses on the equipment degradation process rather than the instantaneous state. It can accurately identify early hidden faults, filter invalid interference, and realize fault source tracing and trend prediction, fully supporting predictive maintenance of equipment.

[0263] V. Core Output Capabilities of the Model: (1) The real-time health level of the equipment and the continuous time-series health evolution curve of the whole cycle can intuitively show the trend of equipment status change; (2) Deterioration trend, evolution slope and cumulative offset data of high-definition images, infrared thermal imaging, acoustic prints, visual vibration, gas, meter readings, temperature and humidity multimodal indicators; (3) Early warning of hidden equipment deterioration, dynamic prompts of deterioration process and risk level classification; (4) Tracing the source of multimodal data coupling anomalies and reviewing the complete evolution path of equipment failures; (5) Equipment health trend prediction, fault window prediction and intelligent operation and maintenance suggestions.

[0264] The equipment status time-series evolution model relies on the quadruped inspection robot's multi-modal perception capabilities across all scenarios, multiple dimensions, time synchronization, and high precision. With time evolution as the main line, equipment physical degradation mechanisms as support, and cross-modal correlation analysis as the core, it breaks through the technical limitations of traditional single-point monitoring, continuously quantifies the gradual evolution process of equipment health status, and fully realizes early hidden danger warning, intelligent error reduction and leakage prevention, accurate fault tracing, and status trend prediction. It is the core technical support for upgrading robot intelligent inspection from "simple data collection and display" to "intelligent equipment diagnosis and predictive operation and maintenance".

[0265] This model is built and applied based on data from previous inspections, and specifically includes the following four stages: (1) Feature vector construction: After each inspection of the quadruped robot, the vibration modal parameters, infrared temperature distribution index, acoustic signature, gas / temperature and humidity data, and causal graph fault confidence are integrated, normalized and spliced ​​to generate a unique multi-dimensional equipment health status vector that represents the current equipment status. (2) Time-series data storage: The equipment number, inspection timestamp, complete health vector, and on-site working condition label are written into the distributed time-series database. A hierarchical time-series dataset is established according to equipment and measurement point to retain the full-cycle inspection data for a long time. (3) Deterioration model training and fitting: Based on the historical health vector dataset in the time series database, the gradient boosting regression tree or long short-term memory LSTM time series machine learning model is used for offline training. The actual equipment maintenance records and fault downtime records are used as labels to fit the deterioration evolution curve of the equipment health index as the running time changes. (4) Model application: After the robot generates a health vector in real time during on-site inspection, it is input into the trained degradation prediction model, compared with the historical degradation curve to calculate the remaining service life (RUL) of the parts, and outputs graded early warning to support predictive maintenance and one-machine-one-policy equipment management in the factory.

[0266] For example, by combining historical meter reading trends, long-term infrared temperature rise slopes, and acoustic degradation trajectories, the remaining useful life (RUL) of the equipment can be predicted, and "predictive maintenance" recommendations can be issued days or weeks before a failure occurs, truly achieving prevention before problems arise.

[0267] The algorithm logic is constrained by the motion characteristics of quadruped robots, based on "spatial alignment", and takes the causal cross-validation of "sound-vibration-temperature-gas-vision" as its core innovation. It ultimately outputs a high-confidence hierarchical early warning and predictive maintenance strategy, which perfectly meets the requirements of modern industry for "precision, reliability and foresight" of embodied intelligent inspection.

[0268] To facilitate understanding of the above embodiments, the following will use a specific application scenario of the above embodiments (intelligent inspection scenario of quadruped robots in machine repair workshops) as an example for illustration: Hardware Deployment: The quadruped robot is equipped with a 2K high-definition camera, a 640×512 infrared thermal imager, a 4-channel acoustic sensor, a high-speed visual vibration camera, and a combustible gas / temperature and humidity composite sensor. The body integrates a 6-axis IMU, the leg joints are equipped with 23-bit high-precision encoders, and a LiDAR is used to achieve spatial positioning; each sensor uses 1ms synchronous triggering for data acquisition. On-site data acquisition and preprocessing: The quadruped robot autonomously walks along the production line to inspect the gearboxes. The IMU and encoder output posture motion data in real time. The walking jitter is eliminated by EIS motion compensation. Multimodal data is spatially aligned based on the point cloud coordinate system. The vibration modes of the gearbox housing and bearing housing are extracted using the phase optical flow method and the three-dimensional digital image correlation method. Fault reasoning verification: Construct a fault causal graph for gearboxes, including three typical faults: bearing pitting, gear tooth breakage, and lubricating oil leakage, and bind four characteristic thresholds: vibration, temperature, abnormal noise, and oil and gas. During a certain inspection, only the infrared temperature rose slightly, and there were no abnormalities in vibration or sound. The graph cross-referenced the data and determined that it was due to environmental hot air interference, thus eliminating false alarms. When the vibration resonance peak increased, the temperature rise in the bearing area exceeded the standard, and the abnormal noise spectrum appeared simultaneously, the graph matched the bearing wear causal link and output a level-two fault warning. Application of the degradation model: The gearbox is inspected daily for 60 consecutive days, and 60 sets of health vectors are stored in the time series database. The degradation curve of the gear bearing is trained based on LSTM. After the health vector of the 58th inspection is input into the model, the remaining running time of the bearing is predicted to be 18 days, triggering a planned maintenance work order to avoid unplanned downtime.

[0269] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments. like Figure 7 As shown, another embodiment of the present invention provides a multimodal perception anomaly early warning system 100 for quadruped robots, comprising: The data acquisition module 110 is used to synchronously acquire multimodal perception data through multiple sensors mounted on the quadruped robot. The multimodal perception data includes at least: high-definition video images, infrared thermal imaging, acoustic signals, visual vibration images, gas concentration data, and temperature and humidity data.

[0270] Spatial alignment module 120 is used to spatially align multimodal sensing data; Feature extraction module 130 is used to extract features from the spatially aligned multimodal sensing data and convert them into multidimensional feature vectors. Causal reasoning module 140 is used to input feature vectors of various dimensions into causal reasoning model, perform multi-source data cross-validation, and output causal reasoning results and equipment health status vector. The causal reasoning model at least supports joint diagnosis of sound-vibration-temperature and / or gas-temperature-visual linkage for leak tracing. The graded early warning module 150 is used to calculate a confidence score based on the causal inference result, perform graded early warnings based on the confidence score, and transmit the early warning signal back to the remote monitoring platform; and... The timing prediction module 160 is used to store the equipment health status vector of each inspection into the timing database, establish a timing evolution model of equipment status, fit the equipment deterioration trend curve, predict the remaining service life of the equipment, and send the prediction results back to the remote monitoring platform.

[0271] As a specific implementation of the quadruped robot multimodal perception anomaly early warning system 100 of the present invention, the above-mentioned functional modules can be physically divided into an execution layer, an edge computing layer, a communication layer, and a management layer, such as... Figure 8 As shown.

[0272] (1) The execution layer consists of a quadruped robot and its upper-mounted sensor equipment. The quadruped robot is equipped with a visible light gimbal, a high-definition camera, a thermal imaging camera, an acoustic sensor, a gas sensor, a temperature and humidity sensor, a 5G module / wireless LAN module, and other upper-mounted equipment, which can inspect the appearance of equipment, tunnel structure, power transmission cables, pipe corridors and water pipes, etc. At the same time, the robot performs preliminary identification and analysis on the collected data and transmits the data to the inspection backend, providing maintenance personnel with decision-making basis and real-time processing solutions.

[0273] (2) Edge computing layer, which consists of the intelligent control module mounted on the quadruped robot and the robot inspection platform running inside. It can issue inspection tasks according to plan and control the operation of sensors. During inspection, the robot can complete the task autonomously and store the data locally. After returning to the vicinity of the charging pile and the network is restored, it can upload the data, ensuring the continuity of equipment monitoring and the stability of robot inspection.

[0274] (3) Communication layer: The quadruped robot and the remote monitoring platform use a dual-mode communication method combining 5G public network and wireless local area network. The wireless local area network is used for batch data transmission after inspection, and the 5G public network is used for remote control takeover in abnormal situations. Specifically, considering the characteristics of long tunnel length and severe signal attenuation, the quadruped robot can use a communication method combining 5G public network and wireless local area network. Wireless local area network base stations are set up near the charging piles in the tunnel. After each inspection, the video stream data collected by the gimbal is transmitted back to the data center for analysis using wireless local area network. During the inspection, the existing 5G public network is used as a backup network to ensure that the robot can be taken over and remotely controlled to the abnormal location for secondary verification as soon as an abnormality occurs on site. All inspection data collected by the robot are integrated and transmitted to the remote monitoring platform.

[0275] (4) Management level: After the quadruped robot's edge data is collected and integrated, it is transmitted to the data center server via a wireless local area network. The server processes and analyzes the data, issues alarms for abnormal data, and centrally displays the inspection data. The software server has a standard interface that can interact with third-party platforms to synchronize inspection content, making it convenient for maintenance personnel to view.

[0276] The quadruped robot in this embodiment operates within a temperature range of -20°C to 55°C for the first time, demonstrating strong adaptability in extreme environments of extreme cold and heat. This quadruped robot possesses excellent obstacle-crossing capabilities, enabling it to quickly and stably traverse obstacles in complex and changing environments, and accurately navigate 45° openwork industrial staircases (considering the shift in center of gravity caused by the upper components, it is recommended that the staircase angle be kept within 35° on-site). Even in extreme visual conditions such as dim lighting, bright light, flickering light, or complete darkness, the inspection robot can still autonomously navigate and complete its designated tasks.

[0277] In terms of interaction, the quadruped robot is equipped with advanced light-symbol interaction, conveying equipment status information to operators through light signals, enhancing the intuitiveness and convenience of operation. Simultaneously, the quadruped robot possesses a safe obstacle-stopping function for random dynamic objects, effectively ensuring the safety of the operation process. Regarding battery life, the quadruped robot performs excellently, with a continuous battery life of 2-3 hours (depending on the load). To meet the needs of long-term operation, the battery supports rapid on-site replacement, greatly improving work efficiency. Furthermore, the quadruped robot has autonomous charging capabilities, achieving precise autonomous charging through integrated positioning technology, providing strong support for continuous operation in various industry scenarios. In terms of protection, the quadruped robot reaches the stringent IP67 protection rating, ensuring stable operation even in harsh environments. Additionally, the quadruped robot is equipped with a real-time monitoring system and an emergency response system, capable of promptly identifying and handling potential problems, ensuring the safety and efficiency of intelligent operations.

[0278] The quadruped robot comes standard with a high-real-time operating system and a high-performance multi-core processor, ensuring sufficient memory and computing power to load motion control programs, navigation algorithms, intelligent sensing module programs, and image acquisition. It supports multi-threaded development, providing an internal parameter of a 1ms motion control cycle and a freely configurable planning cycle (25ms-50ms), facilitating the implementation of algorithms such as MPC and path planning to meet the real-time management and rapid computing requirements of project implementation. The quadruped robot also has the capability to connect to an IoT platform, accepting the relevant management requirements of the IoT platform and adhering to the development requirements of the IoT platform's device information model.

[0279] The quadruped robot is also equipped with a millimeter-wave phased array radar. The millimeter-wave phased array radar adopts a frequency-modulated continuous wave system and integrates a phased array antenna array. It achieves beam spatial scanning by electronically controlling the phase of each radiating unit. The millimeter-wave phased array radar and lidar form a redundant sensing combination. When the lidar fails in dusty or foggy environments, the detection data of the millimeter-wave phased array radar is switched as the main data source for navigation, positioning, and map building. It is combined with a fusion SLAM algorithm to achieve localization and dynamic path planning. The millimeter-wave phased array radar is also used to cover the robot's blind spot and detect obstacles in the blind spot to trigger emergency stop or detour decisions.

[0280] Specifically, millimeter-wave phased array radar is used as the basic navigation system. Advantages of millimeter-wave phased array radar include: strong penetration in rain, fog, dust, and dark environments; and strong speed measurement and long-range multi-target tracking capabilities.

[0281] Phased array millimeter-wave radars are installed as redundant sensing in complex industrial environments such as densely and irregularly distributed steel mill units, rail transit tunnels, coal mines with high dust levels, underground pipe corridors, and explosion-proof environments in complete darkness.

[0282] The core function of millimeter-wave phased array radar is to compensate for the shortcomings of lidar. In environments such as tunnels, dense fog, and heavy dust, lidar is scattered and becomes ineffective, while phased array millimeter-wave radar is not subject to obstruction or interference, enabling real-time detection of distant personnel, equipment, and walls. It also performs multi-sensor fusion with lidar to eliminate perception blind spots and prevent robots from crashing into walls or falling. Millimeter-wave phased array radar technology utilizes the strong penetration and anti-interference capabilities of millimeter waves, combined with the high-precision scanning capability of phased array antennas, to construct an all-weather, highly robust sensing and positioning closed-loop system. The specific implementation logic can be divided into the following four core components:

[0283] (1) Hardware architecture and signal transmission The system employs a frequency-modulated continuous wave (FMCW) radar operating in the millimeter-wave band (such as the K-band) and integrates a phased array antenna. By electronically controlling the phase of each radiating element in the phased array antenna, the radar beam can achieve high-speed and precise spatial scanning and beam pointing without mechanical rotation, thereby efficiently acquiring three-dimensional spatial information of the surrounding environment.

[0284] (2) Multi-source data fusion sensing in complex environments Single sensors are prone to failure in extreme environments, so millimeter-wave radar typically does not operate independently but is fused with multi-source data from lidar, vision cameras, and IMUs (inertial measurement units). Millimeter-wave radar, with its strong penetration and anti-interference capabilities in harsh environments such as dust, rain, fog, and strong light, compensates for the shortcomings of optical and laser sensors, providing robots with stable and continuous environmental characteristic data.

[0285] (3) High-precision positioning and dynamic path planning At the navigation and positioning level, the system uses millimeter-wave radar rotation scanning or phased array to detect fixed beacons (such as reflective road signs) on the road surface, and iteratively updates the robot's real-time pose based on the beacon coordinate differences. At the same time, combined with the self-developed fusion SLAM (simultaneous localization and mapping) algorithm, the robot can achieve centimeter-level high-precision autonomous navigation and dynamic path planning in trackless and complex terrains (such as stairs, slopes, and narrow passages), effectively avoiding obstacles.

[0286] (4) Blind spot detection and safety obstacle avoidance closed loop During motion control, phased array radar is specifically used to cover the robot's own blind spots (such as low obstacles or blind spots to the sides and rear during rotation). By detecting potential collision risks in the blind spots in real time, the system can trigger millisecond-level emergency stop or detour decisions, thereby ensuring the robot's safe operation in complex and dynamic environments.

[0287] In summary, the implementation logic of this navigation scheme is as follows: using millimeter-wave phased array radar as the core of all-weather perception, eliminating blind spots of single perception through multi-sensor fusion algorithms, achieving high-precision positioning by combining beacon iteration and SLAM technology, and finally building a closed loop of "perception-decision-execution" to ensure stable inspection of the robot under various extreme working conditions.

[0288] In one optional implementation, the quadruped robot is equipped with a multi-floor navigation system, achieving precise positioning through LiDAR and SLAM (Simultaneous Localization and Mapping) technology. It is simple to deploy, easy to use, and highly mobile. It supports autonomous stair climbing and can build a 3D environment map in real time. Equipped with an adaptive stair gait algorithm, it can autonomously identify steps and plan foot landing points. Combined with a dynamic obstacle avoidance system, it can autonomously switch tasks in multi-level scenarios, achieving intelligent automatic inspection. Based on the requirements for automatic inspection and obstacle avoidance, it is recommended to reserve a 1-meter passage width on flat ground for the robot to operate. If the robot needs to climb stairs, considering the change in the center of gravity caused by the upper equipment, it is recommended to reserve a 120-centimeter passage width on the stairs, ensuring that the incline of the stairs is within 35 degrees and the height of each step is within 20 centimeters.

[0289] Furthermore, in this embodiment, the quadruped robot also possesses omnidirectional perception capabilities. For example... Figure 9 The diagram shown illustrates the full-body sensor coverage of the quadruped robot 111. Different colored sector areas mark the effective detection angle ranges of each sensor module around the robot. Specific information is as follows: The detailed detection angles in each direction are as follows: Upper right front region of the robot, -45°~12° (covering approximately 57° of the right front); Front region of the robot, -39°~-98° (covering approximately 59° of the front); Front left region of the robot, -125°~178° (covering a relatively large area in front of the left); Region associated with the left front leg of the robot, -6°~127° (for close-range local sensing of the left lower limb); Rear region of the robot, 49°~-108° (large span, covering the rear); Upper right rear region of the robot, -2°~57° (covering approximately 59° of the right rear); Rear left region of the robot, -168°~133° (covering the left rear region); Region associated with the right rear leg of the robot, -117°~176° (for local sensing of the right rear lower limb). Note: Negative values ​​in the data (such as -45°, -98°, etc.) typically represent the downward tilt angle of the sensor in the vertical dimension. Ground blind spot filling and fall prevention: A large downward pitch angle (such as -98°, -108°) can ensure that the robot can clearly see the ground, curb, speed bump or step right below the body, which is crucial for the robot's safe navigation and fall prevention.

[0290] These multi-angle detection zones form a near 360° all-domain perception coverage, enabling the robot to capture environmental information such as obstacles and terrain undulations in real time during movement. This supports the robot in completing actions such as autonomous obstacle avoidance, gait adjustment, and adaptation to complex terrain, ensuring the robot's walking stability and environmental adaptability in unstructured scenarios.

[0291] In one alternative implementation, auxiliary lidar navigation and positioning technology is also included. For example... Figure 10 As shown, installing auxiliary road markers 201 on both sides of tunnel 200 provides a positioning reference for the lidar. This method obtains the displacement information of the robot by judging the displacement of the reflector relative to the radar, accumulates the displacement as an odometer, and calculates the mileage position of the identified road markers. The relative coordinates between the auxiliary road markers 201 and the quadruped robot 111 can be further used to calculate the current mileage position and lateral position of the quadruped robot 111. Combining the lateral and longitudinal position information at the same time, as the quadruped robot 111 moves, it continues to identify the auxiliary road markers 201 on the left and right sides, while performing error compensation to obtain accurate position information.

[0292] In one alternative implementation, the quadruped robot features autonomous charging capabilities. Factors such as light, dirt, and dust in industrial environments can affect the efficiency of traditional charging solutions. The quadruped robot, equipped with autonomous charging equipment, utilizes a fusion positioning system to ensure a smoother and more efficient charging process. Employing laser reflection technology, it supports one-click recharging and can achieve autonomous charging without the need for mapping.

[0293] In one optional implementation, the quadruped robot communication system employs advanced wireless data network transmission technology to ensure seamless wireless network coverage within the inspection area. This system also integrates a wireless encrypted communication network to guarantee data transmission security. On the robot's end, the industrial control computer and video devices connect to the wireless hub via Ethernet interfaces. By using an onboard wireless communication terminal, they can access the local area network for real-time data transmission and processing. If comprehensive wireless network coverage is not yet deployed in the factory area, a solution integrating a wireless access point (AP) base station can be designed. Depending on the actual conditions of the inspection site, a flexible self-organizing network can be implemented, optimizing network signal coverage by selecting omnidirectional or directional antennas while reducing deployment costs. This solution not only facilitates direct network access for the quadruped robot but also ensures efficient execution of inspection tasks.

[0294] For detailed deployment instructions, please refer to [link / reference]. Figure 11The diagram shows the architecture of an IoT system. This IoT system 500 includes a top-level cloud and remote management layer 510, a middle-level field network hub layer 520, and a bottom-level project site equipment layer 530. The cloud and remote management layer 510 includes a cloud server 511, a monitoring computer 512, and an internet connection 513; the field network hub layer 520 includes a station-side monitoring computer 521 and a gateway 522; and the project site equipment layer 530 includes a wireless communication network 531, a weather station 532, an automatic door 533, an outdoor charging room 534, and a quadruped robot 111.

[0295] In another embodiment of the invention, the system also includes an on-site installation and modification scheme to ensure that the quadruped robot can move unimpeded along the inspection path.

[0296] (1) Fire door modification: For fire doors that are manually operated and do not have a drive motor and require manual pushing and pulling, door closers and access control controllers need to be installed to enable automatic door opening during robot inspection. The on-site doors can be modified to Class A fire doors.

[0297] (2) Rodent barrier modification: To address the obstacle to robot passage caused by 30-50cm rodent barriers on-site, we support the automation modification of the rodent barriers, replacing them with a swing gate-type automatic rodent barrier 300, such as... Figure 12 As shown, the system includes a support column 301, a baffle 302, and a rotating shaft 303. The support column 301 is fixed to the ground 400. The baffle 302 is mounted on the support column 301 and connected to the rotating shaft 303. A brushless motor drives the rotating shaft 303 to rotate, causing the baffle 302 to open or close. This design ensures the safety of the on-site production equipment while also allowing for robot mobility.

[0298] In another embodiment of the present invention, the quadruped robot has self-inspection and safety protection functions: The system performs fault self-diagnosis and alarm analysis through a self-test module interface. This is achieved by displaying the current status of each module on the quadruped robot, including the common modules, battery module, network connection module, and safety module, to check if they are functioning correctly. When any of these modules malfunctions, the warning light on the back of the quadruped robot stays on orange, allowing operators at the station to react promptly. Simultaneously, the robot uploads fault information to the local monitoring backend. At this time, the status bar background of the malfunctioning module on the self-test module interface turns red, and the local monitoring backend issues an alarm. Operators in the control room can then promptly check and address the malfunctioning module upon hearing the alarm.

[0299] The quadruped robot uses a laser obstacle avoidance system to detect obstacles and personnel within the scene. It stops and sounds an alarm when it encounters an obstacle. Simultaneously, structural safety features are added to prevent collisions that could cause injury to personnel or equipment. This, along with the obstacle avoidance system, creates a double safety barrier. In the event of obstacle avoidance system failure, it prevents violent collisions with obstacles and triggers a system safety alarm. Flashing warning lights are also installed in prominent locations to alert personnel within the area.

[0300] During autonomous inspection, when the laser obstacle-stopping system detects an obstacle ahead, the quadruped robot will issue an alarm and stop in an emergency, notifying the main control room to remove the obstacle. During this process, the main control room can issue instructions to return, standby, or other inspection tasks that do not affect the system. In standby mode, the system enters a low-power state to save power.

[0301] Meanwhile, due to the special environment in different production scenarios, the quadruped robot body is also designed with extremely strong resistance to electric field interference, and its feasibility has been successfully verified in the ultra-high current rectifier chamber of electrolytic aluminum.

[0302] In this embodiment, the remote monitoring platform adopts a B / S architecture design, specifically configurable as an intelligent management platform. It mainly consists of a real-time monitoring module, a data statistics module, a reporting system, a map editing module, a task management module, a system settings module, and a system integration module. It possesses multiple functions including data acquisition, storage, statistics, retrieval, report output, and intelligent analysis, used for task management, data querying, and equipment monitoring of the quadruped robot. This intelligent management platform features a user-friendly interface, convenient operation, clear and intuitive information display, and a range of powerful functions.

[0303] (1) Video monitoring function: During the inspection, the quadruped inspection robot can collect video images of the surrounding environment through its own integrated gimbal device, and transmit the image information back to the main control room via wireless network according to the needs of the staff, and automatically record the shooting location and equipment name so that the staff can query complete information in the future.

[0304] The video monitoring system provides a clear and intuitive real-time view of the equipment being inspected by the robot, displaying the current horizontal and vertical positions of the pan-tilt unit. It allows for single-image full-screen zooming and double-clicking to close the full-screen view. Inspection points include meters, status indicators, appearance and auxiliary facilities, and the equipment's operating environment.

[0305] (2) Infrared temperature measurement function: The quadruped inspection robot is equipped with an infrared thermal imager with automatic focusing function. It can detect the temperature of equipment in the production scene and display the area with the highest temperature and temperature value in the image in real time. It can automatically analyze and judge the overheating fault of the equipment as required and issue an early warning.

[0306] (3) Alarm function for detection results: Based on the different optional monitoring equipment, the quadruped inspection robot can send alarm information to the management personnel according to the corresponding detection results such as meter readings, infrared temperature, gas detection, dust detection, temperature and humidity detection, noise detection and / or personnel behavior detection, according to the preset alarm conditions or thresholds. The alarm information includes alarm time, identification type, alarm level, inspection value, etc. The robot can also refer to data information, historical trends, historical defects, historical images, etc. to review the equipment alarms and determine the defect type. This enhances the management personnel's ability to control the overall production scene.

[0307] (4) Autonomous Inspection and Task Management Functions: The quadruped inspection robot can perform various inspection modes, including timed inspection, fixed-point inspection, temporary inspection for designated tasks, and remote inspection. The commonly used fixed-point inspection can automatically and uniformly inspect according to the specified path and the specified inspection target point. It is only necessary to set the inspection path and start the automatic inspection so that the robot can automatically complete one inspection. During the inspection process, the robot can automatically and accurately stop and detect each inspection work position. After completing the prescribed actions, it will automatically move to the next inspection target point according to the path. No manual operation or control is required to complete the inspection work. The robot also automatically records and saves the collected data.

[0308] Scheduled inspection: After the set time is reached, the robot can automatically patrol along the predetermined trajectory, stop precisely at the pre-set location, and automatically inspect the equipment to be inspected. After all inspections are completed, it automatically returns to the charging room for autonomous charging.

[0309] Designated inspection point: Operators can set any inspection point within the area. The quadruped inspection robot can plan the optimal path based on the current location and the target point and autonomously run to the target point.

[0310] Remote inspection: Operators can remotely control the robot via mouse and keyboard to move it off the pre-set route to a designated location (within the robot's reach) in the quadruped robot's working environment. By remotely controlling the robot's movement and gimbal actions, camera focusing, operation of the infrared thermal imager, and acquisition of visible light and infrared data, special inspections of the equipment to be inspected can be carried out.

[0311] Quadruped inspection robots can autonomously inspect production equipment. Through data analysis, they can provide operators with information on equipment operating status. By analyzing historical data, they can improve the predictability of equipment and provide effective data support for condition-based maintenance or condition assessment, thereby reducing the occurrence of accidents. Using quadruped inspection robots to inspect production equipment allows operators to stay away from live equipment, ensuring high safety and reliability. At the same time, it provides an innovative technical inspection method for the promotion and application of unattended operation, improving the reliability and stability of the production system.

[0312] (5) Task Report Browsing: Displays data, images, and other information collected from inspection points related to the task, categorized by task. Shows information on completed tasks within a specific time period, including: task name, start time, end time, total time elapsed, total number of inspected devices, total number of suspected abnormal devices, and review status. Selecting certain tasks allows for review of the data within those tasks and export of related inspection task reports.

[0313] (6) Equipment Data Browsing: The equipment data browsing module mainly displays data, images, and other information collected at inspection points from the perspective of the equipment. Equipment data browsing is primarily in tabular form, sequentially querying the collected information for each point included in the current inspection task according to the order of the inspection points. This information is then verified and confirmed. Once all points included in the current inspection task have been confirmed, the reviewer and review time can be entered, at which point the conditions for generating a task report are met.

[0314] (7) Equipment alarm: The equipment alarm displays alarms for each device, including alarm time, identification type, alarm level, inspection value, etc., and can refer to data information, historical trends, historical defects, historical pictures, etc. to review the equipment alarms and determine the defect type.

[0315] (8) "Tag" obstacle avoidance function: When the maintenance area is set, the maintenance equipment and the road occupied are set when the equipment is being maintained in the production scene. When the task is compiled, the relevant inspection points and roads are automatically closed, including all inspection points of the maintenance equipment itself and all inspection points located in the maintenance area to inspect other equipment.

[0316] (9) Inspection monitoring: Inspection monitoring mainly realizes the real-time monitoring function of robot inspection operation, including inspection task information, inspection map, real-time video, inspection results, inspection alarm, robot control, etc.

[0317] (10) Task Creation: The task creation module can create comprehensive inspections, routine inspections, special inspections, special inspections, and custom tasks. Each sub-module integrates functions such as task creation and task distribution. Based on different inspection task types, relevant inspection points are automatically preset, and task names are automatically generated.

[0318] (11) Task Display: The task display shows all issued inspection tasks in a monthly calendar format. Each day's cell on the calendar displays the number of tasks issued that day and their completion status. After selecting a date, you can see all the assigned tasks and their status in detail in the list on the right.

[0319] The quadruped robot possesses edge computing capabilities, enabling it to analyze data using its onboard AI algorithms and models. If the analysis results exceed pre-set alarm thresholds, an alarm is triggered. The quadruped robot has its own database and data synchronization program. While storing various inspection data in its own database, it simultaneously uses its data synchronization program to store inspection and alarm data on a remote monitoring platform in the data center. During inspections, the quadruped robot can issue relevant data acquisition commands to these sensors, obtaining the data collected and performing comprehensive data analysis. The quadruped robot also features automatic charging; when its battery is low during inspections, it automatically returns to the charging station to recharge.

[0320] Among them, the intelligent management platform software of the quadruped robot integrates a variety of advanced technologies such as big data, machine learning, and artificial intelligence. It can perceive multi-dimensional data such as environment, equipment, sound, and temperature in production and inspection scenarios in different industries in real time. Through big data technology, it can deeply mine data information, make predictive calculations and analyses, and display comprehensive data of the inspection process in real time through various data presentation methods. This provides a scientific basis for accurate decision-making, thereby freeing maintenance personnel from tedious work, improving overall maintenance efficiency, reducing maintenance costs, and safeguarding the security and reliability of business systems.

[0321] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms fall within the scope of protection of the present invention.

Claims

1. A multimodal perception anomaly early warning method for a quadruped robot, characterized in that, Includes the following steps: The quadruped robot synchronously collects multimodal perception data using multiple sensors, including at least: high-definition video images, infrared thermal imaging, acoustic signature signals, visual vibration images, gas concentration data, meter readings, and temperature and humidity data. Spatial alignment is performed on the multimodal sensing data; Feature extraction is performed on the spatially aligned multimodal sensing data, and the data is transformed into multidimensional feature vectors. The feature vectors of each dimension are input into the causal inference model, and multi-source data cross-validation is performed to output the causal inference results and equipment health status vector. The causal inference model at least supports joint diagnosis of sound-vibration-temperature and / or gas-temperature-visual linkage for leak tracing. Calculate a confidence score based on the causal reasoning results, issue graded early warnings based on the confidence scores, and transmit the early warning signals back to the remote monitoring platform. The health status vector of the equipment during each inspection is stored in a time-series database, a time-series evolution model of the equipment status is established, the equipment deterioration trend curve is fitted, the remaining service life of the equipment is predicted, and the prediction results are sent back to the remote monitoring platform.

2. The multimodal perception anomaly early warning method for quadruped robots according to claim 1, characterized in that, The spatial alignment includes: The robot's posture data is collected using an inertial measurement unit installed on the quadruped robot, and motion data of the encoders of each joint of the robot is obtained. Based on the posture data and the motion data, motion compensation is performed on the high-definition video image, the infrared thermal image, and the visual vibration image.

3. The multimodal perception anomaly early warning method for quadruped robots according to claim 1, characterized in that, The combined sound-vibration-temperature diagnosis is as follows: when both visual vibration characteristics and acoustic characteristics are abnormal, infrared thermal imaging characteristics are invoked in conjunction. If the temperature of the corresponding part rises abnormally at the same time, it is confirmed as a mechanical wear fault; if the temperature is normal, it is determined to be an external interference. The gas-temperature-visual linkage leak tracing method is as follows: when the gas concentration characteristics are abnormal, the infrared thermal imaging and the high-definition video images at a first preset time before and after the trigger time are automatically traced back. The abnormal temperature difference is identified by the infrared thermal imaging and the leak source status is identified by combining the high-definition video images to locate the leak source.

4. The multimodal perception anomaly early warning method for quadruped robots according to claim 1, characterized in that, The causal reasoning model supports environmental adaptive compensation: using temperature and humidity characteristics as an environmental benchmark, it dynamically corrects the alarm threshold of the infrared thermal imaging and the alarm threshold of the gas concentration data.

5. The multimodal perception anomaly early warning method for quadruped robots according to claim 1, characterized in that, The tiered early warning system includes: When the confidence score is lower than the first preset threshold, a first-level warning is triggered, and the quadruped robot is controlled to perform secondary data collection on the abnormal parts. When the confidence score is between the first preset threshold and the second preset threshold, a level two warning is triggered, and the abnormal data and diagnostic report are sent back to the remote monitoring platform. When the confidence score is higher than the second preset threshold, a level 3 warning is triggered, and the quadruped robot is controlled to start an emergency response, continuously track the source of the anomaly and link the on-site sound and light alarm device. The confidence score is calculated based on the clarity of each sensor data, the degree of interference in the current environment, and the consistency of multimodal cross-validation.

6. The multimodal perception anomaly early warning method for quadruped robots according to claim 5, characterized in that, It also includes performing the following three-mode fusion early warning logic after feature extraction of the voiceprint signal: Threshold warning: When the voiceprint feature value exceeds the preset absolute threshold in real time, a threshold alarm is triggered; Trend increase warning: The sliding window algorithm is used to calculate the cumulative change of the current voiceprint feature value relative to the dynamic health baseline. When the cumulative change exceeds the preset tolerance range, the increase alarm is triggered and the trend increase is judged to be abnormal. Trend growth warning: Calculate the rate of change of voiceprint feature value over time. When the rate of change exceeds a preset threshold, trigger a growth alarm and determine that the trend growth is abnormal. The threshold warning, the trend increase warning, and the trend growth rate warning are fused together using logic gates and a weight matrix to form an adaptive warning strategy, which includes: NOT logic: When the voiceprint feature value instantaneously exceeds the preset absolute threshold but quickly falls back within a second preset time and there is no continuous cumulative change, it is determined to be an external transient interference and filtered out, and the threshold alarm is not triggered. AND logic: When the trend increase is determined to be abnormal and the trend growth rate continues to be positive, the warning level is increased; OR logic: When the threshold warning is triggered, the level 3 warning is triggered.

7. The multimodal perception anomaly early warning method for quadruped robots according to claim 1, characterized in that, The feature extraction includes meter recognition of the high-definition video image, comprising: The high-definition video image is preprocessed, including at least strong light reflection suppression, low light brightness enhancement, and perspective distortion correction; The target detection network locates the region of interest (ROI) on the dial of an image, distinguishing between digital and analog meters. For the digital meter, a character recognition network is used to segment and recognize the numbers; For the pointer-type meter, the coordinates of the dial center, zero mark, full scale and pointer tip are located by key point detection, the angle between the pointer and the zero mark is calculated and the reading is converted according to the range ratio; The system outputs a recognition confidence score. If the confidence score is lower than a preset threshold, the recognition is deemed to have failed. The system then controls the quadruped robot to automatically retake the image and performs a time-series verification on the recognition result. Finally, the system uploads the recognition data and the original image to the remote monitoring platform.

8. A multimodal perception anomaly early warning system for a quadruped robot, characterized in that, include: The data acquisition module is used to synchronously acquire multimodal perception data through multiple sensors mounted on the quadruped robot. The multimodal perception data includes at least: high-definition video images, infrared thermal imaging, acoustic signals, visual vibration images, gas concentration data, meter readings, and temperature and humidity data. A spatial alignment module is used to spatially align the multimodal sensing data; The feature extraction module is used to extract features from the spatially aligned multimodal sensing data and convert them into multidimensional feature vectors. The causal reasoning module is used to input feature vectors of various dimensions into the causal reasoning model, perform cross-validation of multi-source data, and output causal reasoning results and equipment health status vectors. The causal reasoning model at least supports joint diagnosis of sound-vibration-temperature and / or gas-temperature-visual linkage for leak tracing. The graded early warning module is used to calculate a confidence score based on the causal reasoning result, perform graded early warnings based on the confidence score, and send the early warning signal back to the remote monitoring platform. The time-series prediction module is used to store the equipment health status vector of each inspection into the time-series database, establish a time-series evolution model of equipment status, fit the equipment deterioration trend curve, predict the remaining service life of the equipment, and send the prediction results back to the remote monitoring platform.

9. The quadruped robot multimodal perception anomaly early warning system according to claim 8, characterized in that, The quadruped robot is also equipped with a millimeter-wave phased array radar. The millimeter-wave phased array radar adopts a frequency-modulated continuous wave system and integrates a phased array antenna array. It achieves beam spatial scanning by electronically controlling the phase of each radiating element. The millimeter-wave phased array radar and lidar form a redundant sensing combination. When the lidar fails in dusty or foggy environments, the detection data of the millimeter-wave phased array radar is switched as the main data source for navigation, positioning, and map building. It is combined with a fusion SLAM algorithm to achieve localization and dynamic path planning. The millimeter-wave phased array radar is also used to cover the robot's blind spot and detect obstacles in the blind spot to trigger emergency stop or detour decisions.

10. The quadruped robot multimodal perception anomaly early warning system according to claim 8, characterized in that, The quadruped robot communicates with the remote monitoring platform using a dual-mode communication method combining 5G public network and wireless local area network; the wireless local area network is used for batch data transmission after inspection, and the 5G public network is used for remote control takeover in abnormal situations.

Images