Intelligent perception diagnosis system and method, readable storage medium and inspection robot

Abstract

The invention relates to the technical field of robot perception and artificial intelligence, and discloses an intelligent perception diagnosis system and method, a readable storage medium and an inspection robot, and the method specifically comprises the steps: obtaining the first type of perception data of an environment through a first sensor module, analyzing the first type of perception data to recognize a feature region, and generating a detection task, and dynamically adjusting working parameters of a second sensor module according to the detection task so as to perform enhanced detection on the feature region, and fusing the first type of perception data and enhanced detection data obtained from the second sensor module so as to generate a representation of the environment. Through cognitive collaboration, active interference suppression and physical information fusion among the sensors, the perception problem of the robot in the glass-metal mixed environment is fundamentally solved, and the robot is endowed with the preliminary diagnosis capability of the equipment health state.

Description

Technical Field

[0001] This invention relates to the technical field of robot perception and artificial intelligence, and more specifically, to a robot intelligent perception and diagnostic system, method, readable storage medium, and inspection robot based on cognitive collaboration and physical information networks. Background Technology

[0002] Inspection robots are increasingly being used in high-value industrial scenarios such as indoor substations. However, the glass viewing windows and smooth metal casings (GIS equipment) commonly found in these scenarios pose a significant challenge to traditional robot navigation sensors. The laser beams emitted by LiDAR (Light Detection and Ranging) can almost completely penetrate glass, resulting in large-area holes in the point cloud at the perception level. Robots may then "ignore" glass obstacles, making collisions highly likely. Simultaneously, smooth metal surfaces produce strong specular reflections of laser and millimeter-wave radar signals, triggering multipath effects. This causes sensors to generate false "ghost points" in the point cloud that deviate from the actual object positions, severely interfering with obstacle avoidance decisions.

[0003] To address this issue, existing technologies have made various attempts. For example, the visual fusion scheme disclosed in Chinese patent CN113806325A attempts to supplement information using an RGB-D camera, but visual sensors also cannot effectively perceive completely transparent glass. Other solutions attempt to simply use millimeter-wave radar as an aid, such as the technology disclosed in CN111625836A, but it typically uses low-resolution millimeter-wave radar and only performs simple back-end data overlay. This loose "data fusion" does not solve the limitations of each sensor individually, nor does it address the multipath interference problem at its root. These solutions essentially remain at the level of "passive perception" and "post-processing fusion," lacking intelligent collaboration between sensors, the ability to actively intervene in interference sources, and the ability to understand the physical nature of the scene. Therefore, their reliability is insufficient in complex dynamic environments. Summary of the Invention

[0004] To address the aforementioned technical problems in related technologies, this invention provides an intelligent sensing and diagnostic system, method, readable storage medium, and inspection robot, which can solve the above problems.

[0005] To achieve the above-mentioned technical objectives, the technical solution of the present invention is implemented as follows: An intelligent sensing and diagnostic system includes a central processing unit and a first sensor module and a second sensor module communicatively connected to the central processing unit. The central processing unit is configured to: generate a detection task based on features (such as point cloud holes or low-density regions) in a first type of sensing data (such as point cloud) acquired by the first sensor module; and adjust the operating parameters (such as beam direction, scanning mode, or transmission power) of the second sensor module in real time and dynamically according to the detection task to enhance the detection of the three-dimensional spatial region corresponding to the features.

[0006] Furthermore, it also includes an active suppression module, which includes an auxiliary transmitting antenna. The central processing unit is further configured to: identify multipath interference signals from the sensing data of the second sensor module, calculate the parameters of the multipath interference signals, and control the auxiliary transmitting antenna to transmit a cancellation signal. The phase of the cancellation signal is opposite to the phase of the multipath interference signal, so as to generate destructive interference with the multipath interference signal at the receiving end of the second sensor module.

[0007] Furthermore, it also includes a synchronization and communication unit, which generates a PPS signal and distributes it to the first sensor module and the second sensor module through hardware lines to ensure that the data acquisition timestamps of the first sensor module and the second sensor module are absolutely aligned.

[0008] Furthermore, the first sensor module is a lidar (LiDAR), and the second sensor module is a millimeter-wave MIMO array radar.

[0009] Furthermore, the central processing unit uses a physical information neural network (PINN) to fuse the data from the first sensor module and the second sensor module. The loss function of the physical information neural network includes one or more partial differential equations describing physical laws as regularization terms.

[0010] Furthermore, the canonical term of the partial differential equation includes the thermodynamic conduction equation, and the temperature anomaly gradient of the device surface in the network diagnostic environment is used as early warning information; the output of the physical information neural network is a semantic three-dimensional voxel field, and each voxel in the three-dimensional voxel field contains an estimated value of occupancy probability, material label and physical state (such as temperature).

[0011] A smart sensing and diagnostic method includes the following steps: S100: Acquire first-type perception data of the environment through the first sensor module; S200: Analyze the first type of sensing data to identify feature regions and generate a detection task; S300. According to the detection task, dynamically adjust the operating parameters of the second sensor module to enhance the detection of the feature region; S310. Identify the multipath interference signal from the second sensor module and transmit a cancellation signal with a phase opposite to the multipath interference signal to suppress the interference at the physical level. S400: The first type of sensing data and the enhanced detection data obtained from the second sensor module are fused to generate a characterization of the environment.

[0012] Furthermore, the fusion step in S400 is implemented through a physical information neural network, the training process of which is constrained by at least one partial differential equation describing physical laws.

[0013] A readable storage medium storing a computer program, which, when executed by a processor, performs the intelligent sensing and diagnostic method.

[0014] An inspection robot, including the aforementioned intelligent sensing and diagnostic system.

[0015] The beneficial effects of this invention are: (1) Cognitive collaboration, high efficiency and reliability: Through the closed-loop mechanism of guiding millimeter-wave radar to focus and scan, the system resources are intelligently allocated to the most needed areas, which greatly improves the detection efficiency and confirmation confidence of difficult obstacles such as glass.

[0016] (2) Active suppression and elimination of interference: By transmitting destructive interference signals, the strongest multipath interference is suppressed from the physical source. Compared with traditional filtering algorithms, the suppression ratio is higher and the effect is more stable, which significantly reduces obstacle avoidance decision errors caused by "ghosting".

[0017] (3) Physical fusion and deep understanding: The Physical Information Neural Network (PINN) is used for robot perception fusion, which upgrades the system from simple geometric mapping to physical understanding of the scene. This not only improves the robustness of perception, but also realizes the functional leap from "obstacle avoidance" to "diagnosis", bringing a new value dimension to the inspection robot.

[0018] (4) High adaptability and maintainability: The system can learn from the environment by analyzing multipath information to retrieve the reflection characteristics of the environment. At the same time, through the physical consistency check of PINN, it can also perform online diagnosis of minor faults (such as temperature drift) of the sensor itself. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] The present invention will now be described in further detail with reference to the accompanying drawings.

[0021] Figure 1 This is a system architecture diagram of the robot intelligent perception and diagnosis system described in this invention; Figure 2 This is a schematic diagram of the closed-loop control process of the cognitive sensor symbiosis mechanism described in this invention; Figure 3 This is a schematic diagram illustrating the structure and working principle of the active multipath suppression module described in this invention; Figure 4 This is a schematic diagram of the physical information neural network (PINN) described in this invention; Figure 5 This is a schematic diagram illustrating the robot's working application according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0023] This invention discloses a robot intelligent perception and diagnosis system and method that enables cognitive collaboration between sensors, active suppression of physical interference, and scene understanding in combination with physical laws. It aims to fundamentally improve the perception reliability and obstacle avoidance safety of robots in extreme mixed environments such as glass and metal, and endow them with the ability to make preliminary diagnoses of the status of environmental equipment. It is implemented on a wheeled inspection robot platform.

[0024] Example 1: The system in this application consists of a hardware layer and a software layer.

[0025] like Figure 1 As shown, the hardware layer includes a first sensor module 1, a second sensor module 2, an active suppression module 3, a central processing unit 4, and a synchronization and communication unit.

[0026] The first sensor module 1 uses an Ouster OS1-32 32-line LiDAR with a horizontal field of view of 360° and a vertical field of view of ±22.5°, outputting 3D point cloud data at a frequency of 20Hz. It is responsible for performing global environment scanning to acquire high-resolution geometric contour information.

[0027] The second sensor module 2 is a custom-designed dual-frequency millimeter-wave radar for this invention. Developed based on TI's AWR2944 chip, this radar integrates four transmit channels (TX) and four receive channels (RX), forming a 16-channel virtual MIMO array. The radar can operate in a time-division multiplexing configuration in the 60-64 GHz band (for wide field-of-view imaging) and the 77-81 GHz band (for high-resolution detection). Its RF front-end phase shifter supports digital beamforming (DBF), concentrating beam energy within a narrow field of view of ±5°. It possesses all-weather capability and exhibits excellent detection performance on non-metallic materials such as glass.

[0028] The active suppression module 3 consists of a single-channel auxiliary millimeter-wave transmitting antenna and a control circuit board. The control circuit board integrates a high-speed DAC (such as Analog Devices' AD9162) and a programmable phase shifter, and is controlled by the central processing unit 4 via the SPI bus. Upon identifying a critical multipath interference path, this module is configured to transmit a canceling signal with opposite phase, actively eliminating the interference at the physical level, rather than relying solely on passive filtering by backend algorithms.

[0029] The central processing unit (CPU) 4 utilizes the NVIDIA Jetson AGX Orin development kit, which includes a high-performance ARM Cortex-A78AE CPU cluster and a 1792-core Ampere architecture GPU, providing computational support for complex algorithms. As the "brain" of the system, the CPU executes the core cognitive collaboration, active inhibition, and fusion diagnostic algorithms of this invention. It is configured to generate detection tasks based on features (such as point cloud holes) in the data from the first sensor module, and dynamically adjust the operating parameters of the second sensor module accordingly (such as performing beamforming focusing scans), achieving closed-loop task-driven collaboration between sensors—this is the "cognitive symbiosis mechanism." The CPU further employs a Physical Information Neural Network (PINN) as the final fusion and decision-making core. This network, based on the traditional neural network loss function, introduces partial differential equations (PDEs) describing the laws of the physical world as regularization terms, such as the law of electromagnetic wave reflection and the thermodynamic conduction equation. Constrained by these physical laws, the output of PINN is no longer a simple three-dimensional grid, but a "digital twin" scene with physical attributes and semantic labels. This scenario can not only characterize obstacles with extremely high accuracy and completeness for obstacle avoidance, but also infer the physical state of each location, such as material (glass / metal) and surface temperature, thereby enabling early diagnosis of physical anomalies such as local overheating of equipment.

[0030] The synchronization and communication unit uses an STM32H750VB microcontroller as the master clock source. This MCU generates a PPS (Pulse Per Second) signal with an accuracy better than 50 nanoseconds, which is distributed to the first sensor module 1 (LiDAR) and the second sensor module 2 (millimeter-wave radar) via hardware circuitry, ensuring absolute alignment of their data acquisition timestamps. Data transmission between each module and the central processing unit 4 is achieved via gigabit industrial Ethernet.

[0031] The system software is developed based on the ROS 2 (Robot Operating System 2) Galactic framework, which ensures low-latency and highly reliable communication between various algorithm modules.

[0032] Example 2: like Figure 2 The diagram shows the implementation details of the cognitive symbiosis mechanism, which is implemented by a ROS 2 node named Cognitive_Scheduler and includes the following steps: Point cloud hole detection: The Cognitive_Scheduler node subscribes to the / lidar / points topic published by LiDAR1. It first projects the point cloud onto a 2D polar coordinate grid map, with each grid cell recording the number of points. Then, it uses a fast image morphology "closing operation" to fill small, unstructured holes. For large, continuous hole areas that still exist, the DBSCAN clustering algorithm is used to identify them. A hole with an area greater than 0.5 square meters and a regular shape (such as a rectangle) will be classified as a "high-priority suspected glass area".

[0033] Generate Focused Probe Task 201: For each high-priority region, the system calculates its 3D bounding box (center point coordinates, length, width, height, and orientation) in the robot coordinate system. This information is encapsulated into a custom ROS 2 message type, FocusScanTask, and published via the / mmwave / task topic. This message also includes a task ID and a timestamp.

[0034] Execute Focused Scan 202: The driver node of the millimeter-wave radar subscribes to the / mmwave / task topic. Upon receiving the task, the driver immediately issues instructions to the AWR2944 chip: (1) switch to the 79GHz high-resolution working mode; (2) calculate the azimuth and elevation angles that the beam needs to point to based on the coordinates in the task, and update the phase control register of the DBF accordingly; (3) increase the repetition frequency of the Chirp (frequency-modulated continuous wave) from the usual 1kHz to 5kHz, and perform a dense scan of the area within 0.1 seconds to obtain enhanced data with a signal-to-noise ratio improvement of about 15dB.

[0035] Result Confirmation and Feedback 203: Enhanced millimeter-wave point cloud data is published back to the / mmwave / result topic as a FocusScanResult message, which includes the corresponding task ID. The Cognitive_Scheduler node receives this result. If a flat point cluster with an RCS (radar cross section) value matching the characteristics of glass is detected in the area, the confidence level of the area is updated to 95% "confirmed glass obstacle", and its geometric information is published to the global cost map layer.

[0036] Example 3: like Figure 3 The following are the implementation details of active multipath suppression, a function implemented by a ROS 2 node named Multipath_Canceller, which includes the following steps: Strong Multipath Identification 302: This node analyzes conventional millimeter-wave scan data. It maintains a historical trajectory and identifies "ghost points" that appear in more than 10 consecutive frames at a mirror position relative to the robot's stationary position and have an energy value higher than -10 dBsm as strong multipath interference caused by stable mirror reflection.

[0037] Parameter calculation: Using the actual wall (reflective surface) position provided by the lidar point cloud, the transmission path length (delay τ) and phase difference φ relative to the direct wave of the multipath signal are accurately calculated through geometric relationships. Its amplitude A can be directly read from the radar echo.

[0038] The cancellation signal generation and transmission process 303: The Multipath_Canceller node sends the calculated amplitude A and phase (φ+π) to the control circuit of the active suppression module 3 via the SPI bus. The DAC and phase shifter then generate the corresponding analog baseband signal, which is up-converted and transmitted through the auxiliary antenna. The closed-loop delay of this process is controlled within 5 milliseconds, ensuring real-time cancellation. Testing has shown that this method can achieve stable suppression of over 20dB against strong multipath signals.

[0039] Example 4: like Figure 4 The implementation details of the Physical Information Neural Network (PINN) are shown. This Physical Information Neural Network (PINN) 401 is implemented using the TensorFlow framework and runs as a PINN_Fusion ROS 2 node.

[0040] Network structure: An improved MLP (Multilayer Perceptron) structure is adopted, containing 8 fully connected layers, each with 256 neurons, and the activation function is tanh. To accelerate convergence, a Fourier feature embedding layer is added to the network to encode the input coordinates at high frequency.

[0041] Training and Inference: The network is pre-trained offline using simulated data and a small amount of labeled real-world scene data. During robot operation, the network performs online inference. Input layer 402 receives synchronized data frames from various sensor topics.

[0042] Loss Function 403 Details: During the inference phase, the loss function is used for online fine-tuning and anomaly detection. The specific structure of the L_phys term is as follows: Thermal conductivity PDE residuals: .in, Calculated by temperature output difference between two consecutive frames. The Laplace operator performs second-order spatial difference calculations on the temperature field output by the network. The thermal diffusivity α is set to a constant based on the material of the GIS equipment. When f_heat remains above a threshold in a local area, a "temperature gradient anomaly" alarm is triggered.

[0043] Output layer 404 details: The network outputs a 3D voxel field with a resolution of 5cm. The output vector for each voxel is [P_occ, C_metal, C_glass, T_est, F_anomaly], where P_occ is the occupancy probability, C_metal and C_glass are the confidence scores for the material being metal and glass, respectively, T_est is the temperature estimate, and F_anomaly is the anomaly alarm flag. This voxel field is published as ROS 2's sensor_msgs / PointCloud2 and custom diagnostic_msgs / DiagnosticArray messages for use by navigation and monitoring systems.

[0044] Example 5: like Figure 5 The image shows an example of an end-to-end application scenario, where robot 501 performs an inspection task in the GIS room.

[0045] The robot travels along a preset path while the LiDAR continuously scans. When it approaches a GIS device 504 equipped with a glass observation window 503, the Cognitive_Scheduler node detects a point cloud hole in the glass area and immediately triggers a focused scan of the millimeter-wave radar, confirming the precise location and size of the glass obstacle within 100 milliseconds.

[0046] Meanwhile, the smooth metal surface of the GIS device 504 generated strong multipath reflections, creating a false "ghost" obstacle on the side of the robot. The Multipath_Canceller node identified this stable multipath within 3-4 frames and activated the active suppression module 3. In subsequent scans, the "ghost" disappeared from the millimeter-wave data.

[0047] All preprocessed, clean data is fed into the PINN_Fusion node. PINN not only fuses and generates a complete, error-free 3D map, guiding the robot to avoid the glass window at a safe distance of 5cm, but also continuously outputs the surface temperature field of the GIS device 504.

[0048] At inspection point P1, PINN's output showed that although the local temperature of a bus connector was only 45℃ (not exceeding the absolute threshold), its f_heat residual value was two orders of magnitude higher than the surrounding area. The system immediately set the F_anomaly flag to one and issued a diagnostic message to the host computer monitoring system via diagnostic_msgs: "Device ID: GIS-07A, Location: C-phase bus, suspected overheating risk, abnormal temperature gradient," achieving early warning of the fault.

[0049] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent sensing and diagnostic system, characterized in that, The system includes a central processing unit and a first sensor module and a second sensor module that are communicatively connected to the central processing unit. The central processing unit is configured to: generate a detection task based on features in a first type of perception data acquired by the first sensor module, and adjust the operating parameters of the second sensor module in real time and dynamically according to the detection task to enhance the detection of the three-dimensional spatial region corresponding to the features.

2. The intelligent sensing and diagnostic system according to claim 1, characterized in that, It also includes an active suppression module, which includes an auxiliary transmitting antenna. The central processing unit is further configured to: identify multipath interference signals from the sensing data of the second sensor module, calculate the parameters of the multipath interference signals, and control the auxiliary transmitting antenna to transmit a cancellation signal. The phase of the cancellation signal is opposite to the phase of the multipath interference signal, so as to generate destructive interference with the multipath interference signal at the receiving end of the second sensor module.

3. The intelligent sensing and diagnostic system according to claim 1, characterized in that, It also includes a synchronization and communication unit, which generates a PPS signal and distributes it to the first sensor module and the second sensor module through hardware lines to ensure that the data acquisition timestamps of the first sensor module and the second sensor module are absolutely aligned.

4. The intelligent sensing and diagnostic system according to claim 1, characterized in that, The first sensor module is a lidar, and the second sensor module is a millimeter-wave MIMO array radar.

5. The intelligent sensing and diagnostic system according to claim 1, characterized in that, The central processing unit uses a physical information neural network to fuse the data from the first sensor module and the second sensor module. The loss function of the physical information neural network includes one or more partial differential equations describing physical laws as regularization terms.

6. The intelligent sensing and diagnostic system according to claim 5, characterized in that, The canonical term of the partial differential equation includes the thermodynamic conduction equation, and the temperature anomaly gradient on the surface of the device in the network diagnostic environment is used as early warning information; the output of the physical information neural network is a semantic three-dimensional voxel field, and each voxel in the three-dimensional voxel field contains an occupancy probability, material label and physical state estimate.

7. An intelligent sensing and diagnostic method, characterized in that, Includes the following steps: S100: Acquire first-type perception data of the environment through the first sensor module; S200: Analyze the first type of sensing data to identify feature regions and generate a detection task; S300. According to the detection task, dynamically adjust the operating parameters of the second sensor module to enhance the detection of the feature region; S310. Identify the multipath interference signal from the second sensor module and transmit a cancellation signal with a phase opposite to the multipath interference signal to suppress the interference at the physical level. S400: The first type of sensing data and the enhanced detection data obtained from the second sensor module are fused to generate a characterization of the environment.

8. The intelligent sensing and diagnostic method according to claim 7, characterized in that, The fusion step in S400 is implemented through a physical information neural network, the training process of which is constrained by at least one partial differential equation describing physical laws.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the program is run by the processor, it executes the intelligent sensing and diagnostic method as described in any one of claims 7 to 8.

10. An inspection robot, characterized in that, Including the intelligent sensing and diagnostic system as described in any one of claims 1 to 6.

Images