Robot intelligent inspection method and system based on multi-source data fusion

Abstract

The invention provides a robot intelligent inspection method and system based on multi-source data fusion, and relates to the technical field of intelligent inspection, and the method comprises the steps: carrying out the data collection of an inspection region based on an inspection robot; adjusting the enhancement parameter of the equipment appearance image, highlighting the equipment operation sound in the inspection area sound, correcting the equipment temperature distribution, and compensating the area gas concentration; extracting defect features in the equipment appearance image, capturing sound features in equipment operation sound, extracting temperature features in equipment temperature distribution, and learning gas concentration features corresponding to regional gas concentration; and performing space-time fusion on each feature to generate a space-time fusion inspection feature, outputting an equipment fault diagnosis result and adjusting a current inspection path of the inspection robot, thereby cooperatively collecting multi-dimensional information of the equipment, optimizing data quality through adaptive processing, mining associated features through space-time fusion, and adjusting the inspection path in combination with intelligent decision. Fault diagnosis reliability is improved, and safe and stable operation of equipment is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of intelligent inspection technology, and in particular to a robot intelligent inspection method and system based on multi-source data fusion. Background Art

[0002] Traditional robotic inspection methods often rely on data from a single sensor, resulting in limited detection dimensions and poor environmental adaptability. Optical image inspections are easily affected by lighting, leading to missed defects. Acoustic inspections struggle to effectively identify abnormal equipment noises in noisy environments. Thermal infrared temperature measurement is susceptible to interference from ambient temperature, resulting in errors. Gas monitoring, due to its single data dimension, is unable to correlate with equipment operating status. Existing technologies often process data independently, without establishing spatiotemporal correlations between multi-source data. This results in low fault diagnosis accuracy and high false alarm rates. Furthermore, inspection paths are fixed, and the efficiency of re-sampling abnormal areas is low, making it difficult to meet the needs of high-precision, full-scene inspections in complex industrial environments.

[0003] Therefore, it is necessary to provide a robot intelligent inspection method and system based on multi-source data fusion to solve the above technical problems. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention provides a robot intelligent inspection method and system based on multi-source data fusion, which is used to solve the problems of existing inspection methods relying on single data, lacking multi-source association, low detection accuracy, poor environmental adaptability, and inflexible path adjustment.

[0005] The present invention provides a robot intelligent inspection method based on multi-source data fusion, the method comprising: Based on an inspection robot equipped with a multi-source sensor array, all-round data collection is performed on the target inspection area, wherein the multi-source sensor array includes an optical sensor, an acoustic sensor, a thermal infrared sensor and a gas sensor. The optical sensor is used to collect an inspection device appearance image of the target inspection device in the target inspection area, the acoustic sensor is used to capture the inspection area sound of the target inspection area, the thermal infrared sensor is used to obtain the inspection device temperature distribution of the target inspection device, and the gas sensor is used to monitor the inspection area gas concentration of the target inspection area; Automatically adjust the image enhancement parameters of the inspection equipment appearance image based on an adaptive enhancement algorithm, highlight the inspection equipment operation sound in the inspection area sound based on an adaptive filtering algorithm, adaptively correct the inspection equipment temperature distribution based on the inspection equipment heat conduction model, and compensate for the gas concentration in the inspection area in real time based on a Kalman filtering algorithm; Based on the convolutional neural network, the inspection equipment appearance defect features in the inspection equipment appearance image are extracted. The inspection equipment sound features in the operation sound of the inspection equipment are captured by combining the deep belief network and the recurrent neural network. The inspection equipment temperature features in the temperature distribution of the inspection equipment are extracted using the manifold learning method. The inspection area gas concentration features corresponding to the gas concentration in the inspection area are learned based on the time series model. Performing spatiotemporal fusion on the appearance defect features of the inspection equipment, the sound features of the inspection equipment, the temperature features of the inspection equipment, and the gas concentration features of the inspection area to generate spatiotemporal fusion inspection features; According to the spatiotemporal fusion inspection features of all the target inspection devices in the target inspection area, an inspection device fault diagnosis result is output, and the current inspection path of the inspection robot is automatically adjusted.

[0006] Preferably, the optical sensor is used to collect an inspection device appearance image of a target inspection device in the target inspection area, the acoustic sensor is used to capture the inspection area sound of the target inspection area, the thermal infrared sensor is used to obtain the inspection device temperature distribution of the target inspection device, and the gas sensor is used to monitor the inspection area gas concentration of the target inspection area, specifically including: Obtaining a distance between the optical sensor and the target inspection device; if the distance is greater than or equal to a focus adjustment critical value, activating a long-distance wide-angle mode of the optical sensor to capture an overall inspection device image of the target inspection device; if the distance is less than the focus adjustment critical value, activating a short-distance telephoto mode of the optical sensor to capture a detailed inspection device image of the target inspection device; and summarizing the overall inspection device image and the detailed inspection device image to generate an appearance image of the inspection device; Based on the array design of the acoustic sensor, the sound of the inspection area of ​​the target inspection area is collected through beamforming technology and the source direction of the sound of the inspection area is located; The thermal infrared sensor collects a low-resolution thermal infrared image of the target inspection device, converts the low-resolution thermal infrared image into a high-resolution thermal infrared image using a super-resolution reconstruction algorithm, and quantifies the high-resolution thermal infrared image to output the temperature distribution of the inspection device; Based on the micro-electromechanical system technology of the gas sensor, it responds to and identifies each target gas component in the target inspection area, and calculates the gas concentration corresponding to all the target gas components to generate the gas concentration of the inspection area.

[0007] Preferably, the automatic adjustment of image enhancement parameters of the inspection equipment appearance image based on the adaptive enhancement algorithm, the highlighting of the inspection equipment operation sound in the inspection area sound based on the adaptive filtering algorithm, the adaptive correction of the inspection equipment temperature distribution based on the inspection equipment heat conduction model, and the real-time compensation of the gas concentration in the inspection area based on the Kalman filtering algorithm specifically include: In the adaptive enhancement algorithm based on the generative adversarial network, the generator generates an enhanced image of the inspection equipment based on the input appearance image of the inspection equipment, and the discriminator optimizes the enhancement effect of the enhanced image of the inspection equipment through adversarial training; In the adaptive filtering algorithm based on variational mode decomposition, different frequency components in the inspection area sound are separated according to the intrinsic mode function, and the frequency components within the preset operating sound frequency range are extracted to generate the inspection equipment operating sound and perform filtering processing; Establishing a heat conduction model for the inspection equipment and correcting the temperature distribution of the inspection equipment based on the heat dissipation of the target inspection equipment; The Kalman filter algorithm is used to dynamically compensate the gas concentration in the inspection area in combination with the real-time environmental parameters of the target inspection area, including real-time temperature, real-time humidity and real-time air pressure.

[0008] Preferably, the extraction of inspection equipment appearance defect features from the inspection equipment appearance image based on a convolutional neural network, the capture of inspection equipment sound features from the inspection equipment operation sound in combination with a deep belief network and a recurrent neural network, the extraction of inspection equipment temperature features from the inspection equipment temperature distribution using a manifold learning method, and the learning of inspection area gas concentration features corresponding to the inspection area gas concentration based on a time series model specifically include: Constructing a convolutional neural network based on an attention mechanism, and introducing an attention module into the convolutional neural network to automatically identify the appearance defect features of the inspection equipment on the surface of the target inspection equipment; The inspection equipment operating sound is subjected to feature learning through a deep belief network to generate the inspection equipment sound spectrum features, and a recurrent neural network is used to process the time series information of the inspection equipment sound spectrum features and capture dynamic change patterns, thereby extracting the inspection equipment sound features from the inspection equipment operating sound; Using a manifold learning method based on principal component analysis and local linear embedding, the high-dimensional temperature distribution data in the temperature distribution of the inspection equipment is projected into a low-dimensional space to generate the temperature features of the inspection equipment; A time series model based on a long short-term memory network is constructed to learn the long-term and short-term variation trends of the gas concentration in the inspection area, and output the concentration statistical characteristics of the gas concentration in the inspection area, that is, the gas concentration characteristics of the inspection area.

[0009] Preferably, the spatiotemporal fusion of the inspection equipment appearance defect features, the inspection equipment sound features, the inspection equipment temperature features, and the inspection area gas concentration features to generate spatiotemporal fusion inspection features specifically includes: Constructing a Bayesian network based on the probability distribution and conditional probability relationship of the inspection equipment appearance defect characteristics, the inspection equipment sound characteristics, the inspection equipment temperature characteristics, and the inspection area gas concentration characteristics; Based on the Bayesian network, the characteristic causal relationship between the inspection equipment appearance defect characteristics, the inspection equipment sound characteristics, the inspection equipment temperature characteristics and the inspection area gas concentration characteristics is analyzed, and the feature fusion weights corresponding to the inspection equipment appearance defect characteristics, the inspection equipment sound characteristics, the inspection equipment temperature characteristics and the inspection area gas concentration characteristics are determined respectively; Based on the feature fusion weights corresponding to the appearance defect features of the inspection equipment, the sound features of the inspection equipment, the temperature features of the inspection equipment, and the gas concentration features of the inspection area, a spatiotemporal graph convolutional network is constructed in the time and space dimensions; The spatiotemporal graph convolutional network is used to learn and establish the spatiotemporal correlation between the appearance defect characteristics of the inspection equipment, the sound characteristics of the inspection equipment, the temperature characteristics of the inspection equipment and the gas concentration characteristics of the inspection area, and output the spatiotemporal fusion inspection characteristics.

[0010] Preferably, a multi-source data fusion algorithm based on quantum computing is established to map the spatiotemporal fusion inspection features of all the target inspection devices into the quantum state space, and to search and match the potential correlations between the spatiotemporal fusion inspection features by utilizing the superposition and entanglement properties of quantum bits; Blockchain technology is introduced to record and store the matched spatiotemporal fusion inspection features.

[0011] Preferably, the outputting of inspection equipment fault diagnosis results based on the spatiotemporal fusion inspection features of all the target inspection equipment in the target inspection area and automatically adjusting the current inspection path of the inspection robot specifically includes: Using a multi-task neural network based on transfer learning, fault diagnosis is performed on the spatiotemporal fusion inspection features of all target inspection devices, and the inspection device fault diagnosis results are output; Obtaining a three-dimensional inspection area map corresponding to the target inspection area, and automatically adjusting the current inspection path using a reinforcement learning-based path planning algorithm based on the inspection equipment fault diagnosis result and the three-dimensional inspection area map, wherein a secondary inspection path needs to be added for the target inspection equipment with a fault; Using virtual reality and augmented reality technologies, the fault diagnosis results of the inspection equipment are displayed to the operation and maintenance end; Obtain a knowledge base of historical faults of inspection equipment, establish an inspection knowledge graph, and associate and integrate the inspection equipment fault diagnosis results with the knowledge base of historical faults of inspection equipment.

[0012] A robot intelligent inspection system based on multi-source data fusion, the system comprising: a data acquisition module, configured to perform all-round data acquisition of a target inspection area based on an inspection robot equipped with a multi-source sensor array, wherein the multi-source sensor array includes an optical sensor, an acoustic sensor, a thermal infrared sensor, and a gas sensor; the optical sensor is configured to collect an inspection device appearance image of a target inspection device in the target inspection area; the acoustic sensor is configured to capture inspection area sounds of the target inspection area; the thermal infrared sensor is configured to obtain an inspection device temperature distribution of the target inspection device; and the gas sensor is configured to monitor inspection area gas concentration in the target inspection area; a data preprocessing module for automatically adjusting image enhancement parameters of the inspection equipment's appearance image based on an adaptive enhancement algorithm, highlighting the inspection equipment's operating sounds in the inspection area's sounds based on an adaptive filtering algorithm, adaptively correcting the inspection equipment's temperature distribution based on a thermal conduction model of the inspection equipment, and compensating for gas concentration in the inspection area in real time based on a Kalman filtering algorithm; a feature extraction module for extracting appearance defect features of the inspection equipment from the appearance image of the inspection equipment based on a convolutional neural network, capturing the inspection equipment sound features from the operating sound of the inspection equipment by combining a deep belief network with a recurrent neural network, extracting the inspection equipment temperature features from the temperature distribution of the inspection equipment using a manifold learning method, and learning the inspection area gas concentration features corresponding to the inspection area gas concentration based on a time series model; A feature fusion module is used to perform spatiotemporal fusion of the inspection equipment appearance defect features, the inspection equipment sound features, the inspection equipment temperature features, and the inspection area gas concentration features to generate spatiotemporal fusion inspection features; The path adjustment module is used to output the inspection equipment fault diagnosis result according to the spatiotemporal fusion inspection characteristics of all the target inspection equipment in the target inspection area, and automatically adjust the current inspection path of the inspection robot.

[0013] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the steps of the robot intelligent inspection method based on multi-source data fusion as described above.

[0014] A readable storage medium stores a computer program, which, when executed by a processor, is used to implement the steps of the robot intelligent inspection method based on multi-source data fusion as described above.

[0015] Compared with related technologies, the robot intelligent inspection method based on multi-source data fusion and its system provided by the present invention have the following beneficial effects: The present invention is based on an inspection robot equipped with a multi-source sensor array to collect all-round data of the target inspection area, wherein the multi-source sensor array includes an optical sensor, an acoustic sensor, a thermal infrared sensor and a gas sensor, the optical sensor is used to collect the inspection equipment appearance image of the target inspection equipment in the target inspection area, the acoustic sensor is used to capture the inspection area sound of the target inspection area, the thermal infrared sensor is used to obtain the inspection equipment temperature distribution of the target inspection equipment, and the gas sensor is used to monitor the inspection area gas concentration of the target inspection area; based on the adaptive enhancement algorithm, the image enhancement parameters of the inspection equipment appearance image are automatically adjusted, based on the adaptive filtering algorithm, the inspection equipment operation sound in the inspection area sound is highlighted, based on the patrol equipment heat conduction model, the patrol equipment temperature distribution is adaptively corrected, and based on the Kalman filtering algorithm, the patrol area gas concentration is compensated in real time; based on the convolutional neural network, the inspection equipment appearance image is extracted The appearance defect characteristics of the inspection equipment are combined with the deep belief network and the recurrent neural network to capture the inspection equipment sound characteristics in the inspection equipment operation sound, the manifold learning method is used to extract the inspection equipment temperature characteristics in the inspection equipment temperature distribution, and the inspection area gas concentration characteristics corresponding to the gas concentration in the inspection area are learned based on the time series model; the inspection equipment appearance defect characteristics, inspection equipment sound characteristics, inspection equipment temperature characteristics and inspection area gas concentration characteristics are spatiotemporally fused to generate spatiotemporal fused inspection characteristics; according to the spatiotemporal fused inspection characteristics of all target inspection equipment in the target inspection area, the inspection equipment fault diagnosis results are output, and the current inspection path of the inspection robot is automatically adjusted, so that multi-source sensors can collaboratively collect multi-dimensional information of the equipment, optimize data quality through adaptive processing, mine related features through spatiotemporal fusion, and dynamically adjust the inspection path in combination with intelligent decision-making to improve the reliability of fault diagnosis and ensure safe and stable operation of the equipment.

[0016] The present invention adopts multi-source data fusion and intelligent analysis technology. Through the collaborative work of multiple types of sensors, it synchronously collects multi-dimensional information such as equipment appearance characteristics, operating sounds, surface temperature and environmental gases. It breaks through the limitations of traditional single data detection, realizes all-round perception of the equipment's operating status, ensures that no potential abnormal signals are missed, and significantly improves the comprehensiveness and reliability of industrial equipment inspections. The present invention uses an adaptive data preprocessing mechanism to dynamically optimize data quality according to different environmental characteristics. In scenarios such as complex lighting, noise interference, temperature and humidity fluctuations, it can still effectively retain key features and avoid environmental factors from interfering with the detection results. The present invention can accurately capture effective information from various types of data and fully explore characteristics such as subtle defects, abnormal sounds, temperature anomalies and gas changes in equipment. The multi-source data fusion technology of the present invention establishes a spatiotemporal correlation model, deeply analyzes the inherent connections between different types of data, integrates scattered feature information into a complete description of the equipment status, and transforms fault judgment from one-sided speculation to comprehensive analysis, greatly improving the accuracy of fault diagnosis and reducing misjudgments and missed judgments. The intelligent decision-making and path adjustment mechanism of this invention achieves dynamic optimization of the inspection process, can flexibly adjust inspection priorities based on the actual status of the equipment, conduct targeted re-sampling of abnormal areas, and improve the utilization efficiency of inspection resources. By applying knowledge graphs and visualization technologies, this invention makes inspection results more intuitive and easy to understand, providing clear decision-making references for operation and maintenance personnel, helping to quickly develop treatment plans, and effectively ensuring the safe and stable operation of industrial equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flowchart of a robot intelligent inspection method based on multi-source data fusion provided by an embodiment of the present invention; Figure 2 A system block diagram of a robot intelligent inspection system based on multi-source data fusion provided by an embodiment of the present invention; Figure 3 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0019] like Figure 1 FIG. 1 is a flow chart of a robot intelligent inspection method based on multi-source data fusion provided by an embodiment of the present invention. Figure 1The execution subject of the method shown may be a software and / or hardware device. The execution subject of the present application may include but is not limited to at least one of the following: user equipment, network equipment, etc. Among them, user equipment may include but is not limited to computers, smart phones, personal digital assistants (PDAs) and the electronic devices mentioned above. Network equipment may include but is not limited to a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computers. This embodiment does not limit this. It includes steps S1 to S5, as follows: S1, based on the inspection robot equipped with a multi-source sensor array, performing all-round data collection on the target inspection area, wherein the multi-source sensor array includes an optical sensor, an acoustic sensor, a thermal infrared sensor and a gas sensor, the optical sensor is used to collect the inspection equipment appearance image of the target inspection equipment in the target inspection area, the acoustic sensor is used to capture the inspection area sound of the target inspection area, the thermal infrared sensor is used to obtain the inspection equipment temperature distribution of the target inspection equipment, and the gas sensor is used to monitor the inspection area gas concentration of the target inspection area; Optical sensors are sensors that use optical imaging technology to capture images of the appearance and surface details of target inspection equipment. Acoustic sensors capture various sound signals within the target inspection area, including equipment operating sounds and ambient noise. Thermal infrared sensors detect the surface temperature distribution of target inspection equipment by receiving infrared radiation signals from objects. Gas sensors are sensors used to detect the composition and concentration changes of various gases within the target inspection area.

[0020] The inspection robot is equipped with a multi-source sensor array, which achieves all-round perception of the target inspection area through the collaborative work of multiple sensors, breaking through the limitations of the limited detection dimension of a single sensor.

[0021] The multi-source sensor array comprises four core sensing devices: optical, acoustic, thermal infrared, and gas sensors. These sensors acquire equipment and environmental information from different physical dimensions. Optical sensors focus on monitoring the appearance of target equipment, capturing the physical condition of the equipment surface through imaging technology. For example, during substation inspections, they can capture images of transformer bushings and the instrument display status of switchgear, providing raw image data for subsequent appearance defect identification. Acoustic sensors collect sound signals within the target inspection area, including both the inherent sounds generated by equipment operation and background noise, such as motor vibration and pipe airflow. Thermal infrared sensors detect infrared radiation characteristics of objects to generate surface temperature distribution data for target equipment. This provides a visual representation of heat distribution across various parts of the equipment, such as the temperature distribution of connectors in distribution cabinets or the temperature field changes on the surface of reactors. Gas sensors monitor gas composition and concentration changes within the target inspection area. In scenarios such as chemical parks, they can detect the concentration of leaked gases and volatiles in real time, providing data support for environmental safety and equipment sealing status assessment.

[0022] S2, automatically adjusting image enhancement parameters of the inspection equipment appearance image based on an adaptive enhancement algorithm, highlighting the inspection equipment operating sound in the inspection area sound based on an adaptive filtering algorithm, adaptively correcting the inspection equipment temperature distribution based on a thermal conduction model of the inspection equipment, and compensating the gas concentration in the inspection area in real time based on a Kalman filtering algorithm; An adaptive enhancement algorithm is used to process the appearance images of inspection equipment obtained by optical sensors. This algorithm can automatically adjust image enhancement parameters based on the inherent characteristics of the image, such as lighting conditions and contrast. For example, it can enhance image brightness and details in a factory environment with insufficient lighting, and suppress overexposed areas in scenes with direct strong light, ensuring that the appearance features of the equipment are clearly visible.

[0023] For the sounds in the inspection area collected by the acoustic sensor, the operating sounds of the inspection equipment are highlighted through an adaptive filtering algorithm. The algorithm can dynamically identify the effective frequency bands and interference frequency bands in the sound signal, filter out background noise such as mechanical vibration and human activities in a noisy industrial environment, and retain the characteristic sounds of equipment operation, such as abnormal noise from pump operation and characteristic sound patterns of valve opening and closing.

[0024] The temperature distribution data of inspection equipment obtained by thermal infrared sensors is adaptively corrected based on the thermal conduction model of the inspection equipment. This model combines the thermal conductivity characteristics of the equipment material, ambient temperature changes and other factors to eliminate the impact of environmental interference on temperature measurement. For example, during outdoor inspections, the deviation of the ambient temperature difference from the equipment surface temperature reading can be corrected to ensure that the temperature data truly reflects the operating status of the equipment.

[0025] The Kalman filter algorithm is used to perform real-time compensation for the gas concentration data in the inspection area monitored by the gas sensor. By dynamically fusing the gas concentration measurement value with the changing trend of environmental parameters, the measurement fluctuations are smoothed, ensuring the stability and accuracy of the gas concentration data in scenarios with drastic changes in temperature and humidity.

[0026] S3, extracting appearance defect features of the inspection equipment from the appearance image of the inspection equipment based on a convolutional neural network, capturing the inspection equipment sound features in the operating sound of the inspection equipment by combining a deep belief network with a recurrent neural network, extracting the inspection equipment temperature features from the temperature distribution of the inspection equipment using a manifold learning method, and learning the inspection area gas concentration features corresponding to the gas concentration in the inspection area based on a time series model; For the appearance images of inspection equipment, a convolutional neural network is used for feature extraction. This network captures the local details and global features of the image layer by layer through multi-layer convolution operations, and accurately identifies appearance defect features such as cracks, corrosion, and deformation on the equipment surface. For example, during the inspection of wind power equipment, subtle damage features on the blade surface can be extracted.

[0027] For the operating sounds of inspection equipment, feature mining is performed by combining deep belief networks and recurrent neural networks. The deep belief network learns the deep spectral characteristics of sound signals through multi-layer nonlinear transformations, while the recurrent neural network is good at capturing the dynamic changes of sound over time. The combination of the two can effectively extract the sound characteristics of equipment operation, such as the characteristic frequency of motor bearing wear and the soundprint changes of abnormal gear meshing.

[0028] For the temperature distribution data of inspection equipment, the manifold learning method is used for feature extraction. This method maps high-dimensional temperature distribution data to a low-dimensional space, retains key temperature features, and accurately extracts temperature features such as the hotspot distribution and temperature gradient of the equipment, such as the overheating area features of the connectors in the distribution cabinet.

[0029] For the gas concentration data in the inspection area, its characteristic laws are learned based on the time series model. The model analyzes the changing trend of gas concentration over time and extracts gas concentration characteristics such as concentration fluctuation cycle and mutation characteristics, such as the slow increase in gas concentration due to trace leakage in the pipeline.

[0030] S4, performing spatiotemporal fusion on the inspection equipment appearance defect feature, the inspection equipment sound feature, the inspection equipment temperature feature, and the inspection area gas concentration feature to generate a spatiotemporal fusion inspection feature; It is understandable that a spatiotemporal correlation analysis can be performed on the appearance defect features, sound features, temperature features, and gas concentration features of the inspection equipment. The collection moments of different features can be aligned in the time dimension, and the equipment positions corresponding to the features can be matched in the spatial dimension. By establishing an intrinsic correlation model between multi-source features, the scattered feature information can be fused into a unified spatiotemporal fusion inspection feature.

[0031] This fusion processing breaks through the limitations of a single feature and achieves the complementarity and enhancement of multi-dimensional information. For example, it correlates and integrates the overheating temperature characteristics of the equipment with the abnormal sound characteristics and appearance deformation characteristics to more comprehensively reflect the potential fault status of the equipment.

[0032] S5, outputting inspection equipment fault diagnosis results according to the spatiotemporal fusion inspection features of all the target inspection equipment in the target inspection area, and automatically adjusting the current inspection path of the inspection robot.

[0033] Finally, based on the spatiotemporal fusion inspection characteristics of all target inspection devices in the target inspection area, the fault diagnosis results of the inspection equipment can be output through the intelligent diagnosis model to clarify whether there are any abnormalities in the equipment and the type of abnormality.

[0034] At the same time, the inspection robot's current inspection path is automatically adjusted based on the fault diagnosis results. For equipment areas with abnormalities, the inspection frequency is increased or re-routes are planned to achieve precise scheduling of inspection resources. For example, if a distribution cabinet is diagnosed with temperature anomalies and gas leak risks, the robot can automatically adjust its path to conduct a close-range inspection of that area, providing more detailed data support for operation and maintenance decisions.

[0035] In a specific implementation process, the optical sensor is used to collect an inspection device appearance image of the target inspection device in the target inspection area, the acoustic sensor is used to capture the inspection area sound of the target inspection area, the thermal infrared sensor is used to obtain the inspection device temperature distribution of the target inspection device, and the gas sensor is used to monitor the inspection area gas concentration of the target inspection area, specifically including: Obtaining a distance between the optical sensor and the target inspection device; if the distance is greater than or equal to a focus adjustment critical value, activating a long-distance wide-angle mode of the optical sensor to capture an overall inspection device image of the target inspection device; if the distance is less than the focus adjustment critical value, activating a short-distance telephoto mode of the optical sensor to capture a detailed inspection device image of the target inspection device; and summarizing the overall inspection device image and the detailed inspection device image to generate an appearance image of the inspection device; Based on the array design of the acoustic sensor, the sound of the inspection area of ​​the target inspection area is collected through beamforming technology and the source direction of the sound of the inspection area is located; The thermal infrared sensor collects a low-resolution thermal infrared image of the target inspection device, converts the low-resolution thermal infrared image into a high-resolution thermal infrared image using a super-resolution reconstruction algorithm, and quantifies the high-resolution thermal infrared image to output the temperature distribution of the inspection device; Based on the micro-electromechanical system technology of the gas sensor, it responds to and identifies each target gas component in the target inspection area, and calculates the gas concentration corresponding to all the target gas components to generate the gas concentration of the inspection area.

[0036] In actual applications, optical sensors focus on capturing images of the target inspection equipment's appearance and optimizing the imaging effect through a distance-adaptive adjustment mechanism. Specifically, the distance between the optical sensor and the target inspection equipment can be acquired in real time. When the distance is far away, it automatically switches to long-distance wide-angle mode, capturing the entire shape of the equipment. For example, a panoramic image of the transformer in a substation can be taken, clearly showing the overall layout and external structure of the equipment. When the distance is close, the short-range telephoto mode is activated to focus on key details of the equipment, such as capturing the position of the instrument pointer of the switch cabinet and the status of the line connector. By combining the overall image and the detailed image, a complete image of the inspection equipment's appearance is formed, providing comprehensive visual data for subsequent defect identification.

[0037] The acoustic sensor utilizes an array-like design and beamforming technology to achieve directional acquisition of sound signals and localize their source. In complex industrial environments, this technology effectively focuses on the operating sounds of target inspection equipment while suppressing interfering ambient noise. For example, during workshop inspections, it can accurately capture the operating sounds of equipment such as motors and pumps while filtering out irrelevant noise such as human movement and mechanical vibration. By analyzing the spatial orientation of the sound signal, it can also pinpoint the source of abnormal sounds, providing acoustic evidence for locating equipment faults.

[0038] Thermal infrared sensors use temperature imaging technology to obtain equipment temperature distribution data. They first capture low-resolution thermal infrared images, then employ super-resolution reconstruction algorithms to enhance image clarity, making it easier to discern temperature details on the equipment surface. For example, they can clearly visualize temperature differences between connectors within a power distribution cabinet. The reconstructed high-resolution images are then subjected to temperature quantification, converting the image's grayscale information into specific temperature values. Ultimately, the sensors output quantifiable temperature distribution data for the inspected equipment, visually reflecting the thermal status of the equipment.

[0039] Gas sensors, based on micro-electromechanical systems (MEMS) technology, enable high-precision gas monitoring. They offer rapid response capabilities and can accurately identify various gas components within target areas in complex environments, such as volatile gases in chemical parks and insulating gases in substations. By statistically analyzing the concentration of each identified gas component, comprehensive gas concentration data for the inspection area is generated. This not only monitors gas concentrations exceeding standards in real time, but also identifies equipment sealing status and potential leak risks based on concentration trends, providing chemical-level data support for environmental safety and equipment status assessments.

[0040] The method automatically adjusts the image enhancement parameters of the inspection equipment appearance image based on the adaptive enhancement algorithm, highlights the inspection equipment operation sound in the inspection area sound based on the adaptive filtering algorithm, adaptively corrects the inspection equipment temperature distribution based on the inspection equipment heat conduction model, and compensates the gas concentration in the inspection area in real time based on the Kalman filtering algorithm, specifically including: In the adaptive enhancement algorithm based on the generative adversarial network, the generator generates an enhanced image of the inspection equipment based on the input appearance image of the inspection equipment, and the discriminator optimizes the enhancement effect of the enhanced image of the inspection equipment through adversarial training; In the adaptive filtering algorithm based on variational mode decomposition, different frequency components in the inspection area sound are separated according to the intrinsic mode function, and the frequency components within the preset operating sound frequency range are extracted to generate the inspection equipment operating sound and perform filtering processing; Establishing a heat conduction model for the inspection equipment and correcting the temperature distribution of the inspection equipment based on the heat dissipation of the target inspection equipment; The Kalman filter algorithm is used to dynamically compensate the gas concentration in the inspection area in combination with the real-time environmental parameters of the target inspection area, including real-time temperature, real-time humidity and real-time air pressure.

[0041] An adaptive enhancement algorithm based on a generative adversarial network is used to process images of the inspection equipment's exterior captured by optical sensors. This algorithm comprises two core modules: a generator and a discriminator. The generator receives the original image and generates a preliminary enhanced image based on the image's lighting conditions, contrast, and other characteristics. The discriminator compares the preliminary enhanced image with a library of clear samples, evaluates the enhancement effect, and provides feedback to the generator. Continuous adversarial training continuously optimizes the enhancement parameters. For example, in a dimly lit factory environment, the algorithm can automatically enhance image details, making even minor surface defects like cracks and rust visible, effectively addressing the issue of poor image quality under complex lighting conditions.

[0042] An adaptive filtering algorithm based on variational modal decomposition (VMD) is used to extract valid signals from the inspection area sounds captured by acoustic sensors. This algorithm decomposes the mixed sound signal into distinct frequency components using intrinsic mode functions (IMFs), accurately identifying the characteristic frequency bands of equipment operating sounds, such as those of motor operation and pump vibration. By filtering out ambient noise frequency components outside a preset range, such as background noise in the workshop and sounds of personnel, the algorithm highlights the valid characteristics of the operating sounds of the inspection equipment, laying the foundation for subsequent sound feature analysis.

[0043] Adaptive correction is achieved by building a heat conduction model for inspection equipment based on the temperature distribution data captured by thermal infrared sensors. This model incorporates the target equipment's material properties, heat dissipation structure, and real-time heat dissipation to eliminate interference from ambient temperature fluctuations on measurement results. For example, during outdoor substation inspections, this model can correct for the impact of ambient temperature differences on transformer surface temperature measurements, accurately identifying true hot spots caused by poor contact, overload, and other factors, and ensuring that temperature data accurately reflects the equipment's operating status.

[0044] A Kalman filter algorithm dynamically compensates for gas concentration data in inspection areas monitored by gas sensors. This algorithm uses real-time environmental parameters, including temperature, humidity, and air pressure, as auxiliary variables. By establishing a correlation model between gas concentration and environmental factors, it corrects errors in the original measured values. For example, in a chemical plant with drastic temperature and humidity fluctuations, this algorithm can effectively smooth out fluctuating errors in gas concentration measurements, accurately capturing subtle changes in gas concentration caused by equipment leaks, and ensuring the stability and reliability of gas concentration data.

[0045] The method extracts the appearance defect features of the inspection equipment from the appearance image of the inspection equipment based on a convolutional neural network, captures the sound features of the inspection equipment in the operating sound of the inspection equipment by combining a deep belief network with a recurrent neural network, extracts the temperature features of the inspection equipment from the temperature distribution of the inspection equipment using a manifold learning method, and learns the gas concentration features of the inspection area corresponding to the gas concentration in the inspection area based on a time series model. Specifically, the method includes: Constructing a convolutional neural network based on an attention mechanism, and introducing an attention module into the convolutional neural network to automatically identify the appearance defect features of the inspection equipment on the surface of the target inspection equipment; The inspection equipment operating sound is subjected to feature learning through a deep belief network to generate the inspection equipment sound spectrum features, and a recurrent neural network is used to process the time series information of the inspection equipment sound spectrum features and capture dynamic change patterns, thereby extracting the inspection equipment sound features from the inspection equipment operating sound; Using a manifold learning method based on principal component analysis and local linear embedding, the high-dimensional temperature distribution data in the temperature distribution of the inspection equipment is projected into a low-dimensional space to generate the temperature features of the inspection equipment; A time series model based on a long short-term memory network is constructed to learn the long-term and short-term variation trends of the gas concentration in the inspection area, and output the concentration statistical characteristics of the gas concentration in the inspection area, that is, the gas concentration characteristics of the inspection area.

[0046] It's understandable that a convolutional neural network based on an attention mechanism is constructed for feature extraction from images of the inspection equipment's appearance. This network, by introducing an attention module into the traditional convolutional architecture, automatically focuses on key areas of the equipment's surface, specifically capturing defect-related features. For example, during power equipment inspections, it can accurately identify subtle defects such as cracks on insulator surfaces and rust marks on switchgear. By enhancing the characteristic response of defective areas and weakening background interference, it significantly improves the accuracy of identifying appearance defect features, providing a reliable basis for evaluating the equipment's appearance condition.

[0047] For the sound signals from inspection equipment operation, a combined model of a deep belief network and a recurrent neural network is used to extract sound features. The deep belief network performs deep learning on the sound signals through multi-layer nonlinear transformations, generating sound spectrum features containing information such as frequency and amplitude. The recurrent neural network further processes the time series information of these spectrum features to capture the dynamic changes in the equipment's operating sound over time. For example, during motor inspections, it can effectively extract abnormal sound patterns caused by bearing wear and periodic noise characteristics caused by poor gear meshing, thereby characterizing the sound characteristics of mechanical equipment failures.

[0048] A manifold learning method based on principal component analysis and local linear embedding is used to extract temperature features from inspection equipment temperature distribution data. This method first reduces the dimensionality of high-dimensional temperature data through principal component analysis, preserving key temperature variation information. Local linear embedding is then used to preserve the local geometric structure of the data and map the high-dimensional temperature distribution data into a low-dimensional space. For example, during inspections of power distribution cabinets, abnormal temperature features such as connector overheating and line overload can be accurately extracted, clearly demonstrating the thermal distribution patterns of the equipment and providing feature support for overheating fault diagnosis.

[0049] For gas concentration data in inspection areas, a time series model based on a long short-term memory network is constructed for feature learning. This model effectively captures the temporal evolution of gas concentrations, learning both long-term concentration trends, such as the natural diffusion of ambient gases, and short-term concentration fluctuations, such as sudden concentration changes caused by equipment leaks. For example, during chemical park inspections, features such as abnormal upward trends and periodic fluctuations in gas concentration can be extracted to characterize the sealing status of equipment and potential leakage risks, providing key feature input for gas safety monitoring.

[0050] The spatiotemporal fusion of the inspection equipment appearance defect features, the inspection equipment sound features, the inspection equipment temperature features, and the inspection area gas concentration features to generate spatiotemporal fusion inspection features specifically includes: Constructing a Bayesian network based on the probability distribution and conditional probability relationship of the inspection equipment appearance defect characteristics, the inspection equipment sound characteristics, the inspection equipment temperature characteristics, and the inspection area gas concentration characteristics; Based on the Bayesian network, the characteristic causal relationship between the inspection equipment appearance defect characteristics, the inspection equipment sound characteristics, the inspection equipment temperature characteristics and the inspection area gas concentration characteristics is analyzed, and the feature fusion weights corresponding to the inspection equipment appearance defect characteristics, the inspection equipment sound characteristics, the inspection equipment temperature characteristics and the inspection area gas concentration characteristics are determined respectively; Based on the feature fusion weights corresponding to the appearance defect features of the inspection equipment, the sound features of the inspection equipment, the temperature features of the inspection equipment, and the gas concentration features of the inspection area, a spatiotemporal graph convolutional network is constructed in the time and space dimensions; The spatiotemporal graph convolutional network is used to learn and establish the spatiotemporal correlation between the appearance defect characteristics of the inspection equipment, the sound characteristics of the inspection equipment, the temperature characteristics of the inspection equipment and the gas concentration characteristics of the inspection area, and output the spatiotemporal fusion inspection characteristics.

[0051] Establish a multi-source data fusion algorithm based on quantum computing, map the spatiotemporal fusion inspection features of all the target inspection devices into quantum state space, and use the superposition and entanglement properties of quantum bits to search and match potential correlations between the spatiotemporal fusion inspection features; Blockchain technology is introduced to record and store the matched spatiotemporal fusion inspection features.

[0052] First, we can build a Bayesian network based on the probability distribution of inspection equipment appearance defect characteristics, inspection equipment sound characteristics, inspection equipment temperature characteristics, and inspection area gas concentration characteristics, combining the conditional probability relationship between these characteristics. This network quantifies the probability relationship between different features and establishes a probabilistic inference model between these features.

[0053] Based on the constructed Bayesian network, we further analyze the causal relationships between various features and determine the fusion weights for different features. By evaluating the strength of the features indicative of equipment fault conditions, we assign higher weights to key features. For example, during power transformer inspections, the feature of abnormally high winding temperature contributes significantly to fault diagnosis, and its fusion weight is increased accordingly.

[0054] After determining the feature fusion weights, a spatiotemporal graph convolutional network is constructed across both time and space. The temporal dimension correlates the feature change sequences of the same device at different inspection times, while the spatial dimension connects the feature distribution relationships between different devices or different parts of the same device. For example, in a smart factory workshop inspection, the multi-source features of devices such as motors, conveyor belts, and control cabinets on the same production line can be spatially correlated, while the feature change trajectories at different time periods are connected in series to form a complete spatiotemporal feature network structure.

[0055] Through layer-by-layer operations on a spatiotemporal graph convolutional network, the spatiotemporal correlations between features are deeply explored. The network captures spatial feature dependencies through graph convolution operations and learns the evolution of features in the temporal dimension through sequential convolution layers. Ultimately, it outputs spatiotemporal fusion inspection features that integrate multi-dimensional information. For example, the appearance of surface discharge marks on high-voltage switchgear, the acoustic characteristics of abnormal buzzing sounds, the temperature characteristics of overheated connectors, and the gas characteristics of abnormal sulfur hexafluoride concentrations are fused into a comprehensive feature vector to fully reflect the insulation status of the equipment.

[0056] To further improve fusion efficiency and association mining capabilities, a multi-source data fusion algorithm based on quantum computing was established. This algorithm maps the spatiotemporal fusion features of all target inspection devices into quantum state space. By leveraging the superposition and entanglement of qubits, parallel computing is implemented to rapidly search for potential correlation patterns between the features of different devices. For example, similar fault features of similar devices in different regions can be correlated. Furthermore, blockchain technology is introduced to chain-store the matched fusion features, ensuring the integrity and traceability of the feature data through timestamps and encryption mechanisms.

[0057] Outputting inspection equipment fault diagnosis results based on the spatiotemporal fusion inspection features of all the target inspection equipment in the target inspection area and automatically adjusting the current inspection path of the inspection robot specifically include: Using a multi-task neural network based on transfer learning, fault diagnosis is performed on the spatiotemporal fusion inspection features of all target inspection devices, and the inspection device fault diagnosis results are output; Obtaining a three-dimensional inspection area map corresponding to the target inspection area, and automatically adjusting the current inspection path using a reinforcement learning-based path planning algorithm based on the inspection equipment fault diagnosis result and the three-dimensional inspection area map, wherein a secondary inspection path needs to be added for the target inspection equipment with a fault; Using virtual reality and augmented reality technologies, the fault diagnosis results of the inspection equipment are displayed to the operation and maintenance end; Obtain a knowledge base of historical faults of inspection equipment, establish an inspection knowledge graph, and associate and integrate the inspection equipment fault diagnosis results with the knowledge base of historical faults of inspection equipment.

[0058] In practical applications, a multi-task neural network based on transfer learning can be used to analyze spatiotemporal fusion inspection features and output fault diagnosis results for the inspection equipment. Leveraging model parameters pre-trained in similar equipment inspection scenarios, this network rapidly adapts to the current inspection task. Through multi-task learning, it simultaneously identifies the presence of equipment faults, fault type, and severity. For example, during substation inspections, the spatiotemporal fusion features of transformers can be analyzed to accurately diagnose specific fault types, such as winding overheating and bushing damage, addressing diagnostic accuracy issues caused by insufficient training data in new scenarios.

[0059] Based on the diagnostic results and a three-dimensional map of the target inspection area, a reinforcement learning path planning algorithm dynamically adjusts the current inspection route. The three-dimensional inspection area map includes information about equipment location, terrain distribution, and obstacles. The algorithm prioritizes the faulty equipment and plans an optimal secondary inspection route based on the original path. For example, if an anomaly is detected in a power distribution cabinet, the algorithm automatically generates a secondary inspection route that bypasses the obstacle and reaches the target equipment directly. This ensures targeted and detailed inspection of the faulty area, improving inspection efficiency and fault confirmation accuracy.

[0060] The O&M end here refers to the O&M personnel. To help them intuitively understand the equipment status, virtual reality and augmented reality technologies are used to visualize fault diagnosis results. Virtual reality technology creates a three-dimensional virtual scene of the inspection area, allowing O&M personnel to immersively view the location, distribution, and status of faulty equipment. Augmented reality technology overlays fault identification and diagnostic results onto real-time on-site footage. For example, during an on-site inspection, O&M personnel can directly see temperature anomaly area markings and fault cause analysis superimposed on the equipment surface through AR glasses, providing intuitive reference for O&M decision-making.

[0061] At the same time, by accessing the historical fault knowledge base of the inspection equipment, the current fault diagnosis results are correlated and integrated with historical data to construct an inspection knowledge graph. This inspection knowledge graph contains relevant information such as fault type, characteristic manifestations, and treatment solutions. For example, the current fault characteristics of abnormal vibration and temperature rise in the motor are correlated with historical similar fault cases to automatically match maintenance solutions with spare parts requirements.

[0062] like Figure 2 FIG. 1 is a system block diagram of a robot intelligent inspection system based on multi-source data fusion according to an embodiment of the present invention. The system includes: a data acquisition module, configured to perform all-round data acquisition of a target inspection area based on an inspection robot equipped with a multi-source sensor array, wherein the multi-source sensor array includes an optical sensor, an acoustic sensor, a thermal infrared sensor, and a gas sensor; the optical sensor is configured to collect an inspection device appearance image of a target inspection device in the target inspection area; the acoustic sensor is configured to capture inspection area sounds of the target inspection area; the thermal infrared sensor is configured to obtain an inspection device temperature distribution of the target inspection device; and the gas sensor is configured to monitor inspection area gas concentration in the target inspection area; a data preprocessing module for automatically adjusting image enhancement parameters of the inspection equipment's appearance image based on an adaptive enhancement algorithm, highlighting the inspection equipment's operating sounds in the inspection area's sounds based on an adaptive filtering algorithm, adaptively correcting the inspection equipment's temperature distribution based on a thermal conduction model of the inspection equipment, and compensating for gas concentration in the inspection area in real time based on a Kalman filtering algorithm; a feature extraction module for extracting appearance defect features of the inspection equipment from the appearance image of the inspection equipment based on a convolutional neural network, capturing the inspection equipment sound features from the operating sound of the inspection equipment by combining a deep belief network with a recurrent neural network, extracting the inspection equipment temperature features from the temperature distribution of the inspection equipment using a manifold learning method, and learning the inspection area gas concentration features corresponding to the inspection area gas concentration based on a time series model; A feature fusion module is used to perform spatiotemporal fusion of the inspection equipment appearance defect features, the inspection equipment sound features, the inspection equipment temperature features, and the inspection area gas concentration features to generate spatiotemporal fusion inspection features; The path adjustment module is used to output the inspection equipment fault diagnosis result according to the spatiotemporal fusion inspection characteristics of all the target inspection equipment in the target inspection area, and automatically adjust the current inspection path of the inspection robot.

[0063] Figure 2 The apparatus of the embodiment shown can be used to perform Figure 1 The implementation principles and technical effects of the steps in the method embodiment shown are similar and will not be repeated here.

[0064] An electronic device includes a memory and a processor, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the steps of the robot intelligent inspection method based on multi-source data fusion as described above.

[0065] like Figure 3 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. The electronic device 30 includes: a processor 31, a memory 32 and a computer program; The memory 32 is used to store the computer program, which may also be a flash memory. The computer program is, for example, an application program or a functional module for implementing the above method.

[0066] The processor 31 is configured to execute the computer program stored in the memory to implement the various steps performed by the device in the above method. For details, please refer to the relevant description in the above method embodiment.

[0067] Optionally, the memory 32 may be independent or integrated with the processor 31 .

[0068] When the memory 32 is a device independent of the processor 31, the device may further include: The bus 33 is used to connect the memory 32 and the processor 31 .

[0069] A readable storage medium stores a computer program, which, when executed by a processor, is used to implement the steps of the robot intelligent inspection method based on multi-source data fusion as described above.

[0070] The readable storage medium may be a computer storage medium or a communication medium. Communication media include any medium that facilitates the transfer of computer programs from one location to another. Computer storage media may be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium may also be an integral part of the processor. The processor and the readable storage medium may be located in an application-specific integrated circuit (ASIC). In addition, the ASIC may be located in a user device. Of course, the processor and the readable storage medium may also exist as discrete components in a communication device. The readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0071] The present invention also provides a program product, which includes execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and at least one processor executes the execution instructions so that the device implements the methods provided in the various embodiments described above.

[0072] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0073] Through the introduction of the above embodiments, the present invention can collect all-round data of the target inspection area based on the inspection robot equipped with a multi-source sensor array through the robot intelligent inspection method and system based on multi-source data fusion, wherein the multi-source sensor array includes optical sensors, acoustic sensors, thermal infrared sensors and gas sensors, the optical sensor is used to collect the inspection equipment appearance image of the target inspection equipment in the target inspection area, the acoustic sensor is used to capture the inspection area sound of the target inspection area, the thermal infrared sensor is used to obtain the inspection equipment temperature distribution of the target inspection equipment, and the gas sensor is used to monitor the inspection area gas concentration of the target inspection area; based on the adaptive enhancement algorithm, the image enhancement parameters of the inspection equipment appearance image are automatically adjusted, based on the adaptive filtering algorithm, the inspection equipment operation sound in the inspection area sound is highlighted, based on the patrol equipment thermal conduction model, the patrol equipment temperature distribution is adaptively corrected, and based on the Kalman filtering algorithm, the patrol area gas concentration is compensated in real time; based on The convolutional neural network is used to extract the appearance defect features of the inspection equipment from the appearance image of the inspection equipment, and the deep belief network and the recurrent neural network are combined to capture the inspection equipment sound features in the operation sound of the inspection equipment. The manifold learning method is used to extract the inspection equipment temperature features in the temperature distribution of the inspection equipment, and the inspection area gas concentration features corresponding to the gas concentration in the inspection area are learned based on the time series model; the inspection equipment appearance defect features, inspection equipment sound features, inspection equipment temperature features and inspection area gas concentration features are spatiotemporally fused to generate spatiotemporal fused inspection features; based on the spatiotemporal fused inspection features of all target inspection equipment in the target inspection area, the inspection equipment fault diagnosis results are output, and the current inspection path of the inspection robot is automatically adjusted, so that multi-source sensors can collaboratively collect multi-dimensional information of the equipment, optimize data quality through adaptive processing, mine related features through spatiotemporal fusion, and dynamically adjust the inspection path in combination with intelligent decision-making, thereby improving the reliability of fault diagnosis and ensuring safe and stable operation of the equipment.

[0074] The present invention adopts multi-source data fusion and intelligent analysis technology. Through the collaborative work of multiple types of sensors, it synchronously collects multi-dimensional information such as equipment appearance characteristics, operating sounds, surface temperature and environmental gases. It breaks through the limitations of traditional single data detection, realizes all-round perception of the equipment's operating status, ensures that no potential abnormal signals are missed, and significantly improves the comprehensiveness and reliability of industrial equipment inspections. The present invention uses an adaptive data preprocessing mechanism to dynamically optimize data quality according to different environmental characteristics. In scenarios such as complex lighting, noise interference, temperature and humidity fluctuations, it can still effectively retain key features and avoid environmental factors from interfering with the detection results. The present invention can accurately capture effective information from various types of data and fully explore characteristics such as subtle defects, abnormal sounds, temperature anomalies and gas changes in equipment. The multi-source data fusion technology of the present invention establishes a spatiotemporal correlation model, deeply analyzes the inherent connections between different types of data, integrates scattered feature information into a complete description of the equipment status, and transforms fault judgment from one-sided speculation to comprehensive analysis, greatly improving the accuracy of fault diagnosis and reducing misjudgments and missed judgments. The intelligent decision-making and path adjustment mechanism of this invention achieves dynamic optimization of the inspection process, can flexibly adjust inspection priorities based on the actual status of the equipment, conduct targeted re-sampling of abnormal areas, and improve the utilization efficiency of inspection resources. By applying knowledge graphs and visualization technologies, this invention makes inspection results more intuitive and easy to understand, providing clear decision-making references for operation and maintenance personnel, helping to quickly develop treatment plans, and effectively ensuring the safe and stable operation of industrial equipment.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot intelligent inspection method based on multi-source data fusion, characterized in that: The method comprises: Based on an inspection robot equipped with a multi-source sensor array, all-round data collection is performed on the target inspection area, wherein the multi-source sensor array includes an optical sensor, an acoustic sensor, a thermal infrared sensor and a gas sensor. The optical sensor is used to collect an inspection device appearance image of the target inspection device in the target inspection area, the acoustic sensor is used to capture the inspection area sound of the target inspection area, the thermal infrared sensor is used to obtain the inspection device temperature distribution of the target inspection device, and the gas sensor is used to monitor the inspection area gas concentration of the target inspection area; Automatically adjust the image enhancement parameters of the inspection equipment appearance image based on an adaptive enhancement algorithm, highlight the inspection equipment operation sound in the inspection area sound based on an adaptive filtering algorithm, adaptively correct the inspection equipment temperature distribution based on the inspection equipment heat conduction model, and compensate for the gas concentration in the inspection area in real time based on a Kalman filtering algorithm; Based on the convolutional neural network, the inspection equipment appearance defect features in the inspection equipment appearance image are extracted. The inspection equipment sound features in the operation sound of the inspection equipment are captured by combining the deep belief network and the recurrent neural network. The inspection equipment temperature features in the temperature distribution of the inspection equipment are extracted using the manifold learning method. The inspection area gas concentration features corresponding to the gas concentration in the inspection area are learned based on the time series model. Performing spatiotemporal fusion on the appearance defect features of the inspection equipment, the sound features of the inspection equipment, the temperature features of the inspection equipment, and the gas concentration features of the inspection area to generate spatiotemporal fusion inspection features; According to the spatiotemporal fusion inspection features of all the target inspection devices in the target inspection area, an inspection device fault diagnosis result is output, and the current inspection path of the inspection robot is automatically adjusted.

2. The robot intelligent inspection method based on multi-source data fusion according to claim 1 is characterized in that: The optical sensor is used to collect an inspection device appearance image of a target inspection device in the target inspection area, the acoustic sensor is used to capture the inspection area sound of the target inspection area, the thermal infrared sensor is used to obtain the inspection device temperature distribution of the target inspection device, and the gas sensor is used to monitor the inspection area gas concentration of the target inspection area. Specifically, it includes: Obtaining a distance between the optical sensor and the target inspection device; if the distance is greater than or equal to a focus adjustment critical value, activating a long-distance wide-angle mode of the optical sensor to capture an overall inspection device image of the target inspection device; if the distance is less than the focus adjustment critical value, activating a short-distance telephoto mode of the optical sensor to capture a detailed inspection device image of the target inspection device; and summarizing the overall inspection device image and the detailed inspection device image to generate an appearance image of the inspection device; Based on the array design of the acoustic sensor, the sound of the inspection area of ​​the target inspection area is collected through beamforming technology and the source direction of the sound of the inspection area is located; The thermal infrared sensor collects a low-resolution thermal infrared image of the target inspection device, converts the low-resolution thermal infrared image into a high-resolution thermal infrared image using a super-resolution reconstruction algorithm, and quantifies the high-resolution thermal infrared image to output the temperature distribution of the inspection device; Based on the micro-electromechanical system technology of the gas sensor, it responds to and identifies each target gas component in the target inspection area, and calculates the gas concentration corresponding to all the target gas components to generate the gas concentration of the inspection area.

3. The robot intelligent inspection method based on multi-source data fusion according to claim 1 is characterized in that: The method automatically adjusts the image enhancement parameters of the inspection equipment appearance image based on the adaptive enhancement algorithm, highlights the inspection equipment operation sound in the inspection area sound based on the adaptive filtering algorithm, adaptively corrects the inspection equipment temperature distribution based on the inspection equipment heat conduction model, and compensates the gas concentration in the inspection area in real time based on the Kalman filtering algorithm, specifically including: In the adaptive enhancement algorithm based on the generative adversarial network, the generator generates an enhanced image of the inspection equipment based on the input appearance image of the inspection equipment, and the discriminator optimizes the enhancement effect of the enhanced image of the inspection equipment through adversarial training; In the adaptive filtering algorithm based on variational mode decomposition, different frequency components in the inspection area sound are separated according to the intrinsic mode function, and the frequency components within the preset operating sound frequency range are extracted to generate the inspection equipment operating sound and perform filtering processing; Establishing a heat conduction model for the inspection equipment and correcting the temperature distribution of the inspection equipment based on the heat dissipation of the target inspection equipment; The Kalman filter algorithm is used to dynamically compensate the gas concentration in the inspection area in combination with the real-time environmental parameters of the target inspection area, including real-time temperature, real-time humidity and real-time air pressure.

4. The robot intelligent inspection method based on multi-source data fusion according to claim 1 is characterized in that: The method extracts the appearance defect features of the inspection equipment from the appearance image of the inspection equipment based on a convolutional neural network, captures the sound features of the inspection equipment in the operating sound of the inspection equipment by combining a deep belief network with a recurrent neural network, extracts the temperature features of the inspection equipment from the temperature distribution of the inspection equipment using a manifold learning method, and learns the gas concentration features of the inspection area corresponding to the gas concentration in the inspection area based on a time series model. Specifically, the method includes: Constructing a convolutional neural network based on an attention mechanism, and introducing an attention module into the convolutional neural network to automatically identify the appearance defect features of the inspection equipment on the surface of the target inspection equipment; The inspection equipment operating sound is subjected to feature learning through a deep belief network to generate the inspection equipment sound spectrum features, and a recurrent neural network is used to process the time series information of the inspection equipment sound spectrum features and capture dynamic change patterns, thereby extracting the inspection equipment sound features from the inspection equipment operating sound; Using a manifold learning method based on principal component analysis and local linear embedding, the high-dimensional temperature distribution data in the temperature distribution of the inspection equipment is projected into a low-dimensional space to generate the temperature features of the inspection equipment; A time series model based on a long short-term memory network is constructed to learn the long-term and short-term variation trends of the gas concentration in the inspection area, and output the concentration statistical characteristics of the gas concentration in the inspection area, that is, the gas concentration characteristics of the inspection area.

5. The robot intelligent inspection method based on multi-source data fusion according to claim 1 is characterized in that: The spatiotemporal fusion of the inspection equipment appearance defect features, the inspection equipment sound features, the inspection equipment temperature features, and the inspection area gas concentration features to generate spatiotemporal fusion inspection features specifically includes: Constructing a Bayesian network based on the probability distribution and conditional probability relationship of the inspection equipment appearance defect characteristics, the inspection equipment sound characteristics, the inspection equipment temperature characteristics, and the inspection area gas concentration characteristics; Based on the Bayesian network, the characteristic causal relationship between the inspection equipment appearance defect characteristics, the inspection equipment sound characteristics, the inspection equipment temperature characteristics and the inspection area gas concentration characteristics is analyzed, and the feature fusion weights corresponding to the inspection equipment appearance defect characteristics, the inspection equipment sound characteristics, the inspection equipment temperature characteristics and the inspection area gas concentration characteristics are determined respectively; Based on the feature fusion weights corresponding to the appearance defect features of the inspection equipment, the sound features of the inspection equipment, the temperature features of the inspection equipment, and the gas concentration features of the inspection area, a spatiotemporal graph convolutional network is constructed in the time and space dimensions; The spatiotemporal graph convolutional network is used to learn and establish the spatiotemporal correlation between the appearance defect characteristics of the inspection equipment, the sound characteristics of the inspection equipment, the temperature characteristics of the inspection equipment and the gas concentration characteristics of the inspection area, and output the spatiotemporal fusion inspection characteristics.

6. The robot intelligent inspection method based on multi-source data fusion according to claim 5 is characterized in that: Establish a multi-source data fusion algorithm based on quantum computing, map the spatiotemporal fusion inspection features of all the target inspection devices into quantum state space, and use the superposition and entanglement properties of quantum bits to search and match potential correlations between the spatiotemporal fusion inspection features; Blockchain technology is introduced to record and store the matched spatiotemporal fusion inspection features.

7. The robot intelligent inspection method based on multi-source data fusion according to claim 1 is characterized in that: Outputting inspection equipment fault diagnosis results based on the spatiotemporal fusion inspection features of all the target inspection equipment in the target inspection area and automatically adjusting the current inspection path of the inspection robot specifically include: Using a multi-task neural network based on transfer learning, fault diagnosis is performed on the spatiotemporal fusion inspection features of all target inspection devices, and the inspection device fault diagnosis results are output; Obtaining a three-dimensional inspection area map corresponding to the target inspection area, and automatically adjusting the current inspection path using a reinforcement learning-based path planning algorithm based on the inspection equipment fault diagnosis result and the three-dimensional inspection area map, wherein a secondary inspection path needs to be added for the target inspection equipment with a fault; Using virtual reality and augmented reality technologies, the fault diagnosis results of the inspection equipment are displayed to the operation and maintenance end; Obtain a knowledge base of historical faults of inspection equipment, establish an inspection knowledge graph, and associate and integrate the inspection equipment fault diagnosis results with the knowledge base of historical faults of inspection equipment.

8. A robot intelligent inspection system based on multi-source data fusion, applied to the robot intelligent inspection method based on multi-source data fusion according to any one of claims 1 to 7, characterized in that: The system comprises: a data acquisition module, configured to perform all-round data acquisition of a target inspection area based on an inspection robot equipped with a multi-source sensor array, wherein the multi-source sensor array includes an optical sensor, an acoustic sensor, a thermal infrared sensor, and a gas sensor; the optical sensor is configured to collect an inspection device appearance image of a target inspection device in the target inspection area; the acoustic sensor is configured to capture inspection area sounds of the target inspection area; the thermal infrared sensor is configured to obtain an inspection device temperature distribution of the target inspection device; and the gas sensor is configured to monitor inspection area gas concentration in the target inspection area; a data preprocessing module for automatically adjusting image enhancement parameters of the inspection equipment's appearance image based on an adaptive enhancement algorithm, highlighting the inspection equipment's operating sounds in the inspection area's sounds based on an adaptive filtering algorithm, adaptively correcting the inspection equipment's temperature distribution based on a thermal conduction model of the inspection equipment, and compensating for gas concentration in the inspection area in real time based on a Kalman filtering algorithm; a feature extraction module for extracting appearance defect features of the inspection equipment from the appearance image of the inspection equipment based on a convolutional neural network, capturing the inspection equipment sound features from the operating sound of the inspection equipment by combining a deep belief network with a recurrent neural network, extracting the inspection equipment temperature features from the temperature distribution of the inspection equipment using a manifold learning method, and learning the inspection area gas concentration features corresponding to the inspection area gas concentration based on a time series model; A feature fusion module is used to perform spatiotemporal fusion of the inspection equipment appearance defect features, the inspection equipment sound features, the inspection equipment temperature features, and the inspection area gas concentration features to generate spatiotemporal fusion inspection features; The path adjustment module is used to output the inspection equipment fault diagnosis result according to the spatiotemporal fusion inspection characteristics of all the target inspection equipment in the target inspection area, and automatically adjust the current inspection path of the inspection robot.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor runs the computer program stored in the memory, the processor executes the steps of the robot intelligent inspection method based on multi-source data fusion according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, it is used to implement the steps of the robot intelligent inspection method based on multi-source data fusion as described in any one of claims 1 to 7.

Images