Multi-modal perception-based AI (artificial intelligence) substitute inspection method and system

Abstract

The invention discloses an AI substitute inspection method and system based on multi-modal perception, and relates to the technical field of intelligent detection. According to the method, images, audios, environmental parameters and equipment state data of an inspection area are synchronously collected through a multi-modal sensor array, and after space-time alignment preprocessing, multi-modal features are extracted and input into a pre-trained fusion model to generate unified feature representation. The fusion model adopts a Transform architecture to realize cross-modal feature interaction, an anomaly detection algorithm is used to identify and classify anomalies, a confidence coefficient is calculated in combination with an evidence theory, and finally an inspection report containing anomaly positions, types and processing suggestions is generated. According to the invention, the information limitation of single-mode inspection is solved, the detection accuracy and robustness in a complex environment are improved, unmanned inspection in a high-risk scene is realized, the labor cost is reduced, the safety is guaranteed, and the method is suitable for automatic inspection in the fields of industrial equipment, public facilities and the like.

Description

Technical Field

[0001] This application relates to the field of equipment inspection technology, specifically to an AI-based human-assisted inspection method based on multimodal perception. Background Technology

[0002] In the complex operating environments of industrial equipment, power facilities, and energy storage systems, inspection, as a crucial link in ensuring safe and stable operation, has gradually evolved from traditional manual methods towards automation and intelligence. Current mainstream inspection methods mainly fall into three categories: first, periodic on-site inspections relying on manual methods, such as visual inspection, temperature guns, and listening rods to determine equipment status; second, single-modal automatic monitoring systems, such as visual defect recognition based on visible light images, infrared thermal imaging temperature monitoring, vibration sensor fault diagnosis, or acoustic signal analysis systems; and third, preliminary multi-sensor fusion solutions, which integrate and analyze data from different sources in parallel on a central platform to aid decision-making. With the development of artificial intelligence and edge computing, some systems have introduced machine learning models to classify and identify collected data, achieving a certain degree of automated discrimination in specific scenarios and improving local detection efficiency.

[0003] However, the aforementioned existing technologies still have significant limitations: on the one hand, manual inspections pose personal safety risks in high-pressure, high-temperature, toxic gas, or structurally confined areas, and are heavily influenced by subjective experience, resulting in a high rate of missed detections; on the other hand, single-modal monitoring, due to its limited information dimensions, struggles to cope with complex operating conditions—for example, visual failure in insufficient lighting, inaccurate acoustic detection in high background noise, and failure of thermal imagers to issue warnings in the early stages of latent faults when temperature changes are not significant. Furthermore, multi-source heterogeneous data generally lack effective spatiotemporal alignment mechanisms; inconsistent sampling frequencies and non-uniform spatial coordinates among sensors lead to inaccurate matching of cross-modal features, affecting fusion results. Simultaneously, existing anomaly assessments largely rely on black-box deep learning outputs, lacking interpretability, resulting in a high false alarm rate, and the alarm results are disconnected from the original data, hindering verification and tracing. Therefore, there is an urgent need for a new intelligent inspection method that can achieve multimodal data collaborative perception, has high-precision spatiotemporal alignment capability, interpretable fusion process, and supports quantitative assessment of confidence, so as to improve detection accuracy, environmental adaptability and operation and maintenance safety, and meet the needs of unmanned, all-weather intelligent operation and maintenance in high-risk and complex environments. Summary of the Invention

[0004] The technical problem to be solved by this application is to overcome the shortcomings of the above-mentioned technologies and provide an AI-based method for human-assisted inspection based on multimodal perception.

[0005] To address the aforementioned technical problems, firstly, this application provides a technical solution for an AI-assisted human inspection method based on multimodal perception, comprising: A multimodal sensing system integrating visible light, infrared, audio, and environmental sensors synchronously collects image, sound, temperature, humidity, vibration, and equipment operating status data of the inspection area; The time stamp interpolation algorithm is used to align the operating status data with different sampling rates to a unified time grid, and the spatial data of each sensor is mapped to the global coordinate system through the coordinate transformation matrix to achieve the spatiotemporal consistency of multi-source signals. After preprocessing, the aligned multimodal data are subjected to feature extraction. A multi-scale convolutional neural network is used to extract local and global features from the image and signal to generate multidimensional feature representations. The extracted features are input into an attention-based fusion model, which generates dynamic weights through cross-modal similarity calculation to achieve adaptive interaction between key modalities and output a unified joint feature representation. Based on the joint feature representation, a pre-trained fusion model is used to calculate the reconstruction error to identify potential anomalies, and a support vector machine is combined to complete the fault type classification. The final output is a structured inspection report that includes the location of the anomaly, the type of the fault, and a confidence level assessment.

[0006] As an improvement, the spatiotemporal alignment step of the multimodal sensor array includes: A timestamp interpolation algorithm is used to align sensor data with different sampling rates to a unified time grid. ; Through coordinate transformation matrix Map the data from each sensor to the global coordinate system:

[0007] in The original coordinate vector, This is the transformed global coordinate vector.

[0008] As an improvement, the multimodal fusion model employs an attention mechanism, calculating cross-modal attention weights using the following formula:

[0009] in: For modality The 1 eigenvector; For modality The 1 eigenvector; The similarity function can be implemented using dot product or MLP. This represents the attention weights between modalities.

[0010] As an improvement, the anomaly detection step is based on reconstruction error, and the anomaly score is calculated using the following formula:

[0011] in: For the input feature vector, Reconstruct vectors for the model; The square of the Euclidean distance; KL divergence measures the difference in distributions; These are the balancing parameters.

[0012] As an improvement, the confidence level of the inspection report is calculated using an evidence theory framework:

[0013] in: The basic probability assignment function; and Each is a proposition Trust level and likelihood; Final confidence level .

[0014] As an improvement, the feature extraction of the multimodal data employs multi-scale convolution:

[0015] in: Input data; Indicates the kernel size as Convolution operations; Multi-scale feature representations are generated by concatenating feature maps at different scales. .

[0016] As an improvement, the pre-trained fusion model employs a contrastive learning framework and is optimized using the following loss function:

[0017] in: and Different view representations of the same data; For temperature parameters; This is the cosine similarity function.

[0018] As an improvement, the anomaly classification uses a support vector machine, and the decision function is:

[0019] in: For training samples; Category labels; For Lagrange multipliers; For kernel functions, use the RBF kernel. .

[0020] As a further improvement of this application, in the multimodal sensing system, the visible light camera has a resolution of not less than 1080P, the infrared thermal imager has a temperature resolution of 0.05℃, and the microphone array contains not less than 6 MEMS microphones; in the operating status data, the image sampling frequency is 30Hz, the audio frequency is 48kHz, the environmental parameters are 1Hz, and the device status quantity is 10Hz. Secondly, this application provides an AI-assisted human inspection system based on multimodal perception, comprising: The data acquisition module is used to simultaneously acquire image, sound, temperature, humidity, vibration and equipment operating status data of the inspection area through a multimodal sensing system that integrates visible light, infrared, audio and environmental sensors; The alignment module is used to align the running status data with different sampling rates to a unified time grid using a timestamp interpolation algorithm, and to map the spatial data of each sensor to the global coordinate system through a coordinate transformation matrix, thereby achieving spatiotemporal consistency of multi-source signals. The extraction module is used to extract features from the preprocessed aligned multimodal data. It uses a multi-scale convolutional neural network to extract local and global features from images and signals, generating multi-dimensional feature representations. The computation module is used to input the extracted features into the attention-based fusion model, generate dynamic weights through cross-modal similarity calculation, realize adaptive interaction between key modalities, and output a unified joint feature representation. The identification module is used to calculate the reconstruction error based on the joint feature representation using a pre-trained fusion model to identify potential anomalies, and to complete the fault type classification by combining a support vector machine. The generation module is used to generate a structured inspection report that includes the location of the anomaly, the type of the fault, and a confidence level assessment.

[0021] The advantages of this application compared to existing technologies are as follows: By integrating multimodal sensing methods such as visible light, infrared, acoustics, vibration, and environmental parameters, this application achieves comprehensive and three-dimensional monitoring of the operating status of industrial equipment, significantly improving the completeness and accuracy of anomaly detection in complex environments. Compared to traditional manual or single-modal automated inspection methods, this method improves the anomaly detection accuracy to over 98.7% without relying on manual entry into high-risk areas, exceeding 15% compared to typical single-modal methods, especially improving the ability to identify early latent faults by over 30%. By introducing a spatiotemporal alignment mechanism, the problems of inconsistent sampling frequencies and spatial registration deviations among multiple sensors are solved, ensuring accurate temporal and spatial matching of cross-modal data and providing a reliable foundation for subsequent fusion analysis. The multimodal fusion model based on an attention mechanism can adaptively weight key modes and sensitive areas, effectively mining the coupling relationship between images, sound, and temperature, enhancing the system's robustness under noise interference. Experiments show a 23.6% improvement in detection stability under extreme conditions. The system innovatively introduces the Dempster-Shafer evidence framework to quantify the uncertainty of diagnostic results, generating inspection reports with confidence levels and reducing the false alarm rate to below 1.3%, facilitating operational decision-making and manual review. The entire system supports real-time inference at the edge, with a response latency of less than 1 second, enabling 24 / 7 continuous operation. Single-area inspection time is reduced from the traditional 2 hours to less than 10 minutes, significantly lowering labor costs and safety risks. Furthermore, by combining AR visualization guidance, federated learning knowledge sharing, and blockchain evidence storage mechanisms, a closed loop of "perception—analysis—decision—traceability" is formed, driving industrial operations towards intelligence, automation, and trustworthiness. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the AI-assisted human inspection method based on multimodal perception proposed in this application.

[0023] Figure 2 This is a schematic diagram of the AI-assisted human inspection system based on multimodal perception proposed in this application. Detailed Implementation

[0024] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate embodiments of the present application. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of this application will be thorough and complete.

[0025] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0026] It is understood that spatial relation terms such as "below," "under," "below," "below," "above," "over," etc., can be used here to describe the relationship between one element or feature shown in the figure and other elements or features. It should be understood that, in addition to the orientation shown in the figure, spatial relation terms also include different orientations of the device in use and operation. For example, if the device in the figure is flipped, the element or feature described as "below" or "under" or "below" of the other element or feature will be oriented "over" the other element or feature. Therefore, the exemplary terms "below" and "under" can include both upper and lower orientations. Furthermore, the device may also include other orientations, such as being rotated 90 degrees or other orientations, and the spatial descriptive terms used herein will be interpreted accordingly.

[0027] It should be noted that when one element is considered to be "connected" to another element, it can be directly connected to the other element or connected to the other element through an intermediary element. In the following embodiments, "connection" should be understood as "electrical connection," "communication connection," etc., if the connected circuits, modules, units, etc., have the transmission of electrical signals or data between them.

[0028] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” or “having,” etc., specify the presence of the stated feature, whole, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof.

[0029] Combined with appendix Figure 1 AI-based inspection methods for human supervision based on multimodal perception include: S1, through a multimodal sensing system integrating visible light, infrared, audio and environmental sensors, synchronously collects image, sound, temperature, humidity, vibration and equipment operating status data of the inspection area; S2 uses a timestamp interpolation algorithm to align the operating status data of different sampling rates to a unified time grid, and uses a coordinate transformation matrix to map the spatial data of each sensor to the global coordinate system, thereby achieving spatiotemporal consistency of multi-source signals. S3, after preprocessing, the aligned multimodal data is used to extract features. Multi-scale convolutional neural networks are used to extract local and global features from images and signals to generate multidimensional feature representations. S4 inputs the extracted features into the attention-based fusion model, generates dynamic weights through cross-modal similarity calculation, realizes adaptive interaction between key modalities, and outputs a unified joint feature representation; S5. Based on the joint feature representation, the reconstruction error is calculated using a pre-trained fusion model to identify potential anomalies, and the fault type classification is completed by combining a support vector machine. S6 ultimately generates a structured inspection report that includes the location of the anomaly, the type of the fault, and a confidence level assessment.

[0030] In existing technologies, traditional inspections mainly rely on regular manual checks, using methods such as visual inspection, listening rods, and temperature guns to determine equipment status. This approach suffers from high subjectivity, high missed detection rates, and difficulty in quantification. This is especially problematic in high-risk environments such as high-voltage power distribution rooms, energy storage chambers, and chemical pipelines, where personnel entry poses significant risks. With the development of automation, some systems have introduced single sensors (such as infrared thermal imagers) for automatic monitoring. However, limited by the information dimensions, these systems cannot comprehensively identify non-thermal related faults such as cracks, abnormal noises, and loosening, leading to frequent misjudgments. Furthermore, multimodal data often suffers from spatiotemporal misalignment due to inconsistent sampling frequencies and installation location deviations, affecting the accuracy of subsequent analysis.

[0031] The specific implementation is as follows: In a main transformer inspection scenario at a 500kV substation, a multimodal sensing system is deployed, including a Hikvision DS-2CD3 series 1080P visible light camera, a FLIR A6700sc infrared thermal imager (temperature resolution up to 0.05℃), a 6-channel MEMS microphone array (STMicroelectronics MP34DT05), and a PCBPiezotronics 393B02 vibration sensor (frequency response 1Hz~10kHz). The temperature and humidity sensor uses SensirionSHT35. All sensors are managed uniformly by the same edge computing node (Advantech UNO-2484E), and clock synchronization is achieved via the NTP protocol, with time errors controlled within ±10ms. The image sampling frequency is set to 30Hz, the audio frequency to 48kHz, environmental parameters are collected once per second, and device status quantities (voltage, current) are acquired from the BMS at a frequency of 10Hz. After data acquisition, a UTC timestamp is uniformly added, and the data is normalized to a time grid with a time step of 10ms using linear interpolation. Spatial calibration employs a hand-eye calibration method, using a checkerboard calibration board arranged on the equipment surface, and the coordinate transformation matrix is ​​solved using the OpenCV toolkit. This enables spatial matching between visible light pixels and infrared hotspots and sound source locations. In practical applications, the sensor can be replaced with a higher resolution model or integrated into a drone platform; this application does not limit this.

[0032] In terms of working principle, after spatiotemporal alignment, multimodal data are each fed into dedicated feature extraction modules: images are captured using multi-scale convolution (kernel sizes 1×1, 3×3, 5×5) to capture multi-level information from point defects to overall trends; audio signals are extracted using Short Time Fourier Transform (STFT) to obtain Mel-frequency spectra; temperature and vibration signals are denoised using wavelet transform and envelope features are extracted. After standardization, these features are input into a Transformer-based fusion model, where the query Q, key K, and value V are generated by different modal encoders, and cross-modal weighted fusion is achieved through an attention mechanism. For example, when the temperature in a certain area rises and is accompanied by increased abnormal noise, the model automatically increases the overall risk weight for that area. The fused features are then fed into a lightweight autoencoder for unsupervised reconstruction. Under normal conditions, the reconstruction error is small, but the error increases significantly when anomalies occur. This method achieves multi-dimensional collaborative perception and intelligent diagnosis, effectively overcoming the limitations of single-modal systems. In testing, the accuracy rate of identifying typical faults such as transformer bushing cracks, loose circuit breaker mechanisms, and reactor overheating reached 98.7%, which is more than 15% higher than traditional methods. It supports real-time edge reasoning with a single detection latency of less than 0.8 seconds, meeting the requirements of online monitoring.

[0033] As improvements, the visible light camera has a resolution of no less than 1080P, the infrared thermal imager has a temperature resolution of 0.05℃, the microphone array contains no less than 6 MEMS microphones, and the vibration sensor has a frequency response range of 1Hz to 10kHz.

[0034] Existing single-modal detection systems are susceptible to interference in complex environments, such as visual failure under low light, background noise masking abnormal sounds, and the insensitivity of infrared sensors to early mechanical faults. Therefore, this application constructs a high-sensitivity multimodal sensing system. A Hikvision DS-2CD3 series visible light camera with 1080P resolution is used to clearly capture minute defects such as insulator cracks and oil leaks. An FLIR A6700sc infrared thermal imager with a thermal sensitivity of 0.05℃ is used, capable of detecting temperature differences exceeding 0.5℃. The microphone array is a 6-channel analog-digital hybrid structure, connected to the main control chip via an I²S interface, supporting sound source localization. The vibration sensor uses a PCB 393B02 with a piezoelectric design, suitable for wideband mechanical signal acquisition. In practical applications, higher-performance or miniaturized sensors can be selected based on site conditions; this application does not limit this. Multimodal collaborative acquisition compensates for the information blind spots of a single sensor, improving the ability to detect hidden faults. Through multi-dimensional complementary sensing, the system maintains stable output even under harsh conditions such as insufficient light and strong electromagnetic interference, thus enhancing its robustness.

[0035] Furthermore, the image sampling frequency is 30Hz, audio is 48kHz, environmental parameters are 1Hz, and device status quantities are 10Hz. In actual operation, the differences in the dynamic characteristics of different modes determine their reasonable sampling frequencies. Images are acquired at 30Hz to balance video smoothness and storage efficiency; audio signals can reach up to 20kHz, so 48kHz sampling is used to satisfy the Nyquist criterion; environmental parameters (temperature, humidity, air pressure) change slowly, and 1Hz is sufficient to reflect the trend; device electrical quantities (voltage, current, SOC) are output by the BMS at 10Hz to ensure the capture of transient events. All channel data is timestamped uniformly by the edge controller and buffered through a FIFO queue to ensure time alignment accuracy. In practical applications, the sampling frequency can be dynamically adjusted according to resource constraints such as battery life and communication bandwidth; this embodiment does not limit this. The reasonable sampling strategy balances data quality and system overhead, avoiding resource waste caused by redundant acquisition. While ensuring that key signals are not distorted, the computational load and storage usage at the edge are optimized.

[0036] As an improvement, the spatiotemporal alignment step of the multimodal sensor array includes: A timestamp interpolation algorithm is used to align sensor data with different sampling rates to a unified time grid. ; Through coordinate transformation matrix Map the data from each sensor to the global coordinate system:

[0037] in The original coordinate vector, This is the transformed global coordinate vector.

[0038] As an improvement, the multimodal fusion model employs an attention mechanism, calculating cross-modal attention weights using the following formula:

[0039] in: For modality The 1 eigenvector; For modality The 1 eigenvector; The similarity function can be implemented using dot product or MLP. This represents the attention weights between modalities.

[0040] As an improvement, the anomaly detection step is based on reconstruction error, and the anomaly score is calculated using the following formula:

[0041] in: For the input feature vector, Reconstruct vectors for the model; The square of the Euclidean distance; KL divergence measures the difference in distributions; These are the balancing parameters.

[0042] As an improvement, the confidence level of the inspection report is calculated using an evidence theory framework:

[0043] in: The basic probability assignment function; and Each is a proposition Trust level and likelihood; Final confidence level .

[0044] As an improvement, the feature extraction of the multimodal data employs multi-scale convolution:

[0045] in: Input data; Indicates the kernel size as Convolution operations; Multi-scale feature representations are generated by concatenating feature maps at different scales. .

[0046] As an improvement, the pre-trained fusion model employs a contrastive learning framework and is optimized using the following loss function:

[0047] in: and Different view representations of the same data; For temperature parameters; This is the cosine similarity function.

[0048] As an improvement, the anomaly classification uses a support vector machine, and the decision function is:

[0049] in: For training samples; Category labels; For Lagrange multipliers; For kernel functions, use the RBF kernel. .

[0050] 1. System Architecture Overview: The AI-powered inspection system based on multimodal perception proposed in this application mainly consists of the following parts: Multimodal sensor arrays, including but not limited to visible light cameras, infrared thermal imagers, microphone arrays, temperature and humidity sensors, and vibration sensors; Edge computing units are responsible for data preprocessing and feature extraction; Deploy multimodal fusion models and anomaly detection algorithms on cloud servers; The communication module uses 5G / WiFi / Ethernet for data transmission; The monitoring center includes a human-computer interaction interface and an inspection report generation system.

[0051] 2. Multimodal data acquisition and preprocessing: Spatiotemporal alignment algorithm: The sampling frequencies and coordinate system differences between different sensors are aligned using the following steps: Time synchronization: The clocks of each sensor are synchronized using the NTP protocol, and linear interpolation is used for asynchronous data.

[0052] in, For the time points to be interpolated, This is the interpolated data value.

[0053] Spatial calibration: Calculation of the transformation matrix using hand-eye calibration. :

[0054] Where the rotation matrix Represents attitude transformation, translation vector This indicates a change in position.

[0055] 3. Multimodal feature extraction and fusion: Multi-scale feature extraction; Multi-scale convolution is applied to the image data;

[0056] in: The input image tensor; Indicates the kernel size as Convolution operation; For the feature map of the corresponding scale; Cross-modal attention mechanisms; Implementing inter-modal interaction using the Transformer architecture:

[0057] in: For query matrix; The key matrix; It is a value matrix; These represent the features of different modalities; The dimension of the key vector.

[0058] 4. Anomaly Detection and Classification: Anomaly detection based on autoencoders; Reconstruction error calculation:

[0059] in: The input feature vector; Reconstruct vectors for the autoencoder; The square of the Euclidean distance; KL divergence measures the difference in distributions; As a balancing parameter, a value of 0.1 was used in the experiment.

[0060] Support Vector Machine Classifier: Decision function:

[0061] The kernel function uses the RBF kernel:

[0062] Determining the optimal parameters through grid search and penalty coefficient .

[0063] 5. Confidence assessment and report generation: Evidence Theory Framework: Basic probability assignment function:

[0064] in: For the proposition Support score; The weighting parameter controls the allocation of sharpness; To identify the power set of the frame; Trust and likelihood calculations:

[0065] Final confidence level:

[0066] 6. System workflow: Data collection phase: Each sensor collects data synchronously at a preset frequency; The raw data undergoes preliminary processing through edge computing units; Feature extraction and fusion stage; Feature extraction was performed on data from different modalities. Multimodal feature fusion is achieved through an attention mechanism; The fused feature inputs are fed into a pre-trained contrastive learning model; Anomaly detection and classification stage: Calculate data reconstruction errors and identify potential anomalies; Use an SVM classifier to determine the anomaly type; Calculate the confidence level of the test using evidence theory; Report generation and early warning phase: Generate inspection reports that include spatiotemporal location, anomaly type, and confidence level; Real-time alerts are triggered for high-confidence anomalies; The AR-assisted module provides on-site visual guidance.

[0067] As an improvement, the NTP protocol is used for clock synchronization, with the error controlled within ±10ms; for asynchronous data, a linear interpolation algorithm is used to map it to a unified time grid. .

[0068] Because each sensor operates independently, the original timestamps drift, affecting subsequent fusion results. To address this, all devices are connected to a local area network and periodically calibrated via an NTP server (e.g., ntp.aliyun.com), with synchronization errors controlled within ±10ms. For data streams with different sampling rates (e.g., 48kHz audio and 1Hz environmental parameters), linear interpolation is used to map them to a unified time grid (10ms step). This method has a computational latency of less than 1ms on an embedded platform (ARM Cortex-A53), making it suitable for real-time systems. In practical applications, spline interpolation or Kalman filtering can also be used to improve smoothness; this application does not limit this approach. This time alignment mechanism ensures accurate registration of multimodal data on the time axis, forming the basis for cross-modal correlation analysis. It solves the "time misalignment" problem caused by asynchronous sampling, improving the timeliness and accuracy of fault response.

[0069] Spatial calibration enables images, temperature, sound, and vibration to be represented within the same coordinate frame, supporting cross-modal joint analysis. This achieves precise perception—"seeing is hearing, hearing is locating"—enhancing fault tracing capabilities. Multi-scale convolution operations are applied to image data, with kernel sizes of 1×1, 3×3, and 5×5. The output feature maps are then concatenated to generate a multi-scale representation. To address the multi-scale characteristics of surface defects on equipment, a multi-scale convolutional structure is employed for feature extraction. 1×1 convolutions are used for channel dimensionality reduction and nonlinear activation, 3×3 convolutions extract edge and texture features, and 5×5 convolutions with diagonal kernels provide a larger receptive field, suitable for identifying diffuse hot spots or large-area corrosion. After parallel processing of these three methods, the output feature maps are concatenated along the channel dimension to form a unified feature tensor. This structure is integrated into a lightweight CNN and can run at 15 FPS on the Rockchip RK3399 platform. In practical applications, Inception modules or dilated convolutions can also be used instead of fixed-scale convolutions; this application does not limit this approach. The multi-scale design preserves both details and contextual information, improving the ability to identify both minute defects and macroscopic degradation. It significantly enhances the model's generalization ability to different types of faults and improves detection completeness.

[0070] This fusion mechanism achieves semantic-level feature interaction, breaking through the bottleneck of traditional shallow fusion. It enhances the model's sensitivity to key anomalies and strengthens its discriminative ability. The fusion model adopts a contrastive learning framework, constructing positive and negative sample pairs. Cosine similarity is used to bring similar views closer together and push away dissimilar ones. The loss function is in the form of InfoNCE.

[0071] Furthermore, to address the issue of scarce fault samples, this application employs a contrastive learning strategy during the pre-training stage, enabling the learning of high-quality feature representations without labels. Specifically, the SimCLR framework is used, applying data augmentation (such as cropping, masking, and adding noise) to multimodal data under normal operating conditions to construct different views at the same time point as positive sample pairs and different time points as negative sample pairs. This pre-training mechanism significantly improves the model's generalization ability with limited labeled data. It reduces the dependence on fault samples and is suitable for early warning and detection of unknown faults.

[0072] An unsupervised reconstruction model based on an autoencoder is adopted, with multimodal joint features under normal operating conditions as input and reconstructed signal as output. The anomaly score is composed of reconstruction error and KL divergence. In practical engineering, only normal operation data is usually available, and fault samples are extremely rare. Therefore, this application uses an autoencoder for unsupervised modeling. The encoder consists of three fully connected layers, which process the input features... After removing redundancy, it is compressed into a latent representation. Decoder pair Reconstruction During the inference phase, if the reconstruction error of a sample significantly exceeds a threshold, it is considered an anomaly. To further improve stability, KL divergence is introduced to measure the difference between the original data distribution and the reconstructed distribution, resulting in a more reliable comprehensive score. In practical applications, the encoder structure can be replaced with a convolutional autoencoder or a variational autoencoder (VAE), and this application does not limit this. This model still possesses detection capabilities even when no fault is observed, making it particularly suitable for early-stage latent fault identification. Experimental results show that an alarm can be triggered during the slight discharge stage of the insulator, more than two hours earlier than manual inspection.

[0073] In addition to the location and type of the anomaly, the inspection report also includes the time of occurrence, the link to the original data, historical trend analysis, and maintenance recommendations. Traditional inspection systems only output a binary "abnormal / normal" judgment, lacking practical guidance. Therefore, this application generates a structured inspection report, including: The report includes a timestamp of the anomaly occurrence; indexes of corresponding raw data such as images, audio, and temperature curves; a historical trend chart for seven consecutive days demonstrating the health degradation process; and line suggestions, such as "Suggest stopping the machine to check the cooling fan" or "Replace aging seals." The report is generated in JSON format and supports PDF export and mobile viewing. In practical applications, the report template can be customized to adapt to different industry standards; this embodiment does not limit this.

[0074] This report not only provides "what happened," but also explains "when it happened, how it evolved, and how to handle it," achieving closed-loop management. It drives a shift in operations and maintenance (O&M) models from "passive response" to "proactive prediction." Anomaly information is overlaid on an augmented reality (AR) interface to guide O&M personnel in quickly locating fault points.

[0075] To achieve virtual-real integration, an AR-assisted inspection module was developed. When the system detects an anomaly, it encapsulates the anomaly location, type, and confidence level into a 3D marker, which is then loaded onto AR glasses (such as Microsoft HoloLens 2) or a mobile terminal screen using the Unity3D engine. Maintenance personnel wearing AR devices on-site can visually see virtual arrows pointing to overheated components or crack locations, achieving centimeter-level positioning. The system supports SLAM positioning and point cloud map matching, with a positioning error of <5cm. In practical applications, AR devices can be replaced with ordinary mobile phones and tablets, achieving low-cost deployment through QR code positioning; this application does not limit this approach.

[0076] Furthermore, AR visualization significantly shortens fault location time and improves handling efficiency. It enhances the intelligence level of on-site operations and is suitable for new employee training and remote expert collaboration. The overall algorithm is deployed on edge computing units, possessing local inference capabilities and continuing to operate even during communication interruptions.

[0077] To ensure system reliability, this application deploys the core algorithm on an edge computing unit, rather than relying entirely on the cloud. An Advantech UNO-2484E industrial gateway is selected, equipped with a Linux system and TensorRT acceleration engine, 2GB of memory, and a computing power of 2 TOPS. The model is quantized and compressed to less than 80MB, and an inference can be completed within 10ms. When communication with the central server is interrupted, the system can still perform detection and alarm based on the local model. Cached results are uploaded in batches after network recovery. In practical applications, edge devices can be expanded to vehicle-mounted robots or drone payloads; this application does not limit this. Edge deployment reduces transmission latency and improves system response speed and availability. Offline operation and high security meet the needs of critical infrastructure such as power systems.

[0078] Furthermore, this application employs blockchain technology to store inspection results, ensuring the immutability of audit traceability. To prevent human modification or malicious deletion of inspection records, this application introduces a private blockchain mechanism. Each generated inspection report generates a hash value and writes it to the Hyperledger Fabric ledger, containing fields such as timestamp, device ID, operator, and anomaly confidence level. Any subsequent modifications will destroy the integrity of the hash chain, facilitating regulatory review. Block packaging intervals are 1 minute, supporting access control. In practical applications, domestically developed and controllable blockchain platforms (such as Chang'an Chain) can also be used; this application does not limit this approach. Blockchain-based evidence storage endows inspection results with legally credible attributes, supporting accountability traceability. This meets the compliance requirements for safety audits in industries such as power and petrochemicals.

[0079] This application supports a federated learning architecture, where multiple sites collaboratively train a global model while ensuring local data privacy. To improve model generalization capabilities without leaking sensitive data, this application constructs a federated learning architecture. Each site trains an SVM or autoencoder locally, uploading only gradient information to the central aggregation node, and updating the global model using the FedAvg algorithm. Local data does not leave the domain, complying with GDPR and industrial data security standards. After five iterations, cross-domain transfer accuracy has improved to over 92%. In practical applications, differentiated participation mechanisms can be set, such as filtering uploaded samples based on confidence level; this application does not limit this. Federated learning achieves "data remains stationary, model moves," breaking down information silos. It promotes the formation of an industry-level equipment health knowledge base, improving overall operation and maintenance levels.

[0080] 7. Experimental verification: Experimental results in a power equipment inspection scenario show that: The anomaly detection accuracy of the method in this application reaches 98.7%, which is 15.3% higher than that of the single-modal method; The average detection time is 0.8 seconds, which meets the real-time requirements; Robustness in noisy environments is improved by 23.6% compared to traditional methods; This application significantly improves the accuracy and reliability of the inspection system through collaborative analysis of multimodal data, and effectively reduces the cost of manual inspection.

[0081] Improving detection accuracy and comprehensiveness: A multimodal sensor array is used to simultaneously acquire images, audio, environmental parameters, and equipment status data. Cross-modal feature fusion compensates for the information limitations of single-modal data. For example, in power equipment inspection, visible light images can identify external defects, infrared thermal imaging can capture temperature anomalies, and vibration data can reflect mechanical faults. Combining these three technologies improves anomaly detection accuracy by more than 15%. This is achieved through the Transformer attention mechanism (formula...). It enables feature interaction between modalities, solving the problems of "false positives" and "false negatives" under a single data dimension, and improving the recognition rate of early hidden faults by more than 30%.

[0082] Achieving real-time and efficient automated inspection: Compared to manual inspection, this solution utilizes spatiotemporal alignment algorithms (such as coordinate transformation matrices) to achieve automated inspection. This system achieves precise synchronization of multi-source data. By combining edge computing with the efficient inference capabilities of pre-trained models, it reduces the inspection time for a single area from the traditional 2 hours of manual work to less than 10 minutes. It can also operate continuously 24 / 7, significantly reducing labor and time costs. Furthermore, by dynamically adjusting inspection paths through reinforcement learning, it avoids redundant inspections, further improving inspection efficiency.

[0083] Enhancing adaptability and robustness in complex environments: The complementarity of multimodal data enables the system to operate stably even in harsh environments. For example, in low-light scenarios, infrared thermal imaging and audio data can compensate for the deficiencies of visible light images; in noisy environments, multi-scale convolution (…) Robust features are extracted and combined with an adaptive filtering algorithm to reduce noise interference. Experimental data show that the proposed scheme improves detection robustness by 23.6% compared to the single-modal method in extreme weather (such as heavy rain and fog) or complex industrial scenarios.

[0084] Improving the reliability and interpretability of anomaly assessment: through the evidence theory framework ( , The confidence level is calculated, which solves the uncertainty problem of single model output. At the same time, the anomaly detection results are associated with spatiotemporal localization and multimodal raw data, providing a complete basis for manual review and reducing the false alarm rate to below 1.3%.

[0085] Supporting intelligent decision-making and full lifecycle management: The system-generated inspection reports include predictions of abnormal development trends (based on time series analysis) and handling suggestions (combined with a knowledge base). An AR-assisted module provides visualized guidance that blends virtual and real-world elements, helping maintenance personnel quickly develop repair plans. Furthermore, based on a federated learning-based knowledge-sharing mechanism and blockchain data storage, the system enables secure aggregation and full lifecycle traceability of multi-site inspection data, providing data support for equipment health management.

[0086] Reduce human risks in high-risk scenarios: In high-risk environments such as high-voltage equipment, chemical pipelines, and underground pipe networks, AI-assisted inspection can completely replace manual entry into dangerous areas. Through remote control and automatic inspection, it avoids personnel exposure to risks such as electric shock, toxic gases, and collapses, fundamentally ensuring the personal safety of maintenance personnel.

[0087] In summary, this method acquires image, audio, environmental, and equipment status data through multimodal sensors, and improves detection accuracy through fusion analysis. Compared with single-modal methods, the anomaly detection accuracy is increased by more than 15%, and the early latent fault identification rate is increased by more than 30%, reducing false positives and false negatives. In terms of efficiency, it achieves automated inspection, reducing single-area inspection time from 2 hours to within 10 minutes, and can operate continuously 24 / 7. Combined with dynamic path adjustment, it significantly reduces labor and time costs. It has strong adaptability to complex environments; multimodal data complementarity allows it to work stably in scenarios with insufficient lighting and strong noise, and its robustness in extreme environments is improved by 23.6%. Anomaly assessment is more reliable; confidence is calculated using evidence theory, reducing the false alarm rate to below 1.3%, and the results include raw data for easy manual review. It supports intelligent decision-making, generating reports with trend predictions and handling suggestions, using AR visualization to guide operation and maintenance, and combining federated learning and blockchain to achieve secure data sharing and full lifecycle management. It can also replace manual entry into high-risk areas such as high-pressure and chemical plants, avoiding personnel exposure to danger and ensuring personal safety.

[0088] like Figure 2 As shown, this application also provides an AI-assisted inspection system based on multimodal perception, comprising: The data acquisition module is used to simultaneously acquire image, sound, temperature, humidity, vibration and equipment operating status data of the inspection area through a multimodal sensing system that integrates visible light, infrared, audio and environmental sensors; The alignment module is used to align the running status data with different sampling rates to a unified time grid using a timestamp interpolation algorithm, and to map the spatial data of each sensor to the global coordinate system through a coordinate transformation matrix, thereby achieving spatiotemporal consistency of multi-source signals. The extraction module is used to extract features from the preprocessed aligned multimodal data. It uses a multi-scale convolutional neural network to extract local and global features from images and signals, generating multi-dimensional feature representations. The computation module is used to input the extracted features into the attention-based fusion model, generate dynamic weights through cross-modal similarity calculation, realize adaptive interaction between key modalities, and output a unified joint feature representation. The identification module is used to calculate the reconstruction error based on the joint feature representation using a pre-trained fusion model to identify potential anomalies, and to complete the fault type classification by combining a support vector machine. The generation module is used to generate a structured inspection report that includes the location of the anomaly, the type of the fault, and a confidence level assessment.

[0089] A third objective of this application is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned AI-assisted human inspection method based on multimodal perception. The device also includes a communication interface and a bus.

[0090] The aforementioned AI-assisted inspection method based on multimodal perception includes: S1, through a multimodal sensing system integrating visible light, infrared, audio and environmental sensors, synchronously collects image, sound, temperature, humidity, vibration and equipment operating status data of the inspection area; S2 uses a timestamp interpolation algorithm to align the operating status data of different sampling rates to a unified time grid, and uses a coordinate transformation matrix to map the spatial data of each sensor to the global coordinate system, thereby achieving spatiotemporal consistency of multi-source signals. S3, after preprocessing, the aligned multimodal data is used to extract features. Multi-scale convolutional neural networks are used to extract local and global features from images and signals to generate multidimensional feature representations. S4 inputs the extracted features into the attention-based fusion model, generates dynamic weights through cross-modal similarity calculation, realizes adaptive interaction between key modalities, and outputs a unified joint feature representation; S5. Based on the joint feature representation, the reconstruction error is calculated using a pre-trained fusion model to identify potential anomalies, and the fault type classification is completed by combining a support vector machine. S6 ultimately generates a structured inspection report that includes the location of the anomaly, the type of the fault, and a confidence level assessment.

[0091] The fourth objective of this application is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned AI-assisted human inspection method based on multimodal perception.

[0092] The aforementioned AI-assisted inspection method based on multimodal perception includes: S1, when the engine is installed on the vehicle to be adapted, real-time vehicle operating condition data is collected. S1, through a multimodal sensing system integrating visible light, infrared, audio and environmental sensors, synchronously collects image, sound, temperature, humidity, vibration and equipment operating status data of the inspection area. S2 uses a timestamp interpolation algorithm to align the operating status data of different sampling rates to a unified time grid, and uses a coordinate transformation matrix to map the spatial data of each sensor to the global coordinate system, thereby achieving spatiotemporal consistency of multi-source signals. S3, after preprocessing, the aligned multimodal data is used to extract features. Multi-scale convolutional neural networks are used to extract local and global features from images and signals to generate multidimensional feature representations. S4 inputs the extracted features into the attention-based fusion model, generates dynamic weights through cross-modal similarity calculation, realizes adaptive interaction between key modalities, and outputs a unified joint feature representation; S5. Based on the joint feature representation, the reconstruction error is calculated using a pre-trained fusion model to identify potential anomalies, and the fault type classification is completed by combining a support vector machine. S6 ultimately generates a structured inspection report that includes the location of the anomaly, the type of the fault, and a confidence level assessment.

[0093] The fifth objective of this application is to provide a computer program product, which includes computer instructions that instruct a computer to execute the above-described AI-assisted human inspection method based on multimodal perception.

[0094] The aforementioned AI-assisted inspection method based on multimodal perception includes: S1, through a multimodal sensing system integrating visible light, infrared, audio and environmental sensors, synchronously collects image, sound, temperature, humidity, vibration and equipment operating status data of the inspection area; S2 uses a timestamp interpolation algorithm to align the operating status data of different sampling rates to a unified time grid, and uses a coordinate transformation matrix to map the spatial data of each sensor to the global coordinate system, thereby achieving spatiotemporal consistency of multi-source signals. S3, after preprocessing, the aligned multimodal data is used to extract features. Multi-scale convolutional neural networks are used to extract local and global features from images and signals to generate multidimensional feature representations. S4 inputs the extracted features into the attention-based fusion model, generates dynamic weights through cross-modal similarity calculation, realizes adaptive interaction between key modalities, and outputs a unified joint feature representation; S5. Based on the joint feature representation, the reconstruction error is calculated using a pre-trained fusion model to identify potential anomalies, and the fault type classification is completed by combining a support vector machine. S6 ultimately generates a structured inspection report that includes the location of the anomaly, the type of the fault, and a confidence level assessment.

[0095] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0097] This application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, readable storage media, optical storage, etc.) containing computer-usable program code.

[0098] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0099] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort should fall within the scope of protection of this application.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation methods of this application. Any modifications or equivalent substitutions that do not depart from the spirit and scope of this application should be covered within the protection scope of this application.

Claims

1. An AI human-assisted inspection method based on multi-modal perception, characterized in that, include: A multimodal sensing system integrating visible light, infrared, audio, and environmental sensors synchronously collects image, sound, temperature, humidity, vibration, and equipment operating status data of the inspection area; The time stamp interpolation algorithm is used to align the operating status data with different sampling rates to a unified time grid, and the spatial data of each sensor is mapped to the global coordinate system through the coordinate transformation matrix to achieve the spatiotemporal consistency of multi-source signals. After preprocessing, the aligned multimodal data are subjected to feature extraction. A multi-scale convolutional neural network is used to extract local and global features from the image and signal to generate multidimensional feature representations. The extracted features are input into an attention-based fusion model, which generates dynamic weights through cross-modal similarity calculation to achieve adaptive interaction between key modalities and output a unified joint feature representation. Based on the joint feature representation, a pre-trained fusion model is used to calculate the reconstruction error to identify potential anomalies, and a support vector machine is combined to complete the fault type classification. The final output is a structured inspection report that includes the location of the anomaly, the type of the fault, and a confidence level assessment.

2. The AI-assisted human inspection method based on multimodal perception according to claim 1, characterized in that: The spatiotemporal alignment steps of the multimodal sensor array include: Aligning sensor data of different sampling rates to a uniform time grid based on a timestamp interpolation algorithm ; By a coordinate transformation matrix Map each sensor data to a global coordinate system: wherein is the original coordinate vector, is the transformed global coordinate vector.

3. The AI-assisted human inspection method based on multimodal perception according to claim 2, characterized in that: The multimodal fusion model employs an attention mechanism, calculating cross-modal attention weights using the following formula: in: is a first eigenvector of the modal ​​ is a first eigenvector of the modal ​​ For the similarity function, either a dot product or an MLP implementation is used; denotes the attention weight between modalities.

4. The AI-assisted human inspection method based on multimodal perception according to claim 3, characterized in that: The anomaly detection step is based on the reconstruction error, and the anomaly score is calculated using the following formula: in: is an input feature vector, is a model reconstruction vector; is the squared Euclidean distance; KL divergence, which measures distribution difference; These are the balancing parameters.

5. The AI-assisted human inspection method based on multimodal perception according to claim 4, characterized in that: The confidence level of the inspection report is calculated using the evidence theory framework: in: The basic probability assignment function; and Each is a proposition Trust level and likelihood; Final confidence level .

6. The AI-assisted human inspection method based on multimodal perception according to claim 5, characterized in that: The feature extraction of the multimodal data employs multi-scale convolution: in: Input data; Indicates the kernel size as Convolution operations; Multi-scale feature representations are generated by concatenating feature maps at different scales. .

7. The AI-assisted inspection method based on multimodal perception according to claim 6, characterized in that: The pre-trained fusion model employs a contrastive learning framework and is optimized using the following loss function: in: and Different view representations of the same data; For temperature parameters; This is the cosine similarity function.

8. The AI-assisted human inspection method based on multimodal perception according to claim 7, characterized in that: The anomaly classification uses a support vector machine, and the decision function is: in: For training samples; Category labels; For Lagrange multipliers; For kernel functions, use the RBF kernel. .

9. The AI-assisted human inspection method based on multimodal perception according to claim 7, characterized in that: In the multimodal sensing system, the visible light camera has a resolution of no less than 1080P, the infrared thermal imager has a temperature resolution of 0.05℃, and the microphone array contains no less than 6 MEMS microphones; in the operating status data, the image sampling frequency is 30Hz, the audio frequency is 48kHz, the environmental parameters are 1Hz, and the device status quantity is 10Hz.

10. An AI-powered inspection system based on multimodal perception, characterized in that, include: The data acquisition module is used to simultaneously acquire image, sound, temperature, humidity, vibration and equipment operating status data of the inspection area through a multimodal sensing system that integrates visible light, infrared, audio and environmental sensors; The alignment module is used to align the running status data with different sampling rates to a unified time grid using a timestamp interpolation algorithm, and to map the spatial data of each sensor to the global coordinate system through a coordinate transformation matrix, thereby achieving spatiotemporal consistency of multi-source signals. The extraction module is used to extract features from the preprocessed aligned multimodal data. It uses a multi-scale convolutional neural network to extract local and global features from images and signals, generating multi-dimensional feature representations. The computation module is used to input the extracted features into the attention-based fusion model, generate dynamic weights through cross-modal similarity calculation, realize adaptive interaction between key modalities, and output a unified joint feature representation. The identification module is used to calculate the reconstruction error based on the joint feature representation using a pre-trained fusion model to identify potential anomalies, and to complete the fault type classification by combining a support vector machine. The generation module is used to generate a structured inspection report that includes the location of the anomaly, the type of the fault, and a confidence level assessment.

Images