Intelligent inspection method based on the integration of IoT and virtual reality based on AI big model

Abstract

The present invention discloses an intelligent inspection method based on the fusion of the Internet of Things and virtual reality based on an AI big model, specifically relating to the field of industrial intelligent inspection technology, and is used to solve the end-to-end response lag problem caused by multimodal data fusion delay and resource competition under the existing layered architecture; heterogeneous data of the target device are synchronously collected by multi-source sensors and visual devices, and spatiotemporal synchronization data is generated through cross-modal feature extraction and spatiotemporal alignment; computing resources are dynamically allocated to the AI ​​model or rendering pipeline in combination with cross-modal correlation analysis and environmental interference assessment to give priority to the execution of key tasks; the AI ​​big model is used to perform deep correlation reasoning on multimodal data to generate device status characteristics and abnormal area masks; finally, the abnormal characteristics are superimposed on the three-dimensional scene through virtual-reality fusion rendering technology to form an intuitive interactive interface; efficient fusion and real-time interaction of multimodal data are achieved, significantly improving the accuracy of anomaly detection and the decision-making response efficiency of operators.

Description

Technical Field

[0001] The present invention relates to the field of industrial intelligent inspection technology, and more specifically, to an intelligent inspection method based on the fusion of the Internet of Things and virtual reality based on an AI big model. Background Art

[0002] In the field of intelligent industrial inspection, IoT technology deploys multi-source sensors and visual devices to achieve multi-dimensional perception of equipment status. Combined with AI models, it automates analysis and anomaly detection of collected data. Simultaneously, virtual reality technology is being introduced to provide an immersive interactive interface, assisting operators in remotely viewing equipment status and executing decisions. In existing technologies, IoT data collection, AI analysis, and VR interaction are typically implemented in independent modules, with data transfer and task scheduling accomplished through a layered architecture.

[0003] However, due to the resource competition contradiction between the heterogeneous characteristics of IoT multimodal data (including structured sensor data and unstructured visual data) and the high real-time requirements of virtual reality rendering, the existing layered architecture is difficult to achieve coordinated optimization of data fusion, analytical reasoning and interactive rendering, resulting in a significant increase in end-to-end latency from data collection to visual feedback, which in turn reduces the operator's real-time response ability to abnormal equipment conditions and restricts the effectiveness of the intelligent inspection system. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an intelligent inspection method based on the fusion of the Internet of Things and virtual reality based on an AI big model to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The intelligent inspection method based on the integration of IoT and virtual reality based on AI big model includes:

[0007] S1. Collect multi-source sensor data and visual data of the target device through the IoT terminal;

[0008] S2. Perform cross-modal feature extraction and format unification preprocessing on multi-source sensor data and visual data to generate structured time series data and spatial gridded image data;

[0009] S3. Based on the cross-modal causal reasoning parameters and the spatial coordinate system transformation matrix, the structured time series data and the spatial gridded image data are temporally and spatially aligned to generate spatiotemporally synchronized data;

[0010] S4. Dynamically allocate computing resources to the data fusion module or rendering pipeline module driven by the AI ​​large model based on the correlation of multimodal data in the spatiotemporal synchronization data and the impact of environmental interference on rendering quality;

[0011] S5. The data fusion module uses the AI ​​big model to perform cross-modal correlation analysis on spatiotemporal synchronized data to generate device status feature data and abnormal area mask data;

[0012] S6. The rendering pipeline module integrates the equipment abnormality feature data and the defect area mask data into the three-dimensional inspection scene to generate a virtual and real superimposed inspection interactive interface.

[0013] In a preferred embodiment, the multi-source sensor data and visual data of the target device are collected by the IoT terminal, including:

[0014] The temperature, vibration and current signals of the target device are synchronously collected through temperature sensors, vibration sensors and current sensors as multi-source sensing data;

[0015] The RGB camera and depth camera are used to synchronously collect RGB images and depth point clouds of the target device as visual data;

[0016] Based on the industrial Internet of Things clock synchronization protocol, the temperature signal, vibration signal, current signal, RGB image and depth point cloud are timestamp aligned to generate time-synchronized multi-source sensor data and visual data.

[0017] In a preferred embodiment, cross-modal feature extraction and format unification preprocessing are performed on multi-source sensor data and visual data to generate structured time series data and spatial gridded image data, including:

[0018] Time domain feature extraction is performed on the temperature signal, vibration signal and current signal in the multi-source sensor data to generate a time series feature vector containing the mean, variance and spectrum amplitude;

[0019] Perform spatial grid division on the RGB image in the visual data, extract the texture features and color histogram of each grid area, and generate spatial gridded image data;

[0020] Voxelize the depth point cloud in the visual data, map the 3D point cloud to a 2D spatial grid, and supplement the depth information of the spatial gridded image data;

[0021] The time series feature vectors and spatial gridded image data are normalized separately, unified to the same numerical range, and then stored as structured time series data and spatial gridded image data.

[0022] In a preferred embodiment, the structured time series data and the spatial gridded image data are temporally and spatially aligned based on the cross-modal causal inference parameters and the spatial coordinate system conversion matrix to generate temporally and spatially synchronized data, including:

[0023] Through causal correlation analysis of multi-source sensor data and visual data, time offset is extracted to generate cross-modal causal reasoning parameters;

[0024] Calculate the three-dimensional space coordinate system transformation matrix of the target device based on the calibration plate feature point matching;

[0025] Compensate the time offset to the timestamp of structured time series data to generate a time-synchronized sensor data stream;

[0026] Applying a three-dimensional space coordinate system transformation matrix to the spatially gridded image data to generate a spatially aligned visual data stream;

[0027] Fusing temporally synchronized sensory data streams with spatially aligned visual data streams to generate spatiotemporally synchronized data.

[0028] In a preferred embodiment, based on the correlation of multimodal data in spatiotemporal synchronization data and the impact of environmental interference on rendering quality, computing resources are dynamically allocated to the data fusion module or rendering pipeline module driven by the AI ​​large model, including:

[0029] Extract the latent variable distribution of multimodal data in spatiotemporal synchronized data, calculate the Wasserstein distance between latent variable distributions, and generate multimodal latent variable similarity coefficients;

[0030] Generate a degraded virtual scene based on environmental interference parameters, calculate the difference in structural similarity index between the degraded virtual scene and the ideal scene, and generate a rendering quality conflict mapping value;

[0031] When the multimodal latent variable similarity coefficient is less than or equal to the preset similarity threshold and the rendering quality conflict mapping value is greater than or equal to the preset conflict threshold, computing resources are allocated to the data fusion module driven by the AI ​​large model;

[0032] When the multimodal latent variable similarity coefficient is greater than a preset similarity threshold or the rendering quality conflict mapping value is less than a preset conflict threshold, computing resources are allocated to the rendering pipeline module.

[0033] In a preferred embodiment, the preset similarity threshold and the preset conflict threshold are set according to the statistical distribution quantile when the modal complementarity in the historical data is optimal.

[0034] In a preferred embodiment, the data fusion module performs cross-modal correlation analysis on spatiotemporal synchronized data through an AI large model to generate device status feature data and abnormal area mask data, including:

[0035] The multi-head cross-attention mechanism is used to model the cross-modal association of multimodal features in spatiotemporal synchronized data and generate a fused feature vector.

[0036] Calculate the cosine similarity between the fused feature vector and the pre-trained embedding vector of the normal operation status of the device to generate the anomaly confidence in the device status feature data;

[0037] The spatial gridded image data is segmented by a sliding window, the gradient distribution of texture features and depth features within the window is extracted, and candidate masks of abnormal areas are generated;

[0038] Morphological closing operation is used to optimize candidate masks of abnormal regions, eliminate isolated noise points and fill in empty regions to generate abnormal region mask data;

[0039] The abnormality confidence is compared with the preset abnormality confidence threshold. When the abnormality confidence exceeds the threshold, it is marked as an abnormal state of the device.

[0040] In a preferred embodiment, the pre-trained embedding vector is generated by training with historical normal data, and the anomaly confidence threshold is set according to the statistical distribution quantile of historical anomaly samples.

[0041] In a preferred embodiment, the rendering pipeline module fuses the device abnormality feature data and the defect area mask data into a three-dimensional inspection scene to generate a virtual and real superimposed inspection interactive interface, including:

[0042] Convert the abnormal area mask data into a transparency texture on the 3D model surface and fit it to the 3D inspection scene of the target device through UV mapping;

[0043] Convert the temperature mean and vibration root mean square value in the equipment status characteristic data into a thermal cloud map and vibration intensity field, and superimpose them on the corresponding spatial coordinates of the three-dimensional inspection scene;

[0044] Through depth testing and transparency mixed rendering, the defect area mask data is fused with the thermal cloud map and vibration intensity field to generate a virtual and real superimposed inspection interactive interface.

[0045] In a preferred embodiment, the threshold of the transparency texture is set according to the statistical quantile of the mask data of historical normal samples, and the color mapping gradient of the thermal cloud map is dynamically adjusted according to the safe operating temperature range of the equipment.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. Through the deep collaboration of cross-modal data and the dynamic resource optimization mechanism, the data processing efficiency and real-time interaction of virtual and real in industrial inspection scenarios have been significantly improved. In view of the heterogeneous characteristics of multi-source sensor data and visual data, the spatiotemporal alignment technology is used to break through the collaboration bottleneck of the traditional layered architecture, and the precise synchronization of multimodal data in timestamps and spatial references is achieved through the joint calibration of causal reasoning parameters and spatial coordinate transformation matrices. Compared with the modular processing flow, the collaborative mechanism effectively eliminates the fusion delay caused by data silos, so that sensor signals such as temperature and vibration and visual image features can be mapped to a unified spatiotemporal framework in real time, providing a highly consistent input basis for subsequent analysis. At the same time, based on the dynamic resource allocation strategy, the system can adaptively adjust the computing power distribution according to the multimodal correlation strength and the degree of environmental interference, and give priority to responding to key abnormal events while ensuring rendering quality, thereby significantly shortening the full-link delay from data collection to decision feedback.

[0048] 2. Through the interactive optimization design of virtual-real integration, the closed-loop control capability of anomaly detection and on-site feedback during the inspection process is enhanced. With the help of the AI ​​large model's ability to extract cross-modal correlation features, the system can accurately identify subtle anomalies in the equipment's operating status and generate high-confidence defect area masks. Combined with the deep synthesis technology of the 3D rendering engine, abstract data features are converted into intuitive virtual-real superposition scenes, so that complex parameters such as temperature gradients and vibration intensity can be projected onto the equipment model in real time in the form of thermal cloud maps, vector fields, etc. Operators can quickly locate problems without relying on multi-screen switching or abstract reports. At the same time, the adaptive rendering pipeline ensures the upfront visibility of key information through transparency blending and depth testing, maintains the stability and smoothness of the interactive interface under complex working conditions, and provides a highly immersive decision-making environment for remote inspections. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a flow chart of the intelligent inspection method based on the integration of the Internet of Things and virtual reality based on the AI ​​big model of the present invention;

[0050] Figure 2 This is a flow chart of the dynamic resource allocation decision driven by multimodal data of the present invention. DETAILED DESCRIPTION

[0051] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0052] Example: Figure 1The present invention provides an intelligent inspection method based on the integration of the Internet of Things and virtual reality based on an AI large model, which includes the following steps:

[0053] S1. Collect multi-source sensor data and visual data of the target device through the IoT terminal;

[0054] S2. Perform cross-modal feature extraction and format unification preprocessing on multi-source sensor data and visual data to generate structured time series data and spatial gridded image data;

[0055] S3. Based on the cross-modal causal reasoning parameters and the spatial coordinate system transformation matrix, the structured time series data and the spatial gridded image data are temporally and spatially aligned to generate spatiotemporally synchronized data;

[0056] S4. Dynamically allocate computing resources to the data fusion module or rendering pipeline module driven by the AI ​​large model based on the correlation of multimodal data in the spatiotemporal synchronization data and the impact of environmental interference on rendering quality;

[0057] S5. The data fusion module uses the AI ​​big model to perform cross-modal correlation analysis on spatiotemporal synchronized data to generate device status feature data and abnormal area mask data;

[0058] S6. The rendering pipeline module integrates the equipment abnormality feature data and the defect area mask data into the three-dimensional inspection scene to generate a virtual and real superimposed inspection interactive interface.

[0059] S1. Collect multi-source sensor data and visual data of the target device through the IoT terminal. The specific implementation is as follows:

[0060] Temperature, vibration, and current signals from the target device are synchronously collected through temperature sensors, vibration sensors, and current sensors as multi-source sensing data. The temperature sensor is a thermocouple-type sensor installed on the surface of the target device's hotspot area. The temperature signal measurement range is set based on the device's normal operating temperature range and safety thresholds under extreme operating conditions. For example, the temperature range of a thermal power plant boiler pipe is set from -50°C to 300°C. The temperature signal is filtered through a low-pass filter to eliminate high-frequency noise. The filter cutoff frequency is determined by the relationship between the device's thermal conductivity characteristics and the signal sampling rate. It is specifically calculated as fc = 1 / 2πRC, where fc is the filter cutoff frequency, R is the filter resistance value, and C is the capacitance value. In actual applications, it is adjusted based on the device's thermal inertia time constant.

[0061] The vibration sensor is a piezoelectric accelerometer installed at a vibration-sensitive location on the target equipment. The frequency range of the measured vibration signal covers the harmonics of the equipment's mechanical resonance frequency and the operating fundamental frequency. For example, the frequency range of a turbine bearing vibration signal is set to 0.1 Hz to 5 kHz. The vibration signal is converted to vibration acceleration using a root mean square (RMS) algorithm. The effective amplitude range of the vibration acceleration value is determined by the equipment's vibration safety standard, such as the 0.1g to 20g specified in ISO 10816-3. The current sensor is a Hall-effect sensor installed on the power cable of the target equipment. The measured current signal range is determined by the equipment's rated current and the peak short-circuit current. For example, the current range of a distribution transformer is set to 0A to 1000A. The current signal is filtered through an anti-aliasing filter to eliminate power frequency interference. The filter order and cutoff frequency are selected based on the signal bandwidth and the sampling theorem. For example, a 4th-order Butterworth filter with a cutoff frequency of 60Hz is used.

[0062] An RGB camera and a depth camera are used to synchronously capture RGB images and depth point clouds of the target device as visual data. The RGB camera is a global shutter industrial camera. Its resolution and frame rate are set based on the target device's size and motion speed. For example, for power line inspection scenarios, the resolution is set to 1920×1080 pixels and the frame rate is set to 30 fps. The installation position and angle are determined through 3D spatial calibration to ensure coverage of the target device's critical inspection area. The RGB image is adjusted for brightness and color consistency using an automatic white balance and exposure compensation algorithm. The white balance parameters are dynamically adjusted based on the ambient light intensity and color temperature. The depth camera is a time-of-flight sensor. The depth measurement accuracy and working distance are set based on the target surface reflectivity and inspection accuracy requirements. For example, the measurement accuracy for metal pipes is set to ±1mm, and the working distance is set to 0.5 to 5 meters. The depth point cloud data is converted to 3D coordinates with the target device's center as the origin using a coordinate alignment algorithm. The coordinate alignment algorithm calculates a homography matrix based on the coordinates of the feature points on a calibration plate. The calibration plate is a 30 cm × 30 cm checkerboard pattern with a 5 cm spacing between feature points.

[0063] Based on the Industrial Internet of Things (IIoT) clock synchronization protocol, timestamps of temperature, vibration, current, RGB images, and depth point clouds are aligned to generate time-synchronized multi-source sensor and visual data. The IIoT clock synchronization protocol utilizes the IEEE 1588 Precision Time Protocol. The master clock source is deployed at the IoT gateway. Clock synchronization accuracy is set based on network latency and device jitter tolerance. For example, the master-slave clock synchronization period is set to 1 second, and the clock offset compensation threshold is set to ±10 milliseconds. Exceeding the threshold triggers clock resynchronization. The timestamp alignment process includes the following steps: offsetting the timestamps of the temperature, vibration, and current signals relative to the master clock source's time base. The offset is calculated using a linear regression algorithm with a regression window covering the most recent 10 synchronization periods. The acquisition timestamps of the RGB images and depth point clouds are aligned to a common time base using an interpolation algorithm. The interpolation algorithm is selected based on data continuity and real-time requirements, such as cubic spline interpolation, with an interpolation interval set to 1 millisecond based on the sensor sampling rate.

[0064] The time-synchronized multi-source sensor data and visual data are uploaded to the edge computing node through the message queue telemetry transmission protocol. The data encapsulation format is JSON, which includes the timestamp, device ID, data value and data quality flag. The data quality flag is generated according to the signal validity detection rules. For example, when the temperature signal exceeds the range, it is marked as invalid, and when the vibration signal is missing more than three consecutive sampling points, it is marked as abnormal.

[0065] The installation locations of temperature sensors, vibration sensors, and current sensors are determined based on the target device's thermodynamic properties, mechanical vibration modes, and electrical topology. For example, for power transformers, temperature sensors are installed in the winding hotspot area, with the hotspot location determined through infrared thermal imaging pre-scanning. Vibration sensors are installed on the fuel tank casing, with their location determined through finite element modal analysis. Current sensors are installed on the high-voltage side bushing leads, with the installation angle adjusted based on the magnetic field distribution characteristics. The installation position and angle of the RGB camera and depth camera are determined through three-dimensional spatial calibration. The calibration process involves measuring the relative distance between the camera and the target device using a laser rangefinder and calculating internal and external parameters using Zhang Zhengyou's calibration method.

[0066] During the timestamp alignment process, if the sensor or camera clock offset is detected to exceed the preset threshold, such as ±10 milliseconds, the clock resynchronization mechanism is triggered. The resynchronization mechanism includes sending a synchronization request message to the master clock source. The message content contains the current clock offset and jitter value, and updating the local clock register. The register update step is dynamically adjusted according to the offset. For example, microstepping compensation is used when the offset is less than 1 millisecond, and step compensation is used when the offset is greater than 1 millisecond.

[0067] The edge computing node caches and verifies the received time synchronization data. The verification rules are set according to the equipment safety operation specifications and signal physical constraints. For example, the temperature signal verification range is from the lower limit to the upper limit of the equipment material tolerance temperature, the vibration acceleration value verification range is from the lower limit to the upper limit of the equipment mechanical structure safety vibration amplitude, and the current signal verification range is from the equipment no-load current to the short-circuit current peak. If the data verification fails, the abnormal data is discarded and an error log is recorded. The error log content is set according to the fault diagnosis requirements, such as recording the abnormal data type, timestamp and device ID. The multi-source sensor data and visual data that have passed the verification are stored in the time series database. The time series database table structure is designed according to the data analysis requirements, such as including timestamp, device ID, temperature, vibration, current, RGB image path and depth point cloud path. The path information generates a unique identifier through a hash algorithm.

[0068] The data transmission delay in the time synchronization process is optimized through round-trip time measurement and compensation algorithm. The round-trip time measurement interval is set according to network stability, for example, once every 1 second. The compensation algorithm uses the exponentially weighted moving average method, and the weight coefficient is dynamically adjusted according to the volatility of network delay. For example, when the delay standard deviation exceeds 10 milliseconds, the weight is set to 0.2 to quickly respond to changes. When the standard deviation is less than 10 milliseconds, the weight is set to 0.8 to maintain stability.

[0069] S2. Perform cross-modal feature extraction and format unification preprocessing on multi-source sensor data and visual data to generate structured time series data and spatial gridded image data. The specific implementation is as follows:

[0070] Time-domain feature extraction is performed on the temperature, vibration, and current signals from the multi-source sensor data, generating a time series feature vector consisting of the mean, variance, and spectral amplitude. Time-domain feature extraction of the temperature signal is achieved using a sliding window method. The window length is set based on the rate of change of the equipment state. For example, in a thermal power plant boiler temperature monitoring scenario, the window length is set to 10 seconds, based on twice the boiler's thermal inertia time constant. The mean and variance of the data within the window are calculated. The mean reflects the steady-state temperature level, while the variance indicates the degree of temperature fluctuation. Time-domain feature extraction of the vibration signal involves calculating the root mean square (RMS) value of the acceleration signal. The RMS calculation window length matches the period of the equipment's fundamental vibration frequency. For example, for a steam turbine with a fundamental vibration frequency of 50 Hz, the window length is set to 0.02 seconds. Simultaneously, the spectral amplitude of the vibration signal is extracted using a fast Fourier transform. The spectral amplitude corresponds to the frequency range of integer multiples of the equipment's mechanical resonance frequency. The resolution of the spectral amplitude calculation is determined by the sampling rate and window length. For example, the resolution is 1 Hz for a 1 kHz sampling rate. The time domain feature extraction of the current signal includes calculating the effective value of the current and the harmonic distortion rate. The effective value of the current is calculated by the integration method, and the integration interval is the power frequency period. The harmonic distortion rate is calculated as a percentage after extracting the fundamental wave and the amplitude of each harmonic through discrete Fourier transform. The upper limit of the harmonic order is set to 50 according to the electrical standard of the equipment.

[0071] The RGB image in the visual data is spatially gridded, and texture features and color histograms are extracted from each grid area to generate spatially gridded image data. The grid size of the spatial gridding is set according to the dimensions of the key components of the target equipment. For example, the grid size of the transmission line insulator is set to 32×32 pixels, based on the ratio of the average size of the insulator shed structure to the image resolution. Texture feature extraction uses a gray-level co-occurrence matrix algorithm to calculate the contrast, correlation, and energy values ​​of the pixels within the grid. Contrast reflects surface roughness, correlation indicates texture directional consistency, and energy indicates texture uniformity. The parameters for generating the gray-level co-occurrence matrix include the number of grayscale levels and the directional angle. The number of grayscale levels is set to 16, and the directional angles include 0°, 45°, 90°, and 135°. The color histogram is generated by converting the RGB image to the HSV color space and then calculating the distribution of hue and saturation. The number of bins in the hue histogram is set to 16 based on color recognition requirements, and the number of bins in the saturation histogram is set to 8. The conversion formula is the normalization of the hue component in the HSV model.

[0072] The depth point cloud in the visual data is voxelized, mapping the 3D point cloud to a 2D spatial grid to supplement the depth information in the spatially gridded image data. The voxelized grid resolution matches the grid size of the RGB image. For example, a 32×32 pixel 2D grid corresponds to a 0.5 cm×0.5 cm×0.5 cm voxel size in 3D space. The voxel size is set based on the spatial resolution requirements of the target device. The depth point cloud mapping process involves calculating the average depth of all 3D point cloud points within each 2D grid, rounded to two decimal places in millimeters. The standard deviation of the point cloud density within the grid is also recorded. The standard deviation is used to assess the reliability of the depth measurement. The standard deviation threshold is set at 10%, and data exceeding the threshold is marked as low-confidence data. The depth information is supplemented by adding a depth channel to the spatially gridded image data. The depth channel has a value range of 0 to 10,000 mm, corresponding to the target device's minimum detection distance of 0 meters to a maximum detection distance of 10 meters. The value range is determined by the calibration parameters of the depth sensor.

[0073] The time series feature vectors and spatial gridded image data are normalized separately, brought to the same numerical range, and stored as structured time series data and spatial gridded image data. Normalization of the time series feature vectors uses a minimum-maximum scaling method, with the scaling range set to 0 to 1. For example, the original range of the mean temperature is -50°C to 300°C, but after normalization, it is mapped to a floating-point number between 0 and 1. The scaling parameters are initialized based on the global extreme values ​​of the training dataset. Normalization of the spatial gridded image data involves independent scaling of the texture features, color histogram, and depth channel. The contrast and energy values ​​of the texture features are scaled to a range of 0 to 1, the frequency of the color histogram is converted to a probability distribution, and the depth channel values ​​are linearly scaled to a range of 0 to 1 based on a maximum detection distance of 10 meters. The normalization parameters include the minimum and maximum values ​​of each feature. These parameters are determined by offline statistics of historical extreme values ​​in the training dataset and dynamically updated during the online processing phase. The update is triggered when a feature value continuously exceeds the historical extreme value range for more than 10 data windows. The storage format of structured time series data and spatial gridded image data is a multidimensional array. The array dimension of time series data is timestamp × feature dimension, and the array dimension of spatial gridded image data is number of grid rows × number of grid columns × number of feature channels. Metadata tags are attached when data is stored. The metadata tags include device ID, acquisition time and sensor calibration coefficient. The metadata tags are encapsulated in JSON format, and the field type and length are consistent with the data dictionary.

[0074] The sliding window step size for time-domain feature extraction is set based on real-time requirements. For example, the window step size for temperature signals is set to 1 second, and the window step size for vibration signals is set to 0.01 seconds. The setting is based on the minimum time interval for device status updates. During the spatial grid division process, if it is detected that the grid in the edge area of ​​the image contains invalid pixels, the texture features of the adjacent grids are interpolated and filled. The interpolation method is bilinear interpolation, and the interpolation weight is calculated based on the grid distance. During the voxelization process, if there is no valid point cloud data in the grid, the depth value is marked as invalid and the data verification mechanism is triggered. The data verification mechanism generates a replacement value based on the statistical distribution of historical depth data. The replacement value is the moving average of the depth of adjacent grids, and the moving average window length is 5 adjacent grids.

[0075] The dynamic update logic for normalization includes monitoring for shifts in the distribution of feature values. For example, if the mean temperature value exceeds the historical extreme value for ten consecutive windows, the minimum-maximum scaling parameters are recalculated and the storage configuration is updated. The updated parameters are identified by a version number and the update timestamp is recorded. Metadata tags stored in multidimensional arrays are encapsulated in JSON format. The JSON fields include a string type for the device ID, a timestamp type for the acquisition time, and a floating-point array of sensor calibration coefficients. The length of the floating-point array matches the number of sensors, and the order of the array elements corresponds to the sensor installation location. Time series data and spatially gridded image data are stored on SSDs. The storage path is hierarchical, organized according to the device ID and acquisition time. The directory hierarchy is device ID / year / month / day / hour. Data files are named using the timestamp_feature type.format format. For example, the temperature feature file is named 202310151200_TEMP.npy. The format is chosen based on a balance between data compression and read efficiency.

[0076] S3. Based on the cross-modal causal reasoning parameters and the spatial coordinate system conversion matrix, the structured time series data and the spatial gridded image data are temporally and spatially aligned to generate temporally and spatially synchronized data. The specific implementation is as follows:

[0077] Causal correlation analysis of multi-source sensor data and visual data is used to extract time offsets and generate cross-modal causal inference parameters. Causal correlation analysis employs the Granger causality test method. The time window length is set based on prior knowledge of device state changes. For example, the time window between boiler temperature signals and visual hot spot detection data in a thermal power plant is set to 30 seconds, based on twice the boiler's thermal inertia time constant. The number of lags is set to third order based on signal transmission delay. The selection of the number of lags is automatically optimized using the Akaike Information Criterion, with a penalty factor of 2 and a maximum number of lags set to 10 based on the signal sampling rate. The Granger causality test hypothesis test significance level is set at 0.05, based on conventional statistical confidence levels. A causal relationship is considered to exist when the test p-value is less than 0.05. The time offset is calculated using the maximum mutual information criterion. The time lag range for mutual information calculation is set from -5 seconds to +5 seconds, with a step size of 0.1 seconds. The lag range is selected based on the maximum expected deviation in the device's physical response time. The cross-modal causal inference parameters include the time offset between the temperature signal and the hot spot image, and the time offset between the vibration signal and the structural deformation image. After the parameters are generated, they are stored in a key-value pair format. The key is the sensor-camera pair identifier. The identifier format is "sensor type_camera ID". The value is the offset millisecond value, and the millisecond value is retained to three decimal places.

[0078] The target device's 3D coordinate transformation matrix is ​​calculated based on feature point matching on a calibration plate. The calibration plate features a checkerboard pattern measuring 30cm x 30cm, with alternating black and white squares spaced 5cm apart. The plate is fixed to the target device's non-detection area, where a laser rangefinder and a level ensure physical alignment with the device's coordinate system. Feature point matching utilizes a scale-invariant feature transformation algorithm, extracting the calibration plate's corners as feature points. The corner detection threshold is set to 0.01, based on the statistical distribution of contrast in the calibration plate image. The feature descriptor dimension is 128, chosen to balance algorithm computational efficiency with feature discrimination. The calculation of the three-dimensional space coordinate system transformation matrix includes the following steps: fitting the correspondence between the feature points of the calibration plate in the camera coordinate system and the device coordinate system through the least squares method, calculating the initial estimate of the homography matrix, and setting the fitting residual tolerance to 0.5 pixels; optimizing the rotation matrix and translation vector through the iterative nearest point algorithm, and the iterative termination condition is that the sum of squared residuals is less than 0.1 pixels or the maximum number of iterations reaches 100 times; the final output space coordinate system transformation matrix is ​​a 4×4 homogeneous matrix, including the Euler angle accuracy of the rotation matrix set to 0.01 degrees, the translation vector accuracy set to 0.1 mm, and the scaling factor fixed to 1.0.

[0079] Time offsets are compensated for to the timestamps of structured time series data to generate a time-synchronized sensor data stream. Time offset compensation is achieved through linear interpolation, with an interpolation step size set based on the sensor sampling rate. For example, for a temperature signal with a sampling rate of 1 Hz, the interpolation step size is 1 second, and for a vibration signal with a sampling rate of 1 kHz, the interpolation step size is 0.001 seconds. Timestamp adjustment involves adding the original timestamps to the offset to generate an aligned timestamp sequence. Timestamp accuracy is retained to the millisecond level. The continuity of the timestamp sequence is verified using a sliding window overlap ratio set to 50%, with the overlap window verification threshold set to a time difference of less than 1 millisecond. The time-synchronized sensor data stream is stored as a list of time-value pairs, with the timestamp as the key and the value as the normalized sensor reading. The normalization method uses min-max scaling, with the scaling parameters initialized based on the global extreme values ​​of the historical dataset.

[0080] A three-dimensional spatial coordinate system transformation matrix is ​​applied to the spatially gridded image data to generate a spatially aligned visual data stream. The spatial alignment process includes the following steps: Each grid center coordinate in the spatially gridded image data is multiplied by the transformation matrix and mapped to the device's unified coordinate system. The floating-point precision of the coordinate mapping is retained to four decimal places. Grid coordinates that exceed the target device boundary are truncated. The truncation threshold is set based on the device's physical dimensions. For example, the boundary threshold for a transformer tank is 2m long × 1m wide × 1.5m high, based on the dimensions of the device design drawings. The truncated grid coordinates are interpolated using bilinear interpolation, with interpolation weights calculated based on grid distance, using the inverse Euclidean distance weighted formula. The spatially aligned visual data stream is stored as a grid coordinate-eigenvalue dictionary. The grid coordinates are 3D floating-point numbers, and the eigenvalues ​​include texture contrast, color histogram, and depth average. The dictionary hash collision resolution uses a chain address method, and the hash table size is dynamically allocated based on the number of grids.

[0081] The spatiotemporally synchronized data is generated by fusing time-synchronized sensor data streams with spatially aligned visual data streams. The fusion process is based on a joint indexing of timestamps and spatial coordinates. The index key is a timestamp-grid coordinate pair with millisecond accuracy and millimeter accuracy. The index value is the concatenation of the sensor reading and visual features, with the vector dimension being the sum of the number of sensor features and the number of visual features. The spatiotemporally synchronized data is stored as a multidimensional tensor with the dimensions timestep × grid row × grid column × feature channel. The feature channels include temperature mean, vibration RMS value, color histogram, and depth value. The tensor data type is 32-bit floating point. The tensor is stored with metadata tags, including the spatial coordinate system transformation matrix version number, a hash value of the time synchronization parameters, and a data checksum. The checksum is generated using a cyclic redundancy check algorithm with a CRC-32 polynomial. The initial value of the checksum is set to 0xFFFFFFFF during generation, and the final XOR value is set to 0xFFFFFFFF.

[0082] The number of lags in the Granger causality test is automatically selected using the Akaike Information Criterion (IC), with a penalty factor of 2. The maximum number of lags is set to 10 based on the signal sampling rate. A warning log is generated if the number of lags exceeds 10. During the calibration plate feature point matching process, if missing feature points due to occlusion or reflection are detected, they are supplemented by interpolation from adjacent frames. The interpolation method is bicubic interpolation, with interpolation weights calculated based on the spatial distance between feature points. The normalization coefficient for distance calculation is the inverse of the calibration plate's diagonal length. For time offset compensation, linear interpolation switches to spline interpolation when more than three consecutive sampling points are missing. The boundary conditions for spline interpolation are set to natural splines, and the number of spline nodes is dynamically adjusted based on the length of the missing data. During the spatial alignment truncation process, if a grid coordinate exceeds the device boundary, the feature value of that grid is marked invalid. The invalid flag is stored as a binary mask, with a mask bit count of 1 indicating valid and 0 indicating invalid. The mask is stored as a separate bitmap file with a resolution that matches the number of grid cells.

[0083] During the fusion of spatiotemporal synchronized data, if the timestamps or spatial coordinate indices do not match, a spatiotemporal interpolation mechanism is triggered. This mechanism involves linear interpolation in the time dimension and bilinear interpolation in the spatial dimension. Interpolation weights are calculated based on time difference and spatial distance, with the time difference weight coefficient set to 0.5 and the spatial distance weight coefficient set to 0.5. Multidimensional tensors are stored in a layered data format, using the Zlib compression algorithm and a compression level of 6. The compression level is selected based on test results that balance compression ratio and decompression speed. The cyclic redundancy check code of the metadata tag is verified before data transmission. If the verification fails, a data retransmission mechanism is triggered. The maximum number of retransmissions is set to 3, and the retransmission interval is dynamically adjusted based on network latency. The initial retransmission interval is set to 100 milliseconds, and the interval doubles after each retransmission.

[0084] Step S3 achieves spatiotemporal alignment by combining cross-modal causal inference parameters with the spatial coordinate system transformation matrix. Traditional methods rely on hardware clock synchronization or fixed interpolation algorithms, making them difficult to address spatiotemporal offsets of multimodal data in dynamic environments. For example, this occurs when the spatiotemporal references of sensor and visual data differ due to transmission delays or environmental disturbances. Step S3 uses causal correlation analysis (such as Granger causality testing) to derive temporal offsets from the data itself, replacing the rigid constraints of hardware synchronization. Simultaneously, spatial coordinate transformation relationships are established based on feature matching of calibration plates to address spatial reference differences in multimodal data. Compared to existing technologies, data-driven dynamic calibration overcomes hardware synchronization limitations, reduces end-to-end latency, and can adapt to environmental changes, improving data fusion accuracy.

[0085] Figure 2A flowchart of the dynamic resource allocation decision driven by multimodal data in the present invention is given. Based on the correlation of multimodal data in spatiotemporal synchronization data and the impact of environmental interference on rendering quality, computing resources are dynamically allocated to the data fusion module or rendering pipeline module driven by the AI ​​large model. The specific implementation is as follows:

[0086] The latent variable distributions of multimodal data from spatiotemporally synchronized data were extracted, the Wasserstein distance between the latent variable distributions was calculated, and the multimodal latent variable similarity coefficient was generated. The latent variable distributions were generated by encoding the multimodal data into a latent space using a variational autoencoder. The encoder network of the variational autoencoder consisted of three fully connected layers, 128 hidden layer nodes, a Reluctant Luminance (ReLU) activation function, and a 32-dimensional latent space. The input data consisted of the mean temperature, root mean square (RMS) vibration, and color histogram from the spatiotemporally synchronized data. The training process of the variational autoencoder involved: the encoder network generated the mean and variance of the latent space distribution of the input data, sampled the latent variables using a reparameterization technique, and the decoder network reconstructed the latent variables into the input data. The reconstruction loss function was the mean squared error, the regularization loss was the KL divergence, and the total loss function was a weighted sum of the two, with the weight coefficient set to 0.5. The Wasserstein distance is calculated based on the mean and covariance matrix of the latent variable distribution. It is calculated as the weighted sum of the squared Euclidean distance between the two distribution means and the Frobenius norm of the covariance matrix. The weight coefficients are set to 0.7 and 0.3, respectively, based on the importance of the modal data. The weight coefficients are also based on the contribution of the modal data to historical anomaly detection. For example, when the temperature signal contributes more to defect detection than the vibration signal, the temperature modal weight is set to 0.7. The multimodal latent variable similarity coefficient ranges from 0 to 1, with smaller values ​​indicating greater complementarity between the modalities. For example, the similarity coefficient between the temperature signal of a thermal power plant boiler and the hot spot image is 0.2, indicating high complementarity. The coefficients are normalized and mapped to the 0-1 range.

[0087] A degraded virtual scene is generated based on environmental disturbance parameters. The structural similarity index difference between the degraded virtual scene and the ideal scene is calculated to generate a rendering quality conflict map. Environmental disturbance parameters include light intensity, electromagnetic noise level, and ambient humidity. These parameters are collected in real time via IoT sensors. Light intensity is measured in lux, electromagnetic noise level is measured in decibels, and ambient humidity is measured in percentage. A generative adversarial network (GAN) is used to generate the degraded virtual scene. The generator inputs the environmental disturbance parameters and the RGB-D data of the ideal scene. The generator network is a U-Net with a four-layer convolutional encoder and a four-layer deconvolutional decoder with skip connections to preserve details. The discriminator network has five convolutional layers and outputs a scene realism score. The GAN training data consists of scene image pairs containing different environmental disturbance parameters. The training batch size is 32, and the optimizer is Adam with an initial learning rate of 0.0002 and a momentum parameter of 0.5. A gradient clipping threshold of 1.0 is used during training to prevent gradient explosion. The structural similarity index difference is calculated by comparing the brightness, contrast, and structural information of the degraded and ideal scenes. The brightness comparison function is the square of the difference between the two scene means, the contrast comparison function is the square of the product of the two scene standard deviations, and the structural information comparison function is the square of the covariance. The constant term in the calculation formula is set as a stability coefficient to avoid division by zero. The rendering quality conflict mapping value ranges from 0 to 1. A larger value indicates a more severe impact of environmental interference on rendering quality. For example, a sudden increase in light intensity will result in a conflict mapping value of 0.8. The mapping value is normalized to the range of 0-1.

[0088] When the multimodal latent variable similarity coefficient is less than or equal to the preset similarity threshold and the rendering quality conflict mapping value is greater than or equal to the preset conflict threshold, computing resources are allocated to the data fusion module driven by the large AI model. The preset similarity threshold is set based on the statistical distribution quantile at which modal complementarity is optimal in historical data. For example, the 25th percentile is chosen as the threshold of 0.3. The quantile is calculated using linear interpolation, the data distribution model is kernel density estimation, and the bandwidth parameter is set to 0.1. The preset conflict threshold is set based on the statistical distribution quantile at which rendering quality degradation is most severe in historical data. For example, the 75th percentile is chosen as the threshold of 0.6. The statistical period is 30 days, and the data sampling interval is 1 minute. Computing resource allocation includes the number of GPU cores, memory bandwidth, and processor priority. The allocation strategy uses dynamic preemptive scheduling. For example, when the conditions are met, 80% of the GPU cores are switched from the rendering pipeline module to the data fusion module. The granularity of GPU core allocation is based on streaming multiprocessors, memory bandwidth is allocated on a channel-by-channel basis, and processor priority is set using the Linux kernel's real-time scheduling class. The resource allocation switching trigger delay is required to be less than 50 milliseconds. The delay test method is to inject a step signal and measure the response time. When the delay exceeds the limit, an alarm log is triggered.

[0089] When the multimodal latent variable similarity coefficient exceeds a preset similarity threshold or the rendering quality conflict map value falls below a preset conflict threshold, computing resources are allocated to the rendering pipeline module. The resource allocation switching trigger mechanism monitors the rate of change of the similarity coefficient and conflict map value in real time. If the rate of change exceeds 5% per second, the allocation decision is immediately reassessed. The rate of change is calculated using a sliding window difference method with a window size of 10 seconds and a step size of 1 second. The resource allocation priority of the rendering pipeline module is dynamically adjusted based on scene complexity, which is calculated by the number of meshes and the amount of texture detail. For example, complex scenes are allocated 90% of the GPU cores, while simple scenes are allocated 50%. The mesh number threshold is set at 10,000 meshes, and the amount of texture detail is assessed using the anisotropic filtering level. The resource allocation strategy is validated through a combination of offline simulation and online testing. The offline simulation uses historical data playback, while the online testing injects interference signals in real time and monitors system response latency. Verification metrics include resource switching latency and scene rendering frame rate. Test data is stored in a performance monitoring database for troubleshooting and optimization analysis.

[0090] The update mechanism for the preset similarity and conflict thresholds involves periodically analyzing changes in the distribution of historical data with a 24-hour update cycle. The statistical method used is kernel density estimation, with the bandwidth parameter set to 0.1 based on the data variance and the kernel function set to a Gaussian kernel. If the difference between the old and new thresholds exceeds 10% during a threshold update, a manual review process is triggered. Upon successful review, the threshold is updated and a version number is recorded in the format of timestamp_device ID. Historical data is stored in a distributed database, which is sharded based on the device ID hash. Each shard retains the last 30 days of data, and expired data is automatically archived to cold storage. The cold storage format is compressed Parquet columnar storage, using the Snappy compression algorithm and a compression level of 9. The distributed database uses eventual consistency for reads and writes, with a maximum number of replicas of 3 and a majority write strategy for failure recovery.

[0091] The variational autoencoder was trained on spatiotemporally synchronized samples of multimodal sensory and visual data with a batch size of 64. The optimizer was Adam, with an initial learning rate of 0.001 and a learning rate decay strategy of halving every 10 epochs. Training was stopped when the validation set loss did not decrease for five consecutive epochs. The generative adversarial network was trained on pairs of scene images containing different environmental interference parameters. The gradient clipping threshold was 1.0 during training to prevent gradient explosion. Training was stopped when the ratio of the generator to discriminator loss function remained stable in the range of 0.9 to 1.1 for more than 10 epochs. The structural similarity index was calculated using a sliding window method with an 8×8 pixel window size and a sliding step size of 4 pixels. The window weights were Gaussian kernel functions with a standard deviation of 1.5 and a truncation coefficient of 3.

[0092] The exception handling mechanism during resource allocation includes the following: If the similarity coefficient or conflict mapping value exceeds a reasonable range (for example, the similarity coefficient is negative or the conflict mapping value is greater than 1), the abnormal data is discarded and replaced with the last valid value. The replacement value is valid for 5 seconds and is marked invalid after the expiration. If a GPU resource allocation conflict causes a deadlock, the resource release protocol is triggered to forcibly release occupied resources and reallocate them. The resource release protocol timeout is set to 500 milliseconds. After the timeout, an error log is recorded and the resource scheduling service is restarted. The error log is stored in a structured log format with fields including timestamp, device ID, error code, and description. The log files are archived daily. The archive file compression algorithm is GZIP, and the compression level is set to 6.

[0093] Computing resource allocation strategies are validated through a combination of offline simulation and online testing. Offline simulation uses historical data playback at rates of 1x, 2x, and 5x real-time speed to test the resource allocator's stability under varying loads. Online testing injects interference signals in real time and monitors system response latency. Interference signals include step changes in light intensity, electromagnetic noise pulses, and gradual changes in humidity. Verification metrics include resource switching latency (required to be less than 50 milliseconds) and scene rendering frame rate (required to be greater than 30 fps). Test data is stored in a performance monitoring database with a B+ tree index structure and a cost model optimization query optimization strategy.

[0094] Step S4 dynamically allocates computing resources by evaluating multimodal correlation and environmental interference. In existing layered architectures, resource competition between AI reasoning and rendering modules leads to reduced real-time performance. For example, high-load rendering tasks may preempt AI analysis resources, delaying abnormal responses. Step S4 quantifies modal complementarity through latent variable similarity coefficients, evaluates environmental interference based on rendering quality conflict mapping values, and employs hierarchical decision logic (such as "and / or" conditions). Compared to existing technologies, this strategy can dynamically identify critical tasks (such as prioritizing AI analysis when modal complementarity is low), while adjusting rendering quality based on environmental interference, avoiding resource waste caused by static rules. Resource allocation integrates multimodal correlations and environmental status, driving decisions through dual indicators to achieve a balance between computing load and real-time requirements.

[0095] S5. The data fusion module uses the AI ​​big model to perform cross-modal correlation analysis on the spatiotemporal synchronized data to generate device status feature data and abnormal area mask data. The specific implementation is as follows:

[0096] A multi-head cross-attention mechanism is used to model cross-modal correlations among multimodal features in spatiotemporally synchronized data, generating a fused feature vector. The multi-head cross-attention mechanism uses 8 attention heads, each with a dimension of 64. Input features include the mean temperature, root mean square vibration value, and color histogram from the spatiotemporally synchronized data. A linear transformation is performed to generate a query vector, a key vector, and a value vector. The weight matrix of the linear transformation has a dimension of the input feature dimension × 64, and the bias is initialized to zero. Attention weights are calculated by taking the dot product of the query vector and the key vector, scaling by the inverse square root of the attention head dimension. Softmax normalization is then applied and weighted summed with the value vector to output a fused feature vector. The fused feature vector has a dimension of 512. Layer normalization and residual connections are used to optimize training stability. The parameters of layer normalization are learnable vectors, and the skip paths of residual connections are identity mappings.

[0097] Cosine similarity is calculated between the fused feature vector and the pre-trained embedding vector of the device's normal operating state to generate anomaly confidence scores for the device status feature data. The pre-trained embedding vector is trained using historical normal data, which consists of spatiotemporally synchronized data from the device in a fault-free state. The training method is an autoencoder. The encoder network consists of three fully connected layers, 256 hidden nodes, a GELU activation function, and a latent vector dimension of 128. The decoder network is symmetrical, and the reconstruction loss function is the mean squared error. Cosine similarity is calculated by L2-normalizing the fused feature vector and the pre-trained embedding vector, followed by a dot product. The L2 normalization formula is to divide each vector element by its modulus, resulting in a scalar between -1 and 1. Anomaly confidence scores are generated by linearly mapping the cosine similarity to the range 0 to 1 using the (similarity + 1) / 2 mapping function. The anomaly confidence threshold is set to the 95th percentile of historical anomaly samples, for example, a threshold of 0.85. The quantile is calculated using linear interpolation. The historical data coverage period is 6 months, and the data sampling interval is 1 minute.

[0098] The spatially gridded image data is segmented using a sliding window. The gradient distribution of texture and depth features within the window is extracted to generate candidate masks for abnormal regions. The sliding window size is 64×64 pixels, with a sliding step size of 32 pixels. The window size and step size are determined based on the minimum abnormal region size of the target device. For example, the minimum visible size of a crack in a transmission line insulator is 16×16 pixels. The gradient distribution of texture features is calculated using the Sobel operator to calculate the horizontal and vertical gradient magnitudes. The convolution kernel size of the Sobel operator is 3×3, with the horizontal kernel in the range [-1, 0, 1; -2, 0, 2; -1, 0, 1] and the vertical kernel in the range [-1, -2, -1; 0, 0, 0; 1, 2, 1]. The gradient magnitude is calculated as the square root of the sum of the squared horizontal and vertical gradients, normalized to a range of 0 to 255. The gradient distribution of depth features is calculated by calculating the absolute value of the depth difference between adjacent pixels. The depth difference is expressed in millimeters and normalized to a range of 0 to 10,000 mm, corresponding to the maximum range of the depth sensor. The method for generating candidate masks for abnormal areas is as follows: when the weighted sum of texture gradient and depth gradient exceeds the preset threshold, it is marked as a candidate abnormal area. The weight coefficient is set to 0.6 and 0.4 according to the abnormal contribution of texture and depth in historical data. For example, the contribution of texture gradient in crack detection is higher. The preset threshold is the 99% percentile of historical normal samples, and the threshold calculation data window is the past 30 days.

[0099] Morphological closing operations are used to optimize candidate masks of abnormal regions, eliminate isolated noise points, and fill holes to generate abnormal region mask data. The structuring element of the morphological closing operation is a 3×3 rectangular kernel with all elements set to 1. A dilation operation is first performed to fill holes. The dilation operation is defined as traversing the center of the structuring element through the image. If at least one pixel in the area covered by the center is 1, the output pixel is set to 1. An erosion operation is then performed to smooth the boundary. The erosion operation is defined as setting the output pixel to 1 only when the area covered by the structuring element is completely 1. The number of iterations of the dilation and erosion operations is 1. Isolated regions smaller than 5 pixels are removed. Isolated regions are identified using a connected component labeling algorithm with an 8-neighborhood rule. The abnormal region mask data is stored as a binary image, with a pixel value of 1 indicating an abnormal region and a pixel value of 0 indicating a normal region. The image resolution is consistent with the original spatially gridded image data. Metadata tags are attached during storage, including a timestamp, device ID, and anomaly confidence score.

[0100] The update mechanism for pre-trained embeddings includes periodic retraining every 7 days, with the training data automatically expanded to include newly added normal data. This new data is verified as normal by validation rules, including that sensor readings are within historically normal ranges and that the visual data contains no anomalous features. The autoencoder is trained with the AdamW optimizer, with an initial learning rate of 0.0001, a weight decay factor of 0.01, a training batch size of 32, and early stopping when the validation set loss does not decrease for 10 consecutive epochs. The anomaly confidence threshold is dynamically adjusted by calculating the mean and standard deviation of the anomaly confidence distribution over the past 30 days. When the standard deviation of the new data distribution exceeds 10% of the historical standard deviation, the quantile threshold is recalculated, using the data window from the past 7 to 30 days.

[0101] During the sliding window segmentation process, if the image boundary region is detected to be unable to fully cover the window, the boundary is extended using a mirror padding method with a padding width of 16 pixels, with the axis of symmetry of the mirror padding being the image boundary. The kernel size of the morphological closing operation is dynamically adjusted based on the image resolution. For example, when the resolution is greater than 1920×1080, the kernel size is increased to 5×5 to maintain a balance between processing efficiency and accuracy. During the optimization process of the candidate mask for the abnormal region, if a continuous abnormal region with an area less than 10 pixels is detected, it is determined to be noise and removed. The area is calculated by counting pixels, and the removal operation updates the mask value to 0.

[0102] The training data for the multi-head cross-attention mechanism consists of labeled cross-modal correlation samples. The labels are manually annotated feature correlation strengths, with correlation strengths ranging from 0 to 5, with 0 indicating no correlation and 5 indicating strong correlation. The training loss function is a weighted sum of mean squared error and cross entropy, with weight coefficients set to 0.7 and 0.3, respectively, based on the relative importance of modal data in historical anomaly detection. The Sobel operator convolution kernel used in the gradient distribution calculation is separated into horizontal and vertical directions. The gradient amplitude is synthesized after convolution of the image separately. The synthesized gradient image is then denoised using a Gaussian filter with a 5×5 kernel size and a standard deviation of 1.0.

[0103] Anomaly confidence is verified through ROC curve analysis. The true positive rate and false positive rate are calculated based on a test dataset consisting of 1,000 normal samples and 500 abnormal samples. The samples are sourced from historical inspection records and artificially simulated anomaly injection. The accuracy of the anomaly region mask data is evaluated using the intersection-over-union (IoU) metric. The IoU threshold is set at 0.5. Regions above the threshold are considered valid anomaly regions. The evaluation results are recorded in the performance monitoring report.

[0104] A time series database is used to store historical anomaly samples. The database sharding strategy is based on device ID hashing, with each shard storing anomaly data for a single device. The data retention policy is to automatically archive to cold storage after 180 days. The cold storage format is compressed Parquet files, using the Snappy compression algorithm and a compression level of 9. The time series database uses a RESTful API for read and write operations. B+ tree indexes are used for query optimization, with timestamp and device ID as the index fields. The transaction isolation level is Read Committed.

[0105] S6. The rendering pipeline module integrates the equipment abnormality feature data and the defect area mask data into the 3D inspection scene to generate a virtual and real inspection interactive interface. The specific implementation is as follows:

[0106] The abnormal area mask data is converted into a transparency texture on the 3D model surface and then mapped to the target device's 3D inspection scene via UV mapping. The abnormal area mask data is a binary image, with pixel values ​​of 1 representing abnormal areas and 0 representing normal areas. The transparency texture is generated by setting the transparency of areas corresponding to mask pixel values ​​of 1 to 0.8 and areas corresponding to pixel values ​​of 0 to 0.2. The transparency values ​​are set based on the statistical quantile of historical normal sample mask data, for example, the 99th percentile of transparency for normal areas is 0.2. The statistical quantile calculation period is 30 days, and the data sampling interval is 1 minute. The UV mapping process involves aligning the pixel coordinates of the 2D mask image with the texture coordinates of the 3D model surface. The texture coordinate mapping rule is to project the 3D model vertex coordinates onto a 2D plane and then normalize them to the range of 0-1. The projection plane is determined based on the device's primary inspection viewpoint. For example, the projection plane for a thermal power plant boiler is a front orthogonal view. The projection plane is selected based on the installation position and angle of the device inspection camera.

[0107] The mean temperature and RMS vibration values ​​from the equipment status feature data are converted into a thermal cloud map and a vibration intensity field, which are then overlaid on the corresponding spatial coordinates of the 3D inspection scene. The thermal cloud map is generated by mapping the mean temperature to a predefined color gradient ranging from blue (0°C) to red (300°C). The mapping function is linear interpolation, with an interpolation step size of 10°C, based on the equipment's safe operating temperature range. The color gradient is dynamically adjusted based on the upper thermal tolerance limit of the equipment material. For example, if the upper temperature limit of boiler steel is 500°C, the gradient range will expand from blue (0°C) to dark red (500°C). The vibration intensity field is generated by converting the RMS vibration value into arrow length and color intensity. The arrow direction is determined by the vibration sensor's installation orientation. The color intensity range is green (0g) to red (20g), corresponding to the equipment's vibration safety threshold. The arrow length is converted to vibration amplitude at a ratio of 0.1 meter per 1g. The overlay process is implemented through the shader program of the 3D rendering engine. The vertex shader binds temperature and vibration data to the model vertices, and the fragment shader renders colors and arrows based on the interpolation results. The shader code is written in HLSL language and the compilation target is Shader Model 6.5.

[0108] Through depth testing and transparency blending, defect area mask data is fused with thermal cloud images and vibration intensity fields to create a virtual-real overlay inspection interface. The depth buffer comparison function for depth testing is set to less than or equal to ensure that the foreground anomaly overlaps the background model. The depth buffer precision is 24 bits, and the stencil buffer precision is 8 bits. The blending factor for transparency blending is the source transparency (source factor is SrcAlpha) and the target transparency (target factor is OneMinusSrcAlpha). The blending formula is the source color multiplied by the source transparency plus the target color multiplied by (1 minus the source transparency). The blended color channels are adjusted using gamma correction, with a gamma value of 2.2. The fused inspection interface outputs frame buffer data in RGBA format, with a resolution matching the display device, for example, 1920×1080 pixels, a refresh rate of 60Hz, a color depth of 10 bits, and HDR metadata compliant with the ST 2084 standard.

[0109] The transparency threshold of the texture is set based on the statistical quantile of the mask data of historical normal samples. The quantile is calculated using the R-7 algorithm with a domain of 0 to 1. The transparency threshold of the abnormal area is set at the 99.9th percentile of the historical normal samples. The trigger condition for the dynamic adjustment of the color mapping gradient of the thermal cloud map is the material change event in the equipment maintenance record. The event is subscribed to via the MQTT protocol and the rendering parameters are automatically updated. The parameter update delay is required to be less than 100 milliseconds. The hardware requirements of the 3D rendering engine are NVIDIA RTX 4090 GPU with 24GB of video memory. The software requirements are Unity 2022.3.20f1, the rendering pipeline is the Universal Render Pipeline (URP), and the shader compilation tool is Shader Graph 12.1.7.

[0110] During UV mapping, if a mismatch between the texture coordinates of the 3D model surface and the mask image is detected, the texture coordinates are expanded using a mirror fill method with a fill width of 16 pixels and a fill direction along the boundary of the UV-expanded model surface. The filled texture coordinates are then smoothly transitioned using bilinear interpolation. When temperature data is missing, color interpolation for the thermal cloud image switches to neighboring pixel interpolation. The interpolation window size is 3×3 pixels, with a weight of 0.5 for the center pixel and 0.125 for adjacent pixels. The interpolated result is denoised using a Gaussian filter with a Gaussian kernel standard deviation of 1.0. When the vibration data is abnormal (e.g., the amplitude exceeds 20g), the arrow rendering of the vibration intensity field switches to a flashing warning icon with a red color and a flashing frequency of 2Hz. The flashing period is controlled by a timer interrupt with an interrupt accuracy of 1 millisecond.

[0111] The verification method for the virtual-reality inspection interactive interface is to compare the rendering results with manually annotated ground truth data. The ground truth data includes the location of abnormal areas, temperature values, and vibration amplitudes. The comparison metrics are pixel-level accuracy and intersection-over-union (IoU) ratio. The accuracy threshold is set at 95%, and the IoU threshold is set at 0.8. The test dataset contains 1000 sets of normal samples and 500 sets of abnormal samples. The samples are sourced from historical inspection records and artificially simulated anomaly injection. Rendering performance metrics include frame rate stability (required ≥98%) and latency (required ≤20ms). The test data is stored in a performance monitoring database. The database index is a composite key of timestamp and device ID, and query optimization uses B+ tree indexes.

[0112] The transparency texture update mechanism involves asynchronously updating the texture map after receiving new mask data in real time. The update interval is 1 second, the texture compression format is BC7, the compression level is high quality, and the compression algorithm is accelerated by GPU hardware. Dynamic data of the thermal cloud map and vibration intensity field is transmitted through a circular buffer with a capacity of 60 frames. The overflow handling strategy is to discard the oldest frame. The buffer read and write pointers are thread-safe through atomic operations. The model loading of the 3D inspection scene uses Level of Detail technology. The level of detail of the close-range model is 4K texture, and the level of detail of the long-range model is 1K texture. The switching distance is dynamically calculated based on the focal length of the viewing angle. The focal length data is obtained in real time through the viewing cone parameters of the rendering camera.

[0113] This embodiment achieves systematic optimization of industrial inspection efficiency through the spatiotemporal alignment of multimodal data and a dynamic resource allocation mechanism. Existing technologies generally rely on independent modules to process sensor data and visual data, and achieve functional stacking through fixed clock synchronization and static resource allocation, resulting in heterogeneous data processing delays and resource competition conflicts. This embodiment replaces hardware clock synchronization with cross-modal causal reasoning parameters, deriving spatiotemporal offsets from data correlation, and solves the spatiotemporal reference misalignment problem caused by environmental interference between sensors and visual devices; at the same time, it combines the dual decision logic of latent variable similarity coefficients and rendering quality conflict mapping values ​​to dynamically allocate computing resources. Compared with conventional solutions, this embodiment transforms multimodal data processing from isolated execution to collaborative optimization through data-driven dynamic calibration and adaptive decision-making mechanisms, significantly reducing end-to-end latency and improving the real-time response of exceptions. Through the coupled analysis of cross-modal association modeling and environmental interference, a dynamic feedback loop of data-resource-scenario is constructed.

[0114] The calculations involved in the embodiments are all dimensionless numerical calculations, and the preset parameters and thresholds in the calculations are set by those skilled in the art according to actual conditions.

[0115] It should be noted that the present invention can be deployed on the device itself to implement embedded applications, and can also be run on a PC or other terminal with a user interface, thereby meeting various hardware environments and usage requirements.

[0116] The above embodiments can be implemented in whole or in part via software, hardware, firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in the embodiments of this application are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0117] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0118] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0119] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0120] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0121] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0122] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

[0123] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An intelligent inspection method based on the integration of the Internet of Things and virtual reality based on an AI big model, characterized by: include: S1. Collect multi-source sensor data and visual data of the target device through the IoT terminal; S2. Perform cross-modal feature extraction and format unification preprocessing on multi-source sensor data and visual data to generate structured time series data and spatial gridded image data; S3. Based on the cross-modal causal reasoning parameters and the spatial coordinate system transformation matrix, the structured time series data and the spatial gridded image data are temporally and spatially aligned to generate spatiotemporally synchronized data; S4. Dynamically allocate computing resources to the data fusion module or rendering pipeline module driven by the AI ​​large model based on the correlation of multimodal data in the spatiotemporal synchronization data and the impact of environmental interference on rendering quality, including: Extract the latent variable distribution of multimodal data in spatiotemporal synchronized data, calculate the Wasserstein distance between latent variable distributions, and generate multimodal latent variable similarity coefficients; Generate a degraded virtual scene based on environmental interference parameters, calculate the structural similarity index difference between the degraded virtual scene and the ideal scene, and generate the rendering quality conflict mapping RGB value; When the multimodal latent variable similarity coefficient is less than or equal to the preset similarity threshold and the rendering quality conflict mapping value is greater than or equal to the preset conflict threshold, computing resources are allocated to the data fusion module driven by the AI ​​large model; When the multimodal latent variable similarity coefficient is greater than a preset similarity threshold or the rendering quality conflict mapping value is less than a preset conflict threshold, computing resources are allocated to the rendering pipeline module; The rendering quality conflict mapping value indicates the degree of influence of environmental interference on rendering quality; S5. The data fusion module uses the AI ​​large model to perform cross-modal correlation analysis on spatiotemporal synchronized data to generate device status feature data and abnormal area mask data, including: The multi-head cross-attention mechanism is used to model the cross-modal association of multimodal features in spatiotemporal synchronized data and generate a fused feature vector. Calculate the cosine similarity between the fused feature vector and the pre-trained embedding vector of the normal operation status of the device to generate the anomaly confidence in the device status feature data; The spatial gridded image data is segmented by a sliding window, the gradient distribution of texture features and depth features within the window is extracted, and candidate masks of abnormal areas are generated; Morphological closing operation is used to optimize candidate masks of abnormal regions, eliminate isolated noise points and fill in empty regions to generate abnormal region mask data; The abnormality confidence is compared with the preset abnormality confidence threshold. When the abnormality confidence exceeds the threshold, it is marked as an abnormal state of the device; S6. The rendering pipeline module integrates the equipment abnormality feature data and the defect area mask data into the three-dimensional inspection scene to generate a virtual and real superimposed inspection interactive interface.

2. The intelligent inspection method based on the integration of the Internet of Things and virtual reality based on the AI ​​big model according to claim 1 is characterized in that: Collect multi-source sensor data and visual data from target devices through IoT terminals, including: The temperature, vibration and current signals of the target device are synchronously collected through temperature sensors, vibration sensors and current sensors as multi-source sensing data; The RGB camera and depth camera are used to synchronously collect RGB images and depth point clouds of the target device as visual data; Based on the industrial Internet of Things clock synchronization protocol, the temperature signal, vibration signal, current signal, RGB image and depth point cloud are timestamp aligned to generate time-synchronized multi-source sensor data and visual data.

3. The intelligent inspection method based on the integration of the Internet of Things and virtual reality based on the AI ​​big model according to claim 1 is characterized in that: Perform cross-modal feature extraction and format unification preprocessing on multi-source sensor data and visual data to generate structured time series data and spatial gridded image data, including: Time domain feature extraction is performed on the temperature signal, vibration signal and current signal in the multi-source sensor data to generate a time series feature vector containing the mean, variance and spectrum amplitude; Perform spatial grid division on the RGB image in the visual data, extract the texture features and color histogram of each grid area, and generate spatial gridded image data; Perform voxel processing on the depth point cloud in the visual data, map the three-dimensional point cloud to a two-dimensional spatial grid, and supplement the depth information of the spatial gridded image data; The time series feature vectors and spatial gridded image data are normalized separately, unified to the same numerical range, and then stored as structured time series data and spatial gridded image data.

4. The intelligent inspection method based on the integration of the Internet of Things and virtual reality based on the AI ​​big model according to claim 1 is characterized in that: Based on cross-modal causal inference parameters and spatial coordinate system transformation matrices, structured time series data and spatial gridded image data are temporally and spatially aligned to generate spatiotemporally synchronized data, including: Through causal correlation analysis of multi-source sensor data and visual data, time offset is extracted to generate cross-modal causal reasoning parameters; Calculate the three-dimensional space coordinate system transformation matrix of the target device based on the calibration plate feature point matching; Compensate the time offset to the timestamp of structured time series data to generate a time-synchronized sensor data stream; Applying a three-dimensional space coordinate system transformation matrix to the spatially gridded image data to generate a spatially aligned visual data stream; Fusing temporally synchronized sensory data streams with spatially aligned visual data streams to generate spatiotemporally synchronized data.

5. The intelligent inspection method based on the integration of the Internet of Things and virtual reality based on the AI ​​big model according to claim 1 is characterized in that: The preset similarity threshold and the preset conflict threshold are set according to the statistical distribution quantile when the modal complementarity in the historical data is optimal.

6. The intelligent inspection method based on the integration of the Internet of Things and virtual reality based on the AI ​​big model according to claim 1 is characterized in that: The pre-trained embedding vector is generated by training historical normal data, and the anomaly confidence threshold is set according to the statistical distribution quantile of historical anomaly samples.

7. The intelligent inspection method based on the integration of the Internet of Things and virtual reality based on the AI ​​big model according to claim 1 is characterized in that: The rendering pipeline module integrates equipment anomaly feature data and defect area mask data into a 3D inspection scene, generating a virtual-real inspection interactive interface, including: Convert the abnormal area mask data into a transparency texture on the 3D model surface and fit it to the 3D inspection scene of the target device through UV mapping; Convert the temperature mean and vibration root mean square value in the equipment status characteristic data into a thermal cloud map and vibration intensity field, and superimpose them on the corresponding spatial coordinates of the three-dimensional inspection scene; Through depth testing and transparency mixed rendering, the defect area mask data is fused with the thermal cloud map and vibration intensity field to generate a virtual and real superimposed inspection interactive interface.

8. The intelligent inspection method based on the integration of the Internet of Things and virtual reality based on the AI ​​big model according to claim 7 is characterized in that: The threshold of the transparency texture is set according to the statistical quantile of the mask data of historical normal samples, and the color mapping gradient of the thermal cloud map is dynamically adjusted according to the safe operating temperature range of the equipment.

Images