Internet of Things and virtual reality fused intelligent inspection method based on AI large model

Abstract

The invention discloses an intelligent inspection method for fusion of Internet of Things and virtual reality based on an AI large model, particularly relates to the technical field of industrial intelligent inspection, and is used for solving the problem of end-to-end response lag caused by multi-modal data fusion delay and resource competition under an existing layered architecture. The method comprises the following steps: synchronously acquiring heterogeneous data of target equipment through a multi-source sensor and visual equipment, and generating time-space synchronous data through cross-modal feature extraction and time-space alignment; computing resources are dynamically allocated to an AI model or a rendering pipeline in combination with cross-modal correlation analysis and environmental interference assessment, and key tasks are preferentially guaranteed to be executed; performing deep correlation reasoning on the multi-modal data by using an AI large model, and generating equipment state features and an abnormal region mask; and finally, superposing the abnormal features and the three-dimensional scene through a virtual-real fusion rendering technology to form a visual interaction interface. Efficient fusion and real-time interaction of multi-modal data are realized, and the accuracy of anomaly detection and the decision response efficiency of an operator are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial intelligent inspection, and more specifically, to an intelligent inspection method based on the integration of the Internet of Things and virtual reality with an AI large model. Background Art

[0002] In the field of industrial intelligent inspection, Internet of Things technology realizes multi-dimensional perception of equipment status by deploying multi-source sensors and vision devices, and combines artificial intelligence models to automatically analyze the collected data and detect anomalies. At the same time, virtual reality technology is introduced to provide an immersive interaction interface to assist operators in remotely viewing equipment status and making decisions. In the prior art, Internet of Things data collection, artificial intelligence analysis, and virtual reality interaction are usually implemented in an independent modular manner, and data transmission and task scheduling are completed through a hierarchical architecture.

[0003] However, due to the resource competition contradiction between the heterogeneous characteristics of Internet of Things multi-modal data (including structured sensing data and unstructured visual data) and the high real-time requirements of virtual reality rendering, the existing hierarchical architecture is difficult to achieve the collaborative optimization of data fusion, analysis and reasoning, and interactive rendering, resulting in a significant increase in the end-to-end delay from data collection to visual feedback, thereby reducing the operator's real-time response ability to equipment abnormal status and restricting the effectiveness of the intelligent inspection system. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an intelligent inspection method based on the integration of the Internet of Things and virtual reality with an AI large model to solve the problems proposed in the above background art.

[0005] To achieve the above object, the present invention provides the following technical solutions: An intelligent inspection method based on the integration of the Internet of Things and virtual reality with an AI large model, comprising: S1. Collect multi-source sensing data and visual data of a target device through an Internet of Things terminal; S2. Perform cross-modal feature extraction and format unification preprocessing on the multi-source sensing data and visual data to generate structured time-series data and spatially gridded image data; S3. Align the structured time-series data and spatially gridded image data in space and time based on cross-modal causal inference parameters and a spatial coordinate system transformation matrix to generate spatio-temporally synchronized data; S4. Dynamically allocate computing resources to an AI large model-driven data fusion module or a rendering pipeline module according to the relevance of multi-modal data in the spatio-temporally synchronized data and the impact of environmental interference on the rendering quality; S5. The data fusion module performs cross-modal correlation analysis on the spatio-temporally synchronized data through the AI large model to generate equipment status feature data and abnormal area mask data; S6. The rendering pipeline module fuses the device anomaly feature data and the defect area mask data into the 3D inspection scene to generate an inspection interaction interface with virtual-real superposition.

[0006] In a preferred embodiment, multi-source sensing data and visual data of the target device are collected through an Internet of Things terminal, including: The temperature signal, vibration signal, and current signal of the target device are synchronously collected through a temperature sensor, a vibration sensor, and a current sensor as multi-source sensing data; The RGB image and depth point cloud of the target device are synchronously collected through an RGB camera and a depth camera as visual data; Based on the industrial Internet of Things clock synchronization protocol, time stamps are aligned for the temperature signal, vibration signal, current signal, RGB image, and depth point cloud to generate time-synchronized multi-source sensing data and visual data.

[0007] In a preferred embodiment, cross-modal feature extraction and format unification preprocessing are performed on the multi-source sensing data and visual data to generate structured time-series data and spatially gridded image data, including: Time-domain feature extraction is respectively performed on the temperature signal, vibration signal, and current signal in the multi-source sensing data to generate time-series feature vectors including mean, variance, and spectral amplitude; Spatial grid division is performed on the RGB image in the visual data, and the texture features and color histograms of each grid area are extracted to generate spatially gridded image data; Voxelization processing is performed on the depth point cloud in the visual data to map the 3D point cloud to a 2D spatial grid and supplement the depth information of the spatially gridded image data; The time-series feature vectors and the spatially gridded image data are respectively normalized and unified to the same numerical range and then stored as structured time-series data and spatially gridded image data.

[0008] In a preferred embodiment, spatio-temporal alignment is performed on the structured time-series data and the spatially gridded image data based on cross-modal causal inference parameters and a spatial coordinate system transformation matrix to generate spatio-temporally synchronized data, including: The time offset is extracted through the causal correlation analysis of the multi-source sensing data and the visual data to generate cross-modal causal inference parameters; The 3D spatial coordinate system transformation matrix of the target device is calculated based on the calibration board feature point matching; The time offset is compensated to the time stamp of the structured time-series data to generate a time-synchronized sensing data stream; The 3D spatial coordinate system transformation matrix is applied to the spatially gridded image data to generate a spatially aligned visual data stream; Fuse the sensor data stream with time synchronization and the visual data stream with spatial alignment to generate spatio-temporal synchronized data.

[0009] In a preferred embodiment, according to the correlation of multimodal data in the spatio-temporal synchronized data and the impact of environmental interference on the rendering quality, dynamically allocate computing resources to the data fusion module or the rendering pipeline module driven by the AI large model, including: Extract the latent variable distribution of multimodal data in the spatio-temporal synchronized data, calculate the Wasserstein distance between the latent variable distributions, and generate a multimodal latent variable similarity coefficient; Generate a degraded virtual scene based on the environmental interference parameters, calculate the difference in the structural similarity index between the degraded virtual scene and the ideal scene, and generate a rendering quality conflict mapping value; When the multimodal latent variable similarity coefficient is less than or equal to the preset similarity threshold and the rendering quality conflict mapping value is greater than or equal to the preset conflict threshold, allocate computing resources to the data fusion module driven by the AI large model; When the multimodal latent variable similarity coefficient is greater than the preset similarity threshold or the rendering quality conflict mapping value is less than the preset conflict threshold, allocate computing resources to the rendering pipeline module.

[0010] In a preferred embodiment, the preset similarity threshold and the preset conflict threshold are set according to the statistical distribution quantiles when the modal complementarity is optimal in the historical data.

[0011] In a preferred embodiment, the data fusion module performs cross-modal correlation analysis on the spatio-temporal synchronized data through the AI large model to generate device status feature data and abnormal area mask data, including: Perform cross-modal correlation modeling on the multimodal features in the spatio-temporal synchronized data through the multi-head cross-attention mechanism to generate a fused feature vector; Calculate the cosine similarity based on the fused feature vector and the pre-trained embedding vector of the normal operating state of the device to generate the abnormal confidence in the device status feature data; Segment the spatial grid image data through a sliding window, extract the gradient distributions of the texture features and depth features within the window, and generate a candidate abnormal area mask; Optimize the candidate abnormal area mask using morphological closing operations to eliminate isolated noise points and fill in the hole areas, and generate abnormal area mask data; Compare the abnormal confidence with the preset abnormal confidence threshold, and mark it as the abnormal state of the device when the abnormal confidence exceeds the threshold.

[0012] In a preferred embodiment, the pre-trained embedding vector is generated by training with historical normal data, and the abnormal confidence threshold is set according to the statistical distribution quantiles of historical abnormal samples.

[0013] In a preferred embodiment, the rendering pipeline module fuses the device anomaly feature data and the defect area mask data into the 3D inspection scene to generate a virtual-real superposed inspection interaction interface, including: Converting the anomaly area mask data into a transparency texture on the surface of the 3D model and fitting it to the 3D inspection scene of the target device through UV mapping; Converting the temperature mean value and the vibration root mean square value in the device status feature data into a thermal cloud map and a vibration intensity field, and superimposing them on the corresponding spatial coordinates of the 3D inspection scene; Fusing the defect area mask data with the thermal cloud map and the vibration intensity field through depth test and transparency blending rendering to generate a virtual-real superposed inspection interaction interface.

[0014] In a preferred embodiment, the threshold of the transparency texture is set according to the statistical quantile of the mask data of historical normal samples, and the color mapping gradient of the thermal cloud map is dynamically adjusted according to the device safe operating temperature range.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. Through the deep collaboration and dynamic resource optimization mechanism of cross-modal data, the data processing efficiency and the real-time virtual-real interaction in the industrial inspection scene are significantly improved. Aiming at the heterogeneous characteristics of multi-source sensing data and visual data, the spatio-temporal alignment technology is used to break through the collaboration bottleneck of the traditional hierarchical architecture. Through the joint calibration of the causal inference parameters and the spatial coordinate transformation matrix, the precise synchronization of multi-modal data on the time stamp and the spatial reference is realized. Compared with the modular processing flow, the collaboration mechanism effectively eliminates the fusion delay caused by data islands, enabling the sensing signals such as temperature and vibration to be mapped to the unified spatio-temporal framework in real time, providing a highly consistent input basis for subsequent analysis. At the same time, based on the dynamic resource allocation strategy, the system can adaptively adjust the computing power distribution according to the multi-modal association strength and the environmental interference degree, and give priority to responding to key abnormal events on the premise of ensuring the rendering quality, thus greatly shortening the full-link delay from data acquisition to decision feedback.

[0016] 2. Through the interactive optimization design of virtual-real fusion, the closed-loop control ability of anomaly detection and on-site feedback during the inspection process is strengthened. With the help of the AI large model's ability to extract cross-modal correlation features, the system can accurately identify subtle anomalies in the equipment operation state and generate defect area masks with high confidence. Combining with the depth synthesis technology of the 3D rendering engine, the abstract data features are transformed into an intuitive virtual-real overlay scene, enabling complex parameters such as temperature gradients and vibration intensities to be projected onto the equipment model in real time in the form of thermal cloud maps, vector fields, etc. Operators can quickly locate problems without relying on multi-screen switching or abstract reports. At the same time, the adaptive rendering pipeline ensures the front visibility of key information through transparency blending and depth testing, maintaining the stability and smoothness of the interactive interface under complex working conditions and providing a highly immersive decision-making environment for remote inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flowchart of the intelligent inspection method based on the AI large model for the integration of the Internet of Things and virtual reality of the present invention; Figure 2 is a flowchart of the dynamic resource allocation decision driven by multi-modal data of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0019] Embodiment: Figure 1 An intelligent inspection method based on the AI large model for the integration of the Internet of Things and virtual reality of the present invention is given, including the following steps: S1. Collect multi-source sensing data and visual data of the target device through the Internet of Things terminal; S2. Perform cross-modal feature extraction and format unification preprocessing on the multi-source sensing data and visual data to generate structured time-series data and spatially gridded image data; S3. Perform spatio-temporal alignment on the structured time-series data and spatially gridded image data based on the cross-modal causal inference parameters and the spatial coordinate system transformation matrix to generate spatio-temporally synchronized data; S4. Dynamically allocate computing resources to the data fusion module or the rendering pipeline module driven by the AI large model according to the correlation of multi-modal data in the spatio-temporally synchronized data and the influence of environmental interference on the rendering quality; S5. The data fusion module performs cross-modal correlation analysis on the spatio-temporally synchronized data through the AI large model to generate device state feature data and anomaly area mask data; S6. The rendering pipeline module fuses the device abnormal feature data and the defect area mask data into the 3D inspection scene to generate an inspection interaction interface with virtual-real superposition.

[0020] S1. Collect multi-source sensing data and visual data of the target device through the Internet of Things terminal. The specific implementation is as follows: Synchronously collect the temperature signal, vibration signal, and current signal of the target device through a temperature sensor, a vibration sensor, and a current sensor as multi-source sensing data. The temperature sensor is a thermocouple type sensor, installed on the surface of the hot spot area of the target device. The measurement range of the temperature signal is set according to the normal operating temperature range of the device and the safety threshold under extreme working conditions. For example, the temperature range of the boiler pipeline in a thermal power plant is set to -50°C to 300°C. The temperature signal eliminates high-frequency noise through a low-pass filter, and the filter cut-off frequency is determined according to the relationship between the heat conduction characteristics of the device and the signal sampling rate. Specifically, it is calculated by fc = 1 / 2πRC, where fc is the filter cut-off frequency, R is the resistance value of the filter, and C is the capacitance value. In actual applications, it is adjusted according to the thermal inertia time constant of the device.

[0021] The vibration sensor is a piezoelectric acceleration sensor, installed at the vibration-sensitive part of the target device. The frequency range of the measured vibration signal covers the mechanical resonance frequency of the device and the integer multiples of the running fundamental frequency. For example, the frequency range of the vibration signal of the steam turbine bearing is set to 0.1 Hz to 5 kHz. The vibration signal is converted into a vibration acceleration value through the root mean square algorithm, and the effective amplitude range of the vibration acceleration value is set according to the vibration safety standard of the device. For example, it is 0.1g to 20g specified in ISO 10816-3. The current sensor is a Hall effect type sensor, installed at the power cable of the target device. The measurement range of the current signal is determined according to the peak value of the rated current and short-circuit current of the device. For example, the current range of the distribution transformer is set to 0 A to 1000 A. The current signal eliminates power frequency interference through an anti-aliasing filter, and the filter order and cut-off frequency are selected according to the signal bandwidth and the sampling theorem. For example, a 4th-order Butterworth filter is used, and the cut-off frequency is 60 Hz.

[0022] The RGB camera and the depth camera are synchronized to collect the RGB image and the depth point cloud of the target device as visual data. The RGB camera is a global shutter industrial camera, and its resolution and frame rate are set according to the size and motion speed of the target device. For example, for the power transmission line inspection scenario, the resolution is set to 1920×1080 pixels and the frame rate is set to 30fps. The installation position and angle are determined by three-dimensional space calibration to ensure coverage of the key detection areas of the target device. The RGB image is adjusted for brightness and color consistency through automatic white balance and exposure compensation algorithms, and the white balance parameters are dynamically adjusted according to the ambient light intensity and color temperature. The depth camera is a time-of-flight sensor, and the depth measurement accuracy and working distance are set according to the target surface reflectivity and detection accuracy requirements. For example, the measurement accuracy of a metal pipe is set to ±1mm, and the working distance is set to 0.5 meters to 5 meters. The depth point cloud data is converted into three-dimensional space coordinates with the center of the target device as the origin through a coordinate system alignment algorithm. The coordinate system alignment algorithm calculates the homography matrix based on the coordinates of the feature points on the calibration board. The calibration board is a checkerboard pattern with a size of 30cm×30cm and a feature point spacing of 5cm.

[0023] Based on the industrial Internet of Things clock synchronization protocol, the temperature signal, vibration signal, current signal, RGB image, and depth point cloud are timestamp-aligned to generate time-synchronized multi-source sensing data and visual data. The industrial Internet of Things clock synchronization protocol uses the IEEE 1588 Precision Time Protocol. The master clock source is deployed at the Internet of Things gateway, and the clock synchronization accuracy is set according to the network latency and device jitter tolerance. For example, the master-slave clock synchronization period is set to 1 second, and the clock offset compensation threshold is set to ±10 milliseconds. When the threshold is exceeded, a clock resynchronization is triggered. The timestamp alignment process includes the following steps: offset compensation is performed on the timestamps of the temperature signal, vibration signal, and current signal with respect to the time reference of the master clock source, and the offset is calculated through a linear regression algorithm with a regression window of the data of the last 10 synchronization periods; the acquisition timestamps of the RGB image and the depth point cloud are aligned to a unified time reference through an interpolation algorithm. The interpolation algorithm is selected based on the requirements of data continuity and real-time performance. For example, the cubic spline interpolation method is used, and the interpolation interval is set to 1 millisecond according to the sensor sampling rate.

[0024] The time-synchronized multi-source sensing data and visual data are uploaded to the edge computing node through the Message Queuing Telemetry Transport protocol. The data encapsulation format is JSON, which includes the timestamp, device ID, data value, and data quality flag bit. The data quality flag bit is generated according to the signal validity detection rules. For example, when the temperature signal exceeds the range, it is marked as invalid, and when the vibration signal is missing for more than 3 consecutive sampling points, it is marked as abnormal.

[0025] The installation positions of the temperature sensor, vibration sensor, and current sensor are determined according to the thermodynamic characteristics, mechanical vibration modes, and electrical topological structures of the target device. For example, for a power transformer, the temperature sensor is installed in the hot spot area of the winding, and the hot spot position is determined by pre-scanning with infrared thermal imaging; the vibration sensor is installed on the tank shell, and the position is selected through finite element modal analysis; the current sensor is installed on the high-voltage side bushing lead, and the installation angle is adjusted according to the magnetic field distribution characteristics. The installation positions and angles of the RGB camera and depth camera are determined through three-dimensional space calibration. The calibration process includes using a laser rangefinder to measure the relative distance between the camera and the target device, and calculating the internal and external parameters through the Zhang Zhengyou calibration method.

[0026] During the timestamp alignment process, if the clock offset of the sensor or camera is detected to exceed the preset threshold, such as ±10 milliseconds, the clock re-synchronization mechanism is triggered. The re-synchronization mechanism includes sending a synchronization request message to the master clock source. The message content includes the current clock deviation and jitter value, and the local clock register is updated. The register update step size is dynamically adjusted according to the deviation amount. For example, when the deviation is less than 1 millisecond, micro-step compensation is used; when the deviation is greater than 1 millisecond, step-jump compensation is used.

[0027] The edge computing node caches and verifies the received time synchronization data. The verification rules are set according to the device safe operation specifications and signal physical constraints. For example, the verification range of the temperature signal is from the lower limit to the upper limit of the temperature tolerated by the device material; the verification range of the vibration acceleration value is from the lower limit to the upper limit of the safe vibration amplitude of the device mechanical structure; the verification range of the current signal is from the no-load current of the device to the peak short-circuit current. If the data verification fails, the abnormal data is discarded and an error log is recorded. The content of the error log is set according to the fault diagnosis requirements. For example, the type of abnormal data, timestamp, and device ID are recorded. The multi-source sensing data and visual data that pass the verification are stored in the time series database. The table structure of the time series database is designed according to the data analysis requirements. For example, it includes timestamp, device ID, temperature, vibration, current, RGB image path, and depth point cloud path. The path information generates a unique identifier through the hash algorithm.

[0028] The data transmission delay in the time synchronization process is optimized through the round-trip time measurement and compensation algorithm. The round-trip time measurement interval is set according to the network stability. For example, it is measured once every 1 second. The compensation algorithm uses the exponentially weighted moving average method, and the weight coefficient is dynamically adjusted according to the network delay volatility. For example, when the delay standard deviation exceeds 10 milliseconds, the weight is set to 0.2 to quickly respond to changes; when the standard deviation is lower than 10 milliseconds, the weight is set to 0.8 to maintain stability.

[0029] S2. Perform cross-modal feature extraction and format unification preprocessing on the multi-source sensing data and visual data to generate structured time series data and spatially gridded image data. The specific implementation is as follows: Time-domain feature extraction is performed on the temperature signal, vibration signal, and current signal in the multi-source sensing data respectively to generate a time-series feature vector containing the mean, variance, and spectral amplitude. The time-domain feature extraction of the temperature signal is realized by the sliding window method. The window length is set according to the change rate of the device state. For example, in the temperature monitoring scenario of a thermal power plant boiler, the window length is set to 10 seconds, and the setting basis is twice the thermal inertia time constant of the boiler. The mean and variance are calculated for the data within the window. The mean reflects the steady-state temperature level, and the variance characterizes the degree of temperature fluctuation. The time-domain feature extraction of the vibration signal includes calculating the root mean square value of the acceleration signal. The window length for calculating the root mean square value matches the fundamental vibration period of the device. For example, when the fundamental vibration frequency of a steam turbine is 50 Hz, the window length is set to 0.02 seconds. At the same time, the spectral amplitude of the vibration signal is extracted through the fast Fourier transform. The frequency range corresponding to the spectral amplitude is an integer multiple harmonic of the mechanical resonance frequency of the device. The calculation resolution of the spectral amplitude is determined according to the sampling rate and the window length. For example, when the sampling rate is 1 kHz, the resolution is 1 Hz. The time-domain feature extraction of the current signal includes calculating the effective current value and the harmonic distortion rate. The effective current value is calculated by the integration method, and the integration interval is the power frequency period. The harmonic distortion rate is calculated by extracting the fundamental wave and the amplitudes of each harmonic through the discrete Fourier transform and then calculating the percentage. The upper limit of the harmonic order is set to 50 times according to the electrical standard of the device.

[0030] Spatial grid division is performed on the RGB image in the visual data, and the texture features and color histograms of each grid area are extracted to generate spatially gridded image data. The grid size of the spatial grid division is set according to the size of the key components of the target device. For example, the grid size of the transmission line insulator is set to 32×32 pixels, and the setting basis is the ratio of the average size of the insulator petticoat structure to the image resolution. The texture feature extraction uses the gray-level co-occurrence matrix algorithm to calculate the contrast, correlation, and energy values of the pixels within the grid. The contrast reflects the surface roughness, the correlation characterizes the consistency of the texture direction, and the energy value represents the texture uniformity. The generation parameters of the gray-level co-occurrence matrix include the number of gray levels and the direction angle. The number of gray levels is set to 16 levels, and the direction angles include 0°, 45°, 90°, and 135°. The color histogram is obtained by converting the RGB image to the HSV color space and then statistically analyzing the distribution of hue and saturation. The number of intervals of the hue histogram is set to 16 intervals according to the color identification requirement, and the number of intervals of the saturation histogram is set to 8 intervals. The conversion formula is the normalization process of the hue component in the HSV model.

[0031] Voxelize the depth point cloud in the visual data, map the three-dimensional point cloud to a two-dimensional spatial grid, and supplement the depth information of the spatially gridded image data. The grid resolution of the voxelization process is the same as the grid size of the RGB image. For example, a two-dimensional grid of 32×32 pixels corresponds to a voxel size of 0.5 cm×0.5 cm×0.5 cm in three-dimensional space, and the voxel size is set according to the spatial resolution requirements of the target device. The mapping process of the depth point cloud includes calculating the average depth of all three-dimensional point cloud points within each two-dimensional grid. The average depth is retained to two decimal places, with the unit being millimeters. At the same time, the standard deviation of the point cloud density within the grid is recorded. The standard deviation is used to evaluate the reliability of depth measurement, and the standard deviation threshold is set to 10%. When it exceeds the threshold, it is marked as low-confidence data. The way to supplement the depth information is to add a depth channel to the spatially gridded image data. The numerical range of the depth channel is from 0 to 10,000 millimeters, corresponding to the minimum detection distance of 0 meters to the maximum detection distance of 10 meters of the target device, and the numerical range is determined according to the calibration parameters of the depth sensor.

[0032] Normalize the temporal feature vector and the spatially gridded image data separately. After unifying them to the same numerical range, store them as structured temporal data and spatially gridded image data. The normalization of the temporal feature vector uses the min-max scaling method, and the scaling range is set from 0 to 1. For example, the original range of the temperature mean is from -50°C to 300°C, and after normalization, it is mapped to a floating-point number from 0 to 1. The scaling parameters are initialized according to the global extreme values of the training dataset. The normalization of the spatially gridded image data includes independent scaling of the texture features, color histograms, and depth channels. The scaling ranges of the contrast and energy values of the texture features are set from 0 to 1. The frequencies of the color histograms are converted into probability distributions. The values of the depth channels are linearly scaled from the maximum detection distance of 10 meters to 0 to 1. The parameters of the normalization process include the minimum and maximum values of each feature. The parameters are determined by offline statistics of the historical extreme values of the training dataset and are dynamically updated in the online processing stage. The update trigger condition is that the feature values continuously exceed the historical extreme value range for more than 10 data windows. The storage format of the structured temporal data and the spatially gridded image data is a multi-dimensional array. The array dimension of the temporal data is timestamp×feature dimension, and the array dimension of the spatially gridded image data is the number of grid rows×the number of grid columns×the number of feature channels. Metadata tags are attached when storing the data. The metadata tags include the device ID, acquisition time, and sensor calibration coefficient. The metadata tags are encapsulated in JSON format, and the field types and lengths are consistent with the data dictionary.

[0033] The sliding window step size for time-domain feature extraction is set according to real-time requirements. For example, the window step size for temperature signals is set to 1 second, and the window step size for vibration signals is set to 0.01 second. The setting basis is the minimum time interval for device status update. During the spatial grid division process, if it is detected that the grid in the image edge area contains invalid pixels, the texture features of adjacent grids are used for interpolation filling. The interpolation method is the bilinear interpolation method, and the interpolation weights are calculated according to the grid distance. In the voxelization process, if there is no valid point cloud data in the grid, the depth value is marked as invalid and the data verification mechanism is triggered. The data verification mechanism generates a replacement value according to the statistical distribution of historical depth data. The replacement value is the moving average of the depths of adjacent grids, and the moving average window length is 5 adjacent grids.

[0034] The dynamic update logic of the normalization process includes monitoring the distribution shift of feature values. For example, when the temperature mean exceeds the historical extreme range for 10 consecutive windows, the minimum-maximum scaling parameter is recalculated and the storage configuration is updated. The updated parameters are distinguished by version numbers and the update timestamps are recorded. The metadata tags of the multi-dimensional array storage are encapsulated in JSON format. The JSON fields include the string type of the device ID, the timestamp type of the acquisition time, and the floating-point number array of the sensor calibration coefficients. The length of the floating-point number array is the same as the number of sensors, and the order of the array elements corresponds to the sensor installation positions. The storage medium for time-series data and spatially gridded image data is an SSD solid-state drive. The storage path generates a hierarchical directory structure according to the device ID and the acquisition time. The directory hierarchy is device ID / year / month / day / hour. The naming rule for data files is timestamp_feature type.format. For example, the temperature feature file is named 202310151200_TEMP.npy. The format selection is based on the balance between data compression rate and reading efficiency.

[0035] S3. Perform spatio-temporal alignment on the structured time-series data and spatially gridded image data based on the cross-modal causal inference parameters and the spatial coordinate system transformation matrix to generate spatio-temporally synchronized data. The specific implementation is as follows: Extract the time offset through the causal correlation analysis of multi-source sensing data and visual data to generate cross-modal causal inference parameters. The causal correlation analysis uses the Granger causality test method. The length of the time window is set according to the prior knowledge of the device state change. For example, the time window for the boiler temperature signal and the visual hot spot detection data in a thermal power plant is set to 30 seconds, and the setting basis is twice the thermal inertia time constant of the boiler. The number of lags is set to 3 orders according to the signal transmission delay. The selection of the number of lags is automatically optimized by the Akaike information criterion, and the penalty term coefficient of the Akaike information criterion is set to 2. The maximum number of lags is set to 10 according to the signal sampling rate. The significance level of the hypothesis test of the Granger causality test is set to 0.05, and the setting basis of the significance level is the conventional confidence requirement in statistics. When the test p-value is less than 0.05, it is determined that there is a causal relationship. The time offset is calculated by the maximum mutual information criterion, and the time lag range for mutual information calculation is set to -5 seconds to +5 seconds, with a step size of 0.1 second. The selection of the lag range is based on the maximum expected deviation of the physical response time of the device. The cross-modal causal inference parameters include the time offset between the temperature signal and the hot spot image, and the time offset between the vibration signal and the structural deformation image. After the parameters are generated, they are stored in the key-value pair format. The key is the sensor-camera pair identifier, and the identifier format is "sensor type_camera ID", and the value is the offset in milliseconds, and the millisecond value is reserved to three decimal places.

[0036] Calculate the three-dimensional space coordinate system transformation matrix of the target device based on the calibration board feature point matching. The calibration board is a checkerboard pattern with a size of 30 cm × 30 cm, and the alternating spacing between the black and white squares is 5 cm. The calibration board is fixed in the non-detection area of the target device, and the fixed position is ensured to be physically aligned with the device coordinate system through a laser rangefinder and a level. The feature point matching uses the scale-invariant feature transform algorithm, extracts the corner points of the calibration board as feature points, and the corner point detection threshold is set to 0.01. The threshold setting basis is the statistical distribution of the calibration board image contrast. The dimension of the feature descriptor is 128 dimensions, and the dimension selection is based on the balance between the algorithm calculation efficiency and the feature discrimination. The calculation of the three-dimensional space coordinate system transformation matrix includes the following steps: Fit the corresponding relationship of the calibration board feature points in the camera coordinate system and the device coordinate system by the least squares method to calculate the initial estimate of the homography matrix, and the fitting residual tolerance is set to 0.5 pixel; Optimize the rotation matrix and the translation vector by the iterative closest point algorithm, and the iteration termination condition is that the sum of the squared residuals is less than 0.1 pixel or the maximum number of iterations reaches 100 times; The finally output space coordinate system transformation matrix is a 4×4 homogeneous matrix, and the Euler angle accuracy of the rotation matrix is set to 0.01 degree, the translation vector accuracy is set to 0.1 mm, and the scaling factor is fixed to 1.0.

[0037] Compensate the time offset to the timestamps of the structured time-series data to generate a time-synchronized sensor data stream. The time offset compensation is achieved by linear interpolation, and the interpolation step size is set according to the sensor sampling rate. For example, when the sampling rate of the temperature signal is 1 Hz, the interpolation step size is 1 second, and when the sampling rate of the vibration signal is 1 kHz, the interpolation step size is 0.001 second. The timestamp adjustment includes adding the original timestamp and the offset to generate an aligned timestamp sequence. The timestamp accuracy is reserved to the millisecond level, and the continuity of the timestamp sequence is verified by the sliding window overlap rate, which is set to 50%. The verification threshold for the overlapping window is that the time difference is less than 1 millisecond. The time-synchronized sensor data stream is stored as a list of time-value pairs, where the timestamp is the key and the value is the normalized sensor reading. The normalization method uses the min-max scaling method, and the scaling parameters are initialized according to the global extreme values of the historical dataset.

[0038] Apply the three-dimensional space coordinate system transformation matrix to the spatially gridded image data to generate a spatially aligned visual data stream. The spatial alignment process includes the following steps: Multiply the center coordinate of each grid in the spatially gridded image data by the transformation matrix and map it to the unified coordinate system of the device. The floating-point precision of the coordinate mapping is reserved to four decimal places; Truncate the grid coordinates that exceed the boundaries of the target device. The truncation threshold is set according to the physical size of the device. For example, the boundary threshold of the transformer oil tank is 2 m in length × 1 m in width × 1.5 m in height, and the threshold setting is based on the marked dimensions in the device design drawing; Perform interpolation filling on the truncated grid coordinates. The interpolation method is bilinear interpolation, and the interpolation weights are calculated according to the grid distance. The distance calculation formula is the reciprocal weighted Euclidean distance. The spatially aligned visual data stream is stored as a dictionary of grid coordinate-eigenvalue pairs. The grid coordinates are three-dimensional floating-point numbers, and the eigenvalues include texture contrast, color histogram, and depth average value. The hash collision handling of the dictionary uses the chaining method, and the size of the hash table is dynamically allocated according to the number of grids.

[0039] Fuse the time-synchronized sensor data stream and the spatially aligned visual data stream to generate spatio-temporal synchronization data. The fusion process is based on the joint index of the timestamp and the spatial coordinates. The index key is the timestamp-grid coordinate pair. The timestamp accuracy is at the millisecond level, and the grid coordinate accuracy is at the millimeter level. The index value is the concatenated vector of the sensor reading and the visual feature, and the vector dimension is the sum of the sensor feature number and the visual feature number. The spatio-temporal synchronization data is stored as a multi-dimensional tensor. The tensor dimension is time step × grid row × grid column × feature channel. The feature channels include temperature mean, vibration root mean square value, color histogram, and depth value. The data type of the tensor is 32-bit floating-point number. When storing the tensor, additional metadata tags are attached. The metadata includes the version number of the space coordinate system transformation matrix, the hash value of the time synchronization parameters, and the data checksum. The checksum is generated by the cyclic redundancy check algorithm, the polynomial is CRC-32, the initial value during checksum generation is set to 0xFFFFFFFF, and the final exclusive OR value is set to 0xFFFFFFFF.

[0040] The lag order of the Granger causality test is automatically selected by the Akaike information criterion. The penalty coefficient of the Akaike information criterion is set to 2, and the maximum lag order is set to 10 according to the signal sampling rate. When the lag order exceeds 10, a warning log is triggered. During the calibration board feature point matching process, if feature point missing caused by occlusion or reflection is detected, it is supplemented by adjacent frame interpolation. The interpolation method is bicubic interpolation, and the interpolation weight is calculated according to the spatial distance of the feature points. The normalization coefficient for distance calculation is the reciprocal of the calibration board diagonal length. The linear interpolation method for time offset compensation switches to spline interpolation when the data missing exceeds 3 consecutive sampling points. The boundary condition of the spline interpolation is set to natural spline, and the number of spline nodes is dynamically adjusted according to the length of the missing data. In the truncation process of spatial alignment, if the grid coordinates exceed the device boundary, the eigenvalue of the grid is marked as invalid. The invalid mark is stored by a binary mask. The mask bit is 1 for valid and 0 for invalid. The mask is stored as an independent bitmap file, and the bitmap resolution is the same as the number of grids.

[0041] During the fusion process of spatio-temporal synchronized data, if the timestamps or spatial coordinate indices do not match, a spatio-temporal interpolation mechanism is triggered. The spatio-temporal interpolation mechanism includes linear interpolation in the time dimension and bilinear interpolation in the spatial dimension. The interpolation weights are calculated according to the time difference and spatial distance. The time difference weight coefficient is set to 0.5, and the spatial distance weight coefficient is set to 0.5. The storage format of the multi-dimensional tensor is the hierarchical data format, and the compression algorithm is Zlib. The compression level is set to 6, and the selection of the compression level is based on the balance test results of the compression ratio and decompression speed. The cyclic redundancy check code of the metadata tag is verified before data transmission. If the verification fails, a data retransmission mechanism is triggered. The upper limit of the retransmission times is set to 3 times, and the retransmission interval is dynamically adjusted according to the network delay. The initial retransmission interval is set to 100 milliseconds, and the interval doubles after each retransmission.

[0042] Step S3 realizes spatio-temporal alignment through cross-modal causal inference parameters and the spatial coordinate transformation matrix. Traditional methods rely on hardware clock synchronization or fixed interpolation algorithms, and it is difficult to handle the spatio-temporal offsets of multi-modal data in a dynamic environment. For example, the spatio-temporal benchmarks of sensor and visual data are inconsistent due to transmission delays or environmental disturbances. Step S3 derives the time offset from the data itself through causal correlation analysis (such as Granger causality test), replacing the rigid constraint of hardware synchronization; at the same time, a spatial coordinate transformation relationship is established based on the calibration board feature matching to solve the spatial benchmark differences of multi-modal data. Compared with the existing technology, it breaks through the hardware synchronization limit through data-driven dynamic calibration, reduces the end-to-end delay, and can adapt to environmental changes, improving the data fusion accuracy.

[0043] Figure 2The flowchart of the multi-modal data-driven dynamic resource allocation decision of the present invention is given. According to the relevance of multi-modal data in spatio-temporal synchronous data and the impact of environmental interference on rendering quality, computing resources are dynamically allocated to the data fusion module or the rendering pipeline module driven by the AI large model. The specific implementation is as follows: Extract the latent variable distribution of multi-modal data in spatio-temporal synchronous data, calculate the Wasserstein distance between the latent variable distributions, and generate a multi-modal latent variable similarity coefficient. The latent variable distribution is generated by performing latent space encoding on multi-modal data through a variational autoencoder. The encoder network structure of the variational autoencoder is a three-layer fully connected layer, with 128 hidden layer nodes, a ReLU activation function, and a 32-dimensional latent space. The input data is the temperature mean, vibration root mean square value, and color histogram in spatio-temporal synchronous data. The training process of the variational autoencoder includes: the input data generates the mean and variance of the latent space distribution through the encoder network, samples the latent variables through the reparameterization trick, the decoder network reconstructs the latent variables into the input data, the reconstruction loss function is the mean square error, the regularization loss is the KL divergence, and the total loss function is the weighted sum of the two, with the weight coefficient set to 0.5. The calculation of the Wasserstein distance is based on the mean and covariance matrix of the latent variable distribution. The calculation formula is the weighted sum of the square of the Euclidean distance between the means of the two distributions and the Frobenius norm of the covariance matrix. The weight coefficients are set to 0.7 and 0.3 according to the importance of the modal data. The basis for setting the weight coefficients is the contribution degree of the modal data in historical anomaly detection. For example, when the contribution degree of the temperature signal to defect detection is higher than that of the vibration signal, the temperature modal weight is set to 0.7. The value range of the multi-modal latent variable similarity coefficient is from 0 to 1. The smaller the value, the stronger the complementarity between the modalities. For example, the similarity coefficient between the temperature signal and the hot spot image of a thermal power plant boiler is 0.2, indicating a high degree of complementarity. The coefficient is mapped to the 0-1 interval through normalization processing.

[0044] Generate a degraded virtual scene based on environmental interference parameters, calculate the difference in the structural similarity index between the degraded virtual scene and the ideal scene, and generate a rendering quality conflict mapping value. The environmental interference parameters include light intensity, electromagnetic noise level, and environmental humidity. The parameters are collected in real time through Internet of Things sensors. The unit of light intensity is lux, the unit of electromagnetic noise level is decibel, and the unit of environmental humidity is percentage. The generation of the degraded virtual scene uses a generative adversarial network. The generator inputs the environmental interference parameters and the RGB-D data of the ideal scene. The network structure of the generator is U-Net. The encoder part consists of four convolutional layers, and the decoder part consists of four transposed convolutional layers. Skip connections are used to retain detailed information. The network structure of the discriminator is five convolutional layers, and the output is the scene authenticity score. The training data of the generative adversarial network are scene image pairs containing different environmental interference parameters. The training batch size is 32, the optimizer is Adam, the initial learning rate is 0.0002, the momentum parameter is 0.5, and the gradient clipping threshold during training is 1.0 to prevent gradient explosion. The difference in the structural similarity index is calculated by comparing the brightness, contrast, and structural information of the degraded scene and the ideal scene. The brightness comparison function is the square of the difference in the means of the two scenes, the contrast comparison function is the square of the product of the standard deviations of the two scenes, and the structural information comparison function is the square of the covariance. The constant term in the calculation formula is set as a stability coefficient to avoid division by zero. The value range of the rendering quality conflict mapping value is from 0 to 1. The larger the value, the more serious the impact of environmental interference on the rendering quality. For example, the conflict mapping value caused by a sudden increase in light intensity is 0.8, and the mapping value is mapped to the 0-1 interval through normalization processing.

[0045] When the similarity coefficient of the multimodal latent variable is less than or equal to the preset similarity threshold and the rendering quality conflict mapping value is greater than or equal to the preset conflict threshold, allocate computing resources to the data fusion module driven by the AI large model. The preset similarity threshold is set according to the statistical distribution quantile when the modal complementarity is optimal in the historical data. For example, select the 25% quantile as the threshold 0.3. The quantile calculation uses linear interpolation, the data distribution model is kernel density estimation, and the bandwidth parameter is set to 0.1. The preset conflict threshold is set according to the statistical distribution quantile when the rendering quality degradation is the most serious in the historical data. For example, select the 75% quantile as the threshold 0.6, the statistical period is 30 days, and the data sampling interval is 1 minute. The computing resource allocation includes the number of GPU cores, memory bandwidth, and processor priority. The allocation strategy is dynamic preemptive scheduling. For example, when the conditions are met, 80% of the GPU cores are switched from the rendering pipeline module to the data fusion module. The GPU core allocation granularity is in units of streaming multiprocessors, the memory bandwidth is allocated by channel sharding, and the processor priority is set through the real-time scheduling class of the Linux kernel. The resource allocation switching trigger delay requirement is less than 50 milliseconds. The delay test method is to inject a step signal and measure the response time. An alarm log is triggered when the delay exceeds the limit.

[0046] When the multi-modal latent variable similarity coefficient is greater than the preset similarity threshold or the rendering quality conflict mapping value is less than the preset conflict threshold, the computing resources are allocated to the rendering pipeline module. The resource allocation switching trigger mechanism includes real-time monitoring of the change rates of the similarity coefficient and the conflict mapping value. If the change rate exceeds 5% per second, the allocation decision is immediately re-evaluated. The change rate is calculated using the sliding window difference method with a window size of 10 seconds and a step size of 1 second. The resource allocation priority of the rendering pipeline module is dynamically adjusted according to the scene complexity, which is calculated based on the number of meshes and the texture detail level. For example, 90% of the GPU cores are allocated for complex scenes and 50% for simple scenes. The mesh number threshold is set to 10,000 meshes, and the texture detail level is evaluated by the anisotropic filtering level. The resource allocation strategy is verified through a combination of offline simulation and online testing. Offline simulation uses historical data playback, and online testing injects interference signals in real time and monitors the system response delay. The verification metrics include resource switching delay and scene rendering frame rate. The test data is stored in the performance monitoring database for troubleshooting and optimization analysis.

[0047] The update mechanism for the preset similarity threshold and the preset conflict threshold includes periodically counting the distribution changes of historical data with an update period of 24 hours. The statistical method is kernel density estimation, and the bandwidth parameter is set to 0.1 according to the data variance. The kernel function is the Gaussian kernel. When the difference between the new and old thresholds exceeds 10% during threshold update, an artificial review process is triggered. After the review is passed, the thresholds are updated and the version number is recorded. The version number format is timestamp_device ID. The historical data is stored in a distributed database. The database sharding strategy is based on hash distribution of the device ID. Each shard retains the data for the most recent 30 days, and the expired data is automatically archived to cold storage. The cold storage format is compressed Parquet columnar storage with the Snappy compression algorithm and a compression level of 9. The read-write consistency level of the distributed database is eventual consistency, the number of replicas is 3, and the fault recovery strategy is majority write.

[0048] The training data for the variational autoencoder are spatio-temporal synchronous samples of multi-modal sensing data and visual data. The training batch size is 64, the optimizer is Adam, the initial learning rate is 0.001, the learning rate decay strategy is to halve every 10 epochs, and the training stop condition is that the validation set loss does not decrease for 5 consecutive epochs. The training data for the generative adversarial network are scene image pairs containing different environmental interference parameters. The gradient clipping threshold during training is 1.0 to prevent gradient explosion. The training stop condition is that the ratio of the generator and discriminator loss functions stabilizes in the range of 0.9 to 1.1 for more than 10 epochs. The structural similarity index is calculated using the sliding window method with a window size of 8×8 pixels, a sliding step size of 4 pixels, a window weight of the Gaussian kernel function with a standard deviation of 1.5, and the truncation coefficient of the Gaussian kernel set to 3.

[0049] The exception handling mechanism during resource allocation includes: if the similarity coefficient or conflict mapping value exceeds the reasonable range (for example, the similarity coefficient is negative or the conflict mapping value is greater than 1), the abnormal data is discarded and replaced with the previous valid value. The validity period of the replacement value is set to 5 seconds, and it is marked as invalid after the expiration; if a deadlock occurs due to GPU resource allocation conflicts, the resource release protocol is triggered to forcibly release the occupied resources and reallocate them. The timeout of the resource release protocol is set to 500 milliseconds. After the timeout, an error log is recorded and the resource scheduling service is restarted. The storage format of the error log is structured log, and the fields include timestamp, device ID, error code, and description. The log files are archived by day, and the compression algorithm of the archived files is GZIP, and the compression level is set to 6.

[0050] The verification of the computing resource allocation strategy is combined with offline simulation and online testing. Offline simulation uses historical data playback, and the data playback rates are 1 times, 2 times, and 5 times the real-time speed to test the stability of the resource allocator under different loads; online testing injects interference signals in real time and monitors the system response delay. The interference signals include step changes in light intensity, electromagnetic noise pulses, and humidity gradients. The verification metrics include resource switching delay (required to be less than 50 milliseconds) and scene rendering frame rate (required to be greater than 30 fps). The test data is stored in the performance monitoring database, the database index structure is B+ tree, and the query optimization strategy is cost model optimization.

[0051] Step S4 dynamically allocates computing resources through multimodal correlation and environmental interference assessment. In the existing hierarchical architecture, the resource competition contradiction between the AI inference and rendering modules leads to a decline in real-time performance. For example, high-load rendering tasks may preempt AI analysis resources, delaying the abnormal response. Step S4 quantifies the modal complementarity through the latent variable similarity coefficient, combines the rendering quality conflict mapping value to evaluate environmental interference, and adopts a hierarchical decision logic (such as "AND / OR" conditions). Compared with the existing technology, this strategy can dynamically identify critical tasks (such as giving priority to AI analysis when the modal complementarity is low), and at the same time adjust the rendering quality according to environmental interference to avoid resource waste caused by static rules. The resource allocation integrates multimodal correlation relationships and environmental states, and drives decisions through dual metrics to achieve a balance between computing load and real-time requirements.

[0052] S5. The data fusion module performs cross-modal correlation analysis on the spatio-temporally synchronized data through the AI large model to generate device state feature data and abnormal area mask data. The specific implementation is as follows: Cross-modal correlation modeling of multi-modal features in spatio-temporal synchronous data is performed through a multi-head cross-attention mechanism to generate a fused feature vector. The number of attention heads in the multi-head cross-attention mechanism is set to 8, and the dimension of each attention head is 64. The input features include the average temperature, root mean square value of vibration, and color histogram in the spatio-temporal synchronous data. Query vectors, key vectors, and value vectors are generated through linear transformation. The dimension of the weight matrix of the linear transformation is the input feature dimension × 64, and the bias term is initialized to zero. The attention weights are calculated by the dot product of the query vector and the key vector. The scaling factor is the reciprocal of the square root of the attention head dimension. After normalization by the Softmax function, it is weighted and summed with the value vector to output the fused feature vector. The dimension of the fused feature vector is 512. The training stability is optimized through layer normalization and residual connection. The parameters of layer normalization are learnable vectors, and the skip path of the residual connection is the identity mapping.

[0053] The cosine similarity is calculated based on the fused feature vector and the pre-trained embedding vector of the normal operating state of the device to generate the anomaly confidence in the device state feature data. The pre-trained embedding vector is generated through training with historical normal data. The training data is the spatio-temporal synchronous data of the device in the fault-free state. The training method is an autoencoder. The encoder network structure is a three-layer fully connected layer. The number of hidden layer nodes is 256, the activation function is GELU, and the dimension of the latent vector is 128. The decoder network structure is symmetric, and the reconstruction loss function is the mean square error. The calculation of the cosine similarity includes calculating the dot product after L2 normalization of the fused feature vector and the pre-trained embedding vector. The formula for L2 normalization is that each element of the vector is divided by the vector norm length, and the dot product result is a scalar from -1 to 1. The anomaly confidence is generated by linearly mapping the cosine similarity to the interval of 0 to 1. The mapping function is (similarity + 1) / 2. The anomaly confidence threshold is set to the 95% quantile of the historical anomaly samples. For example, the threshold is 0.85. The quantile calculation uses linear interpolation. The historical data coverage period is 6 months, and the data sampling interval is 1 minute.

[0054] The spatial gridded image data is segmented by a sliding window, and the gradient distributions of the texture features and depth features within the window are extracted to generate a candidate mask for the abnormal region. The size of the sliding window is 64×64 pixels, and the sliding step size is 32 pixels. The settings of the window size and step size are based on the minimum abnormal region size of the target device. For example, the minimum visible size of a crack in a transmission line insulator is 16×16 pixels. The gradient distribution of the texture features calculates the gradient magnitudes in the horizontal and vertical directions through the Sobel operator. The convolution kernel size of the Sobel operator is 3×3, the horizontal kernel is [-1,0,1; -2,0,2; -1,0,1], and the vertical kernel is [-1,-2,-1; 0,0,0; 1,2,1]. The formula for the gradient magnitude is the square root of the sum of the squares of the horizontal gradient and the vertical gradient, which is normalized to the range of 0 to 255. The gradient distribution of the depth features is obtained by calculating the absolute value of the depth difference between adjacent pixels. The unit of the depth difference is millimeters, and the normalization range is 0 to 10,000 millimeters, corresponding to the maximum range of the depth sensor. The method for generating the candidate mask for the abnormal region is as follows: When the weighted sum of the texture gradient and the depth gradient exceeds a preset threshold, it is marked as a candidate abnormal region. The weight coefficients are set to 0.6 and 0.4 according to the abnormal contribution degrees of the texture and depth in the historical data. For example, in crack detection, the contribution degree of the texture gradient is higher. The preset threshold is the 99% quantile of the historical normal samples, and the threshold calculation data window is the past 30 days.

[0055] The candidate mask for the abnormal region is optimized by morphological closing operation to eliminate isolated noise points and fill the hole regions, generating the mask data for the abnormal region. The structuring element for the morphological closing operation is a 3×3 rectangular kernel, and all the element values within the kernel are 1. First, the dilation operation is performed to fill the holes. The definition of the dilation operation is to traverse the image with the center of the structuring element. If there is at least one pixel with a value of 1 in the center-covered region, the output pixel is set to 1. Then, the erosion operation is performed to smooth the boundary. The definition of the erosion operation is that the output pixel is set to 1 only when the region covered by the structuring element is all 1. The number of iterations for both the dilation and erosion operations is 1. The isolated regions smaller than 5 pixels after processing are removed. The determination of the isolated regions is achieved through the connected component labeling algorithm, and the connected component adjacency rule is 8-neighborhood. The storage format of the mask data for the abnormal region is a binary image, where the pixel value of 1 represents the abnormal region and 0 represents the normal region. The image resolution is the same as that of the original spatial gridded image data. When storing, metadata tags are attached. The metadata includes the timestamp, device ID, and abnormal confidence level.

[0056] The update mechanism for pre-trained embedding vectors includes periodic retraining with a period of 7 days. The training data is automatically extended to newly added normal data, and the newly added data needs to be verified as in a normal state through verification rules. The verification rules include that the sensor readings are within the historical normal range and the visual data has no abnormal features. The training optimizer for the autoencoder is AdamW, with an initial learning rate of 0.0001, a weight decay coefficient of 0.01, a training batch size of 32, and an early stopping condition that the validation set loss has not decreased for 10 consecutive epochs. The dynamic adjustment method for the anomaly confidence threshold is as follows: Statistically analyze the anomaly confidence distribution in the past 30 days, calculate the mean and standard deviation. When the standard deviation of the new data distribution exceeds 10% of the historical standard deviation, recalculate the quantile threshold, and the window data for recalculation is the data from the past 7 days to 30 days.

[0057] During the sliding window segmentation process, if it is detected that the image boundary region cannot completely cover the window, the mirror padding method is used to expand the boundary, with a padding width of 16 pixels, and the symmetry axis of the mirror padding is the image boundary line. The kernel size of the morphological closing operation is dynamically adjusted according to the image resolution. For example, when the resolution is greater than 1920×1080, the kernel size is increased to 5×5 to maintain the balance between processing efficiency and accuracy. If it is detected during the optimization process of the anomaly region candidate mask that the area of consecutive anomaly regions is less than 10 pixels, it is determined as noise and removed. The area calculation is achieved through pixel counting, and the removal operation updates the mask value to 0.

[0058] The training data for the multi-head cross-attention mechanism is labeled cross-modal association samples, and the label is the manually annotated feature association intensity. The association intensity is divided into levels from 0 to 5, where 0 indicates no association and 5 indicates strong association. The training loss function is the weighted sum of the mean squared error and cross-entropy, and the weight coefficients are set to 0.7 and 0.3, with the setting basis being the importance ratio of the modal data in historical anomaly detection. The Sobel operator convolution kernel in the gradient distribution calculation is separated into horizontal and vertical directions. After convolving the image in each direction respectively, the gradient magnitude is synthesized. The synthesized gradient image is denoised through Gaussian filtering, with a Gaussian kernel size of 5×5 and a standard deviation of 1.0.

[0059] The verification of the anomaly confidence is analyzed through the ROC curve. The calculation of the true positive rate and false positive rate is based on the test data set, which contains 1000 groups of normal samples and 500 groups of abnormal samples. The sample sources are historical inspection records and artificially simulated anomaly injections. The accuracy evaluation of the anomaly region mask data uses the intersection over union (IoU) metric. The IoU threshold is set to 0.5, and if it is higher than the threshold, it is determined as a valid anomaly region. The evaluation results are recorded in the performance monitoring report.

[0060] The historical anomaly samples are stored in a time series database. The database sharding strategy is to shard by device ID hashing, and each shard stores the anomaly data of a single device. The data retention policy is to automatically archive to cold storage after 180 days. The cold storage format is compressed Parquet files, the compression algorithm is Snappy, and the compression level is 9. The read and write interfaces of the time series database are RESTful APIs. Query optimization uses B+ tree indexes, and the index fields are timestamp and device ID. The transaction isolation level is read committed.

[0061] S6. The rendering pipeline module fuses the device anomaly feature data and the defect area mask data into the 3D inspection scene to generate an inspection interaction interface with virtual-real superposition. The specific implementation is as follows: Convert the anomaly area mask data into the transparency texture on the surface of the 3D model and attach it to the 3D inspection scene of the target device through UV mapping. The anomaly area mask data is a binary image, where the pixel value of 1 represents the anomaly area and 0 represents the normal area. The generation method of the transparency texture is as follows: set the transparency of the area corresponding to the mask pixel value of 1 to 0.8, and the transparency of the area corresponding to the pixel value of 0 to 0.2. The transparency value is set according to the statistical quantile of the mask data of historical normal samples. For example, the 99% quantile of the transparency of the normal area is 0.2, the calculation period of the statistical quantile is 30 days, and the data sampling interval is 1 minute. The UV mapping process includes aligning the pixel coordinates of the 2D mask image with the texture coordinates on the surface of the 3D model. The mapping rule of the texture coordinates is to project the vertex coordinates of the 3D model onto the 2D plane and then normalize them to the range of 0-1. The projection plane is determined according to the main detection view of the device. For example, the projection plane of the boiler in a thermal power plant is the front orthographic view. The selection basis of the projection plane is the installation position and angle of the device inspection camera.

[0062] Convert the temperature mean and vibration root mean square value in the device status characteristic data into a thermal cloud map and a vibration intensity field, and superimpose them on the corresponding spatial coordinates in the 3D inspection scenario. The generation method of the thermal cloud map is as follows: Map the temperature mean to a predefined color gradient. The color gradient ranges from blue (0°C) to red (300°C), and the mapping function is linear interpolation. The interpolation step is set to 10°C according to the safe operating temperature range of the device. The basis for the dynamic adjustment of the color gradient is the upper thermal tolerance limit of the device material. For example, when the upper tolerance temperature of boiler steel is 500°C, the gradient range is extended to blue (0°C) to dark red (500°C). The generation method of the vibration intensity field is as follows: Convert the vibration root mean square value into arrow length and color intensity. The arrow direction is determined according to the installation orientation of the vibration sensor. The color intensity ranges from green (0g) to red (20g), corresponding to the vibration safety threshold of the device. The conversion relationship between the arrow length and the vibration amplitude is 0.1 meter per 1g. The superimposition process is implemented through the shader program of the 3D rendering engine. The vertex shader binds the temperature and vibration data to the model vertices, and the fragment shader renders the colors and arrows according to the interpolation results. The shader code is written in HLSL language, and the compilation target is Shader Model 6.5.

[0063] Fuse the defect area mask data with the thermal cloud map and the vibration intensity field through depth testing and transparency blending rendering to generate a virtual-real superimposed inspection interaction interface. The depth buffer comparison function of the depth test is set to less than or equal to, ensuring that the foreground abnormal area covers the background model. The depth buffer precision is 24 bits, and the stencil buffer precision is 8 bits. The blending factors of the transparency blending are set to source transparency (source factor is SrcAlpha) and destination transparency (destination factor is OneMinusSrcAlpha). The blending formula is source color multiplied by source transparency plus destination color multiplied by (1 minus source transparency). The blended color channels are adjusted through gamma correction, and the gamma value is set to 2.2. The fused inspection interaction interface is output as frame buffer data in RGBA format, with a resolution matching the display device, such as 1920×1080 pixels, a refresh rate of 60Hz, a color depth of 10bit, and HDR metadata conforming to the ST 2084 standard.

[0064] The threshold of the transparency texture is set according to the statistical quantile of the mask data of historical normal samples. The quantile calculation uses the R-7 algorithm, and the domain is from 0 to 1. The threshold of the transparency of the abnormal area is set to the 99.9% quantile of historical normal samples. The trigger condition for the dynamic adjustment of the color mapping gradient of the heat cloud map is the material change event in the device maintenance record. The event is subscribed through the MQTT protocol and the rendering parameters are automatically updated. The parameter update delay requirement is less than 100 milliseconds. The hardware dependency of the 3D rendering engine is the NVIDIA RTX 4090 GPU, the video memory requirement is 24GB, the software dependency is Unity 2022.3.20f1, the rendering pipeline is the Universal Render Pipeline (URP), and the shader compilation tool is Shader Graph 12.1.7.

[0065] During the UV mapping process, if it is detected that the texture coordinates of the 3D model surface do not match those of the mask image, the mirror filling method is used to expand the texture coordinates. The filling width is 16 pixels, and the filling direction is along the boundary of the UV unfolding of the model surface. The filled texture coordinates are smoothed through bilinear interpolation. When the temperature data is missing, the color interpolation of the heat cloud map switches to neighboring pixel interpolation. The interpolation window size is 3×3 pixels, the weight of the central pixel is 0.5, and the weight of the adjacent pixels is 0.125. The interpolation result is denoised by Gaussian filtering, and the standard deviation of the Gaussian kernel is 1.0. When the vibration data is abnormal (such as the amplitude exceeding 20g), the arrow rendering of the vibration intensity field switches to a flashing warning icon. The flashing frequency is 2Hz, the icon color is red, and the flashing period is controlled by a timer interrupt with an interrupt precision of 1 millisecond.

[0066] The verification method for the virtual-real overlay inspection and interaction interface is to compare the rendering results with the ground truth data manually marked. The ground truth data includes the location of the abnormal area, temperature values, and vibration amplitudes. The comparison metrics are the pixel-level accuracy and the intersection over union. The accuracy threshold is set to 95%, and the intersection over union threshold is set to 0.8. The test dataset contains 1000 groups of normal samples and 500 groups of abnormal samples. The sample sources are historical inspection records and artificially simulated abnormal injections. The rendering performance metrics include the frame rate stability rate (required to be ≥98%) and the latency (required to be ≤20ms). The test data is stored in the performance monitoring database. The database index is a composite key of the timestamp and the device ID, and the query optimization uses a B+ tree index.

[0067] The update mechanism of the transparency texture includes asynchronously updating the texture map after receiving new mask data in real time. The update interval is 1 second. The texture compression format is BC7, the compression level is high quality, and the compression algorithm is accelerated by GPU hardware. The dynamic data of the thermal cloud map and the vibration intensity field are transmitted through a circular buffer. The buffer capacity is 60 frames. The overflow handling strategy is to discard the oldest frame. The buffer read and write pointers ensure thread safety through atomic operations. The model loading of the 3D inspection scene adopts the LOD technology. The close-range model detail level is 4K texture, and the far-range model detail level is 1K texture. The switching distance is dynamically calculated according to the perspective focal length. The focal length data is obtained in real time through the frustum parameters of the rendering camera.

[0068] In this embodiment, the systematic optimization of industrial inspection efficiency is achieved through the spatio-temporal alignment and dynamic resource allocation mechanism of multi-modal data. The prior art usually relies on independent modules to process sensing data and visual data, and realizes function stacking through fixed clock synchronization and static resource allocation, resulting in the contradiction between heterogeneous data processing delay and resource competition. In this embodiment, cross-modal causal inference parameters are used to replace hardware clock synchronization, and the spatio-temporal offset is deduced from the data correlation, solving the problem of spatio-temporal reference misalignment caused by environmental interference between sensors and visual devices; at the same time, combined with the dual decision-making logic of the latent variable similarity coefficient and the rendering quality conflict mapping value, computing resources are dynamically allocated. Compared with the conventional scheme, in this embodiment, through the data-driven dynamic calibration and adaptive decision-making mechanism, the multi-modal data processing is changed from isolated execution to collaborative optimization, significantly reducing the end-to-end delay and improving the real-time performance of anomaly response. Through the coupled analysis of cross-modal correlation modeling and environmental interference, a dynamic feedback closed-loop of data-resource-scene is constructed.

[0069] The calculations involved in the embodiments are all dimensionless numerical calculations. The preset parameters and threshold selections in the calculations are set by those skilled in the art according to the actual situation.

[0070] It should be noted that the present invention can be deployed on the device itself to achieve embedded applications, or can also run on a PC or other terminals with a user interface, so as to meet various hardware environments and usage requirements.

[0071] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more collections of available media. The available media can be magnetic media (such as floppy disks, hard disks, magnetic tapes), optical media (such as DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0072] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and modules described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0073] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there can be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or modules can be in electrical, mechanical, or other forms.

[0074] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules. They can be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0075] In addition, in each embodiment of the present application, each functional module can be integrated into a processing module, or each module can exist physically alone, or two or more modules can be integrated into one module.

[0076] If the above-mentioned function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0077] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0078] Finally: The above is only the preferred embodiment of the present invention and is not used to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent inspection method for the integration of the Internet of Things and virtual reality based on an AI large model, characterized in that, Including: S1. Collect multi-source sensing data and visual data of the target device through the Internet of Things terminal; S2. Perform cross-modal feature extraction and format unification preprocessing on the multi-source sensing data and visual data to generate structured time-series data and spatially gridded image data; S3. Perform spatio-temporal alignment on the structured time-series data and spatially gridded image data based on cross-modal causal inference parameters and spatial coordinate system transformation matrices to generate spatio-temporal synchronized data; S4. Dynamically allocate computing resources to the data fusion module or rendering pipeline module driven by the AI large model according to the relevance of multi-modal data in the spatio-temporal synchronized data and the impact of environmental interference on the rendering quality; S5. The data fusion module performs cross-modal correlation analysis on the spatio-temporal synchronized data through the AI large model to generate device status feature data and abnormal area mask data; S6. The rendering pipeline module fuses the device abnormal feature data and defect area mask data into the 3D inspection scene to generate a virtual-real overlay inspection interaction interface.

2. The intelligent inspection method based on the integration of the Internet of Things and virtual reality with an AI large model according to claim 1, wherein Collecting multi-source sensing data and visual data of the target device through the Internet of Things terminal includes: Synchronously collect the temperature signal, vibration signal, and current signal of the target device through temperature sensors, vibration sensors, and current sensors as multi-source sensing data; Synchronously collect the RGB image and depth point cloud of the target device through an RGB camera and a depth camera as visual data; Perform timestamp alignment on the temperature signal, vibration signal, current signal, RGB image, and depth point cloud based on the industrial Internet of Things clock synchronization protocol to generate time-synchronized multi-source sensing data and visual data.

3. The intelligent inspection method for the integration of the Internet of Things and virtual reality based on the AI large model according to claim 1, characterized in that, Performing cross-modal feature extraction and format unification preprocessing on the multi-source sensing data and visual data to generate structured time-series data and spatially gridded image data includes: Respectively perform time-domain feature extraction on the temperature signal, vibration signal, and current signal in the multi-source sensing data to generate time-series feature vectors including mean, variance, and spectral amplitude; Perform spatial grid division on the RGB image in the visual data, extract the texture features and color histograms of each grid area to generate spatially gridded image data; Perform voxelization processing on the depth point cloud in the visual data, map the three-dimensional point cloud to a two-dimensional spatial grid, and supplement the depth information of the spatially gridded image data; Perform normalization processing on the time-series feature vectors and spatially gridded image data respectively, and store them as structured time-series data and spatially gridded image data after unifying them to the same numerical range.

4. The intelligent inspection method for the integration of the Internet of Things and virtual reality based on the AI large model according to claim 1, characterized in that, Performing spatio-temporal alignment on the structured time-series data and spatially gridded image data based on cross-modal causal inference parameters and spatial coordinate system transformation matrices to generate spatio-temporal synchronized data includes: Extract the time offset through the causal correlation analysis of multi-source sensing data and visual data to generate cross-modal causal inference parameters; Calculate the three-dimensional spatial coordinate system transformation matrix of the target device based on the calibration board feature point matching; Compensate the time offset to the timestamp of the structured time-series data to generate a time-synchronized sensing data stream; Apply the three-dimensional spatial coordinate system transformation matrix to the spatially gridded image data to generate a spatially aligned visual data stream; Fuse the sensor data stream with time synchronization and the visual data stream with spatial alignment to generate spatio-temporal synchronized data.

5. The intelligent inspection method for the integration of the Internet of Things and virtual reality based on the AI large model according to claim 1, characterized in that, According to the correlation of multimodal data in the spatio-temporal synchronized data and the impact of environmental interference on the rendering quality, dynamically allocate computing resources to the data fusion module or the rendering pipeline module driven by the AI large model, including: Extract the latent variable distribution of multimodal data in the spatio-temporal synchronized data, calculate the Wasserstein distance between the latent variable distributions, and generate a multimodal latent variable similarity coefficient; Generate a degraded virtual scene based on the environmental interference parameters, calculate the structural similarity index difference between the degraded virtual scene and the ideal scene, and generate a rendering quality conflict mapping value; When the multimodal latent variable similarity coefficient is less than or equal to the preset similarity threshold and the rendering quality conflict mapping value is greater than or equal to the preset conflict threshold, allocate computing resources to the data fusion module driven by the AI large model; When the multimodal latent variable similarity coefficient is greater than the preset similarity threshold or the rendering quality conflict mapping value is less than the preset conflict threshold, allocate computing resources to the rendering pipeline module.

6. The intelligent inspection method for the integration of Internet of Things and virtual reality based on the AI large model according to claim 5, characterized in that, The preset similarity threshold and the preset conflict threshold are set according to the statistical distribution quantiles when the modal complementarity is optimal in the historical data.

7. The intelligent inspection method for the integration of Internet of Things and virtual reality based on the AI large model according to claim 1, characterized in that, The data fusion module performs cross-modal correlation analysis on the spatio-temporal synchronized data through the AI large model to generate device status feature data and abnormal area mask data, including: Perform cross-modal correlation modeling on the multimodal features in the spatio-temporal synchronized data through the multi-head cross-attention mechanism to generate a fused feature vector; Calculate the cosine similarity based on the fused feature vector and the pre-trained embedding vector of the normal device operating state to generate the abnormal confidence in the device status feature data; Segment the spatial grid image data through a sliding window, extract the gradient distributions of the texture features and depth features within the window, and generate a candidate abnormal area mask; Optimize the candidate abnormal area mask using morphological closing operations to eliminate isolated noise points and fill in the hole areas, and generate abnormal area mask data; Compare the abnormal confidence with the preset abnormal confidence threshold, and mark it as the device abnormal state when the abnormal confidence exceeds the threshold.

8. The intelligent inspection method for the integration of the Internet of Things and virtual reality based on the AI large model according to claim 7, wherein, The pre-trained embedding vector is generated through training with historical normal data, and the abnormal confidence threshold is set according to the statistical distribution quantiles of historical abnormal samples.

9. The intelligent inspection method for the integration of the Internet of Things and virtual reality based on the AI large model according to claim 1, characterized in that, The rendering pipeline module fuses the device abnormal feature data and the defect area mask data into the 3D inspection scene to generate a virtual-real superposed inspection interaction interface, including: Convert the abnormal area mask data into a transparency texture on the 3D model surface, and fit it to the 3D inspection scene of the target device through UV mapping; Convert the temperature mean and vibration root mean square value in the device status feature data into a thermal cloud map and a vibration intensity field, and superimpose them on the corresponding spatial coordinates of the 3D inspection scene; Fuse the defect area mask data with the thermal cloud map and the vibration intensity field through depth testing and transparency blending rendering to generate a virtual-real superposed inspection interaction interface.

10. The intelligent inspection method for the integration of the Internet of Things and virtual reality based on the AI large model according to claim 9, characterized in that, The threshold of the transparency texture is set according to the statistical distribution quantiles of the mask data of historical normal samples, and the color mapping gradient of the thermal cloud map is dynamically adjusted according to the device safe operating temperature range.

Images