Digital Twin Management Platform and Methodology for Industrial Assets Integrating Vision and IoT

Abstract

This application provides a digital twin management platform and method for industrial assets that integrates vision and IoT, relating to the field of digital twin technology. The method includes: acquiring a rolling shutter camera frame sequence carrying line-by-line exposure time metadata and a sensor time series carrying transmission timestamp logs; calculating intra-frame anisotropic error based on the rolling shutter camera frame sequence, estimating the delay distribution based on the sensor time series, and fusing the two into a deviation statistic; determining whether rolling shutter vibration distortion or variable transmission delay exists based on the deviation statistic; if so, constructing a unified time base, registering the event times in the visual data stream and the IoT data stream to obtain a unified data stream; and generating a digital twin management result for industrial assets based on the unified data stream. This invention can effectively compensate for internal sensor time distortion and network transmission uncertainty, improving the accuracy of multimodal data alignment and the reliability of industrial asset simulation.

Description

Technical Field

[0001] This application relates to the field of digital twin technology, and more specifically, to a digital twin management platform and method for industrial assets that integrates vision and IoT. Background Technology

[0002] Multimodal sensing devices, such as visual sensors and IoT sensors, are widely deployed in the operation monitoring and management of industrial assets. By integrating visual data streams and IoT data streams, digital twins can be constructed in virtual space, enabling full lifecycle management of industrial assets. However, existing multimodal data alignment methods still have shortcomings. On the one hand, rolling shutter cameras are prone to intra-frame time shifts during line-by-line exposure, resulting in temporal distortion within the visual data. On the other hand, IoT sensors introduce uncertain latency fluctuations when transmitting data with the core network via fieldbus. These temporal distortions and transmission uncertainties not only weaken the accuracy of multimodal data alignment but may also lead to deviations in the digital twin model's simulation of the industrial asset's state.

[0003] Therefore, how to compensate for the internal time distortion of sensors and the uncertainty of network transmission in heterogeneous data streams, so as to provide a more accurate multimodal data alignment scheme than existing technologies to improve the reliability of industrial asset simulation, has become an urgent technical problem to be solved in this field. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application provides a digital twin management platform and method for industrial assets that integrates vision and IoT.

[0005] Firstly, this application provides a digital twin management method for industrial assets that integrates vision and IoT, including:

[0006] Acquire visual data streams and IoT data streams of industrial assets; wherein, the visual data stream is a rolling shutter camera frame sequence carrying line-by-line exposure time metadata, and the IoT data stream is a sensor time series carrying transmission timestamp logs;

[0007] Based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata, the geometric deviation perpendicular to the scanning direction is calculated from the visual data stream as the intra-frame anisotropic error; based on the sensor time series and the transmission timestamp log, the delay distribution is estimated from the IoT data stream; the intra-frame anisotropic error and the delay distribution are fused into deviation statistics;

[0008] Based on the aforementioned deviation statistics, it is determined whether there is rolling shutter vibration distortion or variable transmission delay;

[0009] In response to the presence of at least one of rolling shutter vibration distortion or variable transmission delay, a unified time base is constructed based on the deviation statistics, and a first event time and a second event time are extracted from the visual data stream and the IoT data stream respectively under the unified time base. The first event time and the second event time are registered to obtain a unified data stream.

[0010] Based on the unified data flow, digital twin management is performed on the industrial assets to obtain the digital twin management results of the industrial assets.

[0011] Optionally, calculating the geometric deviation perpendicular to the scanning direction as intra-frame anisotropy error from the visual data stream based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata includes:

[0012] High-frequency motion data generated by an inertial measurement unit rigidly connected to a rolling shutter camera is acquired, and the characteristic vibrations of the industrial asset are parameterized based on the high-frequency motion data to obtain initial vibration parameters.

[0013] Starting with the initial vibration parameters, an iterative optimization process is performed with the goal of minimizing image entropy, wherein each iteration includes:

[0014] Based on the vibration parameters of the current iteration and the line-by-line exposure time metadata, a geometric correction function is generated;

[0015] The geometric correction function is applied to the target frame in the rolling shutter camera frame sequence to obtain a candidate corrected image;

[0016] Calculate the image entropy of the candidate corrected image; and update the vibration parameters according to the negative gradient direction of the image entropy;

[0017] When the image entropy converges or reaches the preset number of iterations, the iterative optimization process is terminated, and the geometric deformation corresponding to the finally converged vibration parameter is taken as the geometric deviation.

[0018] Optionally, updating the vibration parameters includes:

[0019] Calculate the first-order gradient vector and second-order Hessian matrix of the image entropy function of the candidate corrected image at the vibration parameters of the current iteration;

[0020] The second-order Hessian matrix is ​​positive definite to obtain the corrected Hessian matrix.

[0021] Based on the first-order gradient vector and the corrected Hessian matrix, the update amount of the vibration parameters is determined, and the vibration parameters are updated accordingly.

[0022] Optionally, the step of performing positive definite correction on the second-order Hessian matrix to obtain the corrected Hessian matrix includes:

[0023] Determine whether the second-order Hessian matrix is ​​a positive definite matrix;

[0024] In response to the fact that the second-order Hessian matrix is ​​a non-positive definite matrix, the following steps are performed:

[0025] Obtain real-time operating condition data synchronized with the candidate corrected image from the IoT data stream;

[0026] Based on the real-time operating data and the preset correlation model characterizing the health status of the industrial assets, the expected vibration parameters corresponding to the real-time operating data are determined.

[0027] Construct a regularization matrix, wherein the penalty strength of the regularization matrix is ​​a function of the difference between the vibration parameters of the current iteration and the expected vibration parameters;

[0028] The regularization matrix is ​​added to the second-order Hessian matrix to obtain the modified Hessian matrix.

[0029] Optionally, estimating the latency distribution from the IoT data stream based on the sensor time series and the transmission timestamp log includes:

[0030] Parse the transmission timestamp log to obtain the arrival and forwarding timestamps of each data point in the sensor time series at multiple intermediate nodes in the data transmission path;

[0031] Based on the arrival and forwarding timestamps of the multiple intermediate nodes, the end-to-end data transmission path is divided into multiple transmission segments, which include at least a fieldbus transmission segment and a core network transmission segment.

[0032] For each of the multiple transmission segments, independently construct a probability distribution sub-model of its delay;

[0033] By fusing multiple probability distribution sub-models, the overall time delay distribution of the multimodal model is obtained.

[0034] Optionally, before dividing the end-to-end data transmission path into multiple transmission segments, the method further includes: extracting delay fingerprints from the links and intermediate nodes in the data transmission path, wherein the delay fingerprint extraction includes:

[0035] For a link consisting of any two adjacent intermediate nodes, the online transmission delay is calculated based on the forwarding timestamp when the data point leaves the previous intermediate node and the arrival timestamp when it arrives at the next intermediate node. Statistical features characterizing the online transmission delay are then extracted to form the link fingerprint of the corresponding link.

[0036] For any intermediate node in the data transmission path, the intra-node processing latency is calculated based on the arrival timestamp of the data point to the intermediate node and the forwarding timestamp of the data point to leave the intermediate node. Statistical features characterizing the intra-node processing latency are then extracted to form the node fingerprint of the intermediate node.

[0037] Optionally, dividing the end-to-end data transmission path into multiple transmission segments includes:

[0038] Cluster the link fingerprint and the node fingerprint respectively, and classify the links that meet the preset link clustering conditions into the same link category, and classify the intermediate nodes that meet the preset node clustering conditions into the same node category;

[0039] Based on the link category and the node category, the data transmission path is divided to obtain multiple transmission segments with different physical meanings.

[0040] Optionally, fusing multiple probability distribution sub-models to obtain the overall time delay distribution of the multimodal model includes:

[0041] Based on multiple probability distribution sub-models, multiple segment combination operation states are identified and defined, wherein each segment combination operation state corresponds to a specific delay mode for each transmission segment.

[0042] For each of the multiple segmented combined operating states, the conditional probability distribution of the overall time delay corresponding to each state is calculated;

[0043] Based on real-time observed latency data, estimate the probability that the data transmission path is in each segmented combination operation state at the current moment;

[0044] The overall time delay distribution of the multimodal mode is constructed by weighting and summing multiple conditional probability distributions according to their corresponding probabilities.

[0045] Optionally, based on the aforementioned deviation statistics, determining whether rolling shutter vibration distortion or variable transmission delay exists includes:

[0046] A cross-modal physical model is established, which is used to characterize the mapping relationship from the shared potential physical state parameters to the intra-frame anisotropic error and the time delay distribution;

[0047] Based on the currently calculated intra-frame anisotropic error, the cross-modal physical model is used for inverse deduction to estimate the corresponding potential physical state parameters;

[0048] Based on the estimated potential physical state parameters, the cross-modal physical model is used to make a positive prediction and generate the expected time delay distribution.

[0049] Calculate the statistical distance between the expected latency distribution and the latency distribution actually estimated from the IoT data stream;

[0050] If the statistical distance is greater than or equal to a preset inconsistency threshold, it is determined that there is rolling shutter vibration distortion or variable transmission delay.

[0051] Secondly, this application provides a digital twin management platform for industrial assets that integrates vision and IoT, including:

[0052] The acquisition module is used to acquire visual data streams and IoT data streams of industrial assets; wherein, the visual data stream is a rolling shutter camera frame sequence carrying line-by-line exposure time metadata, and the IoT data stream is a sensor time series carrying transmission timestamp logs;

[0053] The first processing module is used to calculate the geometric deviation perpendicular to the scanning direction as intra-frame anisotropic error from the visual data stream based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata; estimate the delay distribution from the IoT data stream based on the sensor time series and the transmission timestamp log; and fuse the intra-frame anisotropic error and the delay distribution into deviation statistics.

[0054] The judgment module is used to determine whether there is rolling shutter vibration distortion or variable transmission delay based on the deviation statistics.

[0055] The second processing module is used to respond to the presence of at least one of rolling shutter vibration distortion or variable transmission delay, construct a unified time base based on the deviation statistics, and extract the first event time and the second event time from the visual data stream and the IoT data stream respectively under the unified time base, and register the first event time and the second event time to obtain a unified data stream;

[0056] The management module is used to perform digital twin management of the industrial assets based on the unified data flow, and obtain the digital twin management results of the industrial assets.

[0057] Compared with existing technologies, this application, by introducing intra-frame anisotropic error modeling and multimodal delay distribution estimation mechanisms, can achieve high-precision alignment of visual data and IoT data under a unified time base, overcoming the limitations of traditional methods that rely solely on single timestamp matching. By fusing vibration compensation based on image entropy optimization and transmission delay estimation based on cross-segment probability modeling, this invention has significant advantages in eliminating rolling shutter distortion and network jitter, thereby effectively improving the robustness of heterogeneous data fusion. Based on this, digital twins can more realistically map the operating status of industrial assets, ensuring their reliability under complex operating conditions. Compared with existing technologies, this invention has significant improvements in multimodal alignment accuracy, data consistency, and traceability of asset status simulation in industrial scenarios, providing more stable digital support capabilities for industrial systems. Attached Figure Description

[0058] Figure 1 A flowchart illustrating the industrial asset digital twin management method integrating vision and IoT provided in this application embodiment;

[0059] Figure 2 A flowchart illustrating a method for updating vibration parameters provided in this application embodiment;

[0060] Figure 3 A flowchart illustrating a method for obtaining a modified Hessian matrix, provided as an embodiment of this application;

[0061] Figure 4 A schematic diagram of an industrial asset digital twin management platform integrating vision and IoT provided in an embodiment of this application. Detailed Implementation

[0062] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0063] See Figure 1 The diagram shows a flowchart of a digital twin management method for industrial assets integrating vision and IoT, provided in an embodiment of this application. The method includes steps S101 to S105, wherein:

[0064] S101: Acquire visual data streams and IoT data streams of industrial assets; wherein, the visual data stream is a rolling shutter camera frame sequence carrying line-by-line exposure time metadata, and the IoT data stream is a sensor time series carrying transmission timestamp logs;

[0065] S102: Based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata, calculate the geometric deviation perpendicular to the scanning direction from the visual data stream as the intra-frame anisotropic error; based on the sensor time series and the transmission timestamp log, estimate the delay distribution from the IoT data stream; fuse the intra-frame anisotropic error and the delay distribution into deviation statistics;

[0066] S103: Based on the aforementioned deviation statistics, determine whether there is rolling shutter vibration distortion or variable transmission delay;

[0067] S104: In response to the presence of at least one of rolling shutter vibration distortion or variable transmission delay, based on the deviation statistics, a unified time base is constructed, and a first event time and a second event time are extracted from the visual data stream and the IoT data stream respectively under the unified time base. The first event time and the second event time are registered to obtain a unified data stream.

[0068] S105: Based on the unified data flow, perform digital twin management on the industrial assets to obtain the digital twin management results of the industrial assets.

[0069] Regarding the above S101:

[0070] Specifically, the visual data stream is acquired by a rolling shutter camera installed in the industrial field. During line-by-line exposure, the rolling shutter camera attaches corresponding exposure start and end time information to each scan line, thus forming a frame sequence carrying line-by-line exposure time metadata. To ensure data continuity and accuracy, the rolling shutter camera is calibrated at the factory to ensure stable operation of its internal clock. When used in the industrial field, the camera can also be connected to a host computer via a network interface or bus interface to periodically calibrate the consistency between its internal clock and the system's main clock. This ensures that the line-by-line exposure time metadata accurately reflects the time position of each image acquisition line.

[0071] In practical applications, the camera can employ a common CMOS rolling shutter imaging device, such as an industrial camera with a resolution of 1920×1080. The camera continuously acquires data from the target industrial asset at a fixed frame rate, for example, set to 30 frames per second. When outputting each frame, the camera records the line-by-line exposure time metadata as an additional log or metadata file. This file is linked to the image data via time stamps, thus forming a complete visual data stream. This visual data stream includes not only the image pixel matrix itself but also the time information associated with each frame and even each line of scanning, providing the foundation for subsequent processing.

[0072] Meanwhile, the IoT data stream is acquired by various sensors deployed on industrial assets. These sensors may include temperature sensors, pressure sensors, vibration sensors, current sensors, or other detection units capable of reflecting the asset's operational status. Each sensor records the data sampling time during acquisition and generates a transmission-related timestamp log as the data is transmitted through the communication network. This timestamp log typically includes information about the time the data arrives at different network nodes or gateways, describing the data's propagation along the transmission path.

[0073] During implementation, IoT sensors transmit collected data to edge computing nodes or servers via fieldbus or wireless transmission protocols. To ensure timestamp accuracy, the sensors can employ synchronization mechanisms, such as calibrating their local clocks using Network Time Protocol (NTP) or GPS-based time signals. This ensures that the sampling times of the sensor time series have a consistent reference across different sensors.

[0074] Among them, fieldbuses include CAN bus, Modbus, Profibus, etc.; wireless transmission protocols include Wi-Fi, ZigBee, LoRa, etc.

[0075] For example, in a typical scenario, a temperature sensor collects the surface temperature of industrial equipment at a sampling frequency of once per second, adding a sampling timestamp along with the temperature data; a vibration sensor collects acceleration signals at a sampling frequency of 1000 times per second, generating a corresponding timestamp sequence. When this data enters the gateway via the fieldbus, the gateway appends a forwarding timestamp to the header of the data packet to record the arrival and forwarding times of the data at different communication stages. All this information together constitutes the IoT data stream.

[0076] Therefore, through the above method, the system can simultaneously obtain a rolling shutter camera frame sequence carrying line-by-line exposure time metadata as a visual data stream, and a sensor time series carrying transmission timestamp logs as an IoT data stream. Both data streams include raw observations and time-related annotation information, providing a data foundation for subsequent multimodal alignment and digital twin modeling.

[0077] Regarding S102 above:

[0078] The method provided in this application, after acquiring the visual data stream and the IoT data stream, performs step S102 to perform error analysis and statistical modeling on the visual data stream and the IoT data stream respectively.

[0079] First, by analyzing the rolling shutter camera frame sequence and line-by-line exposure time metadata in the visual data stream, the system can analyze the pixel arrangement relationship along and perpendicular to the scanning direction in the image. Because the sampling times of different lines differ slightly during the line-by-line scanning process of the rolling shutter camera, this time difference will manifest as geometric distortion in the image when the monitored industrial asset is in motion or vibrating.

[0080] To quantify this distortion, this implementation uses line-by-line exposure time metadata as a time reference, comparing the relative position changes of reference feature points in the target image between different lines to obtain the offset perpendicular to the scanning direction. These offsets reflect the deformation of each line of the image relative to an ideal static scene, and after statistical processing, can be used as a characterization of intra-frame anisotropy error.

[0081] For example, in a machine tool monitoring scenario, the edges of the workpiece captured by the camera should exhibit a regular straight-line structure. However, due to the rolling shutter effect, the edge lines may appear locally curved or tilted. By using line-by-line exposure time metadata, the acquisition time of each line of the image can be mapped to a unified time axis. Combined with the continuity of the workpiece's geometric features, the edge positions in the actual acquired images can be compared and analyzed. The resulting offset can then be used as a geometric deviation to measure the inconsistency between lines within a frame.

[0082] Secondly, based on the sensor time series and transmission timestamp logs in the IoT data stream, the system can extract the arrival and forwarding time information of each data point at different communication nodes. By comparing the sampling timestamp and the transmission timestamp, the system can assess the latency of data at different stages such as acquisition, transmission, and forwarding, and form an overall latency distribution. This latency distribution can reflect the uncertainties existing in the transmission of IoT data streams, including random jitter, network congestion, and node processing delays.

[0083] To improve the stability of the analysis, this implementation method can use the statistical results of multiple observations when estimating the time delay distribution, rather than relying solely on a single measurement, in order to avoid bias caused by occasional anomalies.

[0084] For example, in monitoring industrial pump stations, the sampling data generated by pressure sensors typically needs to enter the edge gateway via a fieldbus and then be uploaded to the server via the core network. The latency at different stages of the transmission path varies; some stages exhibit fixed processing delays, while others may vary due to network load fluctuations. By analyzing the sensor time series and the timestamp records of each node, the end-to-end latency distribution can be obtained. This distribution usually does not follow a single pattern but rather exhibits multi-peak or long-tail characteristics; therefore, statistical modeling is a more reasonable approach to characterize it.

[0085] Finally, the intra-frame anisotropic error obtained from the visual data stream and the latency distribution obtained from the IoT data stream are fused to form a deviation statistic. This deviation statistic can comprehensively describe the temporal uncertainties from the perception layer and the communication layer, thus providing a basis for subsequent data alignment and unified processing. During the fusion process, weighted statistics or joint modeling can be performed on deviation information from different time domains and different sources to ensure that the results take into account the characteristics of both types of data.

[0086] Therefore, this implementation can simultaneously obtain the intra-frame distortion characteristics of the image acquisition stage and the time delay characteristics of the sensor data transmission stage, and integrate the two into a unified deviation statistical result, thereby providing reliable data support for subsequent digital twin data registration and management.

[0087] Regarding the above S103:

[0088] After obtaining the deviation statistics, this application determines whether there is rolling shutter vibration distortion or variable transmission delay based on the deviation statistics.

[0089] Specifically, the deviation statistics include intra-frame anisotropic errors from the visual data stream and latency distributions from the IoT data stream. The system can identify whether abnormal states exist by analyzing these two types of deviation data.

[0090] Regarding visual data, the system first compares the ideal geometric structure of the same scene with the actual image features acquired. If the intra-frame anisotropy error shows a significant shift across multiple adjacent frames, or if the shift pattern corresponds to the vibration characteristics of the device, it can be determined that the rolling shutter vibration distortion is relatively obvious. To avoid misjudgments caused by transient noise, the system typically uses a certain number of consecutive frames for statistical analysis.

[0091] For example, when metal components of industrial equipment are subjected to high-frequency vibration, the edges of their images will periodically bend. By observing this periodic shift in the deviation statistics, the system can determine that the rolling shutter imaging has been affected by vibration.

[0092] Regarding IoT data, the system performs volatility analysis based on the latency distribution derived from transmission timestamp logs. If the latency distribution exhibits a clear multi-peak pattern, or if it fluctuates significantly within a short period and is difficult to explain using a fixed latency model, then it can be determined that variable transmission latency exists.

[0093] For example, in a wireless transmission environment in an industrial setting, when network load suddenly increases or interference intensifies, the arrival time of data packets becomes unstable, leading to asymmetrical or long-tailed delay distributions. In such cases, the system can determine that there is uncertainty in the transmission of IoT data streams.

[0094] Furthermore, when the statistical results of the deviation between the visual data stream and the IoT data stream are correlated in the time dimension, the system can make a joint judgment by combining the two types of results.

[0095] For example, if the intra-frame anisotropic error increases at the same time, and the latency distribution of IoT data also shows large fluctuations, it may indicate that industrial assets are in a state of strong interference or unstable operation.

[0096] Using the above method, the system can accurately identify the presence of rolling shutter vibration distortion or variable transmission delay, relying solely on visual and IoT observation data, without requiring complex external calibration. This judgment directly affects the reliability of subsequent data alignment strategies and digital twin modeling.

[0097] Regarding S104 above:

[0098] In one optional implementation, when it is determined from the deviation statistics that there is at least one of rolling shutter vibration distortion or variable transmission delay, the system will execute step S104, that is, construct a unified time base based on the deviation statistics, and perform event time registration on the visual data stream and IoT data stream under the unified time base.

[0099] First, establishing a unified time base refers to creating a consistent time reference among multiple existing independent time sources. Visual data streams rely on camera line-by-line exposure time metadata, while IoT data streams rely on sensor local clocks and timestamp logs during transmission. Due to differences in their acquisition mechanisms and transmission links, simply relying on raw timestamps for pairing often leads to discrepancies.

[0100] This implementation introduces deviation statistics, using intra-frame anisotropic errors from rolling shutter distortion and delay distributions from IoT transmission processes as the basis for time correction, to adjust the original time axis and thus obtain a unified time base.

[0101] In practice, the system first standardizes the start and end times of each line of exposure in the visual data stream so that the time information of each frame can be mapped to a time axis with a unified time base.

[0102] At the same time, the timestamp logs of IoT data streams are corrected based on the statistical characteristics of latency distribution. For example, by eliminating fixed latency components on the network path and compensating for the effects of random jitter, the sensor sampling time is mapped to the equivalent sampling time under a unified time base.

[0103] Next, under a unified time base, the system extracts event times from both the visual data stream and the IoT data stream. Event times in the visual data stream can be moments when a target object undergoes a significant change within an image frame, such as a sudden shift in the position of a workpiece edge or the completion of a movement cycle by a device component. Event times in the IoT data stream can be moments when a sensor detects a state parameter crossing a threshold, a vibration signal reaching a peak, or a sudden change in current / pressure. These event times are all labeled with a unified time base, ensuring comparability.

[0104] After obtaining the first event time from the visual data stream and the second event time from the IoT data stream, the system registers them. Registration is accomplished by calculating the relative difference between the event times and aligning them. To ensure registration stability, the system compares multiple event times within a certain time window and selects the time point with the highest overlap as the alignment benchmark. This not only handles the synchronization of single events but also maintains global consistency between data streams during long-term operation.

[0105] For example, in a scenario monitoring rotating equipment, the visual data stream might record the time point when the rotor passes a specific location, while an IoT vibration sensor might detect a vibration peak at the same moment. Due to rolling shutter vibration distortion and network latency, the timestamps of the two might deviate. By constructing a unified time base and registering the events, the system can align the two, ensuring that the changes in the rotor image and the peak of the vibration signal accurately correspond on a unified time axis.

[0106] Finally, after event time registration, the system generates a unified data stream. This unified data stream integrates the temporal information of both visual and IoT data, enabling subsequent digital twin modeling of industrial assets to be based on a consistent time reference, thereby ensuring the accuracy and reliability of the simulation results.

[0107] Regarding the above S105:

[0108] In one optional implementation, after obtaining the unified data stream, the system executes step S105, that is, based on the unified data stream, performs digital twin management of the industrial assets and outputs the corresponding management results.

[0109] First, a unified data stream is imported into the digital twin modeling engine. This modeling engine can include a multi-layered structure: the first is a physical model layer, used to establish basic parameter relationships based on the physical characteristics of industrial assets, such as the geometry of equipment, kinematic parameters, energy conversion relationships, etc.; the second is a data-driven layer, used to receive observation data from the unified data stream in real time to dynamically update the model status; and the third is a business logic layer, used to generate management results based on the model's operating status, such as health assessments, anomaly warnings, or maintenance recommendations.

[0110] In practical implementation, visual information from the unified data stream can be used to update the spatial state of industrial assets, such as the geometric deformation of equipment appearance, the position of components, and their movement trajectories. Meanwhile, the IoT data stream provides supplementary information on physical quantities for the model, such as temperature distribution, pressure changes, vibration amplitude, and current load.

[0111] By synchronizing the two types of information under a unified time base, digital twins can maintain consistency with actual industrial assets.

[0112] During operation, the system can continuously analyze a unified data stream to reflect the status of industrial assets in real time.

[0113] For example, when visual data reflects fluctuations in the rotational speed of a rotating component, and the IoT sensor simultaneously detects an abnormal increase in current load, the digital twin will combine the two to show that the device may be operating under unbalanced load.

[0114] Furthermore, management results can manifest as generating operation and maintenance logs, pushing anomaly alerts, or triggering preventative maintenance processes.

[0115] Furthermore, digital twin management is not limited to anomaly detection; it can also be used to optimize decision-making.

[0116] For example, in a pump station monitoring scenario, a unified data stream can reflect the pump's operating curves under different conditions. The system archives and compares these curves in a digital twin environment, thereby helping maintenance personnel select more energy-efficient or more stable operating modes. In this way, the digital twin results can support both short-term operational monitoring and long-term planning and management.

[0117] For example, in a production workshop, visual cameras monitor the appearance of equipment, while IoT sensors collect temperature and vibration signals. After forming a unified data stream through the aforementioned steps, the digital twin can present the real-time operating status of key equipment in the workshop and issue warnings to maintenance personnel before the detected temperature exceeds a safe threshold. This management approach not only improves the accuracy of monitoring but also reduces the risk of sudden failures.

[0118] Therefore, this implementation method enables the virtual model of industrial assets to accurately map the actual operating conditions in real time through digital twin management based on unified data flow, and outputs management results that can be used for monitoring, diagnosis, optimization and decision-making, thereby significantly improving the reliability and effectiveness of digital operation and maintenance of industrial assets.

[0119] Optionally, calculating the geometric deviation perpendicular to the scanning direction as intra-frame anisotropy error from the visual data stream based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata includes:

[0120] High-frequency motion data generated by an inertial measurement unit rigidly connected to a rolling shutter camera is acquired, and the characteristic vibrations of the industrial asset are parameterized based on the high-frequency motion data to obtain initial vibration parameters.

[0121] Starting with the initial vibration parameters, an iterative optimization process is performed with the goal of minimizing image entropy, wherein each iteration includes:

[0122] Based on the vibration parameters of the current iteration and the line-by-line exposure time metadata, a geometric correction function is generated;

[0123] The geometric correction function is applied to the target frame in the rolling shutter camera frame sequence to obtain a candidate corrected image;

[0124] Calculate the image entropy of the candidate corrected image; and update the vibration parameters according to the negative gradient direction of the image entropy;

[0125] When the image entropy converges or reaches the preset number of iterations, the iterative optimization process is terminated, and the geometric deformation corresponding to the finally converged vibration parameter is taken as the geometric deviation.

[0126] In order to more accurately calculate the geometric deviation perpendicular to the scanning direction in the image of a rolling shutter camera, thereby improving the ability to characterize intra-frame anisotropic errors, this implementation can overcome the defect of being easily interfered with by noise when relying solely on image pixel analysis, and can better capture dynamic distortion caused by device vibration.

[0127] This implementation first uses an inertial measurement unit to parameterize the characteristic vibrations of industrial assets, obtaining a reasonable initial vibration estimate. Then, it iterative optimization is used to gradually correct the estimate, resulting in final parameters that accurately reflect geometric distortion. This not only improves the convergence speed of error estimation but also enhances robustness under complex operating conditions.

[0128] In practice, a rolling shutter camera is rigidly mounted on the same platform as an inertial measurement unit (IMU) via a structurally fixed design, enabling both to sense the same motion state. The IMU, which may include an accelerometer and a gyroscope, outputs high-frequency motion data that reflects the minute vibrations and attitude changes of the industrial asset during imaging. The system preprocesses this high-frequency motion data, performing noise reduction, integration, and feature extraction to obtain a set of parameters characterizing the equipment's vibrations, such as vibration amplitude, direction, and period. These parameters serve as initial vibration parameters, providing a starting point for subsequent image distortion correction.

[0129] After obtaining the initial vibration parameters, the system performs an iterative optimization process aimed at minimizing image entropy. In each iteration, the system generates a set of geometric correction functions based on the vibration parameters of the current iteration and the line-by-line exposure time metadata. These geometric correction functions describe the positional compensation relationship of pixels perpendicular to the scanning direction. The system then applies these geometric correction functions to the target frame in the rolling shutter camera frame sequence to geometrically correct distorted areas in the image, thereby obtaining candidate corrected images.

[0130] Subsequently, the system evaluates the information content of the candidate correction images, using image entropy as the metric. Image entropy reflects the randomness and blurriness of an image. Higher image entropy indicates more irregularities and distortion in the image; lower image entropy indicates a clearer image structure and effective distortion compensation. Therefore, by using the reduction of image entropy as the optimization objective, a better distortion correction result can be gradually achieved.

[0131] In each iteration, the system adjusts the vibration parameters based on the trend of image entropy changes in the candidate correction images. Specifically, when a decrease in image entropy is detected, the system continues to correct the vibration parameters in that direction; if the entropy value no longer decreases or oscillates, it indicates that the parameters are close to the optimal solution, and the step size needs to be adjusted or convergence needs to be determined. To avoid getting trapped in local optima, the system can also introduce certain perturbations or random search mechanisms during the iteration process to explore potentially better solutions.

[0132] When the image entropy remains stable over several consecutive iterations, or when the number of iterations reaches a preset threshold, the system terminates the iterative optimization process. At this point, the geometric deformation corresponding to the finally converged vibration parameters, i.e., the geometric deviation perpendicular to the scanning direction, is used to characterize the intra-frame anisotropic error.

[0133] For example, in a scenario monitoring machine tool processing, the workpiece edge line captured by the camera should be straight, but due to vibration, the edge may appear curved. Using motion data from the inertial measurement unit, the direction and amplitude of the workpiece vibration can be initially inferred; through image entropy iterative optimization, the system further refines this estimate, restoring the straight structure of the edge line in the corrected image. The resulting geometric deviation not only quantifies the degree of rolling shutter distortion but also provides a reliable basis for subsequent time alignment and digital twin modeling.

[0134] Through the above methods, this embodiment achieves high-precision estimation of intra-frame anisotropic errors in visual data streams, improving the overall adaptability and robustness of the method in this application to industrial asset motion distortion.

[0135] Optional, see Figure 2A flowchart of a method for updating vibration parameters provided in this application embodiment includes steps S201 to S203, wherein:

[0136] S201: Calculate the first-order gradient vector and second-order Hessian matrix of the image entropy function of the candidate corrected image at the vibration parameter of the current iteration;

[0137] S202: Perform positive definite correction on the second-order Hessian matrix to obtain the corrected Hessian matrix;

[0138] S203: Based on the first-order gradient vector and the corrected Hessian matrix, determine the update amount of the vibration parameters and update the vibration parameters.

[0139] To improve the convergence efficiency and stability of vibration parameter updates during image entropy iterative optimization, this application proposes introducing gradient and second-order approximation methods to determine the correction amount in the parameter update stage. This method can ensure the correct optimization direction while avoiding divergence caused by excessive or unreasonable updates, thus more reliably obtaining the optimal vibration parameters characterizing geometric distortion.

[0140] In the specific implementation process, the system first establishes an evaluation function for image entropy for the candidate correction images. To update the vibration parameters, the system needs to analyze the sensitivity of image entropy to changes in vibration parameters. Therefore, the system calculates the rate of change of image entropy in different parameter directions near the vibration parameters of the current iteration. Specifically:

[0141] Based on the current vibration parameters, apply small perturbations in each direction, such as slightly adjusting the vibration amplitude or vibration direction;

[0142] A new geometric correction function is generated for the perturbed vibration parameters and applied to the target frame to obtain a new candidate correction image;

[0143] Calculate the image entropy values ​​of these candidate corrected images respectively, and compare them with the current entropy value;

[0144] Based on the entropy difference, the changing trend of the image entropy function at the current vibration parameter is obtained, and then a first-order gradient vector is formed.

[0145] While obtaining the first-order gradient vector, the system also estimates the second-order change characteristics. Specifically, the system observes the magnitude of image entropy changes under different parameter combinations and approximates the second-order curvature information by comparing the entropy differences before and after multiple perturbation combinations, thereby constructing an initial Hessian matrix. This matrix describes the curvature of the image entropy function in the parameter space and is used to guide the update step size and direction.

[0146] After obtaining the initial Hessian matrix, the system checks whether it satisfies positive definiteness. If a non-positive definite case is found, meaning that the corresponding direction may lead to optimization instability, it is handled through positive definiteness correction. Specifically, a certain adjustment factor is applied to the diagonal elements of the matrix to make the matrix as a whole positive definite, thereby ensuring that parameter updates always move in the direction of decreasing image entropy.

[0147] Next, the system combines the first-order gradient vector with the corrected Hessian matrix to calculate the update of the vibration parameters. In other words, the system considers not only the direction of the current entropy decrease but also the curvature information of the parameter space to determine the update step size. This avoids oscillations caused by relying solely on the gradient and also avoids slow convergence due to excessively small updates.

[0148] In specific applications, such as monitoring high-speed rotating machinery, updating vibration parameters may involve adjusting the rotation frequency and amplitude. When the system detects a rapid decrease in entropy in a certain direction based on the first-order gradient, the curvature information of the Hessian matrix helps determine whether the update amplitude needs to be limited to prevent excessive parameter adjustment that could lead to over-straightening or distortion of the edge structure. Ultimately, the system corrects the current vibration parameters based on the calculated update amount and proceeds to the next iteration.

[0149] Through the above method, this embodiment achieves efficient updating of vibration parameters during image entropy optimization, which can significantly improve the convergence speed while ensuring stability, thereby obtaining the geometric deviation of the rolling shutter image more accurately.

[0150] Optional, see Figure 3 A flowchart of a method for obtaining a modified Hessian matrix provided in this application includes steps S301 to S304, wherein:

[0151] This application determines whether the second-order Hessian matrix is ​​a positive definite matrix; in response to the second-order Hessian matrix being a non-positive definite matrix, the following steps are performed:

[0152] S301: Obtain real-time operating condition data synchronized with the candidate correction image from the IoT data stream;

[0153] S302: Based on the real-time operating data and a preset correlation model characterizing the health status of the industrial asset, determine the expected vibration parameters corresponding to the real-time operating data;

[0154] S303: Construct a regularization matrix, wherein the penalty strength of the regularization matrix is ​​a function of the difference between the vibration parameter of the current iteration and the expected vibration parameter;

[0155] S304: Add the regularization matrix to the second-order Hessian matrix to obtain the modified Hessian matrix.

[0156] This application proposes a technical solution to address the problem of non-positive definite Hessian matrices that may occur during iterative optimization, by combining real-time IoT operating data for positive definite correction.

[0157] The technical problem this solution aims to solve is: how to avoid distortion or divergence in the optimization direction during parameter updates, so that the vibration parameter correction process can remain stable and reliable under complex working conditions.

[0158] When traditional image entropy-based optimization processes rely solely on visual information, the calculated second-order Hessian matrix may be non-positive definite in certain dimensions, leading to uncertainty in parameter update directions and potentially causing the iterative process to deviate from the optimal solution. Therefore, this implementation introduces IoT operating condition data synchronized with the candidate correction images and utilizes an association model characterizing the health status of industrial assets for assistance, thereby incorporating external physical constraints into the optimization process to achieve matrix positive definiteness.

[0159] In the specific implementation process, the system first checks the properties of the second-order Hessian matrix in each iteration. If the detection result shows that the matrix has negative characteristics in some directions, it is determined to be a non-positive definite matrix. At this time, the system initiates the positive definiteness correction process.

[0160] The first step in the correction process is to synchronously acquire IoT operating condition data. The system extracts data points from the IoT data stream that match the timestamps of the candidate correction images, such as temperature, pressure, vibration amplitude, or current load at the corresponding moment. In this way, it can be ensured that the visual data and IoT data reflect the same physical state.

[0161] Subsequently, the system interprets this operating data using a pre-defined industrial asset health status correlation model. This model can be a hybrid model constructed from both physical mechanisms and empirical statistics, or it can incorporate artificial intelligence algorithms to some extent. For example, the model's input consists of real-time collected multi-source data such as temperature, vibration, and rotational speed, and its output is a predicted value for the expected vibration parameters. This predicted value reflects the reasonable vibration pattern that the industrial asset should exhibit under the current operating conditions, thus providing a reference for subsequent regularization.

[0162] Next, the system constructs a regularization matrix. This matrix determines a penalty intensity by comparing the current iterative vibration parameters with the model's predicted vibration parameters. A larger penalty intensity indicates a greater discrepancy between the system's current parameters and the reasonable operating conditions, requiring stronger corrective constraints. In this way, the regularization matrix mathematically adds a stability factor to the original Hessian matrix, preventing the optimization process from deviating from the actual physical state of the equipment.

[0163] After constructing the regularization matrix, the system adds it to the original second-order Hessian matrix to obtain the corrected Hessian matrix. Since the design of the regularization matrix ensures that it is positive definite, the result of the addition is also a positive definite matrix, thus ensuring that the parameter update direction always faces the region where image entropy decreases and remains consistent with the actual operating conditions of the device.

[0164] For example, when monitoring high-speed rotating equipment, if the calculated Hessian matrix is ​​non-positive definite due to instantaneous noise during image entropy optimization, the system will immediately access the vibration acceleration and rotational speed data collected by the IoT sensor at the same time. Based on this, the correlation model determines that the reasonable vibration mode of the equipment at that rotational speed should be within a certain range. If the current iteration parameters significantly deviate from this range, the regularization matrix will enhance the correction strength in that direction, causing the updated Hessian matrix to return to a stable positive definite state.

[0165] The correlation model collects corresponding data on intra-visual geometric deviation and IoT latency distribution under different operating conditions of the device. This data is then used as input-output pairs and trained and fitted using multiple linear regression, support vector regression, or a three-layer feedforward neural network to obtain a model that reflects the relationship between operating parameters and deviations. The model's inputs can be operating variables such as speed, load, and temperature, while the outputs are geometric distortion parameters and latency characteristic values.

[0166] For example, the correlation model could be a rule-based system, where, for instance, if the temperature is greater than 80 degrees Celsius and the current is greater than 5 amperes, the expected vibration frequency is in the range of 100-120 Hz.

[0167] By employing the above methods, this embodiment not only prevents the optimization process from failing due to non-positive definite matrices, but also organically combines IoT operating condition information with visual data, making the optimization results more consistent with the actual operating status of industrial assets. This cross-modal constraint mechanism significantly improves the robustness and reliability of the method in this application under complex operating conditions.

[0168] Optionally, estimating the latency distribution from the IoT data stream based on the sensor time series and the transmission timestamp log includes:

[0169] Parse the transmission timestamp log to obtain the arrival and forwarding timestamps of each data point in the sensor time series at multiple intermediate nodes in the data transmission path;

[0170] Based on the arrival and forwarding timestamps of the multiple intermediate nodes, the end-to-end data transmission path is divided into multiple transmission segments with different physical meanings. The transmission segments include at least a fieldbus transmission segment and a core network transmission segment.

[0171] For each of the multiple transmission segments, independently construct a probability distribution sub-model of its delay;

[0172] By fusing multiple probability distribution sub-models, the overall time delay distribution of the multimodal model is obtained.

[0173] The technical solution proposed in this application aims to address the uncertainty of multi-stage and multi-type delays in the end-to-end transmission of IoT sensor data. By jointly analyzing the sensor time series and transmission timestamp logs, a delay distribution model that is more consistent with actual working conditions can be obtained.

[0174] This approach breaks down the overall transmission process into multiple physically meaningful transmission segments, models each segment independently, and then merges these segment models into a unified distribution. This avoids oversimplification of complex network latency characteristics by a single model. This significantly improves the accuracy and adaptability of latency estimation, providing a reliable basis for subsequent data alignment.

[0175] In the specific implementation process, the system first parses the transmission timestamp log. This log is automatically generated when IoT data is transmitted between different nodes, and typically includes the time when the data packet arrives at the node and the time when the node completes forwarding. The system then maps the log to the sensor time series one by one, thereby obtaining the complete transmission trajectory of each data point across multiple intermediate nodes.

[0176] For example, when a temperature sampling point is sent from the sensor and passes through the fieldbus controller, edge gateway, core switch and finally arrives at the server, the system can extract its arrival time and forwarding time at each of the above intermediate nodes.

[0177] Subsequently, based on these arrival and forwarding times, the system divides the overall data transmission path into multiple segments. Each segment corresponds to a transmission link with clearly defined physical attributes. For example, the part from the sensor to the fieldbus controller is the "fieldbus transmission segment," characterized by low latency but significantly affected by bus occupancy; the part from the edge gateway to the core switch is the "core network transmission segment," characterized by large latency fluctuations and affected by network traffic and routing policies. This division allows for more targeted modeling methods to be used for the characteristics of different segments.

[0178] After the segmentation is completed, the system independently constructs a latency probability distribution sub-model for each transmission segment. The construction process includes: statistically analyzing latency samples from multiple data points within the segment, extracting common feature values ​​such as mean, variance, and the frequency of extreme latency occurrences, and forming a probability distribution sub-model based on these features. This probability distribution sub-model is not limited to a fixed type and can adopt a suitable descriptive method according to the segment characteristics. For example, an approximate normal distribution can be used to represent stable segments, or a long-tailed distribution can be used to represent segments significantly affected by network congestion. This approach closely approximates the actual latency patterns of the segments.

[0179] After establishing the probability distribution sub-models for each segment, the system fuses these sub-models to obtain the overall end-to-end latency distribution. The fusion process can be understood as combining the latency characteristics of each segment according to the order of data transmission along the path. Since different segments may simultaneously exhibit different latency modes, the final overall latency distribution often displays multi-peak or multi-modal characteristics.

[0180] For example, in a pump station monitoring scenario, the fieldbus transmission part has a stable delay, while the core network part has a drastic delay fluctuation. The overall delay distribution after the two are combined is characterized by a main peak and a secondary peak.

[0181] In this way, this embodiment can not only identify and quantify the contribution of each segment to the end-to-end latency, but also generate an overall latency distribution that truly reflects the characteristics of complex networks. Avoiding the oversimplification of complex latency characteristics using a single distribution model in traditional methods, this embodiment makes latency estimation more consistent with actual industrial communication environments, helping to improve the reliability of subsequent data alignment and digital twin modeling.

[0182] Optionally, before dividing the end-to-end data transmission path into multiple physically distinct transmission segments, the method further includes: extracting delay fingerprints from the links and intermediate nodes in the data transmission path, wherein the delay fingerprint extraction includes:

[0183] For a link consisting of any two adjacent intermediate nodes, the online transmission delay is calculated based on the forwarding timestamp when the data point leaves the previous intermediate node and the arrival timestamp when it arrives at the next intermediate node. Statistical features characterizing the online transmission delay are then extracted to form the link fingerprint of the corresponding link.

[0184] For any intermediate node in the data transmission path, the intra-node processing latency is calculated based on the arrival timestamp of the data point to the intermediate node and the forwarding timestamp of the data point to leave the intermediate node. Statistical features characterizing the intra-node processing latency are then extracted to form the node fingerprint of the intermediate node.

[0185] To more accurately distinguish the performance differences between different links and intermediate nodes before dividing the end-to-end data transmission path into multiple physically distinct transmission segments, this application employs a delay fingerprint extraction mechanism. The technical problem this mechanism addresses is how to accurately characterize the delay characteristics of complex industrial communication networks without relying on a single end-to-end measurement, but rather through fine-grained feature extraction of each link and node.

[0186] In this way, more reliable data support can be provided for subsequent segmented modeling.

[0187] The principle is that end-to-end latency is composed of multiple link transmission and node processing stages, and different stages may have drastically different latency patterns. If the latency characteristics of each link and node can be extracted separately and formed into an independent "latency fingerprint," more detailed classification and differentiation can be achieved before overall segmentation. This approach avoids the problem of generalizing all uncertainties, thereby improving the accuracy of latency modeling.

[0188] In its implementation, the system first extracts a fingerprint of link-level latency. Taking any two adjacent intermediate nodes as an example, the system obtains the forwarding timestamp when a data point leaves the previous intermediate node and the arrival timestamp when it arrives at the next intermediate node. The difference between these two times represents the online transmission latency of that link segment. After collecting a sufficient number of data points, the system performs statistical analysis on the latency of the link, extracting features such as average transmission latency, latency fluctuation range, and latency distribution pattern. These statistical features constitute the "link fingerprint" of the link. For example, in a pump station monitoring system, the latency fingerprint of a fieldbus link typically exhibits a low mean and small fluctuations, while the latency fingerprint of a wireless link may exhibit long-tail characteristics and larger fluctuations.

[0189] For example, a link fingerprint can be a four-dimensional vector containing [average latency, latency variance, 95th percentile latency, packet loss rate], which is calculated over a 60-second sliding window.

[0190] Subsequently, the system extracts fingerprints for node-level latency. For any intermediate node, the system obtains the arrival timestamp of the data point when it arrives at the intermediate node, and the forwarding timestamp of the data point leaving after completing processing. The difference between the two is the processing latency within that intermediate node. By statistically analyzing the processing latency of multiple data points, features such as average processing time, jitter amplitude, and processing queue backlog can be extracted, thus forming the "node fingerprint" of that intermediate node.

[0191] For example, in industrial gateway devices, due to the need to perform protocol parsing and data encryption, their node fingerprints typically exhibit a large fixed latency component, while in switch nodes, fingerprint features reflect more instantaneous congestion and buffer queuing.

[0192] In a typical application scenario, as pressure sensor data sequentially passes through the fieldbus, edge gateway, core switch, and finally reaches the server, the system can generate an independent latency fingerprint for each link and each node. By comparing the fingerprint characteristics of different links and nodes, it can be observed that, for example, the fieldbus latency is stable but has a relatively high fixed value, while the wireless link latency is unstable and dispersed; furthermore, the edge gateway node processing latency is constant, while the core switch latency varies with network traffic. These differentiated fingerprints provide a clear basis for subsequently dividing transmission segments with different physical meanings.

[0193] Through the above methods, this embodiment achieves fine-grained characterization of link and node delay features. Its beneficial effects are: it can accurately distinguish the performance differences of different transmission links in complex network environments, thereby making subsequent segmented modeling and overall delay estimation more consistent with reality, and improving the accuracy and robustness of the industrial asset digital twin management method at the communication level.

[0194] Optionally, dividing the end-to-end data transmission path into multiple transmission segments with different physical meanings includes:

[0195] Cluster the link fingerprint and the node fingerprint respectively, and classify the links that meet the preset link clustering conditions into the same link category, and classify the intermediate nodes that meet the preset node clustering conditions into the same node category;

[0196] Based on the link category and the node category, the data transmission path is divided to obtain multiple transmission segments with different physical meanings.

[0197] To avoid excessive model complexity caused by simply treating all links and nodes as independent units in latency modeling, this application adopts transmission path partitioning based on latency fingerprint clustering.

[0198] The technical problem this alternative implementation aims to solve is: how to reasonably classify similar links and nodes in a complex industrial network environment by performing cluster analysis on the delay characteristics of links and nodes, thereby obtaining physically meaningful transmission segments. The principle is that although different links and nodes are physically located, if their delay fingerprints are highly similar in statistical characteristics, they can be considered as belonging to the same category, thus dividing the transmission path into stable and interpretable segments. This improves the accuracy of the delay model while reducing its complexity.

[0199] In the implementation process, the system first clusters the link fingerprints. Each link fingerprint includes statistical characteristics such as average transmission delay, fluctuation range, and distribution pattern. The system sets a set of preset link clustering conditions, such as: average transmission delay difference not exceeding a certain allowable range, fluctuation amplitude at a similar level, and highly consistent distribution pattern. Links that meet the above conditions will be classified into the same category. In this way, even links that physically belong to different sensors can be uniformly regarded as the same type of link if they exhibit similar delay patterns.

[0200] Similarly, the system clusters node fingerprints. Node fingerprints primarily describe the statistical characteristics of processing latency within a node, such as average processing latency, queue backlog characteristics, and volatility. The system sets preset node clustering conditions, such as: similar average processing times, consistent latency jitter, and the presence of a significant long-tail distribution. Nodes meeting these conditions will be grouped into the same node category. Thus, for example, multiple field gateways that may exhibit consistent latency characteristics due to using the same hardware and firmware versions can be clustered into the same node category.

[0201] Understandably, the above conditions can be set after statistical analysis of the initially collected network logs, and can be dynamically adjusted during operation.

[0202] After clustering links and nodes, the system divides the transmission path based on the categories. Specifically, the system groups links belonging to the same link category into one transmission segment, and nodes belonging to the same node category into another transmission segment. Ultimately, the end-to-end data transmission path is divided into multiple physically meaningful segments, each representing a class of links or nodes with similar delay characteristics.

[0203] For example, in a smart factory scenario, sensor data sequentially passes through fieldbus links, multiple gateway nodes, core switch links, and the cloud platform entry node. After fingerprint extraction, the system may find that the fingerprint characteristics of fieldbus links and local industrial Ethernet links are highly similar, both exhibiting stable low latency. Therefore, they can be clustered into the same link category, forming a "field transmission segment".

[0204] For example, the link fingerprint uses the K-means algorithm for clustering, where the cluster size k=3 and the distance metric is Euclidean distance, thereby classifying the links into three categories: low latency, medium latency, and high latency.

[0205] Meanwhile, if the processing delay fingerprints of multiple gateway nodes also exhibit similarities, all possessing a fixed protocol parsing delay, they will be classified into the same node category, forming a "gateway processing segment." In this way, the overall path is divided into several physically distinct parts, such as "on-site transmission segment—gateway processing segment—core network transmission segment—cloud platform access segment."

[0206] Through the above method, this embodiment realizes path segmentation based on time delay fingerprints, which not only avoids the problems of oversimplification or over-subdivision in traditional methods, but also makes each segment correspond to a clear physical meaning, which can significantly improve the interpretability and reliability of subsequent time delay distribution modeling.

[0207] Optionally, fusing multiple probability distribution sub-models to obtain the overall time delay distribution of the multimodal model includes:

[0208] Based on multiple probability distribution sub-models, multiple segment combination operation states are identified and defined, wherein each segment combination operation state corresponds to a specific delay mode for each transmission segment.

[0209] For each of the multiple segmented combined operating states, the conditional probability distribution of the overall time delay corresponding to each state is calculated;

[0210] Based on real-time observed latency data, estimate the probability that the data transmission path is in each segmented combination operation state at the current moment;

[0211] The overall time delay distribution of the multimodal mode is constructed by weighting and summing multiple conditional probability distributions according to their corresponding probabilities.

[0212] To avoid oversimplifying complex delay patterns using a single distribution model and to more accurately describe the delay patterns of industrial communication networks under different operating states, this optional implementation defines multiple segmented combined operating states, calculates the conditional distribution under each state, and then assigns probability weights to different states based on real-time observed delay data, ultimately forming a multimodal overall delay distribution.

[0213] In the specific implementation process, we first use the probability distribution sub-models of each transmission segment that have been established in the previous steps to analyze the typical patterns that they may present one by one.

[0214] For example, fieldbus segments may exhibit two modes: stable low latency and mild congestion, while core network segments may exhibit three modes: normal latency, peak latency, and long-tail latency. The system identifies and defines multiple combined segment operating states by enumerating and combining these modes. Each combined operating state represents the possible latency of the entire end-to-end path at a given moment, such as "fieldbus stable low latency + core network peak latency" or "fieldbus mild congestion + core network normal latency," etc.

[0215] After determining the combined operating states, the system will, for each combined operating state, progressively superimpose the probability distribution sub-models of its corresponding segments to deduce the overall time delay condition distribution under that state.

[0216] During the calculation, the delay samples from each segment are jointly calculated to obtain the distribution of the end-to-end delay under that combination. In this way, the conditional distribution under different combination states can clearly reflect the possible range and probabilistic characteristics of the delay.

[0217] Next, real-time observed latency data is introduced to dynamically estimate the current operating state of the system. For example, within a sliding time window, the latest latency samples are collected and compared with the conditional distributions of each combined operating state to calculate the degree of fit between the samples and each conditional distribution. States with a high degree of fit are assigned a higher probability, while states with a low degree of fit correspond to a lower probability.

[0218] For example, when real-time data exhibits significant long-tail latency characteristics, the system will increase the weight of the combined states related to "core network long-tail latency".

[0219] After estimating the probabilities of each state, the system sums all conditional probability distributions according to their corresponding probabilities to obtain the final multimodal overall time delay distribution. This overall time delay distribution is no longer limited to a single curve, but consists of multiple peaks and intervals, which can simultaneously reflect the delay patterns under normal operation and abnormal conditions.

[0220] For example, in the wireless monitoring scenario of a smart factory, the final multimodal delay distribution may include a main peak (normal delay state) and several sub-peaks (abnormal states caused by congestion or interference).

[0221] Thus, this embodiment achieves multimodal overall latency modeling based on a piecewise probability distribution sub-model. It can take into account the impact of normal and abnormal states on latency, avoiding the one-sidedness of traditional single distribution models, thereby making the digital twin management method for industrial assets more accurate and robust in the face of complex network environments.

[0222] Optionally, based on the aforementioned deviation statistics, determining whether rolling shutter vibration distortion or variable transmission delay exists includes:

[0223] A cross-modal physical model is established, which is used to characterize the mapping relationship from the shared potential physical state parameters to the intra-frame anisotropic error and the time delay distribution;

[0224] Based on the currently calculated intra-frame anisotropic error, the cross-modal physical model is used for inverse deduction to estimate the corresponding potential physical state parameters;

[0225] Based on the estimated potential physical state parameters, the cross-modal physical model is used to make a positive prediction and generate the expected time delay distribution.

[0226] Calculate the statistical distance between the expected latency distribution and the latency distribution actually estimated from the IoT data stream;

[0227] If the statistical distance is greater than or equal to a preset inconsistency threshold, it is determined that there is rolling shutter vibration distortion or variable transmission delay.

[0228] Furthermore, in order to more accurately determine whether there is rolling shutter vibration distortion or variable transmission delay, this application adopts a determination mechanism based on a cross-modal physical model.

[0229] The technical problem this mechanism aims to solve is: how to unify the intra-frame anisotropic error exhibited in visual data and the latency distribution exhibited in IoT data under the same physical state interpretation framework, so as to identify potential anomalies through consistency checks.

[0230] This application establishes a mapping relationship between shared potential physical state parameters and two types of deviations, which enables the establishment of a predictable correspondence between different modal data. Furthermore, by comparing the difference between the expected distribution and the actual observed distribution, abnormal operating conditions can be identified.

[0231] In the specific implementation process, the first step is to establish a cross-modal physical model. This model is based on a set of potential physical state parameters, such as the vibration intensity, frequency, and directivity of the device during operation, as well as the load level and stability of the network link. These parameters are reflected both in the visual modality, such as causing uneven geometric offsets within frames, and in the IoT modality, such as causing fluctuations in network latency distribution.

[0232] Therefore, the model establishes a mapping relationship from potential physical state parameters to visual error and time delay distribution, so that visual and IoT observation data are no longer isolated, but driven by the same physical mechanism.

[0233] During execution, the system will use the cross-modal physical model to perform inverse deduction based on the currently calculated intra-frame anisotropic error.

[0234] For example, the cross-modal physical model consists of a visual side model and an IoT side model, which are interconnected through shared underlying physical state parameters.

[0235] The visual side model takes the intra-frame anisotropic error of the rolling shutter image as input. It can extract the main error patterns through principal component analysis or extract distortion features using a convolutional neural network. Then, it outputs the corresponding potential vibration parameters, such as vibration amplitude, frequency, and direction, through a regression module.

[0236] The IoT side model takes potential physical state parameters as input and uses probabilistic graphical models or queuing theory models to deduce the corresponding time delay distribution, such as a single-peak distribution under normal operating conditions, a bimodal distribution under mild congestion, or a long-tail distribution under severe interference.

[0237] During reverse engineering, the system compares the output of the visual sub-model with the actual calculated intra-frame bias and obtains the most suitable latent physical state parameters through iterative search. During forward prediction, the system inputs the latent physical state parameters into the IoT-side sub-model to obtain the expected latency distribution. By comparing the expected latency distribution with the actual IoT observation distribution, inconsistencies can be determined. The model structure clearly presents the inputs, outputs, and intermediate relationships, enabling those skilled in the art to achieve consistent modeling of cross-modal data.

[0238] For example, the steps of reverse inference include: inputting the actually calculated intra-frame geometric deviations into the model, and finding potential physical state parameters that can explain these deviations through fitting and iteration.

[0239] For example, when periodic bending is detected in a rolling shutter image, the system can deduce the vibration frequency and amplitude range of industrial equipment and obtain a set of candidate potential physical state parameters accordingly.

[0240] Next, the system will use the latent physical state parameters obtained through inverse deduction to perform forward prediction using a cross-modal physical model. The goal of forward prediction is to generate an expected IoT latency distribution under these latent physical state parameters.

[0241] For example, if the derived parameters indicate that the device is in a high-frequency vibration state, the model may predict that the corresponding IoT transmission path will exhibit high latency jitter or obvious multi-peak distribution. In this way, the system can deduce the IoT modal behavior that is consistent with its physical form from the visual modality.

[0242] After generating the expected latency distribution, the system compares it with the latency distribution estimated in the actual IoT data stream. Specifically, the system performs statistical measurements on the differences between the two, including whether the overall shape is consistent, whether the peak position is shifted, and whether the distribution tails are elongated. If the difference is small, it indicates that the observation results of the visual modality and the IoT modality corroborate each other, and the current system is in a consistent physical state; if the difference is large, it indicates that at least one modality's data performance is abnormal.

[0243] To achieve automated judgment, the system sets an inconsistency threshold. When the calculated difference metric is greater than or equal to this inconsistency threshold, it is determined that rolling shutter vibration distortion or variable transmission delay exists. The inconsistency threshold can be obtained through experimental calibration or dynamically adjusted during long-term system operation to adapt to different noise levels and operating conditions in industrial environments.

[0244] For example, in an automated production workshop application scenario, a camera observes periodic jitter at the edges of materials on a conveyor belt, while the latency distribution reported by the IoT sensors simultaneously exhibits an unexpected bimodal characteristic. If there is a significant difference between the expected distribution derived from the cross-modal physical model and the actual IoT observation—for example, the expected distribution is unimodal but the actual distribution is bimodal—the system determines that variable transmission latency exists. This determination can be used for subsequent anomaly alarms and dynamic correction of the digital twin model.

[0245] It is understood that the inconsistency threshold can be determined through statistical calibration. For example, when the device is in a stable operating condition, the IoT latency distribution over a certain time window can be collected, its mean and standard deviation can be calculated, and the mean plus or minus a certain number of standard deviations can be used as the threshold interval. When the statistical distance between the cross-modal physical model's predicted distribution and the actual distribution exceeds the range of this threshold interval, it is judged as an anomaly. This threshold can also be corrected based on historical data during long-term operation.

[0246] Through the above methods, this embodiment can not only identify anomalies within a single modality, but also enhance the reliability of the judgment by utilizing cross-modal consistency. Its beneficial effects are: by introducing a cross-modal physical model, it can link two heterogeneous data types, visual and IoT, and uniformly interpret the observation results within a framework of shared physical state parameters. This allows for more accurate and robust identification of rolling shutter vibration distortion and variable transmission delay, improving the credibility and robustness of digital twin management.

[0247] In an optional embodiment, to enable those skilled in the art to construct and operate a cross-modal physical model for anomaly detection, the following executable implementation path is provided. Structurally, the model consists of a shared coding unit and two decoding paths that work in tandem around the physical state of an industrial asset at the same moment: the shared coding unit represents this physical state; the visual-side decoding path receives this physical state and reconstructs a distortion representation consistent with intra-frame anisotropic errors; the IoT-side decoding path receives the same physical state and reconstructs a distribution representation consistent with end-to-end delay statistics. The shared coding unit can employ a cascaded structure of a lightweight feedforward network and a temporal feature unit. The visual-side decoding path can include a convolutional feature reconstruction unit and a morphological constraint unit, while the IoT-side decoding path can include a distribution morphological reconstruction unit and a segmented contribution aggregation unit. These three components are fixed through parameter connections, enabling the same shared state to simultaneously interpret two types of observations.

[0248] To support the above model, data acquisition and calibration were first performed. Several representative operating and disturbance combinations were set under controllable conditions, such as different rotational speeds, loads, and temperatures, as well as different network occupancy and interference levels. Under each combination, rolling shutter camera frame sequences and progressive exposure time metadata, IoT sensor time series, and arrival / forwarding timestamp logs of each intermediate node were simultaneously acquired, and observable operating condition variables and health tags were recorded for anchoring. After acquisition, the data was aligned using a unified time base, time-drifted samples were removed, and training and validation sample sets covering typical operating conditions were retained. The online runtime used the same acquisition and alignment process as the offline runtime to facilitate threshold updates and lightweight calibration.

[0249] In terms of feature construction, the vision side extracts statistical and temporal features reflecting geometric offset perpendicular to the scanning direction from the rolling shutter camera frame sequence. These features include, but are not limited to, inter-line edge displacement curves and their multi-scale summaries, periodic summaries of feature point cross-line displacement trajectories, and time series of texture sharpness and block alignment of candidate corrected images. The IoT side obtains a standardized distributed representation from latency logs, including several key quantiles, the positions and widths of the main and secondary peaks, tail coverage, jitter amplitude and congestion markers within a short time window, and source segment identifiers and proportions. The features from both sides are then input into the model after temporal alignment and dimensional standardization.

[0250] Training and calibration employ a "single-modal pre-training – cross-modal joint training" approach. In the single-modal pre-training stage, the visual-side decoding path and the IoT-side decoding path are trained separately to achieve stable reconstruction capabilities. Subsequently, a shared encoding unit is introduced for joint training, enabling the simultaneous reconstruction of observations from both sides under the same shared state. External operating condition variables or health scores are used as anchoring information to constrain the identifiability of the shared state, preventing instability caused by excessive freedom. During training, the bimodal reconstruction error and cross-modal consistency index are monitored on the validation set. Early stopping conditions are set, and calibration weights corresponding to specific industrial assets or production lines are retained for easy migration and deployment.

[0251] The reverse inference process follows this procedure: The intra-frame anisotropic error features calculated at the current moment are used as input. First, an initial estimate of the shared state is generated via a lightweight regression head. Then, based on this initial estimate, small-step parameter fine-tuning is performed. In each step, visual features are reconstructed through the visual-side decoding path, and the residual is checked for decrease. Simultaneously, physical constraints are imposed on the shared state, such as not exceeding known operating conditions and maintaining smooth transitions with recent historical states, until the residual no longer decreases significantly or reaches the iteration limit within several rounds. The shared potential state at that moment is then output. Forward prediction feeds the shared state into the IoT-side decoding path, generating a standardized latency distribution representation. This representation includes a probability vector with fixed buckets, a list of key quantiles, structural information of the main and secondary peaks, and tail coverage, facilitating comparison with actual observations.

[0252] Consistency assessment is conducted using a "morphological comparison - index summary" approach. Within a sliding time window, the actual time delay distribution is statistically represented to the same specifications as the prediction. The offset of the main peak's position and width, the difference in the appearance and disappearance of secondary peaks, the deviation of each key quantile, the difference in tail coverage, and the overall binning difference are calculated item by item. These differences are summarized using weights calibrated offline to obtain a single consistency score or stratified conclusion. The inconsistency threshold is set based on offline calibration: the distribution range of consistency scores is statistically analyzed on stable operating condition samples, and the high quantile range is selected as the inconsistency threshold. During online operation, slow rolling updates are performed within an alarm-free window to adapt to seasonal or rhythmic changes. A maximum adjustment range is set, and manual review is triggered when the limit is exceeded to avoid threshold drift.

[0253] In an example of early-stage bearing failure in a pump station, offline testing covered different lubrication and load combinations, simultaneously collecting visual distortion and IoT latency data and completing joint training. During online operation, when a distortion pattern similar to bearing loosening appeared on the visual side, the shared state obtained through reverse deduction manifested as higher vibration intensity and specific directionality. This state, through the decoding path on the IoT side, forward predicted a latency pattern with obvious secondary peaks and thickened tails. If, after comparison with the actual latency distribution, differences exceeding the normal range were observed in the main peak shift, key quantiles, and tail coverage, the consistency score exceeded the threshold. Based on this, the system determined that there was rolling shutter vibration distortion or variable transmission latency, and provided constraints and alarm records to the digital twin management process.

[0254] When the shared state estimation is unstable or the training coverage is insufficient, the system provides degradation and fault tolerance strategies: the vision side gives the risk level according to the rule-based distortion criteria, the IoT side outputs the anomaly score independently according to the segmented time delay model, and finally the conclusion is formed by conservative synthesis rules, so as to maintain a prudent and reliable judgment when the model capability is temporarily insufficient.

[0255] Based on the same inventive concept, this application also provides an industrial asset digital twin management platform that integrates vision and IoT, corresponding to the industrial asset digital twin management method that integrates vision and IoT. Since the principle of the platform in this application is similar to the industrial asset digital twin management method that integrates vision and IoT described above, the implementation of the platform can refer to the implementation of the method, and the repeated parts will not be described again.

[0256] See Figure 4 A schematic diagram of an industrial asset digital twin management platform integrating vision and IoT provided in this application embodiment, including:

[0257] The acquisition module 10 is used to acquire visual data streams and IoT data streams of industrial assets; wherein, the visual data stream is a rolling shutter camera frame sequence carrying line-by-line exposure time metadata, and the IoT data stream is a sensor time series carrying transmission timestamp logs;

[0258] The first processing module 20 is used to calculate the geometric deviation perpendicular to the scanning direction as intra-frame anisotropic error from the visual data stream based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata; estimate the delay distribution from the IoT data stream based on the sensor time series and the transmission timestamp log; and fuse the intra-frame anisotropic error and the delay distribution into deviation statistics.

[0259] The judgment module 30 is used to determine whether there is rolling shutter vibration distortion or variable transmission delay based on the deviation statistics.

[0260] The second processing module 40 is used to respond to the presence of at least one of rolling shutter vibration distortion or variable transmission delay, construct a unified time base based on the deviation statistics, and extract the first event time and the second event time from the visual data stream and the IoT data stream respectively under the unified time base, and register the first event time and the second event time to obtain a unified data stream;

[0261] The management module 50 is used to perform digital twin management of the industrial assets based on the unified data flow, and obtain the digital twin management results of the industrial assets.

[0262] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A digital twin management method for industrial assets integrating vision and IoT, characterized in that, include: Acquire visual data streams and IoT data streams of industrial assets; wherein, the visual data stream is a rolling shutter camera frame sequence carrying line-by-line exposure time metadata, and the IoT data stream is a sensor time series carrying transmission timestamp logs; Based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata, the geometric deviation perpendicular to the scanning direction is calculated from the visual data stream as the intra-frame anisotropic error; based on the sensor time series and the transmission timestamp log, the delay distribution is estimated from the IoT data stream; the intra-frame anisotropic error and the delay distribution are fused into deviation statistics; Based on the aforementioned deviation statistics, it is determined whether there is rolling shutter vibration distortion or variable transmission delay; In response to the presence of at least one of rolling shutter vibration distortion or variable transmission delay, a unified time base is constructed based on the deviation statistics, and a first event time and a second event time are extracted from the visual data stream and the IoT data stream respectively under the unified time base. The first event time and the second event time are registered to obtain a unified data stream. Based on the unified data flow, digital twin management is performed on the industrial assets to obtain the digital twin management results of the industrial assets; The step of estimating the latency distribution from the IoT data stream based on the sensor time series and the transmission timestamp log includes: Parse the transmission timestamp log to obtain the arrival and forwarding timestamps of each data point in the sensor time series at multiple intermediate nodes in the data transmission path; Based on the arrival and forwarding timestamps of the multiple intermediate nodes, the end-to-end data transmission path is divided into multiple transmission segments, which include at least a fieldbus transmission segment and a core network transmission segment. For each of the multiple transmission segments, independently construct a probability distribution sub-model of its delay; By fusing multiple probability distribution sub-models, the overall time delay distribution of the multimodal model is obtained.

2. The industrial asset digital twin management method integrating vision and IoT according to claim 1, characterized in that, The step of calculating the geometric deviation perpendicular to the scanning direction as intra-frame anisotropy error from the visual data stream based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata includes: High-frequency motion data generated by an inertial measurement unit rigidly connected to a rolling shutter camera is acquired, and the characteristic vibrations of the industrial asset are parameterized based on the high-frequency motion data to obtain initial vibration parameters. Starting with the initial vibration parameters, an iterative optimization process is performed with the goal of minimizing image entropy, wherein each iteration includes: Based on the vibration parameters of the current iteration and the line-by-line exposure time metadata, a geometric correction function is generated; The geometric correction function is applied to the target frame in the rolling shutter camera frame sequence to obtain a candidate corrected image; Calculate the image entropy of the candidate corrected image; and update the vibration parameters according to the negative gradient direction of the image entropy; When the image entropy converges or reaches the preset number of iterations, the iterative optimization process is terminated, and the geometric deformation corresponding to the finally converged vibration parameter is taken as the geometric deviation.

3. The industrial asset digital twin management method integrating vision and IoT according to claim 2, characterized in that, The updating of the vibration parameters includes: Calculate the first-order gradient vector and second-order Hessian matrix of the image entropy function of the candidate corrected image at the vibration parameters of the current iteration; The second-order Hessian matrix is ​​positive definite to obtain the corrected Hessian matrix. Based on the first-order gradient vector and the corrected Hessian matrix, the update amount of the vibration parameters is determined, and the vibration parameters are updated accordingly.

4. The industrial asset digital twin management method integrating vision and IoT according to claim 3, characterized in that, The positive definite correction of the second-order Hessian matrix to obtain the corrected Hessian matrix includes: Determine whether the second-order Hessian matrix is ​​a positive definite matrix; In response to the fact that the second-order Hessian matrix is ​​a non-positive definite matrix, the following steps are performed: Obtain real-time operating condition data synchronized with the candidate corrected image from the IoT data stream; Based on the real-time operating data and the preset correlation model characterizing the health status of the industrial assets, the expected vibration parameters corresponding to the real-time operating data are determined. Construct a regularization matrix, wherein the penalty strength of the regularization matrix is ​​a function of the difference between the vibration parameters of the current iteration and the expected vibration parameters; The regularization matrix is ​​added to the second-order Hessian matrix to obtain the modified Hessian matrix.

5. The industrial asset digital twin management method integrating vision and IoT according to claim 1, characterized in that, Before dividing the end-to-end data transmission path into multiple transmission segments, the method further includes: extracting delay fingerprints from the links and intermediate nodes in the data transmission path, wherein the extraction of delay fingerprints includes: For a link consisting of any two adjacent intermediate nodes, the online transmission delay is calculated based on the forwarding timestamp when the data point leaves the previous intermediate node and the arrival timestamp when it arrives at the next intermediate node. Statistical features characterizing the online transmission delay are then extracted to form the link fingerprint of the corresponding link. For any intermediate node in the data transmission path, the intra-node processing latency is calculated based on the arrival timestamp of the data point to the intermediate node and the forwarding timestamp of the data point to leave the intermediate node. Statistical features characterizing the intra-node processing latency are then extracted to form the node fingerprint of the intermediate node.

6. The industrial asset digital twin management method integrating vision and IoT according to claim 5, characterized in that, The process of dividing the end-to-end data transmission path into multiple transmission segments includes: Cluster the link fingerprint and the node fingerprint respectively, and classify the links that meet the preset link clustering conditions into the same link category, and classify the intermediate nodes that meet the preset node clustering conditions into the same node category; Based on the link type and the node type, the data transmission path is divided to obtain the multiple transmission segments.

7. The industrial asset digital twin management method integrating vision and IoT according to claim 6, characterized in that, The step of fusing multiple probability distribution sub-models to obtain the overall time delay distribution of the multimodal model includes: Based on multiple probability distribution sub-models, multiple segment combination operation states are identified and defined, wherein each segment combination operation state corresponds to a specific delay mode for each transmission segment. For each of the multiple segmented combined operating states, the conditional probability distribution of the overall time delay corresponding to each state is calculated; Based on real-time observed latency data, estimate the probability that the data transmission path is in each segmented combination operation state at the current moment; The overall time delay distribution of the multimodal mode is constructed by weighting and summing multiple conditional probability distributions according to their corresponding probabilities.

8. The industrial asset digital twin management method integrating vision and IoT according to claim 1, characterized in that, Based on the aforementioned deviation statistics, determining whether rolling shutter vibration distortion or variable transmission delay exists includes: A cross-modal physical model is established, which is used to characterize the mapping relationship from the shared potential physical state parameters to the intra-frame anisotropic error and the time delay distribution; Based on the currently calculated intra-frame anisotropic error, the cross-modal physical model is used for inverse deduction to estimate the corresponding potential physical state parameters; Based on the estimated potential physical state parameters, the cross-modal physical model is used to make a positive prediction and generate the expected time delay distribution. Calculate the statistical distance between the expected latency distribution and the latency distribution actually estimated from the IoT data stream; If the statistical distance is greater than or equal to a preset inconsistency threshold, it is determined that there is rolling shutter vibration distortion or variable transmission delay.

9. A digital twin management platform for industrial assets integrating vision and IoT, used to implement the digital twin management method for industrial assets integrating vision and IoT as described in any one of claims 1-8, characterized in that, include: The acquisition module is used to acquire visual data streams and IoT data streams of industrial assets; wherein, the visual data stream is a rolling shutter camera frame sequence carrying line-by-line exposure time metadata, and the IoT data stream is a sensor time series carrying transmission timestamp logs; The first processing module is used to calculate the geometric deviation perpendicular to the scanning direction as intra-frame anisotropic error from the visual data stream based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata; estimate the delay distribution from the IoT data stream based on the sensor time series and the transmission timestamp log; and fuse the intra-frame anisotropic error and the delay distribution into deviation statistics. The judgment module is used to determine whether there is rolling shutter vibration distortion or variable transmission delay based on the deviation statistics. The second processing module is used to respond to the presence of at least one of rolling shutter vibration distortion or variable transmission delay, construct a unified time base based on the deviation statistics, and extract the first event time and the second event time from the visual data stream and the IoT data stream respectively under the unified time base, and register the first event time and the second event time to obtain a unified data stream; The management module is used to perform digital twin management of the industrial assets based on the unified data flow, and obtain the digital twin management results of the industrial assets.

Images