Industrial asset digital twin management platform and method fusing vision and IoT

Abstract

The invention provides an industrial asset digital twinning management platform and method fusing vision and IoT, and relates to the technical field of digital twinning, and the method comprises the steps: obtaining a rolling shutter camera frame sequence carrying line-by-line exposure time metadata and a sensor time sequence carrying a transmission timestamp log; calculating an intra-frame anisotropy error based on a rolling shutter camera frame sequence, estimating time delay distribution based on the sensor time sequence, and fusing the intra-frame anisotropy error and the time delay distribution into deviation statistics; judging whether rolling shutter vibration distortion or variable transmission time delay exists or not according to the deviation statistics, if so, constructing a unified time base, and registering event time in the visual data stream and the IoT data stream to obtain a unified data stream; and generating a digital twin management result of the industrial assets based on the unified data stream. According to the method, the time distortion in the sensor and the network transmission uncertainty can be effectively compensated, and the precision of multi-modal data alignment and the reliability of industrial asset simulation are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital twinning, in particular to an industrial asset digital twinning management platform and method fusing vision and IoT. BACKGROUND

[0002] Multi-modal perception devices such as visual sensors and IoT sensors are widely deployed in the operation monitoring and management scenarios of industrial assets. By fusing visual data streams and IoT data streams, a digital twin can be constructed in a virtual space, thereby enabling the full life cycle management of industrial assets. However, existing multi-modal data alignment methods still have deficiencies. On the one hand, a rolling shutter camera is prone to intra-frame time offset during line-by-line exposure, resulting in time distortion within visual data; on the other hand, IoT sensors will introduce uncertain delay fluctuations when transmitting data through field buses and core networks. These time distortions and transmission uncertainties not only weaken the accuracy of multi-modal data alignment, but also may cause deviations in the simulation of industrial asset states by digital twin models.

[0003] Therefore, how to compensate for sensor internal time distortion and network transmission uncertainty in heterogeneous data streams to provide a more accurate multi-modal data alignment scheme than existing technologies for improving the reliability of industrial asset simulation has become a technical problem that needs to be solved in the field. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides an industrial asset digital twinning management platform and method fusing vision and IoT.

[0005] In a first aspect, the present application provides an industrial asset digital twinning management method fusing vision and IoT, comprising:

[0006] obtaining a visual data stream and an IoT data stream of an industrial asset; wherein the visual data stream is a rolling shutter camera frame sequence carrying line-by-line exposure time metadata, and the IoT data stream is a sensor time sequence carrying transmission timestamp logs;

[0007] calculating, based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata, a geometric deviation perpendicular to the scanning direction from the visual data stream as an intra-frame anisotropic error; estimating, based on the sensor time sequence and the transmission timestamp logs, a delay distribution from the IoT data stream; and fusing the intra-frame anisotropic error and the delay distribution into a deviation statistic;

[0008] judging, based on the deviation statistic, whether there is rolling shutter vibration distortion or variable transmission delay;

[0009] in response to at least one of the existence of rolling shutter vibration distortion or variable transmission time delay, constructing a unified time base based on the bias statistics, and extracting first event times and second event times from the visual data stream and the IoT data stream respectively under the unified time base, registering the first event times and the second event times to obtain a unified data stream;

[0010] based on the unified data stream, performing digital twin management on the industrial asset to obtain a digital twin management result of the industrial asset.

[0011] Optionally, the calculating, based on the rolling shutter camera frame sequence and the progressive exposure time metadata, a geometric bias perpendicular to a scanning direction from the visual data stream as an in-frame anisotropic error comprises:

[0012] obtaining high-frequency motion data generated by an inertial measurement unit rigidly connected to the rolling shutter camera, and parameterizing a characteristic vibration of the industrial asset based on the high-frequency motion data to obtain initial vibration parameters;

[0013] starting from the initial vibration parameters, performing an iterative optimization process with the objective of minimizing image entropy, wherein each iteration cycle comprises:

[0014] generating a geometric correction function based on the vibration parameters of the current iteration and the progressive exposure time metadata;

[0015] applying the geometric correction function to a target frame in the rolling shutter camera frame sequence to obtain a candidate corrected image;

[0016] calculating an image entropy of the candidate corrected image; and updating the vibration parameters according to a negative gradient direction of the image entropy;

[0017] when the image entropy converges or reaches a preset number of iterations, terminating the iterative optimization process, and taking a geometric deformation variable corresponding to the vibration parameters finally converged as the geometric bias.

[0018] Optionally, the updating the vibration parameters comprises:

[0019] calculating a first-order gradient vector and a second-order Hessian matrix of an image entropy function of the candidate corrected image at the vibration parameters of the current iteration;

[0020] performing positive definite correction on the second-order Hessian matrix to obtain a modified Hessian matrix;

[0021] determining an update amount of the vibration parameters based on the first-order gradient vector and the modified Hessian matrix, and updating the vibration parameters.

[0022] Optionally, the positive definite modification on the second-order Hessian matrix to obtain a modified Hessian matrix comprises:

[0023] determining whether the second-order Hessian matrix is a positive definite matrix;

[0024] in response to the second-order Hessian matrix being a non-positive definite matrix, performing the following steps:

[0025] obtaining real-time working condition data time-synchronized with the candidate corrected image from the IoT data stream;

[0026] determining an expected vibration parameter corresponding to the real-time working condition data based on the real-time working condition data and a preset correlation model representing a health state of the industrial asset;

[0027] constructing a regularization matrix, wherein a penalty strength of the regularization matrix is a function of a difference between a vibration parameter of a current iteration and the expected vibration parameter;

[0028] adding the regularization matrix to the second-order Hessian matrix to obtain the modified Hessian matrix.

[0029] Optionally, the estimating a time delay distribution from the IoT data stream based on the sensor time series and the transmission timestamp log comprises:

[0030] parsing the transmission timestamp log to obtain arrival and forwarding time stamps of each data point in the sensor time series at multiple intermediate nodes in a data transmission path;

[0031] dividing the end-to-end data transmission path into multiple transmission segments based on the arrival and forwarding time stamps of the multiple intermediate nodes, the transmission segments including at least a field bus transmission segment and a core network transmission segment;

[0032] independently constructing a probability distribution sub-model of a time delay for each transmission segment in the multiple transmission segments;

[0033] fusing the multiple probability distribution sub-models to obtain a multi-modal overall time delay distribution.

[0034] Optionally, before dividing the end-to-end data transmission path into multiple transmission segments, further comprising: extracting a time delay fingerprint of a link and an intermediate node in the data transmission path, the time delay fingerprint extraction comprising:

[0035] For a link segment composed of any two adjacent intermediate nodes, an online transmission delay is calculated based on a forwarding timestamp of a data point leaving a previous intermediate node and an arrival timestamp of the data point arriving at a next intermediate node, and a statistical feature representing a characteristic of the online transmission delay is extracted to form a link fingerprint of the corresponding link;

[0036] For any intermediate node in the data transmission path, an intra-node processing delay is calculated based on an arrival timestamp of a data point arriving at the intermediate node and a forwarding timestamp of the data point leaving the intermediate node, and a statistical feature representing a characteristic of the intra-node processing delay is extracted to form a node fingerprint of the intermediate node.

[0037] Optionally, the dividing the end-to-end data transmission path into multiple transmission segments comprises:

[0038] The link fingerprints and the node fingerprints are respectively clustered, links meeting preset link clustering conditions are divided into a same link category, and intermediate nodes meeting preset node clustering conditions are divided into a same node category;

[0039] Based on the link categories and the node categories, the data transmission path is divided to obtain the multiple physically different transmission segments.

[0040] Optionally, the fusing multiple probability distribution sub-models to obtain a multimodal overall delay distribution comprises:

[0041] Based on multiple probability distribution sub-models, multiple segment combination operating states are identified and defined, wherein each segment combination operating state corresponds to a specific delay mode of each transmission segment;

[0042] For each of the multiple segment combination operating states, a conditional probability distribution of the corresponding overall delay under each state is calculated;

[0043] According to the real-time observed delay data, a probability that the data transmission path is in each segment combination operating state at a current moment is estimated;

[0044] Multiple conditional probability distributions are weighted and summed according to their corresponding probabilities to construct the multimodal overall delay distribution.

[0045] Optionally, based on the deviation statistics, it is judged whether there is a rolling shutter vibration distortion or a variable transmission delay.

[0046] A cross-modal physical model is established, which is used to represent a mapping relationship from a shared latent physical state parameter to the in-frame anisotropy error and the delay distribution;

[0047] based on the current calculated intra-frame anisotropy error, using the cross-modal physical model for reverse deduction, estimate the corresponding latent physical state parameters;

[0048] based on the estimated latent physical state parameters, using the cross-modal physical model for forward prediction, generating an expected time delay distribution;

[0049] calculate the statistical distance between the expected time delay distribution and the actual estimated time delay distribution from the IoT data stream;

[0050] in response to the statistical distance being greater than or equal to a preset inconsistency threshold, it is judged that there is a rolling shutter vibration distortion or a variable transmission time delay.

[0051] In a second aspect, the present application provides an industrial asset digital twin management platform integrating vision and IoT, comprising:

[0052] a collection module for acquiring vision data stream and IoT data stream of an industrial asset; wherein the vision data stream is a rolling shutter camera frame sequence carrying line-by-line exposure time metadata, and the IoT data stream is a sensor time sequence carrying transmission timestamp logs;

[0053] a first processing module for calculating geometric deviation perpendicular to the scanning direction as intra-frame anisotropy error from the vision data stream based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata; estimating the time delay distribution from the IoT data stream based on the sensor time sequence and the transmission timestamp logs; and fusing the intra-frame anisotropy error and the time delay distribution into deviation statistics;

[0054] a judgment module for judging whether there is a rolling shutter vibration distortion or a variable transmission time delay based on the deviation statistics;

[0055] a second processing module for, in response to at least one of a rolling shutter vibration distortion or a variable transmission time delay, constructing a unified time base based on the deviation statistics, and extracting first event time and second event time from the vision data stream and the IoT data stream respectively under the unified time base, and registering the first event time and the second event time to obtain a unified data stream;

[0056] a management module for performing digital twin management on the industrial asset based on the unified data stream, and obtaining a digital twin management result of the industrial asset.

[0057] Compared with the prior art, the application can realize high-precision alignment of visual data and IoT data under a unified time base by introducing intra-frame anisotropic error modeling and multi-modal time delay distribution estimation mechanism, breaking through the limitation of traditional methods relying only on single timestamp matching. By fusing vibration compensation based on image entropy optimization and transmission time delay estimation based on cross-section probability modeling, the application has significant advantages in eliminating rolling shutter distortion and network jitter, thereby effectively improving the robustness of heterogeneous data fusion. On this basis, the digital twin can more realistically map the operating state of the industrial asset, ensuring its reliability under complex working conditions. Compared with the prior art, the application has obvious improvements in multi-modal alignment accuracy, data consistency and traceability of asset state simulation in industrial scenarios, and can provide more stable digital support capability for industrial systems. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flowchart of the industrial asset digital twin management method fusing vision and IoT provided by the embodiment of the application is shown in the figure.

[0059] Figure 2 A flowchart of the method for updating vibration parameters provided by the embodiment of the application is shown in the figure.

[0060] Figure 3 A flowchart of the method for obtaining a corrected Hessian matrix provided by the embodiment of the application is shown in the figure.

[0061] Figure 4 A schematic diagram of the industrial asset digital twin management platform fusing vision and IoT provided by the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all embodiments of the application.

[0063] Referring to Figure 1 A flowchart of the industrial asset digital twin management method fusing vision and IoT provided by the embodiment of the application is shown in the figure, and the method comprises steps S101-S105, wherein:

[0064] S101: acquiring a visual data stream and an IoT data stream of an industrial asset; wherein the visual data stream is a rolling shutter camera frame sequence carrying row-by-row exposure time metadata, and the IoT data stream is a sensor time sequence carrying transmission timestamp logs;

[0065] S102: calculating, from the visual data stream, a geometric bias perpendicular to the scanning direction as intra-frame anisotropy error based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata; estimating, from the IoT data stream, a time delay distribution based on the sensor time sequence and the transmission timestamp log; fusing the intra-frame anisotropy error and the time delay distribution into bias statistics;

[0066] S103: determining, based on the bias statistics, whether there is rolling shutter vibration distortion or variable transmission time delay;

[0067] S104: in response to at least one of the rolling shutter vibration distortion or the variable transmission time delay, constructing a unified time base based on the bias statistics, and extracting first event times and second event times from the visual data stream and the IoT data stream respectively under the unified time base, and registering the first event times and the second event times to obtain a unified data stream;

[0068] S105: performing digital twin management on the industrial asset based on the unified data stream to obtain a digital twin management result of the industrial asset.

[0069] For the above S101:

[0070] Specifically, the visual data stream is obtained by a rolling shutter camera installed in an industrial site. The rolling shutter camera attaches the exposure start and end time information for each scanning line during the line-by-line exposure, thereby forming a frame sequence carrying line-by-line exposure time metadata. In order to ensure the continuity and accuracy of the data, the rolling shutter camera is calibrated when it leaves the factory to ensure that its internal clock can run stably. When used in an industrial site, the camera can also access the host computer through a network interface or a bus interface to regularly calibrate the consistency of its internal clock and the system master clock. In this way, the line-by-line exposure time metadata can truly reflect the time position of each row of image acquisition.

[0071] In a specific application, the camera can use a common CMOS rolling shutter imaging device, such as an industrial camera with a resolution of 1920x1080. The camera continuously acquires the target industrial asset at a fixed frame rate, for example, set to 30 frames per second. The camera records the line-by-line exposure time metadata in the form of an additional log or metadata file when outputting each frame of image, and the file is bound to the image data through a time tag, thereby constituting a complete visual data stream. The visual data stream not only includes the image pixel matrix itself, but also includes time information related to each frame and even each scanning line, providing a basic condition for subsequent processing.

[0072] Meanwhile, the IoT data stream is collected by various sensors deployed on the industrial assets. The sensors can include temperature sensors, pressure sensors, vibration sensors, current sensors, or other detection units that can reflect the operating state of the assets. Each sensor records the time of data sampling at the same time of data collection, and generates a timestamp log related to transmission when the data is transmitted through the communication network. The timestamp log usually includes time information of the data arriving at different network nodes or gateways, which describes the propagation of the data in the transmission path.

[0073] In the implementation process, the IoT sensor transmits the collected data to the edge computing node or server through the field bus or wireless transmission protocol. To ensure the accuracy of the timestamp, the sensor end can use a synchronization mechanism, such as the Network Time Protocol (NTP) or GPS-based time signal, to correct the local clock of the sensor. This ensures that the sampling time of the sensor time series has a consistent reference between different sensors.

[0074] The field bus is, for example, CAN bus, Modbus, Profibus, etc.; the wireless transmission protocol is, for example, Wi-Fi, ZigBee, LoRa, etc.

[0075] For example, in a typical scenario, a temperature sensor collects the temperature of the surface of an industrial device at a sampling frequency of 1 per second, and adds a sampling timestamp when generating temperature data; a vibration sensor collects an acceleration signal at a sampling frequency of 1000 per second, and generates a corresponding timestamp sequence. When these data enter the gateway through the field bus, the gateway will add a forwarding timestamp to the header of the data packet to record the arrival time and forwarding time of the data at different communication links. All these information together constitute the IoT data stream.

[0076] Therefore, in the above manner, the system can simultaneously obtain the rolling shutter camera frame sequence carrying the row-by-row exposure time metadata as the visual data stream, and the sensor time sequence carrying the transmission timestamp log as the IoT data stream. Both of the two data streams include original observation values and time-related label information, which provides a data basis for subsequent multi-modal alignment and digital twin modeling.

[0077] For the above S102:

[0078] The method provided in the present application performs step S102 after obtaining the visual data stream and the IoT data stream, and respectively performs error analysis and statistical modeling on the visual data stream and the IoT data stream.

[0079] First, for the rolling shutter camera frame sequence and line-by-line exposure time metadata in the visual data stream, the system can analyze the pixel arrangement relationship in the image along the scanning direction and perpendicular to the scanning direction. Due to the rolling shutter camera, there is a slight difference in the sampling time of different lines during the line-by-line scanning. When the monitored industrial asset is in motion or vibration state, this time difference will be manifested as geometric distortion in the image.

[0080] In order to quantify this distortion, the present embodiment takes the line-by-line exposure time metadata as the time reference, compares the relative position changes of the reference feature points in the target image between different lines, and obtains the offsets perpendicular to the scanning direction. These offsets can reflect the deformation of each line image relative to the ideal static scene, and after statistical processing, they can be used as a representation of the anisotropic error in the frame.

[0081] For example, in the monitoring scene of a machine tool, the edge of the workpiece photographed by the camera should present a regular straight line structure, but due to the rolling shutter effect, the edge line will appear local bending or tilting. Through the line-by-line exposure time metadata, the acquisition time of each line image can be mapped to a unified time axis, and combined with the continuity of the geometric features of the workpiece, the edge position in the actual acquisition image can be compared and analyzed. The offset obtained from this can be used as a geometric deviation to measure the inconsistency between the lines in the frame.

[0082] Second, for the sensor time series and transmission timestamp logs in the IoT data stream, the system can extract the arrival and forwarding time information of each data point at different communication nodes. By comparing the sampling timestamp and the transmission timestamp, the delay of the data at different stages such as collection, transmission, forwarding, etc. can be evaluated, and the overall delay distribution can be formed. This delay distribution can reflect the uncertainty of the IoT data stream in the transmission process, including random jitter, network congestion, and node processing delay, etc.

[0083] In order to improve the stability of the analysis, the present embodiment can use the statistical results of multiple observations when estimating the delay distribution, rather than relying only on a single measurement, to avoid the deviation caused by occasional anomalies.

[0084] For example, in the scenario of monitoring an industrial pump station, the sampling data generated by the pressure sensor usually needs to pass through the field bus to enter the edge gateway, and then be uploaded to the server through the core network. In the transmission path, the delays of different links have differences, some links show fixed processing delay, and others may change due to network load fluctuations. By analyzing the sensor time series and the timestamp records of each node, the end-to-end delay distribution can be obtained. This distribution usually does not conform to a single rule, but shows multi-peak or long-tail characteristics, so it is more reasonable to use statistical modeling to represent it.

[0085] Finally, the intra-frame anisotropic error obtained from the visual data stream and the time delay distribution obtained from the IoT data stream are fused to form a deviation statistic. The deviation statistic can comprehensively describe the time uncertainty from the perception layer and the communication layer, thereby providing a basis for subsequent data alignment and unified processing. In the fusion process, the deviation information from different time domains and different sources can be weighted and statistically analyzed or jointly modeled to ensure that the results can take into account the characteristics of the two types of data.

[0086] Therefore, the embodiment can simultaneously obtain the intra-frame distortion characteristics of the image acquisition link and the time delay characteristics of the sensor data transmission link, and fuse the two into a unified deviation statistical result, thereby providing reliable data support for subsequent digital twin data registration and management.

[0087] For the above S103:

[0088] After obtaining the deviation statistic, it is determined whether there is a rolling shutter vibration distortion or a variable transmission time delay based on the deviation statistic.

[0089] Specifically, the deviation statistic includes intra-frame anisotropic error from the visual data stream and time delay distribution from the IoT data stream. The system can analyze the two types of deviation data to identify whether there is an abnormal state.

[0090] In terms of visual data, the system first compares the ideal geometric structure with the actual collected image features under the same scene. If the intra-frame anisotropic error continuously appears significant deviation in multiple adjacent frames, or the deviation mode corresponds to the vibration characteristics of the device operation, it can be determined that the rolling shutter vibration distortion is more obvious. In order to avoid misjudgment caused by transient noise, the system usually uses a certain number of consecutive frames for statistics.

[0091] For example, when the metal components of the industrial equipment are under high-frequency vibration, the image edges will periodically bend. By observing the periodic deviation in the deviation statistic, the system can determine that the rolling shutter imaging has been affected by vibration.

[0092] In terms of IoT data, the system performs fluctuation analysis on the time delay distribution derived from the transmission timestamp log. If the time delay distribution presents a clear multi-peak shape, or the fluctuation amplitude is large within a short time and is difficult to explain with a fixed delay model, it can be determined that there is a variable transmission time delay.

[0093] For example, in a wireless transmission environment in an industrial field, when the network load suddenly increases or the interference increases, the arrival time of the data packet will appear unstable, resulting in asymmetric or long-tail characteristics of the time delay distribution. At this time, the system can determine that there is uncertainty in the transmission process of the IoT data stream.

[0094] Further, when the deviation statistics of the visual data stream and the IoT data stream have correlation in the time dimension, the system can comprehensively judge the two types of results.

[0095] For example, when the intra-frame anisotropy error increases in the same period, and the time delay distribution of the IoT data also shows large fluctuations, it may indicate that the industrial asset is in a strong interference or unstable operation state.

[0096] In the above manner, the system can accurately identify whether there is rolling shutter vibration distortion or variable transmission time delay without complex external calibration, only relying on the observation data of vision and IoT. This judgment result will directly affect the reliability of the subsequent data alignment strategy and digital twin modeling.

[0097] For the above S104:

[0098] In an optional implementation, when it is judged based on the deviation statistics that at least one of the rolling shutter vibration distortion or the variable transmission time delay exists, the system will perform step S104, that is, constructing a unified time base based on the deviation statistics, and performing event time registration of the visual data stream and the IoT data stream under the unified time base.

[0099] Firstly, the construction of the unified time base means to establish a consistent time reference between the existing multiple independent time sources. The visual data stream relies on the line-by-line exposure time metadata of the camera, while the IoT data stream relies on the local clock of the sensor and the timestamp log in the transmission process. Due to the differences in the acquisition mechanism and the transmission link between the two, simply relying on the original timestamp for pairing often produces deviation.

[0100] This embodiment introduces deviation statistics, and uses the intra-frame anisotropy error from the rolling shutter distortion and the time delay distribution from the IoT transmission process as the basis for time correction to adjust the original time axis, thereby obtaining a unified time base.

[0101] In specific implementation, the system first standardizes the start and end time of line-by-line exposure in the visual data stream, so that the time information of each frame of image can be mapped to the time axis of the unified time base.

[0102] At the same time, the timestamp log of the IoT data stream is corrected according to the statistical characteristics of the time delay distribution, for example, by eliminating the fixed delay component on the network path and compensating the influence of random jitter, so as to map the sensor sampling time to the equivalent sampling time under the unified time base.

[0103] Next, the system extracts event times from both the visual data stream and the IoT data stream under the unified time base. The event time in the visual data stream can be the moment when the target object in the image frame changes significantly, such as the appearance of a sudden change in the edge of a workpiece, the completion of a motion cycle of a device component, etc. The event time in the IoT data stream can be the moment when the sensor detects that the state parameter crosses the threshold, the vibration signal appears a peak, or the current / voltage changes abruptly. These event times are labeled with a unified time base, making them comparable.

[0104] After obtaining the first event time from the visual data stream and the second event time from the IoT data stream, the system performs registration on them. The registration process can be completed by calculating the relative difference between the event times and aligning them. To ensure the stability of the registration, the system can compare multiple event times within a certain time window and select the time point with the highest coincidence degree as the alignment reference. This not only handles the synchronization of a single event, but also maintains the global consistency between the data streams in long-term operation.

[0105] For example, in the scenario of monitoring a rotating device, the visual data stream may record the time point when the rotor passes a certain position, while the IoT vibration sensor detects a vibration peak at the same time. Due to the existence of rolling shutter vibration distortion and network delay, the timestamps of the two may be biased. Through the construction of a unified time base and the registration of event times, the system can align them, so that the image change of the rotor and the peak of the vibration signal accurately correspond on the unified time axis.

[0106] Finally, after the registration of event times, the system generates a unified data stream. This unified data stream integrates the time information of both visual and IoT data, enabling subsequent industrial asset digital twin modeling to be based on a consistent time reference, thereby ensuring the accuracy and reliability of the simulation results.

[0107] For the above S105:

[0108] In an optional implementation, after obtaining the unified data stream, the system performs step S105, i.e., based on the unified data stream, performing digital twin management on the industrial asset and outputting the corresponding management result.

[0109] First, the unified data stream is imported as input into the digital twin modeling engine. This modeling engine can include multiple layers: one is the physical model layer, which is used to establish the basic parameter relationship according to the physical characteristics of the industrial asset, such as the geometric structure, kinematics parameters, energy conversion relationship, etc. of the device; the second is the data-driven layer, which is used to access the observation data from the unified data stream in real time to dynamically update the model state; the third is the business logic layer, which is used to generate management results such as health assessment, abnormal warning or operation and maintenance suggestions according to the running state of the model.

[0110] In the implementation process, the spatial state of the industrial asset can be updated by using the visual information in the unified data stream, such as the geometric deformation of the appearance of the equipment, the position and motion trajectory of the components, and the like. Meanwhile, the IoT data stream provides supplementary information of physical quantities for the model, such as temperature distribution, pressure change, vibration amplitude, current load, and the like.

[0111] By synchronously updating the two types of information on a unified time base, the digital twin can maintain consistency with the actual industrial asset.

[0112] In operation, the system can continuously analyze the unified data stream to reflect the state of the industrial asset in real time.

[0113] For example, when the visual data reflects that the speed of a rotating component fluctuates, and the IoT sensor simultaneously detects that the current load abnormally rises, the digital twin will comprehensively display that the equipment may be in a partial load operation state.

[0114] Further, the management result can be manifested as generating an operation and maintenance log, pushing an abnormal alarm, or triggering a preventive maintenance process.

[0115] In addition, the digital twin management is not limited to abnormal detection, but can also be used for optimization decision.

[0116] For example, in the monitoring scene of a pump station, the unified data stream can reflect the working curve of the pump body under different working conditions, and the system archives and compares these curves in the digital twin environment, thereby helping the operation and maintenance personnel to select a more energy-saving or more stable operation mode. In this way, the digital twin result can not only support short-term operation monitoring, but also serve long-term planning and management.

[0117] For example, in a production workshop, a visual camera monitors the appearance of the equipment, and an IoT sensor collects temperature and vibration signals. After forming the unified data stream through the foregoing steps, the digital twin can present the running state of the key equipment in the workshop in real time, and issue a warning to the operation and maintenance personnel before the temperature exceeds the safety threshold. This management result not only improves the accuracy of monitoring, but also reduces the risk of sudden failure.

[0118] Therefore, the embodiment enables the virtual model of the industrial asset to accurately map the actual running condition in real time and output a management result that can be used for monitoring, diagnosis, optimization and decision making through digital twin management based on the unified data stream, thereby significantly improving the reliability and effectiveness of digital operation and maintenance of the industrial asset.

[0119] Optionally, the calculating, from the visual data stream, a geometric deviation perpendicular to a scanning direction as intra-frame anisotropy error based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata comprises:

[0120] obtaining high-frequency motion data generated by an inertial measurement unit rigidly connected to the rolling shutter camera, and parameterizing a characteristic vibration of the industrial asset based on the high-frequency motion data to obtain initial vibration parameters;

[0121] starting from the initial vibration parameters, performing an iterative optimization process with the goal of minimizing image entropy, wherein each iteration cycle comprises:

[0122] generating a geometric correction function based on vibration parameters of a current iteration and the line-by-line exposure time metadata;

[0123] applying the geometric correction function to a target frame in the rolling shutter camera frame sequence to obtain a candidate corrected image;

[0124] calculating image entropy of the candidate corrected image; and updating the vibration parameters according to a negative gradient direction of the image entropy;

[0125] when the image entropy converges or reaches a preset number of iterations, terminating the iterative optimization process, and taking a geometric deformation variable corresponding to the final converged vibration parameters as the geometric deviation.

[0126] In this way, in order to more accurately calculate the geometric deviation perpendicular to the scanning direction in the rolling shutter camera image, thereby improving the representation ability of the intra-frame anisotropy error, the embodiment can overcome the defect that pure reliance on image pixel analysis is prone to noise interference, and can better capture dynamic distortion caused by device vibration.

[0127] The embodiment first parameterizes the characteristic vibration of the industrial asset using the inertial measurement unit to obtain a reasonable initial vibration estimate, and then gradually corrects through iterative optimization to obtain final parameters that can accurately reflect geometric distortion. In this way, not only the convergence speed of error estimation is improved, but also the robustness under complex working conditions is enhanced.

[0128] In the specific implementation process, the rolling shutter camera is fixedly installed on the same platform as the inertial measurement unit through a rigid structure, so that both can perceive the same motion state. The inertial measurement unit can include an accelerometer and a gyroscope, and the output high-frequency motion data thereof can reflect the slight vibration and attitude change of the industrial asset during imaging. The system pre-processes these high-frequency motion data, such as denoising, integration and feature extraction, to obtain a set of parameters representing the characteristic vibration of the device, such as vibration amplitude, vibration direction and vibration period, etc. These parameters serve as initial vibration parameters, providing a starting point for subsequent image distortion correction.

[0129] After obtaining the initial vibration parameters, the system performs an iterative optimization process targeting at minimizing the image entropy. In each iteration cycle, the system generates a set of geometric correction functions according to the vibration parameters of the current iteration and the line-by-line exposure time metadata. The geometric correction functions are used to describe the positional compensation relationship of pixels in the direction perpendicular to the scanning direction. Then, the system applies the geometric correction functions to the target frame in the rolling shutter camera frame sequence to perform geometric correction on the distorted areas in the image, thereby obtaining a candidate corrected image.

[0130] Subsequently, the system evaluates the information amount of the candidate corrected image, using the image entropy as the evaluation index. The image entropy can reflect the randomness and blurring degree of the image. When the image entropy is high, it means that there are more irregularities and distortions in the image; when the image entropy decreases, it means that the structure of the image is clearer and the distortion is effectively compensated. Therefore, taking the reduction of the image entropy as the optimization target, a better distortion correction result can be gradually converged.

[0131] In each iteration, the system adjusts the vibration parameters according to the change trend of the image entropy of the candidate corrected image. Specifically, when the image entropy is detected to decrease, the system continues to modify the vibration parameters in this direction; if the entropy value no longer decreases or oscillates, it means that the parameters are close to the optimal solution, and the step size needs to be adjusted or convergence needs to be determined. In order to avoid falling into local optimum, the system can also introduce a certain perturbation or random search mechanism in the iteration process to explore a possibly better solution.

[0132] When the image entropy remains stable for a certain number of iterations or the number of iterations reaches a preset threshold, the system terminates the iterative optimization process. At this time, the geometric deformation variable corresponding to the vibration parameters finally converged, i.e., the geometric deviation perpendicular to the scanning direction, is used to represent the in-frame anisotropic error.

[0133] For example, in the scenario of monitoring the machining process of a machine tool, the edge line of the workpiece captured by the camera should be a straight line, but due to the influence of vibration, the edge will appear curved. Through the motion data of the inertial measurement unit, the direction and amplitude of the workpiece vibration can be preliminarily inferred; and through the image entropy iterative optimization, the system further refines this estimate, so that the edge line restores the straight line structure in the corrected image. The final obtained geometric deviation not only quantifies the degree of rolling shutter distortion, but also provides a reliable basis for subsequent time alignment and digital twin modeling.

[0134] In the above manner, the embodiment realizes high-precision estimation of in-frame anisotropic error in visual data stream, and improves the adaptability and robustness of the overall method of the application to industrial asset motion distortion.

[0135] Optionally, referring to Figure 2A flowchart of a method for updating a vibration parameter is provided in the embodiments of the present application, comprising steps S201-S203, wherein:

[0136] S201: calculating a first-order gradient vector and a second-order Hessian matrix of an image entropy function of the candidate corrected image at the vibration parameter of the current iteration;

[0137] S202: performing positive definite correction on the second-order Hessian matrix to obtain a corrected Hessian matrix;

[0138] S203: determining an update amount of the vibration parameter based on the first-order gradient vector and the corrected Hessian matrix, and updating the vibration parameter.

[0139] In order to improve the convergence efficiency and stability of the vibration parameter update in the image entropy iterative optimization process, the present application introduces a gradient and second-order approximation method to determine the correction amount in the parameter update link. This method can ensure the correct optimization direction while avoiding divergence caused by excessive or unreasonable updates, thereby more reliably obtaining the optimal vibration parameter representing the geometric distortion.

[0140] In the specific implementation process, the system first establishes an evaluation function of image entropy for the candidate corrected image. In order to update the vibration parameter, the system needs to analyze the sensitivity of image entropy to the vibration parameter. For this purpose, the system will calculate the change rate of image entropy in different parameter directions near the vibration parameter of the current iteration. The specific method is as follows:

[0141] On the basis of the current vibration parameter, a slight perturbation is applied along each direction, for example, slightly adjusting the vibration amplitude or vibration direction;

[0142] The perturbed vibration parameter is used to regenerate the geometric correction function and act on the target frame to obtain a new candidate corrected image;

[0143] The image entropy values of these candidate corrected images are calculated respectively, and they are compared with the current entropy value;

[0144] According to the entropy value difference, the change trend of the image entropy function at the current vibration parameter is obtained, and then a first-order gradient vector is formed.

[0145] At the same time of obtaining the first-order gradient vector, the system also estimates the second-order change characteristic. Specifically, the system observes the change amplitude of the image entropy under different parameter combinations, and by comparing the entropy value differences before and after multiple perturbation combinations, the second-order curvature information is approximately obtained, so as to construct an initial Hessian matrix. This matrix can describe the bending degree of the image entropy function in the parameter space, and is used to guide the update step and direction.

[0146] After obtaining the initial Hessian matrix, the system checks whether it satisfies the positive definiteness. If it is found that there is a non-positive definite case, that is, the corresponding direction may cause the optimization to be unstable, it is processed by correction in the form of positive definite. The specific method includes: applying a certain adjustment factor on the diagonal elements of the matrix, so that the matrix as a whole becomes positive definite, so as to ensure that the parameter update is always in the direction of the image entropy decrease.

[0147] Next, the system combines the first-order gradient vector with the corrected Hessian matrix to calculate the update amount of the vibration parameter. In other words, the system not only considers the direction of the current entropy value decrease, but also combines the curvature information of the parameter space to determine the update step. This can avoid the oscillation caused by simply relying on the gradient, and also avoid slow convergence caused by too small update.

[0148] In specific application scenarios, for example, when monitoring high-speed rotating machinery, the update of the vibration parameter may involve adjusting the rotation frequency and amplitude. When the system finds that the entropy value decreases rapidly in a certain direction according to the first-order gradient, the curvature information of the Hessian matrix helps to confirm whether the update amplitude needs to be limited to prevent the edge structure from being straightened too much or distorted due to excessive parameter adjustment. Finally, the system will correct the current vibration parameter according to the calculated update amount and enter the next iteration.

[0149] In the above manner, the embodiment realizes efficient update of the vibration parameter in the image entropy optimization process, can significantly improve the convergence speed while ensuring stability, and thus more accurately obtains the geometric deviation of the rolling shutter image.

[0150] Optionally, referring to Figure 3 A flowchart of a method for obtaining a corrected Hessian matrix provided by an embodiment of the present application includes steps S301-S304.

[0151] The present application judges whether the second-order Hessian matrix is a positive definite matrix; in response to the second-order Hessian matrix being a non-positive definite matrix, the following steps are performed:

[0152] S301: Obtain real-time working condition data time-synchronized with the candidate corrected image from the IoT data stream;

[0153] S302: Determine an expected vibration parameter corresponding to the real-time working condition data based on the real-time working condition data and a preset correlation model representing the health state of the industrial asset;

[0154] S303: Construct a regularization matrix, wherein the penalty strength of the regularization matrix is a function of the difference between the vibration parameter of the current iteration and the expected vibration parameter;

[0155] S304: adding the regularization matrix to the second-order Hessian matrix to obtain the modified Hessian matrix.

[0156] The present application proposes a technical solution of positive definite correction combined with IoT real-time working condition data for the problem of non-positive definite Hessian matrix that may occur in the iterative optimization process.

[0157] The technical problem to be solved by the scheme is how to avoid distortion or divergence of the optimization direction during parameter updating, so that the correction process of the vibration parameter can remain stable and reliable under complex working conditions.

[0158] When the traditional optimization process based on image entropy only relies on visual information, the calculated second-order Hessian matrix may be non-positive definite in some dimensions, resulting in uncertainty of the parameter update direction, and even causing the iterative process to deviate from the optimal solution. Therefore, the present embodiment introduces IoT working condition data synchronized with the candidate correction image, and uses an associated model representing the health state of the industrial asset for assistance, thereby introducing external physical constraints into the optimization process to achieve matrix positive definite.

[0159] In the specific implementation process, the system first checks the characteristics of the second-order Hessian matrix in each iteration. If the detection result shows that the matrix has negative features in some directions, it is determined to be a non-positive definite matrix. At this time, the system starts the positive definite correction process.

[0160] The first step of the correction process is to synchronize the IoT working condition data. The system extracts data points consistent with the timestamp of the candidate correction image from the IoT data stream, such as temperature, pressure, vibration amplitude, or current load at the corresponding time. In this way, it can be ensured that the visual data and IoT data reflect the same physical state.

[0161] Subsequently, the system uses a pre-set industrial asset health state associated model to interpret these working condition data. The model can be a hybrid model constructed by physical mechanism and empirical statistics, or can introduce artificial intelligence algorithms to some extent. For example, the input of the model is the real-time collected temperature, vibration, speed and other multi-source data, and the output is the predicted value of the expected vibration parameter. The predicted value reflects the reasonable vibration pattern that the industrial asset should present under the current working condition, thereby providing a reference for subsequent regularization.

[0162] Next, the system will construct a regularization matrix. The role of this matrix is to determine a penalty strength by comparing the current iteration vibration parameters with the expected vibration parameters predicted by the model. The greater the penalty strength, the greater the gap between the current parameters of the system and the reasonable working conditions, and the stronger the correction constraint is needed. In this way, the regularization matrix is equivalent to adding a stability factor to the original Hessian matrix in mathematics, avoiding the optimization process from deviating from the actual physical state of the device.

[0163] After the construction of the regularization matrix is completed, the system adds it to the original second-order Hessian matrix to obtain the modified Hessian matrix. Since the design of the regularization matrix can ensure that it is positive definite, the result after addition is a positive definite matrix as a whole, thereby ensuring that the parameter update direction is always towards the region where the image entropy decreases, and is consistent with the real working condition of the device.

[0164] For example, when monitoring a high-speed rotating device, if the calculated Hessian matrix is non-positive definite due to transient noise during the image entropy optimization process, the system will immediately call the vibration acceleration and speed data collected by the IoT sensor at the same time. The correlation model determines that the reasonable vibration mode of the device at this speed should be within a certain range. If the current iteration parameters deviate significantly from this range, the regularization matrix will enhance the correction strength in this direction, so that the updated Hessian matrix returns to a stable positive definite state.

[0165] The correlation model can collect corresponding data of geometric deviation and IoT time delay distribution in the visual frame under different working conditions of the device, and use the data as input-output pairs to train and fit through multivariate linear regression, support vector regression or three-layer feedforward neural network, thereby obtaining a model that can reflect the relationship between working condition parameters and deviations. The input of the model can be speed, load, temperature and other working condition variables, and the output is the geometric distortion parameter and the time delay eigenvalue.

[0166] For example, the correlation model can be a rule-based system, for example, if the temperature is greater than 80 degrees Celsius and the current is greater than 5 amperes, the expected vibration frequency is within the range of 100-120 Hz.

[0167] In the above manner, the present embodiment not only prevents the optimization process from failing due to non-positive definite matrix, but also combines IoT working condition information and visual data organically, so that the optimization result is more consistent with the actual running state of the industrial asset. This cross-modal constraint mechanism significantly improves the robustness and reliability of the method of the present application under complex working conditions.

[0168] Optionally, estimating the time delay distribution from the IoT data stream based on the sensor time series and the transmission timestamp log comprises:

[0169] parsing the transmission timestamp log to obtain arrival and forwarding timestamps of each data point in the sensor time series at multiple intermediate nodes in a data transmission path;

[0170] based on the arrival and forwarding timestamps of the multiple intermediate nodes, dividing the end-to-end data transmission path into multiple transmission segments with different physical meanings, the transmission segments at least including a fieldbus transmission segment and a core network transmission segment;

[0171] for each transmission segment in the multiple transmission segments, independently constructing a probability distribution sub-model of its latency;

[0172] fusing the multiple probability distribution sub-models to obtain a multi-modal overall latency distribution.

[0173] The technical solution proposed in the present application aims to solve the uncertainty of multi-link and multi-type delay in the end-to-end transmission process of IoT sensor data. Through joint analysis of the sensor time series and the transmission timestamp log, a latency distribution model that is more consistent with the actual working condition can be obtained.

[0174] In this way, the overall transmission process is divided into multiple transmission segments with clear physical meanings, each segment is independently modeled, and the segment models are fused into an overall distribution, thereby avoiding the over-simplification of a single model to the complex network delay characteristics. This can significantly improve the accuracy and adaptability of latency estimation and provide a reliable basis for subsequent data alignment.

[0175] In the specific implementation process, the system first parses the transmission timestamp log. This log is automatically generated when IoT data is transmitted between different nodes and usually includes the time when the data packet arrives at the node and the time when the node completes forwarding. The system will correspond the log with the sensor time series piece by piece to obtain the complete transmission trajectory of each data point at multiple intermediate nodes.

[0176] For example, when a certain temperature sampling point is sent from the sensor and then passes through the fieldbus controller, the edge gateway, and the core switch, and finally reaches the server, the system can extract its arrival time and forwarding time at each intermediate node mentioned above.

[0177] Subsequently, the system divides the overall data transmission path into multiple segments based on these arrival and forwarding times. Each segment corresponds to a transmission link with clear physical properties. For example, the part from the sensor to the fieldbus controller is the "fieldbus transmission segment", which is characterized by small delay but is significantly affected by bus occupancy; the part from the edge gateway to the core switch is the "core network transmission segment", which is characterized by large delay fluctuations and is affected by network traffic and routing strategy. Through such division, more targeted modeling methods can be used for different characteristics of different segments.

[0178] After the division, the system independently constructs a delay probability distribution sub-model for each transmission segment. The construction process includes: statistics on multiple data point delay samples of the segment, extracting common characteristic values such as mean, variance, and extreme delay frequency, and forming a probability distribution sub-model according to these characteristics. The probability distribution sub-model is not limited to a fixed type, and can be described in a suitable way according to the characteristics of the segment, for example, using an approximate normal distribution to represent a stable segment, or using a long-tailed distribution to represent a segment that is more affected by network congestion. In this way, the actual delay law of the segment can be maximally close.

[0179] After the probability distribution sub-models of the segments are established, the system fuses these probability distribution sub-models to obtain the overall delay distribution of the end-to-end. The fusion process can be understood as superimposing and combining the delay characteristics of each segment according to the order of data in the transmission path. Since different segments may have different delay modes at the same time, the overall delay distribution formed finally often presents a multi-peak or multi-modal characteristic.

[0180] For example, in the pump station monitoring scene, the field bus transmission part has stable delay, and the core network part has severe delay fluctuation. The overall delay distribution after fusion of the two parts is characterized by a main peak and a secondary peak.

[0181] In this way, the embodiment not only can identify and quantify the contribution of each segment to the end-to-end delay, but also can generate an overall delay distribution that can truly reflect the characteristics of a complex network. Avoiding the over-simplification of complex delay characteristics by using a single distribution model in the traditional method, the embodiment makes the delay estimation more consistent with the actual industrial communication environment, and helps to improve the reliability of subsequent data alignment and digital twin modeling.

[0182] Optionally, before the end-to-end data transmission path is divided into multiple transmission segments with different physical meanings, the method further includes: extracting a delay fingerprint of a link and an intermediate node in the data transmission path, and the extraction of the delay fingerprint includes:

[0183] For a link formed by any two adjacent intermediate nodes, based on the forwarding timestamp of a data point leaving a previous intermediate node and the arrival timestamp of the data point arriving at a next intermediate node, the online transmission delay of the link is calculated, and statistical characteristics representing the online transmission delay characteristics of the link are extracted to form a link fingerprint of the link;

[0184] For any intermediate node in the data transmission path, based on the arrival timestamp of a data point arriving at the intermediate node and the forwarding timestamp of the data point leaving the intermediate node, the intra-node processing delay of the intermediate node is calculated, and statistical characteristics representing the intra-node processing delay characteristics of the intermediate node are extracted to form a node fingerprint of the intermediate node.

[0185] In order to more accurately distinguish the performance differences of different links and intermediate nodes before dividing the end-to-end data transmission path into multiple physically different transmission segments, the application adopts a delay fingerprint extraction mechanism. The technical problem solved by the mechanism is how to accurately characterize the delay characteristics of each link and each node through fine-grained feature extraction without relying on a single end-to-end measurement in a complex industrial communication network.

[0186] In this way, more reliable data support can be provided for subsequent segment modeling.

[0187] The principle is that the end-to-end delay is composed of multiple link transmissions and node processing links, and different links may have completely different delay laws. If the delay characteristics of each link and node can be extracted and form independent "delay fingerprints", more detailed classification and differentiation can be achieved before the overall segmentation. This approach avoids the problem of lumping all uncertainties together, thereby improving the accuracy of delay modeling.

[0188] In the specific implementation process, the system first extracts the fingerprint for the link-level delay. Taking any two adjacent intermediate nodes as an example, the system obtains the forwarding timestamp when the data point leaves the previous intermediate node and the arrival timestamp when it arrives at the next intermediate node, and the difference between the two represents the online transmission delay of the link. After collecting a sufficient number of data points, the system performs statistical analysis on the delay of the link to extract features such as average transmission delay, delay fluctuation range, and delay distribution pattern. These statistical features constitute the "link fingerprint" of the link. For example, in a pump station monitoring system, the delay fingerprint of the fieldbus link usually exhibits low mean and small fluctuation, while the delay fingerprint of the wireless link may exhibit long tail characteristics and large fluctuation.

[0189] For example, the link fingerprint can be a four-dimensional vector containing [average delay, delay variance, 95th percentile delay, packet loss rate], which is calculated in a 60-second sliding window.

[0190] Subsequently, the system extracts the fingerprint for the node-level delay. For any intermediate node, the system obtains the arrival timestamp when the data point arrives at the intermediate node and the forwarding timestamp when the intermediate node completes processing and leaves. The difference between the two is the processing delay of the intermediate node. By statistically analyzing the processing delays of multiple data points, features such as average processing time, jitter amplitude, and processing queue accumulation can be extracted to form the "node fingerprint" of the intermediate node.

[0191] For example, in an industrial gateway device, due to the need to perform protocol parsing and data encryption, its node fingerprint usually presents a large fixed delay component, while in a switch node, the fingerprint feature more reflects the instantaneous congestion and cache queuing situation.

[0192] In a typical application scenario, when pressure sensor data successively passes through a field bus, an edge gateway, a core switch and finally reaches a server, the system can generate independent time delay fingerprints for each link and each node. By comparing the fingerprint features of different links and nodes, it can be found that, for example, the field bus delay is stable but the fixed value is high, while the wireless link delay is unstable and the distribution is scattered; for example, the edge gateway node processing delay is constant, while the core switch delay varies with network traffic. These differentiated fingerprints can provide clear basis for subsequent division of transmission segments with different physical meanings.

[0193] In the foregoing manner, the embodiment achieves fine-grained characterization of link and node delay characteristics. Its beneficial effects are that it can accurately distinguish the performance differences of different transmission links in a complex network environment, so that subsequent segment modeling and overall time delay estimation are more in line with actual conditions, and the accuracy and robustness of the industrial asset digital twin management method at the communication level are improved.

[0194] Optionally, the dividing the end-to-end data transmission path into multiple transmission segments with different physical meanings comprises:

[0195] The link fingerprints and the node fingerprints are respectively clustered, links meeting preset link clustering conditions are divided into the same link category, and intermediate nodes meeting preset node clustering conditions are divided into the same node category;

[0196] Based on the link categories and the node categories, the data transmission path is divided to obtain multiple transmission segments with different physical meanings.

[0197] In order to avoid the high model complexity caused by simply regarding all links and nodes as independent units in time delay modeling, the application adopts transmission path division based on time delay fingerprint clustering.

[0198] The technical problem to be solved by the optional embodiment is how to reasonably classify similar links in a complex industrial network environment by clustering analysis of the delay characteristics of links and nodes, so as to obtain transmission segments with physical meanings. The principle is that although different links and nodes have different physical locations, if their time delay fingerprints are highly similar in statistical characteristics, they can be regarded as the same category, and then the transmission path is divided into stable and interpretable segments. This can improve the accuracy of the time delay model while reducing the complexity.

[0199] In the implementation process, the system will first cluster the link fingerprints. Each link fingerprint includes statistical characteristics such as average transmission delay, fluctuation range, distribution form, etc. The system will set a group of preset link clustering conditions, for example: the average transmission delay differs by no more than a certain allowable range, the fluctuation amplitude is in a similar level, the distribution form is highly consistent, etc. Links that meet the above conditions will be classified into the same category. In this way, even if the links belong to different sensors in the physical sense, if they show similar delay patterns, they can be considered as the same type of link.

[0200] Similarly, the system will cluster the node fingerprints. Node fingerprints mainly describe the statistical characteristics of the processing delay in the node, such as average processing delay, queue accumulation characteristics, fluctuation size, etc. The system will set preset node clustering conditions, for example: the average processing time is close, the delay jitter amplitude is consistent, whether there is a significant long-tail distribution, etc. Nodes that meet these conditions will be classified into the same node category. In this way, for example, multiple field gateways may show consistent delay characteristics because they use the same hardware and firmware version, and can be clustered into the same node category.

[0201] It can be understood that the above conditions can be set through statistical analysis of the initial collected network logs, and can be dynamically adjusted during operation.

[0202] After clustering the links and nodes, the system will divide the transmission path based on the categories. Specifically, the system will classify the links belonging to the same link category into one transmission segment, and the nodes belonging to the same node category into another transmission segment. Finally, the end-to-end data transmission path is divided into multiple physically meaningful segments, and each transmission segment represents a type of link or node with similar delay characteristics.

[0203] For example, in a certain smart factory scenario, sensor data passes through field bus links, multiple gateway nodes, core switch links, and cloud platform entry nodes in turn. After fingerprint extraction, the system may find that the fingerprint characteristics of the field bus links and the local industrial Ethernet links are highly similar, both showing stable low delay, so they can be clustered into the same link category to form a "field transmission segment".

[0204] For example, link fingerprints are clustered using the K-means algorithm, with cluster k = 3 and Euclidean distance as the distance metric, thereby dividing the links into three categories: low delay, medium delay, and high delay.

[0205] Meanwhile, if the processing delay fingerprints of multiple gateway nodes also show similarity, all have fixed protocol parsing delay, they will also be divided into the same node category, forming a "gateway processing segment". In this way, the overall path is divided into "field transmission segment-gateway processing segment-core network transmission segment-cloud platform access segment" and several other parts with clear physical meaning.

[0206] In the above manner, the embodiment realizes path division based on delay fingerprint, which not only avoids the problems of over-simplification or over-subdivision in traditional methods, but also makes each segment correspond to clear physical meaning, and can significantly improve the explainability and reliability of subsequent delay distribution modeling.

[0207] Optionally, the step of fusing the plurality of probability distribution sub-models to obtain the multimodal overall delay distribution comprises:

[0208] Based on the plurality of probability distribution sub-models, a plurality of segmented combined operating states are identified and defined, wherein each segmented combined operating state corresponds to a specific delay mode of each transmission segment;

[0209] For each of the plurality of segmented combined operating states, a conditional probability distribution of the overall delay under each state is calculated;

[0210] According to the real-time observed delay data, the probability of the data transmission path being in each segmented combined operating state at the current time is estimated;

[0211] The plurality of conditional probability distributions are weighted and summed according to their corresponding probabilities to construct the multimodal overall delay distribution.

[0212] In order to avoid the over-simplification of a single distribution model to complex delay rules, so as to more accurately describe the delay mode of the industrial communication network under different operating states. The optional embodiment defines a plurality of segmented combined operating states, calculates the conditional distribution under each state, and then combines the real-time observed delay data to give probability weights to different states, and finally forms a multimodal overall delay distribution.

[0213] In the specific implementation process, first, the probability distribution sub-models of each transmission segment established in the previous steps are used to analyze the typical modes that may be presented one by one.

[0214] For example, there can be two modes of stable low delay and light congestion for fieldbus segments, and three modes of regular delay, peak delay and long tail delay for core network segments. The system identifies and defines multiple segment combination operating states by enumerating and combining these modes. Each combination operating state represents the possible delay operating condition of the entire end-to-end path at a certain moment, such as "fieldbus stable low delay + core network peak delay" or "fieldbus light congestion + core network regular delay".

[0215] After determining the combination operating states, the system superimposes the probability distribution sub-models of the corresponding segments for each combination operating state to calculate the overall delay condition distribution under the state.

[0216] In the calculation, the delay samples of each segment are jointly calculated to obtain the distribution of the end-to-end delay under the combination. In this way, the condition distribution under different combination states can clearly reflect the possible interval and probability characteristics of the delay.

[0217] Next, real-time observed delay data are introduced to dynamically estimate the operating state of the current system. For example, in a sliding time window, the latest delay samples are collected and compared with the condition distributions under each combination operating state to calculate the degree of fit between the samples and each condition distribution. The state with a high degree of fit is assigned a higher probability, and the state with a low degree of fit corresponds to a lower probability.

[0218] For example, when the real-time data show obvious long tail delay characteristics, the system increases the weight of the combination state related to "core network long tail delay".

[0219] After estimating the probability of each state, the system weights and sums all the condition probability distributions according to the corresponding probabilities to obtain the final multimodal overall delay distribution. The overall delay distribution is no longer limited to a single curve, but is composed of multiple peaks and intervals, which can reflect the delay laws under both regular operation and abnormal state.

[0220] For example, in the wireless monitoring scene of a smart factory, the final multimodal delay distribution can include a main peak (regular delay state) and several secondary peaks (abnormal states caused by congestion or interference).

[0221] In this way, the embodiment realizes multimodal overall delay modeling based on segment probability distribution sub-models. It can take into account the influence of regular and abnormal states on delay, avoiding the one-sidedness of traditional single distribution models, so that the industrial asset digital twin management method is more accurate and robust when facing complex network environments.

[0222] Optionally, based on the deviation statistics, determining whether there is a rolling shutter vibration distortion or a variable transmission delay comprises:

[0223] a cross-modal physical model is established, which is used to characterize the mapping relationship from shared latent physical state parameters to the intra-frame anisotropy error and the latency distribution;

[0224] Based on the currently calculated intra-frame anisotropy error, the cross-modal physical model is used for reverse deduction to estimate the corresponding latent physical state parameters;

[0225] Based on the estimated latent physical state parameters, the cross-modal physical model is used for forward prediction to generate an expected latency distribution;

[0226] Calculate the statistical distance between the expected latency distribution and the actual estimated latency distribution from the IoT data stream;

[0227] In response to the statistical distance being greater than or equal to a preset inconsistency threshold, it is determined that there is a rolling shutter vibration distortion or a variable transmission latency.

[0228] Further, in order to more accurately determine whether there is a rolling shutter vibration distortion or a variable transmission latency, the application adopts a determination mechanism based on a cross-modal physical model.

[0229] The technical problem to be solved by the mechanism is: how to unify the intra-frame anisotropy error shown in the visual data and the latency distribution shown in the IoT data into the same physical state interpretation framework, so that the consistency test can be used to identify potential abnormalities.

[0230] By establishing the mapping relationship between the shared latent physical state parameters and the two types of deviations, the application can establish a deducible correspondence between different modal data, and then identify abnormal working conditions by comparing the difference between the expected distribution and the actual observation distribution.

[0231] In the specific implementation process, first, a cross-modal physical model needs to be established. This model takes a set of latent physical state parameters as the core, such as the vibration intensity, frequency, and directionality of device operation, as well as the load level and stability of network links. These parameters will be reflected in the visual modality, such as causing intra-frame uneven geometric offset, and will also be reflected in the IoT modality, such as causing fluctuations in network latency distribution.

[0232] Therefore, this model establishes the mapping relationship from latent physical state parameters to visual error and latency distribution, so that visual and IoT observation data are no longer isolated, but are driven by the same physical mechanism.

[0233] In the execution process, the system will use the cross-modal physical model to perform reverse deduction based on the currently calculated intra-frame anisotropy error.

[0234] For example, the cross-modal physical model consists of a vision-side submodel and an IoT-side submodel, which are related to each other through shared latent physical state parameters.

[0235] The vision-side submodel takes the anisotropic error within a rolling shutter image as input, and can extract the main mode of the error through principal component analysis, or extract distortion features using a convolutional neural network, and then output the corresponding latent vibration parameters such as vibration amplitude, frequency and direction through a regression module.

[0236] The IoT-side submodel takes the latent physical state parameters as input, and calculates the corresponding delay distribution pattern through a probabilistic graphical model or a queuing theory model, such as a unimodal distribution under normal conditions, a bimodal distribution under mild congestion, or a long-tail distribution under severe interference.

[0237] In reverse deduction, the system compares the output of the vision submodel with the actual calculated intra-frame deviation, and obtains the most consistent latent physical state parameters through iterative search; in forward prediction, the system inputs the latent physical state parameters into the IoT-side submodel to obtain the expected delay distribution. By comparing the difference between the expected delay distribution and the actual IoT observation distribution, it can be determined whether there is inconsistency. This model structure clearly shows the input, output and intermediate association, and those skilled in the art can implement consistency modeling of cross-modal data based on it.

[0238] For example, the steps of reverse deduction include: inputting the actual calculated intra-frame geometric deviation into the model, and finding the latent physical state parameters that can explain these deviations through fitting and iterative methods.

[0239] For example, when periodic bending is detected in a rolling shutter image, the system can deduce the vibration frequency and amplitude range of the industrial equipment, and obtain a set of candidate latent physical state parameters accordingly.

[0240] Next, the system will use the latent physical state parameters obtained by reverse deduction to perform forward prediction with the help of the cross-modal physical model. The goal of forward prediction is to generate an expected IoT delay distribution under the latent physical state parameters.

[0241] For example, if the deduced parameters indicate that the equipment is in a high-frequency vibration state, the model may predict that the corresponding IoT transmission path will exhibit high delay jitter or obvious multi-modal distribution. In this way, the system can start from the vision modality and deduce the IoT modality performance that is physically consistent.

[0242] After generating the expected latency distribution, the system compares it with the estimated latency distribution in the actual IoT data stream. Specifically, the system statistically measures the difference between the two, including whether the overall shape is consistent, whether the peak position is shifted, whether the distribution tail is lengthened, etc. If the difference is small, it means that the observations of the visual modality and the IoT modality confirm each other, and the current system is in a consistent physical state; if the difference is large, it means that at least one modality's data is abnormal.

[0243] To realize automatic judgment, the system sets an inconsistency threshold. When the calculated difference measure is greater than or equal to the inconsistency threshold, it is determined that there is a rolling shutter vibration distortion or a variable transmission latency. The inconsistency threshold can be obtained through experimental calibration, or it can be dynamically adjusted during long-term operation of the system to adapt to different noise levels and working condition characteristics of different industrial sites.

[0244] For example, in an application scenario of an automated production workshop, the camera observes that the edge of the material on the conveyor belt presents periodic jitter, while the IoT sensor simultaneously feeds back a latency distribution that exhibits unexpected bimodal characteristics. If there is a significant difference between the expected distribution derived from the cross-modal physical model and the actual IoT observation, such as a unimodal expected distribution but a bimodal actual distribution, the system determines that there is a variable transmission latency. This determination result can be used for subsequent abnormal alarm and dynamic correction of the digital twin model.

[0245] It can be understood that the inconsistency threshold can be determined by statistical calibration, for example, when the device is in a stable working condition, the IoT latency distribution in a time window is collected, the mean and standard deviation are calculated, and the mean plus or minus several times the standard deviation is taken as the threshold interval. When the statistical distance between the cross-modal physical model prediction distribution and the actual distribution exceeds the threshold interval range, it is determined to be abnormal. The threshold can also be corrected according to historical data during long-term operation.

[0246] In the above manner, the embodiment not only can judge the abnormality within a single modality, but also can enhance the reliability of the judgment by using cross-modal consistency. Its beneficial effects are that by introducing the cross-modal physical model, the visual and IoT two types of heterogeneous data can be associated, and the observations can be explained under the framework of sharing physical state parameters, so that the rolling shutter vibration distortion and the variable transmission latency can be more accurately and robustly identified, and the credibility and robustness of digital twin management are improved.

[0247] In an alternative embodiment, to enable those skilled in the art to construct and operate a cross-modal physical model for anomaly determination therefrom, the following executable implementation path is provided. The model is structurally composed of a shared encoding unit and two decoding paths, both of which are linked around the industrial asset physical state at the same time: the shared encoding unit characterizes the physical state; the visual side decoding path receives the physical state and reconstructs the distortion characterization consistent with the intra-frame anisotropic error; the IoT side decoding path receives the same physical state and reconstructs the distribution characterization consistent with the end-to-end latency statistics. The shared encoding unit can adopt a serial structure of a light feedforward network and a temporal feature unit, the visual side decoding path can include a convolutional feature reconstruction unit and a morphological constraint unit, and the IoT side decoding path can include a distribution morphological reconstruction unit and a segmented contribution aggregation unit, all of which are fixed through parameter connection relationship, so that the same shared state can explain two types of observations at the same time.

[0248] To support the above-mentioned model, first, data acquisition and calibration are performed. Under controllable working conditions, set several representative combinations of operation and disturbance, such as different rotating speeds, loads and temperatures, and different network occupancy and interference levels; under each combination, simultaneously collect the rolling shutter camera frame sequence and the progressive exposure time metadata, the IoT sensor time series and the arrival / forwarding time stamp log of each intermediate node, and record the observable working condition variables and health labels for anchoring. After the acquisition is completed, the above-mentioned data is aligned with a unified time base, time drift samples are removed, and training and validation sample sets covering typical working conditions are retained; the same acquisition and alignment process as offline is used online to update the threshold and perform light calibration.

[0249] In terms of feature construction, the visual side extracts statistical and temporal features from the rolling shutter camera frame sequence that can reflect the geometric offset perpendicular to the scanning direction, including but not limited to: inter-row edge displacement curve and its multi-scale summary, periodic summary of feature point cross-row displacement trajectory, time series of texture clarity and block alignment of candidate corrected images, etc. The IoT side obtains a standardized distribution representation from the latency log, including several key quantile points, main peak and secondary peak positions and widths, tail coverage, jitter amplitude and congestion markers in a short time window, and source segment identification and proportion. The features on both sides are input into the model after time alignment and dimension standardization.

[0250] The training and calibration adopt the steps of "single-modal pre-training-cross-modal joint training". The single-modal pre-training stage respectively trains the visual side decoding path and the IoT side decoding path, so that they have stable reconstruction capability; then the joint training is performed by introducing a shared encoding unit, so that the same shared state can simultaneously reconstruct the observations on both sides, and the external working condition variables or health scores are used as anchor information to constrain the distinguishability of the shared state, avoiding instability caused by excessive degrees of freedom. The training process monitors the bimodal reconstruction error and cross-modal consistency index on the validation set, sets early stopping conditions and retains the calibration weights corresponding to specific industrial assets or production lines, facilitating migration and deployment.

[0251] The reverse deduction is performed according to the following process: the intra-frame anisotropy error features calculated at the current time are taken as input, and an initial estimate of the shared state is first generated via a lightweight regression head; then small-step parameter fine-tuning is performed starting from the initial estimate, and in each step the visual features are reconstructed by the visual side decoding path and the residual error is checked for reduction, while the shared state is subjected to physical rationality constraints such as not exceeding the known working condition range and maintaining smooth connection with the latest historical state, until the residual error no longer significantly decreases or the iteration upper limit is reached, and the shared latent state at this time is output. The forward prediction sends the above shared state into the IoT side decoding path to generate a standardized time delay distribution representation, which specifically includes a fixed-bucket probability vector, a list of key quantile points, structure information of the main peak and the secondary peak, and tail coverage, etc., facilitating comparison with actual observations.

[0252] The consistency evaluation is performed in the manner of "morphological comparison-index summary". The actual time delay distribution is standardized and represented in the same specification as the prediction within a sliding time window, and the differences in main peak position and width, the appearance and disappearance of secondary peak, the deviation of each key quantile point, the difference in tail coverage, and the overall bucket difference are calculated item by item; the above differences are summarized by weights calibrated offline to obtain a single consistency score or layered conclusions. The setting of the inconsistency threshold comes from offline calibration: the distribution interval of the consistency score is calculated on stable working condition samples, and the high quantile interval is selected as the inconsistency threshold; during online operation, slow rolling update is performed within the no-alarm window to adapt to seasonal or beat changes, while the maximum adjustment amplitude is set and manual review is triggered when the threshold is exceeded to avoid threshold drift.

[0253] In the example of early bearing failure in a pump station, offline tests cover different lubrication and load combinations, with simultaneous collection of visual distortion and IoT latency data and joint training. During online operation, when a distortion pattern similar to bearing looseness appears on the visual side, the shared state derived by reverse deduction is reflected in higher vibration intensity and specific directionality; this state is decoded by the IoT side to predict a latency shape with a clear secondary peak and thickened tail. If the differences in peak shift, key quantile, and tail coverage all exceed the normal interval when compared with the actual latency distribution, the consistency score exceeds the threshold, and the system determines that there is rolling shutter vibration distortion or variable transmission latency, and provides constraint conditions and alarm records to the digital twin management process.

[0254] When the shared state estimation is unstable or the training coverage is insufficient, the system provides degradation and fault-tolerant strategies: the visual side gives a risk level according to the regularization distortion criterion, and the IoT side independently outputs an abnormal score according to the segmented latency model, and finally forms a conclusion with a conservative synthesis rule, so that cautious and reliable judgments can be maintained when the model capacity is temporarily insufficient.

[0255] Based on the same inventive concept, the embodiments of the present application also provide a fusion visual and IoT industrial asset digital twin management platform corresponding to the fusion visual and IoT industrial asset digital twin management method. Since the principles of the platform solving problems in the embodiments of the present application are similar to the fusion visual and IoT industrial asset digital twin management method described above, the implementation of the platform can be referred to the implementation of the method, and the repeated parts will not be described again.

[0256] Referring to Figure 4 The fusion visual and IoT industrial asset digital twin management platform provided in the embodiments of the present application includes:

[0257] The acquisition module 10 is configured to obtain a visual data stream and an IoT data stream of an industrial asset; wherein the visual data stream is a rolling shutter camera frame sequence carrying a row-by-row exposure time metadata, and the IoT data stream is a sensor time sequence carrying a transmission timestamp log;

[0258] The first processing module 20 is configured to calculate, based on the rolling shutter camera frame sequence and the row-by-row exposure time metadata, a geometric deviation perpendicular to a scanning direction as an in-frame anisotropy error from the visual data stream; estimate a latency distribution from the IoT data stream based on the sensor time sequence and the transmission timestamp log; and fuse the in-frame anisotropy error and the latency distribution into a deviation statistic;

[0259] The judgment module 30 is configured to determine, based on the deviation statistic, whether there is rolling shutter vibration distortion or variable transmission latency;

[0260] The second processing module 40 is configured to, in response to at least one of the rolling shutter vibration distortion or the variable transmission time delay existing, construct a unified time base based on the deviation statistics, extract first event times and second event times from the visual data stream and the IoT data stream respectively under the unified time base, register the first event times and the second event times, and obtain a unified data stream.

[0261] The management module 50 is configured to perform digital twin management on the industrial asset based on the unified data stream, and obtain a digital twin management result of the industrial asset.

[0262] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for industrial asset digital twin management fusing vision and IoT, characterized in that, The method comprises: acquiring a visual data stream and an IoT data stream of an industrial asset; wherein the visual data stream is a rolling shutter camera frame sequence carrying line-by-line exposure time metadata, and the IoT data stream is a sensor time sequence carrying a transmission timestamp log; calculating, based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata, a geometric deviation perpendicular to a scanning direction from the visual data stream as an in-frame anisotropic error; estimating, based on the sensor time sequence and the transmission timestamp log, a time delay distribution from the IoT data stream; and fusing the in-frame anisotropic error and the time delay distribution into a deviation statistic; judging, based on the deviation statistic, whether there is rolling shutter vibration distortion or variable transmission time delay; in response to at least one of the rolling shutter vibration distortion or the variable transmission time delay existing, constructing a unified time base based on the deviation statistic, and extracting first event times and second event times from the visual data stream and the IoT data stream respectively under the unified time base, and registering the first event times and the second event times to obtain a unified data stream; based on the unified data stream, performing digital twin management on the industrial asset to obtain a digital twin management result of the industrial asset.

2. The industrial asset digital twin management method fusing vision and IoT according to claim 1, characterized in that, The method further comprises: acquiring high-frequency motion data generated by an inertial measurement unit rigidly connected to the rolling shutter camera, and parameterizing a characteristic vibration of the industrial asset based on the high-frequency motion data to obtain initial vibration parameters; starting from the initial vibration parameters, performing an iterative optimization process with the objective of minimizing image entropy, wherein each iteration cycle comprises: generating a geometric correction function based on the vibration parameters of the current iteration and the line-by-line exposure time metadata; applying the geometric correction function to a target frame in the rolling shutter camera frame sequence to obtain a candidate corrected image; calculating the image entropy of the candidate corrected image; and updating the vibration parameters according to the negative gradient direction of the image entropy; when the image entropy converges or reaches a preset number of iterations, terminating the iterative optimization process, and taking the geometric deformation variable corresponding to the finally converged vibration parameters as the geometric deviation.

3. The industrial asset digital twin management method fusing vision and IoT according to claim 2, characterized in that, The method further comprises: calculating a first-order gradient vector and a second-order Hessian matrix of the image entropy function of the candidate corrected image at the vibration parameters of the current iteration; performing positive definite correction on the second-order Hessian matrix to obtain a corrected Hessian matrix; determining an update amount of the vibration parameters based on the first-order gradient vector and the corrected Hessian matrix, and updating the vibration parameters.

4. The industrial asset digital twin management method fusing vision and IoT according to claim 3, characterized in that, The method further comprises: judging whether the second-order Hessian matrix is a positive definite matrix; in response to the second-order Hessian matrix being a non-positive definite matrix, performing the following steps: acquire real-time operating condition data time-synchronized with the candidate corrected image from the IoT data stream; determine an expected vibration parameter corresponding to the real-time operating condition data based on the real-time operating condition data and a preset correlation model representing a health state of the industrial asset; construct a regularization matrix, wherein a penalty strength of the regularization matrix is a function of a difference between a vibration parameter of a current iteration and the expected vibration parameter; add the regularization matrix to the second-order Hessian matrix to obtain the modified Hessian matrix.

5. The industrial asset digital twin management method fusing vision and IoT according to claim 1, characterized in that, The estimating a delay distribution from the IoT data stream based on the sensor time series and the transmission timestamp log comprises: parsing the transmission timestamp log to obtain arrival and forwarding time stamps of a plurality of intermediate nodes in a data transmission path of each data point in the sensor time series; dividing an end-to-end data transmission path into a plurality of transmission segments based on the arrival and forwarding time stamps of the plurality of intermediate nodes, wherein the transmission segments at least include a field bus transmission segment and a core network transmission segment; independently constructing a probability distribution sub-model of a delay for each transmission segment in the plurality of transmission segments; fusing a plurality of the probability distribution sub-models to obtain a multi-modal overall delay distribution.

6. The industrial asset digital twin management method fusing vision and IoT according to claim 5, characterized in that, Before the dividing the end-to-end data transmission path into the plurality of transmission segments, further comprising: extracting a delay fingerprint of a link and an intermediate node in the data transmission path, wherein the extracting the delay fingerprint comprises: for a link formed by any two adjacent intermediate nodes, calculating an online transmission delay of the link based on a forwarding time stamp of a data point leaving a previous intermediate node and an arrival time stamp of the data point arriving at a next intermediate node, and extracting statistical features representing characteristics of the online transmission delay to form a link fingerprint of the link; for any intermediate node in the data transmission path, calculating an intra-node processing delay of the intermediate node based on an arrival time stamp of a data point arriving at the intermediate node and a forwarding time stamp of the data point leaving the intermediate node, and extracting statistical features representing characteristics of the intra-node processing delay to form a node fingerprint of the intermediate node.

7. The industrial asset digital twin management method fusing vision and IoT according to claim 6, characterized in that, The dividing the end-to-end data transmission path into the plurality of transmission segments comprises: respectively clustering the link fingerprints and the node fingerprints, dividing links satisfying a preset link clustering condition into a same link category, and dividing intermediate nodes satisfying a preset node clustering condition into a same node category; dividing the data transmission path based on the link categories and the node categories to obtain the plurality of transmission segments.

8. The industrial asset digital twin management method fusing vision and IoT according to claim 7, characterized in that, The fusing a plurality of the probability distribution sub-models to obtain a multi-modal overall delay distribution comprises: identifying and defining a plurality of segment combination operating states based on a plurality of the probability distribution sub-models, wherein each segment combination operating state corresponds to a specific delay mode of each transmission segment; for each state in the plurality of segment combination operating states, calculating a conditional probability distribution of an overall delay under each state; estimating a probability that the data transmission path is in each segment combination operating state at a current time according to real-time observed delay data; and outputting the probability that the data transmission path is in each segment combination operating state at the current time. The plurality of conditional probability distributions are weighted-summed according to their corresponding probabilities to construct the overall multi-modal latency distribution.

9. The industrial asset digital twin management method fusing vision and IoT according to claim 1, characterized in that, Based on the bias statistics, it is determined whether there is a rolling shutter vibration distortion or a variable transmission latency, including: A cross-modal physical model is established, which is used to represent the mapping relationship from a shared latent physical state parameter to the intra-frame anisotropy error and the latency distribution; Based on the currently calculated intra-frame anisotropy error, the corresponding latent physical state parameter is estimated by reverse deduction using the cross-modal physical model; Based on the estimated latent physical state parameter, the expected latency distribution is generated by forward prediction using the cross-modal physical model; The statistical distance between the expected latency distribution and the latency distribution actually estimated from the IoT data stream is calculated; In response to the statistical distance being greater than or equal to a preset inconsistency threshold, it is determined that there is a rolling shutter vibration distortion or a variable transmission latency.

10. The industrial asset digital twin management platform fusing vision and IoT, used to implement the method of claim 1-9, characterized in that, Comprising: The acquisition module is used to acquire the visual data stream and the IoT data stream of the industrial asset; wherein the visual data stream is a rolling shutter camera frame sequence carrying line-by-line exposure time metadata, and the IoT data stream is a sensor time sequence carrying transmission timestamp logs; The first processing module is used to calculate the geometric deviation perpendicular to the scanning direction as the intra-frame anisotropy error from the visual data stream based on the rolling shutter camera frame sequence and the line-by-line exposure time metadata; estimate the latency distribution from the IoT data stream based on the sensor time sequence and the transmission timestamp logs; and fuse the intra-frame anisotropy error and the latency distribution into bias statistics; The judgment module is used to determine whether there is a rolling shutter vibration distortion or a variable transmission latency based on the bias statistics; The second processing module is used to, in response to at least one of the rolling shutter vibration distortion or the variable transmission latency, construct a unified time base based on the bias statistics, and extract first event time and second event time from the visual data stream and the IoT data stream respectively under the unified time base, and register the first event time and the second event time to obtain a unified data stream; The management module is used to perform digital twin management on the industrial asset based on the unified data stream to obtain a digital twin management result of the industrial asset.

Images