Visual and laser data time synchronization method for power transmission and transformation equipment digital twinning

Abstract

The application discloses a visual and laser data time synchronization method for power transmission and transformation equipment digital twinning, comprising: establishing an online learning model and training the model to prolong the cycle of the synchronization signal; dynamically compensating clock deviation and data transmission delay by using a timestamp calibration algorithm; and accurately marking the time of visual camera and laser radar data to realize time synchronization of visual and laser data of power transmission and transformation equipment. The application significantly improves the accuracy of data synchronization, realizes accurate time marking of visual camera and laser radar data in the power transmission and transformation equipment digital twinning system, eliminates time errors in the collection process by using the proposed timestamp calibration algorithm, and ensures the consistency of multi-modal data on the time axis. The application not only enhances the real-time perception ability of the digital twinning model to the equipment state, but also provides more accurate data support for operation and maintenance decision-making, thereby providing strong technical support for efficient, stable and safe operation of power transmission and transformation equipment.

Description

Technical Field

[0001] This invention relates to the field of data synchronization processing technology, and in particular to a method for time synchronization of visual and laser data for digital twins of power transmission and transformation equipment. Background Technology

[0002] With the rapid development of power systems towards intelligence and digitalization, digital twin technology for power transmission and transformation equipment has become an important means to achieve efficient operation and maintenance management and condition monitoring. Digital twins construct virtual models of physical equipment by real-time acquisition and analysis of multimodal data (such as visual images, inertial measurement unit (IMU) data, and laser point clouds), thereby enabling dynamic monitoring and predictive maintenance of equipment status. This technology not only improves the safety and reliability of equipment operation but also provides strong support for optimizing operation and maintenance decisions.

[0003] However, in practical applications, the performance of digital twin systems is significantly affected by data synchronization issues, especially in the synchronization of data from sensors such as visual cameras and LiDAR, where existing technologies face numerous challenges. Clock skew and data transmission delays are the main reasons for inconsistencies in the timeline of these sensor data. Due to the different time bases of different sensors, coupled with variations in network transmission conditions, the data acquired from each sensor cannot accurately correspond to the state at the same moment, thus affecting the real-time performance and accuracy of the digital twin model.

[0004] Traditional synchronization techniques, such as those based on Network Time Protocol (NTP) or Precision Time Protocol (PTP), can solve the clock synchronization problem to some extent, but they typically require frequent signal exchanges to maintain synchronization accuracy. This increases communication overhead and can affect synchronization performance due to network jitter. Furthermore, these methods often fail to adequately account for the inherent time delay differences between different types of sensors, making it difficult to meet the high-precision synchronization requirements of digital twins for power transmission and distribution equipment.

[0005] To overcome the above difficulties, there is an urgent need to develop more efficient and accurate time synchronization technology. Specifically, for the characteristics of visual and lidar data, calibration algorithms should be optimized to ensure that even slight time offsets can be effectively corrected, thus guaranteeing the time consistency of all sensor data. Summary of the Invention

[0006] In view of the aforementioned existing problems, the present invention is proposed.

[0007] Therefore, this invention provides a method and system for time synchronization of visual and laser data for digital twins of power transmission and transformation equipment, which solves the problem of significant clock deviation and transmission delay in the time synchronization of visual camera and lidar data, resulting in inconsistency of data on the time axis and affecting the real-time performance and accuracy of the digital twin system.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0009] In a first aspect, the present invention provides a method for time synchronization of visual and laser data for digital twins of power transmission and transformation equipment, comprising: establishing an online learning model and training the model to extend the period of the synchronization signal; based on the trained online learning model, using a timestamp calibration algorithm to dynamically compensate for clock deviation and data transmission delay; and accurately time-marking the visual camera and lidar data to achieve time synchronization of visual and laser data of power transmission and transformation equipment.

[0010] As a preferred embodiment of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in this invention, the step of establishing an online learning model and training the model includes:

[0011] The online learning model comprises a recurrent neural network consisting of an LSTM layer followed by a fully connected layer.

[0012] The online learning model was trained by using the measured data points as the initial training dataset, and the trained online learning model was tested using temperature and timestamps as input features.

[0013] When new labeled data is input into the online learning model, the Adam optimizer is used to update and adapt the online learning model.

[0014] As a preferred embodiment of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in this invention, the time offset defined in the timestamp calibration algorithm includes:

[0015] Due to triggering delay, transmission delay, and clock asynchrony, the generated timestamp differs from the actual sampling time, leading to time inconsistency between the data from the camera and the inertial measurement unit. A time offset t is defined. d To ensure that the data streams from the camera and the inertial measurement unit are kept consistent in time.

[0016] As a preferred embodiment of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in this invention, wherein the feature velocity of the image plane in the timestamp calibration algorithm includes:

[0017] To ensure that the data streams from the camera and the inertial measurement unit are kept in time consistent, the camera sequence should be adjusted according to the time offset t. d Move forward or backward;

[0018] Setting I k and I k+1 These are two consecutive image frames, captured by the camera in a short time interval [t]k ,t k+1 [From C] k Move at a constant speed to C k+1 It can be approximated that the features also move at a constant speed on the image plane during this short time interval. Movement, wherein the constant speed The calculation is as follows:

[0019]

[0020] in, and Features in image plane I k and I k+1 Two-dimensional observations on t k and t k+1 Image plane I k and I k+1 The timestamps of the k-th frame and the (k+1)-th frame.

[0021] As a preferred embodiment of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in this invention, wherein: the timestamp calibration algorithm models the time offset as a visual factor through three-dimensional position parameterization and depth parameterization, including:

[0022] The three-dimensional position parameterization includes parameterizing the time offset into a three-dimensional position P1 = [x1 y1 z1] in the global coordinate system. T If the inertial measurement unit and the camera are out of sync in time, then the constraints of the inertial measurement unit and the visual constraints are also out of sync in the time domain. The camera sequence should be moved forward or backward, that is, the observations of the features should be moved along the timeline.

[0023] The depth parameterization includes parameterizing the time offset relative to the image frame with depth or reciprocal depth. That is, the time offset is first projected into the global coordinate system and then projected back into the image plane in the local camera coordinate system j to find the optimal camera pose and feature observation values ​​and match them with the constraints of the inertial measurement unit.

[0024] As a preferred embodiment of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in this invention, the optimization of the timestamp calibration algorithm with time offset includes:

[0025] The visual factors are added to the local bundle adjustment framework to increase the time offset across the entire state variable, feature P. l Parameterized by 3D position in the global frame or depth relative to a certain image frame;

[0026] The problem is formulated as a cost function that includes the IMU propagation factor, the reprojection factor, and certain prior factors, and time offset calibration is achieved using visual factor factors.

[0027] As a preferred embodiment of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in this invention, the time difference compensation in the timestamp calibration algorithm includes:

[0028] After each optimization with a time offset, the time offset is compensated by shifting the timestamp of the subsequent visual stream;

[0029] The system estimates the time interval δt between the compensated visual measurement and the inertial measurement in subsequent data streams. d The time interval δt d It will be iteratively optimized in subsequent data streams and eventually converge to zero.

[0030] Secondly, the present invention provides a visual and laser data time synchronization system for digital twins of power transmission and transformation equipment, comprising:

[0031] The model building and training module is used to build an online learning model and train the online learning model to extend the period of the synchronization signal;

[0032] The dynamic compensation module is used to dynamically compensate for clock deviation and data transmission delay based on the trained online learning model using a timestamp calibration algorithm.

[0033] The time synchronization module is used to accurately time-mark data from visual cameras and lidar, enabling time synchronization of visual and lidar data from power transmission and transformation equipment.

[0034] Thirdly, the present invention provides an electronic device, comprising:

[0035] Memory and processor;

[0036] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of a visual and laser data time synchronization method for digital twins of power transmission and transformation equipment.

[0037] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment.

[0038] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a method for time synchronization of visual and laser data for digital twins of power transmission and transformation equipment. By dynamically compensating for clock deviations and data transmission delays through an online learning model, the accuracy of data synchronization is significantly improved, achieving precise time stamping of visual camera and lidar data in the digital twin system of power transmission and transformation equipment. In addition, this invention proposes an advanced timestamp calibration algorithm to further eliminate time errors in the acquisition process and ensure the consistency of multimodal data on the time axis. The integrated application of these technologies not only enhances the real-time perception capability of the digital twin model of equipment status but also provides more accurate data support for operation and maintenance decisions, thereby providing a strong technical guarantee for the efficient, stable, and safe operation of power transmission and transformation equipment. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the overall process logic of a visual and laser data time synchronization method for digital twins of power transmission and transformation equipment according to an embodiment of the present invention. Detailed Implementation

[0041] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0042] Example 1

[0043] Reference Figure 1 As one embodiment of the present invention, a method for time synchronization of visual and laser data for digital twins of power transmission and transformation equipment is provided, such as... Figure 1 The specific steps shown are as follows:

[0044] S100: Establish an online learning model and train the model to extend the period of the synchronization signal;

[0045] S200: Based on a trained online learning model, it uses a timestamp calibration algorithm to dynamically compensate for clock skew and data transmission delay;

[0046] S300: Accurately timestamps visual camera and lidar data to achieve time synchronization of visual and lidar data for power transmission and transformation equipment.

[0047] It should be noted that this invention provides a method for synchronizing visual and laser data in digital twins of power transmission and transformation equipment. By dynamically compensating for clock deviations and data transmission delays through an online learning model, the accuracy of data synchronization is significantly improved, enabling precise time stamping of visual camera and lidar data in the digital twin system of power transmission and transformation equipment. In addition, this invention proposes an advanced timestamp calibration algorithm to further eliminate time errors during the acquisition process and ensure the consistency of multimodal data on the time axis. The integrated application of these technologies not only enhances the real-time perception capability of the digital twin model of equipment status but also provides more accurate data support for operation and maintenance decisions, thereby providing a strong technical guarantee for the efficient, stable, and safe operation of power transmission and transformation equipment.

[0048] In this embodiment of the application, the above step S100 of establishing an online learning model and training the model to extend the period of the synchronization signal includes:

[0049] The online learning model consists of a recurrent neural network with an LSTM layer followed by a fully connected layer;

[0050] The online learning model was trained by using the measured data points as the initial training dataset, and the trained online learning model was tested using temperature and timestamps as input features.

[0051] When new labeled data is input into the online learning model, the Adam optimizer is used to update and adapt the online learning model.

[0052] Specifically, distributed networks require frequent synchronization, introducing significant synchronization overhead. Therefore, a clock skew and drift compensation model based on an online LSTM model is used to extend the period required for the synchronization signal and reduce synchronization overhead. The online learning model consists of an LSTM layer followed by a fully connected recurrent neural network. The recurrent neural network is trained using the Adam optimizer with a learning rate of 0.001. Table 1 details the network architecture. The goal is to estimate the oscillator's ppm by utilizing the fact that temperature variation is one of the main causes of clock drift.

[0053] Table 1: Network Architecture.

[0054]

[0055]

[0056] Specifically, to train and evaluate the proposed online LSTM-based method, 1440 measured data points were used as initial training data to train the proposed LSTM model. The remaining data was then divided into two groups: the first group was used to test online predictions, using temperature and timestamps as input features; the second group was used for the LSTM online learning method, i.e., updating and adapting the model as new labeled data arrives.

[0057] Specifically, to evaluate the accuracy of PPM measurements based on single-frequency and LTE, a GPS front-end was connected to improve the stability of the external oscillator. The external oscillator was programmed to scan from -0.5 ppm to 0.5 ppm in 0.025 ppm steps, simulating an oscillator with a fixed bias. The oscillator's PPM was measured using both single-frequency and LTE-based methods, and the results were compared with the corresponding programmed PPM.

[0058] It should be noted that step S100 significantly reduces the communication overhead caused by frequent synchronization in the distributed network, and can more efficiently predict and compensate for clock skew and drift. This not only reduces the frequency of dependence on synchronization signals but also enhances the system's adaptability and response speed. By extending the synchronization period, this method effectively reduces the network load, laying a solid foundation for achieving accurate time stamping of visual camera and LiDAR data in subsequent steps.

[0059] In this embodiment of the application, step S200, based on the trained online learning model, dynamically compensates for clock skew and data transmission delay using a timestamp calibration algorithm, including:

[0060] Specifically, the time offset defined in the timestamp calibration algorithm includes:

[0061] Due to triggering delay, transmission delay, and clock asynchrony, the generated timestamp differs from the actual sampling time, leading to time inconsistency between the data from the camera and the inertial measurement unit. A time offset t is defined. d To ensure that the data streams from the camera and the inertial measurement unit are kept in sync over time, the formula is as follows:

[0062] t IMU =t cam +t d

[0063] It should be noted that the time offset t d This refers to the amount by which we should offset the camera timestamps to ensure that the camera and IMU data streams are time-synchronized. Time offset t d It can be a positive or negative value. If the delay time of the camera sequence is longer than that of the IMU sequence, the time offset t... d It is negative; otherwise, the time offset t dIt is a positive value.

[0064] Specifically, the characteristic velocities of the image plane in the timestamp calibration algorithm include:

[0065] To ensure that the data streams from the camera and the inertial measurement unit are kept in time consistent, the camera sequence should be adjusted according to the time offset t. d Move forward or backward;

[0066] Setting I k and I k+1 These are two consecutive image frames, captured by the camera in a short time interval [t] k ,t k+1 [From C] k Move at a constant speed to C k+1 It can be approximated that the features also move at a constant speed on the image plane during this short time interval. Movement, where the speed is constant The calculation is as follows:

[0067]

[0068] in, and Features in image plane I k and I k+1 Two-dimensional observations on t k and t k+1 Image plane I k and I k+1 The timestamps of the k-th frame and the (k+1)-th frame.

[0069] It should be noted that this embodiment does not move the entire camera or IMU sequence, but rather moves the observations of features specifically on the timeline. For this purpose, feature velocity is introduced to model and compensate for temporal inconsistencies. In a very short time (a few milliseconds), the camera movement can be considered as uniform motion, so in a short time, the features move at an approximately constant speed on the image plane.

[0070] Specifically, the timestamp calibration algorithm models the time offset as a visual factor through 3D position parameterization and depth parameterization, including:

[0071] 3D position parameterization includes parameterizing the time offset into a 3D position P1 = [x1 y1 z1] in the global coordinate system. T If the inertial measurement unit (IMU) and the camera have a time inconsistency, then the IMU constraints and the visual constraints will also be inconsistent in the time domain. The camera sequence should be shifted forward or backward, i.e., the observations of the features should be moved along the timeline. The formula is as follows:

[0072]

[0073] Depth parameterization includes parameterizing the time offset relative to the image frame by depth or inverse depth, with the depth λ in image i as the parameter. i For example, the traditional reprojection error from image i to image j can be written as:

[0074]

[0075] That is, the time offset is first projected into the global coordinate system, and then projected back into the image plane in the local camera coordinate system j, in order to find the optimal camera pose and feature observation values, and match them with the constraints of the inertial measurement unit.

[0076] It should be noted that in the classic Sparse Visual Simultaneous Localization and Mapping (SLAM) algorithm, visual measurements are expressed as (re)projection error in the cost function. This classic (re)projection error is reconstructed by adding a new variable—the time offset. Features have two typical parameterization methods: some algorithms parameterize features as their 3D position in the global coordinate system, while others parameterize them as their depth or reciprocal depth relative to a given image frame. The time offset will be modeled as a visual factor for both parameterization methods.

[0077] Specifically, optimizations to the timestamp calibration algorithm that incorporate time offsets include:

[0078] Several camera frames and IMU measurement data are stored in a bundle, the size of which is usually limited to the range of computational complexity. Local bundle adjustment (BA) jointly optimizes the camera and IMU states as well as feature locations.

[0079] Visual factors are added to the local bundle adjustment framework to increase the time offset across the entire state variable, defined as follows:

[0080]

[0081] The k-th IMU state is determined by its position in the global coordinate system. speed direction and IMU offset b in local coordinate system a b g composition;

[0082] Feature P l The problem is parameterized by the 3D position in the global frame or the depth relative to a certain image frame, and is formulated as a cost function containing the IMU propagation factor, reprojection factor, and certain prior factors. Time offset calibration is achieved using visual factorization, as shown in the formula:

[0083]

[0084] Specifically, the compensation for time differences in the timestamp calibration algorithm includes:

[0085] After each optimization with a time offset, the time offset is compensated by shifting the timestamp of the subsequent visual stream, i.e., t′. cam =t cam +t d The system estimates the time interval δt between the compensated visual measurement and the inertial measurement in subsequent data streams. d Time interval δt d It will be iteratively optimized in subsequent data streams and eventually converge to zero.

[0086] It should be noted that, with the time interval δt d As the time lag decreases, the basic assumption (features move at a constant speed on the image plane over a short period of time) becomes more and more reasonable, and the process gradually compensates for large time lags (e.g., hundreds of milliseconds) from the beginning.

[0087] It should be noted that step S200 above can adjust the timestamp in real time by dynamically compensating for clock deviation and data transmission delay, ensuring the consistency of data from different sensors on the time axis. This not only reduces data mismatch caused by synchronization problems, but also improves the real-time response capability and accuracy of the digital twin system, enabling the data from the visual camera and LiDAR to more accurately reflect the actual equipment status, thereby providing more reliable support for operation and maintenance decisions.

[0088] In this embodiment, step S300 precisely timestamps the data from the visual camera and LiDAR, achieving time synchronization of visual and LiDAR data from power transmission and transformation equipment. This ensures that all collected data can be integrated and analyzed on a unified time reference, greatly improving the accuracy and timeliness of the digital twin model in reflecting the actual operating status of the equipment. Precise time synchronization not only optimizes data quality but also enhances the effectiveness and reliability of operation and maintenance decisions made based on this data, providing a solid technical guarantee for the efficient management and maintenance of smart grids.

[0089] Example 2

[0090] This embodiment provides a vision and laser data time synchronization system for digital twins of power transmission and transformation equipment, including a model building and training module, a dynamic compensation module, and a dynamic compensation module.

[0091] Specifically, the model building and training module is used to build and train online learning models to extend the period of the synchronization signal;

[0092] Specifically, the dynamic compensation module is used to dynamically compensate for clock skew and data transmission delay based on the trained online learning model using a timestamp calibration algorithm;

[0093] Specifically, the dynamic compensation module is used to accurately time-mark the data from the vision camera and lidar, thereby achieving time synchronization of the vision and lidar data of the power transmission and transformation equipment.

[0094] It should be noted that the technical solution of the system for time synchronization of visual and laser data for digital twins of power transmission and transformation equipment is based on the same concept as the technical solution of the method for time synchronization of visual and laser data for digital twins of power transmission and transformation equipment described above. For details not described in detail in the technical solution of the system for time synchronization of visual and laser data for digital twins of power transmission and transformation equipment in this embodiment, please refer to the description of the technical solution of the method for time synchronization of visual and laser data for digital twins of power transmission and transformation equipment described above.

[0095] The above-mentioned unit modules can be embedded in the processor of the computer device in hardware form or independent of it, or they can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0096] This embodiment also provides an electronic device, which includes a processor, a memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of this computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a visual and laser data time synchronization method for digital twins of power transmission and transformation equipment. The display screen of the computer device can be a liquid crystal display screen or an e-ink display screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the casing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0097] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method proposed in the above embodiments.

[0098] The storage medium proposed in this embodiment belongs to the same inventive concept as the method proposed in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0099] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0100] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

[0101] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0102] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0106] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for time synchronization of visual and laser data for power transmission and transformation equipment digital twinning, characterized in that, The method comprises: establishing an online learning model and training the model to extend the period of the synchronization signal; based on the trained online learning model, using a timestamp calibration algorithm to dynamically compensate for clock bias and data transmission delay; accurately time-stamping visual camera and lidar data to achieve time synchronization of visual and laser data of power transmission and transformation equipment; the establishment of an online learning model and training the model comprises: the online learning model comprises a recurrent neural network with an LSTM layer followed by a fully connected layer; the online learning model is trained by using measured data points as initial training data set, and the trained online learning model is tested by using temperature and timestamp as input features; when new labeled data is input into the online learning model, the online learning model is updated and adapted using an Adam optimizer; the definition of time offset in the timestamp calibration algorithm comprises: Due to trigger delay, transmission delay and clock unsynchronization, the generated timestamps are different from the actual sampling time, resulting in time inconsistency between camera and inertial measurement unit data, defining a time offset The data streams of the camera and the inertial measurement unit are kept consistent in time; the feature velocity of the image plane in the timestamp calibration algorithm comprises: To achieve a temporal consistency of the data streams of the camera and the inertial measurement unit, the camera sequence should be adapted according to the time offset forward or backward movement; Setting and are two consecutive image frames, the camera moves at constant speed from to in a short time interval , it can be approximately considered that the feature also moves at constant speed in the image plane in this short time interval, where the calculation of the constant speed is: in, and Features in the image plane and Two-dimensional observations on the surface and Image planes and Between k Frame and the k +1 frame timestamp.

2. The method of claim 1, wherein, in the timestamp calibration algorithm, the time offset is modeled as a visual factor through three-dimensional position parameterization and depth parameterization, which comprises: The three-dimensional position parameterization includes parameterization of the time offset in a global coordinate system as a three-dimensional position If there is a time inconsistency between the inertial measurement unit and the camera, the inertial measurement unit constraints and the visual constraints are also inconsistent in the time domain, the camera sequence should be moved forward or backward, that is, the observation value of the feature is moved on the time line; the depth parameterization comprises parameterizing the time offset with respect to the image frame in depth or inverse depth, i.e. the time offset is first projected into the global coordinate system, and then inversely projected onto the image plane in the local camera coordinate system j to find the optimal camera pose and feature observation value, and matched with the inertial measurement unit constraint.

3. The method of claim 2, wherein, the optimization with time offset in the timestamp calibration algorithm comprises: adding the visual factor to the local bundle adjustment framework to increase the time offset for the entire state variable, and parameterizing the feature through the 3D position in the global frame or the depth with respect to a certain image frame; formulating the problem as a cost function containing IMU propagation factors, re-projection factors and certain prior factors, and using the visual factor to realize time offset calibration.

4. The method of claim 3, wherein, the compensation of time difference in the timestamp calibration algorithm comprises: after each optimization with time offset, the time offset is compensated by moving the timestamp of the subsequent visual stream. The system estimates the time interval between the compensated visual measurements and the inertial measurements in the subsequent data stream , the time interval Will be iteratively optimized in the subsequent data stream, eventually converging to zero.

5. A system using the visual and laser data time synchronization method for power transmission and transformation equipment digital twinning according to any one of claims 1-4, characterized in that, The method comprises: a model establishment and training module for establishing an online learning model and training the online learning model to extend the period of the synchronization signal; a dynamic compensation module for dynamically compensating for clock bias and data transmission delay based on the trained online learning model using a timestamp calibration algorithm; a time synchronization module for accurately time-stamping visual camera and lidar data to achieve time synchronization of visual and laser data of power transmission and transformation equipment.

6. An electronic device comprising: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions, which realize the steps of the visual and laser data time synchronization method for power transmission and transformation equipment digital twin in any one of claims 1-4 when executed by the processor. 7.A computer readable storage medium storing computer executable instructions that, when executed by a processor, perform the steps of the method for visual and laser data time synchronization for power transmission and transformation equipment digital twinning of any one of claims 1-4.

Images