Visual and laser data time synchronization method for digital twinning of power transmission and transformation equipment

Abstract

The invention discloses a visual and laser data time synchronization method for digital twinning of power transmission and transformation equipment, and the method comprises the steps: building an online learning model, and training the model, so as to prolong the period of a synchronization signal; dynamically compensating clock skew and data transmission delay by using a timestamp calibration algorithm; accurate time marking is performed on data of the visual camera and the laser radar, and time synchronization of visual data and laser data of the power transmission and transformation equipment is realized. According to the method, the accuracy of data synchronization is remarkably improved, and accurate time marking of visual camera and laser radar data in a digital twin system of power transmission and transformation equipment is realized; a time stamp calibration algorithm is provided, so that time errors in the acquisition process are eliminated, and the consistency of multi-modal data on a time axis is ensured; the real-time perception capability of the digital twinborn model to the equipment state is enhanced, and more accurate data support is provided for operation and maintenance decision making, so that powerful technical guarantee is provided for efficient, stable and safe operation of power transmission and transformation equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization processing, and in particular to a visual and laser data time synchronization method for digital twins of power transmission and transformation equipment. Background Art

[0002] With the rapid development of power systems towards intelligence and digitalization, digital twin technology for power transmission and transformation equipment has become an important means to achieve efficient operation and maintenance management and status monitoring. Digital twins build virtual models of physical equipment by real-time acquisition and analysis of multimodal data (such as visual images, inertial measurement unit (IMU) data, and laser point clouds), thereby achieving dynamic monitoring and predictive maintenance of equipment status. This technology not only improves the safety and reliability of equipment operation, but also provides strong support for optimizing operation and maintenance decisions.

[0003] However, in practical applications, the performance of digital twin systems is significantly affected by data synchronization issues, especially in the data synchronization of sensors such as visual cameras and lidars, where existing technologies face many challenges. Clock deviation and data transmission delay are the main reasons for the inconsistency of these sensor data on the time axis. Due to the different time bases of different sensors and the changes in network transmission conditions, the data obtained from each sensor cannot accurately correspond to the state at the same time, which in turn affects the real-time and accuracy of the digital twin model.

[0004] Traditional synchronization technologies, such as those based on the Network Time Protocol (NTP) or Precision Time Protocol (PTP), can solve the problem of clock synchronization to a certain extent, but they usually require frequent signal exchanges to maintain synchronization accuracy, which increases communication overhead and may affect the synchronization effect due to network jitter. In addition, these methods often fail to fully consider the inherent delay differences between different types of sensors, making it difficult to meet the needs of digital twins of power transmission and transformation equipment for high-precision synchronization.

[0005] In order to overcome the above difficulties, there is an urgent need to develop more efficient and accurate time synchronization technology. Specifically, based on the characteristics of visual and lidar data, the calibration algorithm should be optimized to ensure that even slight time offsets can be effectively corrected, thereby ensuring the time consistency of all sensor data. Summary of the invention

[0006] In view of the above existing problems, the present invention is proposed.

[0007] Therefore, the present invention provides a visual and laser data time synchronization method and system for digital twins of power transmission and transformation equipment to solve the problem of significant clock deviation and transmission delay in the time synchronization of visual camera and lidar data, which leads to inconsistency of data on the time axis and affects the real-time and accuracy of the digital twin system.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0009] In the first aspect, the present invention provides a method for time synchronization of visual and laser data for digital twins of power transmission and transformation equipment, including: establishing an online learning model and training the model to extend the period of the synchronization signal; based on the trained online learning model, using a timestamp calibration algorithm to dynamically compensate for clock deviation and data transmission delay; accurately time-marking the visual camera and lidar data to achieve time synchronization of visual and laser data of the power transmission and transformation equipment.

[0010] As a preferred solution of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in the present invention, wherein: the establishment of an online learning model and training the model includes:

[0011] The online learning model includes a recurrent neural network with an LSTM layer followed by a fully connected layer;

[0012] Training the online learning model by using the measured data points as an initial training data set, and testing the trained online learning model using temperature and timestamp as input features;

[0013] When new labeled data is input into the online learning model, the Adam optimizer is used to update and adapt the online learning model.

[0014] As a preferred solution of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in the present invention, the time offset defined in the timestamp calibration algorithm includes:

[0015] Due to trigger delay, transmission delay and clock asynchrony, the generated timestamp is different from the actual sampling time, resulting in time inconsistency between the data of the camera and the inertial measurement unit. The time offset t is defined. d Make the camera and inertial measurement unit data streams time-aligned.

[0016] As a preferred solution of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in the present invention, the characteristic speed of the image plane in the timestamp calibration algorithm includes:

[0017] To achieve temporal consistency between the camera and IMU data streams, the camera sequence should be based on the time offset t d Move forward or backward;

[0018] Setting I k and I k+1 are two consecutive image frames, the camera takes pictures at a short time interval [tk ,t k+1 ]From C k Move to C at a constant speed k+1 , it can be approximately considered that the feature also moves at a constant speed on the image plane during this short time interval moving, where the constant speed The calculation is:

[0019]

[0020] in, and The features are respectively k and I k+1 The two-dimensional observations on k and t k+1 The image plane I k and I k+1 The timestamps between the kth frame and the k+1th frame.

[0021] As a preferred solution of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in the present invention, the time offset is modeled as a visual factor through three-dimensional position parameterization and depth parameterization in the timestamp calibration algorithm, including:

[0022] The three-dimensional position parameterization includes the time offset parameterized as a three-dimensional position P1=[x1 y1 z1] in the global coordinate system. T ,If there is a time inconsistency between the IMU and the camera, the IMU constraints and the visual constraints are also inconsistent in the time domain, the camera sequence should be moved forward or backward, that is, the observation value of the feature is moved on the time line;

[0023] The depth parameterization includes parameterizing the time offset relative to the image frame in depth or reciprocal depth, that is, the time offset is first projected into the global coordinate system and then back-projected onto the image plane in the local camera coordinate system j to find the optimal camera pose and feature observation values ​​and match them with the inertial measurement unit constraints.

[0024] As a preferred solution of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in the present invention, the optimization with time offset in the timestamp calibration algorithm includes:

[0025] The visual factor is added to the local bundle adjustment framework to add a time offset to the entire state variable, the feature P l Parameterized by 3D position in a global frame or depth relative to a certain image frame;

[0026] The problem is formulated as a cost function that includes IMU propagation factors, reprojection factors, and certain prior factors, and the visual factor is used to achieve time offset calibration.

[0027] As a preferred solution of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment described in the present invention, the compensation of the time difference in the timestamp calibration algorithm includes:

[0028] After each optimization with time offset, compensating for the time offset by shifting the timestamps of subsequent visual streams;

[0029] The system estimates the time interval δt between the compensated visual measurement and the inertial measurement in the subsequent data stream d , the time interval δt d It will be iteratively optimized in subsequent data streams and eventually converge to zero.

[0030] In the second aspect, the present invention provides a visual and laser data time synchronization system for digital twins of power transmission and transformation equipment, including:

[0031] A model building and training module, used for building an online learning model and training the online learning model to extend the period of the synchronization signal;

[0032] A dynamic compensation module, used to dynamically compensate for clock deviation and data transmission delay using a timestamp calibration algorithm based on the trained online learning model;

[0033] The time synchronization module is used to accurately time-stamp the visual camera and lidar data to achieve time synchronization of the visual and laser data of the power transmission and transformation equipment.

[0034] In a third aspect, the present invention provides an electronic device, comprising:

[0035] Memory and processor;

[0036] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment are implemented.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a visual and laser data time synchronization method for digital twins of power transmission and transformation equipment, dynamically compensates for clock deviations and data transmission delays through an online learning model, significantly improves the accuracy of data synchronization, and realizes accurate time marking of visual camera and lidar data in the digital twin system of power transmission and transformation equipment; in addition, the present invention proposes an advanced timestamp calibration algorithm to further eliminate time errors in the acquisition process and ensure the consistency of multimodal data on the time axis; the integrated application of these technologies not only enhances the real-time perception capability of the digital twin model of the equipment status, but also provides more accurate data support for operation and maintenance decisions, thereby providing a strong technical guarantee for the efficient, stable and safe operation of power transmission and transformation equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0040] Figure 1 This is a logical schematic diagram of the overall process of a method for visual and laser data time synchronization for digital twins of power transmission and transformation equipment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0042] Example 1

[0043] Reference Figure 1 As an embodiment of the present invention, a visual and laser data time synchronization method for digital twins of power transmission and transformation equipment is provided, such as Figure 1 The specific steps shown include:

[0044] S100: establishing an online learning model and training the model to extend the period of the synchronization signal;

[0045] S200: Based on the trained online learning model, it uses the timestamp calibration algorithm to dynamically compensate for clock deviation and data transmission delay;

[0046] S300: Accurately time-stamp visual camera and lidar data to achieve time synchronization of visual and laser data of power transmission and transformation equipment.

[0047] It should be noted that the present invention provides a visual and laser data time synchronization method for digital twins of power transmission and transformation equipment. It dynamically compensates for clock deviations and data transmission delays through an online learning model, significantly improves the accuracy of data synchronization, and realizes accurate time marking of visual camera and lidar data in the digital twin system of power transmission and transformation equipment. In addition, the present invention proposes an advanced timestamp calibration algorithm to further eliminate time errors in the acquisition process and ensure the consistency of multimodal data on the time axis. The integrated application of these technologies not only enhances the real-time perception capability of the digital twin model of the equipment status, but also provides more accurate data support for operation and maintenance decisions, thereby providing a strong technical guarantee for the efficient, stable and safe operation of power transmission and transformation equipment.

[0048] In the embodiment of the present application, the above step S100 establishes an online learning model and trains the model to extend the period of the synchronization signal, including:

[0049] The online learning model consists of a recurrent neural network with an LSTM layer followed by a fully connected layer;

[0050] The online learning model is trained by using the measured data points as the initial training dataset, and the trained online learning model is tested using temperature and timestamp as input features;

[0051] When new labeled data is input into the online learning model, the Adam optimizer is used to update and adapt the online learning model.

[0052] Specifically, the distributed network needs to be synchronized frequently, which introduces a lot of synchronization overhead. Therefore, a clock deviation and drift compensation model based on an online LSTM model is used to extend the period required for the synchronization signal and reduce the synchronization overhead. The online learning model consists of a recursive neural network with an LSTM layer followed by a fully connected layer. The Adam optimizer is used with a learning rate of 0.001 to train the recursive neural network. The network architecture is detailed in Table 1. The goal is to estimate the ppm of the oscillator by taking advantage of the fact that temperature change is one of the main causes of clock drift.

[0053] Table 1: Network architecture.

[0054]

[0055]

[0056] Specifically, in order to train and evaluate the proposed online LSTM-based method, the measured 1440 data points are used as the initial training data for training the proposed LSTM model. The remaining data is then divided into two groups, the first group is used to test online prediction using temperature and timestamp as input features, and the second group is used for the LSTM online learning method, that is, when new labeled data arrives, the model is updated and adapted.

[0057] To evaluate the accuracy of single-frequency and LTE-based PPM measurements, a GPS front end was connected to improve the stability of the external oscillator. The external oscillator was programmed to sweep from -0.5ppm to 0.5ppm in steps of 0.025ppm, simulating an oscillator with a fixed deviation. The PPM of the oscillator was measured using the single-frequency and LTE-based methods, and the measured results were compared with the corresponding programmed PPM.

[0058] It should be noted that the above step S100 can significantly reduce the communication overhead caused by frequent synchronization in the distributed network, and can more efficiently predict and compensate for clock deviation and drift, thereby not only reducing the frequency of dependence on synchronization signals, but also enhancing the adaptability and response speed of the system. By extending the synchronization cycle, this method effectively reduces the network load and lays a solid foundation for the precise time marking of visual camera and lidar data in subsequent steps.

[0059] In the embodiment of the present application, the above step S200 uses a timestamp calibration algorithm to dynamically compensate for clock deviation and data transmission delay based on a trained online learning model, including:

[0060] Specifically, the time offset defined in the timestamp calibration algorithm includes:

[0061] Due to trigger delay, transmission delay and clock asynchrony, the generated timestamp is different from the actual sampling time, resulting in time inconsistency between the data of the camera and the inertial measurement unit. The time offset t is defined. d To achieve the consistency of the data streams of the camera and the inertial measurement unit in time, the formula is expressed as:

[0062] t IMU =t cam +t d

[0063] It should be noted that the time offset t d The time offset t is the amount by which we should offset the camera timestamp so that the camera and IMU data streams are temporally consistent. d Can be positive or negative. If the camera sequence has a longer delay than the IMU sequence, the time offset t d is a negative value, otherwise the time offset t dIs a positive value.

[0064] Specifically, the characteristic speed of the image plane in the timestamp calibration algorithm includes:

[0065] To achieve temporal consistency between the camera and IMU data streams, the camera sequence should be based on the time offset t d Move forward or backward;

[0066] Setting I k and I k+1 are two consecutive image frames, the camera takes pictures at a short time interval [t k ,t k+1 ]From C k Move to C at a constant speed k+1 , it can be approximately considered that the feature also moves at a constant speed on the image plane during this short time interval Moving, where constant speed The calculation is:

[0067]

[0068] in, and The features are respectively k and I k+1 The two-dimensional observations on k and t k+1 The image plane I k and I k+1 The timestamps between the kth frame and the k+1th frame.

[0069] It should be noted that this embodiment does not move the entire camera or IMU sequence, but specifically moves the observation value of the feature on the timeline, and for this purpose, the feature velocity is introduced to model and compensate for the time inconsistency. In a very short time (a few milliseconds), the movement of the camera can be regarded as uniform motion, so in a short time, the feature moves at an approximately constant speed on the image plane.

[0070] Specifically, the time offset is modeled as a visual factor through three-dimensional position parameterization and depth parameterization in the timestamp calibration algorithm, including:

[0071] The 3D position parameterization including the time offset is parameterized in the global coordinate system as the 3D position P1 = [x1 y1z1] T , if the inertial measurement unit and the camera are inconsistent in time, the inertial measurement unit constraints and visual constraints are also inconsistent in the time domain. The camera sequence should be moved forward or backward, that is, the observation value of the feature should be moved on the time line. The formula is expressed as:

[0072]

[0073] The depth parameterization includes the temporal offset relative to the image frame parameterized by the depth or the inverse depth, with the depth λ in image i i For example, the traditional reprojection error from image i to image j can be written as:

[0074]

[0075] That is, the time offset is first projected into the global coordinate system, and then back-projected onto the image plane in the local camera coordinate system j to find the optimal camera pose and feature observations, and match them with the inertial measurement unit constraints.

[0076] It should be noted that in the classic sparse visual simultaneous localization and mapping (SLAM) algorithm, the visual measurement is expressed as a (re)projection error in the cost function, and the classic (re)projection error is reconstructed by adding a new variable, the time offset. There are two typical parameterizations of features. Some algorithms parameterize features as their three-dimensional positions in the global coordinate system, while other algorithms parameterize features as depth or reciprocal depth relative to a certain image frame. The time offset will be modeled as a visual factor for these two parameterizations.

[0077] Specifically, the optimization of the timestamp calibration algorithm with time offset includes:

[0078] Several camera frames and IMU measurement data are stored in a bundle. The size of the bundle is usually limited by the computational complexity. Local bundle adjustment (BA) jointly optimizes the camera and IMU states and feature locations.

[0079] The visual factor is added to the local bundle adjustment framework to add a time offset to the entire state variable, which is defined as:

[0080]

[0081] Among them, the kth IMU state is composed of the position in the global coordinate system speed direction and the IMU bias b in the local coordinate system a , b g composition;

[0082] Feature P l The problem is parameterized by the 3D position in the global frame or the depth relative to a certain image frame, and is formulated as a cost function that includes IMU propagation factors, reprojection factors, and certain prior factors. The visual factor factor is used to achieve time offset calibration, and the formula is:

[0083]

[0084] Specifically, the compensation of time difference in the timestamp calibration algorithm includes:

[0085] After each optimization with time offset, the time offset is compensated by shifting the timestamps of the subsequent visual streams, i.e., t′ cam =t cam +t d ; The system estimates the time interval δt between the compensated visual measurement and the inertial measurement in the subsequent data stream d , time interval δt d It will be iteratively optimized in subsequent data streams and eventually converge to zero.

[0086] It should be noted that as the time interval δt d As the size of the image is reduced, the basic assumption (features move at a constant speed on the image plane over a short period of time) becomes more and more reasonable, and the process gradually compensates for it from coarse to fine even if there are huge time offsets (e.g., hundreds of milliseconds) at the beginning.

[0087] It should be noted that the above step S200 can adjust the timestamp in real time by dynamically compensating for clock deviation and data transmission delay, ensuring the consistency of data from different sensors on the timeline. It not only reduces data mismatches caused by synchronization problems, but also improves the real-time response capability and accuracy of the digital twin system, so that the data of the visual camera and lidar can more accurately reflect the actual equipment status, thereby providing more reliable support for operation and maintenance decisions.

[0088] In the embodiment of the present application, the above step S300 accurately time-marks the visual camera and lidar data, realizes the time synchronization of the visual and laser data of the power transmission and transformation equipment, ensures that all collected data can be integrated and analyzed on a unified time basis, and greatly improves the accuracy and timeliness of the digital twin model in reflecting the actual operating status of the equipment. Accurate time synchronization not only optimizes the quality of the data, but also enhances the effectiveness and reliability of the operation and maintenance decisions made based on these data, providing a solid technical guarantee for the efficient management and maintenance of smart grids.

[0089] Example 2

[0090] In this embodiment, a visual and laser data time synchronization system for digital twins of power transmission and transformation equipment is provided, including a model building and training module, a dynamic compensation module, and a dynamic compensation module;

[0091] Specifically, the model building and training module is used to build an online learning model and train the online learning model to extend the period of the synchronization signal;

[0092] Specifically, the dynamic compensation module is used to dynamically compensate for clock deviation and data transmission delay using a timestamp calibration algorithm based on a trained online learning model;

[0093] Specifically, the dynamic compensation module is used to accurately time-mark the visual camera and lidar data to achieve time synchronization of the visual and laser data of the power transmission and transformation equipment.

[0094] It should be noted that the technical solution of the system for visual and laser data time synchronization for digital twins of power transmission and transformation equipment and the technical solution of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment mentioned above belong to the same concept. For details not described in detail in the technical solution of the system for visual and laser data time synchronization for digital twins of power transmission and transformation equipment in this embodiment, please refer to the description of the technical solution of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment mentioned above.

[0095] The above-mentioned unit modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above-mentioned modules.

[0096] This embodiment also provides an electronic device, which includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless method can be implemented through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a visual and laser data time synchronization method for digital twins of power transmission and transformation equipment is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0097] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method proposed in the above embodiment is implemented.

[0098] The storage medium proposed in this embodiment and the method proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0099] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (RandomAccess Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.

[0100] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0101] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0102] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0106] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A visual and laser data time synchronization method for digital twins of power transmission and transformation equipment, characterized in that: include: Establish an online learning model and train the model to extend the period of the synchronization signal; Based on the trained online learning model, dynamically compensating for clock deviation and data transmission delay using a timestamp calibration algorithm; Accurately time-stamp visual camera and lidar data to achieve time synchronization of visual and laser data of power transmission and transformation equipment.

2. The visual and laser data time synchronization method for digital twins of power transmission and transformation equipment according to claim 1 is characterized in that: The establishing of the online learning model and training of the model comprises: The online learning model includes a recurrent neural network with an LSTM layer followed by a fully connected layer; Training the online learning model by using the measured data points as an initial training data set, and testing the trained online learning model using temperature and timestamp as input features; When new labeled data is input into the online learning model, the Adam optimizer is used to update and adapt the online learning model.

3. The visual and laser data time synchronization method for digital twins of power transmission and transformation equipment according to claim 2 is characterized in that: The time offset defined in the timestamp calibration algorithm includes: Due to trigger delay, transmission delay and clock asynchrony, the generated timestamp is different from the actual sampling time, resulting in time inconsistency between the data of the camera and the inertial measurement unit. The time offset t is defined. d Make the camera and inertial measurement unit data streams time-aligned.

4. The visual and laser data time synchronization method for digital twins of power transmission and transformation equipment according to claim 3 is characterized in that: The characteristic speed of the image plane in the timestamp calibration algorithm includes: To achieve temporal consistency between the camera and IMU data streams, the camera sequence should be based on the time offset t d Move forward or backward; Setting I k and I k+1 are two consecutive image frames, the camera takes pictures at a short time interval [t k ,t k+1 ]From C k Move to C at a constant speed k+1 , it can be approximately considered that the feature also moves at a constant speed on the image plane during this short time interval moving, where the constant speed The calculation is: in, and The features are respectively k and I k+1 The two-dimensional observations on k and t k+1 The image plane I k and I k+1 The timestamps between the kth frame and the k+1th frame.

5. The visual and laser data time synchronization method for digital twins of power transmission and transformation equipment according to claim 4 is characterized in that: Modeling the time offset as a visual factor through three-dimensional position parameterization and depth parameterization in the timestamp calibration algorithm includes: The three-dimensional position parameterization includes parameterizing the time offset into a three-dimensional position P1=[x1y1z1] in the global coordinate system. T ,If there is a time inconsistency between the IMU and the camera, the IMU constraints and the visual constraints are also inconsistent in the time domain, the camera sequence should be moved forward or backward, that is, the observation value of the feature should be moved on the time line; The depth parameterization includes parameterizing the time offset relative to the image frame in depth or reciprocal depth, that is, the time offset is first projected into the global coordinate system and then back-projected onto the image plane in the local camera coordinate system j to find the optimal camera pose and feature observation values ​​and match them with the inertial measurement unit constraints.

6. The visual and laser data time synchronization method for digital twins of power transmission and transformation equipment according to claim 5 is characterized in that: The optimization of the timestamp calibration algorithm with time offset includes: The visual factor is added to the local bundle adjustment framework to add a time offset to the entire state variable, the feature P l Parameterized by 3D position in a global frame or depth relative to a certain image frame; The problem is formulated as a cost function that includes IMU propagation factors, reprojection factors, and certain prior factors, and the visual factor is used to achieve time offset calibration.

7. The visual and laser data time synchronization method for digital twins of power transmission and transformation equipment according to claim 6 is characterized in that: The compensation of the time difference in the timestamp calibration algorithm includes: After each optimization with time offset, compensating for the time offset by shifting the timestamps of subsequent visual streams; The system estimates the time interval δt between the compensated visual measurement and the inertial measurement in the subsequent data stream d , the time interval δt d It will be iteratively optimized in subsequent data streams and eventually converge to zero.

8. A system using the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment as described in any one of claims 1 to 7, characterized in that: include: A model building and training module, used for building an online learning model and training the online learning model to extend the period of the synchronization signal; A dynamic compensation module, used to dynamically compensate for clock deviation and data transmission delay using a timestamp calibration algorithm based on the trained online learning model; The time synchronization module is used to accurately time-stamp the visual camera and lidar data to achieve time synchronization of the visual and laser data of the power transmission and transformation equipment.

9. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the visual and laser data time synchronization method for digital twins of power transmission and transformation equipment as described in any one of claims 1 to 7.

Images