A data multi-scale fusion method for power transmission and transformation equipment digital twinning

Abstract

The application discloses a kind of data multiscale fusion methods for power transmission and transformation equipment digital twinning, it is related to power equipment technical field, including using laser radar and camera to obtain power transmission and transformation equipment data and calibrating, obtain the data of power transmission and transformation equipment after calibration;The target detector is used to the target detection of the data of power transmission and transformation equipment after calibration, and detection result is obtained;Data fusion is carried out based on the detection result, and observation set is obtained;The observation set is input into BiLSTM network to classify the state of power transmission and transformation equipment and optimize the BiLSTM network, and the state evaluation result of power transmission and transformation equipment is obtained.The application adopts multi-angle joint calibration technology, respectively applies YOLOv8 target detector to laser and visual data image classification, and uses bidirectional long short-term memory BiLSTM model to carry out quality evaluation to the detected target, and uses Adam optimizer to carry out hyperparameter optimization, improves the accuracy of quality detection.

Description

Technical Field

[0001] This invention relates to the field of power equipment technology, and in particular to a multi-scale data fusion method for digital twins of power transmission and transformation equipment. Background Technology

[0002] Power transmission and transformation equipment is prone to failure due to continuous operation under various working conditions. Failure of any of these devices can lead to subsequent failures in other devices, ultimately causing the paralysis of the entire power system. To ensure the optimal state of the system, these power devices must be continuously monitored to improve the overall efficiency and functionality of the system. Effective condition monitoring technology can improve the reliability and service life of equipment. Due to continuous operation, any electrical equipment requires regular maintenance, which is an essential part of its life cycle. Currently, advanced technologies for power equipment fault monitoring include inverse problem theory, image processing and computer vision methods, advanced signal processing technology, model-based prediction and health management, digital twin methods, and artificial intelligence methods.

[0003] However, traditional mathematical models or single-sensor monitoring methods cannot meet the current technological development needs of power transmission and transformation equipment. It is necessary to fuse multi-sensor data and combine deep learning methods to train models based on historical data and real-time observations, identify specific fault modes and detect deviations from normal real-time operating conditions, minimize power grid faults, and ensure smooth interaction with the power grid. Regarding current multi-sensor data fusion technologies, traditional RGB-D and binocular vision sensors, although low in cost and simple to deploy, have the disadvantages of limited application scope and computational complexity. Many labeled image datasets can be used for monitoring power equipment, but interpreting lidar point cloud data is quite difficult, mainly because it is difficult to separate many target equipment groups, sparse distribution types, and the ability to identify equipment status only from circulation. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the problem to be solved by this invention is how to provide a method for assessing the status of power transmission and transformation equipment, predicting faults, and maintaining the equipment by setting up a digital twin framework based on laser and visual data, studying the fusion and optimization of multimodal data, and using deep learning methods.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide a multi-scale data fusion method for digital twins of power transmission and transformation equipment, comprising: acquiring and calibrating power transmission and transformation equipment data using lidar and cameras to obtain calibrated power transmission and transformation equipment data; performing target detection on the calibrated power transmission and transformation equipment data using a target detector to obtain detection results; performing data fusion based on the detection results to obtain an observation set; inputting the observation set into a BiLSTM network for power transmission and transformation equipment state classification and optimizing the BiLSTM network to obtain a state assessment result of the power transmission and transformation equipment.

[0008] As a preferred embodiment of the multi-scale data fusion method for digital twins of power transmission and transformation equipment described in this invention, the power transmission and transformation equipment data includes lidar data acquired by lidar and image data acquired by a camera; the lidar and camera lenses are mounted on the same vertical line, the vertical position of the camera lens is higher than that of the lidar, and the vertical view of the camera lens covers the lidar view.

[0009] As a preferred embodiment of the multi-scale data fusion method for digital twins of power transmission and transformation equipment described in this invention, the method for obtaining the calibrated power transmission and transformation equipment data includes the following steps: selecting an equivalent angle of N in the horizontal direction to obtain N+1 calibration positions; setting radar angle reflectors at the N+1 calibration positions to obtain a set of radar calibration angles, as shown in the following formula:

[0010] Ω={α1,α2,…,α N ,α N+1}

[0011] Where Ω represents the set of radar calibration angles; α N Let the radar calibration angle be the Nth calibration position; image the radar angle reflector at the N+1 calibration positions to obtain the set of horizontal positions of the camera calibration pixels, as shown in the following formula:

[0012] X = {x1, x2, ..., x} N ,x N+1}

[0013] Where X is the set of horizontal positions of the camera calibration pixels; x N The horizontal position of the camera calibration pixel at the Nth calibration position is determined. Based on the radar calibration angle set Ω and the horizontal position set X of the camera calibration pixels, the horizontal field of view and image resolution are calculated using the following formulas:

[0014]

[0015] Where, θ νWhere is the horizontal position of the camera; W is the number of horizontal pixels; the pixel angular density is calculated by linear interpolation using the angular density of pixels in the surrounding area, as shown in the following formula:

[0016]

[0017] Where, α x Let ρ(x) be the lidar angle corresponding to pixel x; ρ(x) be the pixel angular density in the interval where pixel x is located; the tangential distortion and mirror distortion of the image data acquired by the camera are corrected using the pixel angular density to obtain the corrected image data.

[0018] As a preferred embodiment of the multi-scale data fusion method for digital twins of power transmission and transformation equipment described in this invention, wherein: when x n-1 <x<x n When the pixel angular density satisfies the following formula:

[0019]

[0020] Where Δα is the angular range between calibration points; α n-1 ρ is the lidar angle at the (n-1)th calibration position; n (x) is the pixel angular density function for the nth calibration interval; x is the average pixel angular density of the nth calibration interval; n The horizontal position of the camera calibration pixel at the nth calibration position; when (x n-1 ,x n The angular density of the pixels in the image varies uniformly, and the pixel angular density satisfies the following formula:

[0021]

[0022] The specific formulas for the left and right edges of a pixel image are as follows:

[0023]

[0024] in, This represents the average pixel corner density at the left edge of the pixel image; This represents the average pixel angular density at the right edge of the pixel image.

[0025] As a preferred embodiment of the multi-scale data fusion method for digital twins of power transmission and transformation equipment described in this invention, the method involves the following steps: using a target detector to perform target detection on the calibrated power transmission and transformation equipment data to obtain the detection result; using a YOLOv8 target detector to perform target detection on the lidar data acquired by the lidar to obtain the lidar target detection result; and using a YOLOv8 target detector to perform target detection on the corrected image data acquired by the camera to obtain the camera target detection result.

[0026] As a preferred embodiment of the multi-scale data fusion method for digital twins of power transmission and transformation equipment described in this invention, the following steps are included in the data fusion based on the detection results to obtain the observation set: obtaining the spatial velocity and position of objects in the lidar target detection results, obtaining the azimuth information and object classification in the camera target detection results; and using the spatial velocity, position, azimuth information, and object classification of the objects to perform maximum a posteriori estimation to obtain the optimal fusion azimuth angle, as shown in the following formula:

[0027]

[0028] Where, θ f The optimal fusion azimuth angle is used to calculate the target measurement data, and the specific formula is as follows:

[0029]

[0030] z m =[xyv x v y "type"] T

[0031] Among them, z m For target measurement data; For radar points; For image measurement; an observation set is generated based on the target measurement data and the object classification, using the following formula:

[0032]

[0033] Among them, z k For the observation set.

[0034] As a preferred embodiment of the multi-scale data fusion method for digital twins of power transmission and transformation equipment described in this invention, the BiLSTM network consists of two different LSTM networks, including a forward I / P sequence and a reverse input sequence; the hidden layer HL and the output of the BiLSTM network are similar in opposite aspects, as shown in the following formula:

[0035] fo t =σ(WE xfo x t +WE hfo h t-1 +WE gfo m t-1 +de fo )

[0036] in t =σ(WE xin x t +WE hin h t-1 +WE gin m t-1 +de in )

[0037] ou t =σ(WE xou x t +WE hou h t-1 +WE gou m t-1 +de ou )

[0038] M′ t =tanh(WE xm x t +WE xh +de g )

[0039] M t =fo t ·m t-1 +in t ·M′ t

[0040] h t =ou t ·tanh(M t )

[0041] Where WE* is the weighting matrix; b* represents the bias of the three gating mechanisms and the I / P converter; tanh is the activation function; fo t Forget gate; M′t is the gate from x t and h t-1 The new data obtained is M′t;in t For I / P gates, select to integrate data into system memory; out t Choose to filter the data in memory before outputting the result gate.

[0042] Secondly, to further address the safety issues existing in power equipment, this invention provides a multi-scale data fusion system for digital twins of power transmission and transformation equipment. The system includes: a data calibration module for acquiring lidar data and image data using lidar and a camera, selecting calibration positions and performing calculations to obtain pixel angular density, and calibrating the image data to obtain corrected image data; a target detection module for performing target detection on the lidar data and corrected image data using a YOLOv8 target detector to obtain lidar target detection results and camera target detection results; a data fusion module for performing maximum a posteriori estimation based on the lidar target detection results and camera target detection results to obtain the optimal fusion azimuth angle and target measurement data, and performing data fusion to generate an observation set; and a state assessment module for inputting the observation set into a BiLSTM network for power transmission and transformation equipment state classification, and optimizing the BiLSTM network using an Adam optimizer to obtain the state assessment results of the power transmission and transformation equipment.

[0043] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements any step of the data multi-scale fusion method for digital twins of power transmission and transformation equipment as described in the first aspect of the present invention.

[0044] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the data multi-scale fusion method for digital twins of power transmission and transformation equipment as described in the first aspect of the present invention.

[0045] The beneficial effects of this invention are as follows: This invention proposes a digital twin technology based on multi-sensor fusion, employing multi-angle joint calibration technology. It uses a YOLOv8 target detector to classify laser and visual data, and a bidirectional long short-term memory (BiLSTM) model to assess the quality of detected targets. Furthermore, it uses the Adam optimizer for hyperparameter optimization, improving the accuracy of quality detection. By fusing LiDAR and camera data and employing deep learning methods for equipment quality detection and classification, it demonstrates strong detection capabilities for power transmission and transformation equipment and can be widely applied in fault detection and intelligent monitoring scenarios. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0047] Figure 1 This is a schematic diagram of the framework of the multi-scale data fusion method for digital twins of power transmission and transformation equipment in Example 1.

[0048] Figure 2 This is a schematic diagram of the BiLSTM network architecture in Example 1.

[0049] Figure 3 This is a schematic diagram of the computer device in Example 3. Detailed Implementation

[0050] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0051] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0053] Example 1

[0054] Reference Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides a multi-scale data fusion method for digital twins of power transmission and transformation equipment.

[0055] Existing multi-sensor data fusion methods mainly suffer from the following problems: Traditional RGB-D and binocular vision sensors, although low in cost and simple to deploy, have the disadvantages of limited application scope and complex computation; many labeled image datasets can be used for monitoring power equipment, but interpreting lidar point cloud data is quite difficult, mainly because it is difficult to separate many target equipment groups, sparse distribution types, and the status of equipment can only be identified from the circulation.

[0056] This application provides a method that can effectively solve the problems mentioned above. The following will describe in detail how to implement this multi-scale data fusion method for digital twins of power transmission and transformation equipment in conjunction with several embodiments.

[0057] Figure 1A schematic diagram illustrating the framework of a multi-scale data fusion method for digital twins of power transmission and transformation equipment is shown, including:

[0058] S1: Use lidar and cameras to acquire data of power transmission and transformation equipment and perform calibration to obtain calibrated data of power transmission and transformation equipment.

[0059] Preferably, the data from the power transmission and transformation equipment includes lidar data acquired by lidar and image data acquired by the camera.

[0060] Specifically, the lidar and camera lens are mounted on the same vertical line, with the camera lens positioned higher than the lidar and its vertical view covering the lidar view.

[0061] It should be noted that, since image data often exhibits tangential and mirror distortion, distortion is corrected in the real world to enhance camera imaging. Through calibration, the image pixel coordinates are matched with the angular coordinates of the object, with the LiDAR and camera lens positioned on a vertical line to ensure that the sensor angles roughly coincide. The camera is mounted on the radar, and considering that the camera's vertical position is larger than the radar's, the camera's vertical view should cover the radar's view. Joint calibration starts from the sensor's 0° positioning, and data visualization from the image and radar allows for fine-tuning of the radar and camera views.

[0062] Preferably, the present invention employs a vertically collinear arrangement of the lidar and the camera, placing the camera at a higher position and ensuring that its view covers the radar view. By ensuring that the sensor angles coincide, the complexity of multi-source data registration is significantly reduced. Utilizing the higher spatial position of the camera, a larger top-down angle can be obtained, reducing occlusion issues. The vertically collinear arrangement also simplifies coordinate system transformation and improves data fusion accuracy.

[0063] Furthermore, if the horizontal direction of the camera is greater than that of the lidar, then the lidar view is selected for calibration.

[0064] If the horizontal direction of the camera is less than or equal to that of the LiDAR, then the calibration view is selected as the overlap between the two. Considering the field of view (FOV) of multiple cameras, the horizontal direction of the camera is generally smaller than that of the LiDAR.

[0065] Preferably, obtaining the calibrated power transmission and transformation equipment data includes the following steps: selecting an equivalent angle of N in the horizontal direction to obtain N+1 calibration positions, where N is an even number.

[0066] Radar angle reflectors are set at N+1 calibration locations to obtain the radar calibration angle set, as shown in the following formula:

[0067] Ω={α1,α2,…,α N ,α N+1}

[0068] Where Ω represents the set of radar calibration angles; α N The radar calibration angle is the Nth calibration position.

[0069] Imaging the radar angle reflector at N+1 calibration locations yields the set of horizontal positions for the camera calibration pixels, as shown in the following formula:

[0070] X = {x1, x2, ..., x} N ,x N+1}

[0071] Where X is the set of horizontal positions of the camera calibration pixels, and the horizontal positions of the camera calibration pixels are defined by the pixel view of the radar corner reflector; x N The horizontal position of the camera calibration pixel at the Nth calibration position, where x N / 2+1 =0 is the X-coordinate of the image center, with the X-axis on the right being positive and the X-axis on the left being negative.

[0072] Based on the radar calibration angle set Ω and the horizontal position set X of the camera calibration pixels, the horizontal field of view and image resolution are calculated using the following formulas:

[0073]

[0074] Where, θ ν denoted as the horizontal position of the camera; W represents the horizontal pixel count, where the image resolution is defined as H×W, and H represents the vertical pixel count.

[0075] The pixel angular density is obtained by linear interpolation using the angular density of pixels in the surrounding area. The specific formula is as follows:

[0076]

[0077] Where, α x Let ρ(x) be the LiDAR angle corresponding to pixel x; ρ(x) is the pixel angular density in the interval where pixel x is located, where the degree of distortion changes continuously.

[0078] The tangential and mirror distortion of the image data acquired by the camera are corrected using pixel angular density to obtain the corrected image data.

[0079] Furthermore, when x n-1 <x<x n At that time, the pixel angular density satisfies the following formula:

[0080]

[0081] Where Δα is the angular range between calibration points; α n-1 ρ is the lidar angle at the (n-1)th calibration position;n (x) is the pixel angular density function for the nth calibration interval; x is the average pixel angular density of the nth calibration interval; n The horizontal position of the camera calibration pixel at the nth calibration position.

[0082] Specifically, when (x n-1 ,x n The angular density of pixels in the image varies uniformly, and the pixel angular density satisfies the following formula:

[0083]

[0084] Furthermore, the specific formulas for the left and right edges of the pixel image are as follows:

[0085]

[0086] in, This represents the average pixel corner density at the left edge of the pixel image; This represents the average pixel angular density at the right edge of the pixel image.

[0087] It should be noted that, based on interpolation and table lookup, the approximate coordinate mapping method is suitable for distortion correction images and image distortion. However, it requires setting more calibration points, reducing the calibration position interval, and improving the accuracy of coordinate transformation.

[0088] Preferably, this invention proposes a novel calibration method based on N+1 calibration positions. By selecting an equivalent angle in the horizontal direction, the symmetry of the calibration point distribution is ensured. The concept of pixel angular density is used for linear interpolation calculation, which effectively solves the nonlinear distortion problem in traditional methods. When within the calibration interval, a pixel angular density function is introduced, making distortion correction more accurate.

[0089] S2: Use a target detector to perform target detection on the calibrated power transmission and transformation equipment data to obtain the detection results.

[0090] Preferably, the target detection of the calibrated power transmission and transformation equipment data using a target detector to obtain the detection results includes the following steps: using a YOLOv8 target detector to perform target detection on the LiDAR data acquired by the LiDAR, identifying the power transmission and transformation equipment in the image, and obtaining the LiDAR target detection results.

[0091] The YOLOv8 object detector is used to detect objects in the corrected image data acquired by the camera, identifying power transmission and transformation equipment in the image and obtaining the camera object detection results. The YOLO algorithm has achieved excellent results in the field of computer vision, and YOLOv8 is a latest image recognition algorithm that can provide higher detection speed and accuracy compared with previous YOLO models.

[0092] S3: Based on the detection results, data fusion is performed to obtain the observation set.

[0093] Preferably, the data fusion based on the detection results to obtain the observation set includes the following steps: obtaining the spatial velocity and position of objects in the lidar target detection results, and obtaining the azimuth information and object classification in the camera target detection results.

[0094] The optimal fusion azimuth is obtained by using the object's spatial velocity, position, azimuth information, and object classification to perform maximum a posteriori estimation. The specific formula is as follows:

[0095]

[0096] Where, θ f This is the optimal azimuth angle for fusion.

[0097] The target measurement data is obtained by calculation using the optimal fusion azimuth angle, and the specific formula is as follows:

[0098]

[0099] z m =[xyv x v y "type"] T

[0100] Among them, z m For the target measurement data, where z m object type and Consistent; [xyv x v y "type" indicates the object's state; For radar points, among which and Objects detected by lidar; For image measurement, where and The objects detected by the camera.

[0101] The observation set is generated based on target measurement data and object classification, using the following formula:

[0102]

[0103] Among them, z k For the observation set.

[0104] Preferably, the present invention employs the maximum a posteriori estimation method in the data fusion stage, achieving complementary advantages by fusing the spatial information of lidar and the classification information of cameras; it introduces the concept of optimal fusion azimuth angle to improve the accuracy of multi-source data fusion; and it generates a more comprehensive observation set by combining target measurement data with object classification information.

[0105] S4: Input the observation set into the BiLSTM network to classify the state of the power transmission and transformation equipment and optimize the BiLSTM network to obtain the state assessment results of the power transmission and transformation equipment.

[0106] Preferred, such as Figure 2 The diagram shows the architecture of a BiLSTM network, which consists of two different LSTM networks: a forward I / P sequence and a backward input sequence. The results of the two LSTM networks are concatenated and used to predict the I / P sequence.

[0107] Specifically, the hidden layer HL and output of a BiLSTM network are similar in opposite aspects, as shown in the following formula:

[0108] fo t =σ(WE xfo x t +WE hfo h t-1 +WE gfo m t-1 +de fo )

[0109] in t =σ(WE xin x t +WE hin h t-1 +WE gin m t-1 +de in )

[0110] ou t =σ(WE xou x t +WE hou h t-1 +WE gou m t-1 +de ou )

[0111] M′ t =tanh(WE xm x t +WE xh +de g )

[0112] M t =fot ·m t-1 +in t ·M′ t

[0113] h t =ou t ·tanh(M t )

[0114] Where WE* is the weighting matrix; b* represents the bias of the three gating mechanisms and the I / P converter; tanh is the activation function, also known as the hyperbolic tangent function; fo t Forget gate; M′t is the gate from x t and h t-1 The new data obtained is M′t;in t For I / P gates, select to integrate data into system memory; out t Choose to filter the data in memory before outputting the result gate.

[0115] Furthermore, optimizing the BiLSTM network refers to iteratively optimizing the parameters of the BiLSTM network using the Adam optimizer to obtain an optimized BiLSTM network, thereby improving the recognition rate.

[0116] Specifically, the pseudocode for the Adam optimizer includes:

[0117] a=0.001, β1=0.9, β2=0.999, η=10 -8 (default value)

[0118] m0←0 (Initialize the first moment vector)

[0119] ν0←0 (Initialize the second moment vector)

[0120] i←0 (Initialization steps)

[0121] When Θ i Unconverged

[0122] i←i+1

[0123] (Obtain the gradient of step i)

[0124] m i ←β ι ·m i-1 +(1-β1)·g i (The first primitive moment estimate with upgrade bias)

[0125] (The second primitive moment estimate with upgrade bias)

[0126] (Calculate the first moment estimate after bias correction)

[0127] (Second raw moment estimate after bias correction)

[0128] (Update parameters)

[0129] End the loop

[0130] Return Θ i (Result parameters)

[0131] It should be noted that Adam is particularly suitable for detection and classification models, and can handle the problem of sparse model parameters and text gradients. Adam is a first-order optimization technique that can replace the classic stochastic gradient descent algorithm, iteratively updating the NN parameters based on the training data. During training, the stochastic gradient descent algorithm is used to update parameters while maintaining the learning rate, so the learning rate does not change. Adam evaluates the independent adaptive learning rate of different parameters by evaluating the gradient estimates at the first and second time steps.

[0132] Specifically, obtaining the state assessment results of power transmission and transformation equipment means inputting the observation set into the optimized BiLSTM network to obtain the state assessment results of the power transmission and transformation equipment.

[0133] It should be noted that the optimized BiLSTM network is used to identify the areas to be actually detected in visual and laser data, and generate a virtual quality map. The virtual quality map records the quality prediction results in the output, and problem boundaries can be extracted from the virtual quality map for subsequent quality inspection or defect correction.

[0134] In summary, this invention proposes a digital twin technology based on multi-sensor fusion. It employs multi-angle joint calibration technology, applies a YOLOv8 target detector to classify laser and visual data separately, uses a bidirectional long short-term memory (BiLSTM) model to assess the quality of detected targets, and uses the Adam optimizer for hyperparameter optimization, thereby improving the accuracy of quality detection. By fusing LiDAR and camera data and employing deep learning methods for equipment quality detection and classification, it demonstrates strong detection capabilities for power transmission and transformation equipment and can be widely applied in fault detection and intelligent monitoring scenarios.

[0135] Example 2, an embodiment of the present invention, provides a multi-scale data fusion system for digital twins of power transmission and transformation equipment, comprising: a data calibration module, used to acquire lidar data and image data using lidar and camera, select calibration positions and perform calculations to obtain pixel angular density and calibrate the image data to obtain corrected image data; a target detection module, used to perform target detection on lidar data and corrected image data using a YOLOv8 target detector to obtain lidar target detection results and camera target detection results; a data fusion module, used to perform maximum a posteriori estimation based on lidar target detection results and camera target detection results to obtain the optimal fusion azimuth angle and target measurement data, and perform data fusion to generate an observation set; and a state assessment module, used to input the observation set into a BiLSTM network for power transmission and transformation equipment state classification, and use the Adam optimizer to optimize the BiLSTM network to obtain the state assessment results of the power transmission and transformation equipment.

[0136] Example 3 is an embodiment of the present invention, which differs from the previous embodiment in that:

[0137] like Figure 3 As shown, if the function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0139] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0140] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0141] Example 4 is an embodiment of the present invention, which provides a multi-scale data fusion method for digital twins of power transmission and transformation equipment. In order to verify the beneficial effects of the present invention, a simulation experiment is conducted for scientific demonstration.

[0142] This example simulates the use of a LiDAR and a camera as data acquisition devices. The LiDAR is installed next to the power transmission and transformation equipment and has a 360-degree horizontal scanning capability to acquire 3D point cloud data of the environment. The camera is installed above the LiDAR, with its lens configured vertically and collinearly with the LiDAR, and the camera's field of view covers the LiDAR's field of view. Spatial alignment of the data between the LiDAR and the camera is achieved through precise calibration to ensure that there is no distortion in the angle and field of view of both.

[0143] First, by vertically collinearly configuring the LiDAR and the camera, multiple positions are set using calibration equipment, and calibration is performed using the calibration pixel information of the LiDAR angle reflector and the camera to ensure that the LiDAR and the camera have the same viewing angle. The calibration process includes imaging the LiDAR angle reflector at different calibration positions to obtain image coordinates and radar angle reflection data, and then correcting image distortion and performing pixel angular density interpolation correction.

[0144] Secondly, after data calibration, the YOLOv8 target detector is used to perform target detection on the data collected by the LiDAR and camera. Target detection is performed on the LiDAR data to extract the spatial position and velocity information of the device; target detection is performed on the camera image data to obtain the azimuth angle and category information of the object; the detection results of the LiDAR and camera are fused, and the azimuth angle of the fused device is evaluated by maximum a posteriori estimation (MAP), and the target measurement data is calculated to finally generate the fused observation set; finally, the observation set is input into a BiLSTM network for state classification, and the state evaluation result of the device is output. In this process, the network parameters are optimized by the Adam optimizer to improve the recognition accuracy and precision. Table 1 shows the experimental data of this invention in three cases: LiDAR and camera fusion, LiDAR alone, and camera alone.

[0145] Table 1 Experimental data of the present invention under different devices

[0146] Experimental subjects Equipment positioning accuracy (m) Detection speed (m / s) State classification accuracy (%) LiDAR and camera fusion 0.08 0.95 98 standalone lidar 0.15 0.80 90 standalone camera 0.12 0.75 85

[0147] As can be seen from the table above, the device location accuracy after data fusion from LiDAR and camera is 0.08 meters, which is a significant improvement compared to using LiDAR and camera alone. This indicates that the accuracy of device positioning is greatly improved after fusing multi-source data. In terms of detection speed, the detection speed of the method of this invention is 0.95 meters per second, which is also improved compared to using LiDAR and camera alone, indicating that data fusion improves the real-time performance of device status monitoring. In terms of status classification accuracy, the method of this invention also shows a significant improvement, reaching 98%, while the accuracy of using LiDAR and camera alone is 90% and 85%, respectively. This shows that through the fusion of multi-source data, the BiLSTM network can more accurately identify the status of the device, improving the reliability and intelligence level of the monitoring system.

[0148] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multi-scale data fusion method for digital twins of power transmission and transformation equipment, characterized in that: include: Data on power transmission and transformation equipment is acquired and calibrated using lidar and cameras to obtain calibrated data on the power transmission and transformation equipment. The calibrated power transmission and transformation equipment data is subjected to target detection using a target detector to obtain detection results. Based on the detection results, data fusion is performed to obtain an observation set; The observation set is input into the BiLSTM network for power transmission and transformation equipment status classification and the BiLSTM network is optimized to obtain the status assessment results of the power transmission and transformation equipment. The data fusion based on the detection results to obtain the observation set includes the following steps: The spatial velocity and position of the object are obtained from the detection results of the lidar, and the azimuth information and object classification are obtained from the detection results of the camera. The optimal fusion azimuth is obtained by using the object's spatial velocity, position, azimuth information, and object classification to perform maximum a posteriori estimation. The specific formula is as follows: Where, θ f The optimal fusion azimuth angle; The target measurement data is obtained by calculation using the optimal fusion azimuth angle, as shown in the following formula: Among them, z m For target measurement data; For radar points; For image measurement; An observation set is generated based on the target measurement data and the object classification, using the following formula: Among them, z k For the observation set; The BiLSTM network consists of two different LSTM networks, including a forward I / P sequence and a backward input sequence; The hidden layer HL and output of the BiLSTM network are similar in opposite aspects, as shown in the following formula: fo t =σ(WE xfo x t +WE hfo h t-1 +WE gfo m t-1 +de fo ) in t =σ(WE xin x t +WE hin h t-1 +WE gin m t-1 +de in ) ou t =σ(WE xou x t +WE hou h t-1 +WE gou m t-1 +de ou ) M′ t = tanh(WE) xm x t +WE xh +deg) M t =fo t ·m t-1 +in t ·M′ t h t =ou t tanh(M) t ) Where WE* is the weighting matrix; b* represents the bias of the three gating mechanisms and the I / P converter; tanh is the activation function; fo t Forgotten Gate; M ′ t is from x t and h t-1 New data M obtained ′ t;in t Choose to integrate data into system memory for I / P gates; t Choose to filter the data in memory before outputting the result gate.

2. The multi-scale data fusion method for digital twins of power transmission and transformation equipment as described in claim 1, characterized in that: The power transmission and transformation equipment data includes lidar data acquired by lidar and image data acquired by camera; The lidar and camera lens are mounted on the same vertical line, with the camera lens positioned vertically higher than the lidar and its vertical view covering the lidar view.

3. The multi-scale data fusion method for digital twins of power transmission and transformation equipment as described in claim 2, characterized in that: The process of obtaining the calibrated power transmission and transformation equipment data includes the following steps: By selecting an equivalent angle of N in the horizontal direction, N+1 calibration positions are obtained; Radar angle reflectors are set at the N+1 calibration locations to obtain the radar calibration angle set, as shown in the following formula: Ω={α1,α2,…,α N ,a N+1 } Where Ω represents the set of radar calibration angles; α N The radar calibration angle for the Nth calibration position; Imaging the radar angle reflector at the N+1 calibration positions yields the set of horizontal positions of the camera calibration pixels, as shown in the following formula: X={x1,x2,…,x N , x N+1 } Where X is the set of horizontal positions of the camera calibration pixels; x N Set the horizontal position of the camera calibration pixel at the Nth calibration position; Based on the radar calibration angle set Ω and the horizontal position set X of the camera calibration pixels, the horizontal field of view and image resolution are calculated using the following formulas: Where, θ ν represents the horizontal position of the camera; W represents the horizontal pixel count. The pixel angular density is obtained by linear interpolation using the angular density of pixels in the surrounding area. The specific formula is as follows: Where, α x Let be the lidar angle corresponding to pixel x; ρ(x) is the pixel angular density of the interval where pixel x is located; The pixel angular density is used to correct tangential and mirror distortion in the image data acquired by the camera, resulting in corrected image data.

4. The multi-scale data fusion method for digital twins of power transmission and transformation equipment as described in claim 3, characterized in that: When x n-1 <x<x n When the pixel angular density satisfies the following formula: Where Δα is the angular range between calibration points; α n-1 ρ is the lidar angle at the (n-1)th calibration position; n (x) is the pixel angular density function for the nth calibration interval; x is the average pixel angular density of the nth calibration interval; n Calculate the horizontal position of the camera calibration pixel at the nth calibration position; When (x) n-1 ,x n The angular density of pixels in a given region varies uniformly, and the pixel angular density satisfies the following formula: The specific formulas for the left and right edges of a pixel image are as follows: in, This represents the average pixel corner density at the left edge of the pixel image; This represents the average pixel angular density at the right edge of the pixel image.

5. The multi-scale data fusion method for digital twins of power transmission and transformation equipment as described in claim 4, characterized in that: The process of performing target detection on the calibrated power transmission and transformation equipment data using a target detector to obtain the detection results includes the following steps: The YOLOv8 target detector is used to perform target detection on the lidar data acquired by the lidar, and the detection results of the lidar are obtained. The YOLOv8 object detector is used to perform object detection on the corrected image data acquired by the camera, and the camera's detection results are obtained.

6. A multi-scale data fusion system for digital twins of power transmission and transformation equipment, based on the multi-scale data fusion method for digital twins of power transmission and transformation equipment as described in any one of claims 1 to 5, characterized in that: include, The data calibration module is used to acquire LiDAR data and image data using LiDAR and camera, select calibration positions and perform calculations to obtain pixel angular density, and calibrate the image data to obtain corrected image data. The target detection module is used to perform target detection on lidar data and corrected image data using the YOLOv8 target detector, and obtain the detection results of lidar and camera. The data fusion module is used to perform maximum a posteriori estimation based on the detection results of lidar and camera, obtain the optimal fusion azimuth angle and target measurement data, and perform data fusion to generate an observation set; The state assessment module is used to input the observation set into the BiLSTM network for state classification of power transmission and transformation equipment, and to optimize the BiLSTM network using the Adam optimizer to obtain the state assessment results of the power transmission and transformation equipment.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the data multi-scale fusion method for digital twins of power transmission and transformation equipment as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the data multi-scale fusion method for digital twins of power transmission and transformation equipment as described in any one of claims 1 to 5.

Images