Data multi-scale fusion method for digital twinning of power transmission and transformation equipment

Abstract

The invention discloses a data multi-scale fusion method for digital twinning of power transmission and transformation equipment, which relates to the technical field of power equipment and comprises the following steps: acquiring data of the power transmission and transformation equipment by using a laser radar and a camera and calibrating the data to obtain the calibrated data of the power transmission and transformation equipment; performing target detection on the calibrated power transmission and transformation equipment data by using a target detector to obtain a detection result; performing data fusion based on the detection result to obtain an observation set; and inputting the observation set into a BiLSTM network to perform state classification of the power transmission and transformation equipment, and optimizing the BiLSTM network to obtain a state evaluation result of the power transmission and transformation equipment. According to the method, a multi-angle joint calibration technology is adopted, a YOLOv8 target detector is respectively applied to carry out image classification on laser and visual data, a bidirectional long-short term memory BiLSTM model is adopted to carry out quality evaluation on a detected target, and an Adam optimizer is used to carry out hyper-parameter optimization, so that the accuracy of quality detection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power equipment, and in particular to a multi-scale data fusion method for digital twins of power transmission and transformation equipment. Background Art

[0002] Power transmission and transformation equipment is prone to failure due to its continuous operation under different working conditions. Any failure in these devices may lead to subsequent failures in other equipment and ultimately paralyze the entire power system. In order to ensure the optimal state of the system, these power equipment must be continuously monitored to improve the overall efficiency and functionality of the system. Effective condition monitoring technology can improve the reliability and service life of equipment. Any electrical equipment requires regular maintenance operations due to its continuous operation, which is an essential part of its life cycle. At present, the advanced technologies of power equipment fault monitoring technology include inverse problem theory, image processing and computer vision methods, advanced signal processing technology, model-based prediction and health management, digital twin methods, and artificial intelligence methods.

[0003] However, traditional mathematical models or single sensor monitoring methods cannot adapt to the current technological development needs of power transmission and transformation equipment. They need to be fused based on multi-sensor data and combined with deep learning methods. By training models based on historical data and real-time observations, specific fault modes can be identified and deviations from normal real-time operating conditions can be detected to minimize grid failures and ensure smooth interaction with the grid. Regarding current multi-sensor data fusion technology, traditional RGB-D and binocular vision sensors, although low cost and simple deployment, have the disadvantages of small application range and complex calculations. Many labeled image datasets can be used for monitoring power equipment, but interpreting lidar point cloud data is quite difficult, mainly because it is difficult to separate many target equipment groups, sparse distribution types, and the status of equipment can only be identified from the circulation. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is how to provide a method for the operating status of power transmission and transformation equipment, set up a digital twin framework based on laser and visual data, study the fusion and optimization of multimodal data, and use deep learning methods to complete the status assessment, fault prediction and maintenance of power transmission and transformation equipment.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In the first aspect, an embodiment of the present invention provides a multi-scale data fusion method for digital twins of power transmission and transformation equipment, which includes using a lidar and a camera to acquire power transmission and transformation equipment data and calibrate it to obtain calibrated power transmission and transformation equipment data; using a target detector to perform target detection on the calibrated power transmission and transformation equipment data to obtain a detection result; performing data fusion based on the detection result to obtain an observation set; inputting the observation set into a BiLSTM network to classify the status of the power transmission and transformation equipment and optimizing the BiLSTM network to obtain a status evaluation result of the power transmission and transformation equipment.

[0008] As a preferred solution of the multi-scale data fusion method for digital twins of power transmission and transformation equipment described in the present invention, the power transmission and transformation equipment data includes lidar data obtained by the lidar and image data obtained by the camera; the lidar and camera lenses are installed on the same vertical line, the vertical position of the camera lens is higher than the lidar and the vertical view of the camera lens covers the lidar view.

[0009] As a preferred solution of the data multi-scale fusion method for digital twins of power transmission and transformation equipment described in the present invention, the method of obtaining the calibrated power transmission and transformation equipment data includes the following steps: selecting an equivalent angle of N in the horizontal direction to obtain N+1 calibration positions; setting a radar angle reflector at the N+1 calibration position to obtain a radar calibration angle set. The specific formula is as follows:

[0010] Ω={α1,α2,…,α N ,α N+1}

[0011] Where Ω is the radar calibration angle set; α N is the radar calibration angle at the Nth calibration position; the radar angle reflector is imaged at the N+1 calibration position to obtain a horizontal position set of camera calibration pixels, and the specific formula is as follows:

[0012] X={x1,x2,…,x N ,x N+1}

[0013] Where X is the horizontal position set of camera calibration pixels; x N is the horizontal position of the camera calibration pixel at the Nth calibration position; according to the radar calibration angle set Ω and the horizontal position set X of the camera calibration pixel, the horizontal field of view angle and image resolution are calculated. The specific formula is as follows:

[0014]

[0015] Among them, θ νis the horizontal position of the camera; W is the number of horizontal pixels; the pixel angular density is obtained by linearly interpolating the pixels using the angular density of the pixels in the nearby area. The specific formula is as follows:

[0016]

[0017] Among them, α x is the laser radar angle corresponding to pixel x; ρ(x) is the pixel angle density of the interval where pixel x is located; the image data acquired by the camera is corrected for tangential distortion and mirror distortion using the pixel angle density to obtain corrected image data.

[0018] As a preferred solution of the data multi-scale fusion method for digital twin of power transmission and transformation equipment described in the present invention, when x n-1 <x<x n When , the pixel angle density satisfies the following formula:

[0019]

[0020] Where Δα is the angle range between the calibration points; α n-1 is the laser radar angle of the n-1th calibration position; ρ n (x) is the pixel angle density function of the nth calibration interval; is the average value of pixel angle density in the nth calibration interval; x n is the horizontal position of the camera calibration pixel at the nth calibration position; when (x n-1 ,x n ] changes uniformly, and the pixel angular density satisfies the following formula:

[0021]

[0022] The specific formulas for the left and right edges of a pixel image are as follows:

[0023]

[0024] in, is the average value of the pixel angular density at the left edge of the pixel image; is the average value of the pixel angular density at the right edge of the pixel image.

[0025] As a preferred scheme of the multi-scale data fusion method for digital twins of power transmission and transformation equipment described in the present invention, wherein: using a target detector to perform target detection on the calibrated power transmission and transformation equipment data, and obtaining the detection result includes the following steps: using a YOLOv8 target detector to perform target detection on the lidar data acquired by the lidar to obtain the lidar target detection result; using a YOLOv8 target detector to perform target detection on the corrected image data acquired by the camera to obtain the camera target detection result.

[0026] As a preferred solution of the data multi-scale fusion method for digital twins of power transmission and transformation equipment described in the present invention, data fusion is performed based on the detection results to obtain an observation set, which includes the following steps: obtaining the spatial velocity and position of the object in the laser radar target detection result, and obtaining the azimuth information and object classification in the camera target detection result; using the spatial velocity, position, azimuth information and object classification of the object to perform maximum a posteriori estimation evaluation to obtain the optimal fusion azimuth, and the specific formula is as follows:

[0027]

[0028] Among them, θ f is the optimal fusion azimuth; the target measurement data is obtained by calculation using the optimal fusion azimuth, and the specific formula is as follows:

[0029]

[0030] z m =[xyv x v y "type"] T

[0031] Among them, z m Measure data for the target; For radar points; For image measurement; an observation set is generated based on the target measurement data and the object classification, and the specific formula is as follows:

[0032]

[0033] Among them, z k is the observation set.

[0034] As a preferred solution of the data multi-scale fusion method for digital twins of power transmission and transformation equipment described in the present invention, the BiLSTM network is composed of two different LSTM networks, including a forward I / P sequence and a reverse input sequence; the hidden layer HL and the output of the BiLSTM network are similar in opposite aspects, and the specific formula is as follows:

[0035] fo t =σ(WE xfo x t +WE hfo h t-1 +WE gfo m t-1 +de fo )

[0036] in t =σ(WE xin x t +WE hin h t-1 +WE gin m t-1 +de in )

[0037] ou t =σ(WE xou x t +WE hou h t-1 +WE gou m t-1 +de ou )

[0038] M′ t =tanh(WE xm x t +WE xh +de g )

[0039] M t =fo t ·m t-1 +in t ·M′ t

[0040] h t =ou t ·tanh(M t )

[0041] Among them, WE* is the weighting matrix; b* is the bias of the three gating mechanisms and the I / P converter; tanh is the activation function; fo t is the forget gate; M′t is the t and h t-1 The new data obtained M′t;in t Select for I / P gate to integrate data into system memory; out t Select the result gate to filter the data in the memory and then output it.

[0042] In the second aspect, in order to further solve the safety problems existing in power equipment, the present invention provides a data multi-scale fusion system for digital twins of power transmission and transformation equipment, which includes: a data calibration module, which is used to obtain lidar data and image data using lidar and camera, select calibration positions and perform calculations, obtain pixel angular density and calibrate image data, and obtain corrected image data; a target detection module, which is used to perform target detection on lidar data and corrected image data using YOLOv8 target detector, and obtain lidar target detection results and camera target detection results; a data fusion module, which is used to perform maximum a posteriori estimation evaluation based on the lidar target detection results and camera target detection results, obtain the optimal fusion azimuth and target measurement data, perform data fusion, and generate an observation set; a state evaluation module, which is used to input the observation set into a BiLSTM network for power transmission and transformation equipment state classification, and optimize the BiLSTM network using an Adam optimizer to obtain a state evaluation result of the power transmission and transformation equipment.

[0043] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, it implements any step of the multi-scale data fusion method for digital twins of power transmission and transformation equipment as described in the first aspect of the present invention.

[0044] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the multi-scale data fusion method for digital twins of power transmission and transformation equipment as described in the first aspect of the present invention.

[0045] Beneficial effects of the invention: The present invention proposes a digital twin technology based on multi-sensor fusion, adopts multi-angle joint calibration technology, applies YOLOv8 target detector to perform image classification on laser and visual data respectively, and adopts bidirectional long short-term memory BiLSTM model to perform quality assessment on detected targets, and uses Adam optimizer to perform hyperparameter optimization, thereby improving the accuracy of quality detection; by fusing lidar and camera data, and adopting deep learning methods to perform equipment quality detection and classification, it exhibits strong detection capabilities for power transmission and transformation equipment, and can be widely used in fault detection and intelligent monitoring scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0047] Figure 1 This is a schematic diagram of the framework of the multi-scale data fusion method for digital twins of power transmission and transformation equipment in Example 1.

[0048] Figure 2 Schematic diagram of the architecture of the BiLSTM network in Example 1.

[0049] Figure 3 This is a schematic diagram of the structure of the computer device in Example 3. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0053] Example 1

[0054] Reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides a multi-scale data fusion method for digital twins of power transmission and transformation equipment.

[0055] Existing multi-sensor data fusion methods have the following main problems: traditional RGB-D and binocular vision sensors, although low-cost and easy to deploy, have the disadvantages of small application range and complex calculation; many labeled image datasets can be used for monitoring power equipment, but interpreting lidar point cloud data is quite difficult, mainly because of the difficulty in separating many target equipment groups, sparse distribution types and the ability to only identify the status of equipment from the circulation.

[0056] The present application provides an effective solution to the above-mentioned problems. Next, we will combine multiple embodiments to elaborate on how to implement the multi-scale data fusion method for digital twins of power transmission and transformation equipment.

[0057] Figure 1The schematic diagram of the framework of the multi-scale data fusion method for digital twins of power transmission and transformation equipment is shown, including:

[0058] S1: Use laser radar and camera to obtain power transmission and transformation equipment data and calibrate it to obtain calibrated power transmission and transformation equipment data.

[0059] Preferably, the power transmission and transformation equipment data includes lidar data acquired by a lidar and image data acquired by a camera.

[0060] Specifically, the laser radar and the camera lenses are installed on the same vertical line, the vertical position of the camera lens is higher than the laser radar and the vertical view of the camera lens covers the laser radar view.

[0061] It should be noted that since image data usually has tangential distortion and mirror distortion, the distortion is modified in the real world to enhance camera imaging; through calibration, the image pixel coordinates are matched with the angular coordinates of the object, where the lidar and camera lenses are located on a vertical line to ensure that the angles of the sensors roughly coincide; the camera is mounted on the radar, and considering that the vertical position of the camera is larger than that of the radar, the vertical view of the camera should cover the radar view; the joint calibration starts with the 0° positioning of the sensor, and the data visualization from the image and radar allows fine-tuning of the radar and camera views.

[0062] Preferably, the present invention adopts a vertically collinear arrangement of the laser radar and the camera, so that the camera is located at a higher position and the view covers the radar view, and the complexity of multi-source data alignment is significantly reduced by ensuring that the sensor angles overlap. By utilizing the higher spatial position of the camera, a larger bird's-eye view can be obtained, reducing occlusion problems. The vertical collinear arrangement also simplifies the coordinate system conversion and improves the data fusion accuracy.

[0063] Furthermore, if the camera is larger than the lidar in horizontal direction, the radar view is selected for calibration.

[0064] If the horizontal direction of the camera is less than or equal to that of the lidar, the calibration view is selected as the overlap of the two, taking into account the field of view FOV of multiple cameras, the horizontal direction of the camera is generally smaller than that of the lidar.

[0065] Preferably, obtaining the calibrated data of the power transmission and transformation equipment includes the following steps: selecting an equivalent angle of N in the horizontal direction, and then obtaining N+1 calibration positions, wherein N is an even number.

[0066] Set radar angle reflectors at N+1 calibration positions to obtain a radar calibration angle set. The specific formula is as follows:

[0067] Ω={α1,α2,…,α N ,α N+1}

[0068] Where Ω is the radar calibration angle set; α N is the radar calibration angle of the Nth calibration position.

[0069] The radar angle reflector is imaged at N+1 calibration positions to obtain the horizontal position set of the camera calibration pixels. The specific formula is as follows:

[0070] X={x1,x2,…,x N ,x N+1}

[0071] Where X is the set of horizontal positions of camera calibration pixels, where the horizontal positions of camera calibration pixels are defined by the pixel view of the radar corner reflector; N is the horizontal position of the camera calibration pixel at the Nth calibration position, where x N / 2+1 =0 is the X coordinate of the center of the image, the right side of the X axis is positive and the left side is negative.

[0072] According to the radar calibration angle set Ω and the horizontal position set X of the camera calibration pixels, the horizontal field of view angle and image resolution are calculated. The specific formula is as follows:

[0073]

[0074] Among them, θ ν is the horizontal position of the camera; W is the number of horizontal pixels, where the image resolution is defined as H×W, and H is the number of vertical pixels.

[0075] The pixel angle density is obtained by linearly interpolating the pixels using the angle density of the pixels in the nearby area. The specific formula is as follows:

[0076]

[0077] Among them, α x is the lidar angle corresponding to pixel x; ρ(x) is the pixel angle density in the interval where pixel x is located, where the degree of distortion changes continuously.

[0078] The pixel angular density is used to correct the tangential distortion and mirror distortion of the image data acquired by the camera to obtain the corrected image data.

[0079] Furthermore, when x n-1 <x<x n When , the pixel angular density satisfies the following formula:

[0080]

[0081] Where Δα is the angle range between the calibration points; α n-1 is the laser radar angle of the n-1th calibration position; ρn (x) is the pixel angle density function of the nth calibration interval; is the average value of pixel angle density in the nth calibration interval; x n Horizontal position of the camera calibration pixel for the nth calibration position.

[0082] Specifically, when (x n-1 ,x n ] changes uniformly, and the pixel angular density satisfies the following formula:

[0083]

[0084] Furthermore, the specific formulas for the left and right edges of the pixel image are as follows:

[0085]

[0086] in, is the average value of the pixel angular density at the left edge of the pixel image; is the average value of the pixel angular density at the right edge of the pixel image.

[0087] It should be noted that the approximate coordinate mapping method is suitable for distortion-corrected images and image distortion based on interpolation and table lookup. At the same time, more calibration points need to be set to reduce the calibration position interval and improve the accuracy of coordinate transformation.

[0088] Preferably, the present invention proposes a new calibration method based on N+1 calibration positions, which ensures the symmetry of the distribution of calibration points by selecting equivalent angles in the horizontal direction; uses the concept of pixel angle density to perform linear interpolation calculations, effectively solving the nonlinear distortion problem in traditional methods; and introduces a pixel angle density function when in the calibration interval, making the distortion correction more accurate.

[0089] S2: Use the target detector to perform target detection on the calibrated power transmission and transformation equipment data to obtain the detection results.

[0090] Preferably, the target detector is used to perform target detection on the calibrated power transmission and transformation equipment data, and obtaining the detection result includes the following steps: using the YOLOv8 target detector to perform target detection on the lidar data acquired by the lidar, identifying the power transmission and transformation equipment in the picture, and obtaining the lidar target detection result.

[0091] The YOLOv8 target detector is used to perform target detection on the calibrated image data obtained by the camera, identify the power transmission and transformation equipment in the picture, and obtain the camera target detection result. The YOLO algorithm has achieved outstanding results in the field of computer vision, and YOLOv8 is the latest image recognition algorithm. Compared with the previous YOLO model, it can provide higher detection speed and accuracy.

[0092] S3: Perform data fusion based on the detection results to obtain the observation set.

[0093] Preferably, data fusion is performed based on the detection results to obtain an observation set, which includes the following steps: obtaining the spatial speed and position of the object in the laser radar target detection results, and obtaining the azimuth information and object classification in the camera target detection results.

[0094] The maximum a posteriori estimation is performed using the spatial velocity, position, azimuth information of the object and the object classification to obtain the optimal fusion azimuth. The specific formula is as follows:

[0095]

[0096] Among them, θ f is the optimal fusion azimuth.

[0097] The target measurement data is obtained by calculation using the optimal fusion azimuth. The specific formula is as follows:

[0098]

[0099] z m =[xyv x v y "type"] T

[0100] Among them, z m is the target measurement data, where z m The object type and consistent; [xyv x v y "type" is the object status; is the radar point, where and Objects discovered by LiDAR; is the image measurement, where and Objects detected by the camera.

[0101] The observation set is generated based on the target measurement data and object classification. The specific formula is as follows:

[0102]

[0103] Among them, z k is the observation set.

[0104] Preferably, the present invention adopts the maximum a posteriori estimation method in the data fusion link, and realizes complementary advantages by fusing the spatial information of the lidar and the classification information of the camera; introduces the concept of the optimal fusion azimuth angle to improve the accuracy of multi-source data fusion; and generates a more comprehensive observation set by combining the target measurement data with the object classification information.

[0105] S4: Input the observation set into the BiLSTM network to classify the status of the power transmission and transformation equipment and optimize the BiLSTM network to obtain the status evaluation results of the power transmission and transformation equipment.

[0106] Preferably, Figure 2 The figure shows the architecture diagram of the BiLSTM network. The BiLSTM network is composed of two different LSTM networks, including a forward I / P sequence and a reverse input sequence. The results of the two LSTM networks are connected and used to predict the I / P sequence.

[0107] Specifically, the hidden layer HL and the output of the BiLSTM network are similar in opposite aspects. The specific formula is as follows:

[0108] fo t =σ(WE xfo x t +WE hfo h t-1 +WE gfo m t-1 +de fo )

[0109] in t =σ(WE xin x t +WE hin h t-1 +WE gin m t-1 +de in )

[0110] ou t =σ(WE xou x t +WE hou h t-1 +WE gou m t-1 +de ou )

[0111] M′ t =tanh(WE xm x t +WE xh +de g )

[0112] M t =fot ·m t-1 +in t ·M′ t

[0113] h t =ou t ·tanh(M t )

[0114] Among them, WE* is the weighting matrix; b* is the bias of the three gating mechanisms and the I / P converter; tanh is the activation function, also known as the hyperbolic tangent function; fo t is the forget gate; M′t is the t and h t-1 The new data obtained M′t;in t Select for I / P gate to integrate data into system memory; out t Select the result gate to filter the data in the memory and then output it.

[0115] Furthermore, optimizing the BiLSTM network refers to iteratively optimizing the parameters of the BiLSTM network using the Adam optimizer to obtain an optimized BiLSTM network, thereby improving the recognition rate.

[0116] Specifically, the pseudo code of the Adam optimizer includes:

[0117] a=0.001, β1=0.9, β2=0.999, η=10 -8 (default value)

[0118] m0←0 (initialize the first moment vector)

[0119] ν0←0 (initialize the second moment vector)

[0120] i←0 (initialization step)

[0121] When θ i Not converged

[0122] i←i+1

[0123] (Get the gradient of step i)

[0124] m i ←β ι ·m i-1 +(1-β1)·g i (Upgrading biased first raw moment estimates)

[0125] (Upgrading biased second raw moment estimates)

[0126] (Compute bias-corrected first moment estimates)

[0127] (Bias-corrected second raw moment estimate)

[0128] (Update parameters)

[0129] End the loop

[0130] Return to Θ i (result parameter)

[0131] It should be noted that Adam is particularly suitable for detection and classification models, and can handle model parameter and text gradient sparsity problems; Adam is a first-order optimization technique that can replace the classic stochastic gradient descent algorithm and iteratively update NN parameters based on training data; during the training process, the stochastic gradient descent algorithm is used to maintain the learning rate to update the parameters, and the learning rate will not change; Adam evaluates the independent adaptive learning rates of different parameters by evaluating the first and second moment estimates of the gradient.

[0132] Specifically, obtaining the status evaluation result of the power transmission and transformation equipment refers to inputting the observation set into the optimized BiLSTM network to obtain the status evaluation result of the power transmission and transformation equipment.

[0133] It should be noted that the optimized BiLSTM network is used to identify the area to be actually inspected in the visual and laser data, and to generate a virtual quality map, which records the quality prediction results in the output. The problem boundaries can then be extracted from the virtual quality map for subsequent quality inspection or defect correction.

[0134] In summary, the present invention proposes a digital twin technology based on multi-sensor fusion, adopts multi-angle joint calibration technology, applies YOLOv8 target detector to perform image classification on laser and visual data respectively, and adopts bidirectional long short-term memory BiLSTM model to perform quality assessment on detected targets, and uses Adam optimizer to perform hyperparameter optimization, thereby improving the accuracy of quality detection; by fusing lidar and camera data, and adopting deep learning methods to perform equipment quality detection and classification, it shows strong detection capability for power transmission and transformation equipment, and can be widely used in fault detection and intelligent monitoring scenarios.

[0135] Embodiment 2 is an embodiment of the present invention, which provides a data multi-scale fusion system for digital twins of power transmission and transformation equipment, including: a data calibration module, which is used to obtain lidar data and image data using lidar and a camera, select a calibration position and perform calculations, obtain pixel angular density and calibrate the image data, and obtain corrected image data; a target detection module, which is used to perform target detection on the lidar data and the corrected image data using a YOLOv8 target detector, and obtain lidar target detection results and camera target detection results; a data fusion module, which is used to perform maximum a posteriori estimation evaluation based on the lidar target detection results and the camera target detection results, obtain the optimal fusion azimuth and target measurement data, and perform data fusion to generate an observation set; a state evaluation module, which is used to input the observation set into a BiLSTM network for power transmission and transformation equipment state classification, and optimize the BiLSTM network using an Adam optimizer to obtain a state evaluation result of the power transmission and transformation equipment.

[0136] Embodiment 3 is an embodiment of the present invention, which is different from the previous embodiment in that:

[0137] like Figure 3 As shown, if the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0138] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0139] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0140] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0141] Example 4 is an embodiment of the present invention, which provides a multi-scale data fusion method for digital twins of power transmission and transformation equipment. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0142] This example simulates the use of laser radar and camera as data acquisition equipment. The laser radar is installed next to the power transmission and transformation equipment and has a 360-degree horizontal scanning capability to obtain three-dimensional point cloud data of the environment. The camera is installed above the laser radar, with the lens and the laser radar in a vertical collinear configuration, and the camera's field of view covers the laser radar's field of view. Between the laser radar and the camera, spatial alignment of the data is achieved through precise calibration to ensure that the angles and fields of view of the two are distortion-free.

[0143] Firstly, the laser radar and camera are configured vertically in a collinear manner, and multiple positions are set up using calibration equipment. Calibration is performed using the calibration pixel information of the laser radar angle reflector and the camera to ensure that the viewing angles of the laser radar and the camera are consistent. The calibration process includes imaging the laser radar angle reflectors at different calibration positions, obtaining image coordinates and radar angle reflection data, and then correcting image distortion and performing interpolation correction of pixel angular density.

[0144] Secondly, after the data calibration is completed, the YOLOv8 target detector is used to perform target detection on the data collected by the lidar and the camera, wherein target detection is performed on the lidar data to extract the spatial position and speed information of the device; target detection is performed on the camera image data to obtain the azimuth and category information of the object; the results of the lidar and camera detection are fused, the fused azimuth is evaluated by the maximum a posteriori estimation MAP, and the target measurement data is calculated to finally generate a fused observation set; finally, the observation set is input into the BiLSTM network for state classification, and the state evaluation result of the device is output. In this process, the network parameters are optimized by the Adam optimizer to improve the recognition precision and accuracy. Table 1 shows the experimental data of the present invention in three cases: lidar and camera fusion, single lidar, and single camera.

[0145] Table 1 Experimental data of the present invention under different equipment

[0146] Experimental subjects Equipment location accuracy (m) Detection speed (m / s) State classification accuracy (%) LiDAR and camera fusion 0.08 0.95 98 Single LiDAR 0.15 0.80 90 Single camera 0.12 0.75 85

[0147] It can be seen from the above table that the equipment position accuracy after the fusion of laser radar and camera data is 0.08 meters, which is significantly improved compared with the accuracy of using laser radar and camera alone, indicating that the accuracy of equipment positioning is greatly improved after fusing multi-source data; in terms of detection speed, the detection speed of the method of the present invention is 0.95 meters per second, which is improved compared with the use of laser radar and camera alone, indicating that data fusion improves the real-time performance of equipment status monitoring; in terms of status classification accuracy, the method of the present invention also has a significant improvement in status classification accuracy, reaching 98%, while the accuracy of using laser radar and camera alone is 90% and 85% respectively, indicating that through the fusion of multi-source data, the BiLSTM network can more accurately identify the status of the equipment, thereby improving the reliability and intelligence level of the monitoring system.

[0148] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-scale data fusion method for digital twins of power transmission and transformation equipment, characterized by: include: Use laser radar and camera to obtain and calibrate the data of power transmission and transformation equipment to obtain the calibrated data of power transmission and transformation equipment; Using a target detector to perform target detection on the calibrated power transmission and transformation equipment data to obtain a detection result; Perform data fusion based on the detection results to obtain an observation set; The observation set is input into the BiLSTM network to classify the state of the power transmission and transformation equipment and the BiLSTM network is optimized to obtain the state evaluation result of the power transmission and transformation equipment.

2. The multi-scale data fusion method for digital twins of power transmission and transformation equipment according to claim 1, characterized in that: The power transmission and transformation equipment data includes laser radar data acquired by the laser radar and image data acquired by the camera; The lidar and camera lenses are mounted on the same vertical line, the camera lens' vertical position is higher than the lidar and the camera lens' vertical view covers the lidar view.

3. The multi-scale data fusion method for digital twins of power transmission and transformation equipment according to claim 2, characterized in that: The step of obtaining the calibrated power transmission and transformation equipment data comprises the following steps: Select an equivalent angle of N in the horizontal direction to obtain N+1 calibration positions; Radar angle reflectors are set at the N+1 calibration positions to obtain a radar calibration angle set. The specific formula is as follows: Ω={α1,α2,…,α N ,a N+1 } Where Ω is the radar calibration angle set; α N is the radar calibration angle of the Nth calibration position; The radar angle reflector is imaged at the N+1 calibration positions to obtain a set of horizontal positions of camera calibration pixels. The specific formula is as follows: X={x1,x2,…,x N , x N+1 } Where X is the horizontal position set of camera calibration pixels; x N The horizontal position of the camera calibration pixel for the Nth calibration position; According to the radar calibration angle set Ω and the horizontal position set X of the camera calibration pixels, the horizontal field of view angle and image resolution are calculated. The specific formula is as follows: Among them, θ ν is the horizontal position of the camera; W is the number of horizontal pixels; The pixel angle density is obtained by linearly interpolating the pixels using the angle density of the pixels in the nearby area. The specific formula is as follows: Among them, α x is the laser radar angle corresponding to pixel x; ρ(x) is the pixel angle density of the interval where pixel x is located; The pixel angular density is used to correct the tangential distortion and mirror distortion of the image data acquired by the camera to obtain corrected image data.

4. The multi-scale data fusion method for digital twins of power transmission and transformation equipment according to claim 3, characterized in that: When x n-1 <x<x n When , the pixel angle density satisfies the following formula: Where Δα is the angle range between the calibration points; α n-1 is the laser radar angle of the n-1th calibration position; ρ n (x) is the pixel angle density function of the nth calibration interval; is the average value of pixel angle density in the nth calibration interval; x n The horizontal position of the camera calibration pixel for the nth calibration position; When (x n-1 ,x n ] changes uniformly, and the pixel angular density satisfies the following formula: The specific formulas for the left and right edges of a pixel image are as follows: in, is the average value of the pixel angular density at the left edge of the pixel image; is the average value of the pixel angular density at the right edge of the pixel image.

5. The multi-scale data fusion method for digital twins of power transmission and transformation equipment according to claim 4, characterized in that: Using a target detector to perform target detection on the calibrated power transmission and transformation equipment data to obtain a detection result includes the following steps: Use the YOLOv8 target detector to perform target detection on the lidar data acquired by the lidar to obtain the lidar target detection result; The YOLOv8 target detector is used to perform target detection on the calibrated image data obtained by the camera to obtain the camera target detection result.

6. The multi-scale data fusion method for digital twins of power transmission and transformation equipment according to claim 5, characterized in that: Performing data fusion based on the detection results to obtain an observation set includes the following steps: Obtaining the spatial speed and position of the object in the laser radar target detection result, and obtaining the azimuth information and object classification in the camera target detection result; The maximum a posteriori estimation is performed using the spatial velocity, position, azimuth information of the object and the object classification to obtain the optimal fusion azimuth. The specific formula is as follows: Among them, θ f is the optimal fusion azimuth; The target measurement data is obtained by calculating using the optimal fusion azimuth angle. The specific formula is as follows: z m =[x y v x v y "type"] T Among them, z m Measure data for the target; For radar points; Measure for images; An observation set is generated based on the target measurement data and the object classification. The specific formula is as follows: Among them, z k is the observation set.

7. The multi-scale data fusion method for digital twins of power transmission and transformation equipment according to claim 6, characterized in that: The BiLSTM network is composed of two different LSTM networks, including a forward I / P sequence and a reverse input sequence; The hidden layer HL and output of the BiLSTM network are similar in reverse, and the specific formula is as follows: fo t =σ(WE xfo x t +WE hfo h t-1 +WE gfo m t-1 +de fo ) in t =σ(WE xin x t +WE hin h t-1 +WE gin m t-1 +de in ) ou t =σ(WE xou x t +WE hou h t-1 +WE gou m t-1 +de ou ) M' t =tanh(WE xm x t +WE xh +de g ) M t =fo t ·m t-1 +in t ·M’ t h t =ou t ·tanh(M t ) Among them, WE* is the weighting matrix; b* is the bias of the three gating mechanisms and the I / P converter; tanh is the activation function; fo t is the forget gate; M′t is the t and h t-1 The new data obtained M′t;in t Select for I / P gate to integrate data into system memory; out t Select the result gate to filter the data in the memory and then output it.

8. A data multi-scale fusion system for digital twins of power transmission and transformation equipment, based on the data multi-scale fusion method for digital twins of power transmission and transformation equipment according to any one of claims 1 to 7, characterized in that: include, A data calibration module is used to obtain lidar data and image data using a lidar and a camera, select a calibration position and perform calculations, obtain pixel angular density, calibrate the image data, and obtain corrected image data; The target detection module is used to perform target detection on the lidar data and the calibrated image data using the YOLOv8 target detector to obtain the lidar target detection results and the camera target detection results; The data fusion module is used to perform maximum a posteriori estimation based on the lidar target detection results and the camera target detection results, obtain the best fusion azimuth and target measurement data, perform data fusion, and generate an observation set; The state assessment module is used to input the observation set into the BiLSTM network to classify the state of the power transmission and transformation equipment, and use the Adam optimizer to optimize the BiLSTM network to obtain the state assessment results of the power transmission and transformation equipment.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multi-scale data fusion method for digital twins of power transmission and transformation equipment described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-scale data fusion method for digital twins of power transmission and transformation equipment described in any one of claims 1 to 7 are implemented.

Images