Method for recognizing conductor galloping based on video stabilization compensation and multi-target cooperative positioning

Abstract

The application provides a conductor dancing identification method based on video stabilization compensation and multi-target cooperative positioning, relates to data processing technology, performs convolution memory processing on a fuzzy robust feature sequence in a time dimension to extract full picture timing features, and performs distribution modeling on a continuous video frame sequence to extract full picture statistical features; the full picture timing features, the full picture statistical features and an overall picture offset vector are input into a regression network for fusion mapping, and an overall jitter compensation amount is output; the conductor center point coordinates are offset based on the overall jitter compensation amount, aligned coordinates are obtained, and conductor center point displacement features and conductor distance change features are calculated based on the aligned coordinates of adjacent frames; the conductor center point displacement features and the conductor distance change features are converted into multi-modal representations and are subjected to attention weighted fusion, a conductor dancing probability is determined based on the fusion features, and the real multi-modal dancing state of the conductor can be stripped from environmental interference in a complex environment, so that the conductor is better identified.

Description

Technical Field

[0001] This invention relates to data processing technology, and more particularly to a method for identifying conductor galloping based on video image stabilization compensation and multi-target collaborative localization. Background Technology

[0002] With the continuous development of power systems, the safe operation of transmission lines is of paramount importance. Conductor galloping, induced by external factors such as wind and icing, can easily lead to serious potential power risks, including phase-to-phase flashover, hardware fatigue, or line tripping. Therefore, identifying and preventing conductor galloping is of great significance for ensuring the safe and stable operation of the power grid.

[0003] Currently, due to the complex environment on site, jitter cancellation and target motion analysis are usually processed as separate modules. This separate design relies heavily on the clear identification of fixed backgrounds in the distance, such as poles and spacers, for displacement calculation. Once the compensation fails, such as the background detection fails under blurred conditions, it will directly pollute the subsequent motion analysis, leading to a decrease in overall recognition accuracy. Therefore, it is very easy to fail due to noise interference, and the error of the compensation failure will cascade and pollute the subsequent motion analysis stage. At the same time, traditional technologies often use single-modal methods, such as pure key point tracking or black box temporal models, which lack the integration of multi-source semantic information, resulting in poor interpretability and weak generalization ability. It fails to effectively decouple the overall camera shake from the actual cable movement in a logical manner. Single-modal pixel displacement calculation is very prone to misjudgment of movement under environmental interference such as strong winds, which seriously limits the robustness and reliability under adverse weather and long-distance shooting conditions.

[0004] Therefore, how to remove environmental interference in complex environments in order to better identify the true multimodal galloping state of conductors has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a conductor galloping recognition method based on video image stabilization compensation and multi-target cooperative localization, which can remove environmental interference in complex environments, thereby better identifying the true multimodal galloping state of conductors.

[0006] A first aspect of the present invention provides a method for conductor galloping recognition based on video image stabilization compensation and multi-target cooperative localization, comprising:

[0007] Feature extraction is performed on a continuous video frame sequence to obtain a fuzzy robust feature sequence, and the overall image offset vector of adjacent frames in the continuous video frame sequence is calculated.

[0008] The fuzzy robust feature sequence is subjected to convolutional memory processing in the time dimension to extract full-image temporal features, and the continuous video frame sequence is subjected to distribution modeling to extract full-image statistical features;

[0009] The full-image temporal features, the full-image statistical features, and the overall image offset vector are input into a regression network for fusion mapping, and the overall jitter compensation amount is output.

[0010] Based on the overall jitter compensation amount, the coordinates of the conductor center point are canceled to obtain the alignment coordinates, and the displacement characteristics of the conductor center point and the distance change characteristics between conductors are calculated based on the alignment coordinates of adjacent frames.

[0011] The displacement features of the center point of the conductor and the distance change features between the conductors are converted into multimodal representations and then fused with attention weighting. The probability of conductor galloping is determined based on the fused features.

[0012] Optionally, in one possible implementation of the first aspect, the step of extracting features from a continuous video frame sequence to obtain a fuzzy robust feature sequence, and calculating the overall image offset vector of adjacent frames in the continuous video frame sequence, includes:

[0013] A continuous sequence of video frames is input into a pre-trained convolutional neural network to extract a fuzzy, robust feature sequence with edge and gradient structure information.

[0014] The dense optical flow field of adjacent frames in the continuous video frame sequence is calculated based on the dense optical flow algorithm. The average value of the displacement vector of all pixels in the dense optical flow field is obtained to obtain the overall image offset vector.

[0015] Optionally, in one possible implementation of the first aspect, the step of performing convolutional memory processing on the fuzzy robust feature sequence in the time dimension to extract full-image temporal features, and performing distribution modeling on the continuous video frame sequence to extract full-image statistical features, includes:

[0016] The fuzzy robust feature sequence is input into a convolutional long short-term memory network, and the spatiotemporal sequence representation is extracted by updating the memory unit state and hidden state. The temporal features of the whole image are determined based on the spatiotemporal sequence representation.

[0017] Based on the contrastive learning framework, feature encoding is performed on sample pairs of the same frame in the continuous video frame sequence under historical conditions to extract feature vector space representing deep distribution similarity.

[0018] The mean, standard deviation, skewness, and kurtosis in the continuous video frame sequence are combined to obtain image distribution statistics that characterize pixel-level statistical anomalies. The feature vector space is then concatenated with the image distribution statistics to generate full-image statistical features.

[0019] Optionally, in one possible implementation of the first aspect, the step of inputting the full-image temporal features, the full-image statistical features, and the overall image offset vector into a regression network for fusion mapping and outputting the overall jitter compensation amount includes:

[0020] The temporal features of the whole image, the statistical features of the whole image, and the overall image offset vector are concatenated along the feature dimension and input into a multilayer perceptron for regression calculation to generate the initial image motion compensation amount.

[0021] The initial motion compensation amount is smoothed over time using a Kalman filter to remove high-frequency noise, thus obtaining the overall jitter compensation amount.

[0022] Optionally, in one possible implementation of the first aspect, the step of canceling the coordinates of the conductor center point based on the overall jitter compensation amount to obtain aligned coordinates, and calculating the displacement characteristics of the conductor center point and the distance change characteristics between conductors based on the aligned coordinates of adjacent frames, includes:

[0023] An instance segmentation network is used to detect the continuous video frame sequence, obtain the bounding box of the conductor segment, and calculate the coordinates of the conductor center point based on the extreme coordinates of the bounding box.

[0024] Subtract the overall jitter compensation amount from the coordinates of the center point of the conductor to obtain the alignment coordinates;

[0025] The Hungarian matching algorithm is used to match and associate the conductor identities of the aligned coordinates between adjacent frames in order to maintain the continuity of the conductor trajectory in the time series.

[0026] The difference in alignment coordinates of the same conductor identity in adjacent frames is calculated as the displacement feature of the conductor center point of a single conductor.

[0027] Calculate the Euclidean distance between the alignment coordinates of different conductor identities in the same frame, and obtain the difference of the Euclidean distance between adjacent frames as the feature of distance change between conductors.

[0028] Optionally, in one possible implementation of the first aspect, the step of converting the displacement features of the conductor center point and the distance variation features between the conductors into multimodal representations and performing attention-weighted fusion, and determining the conductor galloping probability based on the fused features, includes:

[0029] Displacement time series modeling and statistical modeling are performed on the displacement characteristics of the conductor center point and the distance change characteristics between the conductors, respectively, to generate the time series characteristics of the displacement characteristics of the conductor center point and the statistical characteristics of the distance change characteristics between the conductors;

[0030] A spatial relationship graph of the conductors is constructed using the identified conductor segments as nodes and the relative distance between conductors as edges. The graph attention network is used to dynamically allocate neighborhood weights and extract the graph structure features of mutual influence between conductors.

[0031] The temporal features, statistical features, and graph structure features are input into a multimodal attention network, and feature fusion is performed by adaptively allocating feature weights through a transformer mechanism to obtain fused features.

[0032] The fused features are input into a fully connected layer, and the conductor dancing probability is obtained by mapping through an activation function.

[0033] Optionally, in one possible implementation of the first aspect, it also includes:

[0034] When the probability of conductor galloping exceeds the preset warning value, the corresponding conductor segment is selected as the segment to be inspected.

[0035] The drone is controlled to collect the first state diagram of each interval device in the section to be inspected, and the preset state diagram corresponding to the interval device is retrieved.

[0036] The preset state diagram and the first state diagram are compared to obtain the deviation range, and the sub-inspection segment in the segment to be inspected is determined based on the deviation range.

[0037] The drone is controlled to perform inspection and data collection on the sub-inspection section, and the resulting images are sent to the management terminal.

[0038] Optionally, in one possible implementation of the first aspect, determining the sub-inspection segment in the segment to be inspected based on the deviation magnitude includes:

[0039] The first state diagram is sorted in descending order based on the deviation amplitude to obtain the amplitude sequence;

[0040] The first two first state diagrams in the amplitude sequence are selected as selected state diagrams, and the interval device corresponding to the selected state diagram is used as the deviation device.

[0041] The transmission line located between the deviation devices in the section to be inspected is selected as the sub-inspection section.

[0042] Optionally, in one possible implementation of the first aspect, the control of the UAV to perform inspection and data collection on the sub-inspection section, and to send the obtained image to be analyzed to the management terminal, includes:

[0043] The interval device corresponding to the first first state diagram in the amplitude sequence is taken as the starting device, and the interval device corresponding to the second first state diagram in the amplitude sequence is taken as the ending device.

[0044] The drone is controlled to fly and collect data along the sub-inspection section from the starting device until it reaches the cutoff device, at which point the obtained image to be analyzed is sent to the management terminal.

[0045] A second aspect of the present invention provides a storage medium storing a computer program, which, when executed by a processor, is used to implement the first aspect of the present invention and various methods possibly involved in the first aspect.

[0046] The beneficial effects of this invention are as follows:

[0047] 1. This invention extracts low-level anti-blurring features and combines them with optical flow algorithms to calculate the overall image offset. It further integrates the temporal variation patterns and pixel statistical features of the image to comprehensively calculate the camera's mechanical shaking. This multi-dimensional fusion compensation method avoids the problem of image stabilization failing in adverse weather or with blurred backgrounds. It effectively isolates interference from complex environments and camera shake. After eliminating image shake errors, it not only focuses on the displacement of individual conductors but also constructs a spatial relationship network between conductors. It comprehensively analyzes the dynamic changes in the relative distances of multiple conductors and uses an attention mechanism to adaptively weight and fuse these physical features. This allows the system to better identify the mutual pulling and abnormal approach states of conductors in space, effectively avoiding misjudgments of conductor dancing based solely on single displacement data, thus improving reliability.

[0048] 2. When the present invention detects a high risk of conductor galloping, it will prioritize controlling the drone to collect status images of each interval device on the line. By comparing the tilt amplitude of these devices, the severity of the stress abnormality can be judged, thereby quickly locating the line segment that needs to be inspected. There is no need to scan the entire long line, which effectively reduces the amount of invalid data collection and data processing. Attached Figure Description

[0049] Figure 1 The flowchart shows the guide wire galloping recognition method based on video image stabilization compensation and multi-target cooperative localization provided by the present invention.

[0050] Figure 2 This is a schematic diagram of the preset state diagram provided by the present invention;

[0051] Figure 3 This is a schematic diagram illustrating the comparison deviation amplitude provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein.

[0054] It should be understood that in the various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0055] It should be understood that in this invention, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0056] It should be understood that in this invention, "multiple" refers to two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B, and C", "Contains A, B, and C" means that all three A, B, and C are contained; "Contains A, B, or C" means that one of A, B, and C is contained; "Contains A, B, and / or C" means that any one, two, or three of A, B, and C are contained.

[0057] It should be understood that in this invention, "B corresponding to A", "B corresponding to A", "A and B correspond", or "B and A correspond" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Matching A and B is defined as a similarity between A and B that is greater than or equal to a preset threshold.

[0058] Depending on the context, "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection."

[0059] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0060] This invention provides a method for conductor galloping recognition based on video image stabilization compensation and multi-target cooperative localization, such as... Figure 1 As shown, steps S1-S5 are included:

[0061] S1, extract features from the continuous video frame sequence to obtain a fuzzy robust feature sequence, and calculate the overall image offset vector of adjacent frames in the continuous video frame sequence.

[0062] It should be noted that in the real physical environment of field power transmission line inspections, surveillance cameras often face harsh conditions such as heavy fog, heavy rain, or dirty lenses, which can lead to severe blurring of distant scenes in the captured video footage. Traditional conductor galloping recognition technology typically employs a separate design, either attempting to sharpen the image using deblurring algorithms or directly relying on a clear, static background, such as a power transmission tower, to calculate the camera's shake. However, this separate design has drawbacks. In extreme weather conditions, deblurring algorithms are prone to failure and may even distort the original image structure, while the severely blurred background cannot be effectively identified by feature point matching algorithms. Therefore, if the initial image stabilization compensation fails, subsequent conductor motion analysis will be completely corrupted. This solution extracts low-level structural features that are insensitive to blur degradation from the original blurred image and simultaneously captures the macroscopic physical offset trend of the image, enabling the system to firmly grasp the motion information of the image even under harsh visual conditions.

[0063] The continuous video frame sequence can be images continuously acquired by monitoring equipment installed at the power transmission line site and arranged in chronological order. That is, images are acquired and sorted sequentially according to time to obtain the continuous video frame sequence. For example... .

[0064] in, Indicates the first The frame image, which is usually shot from a distance, has problems such as overall blurriness, low contrast, and thin wires.

[0065] Understandably, the system receives a continuous sequence of video frames transmitted from the front-end camera and inputs this sequence frame by frame into a pre-trained convolutional neural network model. During training, this model is injected with a large number of motion blur and Gaussian blur noise samples, thus giving its shallow convolutional kernels extremely strong resistance to blur interference. Through the forward propagation calculation of this convolutional neural network, the system directly outputs a low-level feature map that retains key structural information, and sorts it chronologically to obtain a blur-robust feature sequence. Simultaneously, the system calls a dense optical flow estimation algorithm to input the current frame and the previous frame into an optical flow field calculation function, calculating the horizontal and vertical displacement of each point in the image pixel by pixel. Subsequently, the system performs a global statistical aggregation operation on the optical flow displacement data of all pixels in the entire image, such as calculating the median or mean, filtering out interference from locally moving objects, and finally calculating a two-dimensional numerical coordinate representing the global background displacement trend, i.e., the overall image offset vector, which is stored in a cache for subsequent use by the fusion compensation network.

[0066] In some embodiments, step S1 (extracting features from a continuous video frame sequence to obtain a fuzzy robust feature sequence, and calculating the overall image offset vector of adjacent frames in the continuous video frame sequence) includes S11-S12:

[0067] S11, inputs a continuous video frame sequence into a pre-trained convolutional neural network to extract a fuzzy robust feature sequence with edge and gradient structure information.

[0068] Understandably, the system extracts multiple consecutive frames of images in time-stamp order and inputs them frame by frame into a pre-trained convolutional neural network model. During the training phase, this model is injected with a large number of motion blur and Gaussian blur noise samples, thus giving its shallow convolutional kernels extremely strong resistance to blur interference. Through the forward propagation calculation of this convolutional neural network, the system directly outputs the low-level feature map that retains key structural information. Then, the fuzzy robust feature sequences are obtained by sorting them according to time.

[0069]

[0070] in, For the first Low-level feature map of the frame. Represents the set of real numbers. This indicates that the underlying feature map has one dimension. The real tensor space, 64 represents the number of feature channels of the underlying feature map, H represents the height resolution of the original input video frame image, and W represents the width resolution of the original input video frame image.

[0071] The specific process involves extracting consecutive frames of two-dimensional image data based on timestamps and converting them into multi-dimensional tensors. These tensors are then fed sequentially into the input layer of a pre-trained convolutional neural network. The tensor data passes through multiple convolutional and pooling layers with specific weight parameters. During the forward computation, the bottom-level convolutional kernels scan the image through a sliding window, performing inner product calculations on the pixel matrix to activate and extract anti-blurring edge contour feature maps. Finally, the network output layer sorts these bottom-level feature maps according to time sequence, generating a fuzzy robust feature sequence.

[0072] Among these methods, Gaussian noise can be actively superimposed on the image to simulate blur perturbation. The convolutional neural network can be an improved structure of the ResNet-18 network. After pre-training on the ImageNet dataset, it is further trained using data containing blur enhancement. The training here is an existing technique and will not be elaborated on.

[0073] S12, calculate the dense optical flow field of adjacent frames in the continuous video frame sequence based on the dense optical flow algorithm, and obtain the average value of the displacement vector of all pixels in the dense optical flow field to obtain the overall image offset vector.

[0074] It should be noted that, due to the susceptibility of cameras to wind and other factors during long-distance shooting, overall video shake is common. Therefore, this invention introduces overall image offset estimation in the preprocessing stage. To eliminate interference from camera shake caused by strong winds or mechanical vibrations, it is necessary to quantify this overall background displacement. This solution directly employs a dense optical flow algorithm, which does not rely on individual feature points but rather calculates the motion displacement of all pixels in the image and takes the arithmetic mean. This effectively utilizes macroscopic statistical principles. The anisotropic displacements caused by birds flying by or individual branches swaying in the wind are completely diluted and canceled out in the massive calculation of the overall pixel average. The final, unique average offset vector can well represent the absolute physical offset caused by the camera's own body shake.

[0075] Understandably, the system extracts the image grayscale matrix data of the current frame and the previous frame from a continuous video frame sequence. It then calls a dense optical flow calculation function, which mathematically approximates the neighborhood of each pixel to calculate the horizontal and vertical offsets of each pixel in the current frame relative to the corresponding pixel in the previous frame. This generates a dense optical flow field matrix in memory containing the motion vectors of all pixels in the entire image. Next, the system's matrix operation module flattens this dense optical flow field tensor and uses a mean operator to calculate the arithmetic mean of all horizontal and vertical components. Finally, it outputs a single two-dimensional vector coordinate parameter, which is formally established and output as the overall image offset vector representing the amplitude of camera hardware shake.

[0076] For adjacent frames and The dense optical flow field is calculated, preferably using the Falkenberg dense optical flow algorithm. The global offset vector is obtained by averaging the displacement vectors of all pixels in the optical flow field.

[0077]

[0078] in, For the first Frame relative to the first The global offset vector of the frame. This represents the total number of pixels in the image that are involved in the calculation. In the first The dense optical flow at the nth pixel position, calculated by the Falnebeck algorithm, is obtained from the nth pixel position. Frame to the The frame's displacement vector, then over all pixels. Summing from 1 to N, This indicates an averaging operation, used to average the local displacement vectors of all pixels to obtain a global offset vector that represents the motion trend of the entire frame. This global offset vector is used to depict the overall motion of the image and does not represent the actual dancing of the conductor. The calculation of dense optical flow field is existing technology and will not be elaborated here.

[0079] S2, perform convolutional memory processing on the fuzzy robust feature sequence in the time dimension to extract full-image temporal features, and perform distribution modeling on the continuous video frame sequence to extract full-image statistical features.

[0080] It's important to note that to counteract camera shake in harsh environments, relying solely on the optical flow offset vector calculated in previous steps based on adjacent frames is insufficient. In strong winds outdoors, the footage not only contains the camera's own irregular shaking but also includes violent swaying of wires blown by the wind and foreground interference such as birds and leaves. If only simple geometric displacement between two frames is used to compensate for shake, it is easily skewed by the large area of ​​moving foreground in the frame, leading to directional errors in image stabilization. The camera's own shake caused by wind load or mechanical resonance often exhibits a specific low-frequency periodicity on the time axis and shows specific clustering characteristics in the probability distribution of motion amplitude. Therefore, in order to completely separate the real camera shake in the background from the fake shake of objects in the foreground, this solution uses a convolutional memory network to capture the dynamic oscillation rhythm of the image over a long time span based on spatial offset. It also uses distribution modeling to extract statistical patterns such as motion skewness and kurtosis of the whole image. This allows for the creation of a rich and three-dimensional background motion profile for the current video segment from the perspective of macroscopic temporal evolution and probability distribution, providing a foundation for the subsequent calculation of absolute image stabilization compensation values ​​to resist interference.

[0081] Convolutional memory processing refers to the deep learning operation that uses a recurrent neural network model with gated memory units and convolutional operations to process data streams with a time dimension, enabling it to extract spatial features while memorizing the motion state of historical frames and influencing subsequent outputs.

[0082] Understandably, the system receives the fuzzy robust feature sequence output from the preceding steps. Since this sequence is a four-dimensional tensor containing time, channel, and spatial dimensions, the system sequentially inputs it into a pre-constructed convolutional long short-term memory network layer in time step order. Inside this network, convolution operations are responsible for maintaining local perception of the two-dimensional spatial structure, while the forget gate and input gate of the long short-term memory unit are responsible for filtering and memorizing the periodic swaying patterns across frames. As the forward propagation of all video frame sequences ends, the system extracts the hidden state tensor of the last layer of the network, flattens it after global average pooling, and obtains a one-dimensional full-image temporal feature vector containing deep dynamic memory. At the same time, the system's statistical analysis module synchronously reads the grayscale change matrix or optical flow matrix of the continuous video frame sequence, calculates the variance of these matrix data on the time axis across the entire image to measure the intensity of fluctuations, calculates skewness to measure the asymmetry of motion direction, and calculates kurtosis to measure the proportion of extreme motion caused by sudden changes in wind force. The system normalizes the calculated scalar values ​​such as variance, skewness, and kurtosis, and then splices them together in a fixed order to finally generate a full-image statistical feature that can quantify the dynamic distribution pattern of the image.

[0083] In some embodiments, step S2 (performing convolutional memory processing on the fuzzy robust feature sequence in the time dimension to extract full-image temporal features, and performing distribution modeling on the continuous video frame sequence to extract full-image statistical features) includes S21-S23:

[0084] S21, the fuzzy robust feature sequence is input into the convolutional long short-term memory network, and the spatiotemporal sequence representation is extracted by updating the memory unit state and hidden state. The temporal features of the whole image are determined based on the spatiotemporal sequence representation.

[0085] Understandably, the system extracts a fuzzy robust feature sequence with four dimensions: spatial width, height, number of channels, and number of time frames, from the preceding steps. The system inputs frame by frame into a convolutional long short-term memory network (LSTM) according to the time step sequence. Within the network's computational units, the input gate, forget gate, and output gate all call convolution operators to perform convolution filtering calculations on the input two-dimensional feature map and the hidden state of the previous time step. Based on the activation values ​​output by the convolution calculation, the network determines which historical periodic oscillation patterns to retain in the memory unit state and which newly added displacement information of the current frame to update and absorb. As the time step progresses, the system continuously iterates and updates the memory unit state and hidden state. After the entire continuous video frame sequence has been traversed and input, the system extracts the hidden state tensor set output by the last layer of the network as the spatiotemporal sequence representation. Subsequently, the system calls the global average pooling layer in the underlying deep learning framework to reduce the dimensionality of this spatial matrix in both length and width dimensions, finally deriving a fixed-length one-dimensional full-image temporal feature. .

[0086]

[0087] in, The output is the full-map time-series feature matrix. Represents the set of real numbers. This indicates that the feature matrix is ​​a matrix with dimension 1. The real tensor space, This represents the total number of frames in the video sequence, and 512 represents the dimension of the temporal feature vector corresponding to each frame.

[0088] Specifically, the state update formula for a convolutional long short-term memory network is as follows:

[0089]

[0090]

[0091] in, For the first The state of the memory unit at any given moment is used to store long-term temporal information. For the first The forget gate control vector at each time step determines when to remember from the previous time step. How much information is retained? This represents element-wise multiplication, used in gating mechanisms to independently weight each channel of a tensor. For the first The state of the memory unit at any given moment. For the first The input gate control vector at time step 1 determines the current input features. How much is written into new memories? This is a hyperbolic tangent activation function, used to compress values ​​to the interval [-1, 1], enhancing its nonlinear expressive power. The convolution weight matrix is ​​the input features to the memory unit. This represents the convolution operation, used to extract local features in a spatial dimension. The fuzzy robust feature map input for frame T is the bottom-level feature map. This is the convolution weight matrix that represents the hidden state from the previous time step to the current memory unit. For the first The hidden state at any given moment; For the first The hidden state at each moment is used as the output of the current moment and passed to the next moment. For the first The output gate control vector at each moment determines the current memory. How many were output to the hidden state? .

[0092] S22, Based on the contrastive learning framework, feature encoding is performed on sample pairs of the same frame in the continuous video frame sequence under historical conditions to extract the feature vector space representing deep distribution similarity.

[0093] It's important to note that the distribution of images captured by surveillance cameras is extremely complex. It includes not only the actual movement of wires but also abrupt changes in lighting, cloud movement, and the camera's own erratic shaking. If only conventional feature extraction networks are used, the model is often misled by these superficial environmental noises, making it difficult to distinguish between genuine physical motion and mere environmental light and shadow disturbances. Therefore, a contrastive learning framework is introduced. By constructing sample pairs of the current frame and historical images from the same viewpoint under different environmental conditions in the underlying logic, the neural network can search for the invariant intrinsic connections between these images. During training and inference, the network is required to narrow the distance to essentially similar background distributions while widening the distance to anomalous distributions with different motion patterns. This mechanism allows the system to penetrate surface environmental camouflage and uncover the deepest probability distribution patterns in the video footage.

[0094] Among them, historical conditions refer to the environmental conditions of historical records. Sample pairs under historical conditions refer to the comparison data pairs formed by combining the video frames collected at the current moment with the static or specific background video frames collected at the same angle under different lighting or light wind conditions stored in the system's historical database. That is, sample pairs formed under different blur and offset enhancement conditions.

[0095] Understandably, the process involves retrieving the current continuous video frame sequence and extracting historical condition frames corresponding to the current monitoring perspective from a local or cloud-based historical database. These frames are then combined into one-to-one or one-to-many sample pairs using an underlying data pairing script and input into the batch. The SimCLR contrastive learning framework is then used to encode the sample pairs formed under different blur and offset enhancement conditions of the same frame, learning a stable embedding representation of the background distribution.

[0096] The contrastive learning loss function is defined as follows:

[0097] in, To compare the learning loss function values, which are used to measure the discriminative power between positive and negative sample pairs in the feature space, This is the cosine similarity function, used to calculate the directional similarity between two feature vectors. and The feature representation of positive sample pairs obtained by encoding the same frame image under different blur and offset enhancement conditions. Let r be the feature representation of all other samples (including negative samples) in the current batch. This is a temperature coefficient used to control the smoothness of the similarity distribution; the smaller the value, the more the model focuses on the most similar samples. It is an exponential function used to map similarity to the positive real number domain and amplify differences. This indicates all samples within the current batch. To sum, that is, the number of samples, for example, if the number of samples is 10... It is 10; forming the normalized denominator. () is the natural logarithm function, used to convert probability ratios into optimizable loss values.

[0098] Specifically, the server's statistical modeling module retrieves the current continuous video frame sequence and extracts historical condition frames corresponding to the current monitoring perspective from local or cloud-based historical databases. Using an underlying data pairing script, these frames are combined into one-to-one or one-to-many sample pairs for input batches. The system then synchronously inputs these sample pairs into a pre-trained bi-graph encoder network based on a contrastive learning framework. Inside the encoder, a series of multilayer perceptrons and nonlinear activation functions perform deep feature extraction and spatial mapping on the input image matrix. Through the network's forward propagation, each high-dimensional image frame is compressed and encoded into a fixed-length one-dimensional feature vector. The system concatenates and summarizes these vectors according to a preset dimensional order, constructing a feature vector space tensor in GPU memory. The cosine and Euclidean distances within this tensor represent the deep distributional similarity between the current video sequence and the historical baseline.

[0099] S23, combine the mean, standard deviation, skewness and kurtosis in the continuous video frame sequence to obtain the image distribution statistics that characterize pixel-level statistical anomalies, and concatenate the feature vector space with the image distribution statistics to generate full-image statistical features.

[0100] It's important to note that while S22 extracts a highly abstract feature space through contrastive learning, deep learning networks are essentially black boxes. Their high-dimensional output vectors often lack intuitive physical and mathematical interpretation. When dealing with extreme physical phenomena like strong winds causing power lines to sway violently, or extreme abrupt changes such as sudden large-area pixel displacement or localized glare, statistical metrics are crucial for quantification. The mean and standard deviation can macroscopically reflect the overall brightness and contrast shifts in an image; skewness can extremely sensitively capture the asymmetry of image movement direction, such as the shift caused by a unidirectional wind; and kurtosis measures the severity of image tearing caused by extreme anomalies, such as sudden gusts of wind.

[0101] Understandably, the system reads the original pixel matrix or optical flow displacement matrix of a continuous video frame sequence. For the values ​​in the matrix, it executes the mean calculation function, standard deviation calculation function, skewness calculation formula, and kurtosis calculation formula on the time axis or spatial dimension, respectively. The resulting floating-point results, representing physical-level statistical anomalies, are normalized and encapsulated into a fixed-length image distribution statistic. This is existing technology and will not be elaborated upon here. Subsequently, the system's tensor fusion engine extracts the feature vector space output by S22 and the newly obtained image distribution statistic array from memory. It then concatenates these two data blocks along the feature channel dimension axis to obtain the full-image statistical features for subsequent use by the image stabilization compensation regression network.

[0102] It's easy to understand that SimCLR contrastive learning learns the common features of images under different environmental conditions, with a loss function of L, ultimately resulting in a feature vector space. The four statistical values—mean μ, standard deviation σ, skewness, and kurtosis—directly reflect the brightness center, dispersion, skewness (whether there are more bright / dark areas), and tail (whether there is extreme noise) of the entire image. These are explicit distribution fingerprints that contrastive learning cannot directly learn. (Full image statistical features) It concatenates the feature vector space obtained from SimCLR contrastive learning with the four values ​​of the statistics to obtain a feature vector space that simultaneously focuses on deep distribution similarity and pixel-level statistical anomalies.

[0103] Among these, statistical measures that are directly calculated for image pixels or gradients include the mean μ, standard deviation σ, skewness skewness kurtosis .

[0104]

[0105] in, Skewness, reflecting the asymmetry of the data distribution, Excess kurtosis reflects how sharp or flat the data distribution is relative to a normal distribution. The total number of pixels involved in the calculation. Indicates the first grayscale value of each pixel. The mean of all pixel values. The standard deviation of all pixel values. The values ​​are standardized pixel values. The superscripts 3 and 4 represent the third and fourth central moments, respectively, which are used to characterize the distribution shape. −3 means subtracting the standard normal distribution kurtosis baseline value of 3 to make the calculation result excess kurtosis. If the result is 0, it means that it conforms to a normal distribution. A value greater than 0 means it is sharper, and a value less than 0 means it is flatter.

[0106] S3, input the full-image temporal features, the full-image statistical features, and the overall image offset vector into the regression network for fusion mapping, and output the overall jitter compensation amount.

[0107] It's important to note that determining the true extent of camera shake is the most critical prerequisite for conductor galloping recognition. Existing image stabilization technologies typically rely on single-dimensional information, such as simple optical flow displacement between two adjacent frames, or attempt to track distant towers as stationary reference points. However, in the harsh conditions of strong winds and heavy rain, towers are often obscured by rain and fog, making them unrecognizable. Simultaneously, large areas of galloping conductors in the image severely interfere with optical flow calculations, causing single-dimensional image stabilization compensation to easily fail and produce misjudgments. To overcome this problem, this solution no longer considers image displacement in isolation, but rather integrates the overall image offset vector representing the macroscopic motion trend, the full-image temporal features representing the deep mechanical oscillation cycle, and the full-image statistical features representing the probability of sudden environmental changes. By introducing a regression network for high-dimensional fusion, the system enables these three types of data from different sources to correct each other at the underlying logic level. For example, when there is a huge offset in the optical flow display, the network will combine temporal features to determine whether it is a normal pendulum trajectory of the camera being blown by the wind or a statistical data anomaly caused by the lens being blocked by foreign objects. This allows the system to extract extremely accurate camera background shake data under any harsh working conditions, cutting off the pollution of background jitter to subsequent conductor motion analysis from the source.

[0108] Understandably, the system first retrieves the one-dimensional full-image temporal feature vector, the one-dimensional full-image statistical feature vector, and the two-dimensional overall image offset vector generated in the previous steps from the cache. It then calls the tensor concatenation operator at the bottom layer of the deep learning framework to concatenate these three tensors of different sizes along the feature channel dimension, synthesizing them into a super-long one-dimensional comprehensive feature tensor containing all background motion semantics. Subsequently, the system inputs this comprehensive feature tensor to the front end of a pre-built and trained regression network. Inside the regression network, this tensor passes through multiple fully connected layers. In each layer, the input data undergoes matrix multiplication with the network's weight matrix, and a bias term is added. Then, a non-linear transformation is performed through an activation function to achieve deep fusion mapping of multimodal data. When the data flows to the network's final output layer, the system uses a linear activation function for numerical regression calculation, directly outputting a continuous two-dimensional floating-point vector containing horizontal and vertical correction values. To further eliminate high-frequency noise abrupt changes during the prediction process, the system inputs the two-dimensional vector into the internal Kalman filter algorithm module for state smoothing and optimal estimation updates, ultimately deriving a smooth and stable overall jitter compensation amount.

[0109] In some embodiments, step S3 (inputting the full-image temporal features, the full-image statistical features, and the overall image offset vector into a regression network for fusion mapping, and outputting the overall jitter compensation amount) includes:

[0110] S31, the full-image temporal features, the full-image statistical features, and the overall image offset vector are concatenated along the feature dimension and input into the multilayer perceptron for regression calculation to generate the initial image motion compensation amount.

[0111] It should be noted that although the preceding steps extracted extremely rich background motion information in three different dimensions, these are isolated from each other in terms of data form and physical meaning. Simple optical flow offset is easily affected by local foreign objects, simple temporal memory lacks the scale sense of instantaneous displacement, and simple statistical features can only reflect probability distribution and cannot be directly converted into coordinates. Therefore, this solution adopts a strategy of feature concatenation and deep regression mapping in its underlying architecture. By concatenating them in the feature dimension, the system then introduces a multilayer perceptron as the processing center. The multilayer perceptron has extremely powerful nonlinear fitting capabilities and can automatically learn the complex cross relationships between these three features from massive amounts of concatenated data. For example, when the skewness is extremely large and the optical flow offset is severe, it is determined to be real wind swaying, thereby calculating a continuous value belonging to the camera's own mechanical offset, thus avoiding the problem that traditional single-point image stabilization algorithms are prone to getting trapped in local optima.

[0112] Among them, the initial frame motion compensation amount

[0113]

[0114] in, Let be the initial motion compensation amount for frame T, and let represent the overall motion vector of the frame that needs to be removed from the wire detection results. A multilayer perceptron, a type of feedforward neural network, is used to map input features to an output space to regress and obtain compensation. This indicates the temporal features of the entire graph. Statistical characteristics of the whole map and global offset vector The features are concatenated along the feature dimension to form a fused feature vector, which is then used as input to the MLP. The full-image temporal features output by ConvLSTM include temporal variation information of the video background and overall structure. The full-image statistical features, composed of the SimCLR feature vector space (i.e., the embedding representation) and image distribution statistics (mean, standard deviation, skewness, kurtosis), reflect the stability and morphological characteristics of the image pixel distribution. The overall image offset vector, calculated and averaged by the Farneback optical flow algorithm, represents the overall displacement trend between adjacent frames.

[0115] Understandably, the system reads and places the one-dimensional full-image temporal features, one-dimensional full-image statistical features, and two-dimensional overall image offset vector of varying lengths. By invoking a deep learning underlying framework, such as tensor concatenation with the feature dimension as the connection axis, these three independent data blocks are concatenated end-to-end. Then, the system inputs this fused vector into the input layer of a pre-trained multilayer perceptron model. The data flows through multiple hidden layers within the multilayer perceptron. In each layer, the input vector is multiplied by the fully connected weight matrix of that layer, and a bias term is added. It then passes through a non-linear activation function. When the data finally reaches the output layer without non-linear activation, the network performs a linear regression mapping, directly outputting a two-dimensional continuous floating-point coordinate representing the image's XY-axis offset. The system extracts this coordinate and assigns it to the initial image motion compensation value. Here, the feature dimension refers to the mathematical axis in the deep learning data tensor structure used to characterize the number of different semantic attributes or channels contained in the data.

[0116] S32, the initial image motion compensation amount is smoothed over time using a Kalman filter to filter out high-frequency noise and obtain the overall jitter compensation amount.

[0117] It is understandable that the Kalman filter is an autoregressive mathematical algorithm that can use the state equation of a linear system to make optimal estimates and trajectory predictions of the dynamic state of the system through the system input and output observation data. Its processing principle is existing technology and will not be elaborated here. Therefore, the motion compensation amount of the initial image is smoothed in time series by the Kalman filter to filter out high-frequency noise and obtain the overall jitter compensation amount.

[0118] S4. Based on the overall jitter compensation amount, the coordinates of the conductor center point are canceled to obtain the alignment coordinates, and the displacement characteristics of the conductor center point and the distance change characteristics between conductors are calculated based on the alignment coordinates of adjacent frames.

[0119] It should be noted that cameras inevitably shake in strong winds, causing significant relative displacement of the conductors in the image. Directly measuring the conductor's motion in the video feed would yield a result that is the sum of the conductor's actual movement and the camera's mechanical jitter—data with substantial errors unusable for assessing line safety. This solution has already extracted the camera's overall jitter compensation in previous steps; therefore, this step focuses on error elimination and motion restoration. By subtracting the camera's jitter offset in a mathematical coordinate system, the system anchors the conductor's coordinates to a stationary virtual physical reference frame. After alignment, the system calculates not only the absolute displacement of a single conductor but also the relative distance changes between multiple conductors. In strong winds, multiple conductors may swing synchronously in the same direction; while individual displacements may be large, collisions are not guaranteed. The real danger lies in the drastic reduction in relative distance between conductors, which can easily trigger phase-to-phase flashovers.

[0120] Among them, the coordinates of the center point of the conductor can be the original pixel position data of the geometric center position of the conductor segment calculated after the outline of the transmission conductor is identified in the original two-dimensional video image using target detection or instance segmentation algorithms, that is, the geometric center position data of the conductor segment.

[0121] Understandably, the system identifies all conductor segments in the image and calculates the center point coordinates of each conductor using a geometric centroid algorithm. Next, the system retrieves the two-dimensional vector of the overall jitter compensation for the current frame output in step S3 and performs coordinate vector subtraction, subtracting the overall jitter compensation from the conductor center point coordinates. This eliminates coordinate distortion caused by overall image translation, outputting reliable alignment coordinates. Subsequently, using a target tracking algorithm, such as the Hungarian matching algorithm, combined with the spatial location features of previous and subsequent frames, the system associates and binds the same conductor entity in the current frame with the same conductor in the previous frame. For the successfully associated conductor, the system calculates its Euclidean distance and orientation angle between the alignment coordinates of the current and previous frames, using the results as the conductor center point displacement feature. Simultaneously, for multiple successfully identified conductors in the current frame, the system calculates the Euclidean distance between their alignment coordinates pairwise, compares the difference between the distance matrix of the current frame and the distance matrix of the previous frame, extracts the increase or decrease in distance, generates the distance change feature between conductors, and synchronously outputs these two sets of feature data to the subsequent dance discrimination network.

[0122] In some embodiments, step S4 (completing the offsetting process of the conductor center point coordinates based on the overall jitter compensation amount to obtain aligned coordinates, and calculating the conductor center point displacement characteristics and inter-conductor distance change characteristics based on the aligned coordinates of adjacent frames) includes S41-S45:

[0123] S41, an instance segmentation network is used to detect the continuous video frame sequence, obtain the bounding box of the conductor segment, and calculate the coordinates of the center point of the conductor based on the extreme coordinates of the bounding box.

[0124] Understandably, a continuous sequence of video frames is input into a pre-trained instance segmentation network. The network extracts backbone features and predicts masked branches, outputting the detection results of all wire targets in the frame and extracting the bounding boxes surrounding each wire's pixel set. For each bounding box, the minimum and maximum pixel values ​​in the horizontal and vertical directions are extracted. The horizontal maximum and minimum values ​​are added together and divided by two to obtain the x-coordinate of the center point, and the vertical maximum and minimum values ​​are added together and divided by two to obtain the y-coordinate of the center point. The system combines these two coordinates and uses them as the center point coordinates of that wire.

[0125] The object detection and segmentation network performs wire detection on the video frames. The preferred network is YOLOv8-seg.

[0126] For example, detecting the bounding box of a conductor segment. And calculate the coordinates of its center point:

[0127]

[0128] in, For the first The first frame The coordinates of the center point of the root conductor segment This represents the minimum pixel coordinates of the wire segment bounding box along the horizontal axis of the image. This represents the maximum pixel coordinates of the wire segment bounding box along the horizontal axis of the image. This represents the minimum pixel coordinates of the wire segment bounding box along the vertical axis of the image. This represents the maximum pixel coordinates of the wire segment bounding box along the vertical axis of the image. This means taking the average of the horizontal coordinates to obtain the horizontal coordinate of the center point; This means taking the average of the vertical coordinates to obtain the vertical coordinate of the center point.

[0129] S42, Subtract the overall jitter compensation amount from the coordinates of the center point of the conductor to obtain the alignment coordinates.

[0130] It is easy to understand that by subtracting the overall jitter compensation amount from the coordinates of the center point of the conductor, the alignment coordinates are obtained. That is, a vector subtraction traversal operation is performed for each center point of the conductor, that is, the global motion offset is subtracted from the original center point coordinates to eliminate the overall displacement of the image caused by camera jitter and achieve spatial alignment.

[0131] S43, the Hungarian matching algorithm is used to match and associate the conductor identities of the aligned coordinates between adjacent frames to maintain the continuity of the conductor trajectory in the time series.

[0132] Understandably, when processing the current frame, the system first retrieves the historical target list from the previous frame, which contains multiple conductor identities and their corresponding alignment coordinates, and then reads the alignment coordinate list of all unknown identities just calculated in the current frame. The system creates a two-dimensional cost matrix, calculates the Euclidean distance between each conductor alignment coordinate in the previous frame and each conductor alignment coordinate in the current frame, and fills these distance values ​​into the corresponding positions in the matrix. Subsequently, the Hungarian matching function is used, passing this cost matrix as a parameter. This function internally performs row and column addition and subtraction operations and searches for independent zero elements, ultimately calculating a one-to-one assignment matrix that minimizes the global distance sum. The system parses the output of this assignment matrix and accurately inherits and assigns the conductor identity labels from the previous frame to the corresponding conductor targets that are successfully matched in the current frame, thus completing the identity association in the time series, i.e., using the existing Hungarian matching algorithm to achieve conductor identity matching and association.

[0133] S44, calculate the difference in alignment coordinates of the same conductor identity in adjacent frames as the displacement feature of the conductor center point of a single conductor.

[0134] It is understandable that the horizontal displacement is obtained by subtracting the horizontal component of the previous frame from the horizontal component of the alignment coordinates of the current frame, and the vertical displacement is obtained by subtracting the vertical component of the previous frame from the vertical component of the alignment coordinates of the current frame, thus obtaining the displacement characteristics of the center point of a single conductor.

[0135] Displacement of the center point of a single conductor

[0136] in, For the first The first wire Frame relative to the first The displacement vector of the center point of the frame. For the Tth frame The coordinates of the alignment center point of the conductor after jitter compensation. For the first The first frame The coordinates of the alignment center point of the conductor after jitter compensation.

[0137] S45, calculate the Euclidean distance between the alignment coordinates of different conductor identities in the same frame, and obtain the difference of the Euclidean distance between adjacent frames as the distance change feature between conductors.

[0138] The change in distance between different conductors,

[0139]

[0140]

[0141] in, For the first The first frame Root and First The Euclidean distance between the root conductors For the first The first frame The x-coordinate of the center point after traverse compensation. For the first The first frame The ordinate of the center point after traverse compensation. For the first The first frame The x-coordinate of the center point after traverse compensation. For the first The first frame The ordinate of the center point after traverse compensation. For the first Frame relative to the first The first frame With the The change in distance between the wires This represents the distance between two identical wires in the previous frame, with a focus on the lateral component, which is used to characterize the wire dancing properties.

[0142] S5, the displacement features of the center point of the conductor and the distance change features between the conductors are converted into multimodal representations and then fused with attention weighting. The probability of conductor galloping is determined based on the fused features.

[0143] It should be noted that traditional methods for quantifying conductor displacement often rely on single-modal features such as pure pixel offsets or black-box time series. Measurements under ambiguous conditions are extremely inaccurate, and the lack of semantic information integration leads to delayed judgments and high misjudgment rates regarding secondary risks caused by conductor galloping. Conductor galloping not only manifests as low-frequency, large-amplitude oscillations on the time axis but also as multiple conductors moving closer together or pulling on each other in spatial topology. If changes in a single dimension are viewed in isolation, background interference such as wind blowing through tree branches can easily be misjudged as galloping. To completely address this issue, this solution transforms the extracted physical displacement and distance differences into three higher-order representations: time series, statistical distribution, and spatial graph structure. An attention mechanism is used to adaptively evaluate the reliability of these three types of evidence in the current scenario and perform weighted integration. This comprehensive judgment mechanism, with its multi-faceted verification, avoids the limitations of single-modal methods, making the final output galloping probability highly physically interpretable and robust against interference.

[0144] Understandably, the system invokes a temporal modeling network and a statistical modeling function to extract the memory patterns of the center point displacement features and the distance variation features between conductors along the time axis, as well as statistical features such as mean and variance. Simultaneously, the system constructs a conductor structure graph using each conductor as a node and the distance between them as edges, and inputs this graph into a graph attention network. By aggregating neighbor node information, the system models the mutual influence relationships between conductors, thereby generating graph structure features. At this point, the original displacement is transformed into a multimodal representation containing temporal features, statistical features, and graph structure features. Next, the system synchronously inputs these three sets of features into a three-modal attention fusion module based on a transformer architecture. Within this module, the system calculates the inner product of the query matrix and the key matrix to obtain an attention score. This score is then used to perform a weighted summation of the value matrix, dynamically amplifying the weights of key modes and suppressing noisy modes. Finally, the system concatenates and maps these features to output representative fused features. Finally, the fused feature is input into the fully connected layer at the end of the network for linear dimensionality reduction, and then passed through the Sigmoid activation function for nonlinear mapping. The activation function compresses any real number output by the fully connected layer into an open interval between zero and one. The resulting value is extracted and used as the conductor galloping probability to characterize the current line danger level for output warning.

[0145] In some embodiments, step S5 (converting the conductor center point displacement features and the inter-conductor distance change features into multimodal representations and performing attention-weighted fusion, and determining the conductor galloping probability based on the fused features) includes:

[0146] S51, perform displacement time-series modeling and statistical modeling on the displacement characteristics of the conductor center point and the distance change characteristics between the conductors respectively, and generate the time-series characteristics of the displacement characteristics of the conductor center point and the statistical characteristics of the distance change characteristics between the conductors.

[0147] Among them, displacement time series modeling refers to the data processing process of using low-level algorithms such as recurrent neural networks or one-dimensional convolution to mine the periodic time rhythm of the continuous evolution of the physical coordinates of a conductor over time. Statistical modeling refers to the data processing operation of using mathematical statistical formulas to calculate the distribution of distance changes within a specific time window, such as mean, variance, and skewness.

[0148] Understandably, the system retrieves the displacement feature vectors of the conductor center points from multiple frames arranged over time and inputs them as sequential data into a pre-constructed Long Short-Term Memory (LSTM) network. Within this network, the data undergoes iterative updates through input and forget gates, extracting the final hidden state tensor that captures the macroscopic motion cycle, which the system saves as a temporal feature. Simultaneously, the system extracts the feature sequence of distance changes between conductors within the same time window. Variance calculation, standard deviation derivation, and maximum range comparison are then performed on this sequence matrix. The system normalizes these calculated discrete mathematical scalars representing the intensity of physical fluctuations and concatenates them in a fixed order to form a one-dimensional statistical feature, which is then output synchronously with the temporal feature to the next processing node.

[0149] S52 constructs a spatial relationship graph of the conductors using the identified conductor segments as nodes and the relative distance between conductors as edges, and dynamically allocates neighborhood weights through a graph attention network to extract the graph structure features of mutual influence between conductors.

[0150] It's important to note that multiple power transmission lines form a physically interconnected topological system in three-dimensional space. When a dangerous phase-to-phase flashover occurs, the relative spatial distance between two or more conductors often decreases dramatically. Traditional analysis methods focus only on individual conductors and cannot capture this cross-target physical spatial linkage. This solution uses graph theory to abstract the scattered conductor entities in the image into a network of nodes within a graph network. Based on this, a graph attention network is used. Since the probability of collision varies drastically depending on the wind direction and location of the conductors, the graph attention network can simulate the spatial pulling and approximation states between conductors in the physical world, automatically allocating computational focus and feature weights to conductor pairs whose distance is abnormally decreasing. This avoids the problem of single-target tracking.

[0151] Understandably, each conductor assigned an independent identity within the current frame is mapped to a node object in the graph, and their respective coordinate features are assigned to the nodes. Simultaneously, the distance variation features between conductors are assigned to the virtual edge objects connecting different nodes. After construction, the system inputs this graph data structure into the input layer of the graph attention network. In the hidden layer of the network, for each central node, the algorithm calculates its inner product attention score with all associated neighboring nodes using a multilayer perceptron. This score is then normalized using a normalized exponential function to generate neighborhood weights representing importance. Subsequently, based on these neighborhood weight coefficients, the state feature matrices of all neighboring nodes are weighted and summed to update the state matrix of the current central node. After multilayer graph network aggregation calculation, the system globally pools and flattens all graph node features, ultimately outputting graph structure features representing the spatial hazard morphology of the entire region. Specifically, the displacement features of the central point and the variation features of conductor spacing are modeled temporally and statistically, respectively, and a conductor structure graph is constructed. The graph attention network models the mutual influence relationships between conductors.

[0152] S53, the temporal features, statistical features and graph structure features are input into a multimodal attention network, and feature fusion is performed by adaptively allocating feature weights through a transformer mechanism to obtain fused features.

[0153] It should be noted that while temporal features, statistical features, and the aforementioned graph structure features have been obtained, the contribution of these three modalities to the final gossip warning is not constant. For example, during sudden gusts, the statistical features are more crucial, while during sustained light wind resonance, temporal features are more important. If a traditional method of simply concatenating arrays or calculating the mean is used, it is extremely easy for critical danger signals to be drowned out by the normal noise of other modalities. Therefore, a transformer attention mechanism is used. This mechanism can spontaneously compare and find the most core evidence chain in massive multimodal data. It dynamically calculates the intrinsic dot product correlation between features of different modalities, assigning a higher data weight to the modality that better reflects the essence of the current gossip, while suppressing the influence of interfering modalities. This adaptive cross-fusion mechanism ensures that the system can consistently extract decisive gossip discrimination features under extremely complex field wind conditions.

[0154] Understandably, by inputting temporal features, statistical features, and graph structure features into the trimodal attention fusion module, and performing weighted fusion based on the Transformer attention mechanism, the fused features are output. Specifically, temporal features, statistical features, and graph structure features are dimensionally aligned and combined into a feature sequence matrix, which is then input into the trimodal attention fusion module. Within this module, the system uses a transformer to perform matrix multiplication operations with three pre-trained independent weight matrices, mapping them to generate corresponding query vector matrices, key vector matrices, and value vector matrices. By calculating the dot product of the query vector and the transpose of the key vector, an attention score matrix reflecting the interrelationship between different modal features is obtained. This score matrix is ​​then normalized using a normalized exponential function to calculate adaptive attention weight coefficients. The system then performs a weighted summation matrix multiplication of these weight coefficients with the corresponding value vector matrix, completing the deep cross-fusion of features across different modalities. Finally, a fused feature with three physical semantics is output for subsequent use.

[0155] S54, the fused features are input into the fully connected layer, and the wire dancing probability is obtained by mapping through the activation function.

[0156] It should be noted that while the fused feature tensor is an excellent data representation for computers, it is completely unreadable for front-end power grid maintenance personnel or alarm dispatch systems. Therefore, this solution uses a fully connected layer to perform the final feature convergence and linear dimensionality reduction classification, while the activation function can smoothly compress the unbounded real numbers output by the fully connected layer into the standard interval of zero to one, which represents statistical probability.

[0157] The wire dancing probability is output through a fully connected layer and a Sigmoid function:

[0158]

[0159] in, The probability value for conductor galloping is [0, 1], used to determine whether conductor galloping exists in the current video frame. FC is a fully connected layer used to fuse high-dimensional features. Mapped to a one-dimensional scalar space, σ() is the Sigmoid activation function, used to map any real number to the interval between 0 and 1, giving it a probabilistic meaning. The overall expression represents the regression of the probability of dancing occurring from the fused features through linear transformation and nonlinear activation.

[0160] Based on the above embodiments, A1-A4 are also included:

[0161] A1, When the probability of conductor galloping is greater than the preset warning value, the corresponding conductor segment is selected as the segment to be inspected.

[0162] Among them, the preset warning value can be a value that is set in advance by people based on the actual situation.

[0163] It is easy to understand that when the probability of conductor galloping is greater than the preset warning value, it means that the probability of conductor galloping in that section of the line is relatively high, which is likely to cause transmission line faults. Generally, transmission lines are made of multiple aluminum wires twisted together. When galloping occurs, it is very easy for some aluminum wires on the outside of the line to break due to external forces. Therefore, under stable environmental conditions, drones can be dispatched to the corresponding conductor section for inspection, and the corresponding conductor section will be designated as the section to be inspected.

[0164] A2, control the drone to collect the first state diagram of each interval device in the section to be inspected, and retrieve the preset state diagram corresponding to the interval device.

[0165] It should be noted that existing technologies for detecting aluminum wire breakage or severe fatigue damage in transmission lines often require controlling drones to conduct continuous, carpet-like inspections and photography along the entire transmission line, which is hundreds of meters long. This method not only generates massive amounts of invalid image data, increasing the processing load on the backend algorithm, but also makes it difficult to accurately locate subtle strand breakage issues in a timely manner. In real-world operating conditions, spacers and other spacing devices are fixed to various positions on the transmission conductor using specialized metal clamps, playing a crucial role in maintaining conductor tension balance. Once the conductor experiences violent vibrations, if a portion of the outer aluminum wire breaks at a certain point, such as on one side of the clamp, the mechanical tension at that point will instantly become unbalanced. This uneven stress can directly cause slippage of the connected spacer clamps or force the entire spacing device to reverse or twist significantly, resulting in an abnormal tilted posture. Therefore, this solution prioritizes using the spacing devices as mechanical stress indicators for the entire line for directional data acquisition. By acquiring images only of these key stress points, it is possible to identify potential breakage hazards in the conductor itself.

[0166] Therefore, we will prioritize controlling the drone to collect the first state diagram of each interval device at the section to be inspected, and retrieve the preset state diagram corresponding to the interval device, wherein the interval device can be a spacer bar.

[0167] A3. Compare the preset state diagram and the first state diagram to obtain the deviation range, and determine the sub-inspection segment in the section to be inspected based on the deviation range.

[0168] It's easy to understand that the preset state diagram can be a pre-set standard image of a spacer bar. For example, a spacer bar used to fix four transmission lines is typically vertical, appearing perpendicular to the image boundary, i.e., perpendicular to the ground. (See [reference]). Figure 2The spacer bars in the preset state diagram can be side views captured at that location, such as vertical rectangles. Therefore, by directly comparing the spacer bars in the first state diagram with the preset state diagram, the deviation between the two can be obtained. For comparisons between tilted and non-tilted states, see [link to documentation]. Figure 3 The two clamps on the upper side of the spacer that fixes the four branch lines have shifted, and the spacer with the larger displacement is closer to the location of the breakage.

[0169] In some embodiments, step A3 (determining the sub-inspection segment in the segment to be inspected based on the deviation magnitude) includes A31-A33:

[0170] A31, based on the deviation amplitude, sort the first state diagram in descending order to obtain the amplitude sequence.

[0171] A32, select the first two first state diagrams in the amplitude sequence as selected state diagrams, and use the interval device corresponding to the selected state diagram as the deviation device.

[0172] It is easy to understand that the spacer clamps closer to the breakage point on the multiple spacers on the transmission line are more likely to slip and are the first to be affected, forcing the entire spacer device to undergo obvious reversal and twisting.

[0173] Therefore, the server selects the first two first state diagrams in the amplitude sequence as selected state diagrams, and uses the interval device corresponding to the selected state diagram as the deviation device.

[0174] A33, Select the power transmission line located between the deviation devices in the section to be inspected as the sub-inspection section.

[0175] Through the above implementation method, this solution does not require a detailed and complete scan of all transmission lines from beginning to end. Since the fractured parts need to be inspected by drones at close range, a lot of data collection time is required. However, by directly identifying the tilt state of the spacer bars at a certain distance through shooting, we can locate the sub-inspection section that actually needs to be inspected, which not only reduces the amount of data collection, but also the amount of data processing.

[0176] It's easy to understand that spacers are used to separate multiple branch lines in a transmission line to prevent them from coming into contact with each other during galloping. Spacers can be fixed with clamps. Therefore, if multiple aluminum wires break on the outside of the conductor, the spacer may experience movement at the clamping points due to the force being applied on one side and the other side being broken and unloaded. When a spacer has four fixed clamping points, if one of them moves, it indicates that there is an abnormality in the branch line where the clamping point is located.

[0177] In addition to the above, it also includes:

[0178] The fixed point where the deviation occurs in the deviation device is obtained, and the branch circuit where the fixed point is located is taken as the branch to be inspected.

[0179] It is not difficult to understand that the deviation device is a large-amplitude interval device, for example, used to fix 4 lines. The server will obtain the fixed point where the deviation occurs and take the branch circuit where the point is located as the branch to be inspected.

[0180] The transmission line located between the deviation devices in the branch to be inspected is selected as the branch sub-inspection section.

[0181] A4, control the drone to perform inspection and data collection on the sub-inspection section, and send the obtained images to be analyzed to the management terminal.

[0182] In some embodiments, step A4 (controlling the drone to perform inspection and data collection on the sub-inspection section, obtaining the image to be analyzed and sending it to the management terminal) includes A41-A42:

[0183] A41, obtain the interval device corresponding to the first first state diagram in the amplitude sequence as the starting device, and take the interval device corresponding to the second first state diagram in the amplitude sequence as the ending device.

[0184] It is easy to understand that the larger the tilt amplitude, the more it is affected first and the greater the impact. This indicates that the damaged part is closer to the interval device with the largest tilt amplitude. Therefore, the interval device corresponding to the first first state diagram is taken as the starting device, and the interval device corresponding to the second first state diagram in the amplitude sequence is taken as the ending device.

[0185] A42 controls the drone to fly and collect data along the sub-inspection section from the starting device until it reaches the cutoff device, at which point the obtained image to be analyzed is sent to the management terminal.

[0186] It's easy to understand that there's a corresponding flight sequence during flight, which facilitates quick location of the damaged area.

[0187] The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the methods provided in the various embodiments described above.

[0188] The readable storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of computer programs from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application-Specific Integrated Circuit (ASIC). Alternatively, the ASIC can be located in a user equipment. Of course, the processor and the readable storage medium can also exist as discrete components in a communication device. The readable storage medium can be a read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0189] The present invention also provides a program product including executable instructions stored in a readable storage medium. At least one processor of the device can read the executable instructions from the readable storage medium, and the at least one processor executes the executable instructions to cause the device to implement the methods provided in the various embodiments described above.

[0190] In the embodiments of the above-described device, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing conductor galloping based on video image stabilization compensation and multi-target cooperative localization, characterized in that, include: Feature extraction is performed on a continuous video frame sequence to obtain a fuzzy robust feature sequence, and the overall image offset vector of adjacent frames in the continuous video frame sequence is calculated. The fuzzy robust feature sequence is subjected to convolutional memory processing in the time dimension to extract full-image temporal features, and the continuous video frame sequence is subjected to distribution modeling to extract full-image statistical features; The full-image temporal features, the full-image statistical features, and the overall image offset vector are input into a regression network for fusion mapping, and the overall jitter compensation amount is output. Based on the overall jitter compensation amount, the coordinates of the conductor center point are canceled to obtain the alignment coordinates, and the displacement characteristics of the conductor center point and the distance change characteristics between conductors are calculated based on the alignment coordinates of adjacent frames. The displacement features of the center point of the conductor and the distance change features between the conductors are converted into multimodal representations and then fused with attention weighting. The probability of conductor galloping is determined based on the fused features.

2. The method according to claim 1, characterized in that, The step of extracting features from a continuous video frame sequence to obtain a fuzzy robust feature sequence, and calculating the overall image offset vector of adjacent frames in the continuous video frame sequence, includes: A continuous sequence of video frames is input into a pre-trained convolutional neural network to extract a fuzzy, robust feature sequence with edge and gradient structure information. The dense optical flow field of adjacent frames in the continuous video frame sequence is calculated based on the dense optical flow algorithm. The average value of the displacement vector of all pixels in the dense optical flow field is obtained to obtain the overall image offset vector.

3. The method according to claim 1, characterized in that, The step of performing convolutional memory processing on the fuzzy robust feature sequence in the time dimension to extract full-image temporal features, and performing distribution modeling on the continuous video frame sequence to extract full-image statistical features, includes: The fuzzy robust feature sequence is input into a convolutional long short-term memory network, and the spatiotemporal sequence representation is extracted by updating the memory unit state and hidden state. The temporal features of the whole image are determined based on the spatiotemporal sequence representation. Based on the contrastive learning framework, feature encoding is performed on sample pairs of the same frame in the continuous video frame sequence under historical conditions to extract feature vector space representing deep distribution similarity. The mean, standard deviation, skewness, and kurtosis in the continuous video frame sequence are combined to obtain image distribution statistics that characterize pixel-level statistical anomalies. The feature vector space is then concatenated with the image distribution statistics to generate full-image statistical features.

4. The method according to claim 1, characterized in that, The step of inputting the full-image temporal features, the full-image statistical features, and the overall image offset vector into a regression network for fusion mapping, and outputting the overall jitter compensation amount, includes: The temporal features of the whole image, the statistical features of the whole image, and the overall image offset vector are concatenated along the feature dimension and input into a multilayer perceptron for regression calculation to generate the initial image motion compensation amount. The initial motion compensation amount is smoothed over time using a Kalman filter to remove high-frequency noise, thus obtaining the overall jitter compensation amount.

5. The method according to claim 1, characterized in that, The process of canceling the coordinates of the conductor center point based on the overall jitter compensation amount to obtain aligned coordinates, and calculating the displacement characteristics of the conductor center point and the distance change characteristics between conductors based on the aligned coordinates of adjacent frames, includes: An instance segmentation network is used to detect the continuous video frame sequence, obtain the bounding box of the conductor segment, and calculate the coordinates of the conductor center point based on the extreme coordinates of the bounding box. Subtract the overall jitter compensation amount from the coordinates of the center point of the conductor to obtain the alignment coordinates; The Hungarian matching algorithm is used to match and associate the conductor identities of the aligned coordinates between adjacent frames in order to maintain the continuity of the conductor trajectory in the time series. The difference in alignment coordinates of the same conductor identity in adjacent frames is calculated as the displacement feature of the conductor center point of a single conductor. Calculate the Euclidean distance between the alignment coordinates of different conductor identities in the same frame, and obtain the difference of the Euclidean distance between adjacent frames as the feature of distance change between conductors.

6. The method according to claim 1, characterized in that, The step of converting the displacement features of the conductor center point and the distance variation features between conductors into multimodal representations and performing attention-weighted fusion, and determining the conductor galloping probability based on the fused features, includes: Displacement time series modeling and statistical modeling are performed on the displacement characteristics of the conductor center point and the distance change characteristics between the conductors, respectively, to generate the time series characteristics of the displacement characteristics of the conductor center point and the statistical characteristics of the distance change characteristics between the conductors; A spatial relationship graph of the conductors is constructed using the identified conductor segments as nodes and the relative distance between conductors as edges. The graph attention network is used to dynamically allocate neighborhood weights and extract the graph structure features of mutual influence between conductors. The temporal features, statistical features, and graph structure features are input into a multimodal attention network, and feature fusion is performed by adaptively allocating feature weights through a transformer mechanism to obtain fused features. The fused features are input into a fully connected layer, and the conductor dancing probability is obtained by mapping through an activation function.

7. The method according to claim 1, characterized in that, Also includes: When the probability of conductor galloping exceeds the preset warning value, the corresponding conductor segment is selected as the segment to be inspected. The drone is controlled to collect the first state diagram of each interval device in the section to be inspected, and the preset state diagram corresponding to the interval device is retrieved. The preset state diagram and the first state diagram are compared to obtain the deviation range, and the sub-inspection segment in the segment to be inspected is determined based on the deviation range. The drone is controlled to perform inspection and data collection on the sub-inspection section, and the resulting images are sent to the management terminal.

8. The method according to claim 7, characterized in that, The step of determining the sub-inspection segment in the section to be inspected based on the deviation amplitude includes: The first state diagram is sorted in descending order based on the deviation amplitude to obtain the amplitude sequence; The first two first state diagrams in the amplitude sequence are selected as selected state diagrams, and the interval device corresponding to the selected state diagram is used as the deviation device. The transmission line located between the deviation devices in the section to be inspected is selected as the sub-inspection section.

9. The method according to claim 8, characterized in that, The controlled drone performs inspection and data collection on the sub-inspection section, and sends the obtained images to be analyzed to the management terminal, including: The interval device corresponding to the first first state diagram in the amplitude sequence is taken as the starting device, and the interval device corresponding to the second first state diagram in the amplitude sequence is taken as the ending device. The drone is controlled to fly and collect data along the sub-inspection section from the starting device until it reaches the cutoff device, at which point the obtained image to be analyzed is sent to the management terminal.

10. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, is used to implement the method described in any one of claims 1 to 9.

Images