Video stabilization method for transmission line visual detection scene

Abstract

This invention discloses a video stabilization method for power transmission line visualization inspection scenarios, comprising: collecting outdoor environmental images and labeling non-moving objects; establishing original image data and target image data; constructing and training an original target detection model to obtain a final target detection model; detecting non-moving objects in real-time video; selecting motion estimation targets and calculating motion vectors; calculating global motion vectors; processing with Kalman filtering; and performing motion compensation. This invention is particularly applicable to power transmission line visualization inspection scenarios, avoiding weakening of target objects, such as the jitter of conductor galloping, and improving the accuracy of conductor galloping detection. This method can provide the position of non-moving objects, enabling the block matching algorithm to focus more accurately on the corresponding position, improving the accuracy of motion vectors, and enhancing the subsequent image stabilization effect.

Description

Technical Field

[0001] This invention belongs to the field of power transmission engineering technology, and in particular relates to a video stabilization method for visual inspection scenarios of power transmission lines. Background Technology

[0002] With the development and advancement of information technology, concepts such as digital society and new infrastructure have been proposed. As the lifeblood of cities, power transmission lines play a vital role in this transformation. Therefore, intelligent visual detection technology for power transmission lines is widely used. However, due to external factors, the monitoring videos of power transmission lines often suffer from jitter, severely affecting the detection of abnormal events such as conductor galloping.

[0003] Video stabilization technology is typically used to overcome this problem. Video stabilization involves using algorithms to process raw video sequences captured by video equipment and remove jitter. The purpose of video stabilization is twofold: firstly, to improve the viewing experience for the human eye, facilitating manual observation and judgment; and secondly, to serve as a preprocessing stage for many other subsequent processes, such as detection, tracking, and compression. Currently, the most commonly used method is electronic (digital) stabilization, which is based on motion estimation between consecutive video images, followed by motion filtering and motion compensation processing on each frame of the video to obtain a stable image. Generally speaking, electronic (digital) stabilization methods include three steps: motion estimation, motion compensation, and image inpainting.

[0004] One of the main factors restricting the development of digital image stabilization technology is motion estimation. The basic idea of ​​motion estimation is to divide each frame of an image sequence into many non-overlapping macroblocks, assuming that all pixels within a macroblock have the same displacement. Then, for each macroblock within a given search range to a reference frame, a matching block is found based on certain matching criteria. The relative displacement between the matching block and the current block is the motion vector. During video compression, only the motion vector and residual data need to be saved to completely recover the current block, obtaining the motion vector (also called motion vector). This process is called motion estimation. Existing technologies typically use local motion vectors to approximate global motion vectors, such as the block matching method mentioned above, as well as optical flow and feature point detection methods for calculating motion vectors. However, the regions selected or calculated by these methods are random and lack fixed properties. This can lead to inaccuracies in global motion estimation. For example, when both the camera and the guide wire shake simultaneously, if motion vector calculations are performed based on the guide wire region between frames, the shake of the guide wire may be weakened after image stabilization, resulting in a decrease in the accuracy of further guide wire vibration detection.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background technology of this application, and therefore may include prior art that is not known to those skilled in the art. Summary of the Invention

[0006] This invention addresses the problem that existing motion estimation algorithms typically use local motion vectors to approximate global motion vectors, and the selected or calculated regions are random and lack fixed properties. This leads to inaccurate global motion estimation, especially since image stabilization may weaken conductor jitter, resulting in a further decrease in the accuracy of conductor galloping detection. Therefore, this invention designs and provides a video image stabilization method for visual inspection scenarios of power transmission lines.

[0007] To achieve the above-mentioned objectives, the present invention employs the following technical solution:

[0008] A video stabilization method for visual inspection of power transmission lines, characterized by the following steps:

[0009] Step S1: Collect outdoor environment images captured by the data acquisition unit and label non-moving objects in them; obtain a dataset of labeled images of non-moving objects.

[0010] Step S2: Select the labeled image dataset as the original image data; remove labeled object images with an area smaller than the set removal area from the original image data, and use the removed labeled image dataset as the target image data; divide the target image data into training image dataset and test image dataset;

[0011] Step S3: Construct the original target detection model for non-moving object detection;

[0012] Step S4: Use the original image data and the training image dataset of the target image data to train the original target detection model for non-moving object detection to obtain the final target detection model;

[0013] Step S5: The real-time sampling data acquisition unit monitors and captures video within the monitoring range, extracts continuous current frames and reference frames from the video, inputs the current frame and reference frame images into the final target detection model, and uses the final target detection model to detect non-moving objects.

[0014] Step S6: Sort the non-moving object targets detected in both the current frame and the reference frame according to their confidence scores, and select the non-moving object targets with confidence scores higher than the set confidence scores as motion estimation targets; use the block matching method to calculate the motion vector of each motion estimation target between the current frame and the reference frame;

[0015] Step S7: Calculate the average value of the motion vector of each motion estimation target, and use the average value of the motion vector of each motion estimation target as the global motion vector;

[0016] Step S8: Use the Kalman Filter algorithm to process the global motion vector. The process is performed to obtain the global motion vector to be compensated.

[0017] Step S9: Perform motion compensation, which can be expressed as:

[0018] Here, is a pixel in the image before compensation, and is a pixel in the image after compensation.

[0019] In some embodiments of this application, the non-moving object includes towers, houses, chimneys, and / or bridges.

[0020] In some embodiments of this application, the outdoor environment image is a 2K image, and the culling area is set to 60 pixels × 60 pixels.

[0021] In some embodiments of this application, constructing a raw object detection model for non-moving object detection includes replacing the backbone network in Faster R-CNN with a ResNet50 model instead of a VGG-16 model.

[0022] In some embodiments of this application, after motion compensation is completed, image inpainting is performed using the mosaic method to obtain a stable video.

[0023] In some embodiments of this application, training a constructed original object detection model for non-moving object detection using the original image data and the training image dataset of the target image data to obtain the final object detection model includes the following steps:

[0024] Step S41: Input the images from the training image dataset in the target image data into the original target detection model with ResNet50 as the backbone network. After feature extraction by ResNet50, perform a 3 × 3 convolution operation on the feature map output by the Conv_4 layer of ResNet50 to generate a feature map with 256 channels and a size of (H / 16) × (W / 16).

[0025] Step S42: After the 3 × 3 convolution operation, perform two 1 × 1 convolution operations on the feature map ((H / 16) × (W / 16) × 256) to predict the positive and negative shapes of the predicted bounding box and the coordinate offset of the predicted bounding box, where 256 is the number of channels.

[0026] Step S43: Generate suggestions in the suggestion layer, and perform correction, elimination and NMS (Non-Maximum Suppression) filtering on predicted bounding boxes with positive attributes, and finally select a set number of target class predicted bounding boxes that are greater than the threshold p.

[0027] Step S44: Perform region of interest pooling operation to map the selected target class prediction bounding boxes onto the feature map output by the Conv_5 layer of ResNet50, and find the part of the feature map corresponding to each selected target class prediction bounding box;

[0028] Step S45: Perform target bounding box regression and target classification on the feature map output by the region of interest pooling operation;

[0029] Step S46: Determine whether the original target detection model has reached the first convergence level;

[0030] Step S47: When the original object detection model reaches the first convergence level, stop training and save the original object detection model as a target training model;

[0031] Step S48: Input the image from the original image data into the first training model output in step S47, extract features through ResNet50, repeat steps S41 to S45 above, and determine whether the first training model has reached the second convergence level.

[0032] Step S49: When the target training model fully converges, stop training and save the target training model as a secondary target training model;

[0033] Step S50: The secondary training model output from the original image data input step is fine-tuned to adapt the parameters to the target image data until the secondary training model converges again. Then, the fine-tuning is stopped and the secondary target training model is saved as the final target detection model.

[0034] In some embodiments of this application, in step S42, the 1 × 1 convolution result is divided into two branches. The upper branch generates a feature map with (H / 16) × (W / 16) × 18 channels. The 18 channels mean that there are 9 predicted bounding boxes at each pixel position, and each predicted bounding box corresponds to the probability of being a target and the probability of being background. The lower branch generates a feature map with (H / 16) × (W / 16) × 36 channels. The 36 channels mean that there are 9 predicted bounding boxes at each pixel position, and each predicted bounding box corresponds to four independent feature values, which are respectively the offset tx of the x-coordinate of the center point, the offset ty of the y-coordinate of the center point, the horizontal offset tw of the bounding box, and the vertical offset th of the bounding box.

[0035] In some embodiments of this application, three sets of aspect ratios (ratio = [0.5, 1, 2]) and three scales (scale = [8, 16, 32]) are defined to form the prediction bounding box.

[0036] In some embodiments of this application, determining whether the original object detection model has reached the first degree of convergence is to determine whether it has reached slight convergence: determining whether, in each period of the training process, there exists an absolute value of the difference between the loss generated by the previous batch size training and the loss generated by the current batch size training that is less than a first set loss value.

[0037] In some embodiments of this application, the first set loss value is 0.1.

[0038] Compared with the prior art, the advantages and positive effects of the present invention are:

[0039] Compared with existing technologies, the video stabilization method for power transmission line visualization inspection scenarios provided in this application is particularly suitable for such scenarios. This method uses a trained final object detection model to detect four types of non-moving objects in the surveillance video: towers, buildings, chimneys, and bridges. Motion vectors are then calculated using these non-moving objects, and global motion estimation is performed. This eliminates video jitter while ensuring the authenticity and reliability of normally moving objects, especially conductors, reducing inaccuracies in motion compensation caused by the superposition of moving objects and video jitter. Furthermore, the final object detection model can provide the positions of non-moving objects, enabling the block matching algorithm to focus more accurately on these positions, improving the accuracy of motion vectors and thus enhancing the subsequent stabilization effect.

[0040] Other features and advantages of the present invention will become clearer after reading the detailed embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating one embodiment of the video stabilization method for visual inspection of power transmission lines provided in this application;

[0043] Figure 2 The flowchart illustrates how the video stabilization method for power transmission line visualization detection scenarios provided in this application uses the original image data and the target image data training image dataset to train the original target detection model for non-moving object detection, thus obtaining the final target detection model.

[0044] Figure 3This is a schematic diagram of the original object detection model with the ResNet50 model as the backbone network. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that in the description of this invention, the terms "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," which indicate directional or positional relationships, are based on the directional or positional relationships shown in the accompanying drawings. These are merely for ease of description and do not indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation; therefore, they should not be construed as limitations on this invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0047] Existing motion estimation algorithms typically use local motion vectors to approximate global motion vectors. The selected or calculated regions are random and lack fixed properties, leading to inaccurate global motion estimation. In particular, image stabilization may weaken conductor jitter, further reducing the accuracy of conductor galloping detection. This paper designs and provides a video image stabilization method for visual detection of transmission lines. Conductor galloping refers to a low-frequency, large-amplitude self-excited vibration of non-circular cross-section transmission lines caused by wind. Conductor galloping is a common phenomenon in transmission and distribution lines, occurring at various voltage levels under suitable conditions. It possesses enormous energy, and the galloping trajectory is elliptical within a cross-section perpendicular to the conductor axis. Peak values ​​of antinodes on the conductor range from tens of centimeters to twelve or thirteen meters, with maximum amplitudes reaching 5 to 300 times the diameter of the transmission line. Prolonged, high-energy, and large-amplitude conductor galloping will increase the stress on conductors, fittings, crossarms, poles, etc., causing direct damage or fatigue failure to conductors, fittings, and even poles, reducing their service life.

[0048] Transmission line condition monitoring devices can detect conductor galloping. These devices include data acquisition units, data monitoring terminals, transmission line condition monitoring equipment, and a transmission line condition monitoring master station system. The data acquisition units are information measurement devices based on various principles, installed on conductors, ground wires (including optical fiber composite overhead ground wire, OPGW), insulators, towers, and foundations. They transmit measurement information to the data monitoring terminal via a communication network and respond to the terminal's commands. The data monitoring terminal collects information from each data acquisition unit, stores and processes it on-site, and exchanges information with the master station system. The transmission line condition monitoring equipment is a measurement device capable of collecting real-time information on the transmission line itself, meteorological environment, and corridor conditions, and transmitting this information to the master station system via a communication network. The transmission line condition monitoring master station system is a computer system capable of accessing condition monitoring information from various transmission equipment, and performing centralized storage, unified processing, and application. The master station system typically includes an information access front-end, a centralized database, data services, data processing, and various condition monitoring application modules.

[0049] The video stabilization method for power transmission line visualization detection scenarios provided in this application can be implemented by any one of a data acquisition unit, a data monitoring terminal, a power transmission line condition monitoring device, and a master station system. In a preferred embodiment, considering data processing capabilities, it can be implemented by the master station system, for example.

[0050] The video stabilization method for power transmission line visualization inspection scenarios provided in this application includes, for example: Figure 1 The steps are shown.

[0051] Step S1: Collect outdoor environment images captured by the data acquisition unit and label four types of non-moving objects: poles, houses, chimneys, and bridges; obtain the labeled image dataset of the four types of non-moving objects: poles, houses, chimneys, and bridges.

[0052] In some optional embodiments of this application, the outdoor environment image is a 2K image, which consists of 2048×1080 pixels, where 2048 represents the number of pixels in the horizontal direction and 1048 represents the number of pixels in the vertical direction. 2D bounding boxes are used to label four types of non-moving objects in the outdoor environment image: poles, buildings, chimneys, and bridges. When labeling chimneys, only the chimney itself is labeled, excluding smoke or water vapor. For bridges, only images taken from a side view of the bridge in the outdoor environment image are labeled; bridges taken from a top-down view are not labeled. The labeling can be done by professional data labelers.

[0053] Step S2: Select the labeled image dataset as the original image data; remove labeled object images with an area smaller than the set removal area from the original image data, and use the removed labeled image dataset as the target image data; divide the target image data into training image dataset and test image dataset.

[0054] The removal area is set as the critical area for recognizing the object's feature model. In some embodiments of this application, for 2K images, the removal area is set to consist of 60 × 60 pixels, where 60 represents the number of pixels in the horizontal and vertical directions. Statistical analysis of each non-moving object in the original image data reveals that for 2K images, non-moving objects within bounding boxes with an area smaller than 60 × 60 pixels exhibit difficulty in feature model recognition. Therefore, setting the removal area to consist of 60 × 60 pixels results in high-quality target image data.

[0055] Step S3: Construct the original target detection model for non-moving object detection.

[0056] Building a target detection model for non-moving object detection specifically includes the following steps:

[0057] Replace the VGG-16 model with the ResNet50 model in Faster R-CNN.

[0058] In some embodiments of this application, the object detection model is an improvement upon Faster R-CNN (Region-based Convolutional Neural Network). The basic structure of a traditional Faster R-CNN includes a feature extraction part, a Region Proposal Network (RPN), a Proposal Layer, and a Region of Interest (ROI) pooling part. The feature extraction part uses convolution and pooling to extract feature maps from the image to be processed. The Region Proposal Network obtains the approximate location of the target from the feature map through network training. The Proposal Layer uses the approximate location obtained by the Region Proposal Network to continue training and obtain a more precise location. The Region of Interest pooling part uses the more precise location obtained to extract the target to be classified from the feature map and pools it into a fixed-length dataset.

[0059] Traditional feature extraction uses the VGG-16 (Visual Geometry Group) model as its backbone network, which includes 16 layers: 13 convolutional layers and 3 fully connected layers. The VGG-16 model has strong fitting capabilities, predicting text boxes and confidence scores on feature maps generated after downsampling the input image by a factor of 16. However, its feature extraction capabilities are limited and do not adequately match the feature extraction requirements of this application. To improve feature extraction capabilities, in some embodiments of this application, the backbone network in Faster R-CNN is replaced by a deep residual network, namely the ResNet50 model, instead of the VGG-16 model.

[0060] Step S4: Use the original image data and the training image dataset of the target image data to train the original target detection model for non-moving object detection to obtain the final target detection model.

[0061] The training process is described below, including the following: Figure 2 The steps are shown.

[0062] The size of the images in the training image dataset can be denoted as H × W, where H represents the number of pixels in the horizontal direction and W represents the number of pixels in the vertical direction. For example, H is 2048 pixels and W is 1080 pixels.

[0063] Step S41: As Figure 3 As shown, images from the training image dataset in the target image data are input into the original object detection model using ResNet50 as the backbone network. ResNet50 performs feature extraction. A 3 × 3 convolution operation is performed on the feature map output from the Conv_4 layer (conv is short for convolution) of ResNet50, generating a feature map with 256 channels and a size of (H / 16) × (W / 16), which can be represented as ((H / 16) × (W / 16) × 256). The processing steps before the Conv_4 layer are consistent with the traditional ResNet50 model and will not be repeated here. Figure 3 The diagram shows Conv_1, Conv_2, and Conv_3 layers. In the diagram, conv represents convolution, relu represents the activation function, fullconnection represents full connection, reshape represents the reconstruction layer, softmax represents the softmax function, proposal represents the proposal layer, ROI pooling represents region of interest pooling, bbox represents bounding box regression, and class represents the classifier. The definitions of these terms are well-known in the field of neural network technology and will not be elaborated upon here.

[0064] Step S42: After the 3 × 3 convolution operation, perform two 1 × 1 convolution operations on the ((H / 16) × (W / 16) × 256) feature map. The 1 × 1 convolution result is divided into two branches. The upper branch generates a (H / 16) × (W / 16) × 18 channel feature map, where 18 channels means that there are 9 predicted bounding boxes (anchors) at each pixel position. Each predicted bounding box (anchor) corresponds to two independent feature values, which are the probability of being the target and the probability of being the background, respectively. The lower branch generates a (H / 16) × (W / 16) × 36 channel feature map, where 36 channels means that there are 9 predicted bounding boxes (anchors) at each pixel position. Each predicted bounding box (anchor) corresponds to four independent feature values, which are the x-coordinate offset tx of the center point, the y-coordinate offset ty of the center point, the horizontal offset tw of the bounding box, and the vertical offset th of the bounding box.

[0065] Three aspect ratios (ratio = [0.5, 1, 2]) and three scales (scale = [8, 16, 32]) are defined to construct the predicted bounding boxes (anchors). These two sets of parameters can be combined to create nine different shapes and sizes of bounding boxes. For the predicted bounding boxes generated by the upper branch, the highest predicted probability (p) is determined by the relatively larger probability between the probability of being a target and the probability of being background. The category corresponding to this probability is the category of the predicted bounding box, i.e., target-class predicted bounding boxes and background-class predicted bounding boxes.

[0066] After the above steps, the positive or negative sign of the predicted bounding box can be predicted (e.g., the predicted bounding box for the target class is positive, and the predicted bounding box for the background class is negative); at the same time, the coordinate offset of the predicted bounding box can also be predicted.

[0067] Step S43: Generate suggestions in the suggestion layer, and perform correction, elimination and NMS (Non-Maximum Suppression) filtering on predicted bounding boxes with positive attributes, and finally select a set number of target class predicted bounding boxes that are greater than the threshold p.

[0068] Step S44: Perform Region of Interest (ROI) pooling to map the selected target class prediction bounding boxes onto the feature map output by the Conv_5 layer of ResNet50, and find the part of the feature map corresponding to each selected target class prediction bounding box.

[0069] Step S45: Regress the target bounding box and classify the target on the feature map output by the Region of Interest (ROI pooling) operation.

[0070] Step S46: Determine whether the original target detection model has reached the first convergence level.

[0071] The process determines whether the original object detection model has reached the first degree of convergence, which represents a state of slight convergence. More specifically, it determines whether, in each epoch of the training process (the process of passing the complete training image dataset through the original object detection model once and returning once), the absolute value of the difference between the loss generated by the previous batch size training and the loss generated by the current batch size training is less than a first set loss value (for example, 0.1). Slight convergence refers to an interval in the training process of the original object detection model where the loss continues to decrease, but the decrease is gradual, and the absolute value of the difference between the loss of the previous epoch and the loss of the current epoch is less than 0.1.

[0072] Step S47: When the original object detection model reaches the first convergence level, i.e. slight convergence, stop training and save the original object detection model as a target training model.

[0073] Step S48: Input the image from the original image data into the first training model output in step S47, extract features through ResNet50, repeat steps S41 to S45, and determine whether the first training model has reached the second convergence level.

[0074] To determine whether a trained model has reached the second degree of convergence, which represents the state of complete convergence, i.e. the ideal convergence state.

[0075] Step S49: When the target training model fully converges, stop training and save the target training model as a secondary target training model.

[0076] Step S50: Input the image from the original image data into the secondary training model output in step S49, and fine-tune the secondary training model to adapt the parameters to the target image data until the secondary training model converges again. Stop fine-tuning and save the secondary target training model as the final object detection model. The final object detection model can be used to perform detection using a test image dataset.

[0077] Step S5: The real-time sampling data acquisition unit monitors and captures video within the monitoring range, extracts continuous current frames and reference frames from the video, and inputs the current frame and reference frame images into the final target detection model. The final target detection model is then used to detect four types of non-moving objects: poles, buildings, chimneys, and bridges.

[0078] Step S6: Sort the non-moving object targets (towers, houses, chimneys, and bridges) detected in both the current frame and the reference frame according to their confidence scores, and select the non-moving object targets with confidence scores higher than the set confidence scores (for example, targets with confidence scores greater than 0.7) as motion estimation targets; use the block matching method to calculate the motion vector of each motion estimation target between the current frame and the reference frame; the block matching method uses the model disclosed in the prior art, and this application has not made any improvements to this part, so it will not be described in detail here.

[0079] Step S7: Calculate the average value of the motion vectors of each motion-estimated target, and use the average value of the motion vectors of each motion-estimated target as the global motion vector.

[0080]

[0081] in Let represent the global motion vector, n represent the number of motion estimation targets, and represent the motion vector of the motion estimation target.

[0082] Step S8: Use the Kalman Filter algorithm to process the global motion vector. The process is performed to obtain the global motion vector to be compensated.

[0083] Step S9: Perform motion compensation. The motion compensation process can be expressed as follows:

[0084]

[0085] in, This is a pixel in the image before compensation, and this is a pixel in the image after compensation. After motion compensation, the image is repaired using the mosaic method to obtain the final stabilized video.

[0086] Compared with existing technologies, the video stabilization method for power transmission line visualization inspection scenarios provided by the above embodiments of this application is particularly suitable for power transmission line visualization inspection scenarios. This method uses a trained final target detection model to detect four types of non-moving objects in the monitored video: towers, buildings, chimneys, and bridges. Motion vectors are then calculated using these non-moving objects, and global motion estimation is performed. This eliminates video jitter while ensuring the authenticity and reliability of normally moving objects, especially conductors, reducing inaccuracies in motion compensation caused by the superposition of moving objects and video jitter. Furthermore, the final target detection model can provide the positions of non-moving objects, enabling the block matching algorithm to focus more accurately on the corresponding positions, improving the accuracy of motion vectors, and thus enhancing the subsequent stabilization effect.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions claimed by the present invention.

Claims

1. A video stabilization method for visual inspection scenarios of power transmission lines, characterized in that, The method comprises the following steps: Step S1: Collecting outdoor environment images monitored and photographed by a data acquisition unit, and labeling non-moving objects in the images; and obtaining a labeled image dataset of the non-moving objects; Step S2: Selecting the labeled image dataset as original image data; removing labeled object images with a labeled box area of the non-moving objects less than a set removal area from the original image data, and taking the removed labeled image dataset as target image data; and dividing the target image data into a training image dataset and a test image dataset; Step S3: Constructing an original target detection model for non-moving object detection; Step S4: Training the constructed original target detection model for non-moving object detection using the original image data and the training image dataset of the target image data to obtain a final target detection model; Step S5: Real-time sampling of a video of a monitoring range monitored and photographed by the data acquisition unit, extracting continuous current frames and reference frames from the video, inputting the current frame and the reference frame image into the final target detection model, and detecting non-moving object targets by using the final target detection model; Step S6: Sorting the non-moving object targets detected in the current frame and the reference frame according to confidence levels, and selecting non-moving object targets with a confidence level higher than a set confidence level as motion estimation targets; Using a block matching method to calculate the motion vector of each motion estimation target between the current frame and the reference frame; Step S7: Solving the average value of the calculated motion vector of each motion estimation target, and taking the average value of the calculated motion vector of each motion estimation target as a global motion vector; Step S8: processing the global motion vector to be compensated using a Kalman Filter algorithm ;​ Step S9: Perform motion compensation, which can be expressed as ; wherein, is one pixel point in the compensated image before, is one pixel point in the compensated image after.

2. The video stabilization method for transmission line visual inspection scene according to claim 1, characterized in that, The non-moving objects include towers, houses, chimneys, and / or bridges.

3. The method for video stabilization of transmission line visual inspection scenarios according to claim 1, characterized in that, The outdoor environment images are 2K images, and the set removal area is 60 pixels x 60 pixels.

4. The video stabilization method for transmission line visual inspection scene according to claim 1, wherein, The original target detection model for non-moving object detection is constructed by replacing the backbone network in Faster R-CNN with a ResNet50 model.

5. The method for video stabilization of transmission line visual inspection scenes according to claim 1, wherein, After motion compensation, the image is repaired using a mosaic method to obtain a stabilized video.

6. The video stabilization method for visual detection of a power transmission line scene according to any one of claims 1 to 5, characterized in that: The original target detection model for non-moving object detection is trained using the original image data and the training image dataset of the target image data to obtain a final target detection model, which comprises the following steps: Step S41: Inputting images in the training image dataset in the target image data into the original target detection model with ResNet50 as a backbone network, performing feature extraction by ResNet50, performing a 3x3 convolution operation on a feature map output by a Conv_4 layer of ResNet50, and generating a feature map with a channel number of 256 and a size of (H / 16) x (W / 16), wherein H represents the number of pixels in the horizontal direction, and W represents the number of pixels in the vertical direction; Step S42: After the 3 × 3 convolution operation, two 1 × 1 convolution operations are respectively performed on the feature map of ((H / 16) × (W / 16) × 256) to predict the positive and negative of the prediction bounding box and the coordinate offset of the prediction bounding box, wherein 256 is the number of channels; Step S43: generating proposals in the proposal layer, correcting, rejecting and NMS (Non Maximum Suppression) filtering the prediction bounding boxes with positive attributes, and finally selecting a certain number of target class prediction bounding boxes greater than a threshold p; Step S44: performing a region of interest pooling operation to map the selected target class prediction bounding boxes to the feature map output by the Conv_5 layer of the ResNet50, and finding the part of the feature map corresponding to each selected target class prediction bounding box; Step S45: performing regression of the target prediction bounding box and classification of the target on the feature map output by the region of interest pooling operation; Step S46: determining whether the original target detection model reaches a first convergence degree; Step S47: when the original target detection model reaches the first convergence degree, stopping training and saving the original target detection model as a first target training model; Step S48: inputting the images in the original image data into the first training model output by step S47, performing feature extraction by the ResNet50, repeating the above steps S41 to S45, and determining whether the first target training model reaches a second convergence degree; Step S49: when the first target training model completely converges, stopping training and saving the first target training model as a second target training model; Step S50: inputting the images in the original image data into the second training model output by step S49, fine-tuning the second training model to adapt the parameters to the target image data, until the second training model converges again, stopping fine-tuning and saving the second target training model as a final target detection model.

7. The video stabilization method for power line visual detection scene according to claim 6, wherein: In step S42, the 1 × 1 convolution result is divided into two branches, the upper branch generates a feature map of (H / 16) × (W / 16) × 18 channels, and the 18 channels refer to: each pixel position has 9 prediction bounding boxes, and each prediction bounding box corresponds to a probability of being a target and a probability of being a background; the lower branch generates a feature map of (H / 16) × (W / 16) × 36 channels, and the 36 channels refer to: each pixel position has 9 prediction bounding boxes, and each prediction bounding box corresponds to four independent feature values, which are respectively the x coordinate offset tx of the center point, the y coordinate offset ty of the center point, the horizontal direction offset tw of the bounding box and the vertical direction offset th of the bounding box.

8. The video stabilization method for power line visual detection scene according to claim 7, wherein: Three aspect ratios ratio = [0.5, 1, 2] and three scales scale = [8, 16, 32] are defined to constitute the prediction bounding box.

9. The video stabilization method for transmission line visual detection scene according to claim 8, characterized in that: The judgment of whether the original target detection model reaches the first convergence degree is to judge whether it reaches slight convergence: whether the absolute value of the difference between the loss generated by the previous batch size training and the loss generated by the current batch size training in each period of the training process is less than the first set loss value.

10. The video stabilization method for transmission line visual inspection scene according to claim 9, characterized in that, The first set loss value is 0.1.

Images