A method and apparatus for wire tracking
By constructing a lightweight conductor detection model and optimizing the hybrid loss function, combined with bidirectional optical flow tracking to confirm the observation point, the problems of inaccurate conductor contour detection and large tracking deviation were solved, and efficient conductor tracking in complex backgrounds was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG SENTER ELECTRONICS
- Filing Date
- 2022-07-12
- Publication Date
- 2026-04-10
AI Technical Summary
Existing traverse tracking methods suffer from inaccurate traverse contour detection and large tracking deviations, making them particularly difficult to track effectively in complex environments.
A lightweight wire detection model is constructed using depthwise separable convolution. A multi-receptive field fusion module with holed depthwise separable convolution is combined with the model to optimize the model using a hybrid loss function of Focal loss and Dice loss. Observation points are confirmed by bidirectional optical flow tracking, and invalid observation points are eliminated to improve tracking robustness.
It improves the accuracy and robustness of conductor tracking, enabling effective detection and tracking of transmission lines in complex environments.
Smart Images
Figure CN115272920B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of electric power, and in particular to a wire tracking method and device. BACKGROUND
[0002] Target tracking is an important research problem in the field of computer vision. In the field of target tracking, the research on the tracking technology of power transmission wires has important significance for the safe maintenance of power transmission channels. There are two important technical points for wire tracking technology. One is to accurately detect and identify the wire, which plays a very important role in subsequent wire tracking. The other is to select appropriate and accurate tracking points for accurate tracking of the wire.
[0003] For wire detection and tracking in power transmission channel images, existing methods often rely on edge detection and direct detection to detect wires. However, due to the very long and thin power transmission wires and the often low imaging quality of long-range wires, when the background of the power transmission channel is not only the sky (for example, the background is mountains or large buildings), such methods cannot cope with complex wire backgrounds and cannot completely and effectively extract the wire contour, resulting in failure of subsequent wire tracking. In addition, traditional key point tracking can use sparse optical flow algorithm (Lucas Kanade, LK) to achieve tracking, but when tracking the observation points of the wires in the power transmission channel scene, due to the very similar observation point positions and pixel values of different points on the wire, directly using the LK algorithm may cause local observation point drift or large prediction deviation, which is not conducive to wire monitoring in the power transmission channel. SUMMARY
[0004] The embodiments of the present application provide a wire tracking method and device, which are used to solve the technical problem that the existing wire tracking method detects inaccurate wire contours and has large tracking deviation.
[0005] The embodiments of the present application adopt the following technical solutions:
[0006] On the one hand, the embodiments of the present application provide a wire tracking method, which includes: constructing and training a wire detection model; optimizing the trained wire detection model through a preset hybrid loss function; identifying a first frame of video image in a power transmission channel video through the optimized wire detection model to obtain a wire contour image corresponding to the first frame of video image; determining a set of wire observation points in the first frame of video image according to the wire contour image; eliminating invalid wire observation points in the set of wire observation points to obtain a set of valid observation points; and tracking the wire in the power transmission channel video according to the set of valid observation points.
[0007] The embodiment of the application only detects the first frame image through the conductor detection model, and then constructs a conductor tracking observation point through a conductor and parallel line intersection strategy; the conductor in the power transmission channel video is tracked according to an effective observation point set; finally, the tracking robustness is ensured through a bidirectional optical flow prediction strategy. The fast tracking of the application mainly reflects that the model is only run for one image. If the model is run for each image, it will be very slow.
[0008] In a feasible implementation, the conductor detection model is constructed and trained, specifically including: constructing a multi-level residual feature extraction module; wherein the multi-level residual feature extraction module is composed of multiple residual modules, each residual module including a depthwise separable convolution layer, a batch normalization layer, and an activation layer; constructing a multi-receptive field feature extraction module; wherein the multi-receptive field feature extraction module includes a dilated depthwise separable convolution layer and a global pooling layer; constructing a feature fusion module to fuse the multi-receptive field feature map output by the multi-receptive field feature extraction module with the shallow feature map extracted by the first residual module in the multi-level residual feature extraction module to obtain a feature fusion image; wherein the shallow feature map is upsampled to the same size of the multi-receptive field feature map before feature fusion; constructing a decoding prediction module to decode and predict the feature fusion image to obtain an output image; the multi-level residual feature extraction module, the multi-receptive field feature extraction module, the feature fusion module, and the decoding prediction module constitute the conductor detection model; and the conductor detection model is trained through a training data set.
[0009] In a feasible implementation, the multi-level residual feature extraction module is constructed, specifically including: connecting a first depthwise separable convolution layer, a batch normalization layer, an activation layer, and a second depthwise separable convolution layer in sequence; wherein the first depthwise separable convolution layer is used for depthwise separable convolution processing on an input image and inputs the processing result into the batch normalization layer for normalization processing; the activation layer is used for activating preset important features in the output features of the batch normalization layer and suppressing preset unimportant features, and the second depthwise separable convolution layer is used for depthwise separable convolution processing on the features output by the activation layer again; an addition layer is constructed to add the input image and the output result of the second depthwise separable convolution layer; the first depthwise separable convolution layer, the batch normalization layer, the activation layer, the second depthwise separable convolution layer, and the addition layer constitute a residual module; and multiple residual modules are connected to constitute the multi-level residual feature extraction module.
[0010] In a feasible implementation, a multi-receptive field feature extraction module is constructed, specifically including: a first hollow depth separation convolutional layer, a second hollow depth separation convolutional layer, a third hollow depth separation convolutional layer, a fourth hollow depth separation convolutional layer, and a global pooling layer are respectively constructed to perform multi-receptive field feature extraction on an input image; the first hollow depth separation convolutional layer is a 1*1 hollow depth separation convolutional layer; the second hollow depth separation convolutional layer is a 3*3 hollow depth separation convolutional layer with a hollow rate of 1; the third hollow depth separation convolutional layer is a 3*3 hollow depth separation convolutional layer with a hollow rate of 3; the fourth hollow depth separation convolutional layer is a 3*3 hollow depth separation convolutional layer with a hollow rate of 5; a cascade layer is constructed to splice the extracted multi-receptive field features; a 1*1 convolutional block is constructed to fuse the spliced multi-receptive field features; the first hollow depth separation convolutional layer, the second hollow depth separation convolutional layer, the third hollow depth separation convolutional layer, the fourth hollow depth separation convolutional layer, the global pooling layer, the cascade layer, and the 1*1 convolutional block constitute the multi-receptive field feature extraction module.
[0011] In a feasible implementation, the trained wire detection model is optimized by a preset hybrid loss function, specifically including: a Focal loss loss function is constructed according to the prediction accuracy of the wire contour prediction image output by the trained wire detection model; a Dice loss loss function is constructed according to the similarity between the wire contour prediction image and the corresponding training sample; the hybrid loss function loss is obtained according to loss=w1*Focalloss+w2*Diceloss; wherein ω1 is the loss balance factor of the Focal loss loss function, and ω2 is the loss balance factor of the Dice loss loss function; the trained wire detection model is parameter-optimized until the hybrid loss function converges, and an optimized wire detection model is obtained.
[0012] In an implementation, the set of wire observation points is determined according to the wire contour image and the first frame of video image, specifically including: determining a minimum circumscribed rectangle of a region where the wire is located in the first frame of video image according to wire features in the wire contour image; drawing equidistant parallel lines within the minimum circumscribed rectangle of the video image; determining coordinates of intersection points between the parallel lines and the wire in the video image to obtain a set of local wire tracking points; generating a set of candidate tracking points in the minimum circumscribed rectangle region by a Shi-Tomasi corner detection algorithm; combining the set of local wire tracking points and the set of candidate tracking points to obtain a set of wire candidate observation points; generating a wire mask according to a wire track in the wire contour image, and screening points in the set of wire candidate observation points that exist on the wire track according to the wire mask to obtain the set of wire observation points; and storing coordinates of all wire observation points in the set of wire observation points.
[0013] In an implementation, invalid wire observation points are removed from the set of wire observation points to obtain a set of valid observation points, specifically including: taking points in the set of wire observation points in the first frame of video image as tracking points, and performing forward optical flow tracking on the tracking points to obtain a set of predicted wire observation points in the second frame of video image; taking points in the set of predicted wire observation points in the second frame of video image as tracking points, and performing reverse optical flow tracking on the tracking points to obtain a set of predicted wire observation points in the first frame of video image; calculating Euler distances of corresponding points in the set of wire observation points and the set of predicted wire observation points in the first frame of video image; if the Euler distance is greater than a preset threshold, determining that the corresponding wire observation point is an invalid observation point and removing it; and removing all invalid observation points in the set of wire observation points in the first frame of video image to obtain the set of valid observation points.
[0014] In an implementation, the wire in the power transmission channel video is tracked according to the set of valid observation points, specifically including: performing forward optical flow tracking according to the set of valid observation points in the first frame of video image to obtain a set of predicted valid observation points in the second frame of video image; calculating a position and a size of a tracking box of the second frame of video image according to the set of predicted valid observation points in the second frame of video image, and similarly calculating a position and a size of a tracking box of each frame of video image to track the wire in the power transmission channel video.
[0015] In a feasible implementation, before the conductor detection model is constructed and trained, the method further comprises: collecting a preset number of power transmission channel images containing power transmission conductors; labeling a conductor area mask for each power transmission conductor in the power transmission channel images through Labelme software to obtain a labeled image corresponding to each power transmission channel image; and grouping the preset number of power transmission channel images and the corresponding labeled images to form a training data set of the conductor detection model.
[0016] In another aspect, the embodiments of the present application also provide a conductor tracking device, which comprises: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions capable of being executed by the at least one processor, so that the at least one processor can execute the conductor tracking method according to any of the above-mentioned embodiments.
[0017] In the conductor tracking method and device provided by the embodiments of the present application, a lightweight conductor detection model based on deep split convolution is first proposed, which extracts features through deep split convolution while maintaining low computational complexity. To solve the problem of conductor detection in complex background, a multi-receptive field fusion module of hollow deep split convolution is proposed to improve the ability of the model to distinguish conductors from background. To solve the problem of small conductor ratio, a hybrid loss function based on Focalloss and Dice loss is proposed to optimize the conductor detection model and enhance conductor detection. Finally, to solve the problem of tracking drift of conductor observation points caused by one-way optical flow, a new observation point selection method is proposed, and a bidirectional optical flow tracking confirmation method is used to improve the robustness of optical flow tracking. Overall, the accuracy of conductor tracking is improved. BRIEF DESCRIPTION OF DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor. In the drawings:
[0019] Figure 1 A conductor tracking method flowchart provided by the embodiments of the present application;
[0020] Figure 2 A conductor detection model schematic diagram provided by the embodiments of the present application;
[0021] Figure 3 A residual module schematic diagram provided by the embodiments of the present application;
[0022] Figure 4 A multi-receptive field feature extraction module provided for an embodiment of the present application is shown in the figure;
[0023] Figure 5 A wire observation point construction diagram provided for an embodiment of the present application is shown in the figure;
[0024] Figure 6 A structure diagram of a wire tracking device provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION
[0025] In order to enable those skilled in the art to better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.
[0026] An embodiment of the present application provides a wire tracking method, as shown in the figure, the wire tracking method specifically includes steps S101-S106: Figure 1
[0027] S101, constructing and training a wire detection model.
[0028] Specifically, a multi-level residual feature extraction module is constructed. The multi-level residual feature extraction module is composed of multiple residual modules, and each residual module includes a depthwise separable convolution layer, a batch normalization layer and an activation layer.
[0029] Further, a multi-receptive field feature extraction module is constructed. The multi-receptive field feature extraction module includes a dilated depthwise separable convolution layer and a global pooling layer.
[0030] Further, a feature fusion module is constructed. The multi-receptive field feature maps output by the multi-receptive field feature extraction module are fused with the shallow feature maps extracted by the first residual module in the multi-level residual feature extraction module to obtain a feature fusion image. Before feature fusion, the shallow feature maps are upsampled to the same size of the multi-receptive field feature maps.
[0031] Further, a decoding prediction module is constructed to decode and predict the feature fusion image to obtain an output image. The multi-level residual feature extraction module, the multi-receptive field feature extraction module, the feature fusion module and the decoding prediction module constitute the wire detection model.
[0032] As a feasible implementation manner, Figure 2 A wire detection model provided for an embodiment of the present application is shown in the figure. As shown in the figure, Figure 2 As shown, the original image is input into the multi-level residual feature extraction module composed of 3 residual modules to perform multi-layer feature map extraction, and the extracted deep layer feature map is input into the multi-receptive field feature extraction module to perform multi-receptive field feature extraction and fusion. Then, the shallow layer feature map output by the first residual module in the multi-level residual feature extraction module is up-sampled until the size is the same as the multi-receptive field feature map output by the multi-receptive field feature extraction module, and then the up-sampled shallow layer feature map and the multi-receptive field feature map are fused by the feature fusion module to obtain a feature fusion image. Finally, the feature fusion image is decoded by the decoding prediction module to obtain a predicted wire contour image.
[0033] In one embodiment, the multi-level residual feature extraction module, the multi-receptive field feature extraction module and the feature fusion module are the encoding modules of the wire detection model, and the decoding prediction module is the decoding module of the wire detection model.
[0034] As a feasible implementation, the multi-level residual feature extraction module is constructed, specifically including: connecting a first depth separation convolutional layer, a batch normalization layer, an activation layer and a second depth separation convolutional layer in sequence; wherein the first depth separation convolutional layer is used for depth separation convolutional processing on the input image, and the processing result is input into the batch normalization layer for normalization processing; the activation layer is used for activating preset important features in the output features of the batch normalization layer, and suppressing preset unimportant features, and the second depth separation convolutional layer is used for depth separation convolutional processing on the features output by the activation layer. An addition layer is constructed to add the input image and the output result of the second depth separation convolutional layer. The first depth separation convolutional layer, the batch normalization layer, the activation layer, the second depth separation convolutional layer and the addition layer constitute a residual module. A plurality of residual modules are connected to constitute a multi-level residual feature extraction module.
[0035] In one embodiment, Figure 3 A residual module schematic diagram provided for the embodiments of the present application is as follows: Figure 3As shown, the input image is fed into a first depthwise segregating convolutional layer for depthwise segregating convolution processing, and the processing result is fed into a batch normalization layer for normalization processing. Then, an activation layer activates preset important features in the output features of the batch normalization layer and suppresses preset unimportant features. A second depthwise segregating convolutional layer then performs depthwise segregating convolution processing on the features output by the activation layer. Finally, the input image is added to the output of the second depthwise segregating convolutional layer to obtain the output feature map. The depthwise segregating convolutional layer reduces the number of parameters and accelerates computation by splitting the ordinary convolution process into depthwise convolution and pointwise convolution. The activation layer activates some features and suppresses some unimportant features on the feature map, and the batch normalization layer normalizes the activated output feature map to stabilize model training. This application adds the convolution input and the activation layer output to form a residual structure. The identity branch design in the residual structure can greatly alleviate the gradient vanishing problem caused by an overly deep model.
[0036] As a feasible implementation method, a multi-receptive field feature extraction module is constructed, specifically including: constructing a first dilated depth-separating convolutional layer, a second dilated depth-separating convolutional layer, a third dilated depth-separating convolutional layer, a fourth dilated depth-separating convolutional layer, and a global pooling layer to extract multi-receptive field features from the input image. Then, a cascaded layer is constructed to concatenate the extracted multi-receptive field features. Finally, a 1*1 convolutional block is constructed to fuse the concatenated multi-receptive field features. The first dilated depth-separating convolutional layer, the second dilated depth-separating convolutional layer, the third dilated depth-separating convolutional layer, the fourth dilated depth-separating convolutional layer, the global pooling layer, the cascaded layer, and the 1*1 convolutional block constitute the multi-receptive field feature extraction module.
[0037] In one embodiment, Figure 4 This is a schematic diagram of a multi-receptive field feature extraction module provided in an embodiment of this application, as shown below. Figure 4 As shown, DW represents dilated depth separation convolution, R represents the dilation rate, Cat represents the concatenation operation, and Pooling represents the global pooling layer. Specifically, for a feature map with a stride of 16, different receptive field features are extracted through different dilated depth separation convolution layers and concatenated together. Then, a 1*1 convolution is used to fuse the multiple receptive field features. This method effectively increases the receptive field of the feature map without increasing the computational cost. The first dilated depth separation convolution layer is a 1*1 dilated depth separation convolution layer; the second dilated depth separation convolution layer is a 3*3 dilated depth separation convolution layer with a dilation rate of 1; the third dilated depth separation convolution layer is a 3*3 dilated depth separation convolution layer with a dilation rate of 3; and the fourth dilated depth separation convolution layer is a 3*3 dilated depth separation convolution layer with a dilation rate of 5.
[0038] Further, a preset number of power transmission channel images containing power transmission conductors are collected. Each power transmission conductor in the power transmission channel images is labeled with a conductor region mask through Labelme software to obtain a labeled image corresponding to each power transmission channel image. The preset number of power transmission channel images and the corresponding labeled images are combined to form a training data set of the conductor detection model. The conductor detection model is trained through the training data set.
[0039] In S102, the trained conductor detection model is optimized through a preset hybrid loss function.
[0040] Specifically, a Focal loss loss function is constructed according to the prediction accuracy of the conductor contour prediction image output by the trained conductor detection model. A Dice loss loss function is constructed according to the similarity between the conductor contour prediction image and the corresponding training sample. Then
[0041] loss=w1*Focalloss+w2*Diceloss, to obtain a hybrid loss function. Wherein, ω1 is a loss balance factor of the Focal loss loss function, and ω2 is a loss balance factor of the Dice loss loss function. The trained conductor detection model is parameter-optimized until the hybrid loss function converges, and an optimized conductor detection model is obtained.
[0042] In one embodiment, considering that the conductor is very long and thin, and occupies a very small area of the picture, this poses a great challenge to the model in distinguishing the conductor from the background pixels and the model in making unbiased predictions. Uneven sample distribution leads to the loss of the background area, which is easy to distinguish and occupies a much larger proportion than the conductor, drowning out the conductor loss during the training of the model using the conventional cross-entropy loss. The model trained in this way will be biased towards background prediction and have weak generalization performance. To solve this problem, the present application proposes a hybrid loss function combining Focal loss and Dice loss to train the conductor detection model, and the loss balance factors ω1 and ω2 are both set to 0.5 in the present application.
[0043] In S103, the first frame video image in the power transmission channel video is recognized through the optimized conductor detection model to obtain a conductor contour image corresponding to the first frame video image.
[0044] Specifically, the present application only detects the first frame video image in the power transmission channel video through the conductor detection model, rather than detecting each frame of image, to realize rapid tracking of the conductor.
[0045] In S104, a set of conductor observation points is determined in the corresponding video image according to the conductor contour image.
[0046] Specifically, based on the conductor features in the conductor contour image, the minimum bounding rectangle of the region containing the conductor in the first frame of the video image is determined. Evenly spaced parallel lines are drawn within the minimum bounding rectangle of the video image. The coordinates of the intersection points between the parallel lines and the conductor in the video image are determined, resulting in a local conductor tracking point set. Within the minimum bounding rectangle region, a candidate tracking point set is generated using the Shi-Tomasi corner detection algorithm. The local conductor tracking point set and the candidate tracking point set are combined to form a conductor candidate observation point set.
[0047] Furthermore, based on the traverse trajectory in the traverse contour image, a traverse mask is generated. Then, based on the traverse mask, points existing on the traverse trajectory are selected from the candidate observation point set to obtain the traverse observation point set. This set stores the position coordinates of all traverse observation points.
[0048] As a feasible implementation method, Figure 5 This is a schematic diagram of a conductor observation point construction provided in an embodiment of this application, as shown below. Figure 5 As shown, based on the conductor contour image output by the conductor detection model, we can determine the minimum bounding rectangle of the region where all conductors are located in the original image. Within this minimum bounding rectangle, we draw parallel lines of equal spacing and thickness. These parallel lines intersect the detected conductor trajectory, creating intersection points, which are the local conductor tracking points. Furthermore, within this minimum bounding rectangle region, candidate tracking points are generated using the Shi-Tomasi corner detection algorithm, and these intersection points together form candidate conductor observation points. To further filter candidate observation points that are only on the conductors, a conductor mask is generated based on the conductor detection trajectory. Based on this conductor mask, points that are only on the conductors are selected from the candidate conductor observation points; these points are called conductor observation points.
[0049] S105. In the set of observation points for the traverse, invalid observation points are removed to obtain the set of valid observation points.
[0050] Specifically, points in the traverse observation point set of the first frame video image are used as tracking points, and forward optical flow tracking is performed on these tracking points to obtain the predicted traverse observation point set of the second frame video image. Points in the predicted traverse observation point set of the second frame video image are used as tracking points, and backward optical flow tracking is performed on these tracking points to obtain the predicted traverse observation point set of the first frame video image. The Eulerian distance between corresponding points in the traverse observation point set of the first frame video image and the predicted traverse observation point set is calculated. If the Eulerian distance is greater than a preset threshold, the corresponding traverse observation point is determined as an invalid observation point and is removed. After removing all invalid observation points from the traverse observation point set of the first frame video image, the valid observation point set is obtained.
[0051] As a feasible implementation, the traditional key point tracking can be implemented using a Lucas Kanade sparse optical flow algorithm (LK algorithm), but in the tracking of conductor observation points in a power transmission channel scene, directly using the LK algorithm can cause local observation point drift or large prediction deviation, which is mainly caused by the fact that the observation points of different points on the conductor are very similar. In order to alleviate this problem, the present application predicts the positions of all conductor observation points in the t+1 image from the t image for all conductor observation points, and then predicts the positions of the conductor observation points in the t image from the positions of all conductor observation points predicted in the t+1 image. If the distance error between the original conductor observation point position in the t image and the predicted conductor observation point position is within a certain range, the conductor observation point is considered valid, otherwise the conductor observation point is removed.
[0052] S106, tracking the conductor in the power transmission channel video according to the set of valid observation points.
[0053] Specifically, forward optical flow tracking is performed according to the set of valid observation points of the first frame of video images to obtain a predicted set of valid observation points of the second frame of video images.
[0054] Further, the position and size of the tracking frame of the second frame of video images are calculated according to the predicted set of valid observation points of the second frame of video images, and the position and size of the tracking frame of each frame of video images are obtained in the same way to track the conductor in the power transmission channel video.
[0055] In addition, the present application also provides a conductor tracking device, as shown in the Figure 6 The conductor tracking device 600 specifically comprises:
[0056] at least one processor 601; and a memory 602 communicatively connected with the at least one processor 601; wherein
[0057] The memory 602 stores instructions executable by the at least one processor 601, so that the at least one processor 601 can perform:
[0058] constructing and training a conductor detection model;
[0059] optimizing the trained conductor detection model through a preset hybrid loss function;
[0060] identifying a first frame of video images in a power transmission channel video through the optimized conductor detection model to obtain a conductor contour image corresponding to the first frame of video images;
[0061] determining a set of conductor observation points in the first frame of video images according to the conductor contour image;
[0062] In the conductor observation point set, invalid conductor observation points are removed to obtain a valid observation point set;
[0063] According to the valid observation point set, the conductor in the power transmission channel video is tracked.
[0064] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment mainly describes the difference from other embodiments. Especially, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.
[0065] The above describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown or sequential order in order to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous or necessary.
[0066] The above only describes the embodiments of the present application and does not limit the present application. The embodiments of the present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiments of the present application shall be included in the scope of the claims of the present application.
Claims
1. A method of wire tracking, characterized by, The method comprises: constructing and training a lead detection model, specifically comprising: constructing a multi-level residual feature extraction module; wherein the multi-level residual feature extraction module is composed of multiple residual modules, each residual module comprising a depth separable convolution layer, a batch normalization layer and an activation layer; constructing a multi-receptive field feature extraction module; wherein the multi-receptive field feature extraction module comprises a dilated depth separable convolution layer and a global pooling layer; constructing a feature fusion module to fuse the multi-receptive field feature maps output by the multi-receptive field feature extraction module with the shallow feature maps extracted by the first residual module in the multi-level residual feature extraction module to obtain a feature fusion image; wherein the shallow feature maps are upsampled to the same size of the multi-receptive field feature maps before feature fusion; constructing a decoding prediction module to decode and predict the feature fusion image to obtain an output image; the multi-level residual feature extraction module, the multi-receptive field feature extraction module, the feature fusion module and the decoding prediction module constitute the lead detection model; training the lead detection model through a training data set; optimizing the trained lead detection model through a pre-set hybrid loss function; recognizing the first frame of video image in the power transmission channel video through the optimized lead detection model to obtain a lead contour image corresponding to the first frame of video image; determining a lead observation point set in the first frame of video image according to the lead contour image, specifically comprising: determining the minimum circumscribed rectangle of the region where the lead is located in the first frame of video image according to the lead features in the lead contour image; drawing equally spaced parallel lines within the minimum circumscribed rectangle of the first video image; determining the intersection position coordinates of the parallel lines and the lead in the first video image to obtain a local lead tracking point set; generating a candidate tracking point set in the minimum circumscribed rectangle region through the Shi-Tomasi corner detection algorithm; combining the local lead tracking point set and the candidate tracking point set into a lead candidate observation point set; generating a lead mask according to the lead trajectory in the lead contour image, and screening out the points existing on the lead trajectory from the lead candidate observation point set according to the lead mask to obtain the lead observation point set; wherein the lead observation point set stores the position coordinates of all lead observation points; eliminating invalid lead observation points from the lead observation point set to obtain a valid observation point set; tracking the lead in the power transmission channel video according to the valid observation point set.
2. The method of claim 1, wherein, constructing a multi-level residual feature extraction module, specifically comprising: The first deep separation convolutional layer, the batch normalization layer, the activation layer and the second deep separation convolutional layer are sequentially connected; wherein, the first deep separation convolutional layer is used for deep separation convolutional processing on the input image, and the processing result is input into the batch normalization layer for normalization processing; the activation layer is used for activating preset important features in the output features of the batch normalization layer, and suppressing preset unimportant features, and the second deep separation convolutional layer is used for deep separation convolutional processing on the features output by the activation layer again; An addition layer is constructed to add the input image and the output result of the second deep separation convolutional layer; The first deep separation convolutional layer, the batch normalization layer, the activation layer, the second deep separation convolutional layer and the addition layer constitute a residual module; A plurality of residual modules are connected to constitute the multi-level residual feature extraction module.
3. The method of claim 1, wherein, A multi-receptive field feature extraction module is constructed, specifically including: First, second, third and fourth hollow deep separation convolutional layers and a global pooling layer are respectively constructed to extract multi-receptive field features from the input image; The first hollow deep separation convolutional layer is a 1*1 hollow deep separation convolutional layer; the second hollow deep separation convolutional layer is a 3*3 hollow deep separation convolutional layer with a hollow rate of 1; the third hollow deep separation convolutional layer is a 3*3 hollow deep separation convolutional layer with a hollow rate of 3; and the fourth hollow deep separation convolutional layer is a 3*3 hollow deep separation convolutional layer with a hollow rate of 5; A cascade layer is constructed to splice the extracted multi-receptive field features; A 1*1 convolution block is constructed to fuse the spliced multi-receptive field features; The first, second, third and fourth hollow deep separation convolutional layers, the global pooling layer, the cascade layer and the 1*1 convolution block constitute the multi-receptive field feature extraction module.
4. The method of claim 1, wherein, The trained wire detection model is optimized by a preset hybrid loss function, specifically including: According to the prediction accuracy of the wire contour prediction image output by the trained wire detection model, a Focalloss loss function is constructed; According to the similarity between the wire contour prediction image and the corresponding training sample, a Dice loss loss function is constructed; According to , obtain the mixed loss function loss; wherein ω1 is a loss balance factor of the Focal loss loss function, and ω2 is a loss balance factor of the Dice loss loss function. The trained wire detection model is parameter-optimized until the hybrid loss function converges, and an optimized wire detection model is obtained.
5. The method of claim 1, wherein, In the wire observation point set, invalid wire observation points are removed to obtain a valid observation point set, specifically including: Points in the wire observation point set of the first frame of video image are taken as tracking points, and the tracking points are forward optical flow tracked to obtain a predicted wire observation point set of the second frame of video image; Points in the predicted wire observation point set of the second frame of video image are taken as tracking points, and the tracking points are reverse optical flow tracked to obtain a predicted wire observation point set of the first frame of video image; calculating Euler distance of corresponding points in the set of wire observation points of the first frame of video images and the set of predicted wire observation points; if the Euler distance is greater than a preset threshold, determining that the corresponding wire observation point is an invalid observation point, and performing elimination; after eliminating all invalid observation points in the set of wire observation points of the first frame of video images, obtaining the set of valid observation points.
6. A method of wire tracking according to claim 5, wherein, According to the set of valid observation points, tracking the wire in the power transmission channel video, specifically including: performing forward optical flow tracking according to the set of valid observation points of the first frame of video images to obtain a set of predicted valid observation points of a second frame of video images; calculating the position and size of a tracking frame of the second frame of video images according to the set of predicted valid observation points of the second frame of video images, and similarly calculating the position and size of a tracking frame of each frame of video images to track the wire in the power transmission channel video.
7. The method of claim 1, wherein, Before constructing and training the wire detection model, the method further includes: collecting a preset number of power transmission channel images containing power transmission wires; through Labelme software, labeling a wire region mask for each power transmission wire in the power transmission channel images to obtain a corresponding labeled image for each power transmission channel image; combining the preset number of power transmission channel images and the corresponding labeled images to form a training data set of the wire detection model.
8. A wire tracking device, characterized by The device includes: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a wire tracking method according to any one of claims 1-7.
Citation Information
Patent Citations
Power transmission line track detection method and equipment
CN113850841A
Computer vision method and intelligent camera system for bridge health diagnosis
CN113971660A