Power equipment inspection segmentation and tracking method based on mask memory propagation
An adaptive mask attention decision network propagated through mask memory propagation solves the problem of segmentation inaccuracy in twin network segmentation and tracking algorithms when target appearance and background clutter change abruptly, achieving high-precision and stable tracking of power equipment.
Patent Information
- Application Number
- CN202311637844.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-02
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2043-12-02
AI Technical Summary
Existing segmentation and tracking algorithms based on Siamese networks suffer from reduced segmentation accuracy when faced with abrupt changes in target appearance and background clutter, leading to unstable tracking performance.
An adaptive mask attention decision network based on mask memory propagation is adopted. By extracting the depth features of the target template image, the target region of neighboring frames and the current frame image, the attention module of mask propagation is used to store the memory library, calculate the similarity matrix and attention map, and combine the attention refinement path and mask evaluation fusion mechanism to generate a more accurate target mask.
It improves the positioning accuracy and robustness of power equipment segmentation tracking, suppresses the influence of background clutter, and obtains more accurate target tracking results.
Smart Images
Figure CN117593544B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of power inspection and maintenance, specifically relating to a power equipment inspection segmentation and tracking method based on mask memory propagation. Background Technology
[0002] With the continuous growth of electricity demand in my country, the number of devices in the power system is increasing. To improve the stable operation of power equipment, continuous monitoring is essential. Typically, large-scale intelligent terminals such as drones, robots, and PTZ cameras are used to replace human eyes for 24-hour real-time monitoring of the equipment. Through deep learning-based computer vision technology, visual calculations such as target detection and tracking are performed on the images monitored by these intelligent terminals, thereby accurately and in real-time detecting any anomalies in the power equipment.
[0003] In recent years, segmentation and tracking algorithms based on Siamese networks have improved the accuracy and stability of video target segmentation and tracking tasks. However, abrupt changes in target appearance and background clutter can still affect segmentation accuracy, thus impacting tracking performance. This leads to potential tracking failures in real-world tracking environments. Therefore, there is an urgent need to propose a tracking method that can effectively guarantee accurate localization and high robustness. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention aims to solve the problem that segmentation accuracy is still affected by abrupt changes in target appearance and background clutter in segmentation and tracking algorithms based on Siamese networks. A new method for segmenting and tracking power equipment based on a memory propagation adaptive mask attention decision network is proposed.
[0005] The present invention adopts the following technical solution.
[0006] A method for segmentation and tracking of power equipment based on mask memory propagation, the specific steps of which are as follows:
[0007] Step S1: Based on the location information of the power equipment, extract the depth features of the target template image, the target region of the neighboring frame, and the current frame image as features of the target template, the target region of the neighboring frame, and the search region of the current frame, respectively.
[0008] Step S2: The attention module propagating through the mask stores the target template of the tracked device and the depth features and mask of the target region in the adjacent frame into a memory library.
[0009] Step S3: Use the features of the search region in the current frame to perform similarity calculations with the memory library to obtain the similarity matrix.
[0010] Step S4: Multiply the target template mask and the target mask of the neighboring frame with the similarity matrix to obtain the attention map;
[0011] The calculation of the attention map is: M i =S i *v i , where S i is the similarity matrix, v i represents the adjacent frame target region mask and the target template mask in the memory bank, and * denotes matrix multiplication.
[0012] Step S5: Perform element-wise multiplication on the attention map and the search region feature of the current frame to obtain the affinity feature, and then perform depth cross-correlation calculation with the target template feature to obtain the response map.
[0013] Step S6: Extract the maximum response channel in the response map and input it into the attention refinement path to obtain two different masks.
[0014] Step S7: Input into the mask evaluation and fusion mechanism to calculate the final target mask, and feed back to step S2 to update the memory bank, so as to obtain the position of the tracked power equipment according to the final mask.
[0015] Further, step S1 is specifically:
[0016] Step S11: Intercept a target region of a fixed size in the i-th frame (i=1) of the video sequence according to the power equipment, and extract the deep feature using the pre-trained ResNet-50 network as the feature of the target template;
[0017] Step S12: Read the j-1-th frame (i<j) of the power equipment in the video sequence, crop the adjacent frame target region, and extract the deep feature of the adjacent frame target region using the pre-trained ResNet-50 network to obtain the feature of the adjacent frame target region
[0018] Step S13: Read the j-th frame (i<j) of the power equipment in the video sequence, crop the search region of the current frame, and extract the deep feature of the search region of the current frame using the pre-trained ResNet-50 network to obtain the feature of the search region of the current frame.
[0019] Further, step S3 is specifically:
[0020] Step S31: The similarity matrix is calculated as: S i =softmax(R(F t )*R(A i )), where F t and A i are respectively the search region feature of the current frame and the stored feature in the memory bank, A i represents the adjacent frame target region feature and the target template feature in the memory bank, * is matrix multiplication, R(F t) and R(A i ) represents the dimension of reshaping the features of the current frame search region and the features stored in the memory bank.
[0021] Step S32: Combine the similarity matrix S i Softmax normalization maps the real-field representation of the matrix to the effective real-field space [0, 1] representing the probability distribution.
[0022] Furthermore, step S5 specifically includes:
[0023] Step S51: The affinity characteristic is calculated as follows: Where M i It is an attention map, F t Features representing the search region in the current frame Represents element-wise product;
[0024] Step S52: The response map is calculated as follows: Where F' t F0 and F0 represent the two affinity features and the target template feature, respectively. Represents a deep cross-correlation operation;
[0025] Furthermore, step S6 specifically includes:
[0026] Step S61: Extract the maximum response from the 17×17×256 response map to obtain a 1×1×256 response bar, flatten it and perform a deconvolution operation to obtain feature e1 with a size of 15×15×32, and then input it into the attention refinement path.
[0027] Step S62: The attention refinement path consists of three refinement modules, each consisting of multiple upsampling layers and skip connections, merging different resolution characteristics to generate the corresponding mask;
[0028] Step S63: Each refinement module also includes an efficient channel attention module to enhance the encoding of low-resolution features. By calculating attention weights, it pays more attention to low-order features that are sensitive to edge details, and finally further refines the segmentation contour.
[0029] Step S64: The efficient channel attention module utilizes fast 1D convolutions to achieve local cross-channel interaction, where the kernel size determines the degree of cross-channel interaction, thereby enhancing the encoding of feature channels. The adaptive selection of the kernel size is calculated as follows: There is a mapping relationship between the number of channels C and the kernel size k. b and γ are hyperparameters set to 1 and 2, respectively.
[0030] Step S64: Finally, the Sigmoid activation function generates two different masks, namely the "template propagation mask" and the "neighbor frame propagation mask".
[0031] Furthermore, step S7 specifically includes:
[0032] Step S71: Input the paired target template RGB images and masks, and the paired neighboring frame target region RGB images and masks into the quality evaluation network;
[0033] Step S72: The quality assessment network consists of a ResNet-18 neural network with shared weights and fully connected layers;
[0034] Step S73: Extract features from the two pairs of inputs through the ResNet-18 neural network to obtain the corresponding Feature1 and Freture2. Add them element by element to obtain the Feature, merge them and enter the fully connected layer.
[0035] Step S74: After the Feature is fed into the fully connected layer, two quality scores are output, corresponding to the quality of the two input pairs respectively. These quality scores serve as weights for subsequent mask fusion. The mask fusion calculation method is as follows: mask t = s1×mask1+s2×mask2, where s1 and s2 are mass fractions.
[0036] The beneficial effects of the present invention are as follows: Compared with the prior art, the present invention has the following advantages:
[0037] (1) This invention designs a mask propagation method for use in segmentation tracking networks. By learning the intrinsic correlation between each frame, the mask information of historical frames is mined. By retrieving the memory library of stored features and historical frame masks, a target attention map is generated to highlight the target region on the frame features, thereby suppressing the adverse effects of background clutter.
[0038] (2) This invention designs an attention refinement path to further refine the segmentation contour during the mask generation process. A lightweight attention mechanism is introduced to calculate the weights of the low-level features, focusing more on the low-level features that are sensitive to edge details, thereby obtaining better segmentation results.
[0039] (3) This invention proposes a mask fusion mechanism to improve the accuracy of the mask. By using a quality assessment network and mask perception fusion, a more accurate tracking result than that of a bounding box is obtained, thereby improving the target localization accuracy. Attached Figure Description
[0040] Figure 1 This is a flowchart of the present invention.
[0041] Figure 2 This is a structural diagram of the efficient channel attention module in the refinement module of the present invention.
[0042] Figure 3This is a comparison diagram of the tracking effect of the present invention and other algorithms on the OTB100 general dataset. Detailed Description of the Embodiments
[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention, and shall not be construed as limiting the protection scope of the present application.
[0044] The objective of the present invention is to provide a power equipment inspection segmentation and tracking method based on mask memory propagation, the specific steps are as follows:
[0045] Step S1: According to the position information of the power equipment, extract the deep features of the target template image, the adjacent frame target area and the current frame image respectively as the features of the target template, the adjacent frame target area and the current frame search area.
[0046] The method of step S1 is specifically as follows:
[0047] Step S11: Intercept a target area of fixed size in the i-th frame image (i=1) of the video sequence according to the power equipment, and use the pre-trained ResNet-50 network to extract deep features as the features of the target template; in some embodiments, the target area is set to a width, height and channel number of 127×127×3;
[0048] Step S12: Read the j-1-th frame image (i<j) of the power equipment in the video sequence and crop the adjacent frame target area, and use the pre-trained ResNet-50 network to extract the deep features of the adjacent frame target area to obtain the features of the adjacent frame target area; in some embodiments, the adjacent frame target area is set to a width, height and channel number of 127×127×3;
[0049] Step S13: Read the j-th frame image (i<j) of the power equipment in the video sequence and crop the current frame search area, and use the pre-trained ResNet-50 network to extract the deep features of the current frame search area to obtain the features of the current frame search area; the purpose of cropping is to make the size of the current frame search area input into the feature extraction network fixed, and in some embodiments, the current frame search area is set to a width, height and channel number of 255×255×3.
[0050] Step S2: Store the deep features and masks of the target template of the tracked device and the adjacent frame target area into a memory bank through the attention module of mask propagation.
[0051] Step S3: Calculate the similarity matrix by performing operations on the features of the current frame search area and the memory bank respectively.
[0052] The specific method in step S3 is as follows:
[0053] Step S31: The similarity matrix is calculated as: S i =softmax(R(F t )*R(A i )), where F t and A i These are the current frame search region features and the stored features in the memory bank, A i Represents the target region features and target template features of neighboring frames in the memory library, * indicates matrix multiplication, R(F t ) and R(A i ) represents the dimensions of reshaping the current frame search region features and the features stored in the memory bank, ultimately obtaining the current frame search region feature size as (31×31)×256 and the memory bank feature size as 256×(15×15).
[0054] Step S32: Combine the similarity matrix S i Softmax normalization maps the real-valued representation of the matrix to the effective real-valued space [0, 1] representing the probability distribution. The final similarity matrix S... i The dimensions are (31×31)×(15×15).
[0055] Step S4: Multiply the target template mask and the target mask of the neighboring frame with the similarity matrix to obtain the attention map;
[0056] The attention map is calculated as: M i =S i *v i S i It is a similarity matrix, v i This represents the target region mask and target template mask in the memory library, with * indicating matrix multiplication. Attention graph M i The dimensions are 31×31.
[0057] Step S5: Perform element-wise multiplication of the attention map with the features of the current frame search region to obtain the affinity features, and then perform deep cross-correlation calculation with the target template features to obtain the response map.
[0058] The specific method in step S5 is as follows:
[0059] Step S51: The affinity characteristic is calculated as follows: Where M i It is an attention map, F t Features representing the search region in the current frame Represents element-wise product, two affinity features F' t The dimensions are 31×31×256;
[0060] Step S52: The response map is calculated as follows: Where F' t F0 and F0 represent the two affinity features and the target template feature, respectively. Represents a deep cross-correlation operation;
[0061] Step S6: Extract the maximum response channel from the response map and input it into the attention refinement path to obtain two different masks.
[0062] The specific method in step S6 is as follows:
[0063] Step S61: Extract the maximum response from the 17×17×256 response map to obtain a 1×1×256 response bar, flatten it and perform a deconvolution operation to obtain feature e1 with a size of 15×15×32, and then input it into the attention refinement path.
[0064] Step S62: The attention refinement path consists of three refinement modules, each consisting of multiple upsampling layers and skip connections, merging different resolution characteristics to generate the corresponding mask;
[0065] Step S63: Each refinement module also includes an efficient channel attention module to enhance the encoding of low-resolution features. By calculating attention weights, it pays more attention to low-order features that are sensitive to edge details, and finally further refines the segmentation contour.
[0066] Step S64: The efficient channel attention module utilizes fast 1D convolutions to achieve local cross-channel interaction, where the kernel size determines the degree of cross-channel interaction, thereby enhancing the encoding of feature channels. The adaptive selection of the kernel size is calculated as follows: There is a mapping relationship between the number of channels C and the kernel size k. b and γ are hyperparameters set to 1 and 2, respectively.
[0067] Step S64: Finally, the Sigmoid activation function generates two different masks, namely the "template propagation mask" and the "neighbor frame propagation mask".
[0068] Step S7: Input to the mask evaluation and fusion mechanism to calculate the final target mask, and simultaneously feed back to step S2 to update the memory library, and obtain the location of the tracked power device based on the final mask.
[0069] Step S7 specifically includes:
[0070] Step S71: Input the paired target template RGB images and masks, and the paired neighboring frame target region RGB images and masks into the quality evaluation network;
[0071] Step S72: The quality assessment network consists of a ResNet-18 neural network with shared weights and fully connected layers;
[0072] Step S73: Extract features from the two pairs of inputs through the ResNet-18 neural network to obtain the corresponding Feature1 and Freture2. Add them element by element to obtain the Feature, merge them and enter the fully connected layer.
[0073] Step S74: After the Feature is fed into the fully connected layer, two quality scores are output, corresponding to the quality of the two input pairs respectively. These quality scores serve as weights for subsequent mask fusion. The mask fusion calculation method is as follows: mask t = s1×mask1+s2×mask2, where s1 and s2 are mass fractions.
[0074] The experiments of this invention were implemented on a computer with an Intel i7-10700 CPU (2.90GHz), 16GB RAM, and an NVIDIA GeForce GTX 1650 GPU, using the Python 3.7 programming language on the PyTorch framework. Figure 1 This is a flowchart for tracking power equipment in a real-world scenario. Figure 3 This paper compares the method of this invention (Ours) with other advanced target tracking algorithms (DaSiamRPN, TADT, SiamMask, GradNet, DeepSRDCF, SiamRPN, SiamDWfc, SRDCF, Staple, SiamFC) on the OTB100 general dataset. It shows that the method has excellent performance in both success rate and accuracy. It can be seen that the present invention has more accurate positioning accuracy and better robustness, and can efficiently handle power equipment tracking tasks.
[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for segmenting and tracking power equipment during inspection based on mask memory propagation, characterized in that, The specific steps are as follows: Step S1: Based on the location information of the power equipment, extract the depth features of the target template image, the target region of the neighboring frame, and the current frame image as features of the target template, the target region of the neighboring frame, and the search region of the current frame, respectively. Step S2: The attention module propagating through the mask stores the target template of the tracked device and the depth features and mask of the target region in the adjacent frame to obtain a memory library; Step S3: Use the features of the search region in the current frame to perform similarity matrix operations with the memory library respectively; Step S4: Multiply the target template mask and the target mask of the neighboring frame with the similarity matrix to obtain the attention map; The attention map is calculated as follows: Mi = Si * vi, where Si is the similarity matrix, vi represents the target region mask and target template mask in the memory library, and * represents matrix multiplication; Step S5: Perform element-wise multiplication of the attention map with the features of the current frame search region to obtain the affinity feature, and then perform deep cross-correlation calculation with the target template feature to obtain the response map; Step S6: Extract the maximum response channel from the response map and input it into the attention refinement path to obtain two different masks; Specifically, the attention refinement path consists of three refinement modules. Each refinement module comprises multiple upsampling layers and skip connections, merging different resolution features to generate corresponding masks. Each refinement module also includes an efficient channel attention module that enhances the encoding of low-resolution features by calculating attention weights to focus more on low-order features sensitive to edge details, ultimately further refining the segmentation contour. Finally, two different masks are generated through the Sigmoid activation function: the "template propagation mask" and the "neighbor frame propagation mask". Step S7: Input to the mask evaluation and fusion mechanism to calculate the final target mask, and simultaneously feed back to step S2 to update the memory library, and obtain the location of the tracked power device based on the final mask; Step S7 is as follows: Step S71: Input the paired target template RGB images and masks, and the paired neighboring frame target region RGB images and masks into the quality evaluation network; Step S72: The quality assessment network consists of a ResNet-18 neural network with shared weights and fully connected layers; Step S73: Extract features from the two pairs of inputs through the ResNet-18 neural network to obtain the corresponding Feature1 and Freture2. Add them element by element to obtain the Feature, merge them and enter the fully connected layer. Step S74: After the Feature is fed into the fully connected layer, two quality scores are output, which correspond to the quality of the two pairs of inputs respectively. The quality scores are used as the subsequent mask fusion weights. The mask fusion calculation method is: maskt=s1×mask1+s2×mask2, where s1 and s2 are the quality scores.