Defect deduplication model building and defect deduplication method for power transmission inspection images

Abstract

The present application relates to a power transmission inspection image defect deduplication model building and defect deduplication method, which is applied to the technical field of power transmission line defect detection, comprising: by constructing a defect deduplication model of power transmission inspection image, improving the existing MOCO v3 training method to train the model, based on the built model, fitting the image projection transformation parameter matrix through the nearest neighbor algorithm and RANSAC algorithm, judging whether two power transmission inspection images constitute repetition by evaluating the fitting result, deleting the repeated images, in the present scheme, the shooting order of the inspection image is reasonably utilized, only a certain number of adjacent images are determined, effectively avoiding that different equipment pictures with similar forms are mistaken as the same target and removed by the deduplication algorithm, thereby solving the problem that the existing defect recognition system can recognize a large number of repeated potential defects from the photos, and the artificial auditing workload of defect analysis is large.

Description

Technical Field

[0001] This invention relates to the field of power transmission line defect detection technology, specifically to the construction of a defect deduplication model and a defect deduplication method for power transmission inspection images. Background Technology

[0002] Defects and hidden dangers in power transmission lines can lead to power grid operation accidents, causing power outages, further equipment damage, and other problems. Therefore, regular inspections of power transmission lines are necessary to promptly identify and address defects / potential hazards. Currently, the process for power transmission line inspections using drones and helicopters is as follows: First, the aircraft inspects the target line and towers along a pre-determined route. During this process, the drone pilot or helicopter inspector takes photos of every potentially defective piece of equipment on the line and towers. Then, all the photos are uploaded to an inspection image analysis system, where an intelligent model identifies potential defects. Next, the defects identified by the model are manually reviewed, and the remaining images are randomly checked to find all defects in the images. A defect report is then created (each line on the defect report describes a defect, including the tower, location, specific details, and severity level). Finally, the maintenance unit accepts the defect report and arranges a repair plan to eliminate the defects. During power line inspections, drones and helicopters take multiple photos of the same target to ensure image quality, resulting in a large number of duplicate photos. This causes two problems: First, when uploading the photos to the defect identification system for further processing, more data needs to be uploaded, consuming more bandwidth and data transmission time. Second, the defect identification system will identify a large number of duplicate potential defects from the photos, increasing the workload of manual review for defect analysis.

[0003] Potential technologies that could meet this need include image deduplication and image registration. Current image deduplication algorithms typically target the entire image, extracting image features and comparing them with features from other images to find similar images. When applied to power transmission inspection image deduplication, these technologies face several challenges: First, components on the same tower may have similar materials and shapes, and images taken at the same location are highly likely to have similar backgrounds. Therefore, images of different components may be misclassified as duplicates due to overly similar extracted features. Second, during the shooting process, due to shooting specifications and aircraft movement, different images of the same location may have differences in angle, orientation, and focal length. The algorithm may identify these differences as different photographs. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a defect deduplication model and method for power transmission inspection images, in order to solve the problems in the prior art where images taken of different components may be mistakenly identified as duplicate images and thus deleted due to overly similar extracted features. Furthermore, due to differences in angle, orientation, and focal length among different images of the same part taken due to shooting specifications and aircraft displacement, existing algorithms may identify them as different photos and not delete them. Therefore, this invention fails to address the problem that defect identification systems identify a large number of duplicate potential defects from photos, resulting in a large workload for manual review of defect analysis.

[0005] According to a first aspect of the present invention, a method for constructing a defect deduplication model for power transmission inspection images is provided, comprising:

[0006] Acquire general images and power transmission inspection images, preprocess the general images and power transmission inspection images to obtain slices for training;

[0007] The slices used for training are input into the built MAE model for training to obtain the pre-trained model;

[0008] After performing random cropping, random rotation, and random projective transformation on the power transmission inspection images, projective transformation images of multiple power transmission inspection images are obtained. Among the multiple projective transformation images, two projective transformation images with the same number of grid points at the same coordinates greater than a preset threshold are selected.

[0009] Random color jittering, graying, blurring, and erasing operations are performed on two projective transformation images to obtain two slices with preset resolutions and the coordinates of the same grid points in the two slices are recorded.

[0010] The coordinates of two slices and their common grid points are input into the pre-trained model, and the pre-trained model is trained using the improved MOCO v3 training method to obtain the final model.

[0011] Preferably,

[0012] The improvements to the MOCO v3 training method include:

[0013] The original global self-attention module is replaced by a local self-attention module with a window width of 16. The local self-attention module includes 12 modules, and each module is set to roll every X intervals. The rolling setting includes: performing rolling operations with step sizes of -8 and 8 on the feature maps it acquires.

[0014] Preferably,

[0015] The improvements to the MOCO v3 training method also include:

[0016] The feature vectors of the same grid points in two slices are obtained by using bilinear interpolation. Comparative learning is performed on a grid point basis. The "projection head" and "prediction head" in the learning process are kept consistent with the parameters at the image level.

[0017] The positive samples in the contrastive learning are features extracted from grid points on another projective transformation image corresponding to the same position of itself, and the negative samples are features from other grid points on the original power transmission inspection image that are more than 24 pixels away from itself, as well as features from other samples.

[0018] The method for obtaining the feature values ​​of the same grid points in two slices includes: feeding the two slices into the ViT network respectively, obtaining the feature maps of the slices, and using bilinear interpolation to obtain the feature vectors of the corresponding positions of specific grid points in the feature maps.

[0019] Preferably,

[0020] The preprocessing of general images and power transmission inspection images to obtain slices for training includes:

[0021] The general image is scaled down to a side length of no more than 800. The power transmission inspection image is cropped and scaled to generate 800*800 slices. 3-5 slices are generated from each original power transmission inspection image.

[0022] Images are randomly selected in a 7:3 ratio between general images and power transmission inspection images. The randomly selected images are cropped according to a preset cropping method, and the cropped images are scaled to 256*256 to obtain slices for MAE model training.

[0023] Preferably,

[0024] After performing random cropping, random rotation, and random projective transformation operations on the power transmission inspection images, the resulting projective transformation images of multiple power transmission inspection images include:

[0025] The power transmission inspection images are cropped according to a preset cropping method, and the coordinates of the four vertices of the cropped area (top left, bottom left, top right, and bottom right) are recorded.

[0026] Rotate the cropping area around the center point at a random angle, and record the coordinates of the top left and top right vertices of the cropping area after rotation;

[0027] The coordinates of the top left and top right vertices are randomly moved to obtain the projective transformation coordinates of the four vertices of the cropped area to the four vertices of a 512*512 square. The original power transmission inspection image is then projectively transformed according to the projective transformation coordinates of the four vertices to obtain the projective transformation image of the power transmission inspection image. The above steps are repeated to generate multiple projective transformation images for training the improved MOCO v3 method.

[0028] According to a second aspect of the present invention, a method for deduplicating defects in power transmission inspection images is provided. The method is based on a final model obtained using the defect deduplication model building method for power transmission inspection images described in any one of the preceding claims. The method includes:

[0029] All the captured inspection images of the towers are input into the final model, and adjacent inspection images are compared pairwise using the nearest neighbor algorithm according to the order of shooting.

[0030] Based on the comparison results, the projective transformation matrices of two adjacent inspection images are estimated to determine whether there is an overlapping region. This estimation of the projective transformation matrices of two adjacent inspection images to determine whether there is an overlapping region includes:

[0031] Obtain effective feature pairs from two adjacent inspection images; the effective feature pair is a pair of grid points from the two images, and the feature distance between them is less than the distance from each grid point to any other grid point in the same image;

[0032] If the number of valid feature pairs exceeds a preset threshold, the RANSAC algorithm is used to fit the projective transformation matrices of the two images to obtain the final projective transformation matrix. Based on the final projective transformation matrix, it is determined whether there is an overlapping region between the two images.

[0033] If there are overlapping areas, and the overlapping area is larger than the set threshold, then delete the inspection image with poor clarity.

[0034] Preferably,

[0035] The step of comparing adjacent inspection images pairwise using the nearest neighbor algorithm includes:

[0036] The adjacent inspection images are divided into squares by a pre-set resolution, and a set of 256-dimensional features are extracted from each square by mean pooling, projection layer and L2 normalization.

[0037] The feature points of the nearest neighboring images to each grid point are found using the random projection hash method. The same method is used to find the grid points in the same image whose spatial distance exceeds a given threshold and whose feature distance is the nearest grid point. The feature distance to itself is recorded to obtain the comparison results.

[0038] Preferably,

[0039] The process of fitting the projective transformation matrices of the two images using the RANSAC algorithm to obtain the final projective transformation matrix includes:

[0040] Four pairs of valid features are randomly selected, and their projective transformation matrices are calculated.

[0041] Transform the coordinates of all valid feature pairs in the original image to the target image using a projective transformation matrix, and calculate the distance between valid feature pairs in the target image;

[0042] Arrange the distances of the effective feature pairs in ascending order, take the top 50% of the distances, and calculate their mean distance.

[0043] Repeat the above steps N times, select the target image with the smallest mean distance, take the 50% feature pairs with the smallest distance under the target image, and adjust the target image based on these feature pairs using the least square distance to obtain the final projective transformation matrix.

[0044] Preferably,

[0045] The step of determining whether two images have overlapping regions based on the final projective transformation matrix includes:

[0046] In the final projective transformation matrix, if the average coordinate error of the 50% feature pairs with the smallest distance does not exceed X pixels, and the scaling ratio of the projective transformation is between [1 / K, K], then it is determined that there is an overlapping region between the two images.

[0047] Preferably,

[0048] If the overlapping area is greater than a set threshold, a patrol image with poor clarity will be deleted, including:

[0049] After determining whether there is an overlapping area between the nth image and the (n-1)th image, if there is an overlapping area between the nth image and the (n-1)th image, and the area of ​​the overlapping area exceeds the first given threshold, then the image with poor clarity is deleted.

[0050] If the ratio of the overlapping area of ​​the (n-1)th image with the nth image or the (n-2)th image to the area of ​​the (n-1)th image exceeds a second given threshold, then the (n-1)th image is deleted.

[0051] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0052] This application constructs a defect deduplication model for power transmission inspection images. It improves upon the existing MOCO v3 training method to train the model, ensuring that features extracted from the same location under different shooting conditions are as similar as possible, while simultaneously maximizing the differences in features from different locations. Based on the constructed model, the projective transformation parameter matrix of the image is fitted using the nearest neighbor algorithm and the RANSAC algorithm. By evaluating the fitting results, it determines whether two power transmission inspection images constitute duplicates and deletes the duplicate images. This scheme makes reasonable use of the shooting order of the inspection images, judging only a few adjacent images, effectively avoiding the misidentification of images of different equipment with similar shapes (such as left, middle, and right phase insulators) as the same target and their removal by the deduplication algorithm. This solves the problem that existing defect identification systems identify a large number of duplicate potential defects from photos, and the manual review workload for defect analysis is too large.

[0053] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0054] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0055] Figure 1 This is a flowchart illustrating a method for building a defect deduplication model of power transmission inspection images according to an exemplary embodiment.

[0056] Figure 2 This is a schematic diagram illustrating a model training process according to another exemplary embodiment;

[0057] Figure 3 This is a flowchart illustrating a method for deduplicating defects in power transmission inspection images according to another exemplary embodiment. Detailed Implementation

[0058] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0059] Example 1

[0060] Figure 1 This is a flowchart illustrating a method for building a defect deduplication model of power transmission inspection images according to an exemplary embodiment, such as... Figure 1 As shown, it includes:

[0061] S1. Obtain general images and power transmission inspection images, preprocess the general images and power transmission inspection images to obtain slices for training;

[0062] S2, input the slices used for training into the built MAE model for training to obtain the pre-trained model;

[0063] S3. After performing random cropping, random rotation and random projective transformation on the power transmission inspection images, projective transformation images of multiple power transmission inspection images are obtained. Among the multiple projective transformation images, two projective transformation images with the number of grid points with the same coordinates greater than a preset threshold are selected.

[0064] S4, perform random color jittering, graying, blurring and erasing operations on two projective transformation images to obtain two slices with preset resolution and record the coordinates of the same grid points in the two slices;

[0065] S5. Input the coordinates of the two slices and their common grid points into the pre-trained model, and train the pre-trained model using the improved MOCO v3 training method to obtain the final model.

[0066] Understandably, this application acquires general images and power transmission inspection images, preprocesses them to obtain slices for training, and then inputs these slices into a pre-built MAE model for training, resulting in a pre-trained model. MAE refers to the masked autoencoders method (see the paper https: / / arxiv.org / abs / 2111.06377), which refers to random mask reconstruction (randomly removing 75% of the image content and using the remaining 25% to predict the pixel values ​​of the removed 75%). The prediction loss is MSE (mean squared error). (error); After performing random cropping, random rotation, and random projective transformation on the power transmission inspection images, projective transformation images of multiple power transmission inspection images are obtained. Among the multiple projective transformation images, two projective transformation images with the number of grid points with the same coordinates greater than a preset threshold are selected. Random color jittering, graying, blurring, and erasing operations are performed on the two projective transformation images to obtain two 512*512 slices. The coordinates of the same grid points in the two slices are recorded. The erasing method is to randomly generate a square region and set the pixel value in the region to the mean of the red, green, and blue channels. The expected area of ​​the erased region is one-third of the slice area. The coordinates of the two slices and their same grid points are input into the pre-trained model. The pre-trained model is trained using the improved MOCO v3 training method to obtain the final model. MOCO v3 is a self-supervised learning framework. For each image, two different slices are generated through random transformation. For each slice, a feature map is extracted based on the feature extraction network (backbone). After global average pooling, it is fed into the "projection head" composed of two MLP layers. The first layer ("projection head") is used to obtain an N-dimensional feature vector. This vector is then fed into a "prediction head" consisting of two additional MLP layers to obtain a feature vector of the same dimension. This prediction head is used to predict the feature vector extracted by another slice's "projection head," and the model is trained based on cosine similarity loss. This method uses power transmission inspection images for unsupervised training of the feature extraction model. Compared to existing feature extraction models trained with supervised access based on public data, this method has stronger discriminative power transmission inspection features and achieves better results.

[0067] Preferably,

[0068] The improvements to the MOCO v3 training method include:

[0069] The original global self-attention module is replaced by a local self-attention module with a window width of 16. The local self-attention module includes 12 modules, and each local self-attention module is set to scroll every X intervals. The scrolling setting includes: performing scrolling operations with step sizes of -8 and 8 on the feature maps it has acquired.

[0070] In this scheme, the improvements to the existing MOCO v3 training method are mainly in two aspects. First, to enable the ViT model to adapt to high-resolution images, a local self-attention module with a window width of 16 (corresponding to the original image resolution of 256) is used to replace the original self-attention module. At the same time, in its 2nd, 5th, 8th, and 11th blocks, before and after the self-attention module, a rolling operation with a stride of -8 and 8 is performed on the feature map (torch.roll, for an array of length N, the index of element i changes to (i+ / -8)%N after rolling), so that the receptive field of the model is not limited to the local area with a window width of 16. It is worth emphasizing that the modules performing the rolling operation here can be local self-attention modules in the order of 1, 4, 7, 10, or 3, 6, 9, 12, in addition to 2, 5, 8, 11. The order of the modules performing the rolling operation here should not be a limitation of this application.

[0071] Preferably,

[0072] The improvements to the MOCO v3 training method also include:

[0073] The feature vectors of the same grid points in two slices are obtained by using bilinear interpolation. Comparative learning is performed on a grid point basis. The "projection head" and "prediction head" in the learning process are kept consistent with the parameters at the image level.

[0074] The positive samples in the contrastive learning are features extracted from grid points on another projective transformation image corresponding to the same position of itself, and the negative samples are features from other grid points on the original power transmission inspection image that are more than 24 pixels away from itself, as well as features from other samples.

[0075] The method for obtaining the feature values ​​of the same grid points in two slices includes: feeding the two slices into the ViT network respectively, obtaining the feature maps of the slices, and using bilinear interpolation to obtain the feature vectors of the corresponding positions of specific grid points in the feature maps;

[0076] Understandably, another improvement to MOCO v3 is the use of pixel-level comparison, as shown in the attached image. Figure 2As shown, bilinear interpolation is used to obtain the feature vectors of the same grid points in two slices. Comparative learning is performed on a grid-point basis. During the learning process, the "projection head" and "prediction head" maintain consistent parameters with the image level. Positive samples are features extracted from grid points on the other slice corresponding to the same location, while negative samples are other grid points more than 24 pixels away in the original image, as well as features from other samples. The feature values ​​of the same grid points in the two slices are obtained as follows: The two slices are fed into a ViT network to obtain their feature maps. Then, `torch.nn.functional.grid_sample` is used to obtain the feature vector corresponding to a specific grid point. Simply put, `torch.nn.functional.grid_sample` provides an input and a grid, and then, based on the coordinate information provided by each position in the grid (the coordinates of the pixels in the input), the pixel values ​​of the corresponding positions in the input are filled into the specified positions in the grid to obtain the final output. Bilinear interpolation is used. The features of other samples are obtained by averaging the features generated by the ViT network and then feeding them into the "projection head". Using pixel-level comparison to train the model forces the network to extract features from the same location under different shooting conditions as similar as possible, while making features from different locations as different as possible. The dimensions of both the projection layer and the prediction layer are set to 256.

[0077] Preferably,

[0078] The preprocessing of general images and power transmission inspection images to obtain slices for training includes:

[0079] The general image is scaled down to a side length of no more than 800. The power transmission inspection image is cropped and scaled to generate 800*800 slices. 3-5 slices are generated from each original power transmission inspection image.

[0080] Images are randomly selected in a 7:3 ratio between general images and power transmission inspection images. The randomly selected images are cropped according to a preset cropping method, and the cropped images are scaled to 256*256 to obtain slices for MAE model training.

[0081] It is understandable that image preprocessing includes data preprocessing before pre-training and data processing during pre-training. Data preprocessing before pre-training includes: scaling the general image to a side length not exceeding 800 pixels, and generating 800*800 slices from the power transmission inspection image through cropping and scaling, with 3-5 slices generated from each original image. Data processing during pre-training includes: randomly selecting images according to a 7:3 ratio of general images to power transmission inspection slices, setting a cropping method, and then scaling the cropped images to 256*256 pixels to obtain slices for training. The cropping method is as follows: the side length of the cropped region is obtained by first taking the logarithm, then uniformly sampling, and finally calculating exponentially, with a minimum value of 256 and a maximum value of 800; the aspect ratio of the region is obtained in a similar manner, with a minimum value of 0.75 and a maximum value of 1.33. Compared to the RandomResizedCrop operator in the PyTorch framework, this method randomly generates region side lengths rather than region areas, thus potentially resulting in more slices with smaller areas.

[0082] Preferably,

[0083] After performing random cropping, random rotation, and random projective transformation operations on the power transmission inspection images, the resulting projective transformation images of multiple power transmission inspection images include:

[0084] The power transmission inspection images are cropped according to a preset cropping method, and the coordinates of the four vertices of the cropped area (top left, bottom left, top right, and bottom right) are recorded.

[0085] Rotate the cropping area around the center point at a random angle, and record the coordinates of the top left and top right vertices of the cropping area after rotation;

[0086] The coordinates of the top left and top right vertices are randomly moved to obtain the projective transformation coordinates of the four vertices of the cropped area to the four vertices of a square with a side length of 512*512. The original power transmission inspection image is then projectively transformed according to the projective transformation coordinates of the four vertices to obtain the projective transformation image of the power transmission inspection image. The above steps are repeated to generate multiple projective transformation images for training the improved MOCO v3 method.

[0087] Understandably, the above cropping method is used to crop the power transmission inspection image, recording the coordinates of the top left, bottom left, top right, and bottom right points of the cropped area. Then, the cropped area is rotated around the center point at a random angle (not exceeding 15 degrees), and the coordinates of the top left and top right vertices of the rotated cropped area are recorded. The coordinates of the top left and top right vertices are then randomly moved (not exceeding 1 / 10 of the side length), obtaining the projective transformation coordinates of the four vertices of the cropped area to the four vertices of a 512*512 square. The original power transmission inspection image is then projectively transformed according to the projective transformation coordinates of the four vertices to obtain the projective transformation image of the power transmission inspection image. The above steps are repeated to generate multiple projective transformation images until the number of two identical grid points in the generated multiple projective transformation images is not less than 16. The two selected projective transformation images are then randomly color-dithered, grayed out, blurred, and erased. The parameters for random color jitter, graying, and blurring are the same as those for the MOCO v2 method (public method). The erasing method is to randomly generate a square region, set the pixel value within the region to the average value of the red, green, and blue channels, and the expected area of ​​the erasing region is one-third of the slice area. After the above steps, two 512*512 slices are generated for each original image, along with the coordinates of the common grid points within them.

[0088] Example 2

[0089] This embodiment also discloses a method for deduplicating defects in power transmission inspection images, as shown in the attached figure. Figure 3 As shown, it includes:

[0090] The method is based on the final model obtained by the defect deduplication model construction method for power transmission inspection images described in any one of the above-mentioned methods, and the method includes:

[0091] S101, input all the inspection images of the tower captured into the final model, and compare adjacent inspection images pairwise according to the shooting order using the nearest neighbor algorithm;

[0092] S102, based on the comparison results, the projective transformation matrices of two adjacent inspection images are estimated to determine whether there is an overlapping region. The estimation of the projective transformation matrices of two adjacent inspection images to determine whether there is an overlapping region includes:

[0093] Obtain effective feature pairs from two adjacent inspection images; the effective feature pair is a pair of grid points from the two images, and the feature distance between them is less than the distance from each grid point to any other grid point in the same image;

[0094] If the number of valid feature pairs exceeds a preset threshold, the RANSAC algorithm is used to fit the projective transformation matrices of the two images to obtain the final projective transformation matrix. Based on the final projective transformation matrix, it is determined whether there is an overlapping region between the two images.

[0095] S103, If there is an overlapping area and the overlapping area is larger than the set threshold, delete the inspection image with poor clarity.

[0096] Understandably, for the final model established above, all the inspection images of the towers to be deduplicated are input into the final model. Following the shooting order, the nearest neighbor algorithm is used to compare adjacent inspection images pairwise. Based on the comparison results, the projective transformation matrix of two adjacent inspection images is estimated to determine if there is an overlapping region. Specifically, if a pair of grid points comes from two different images and their feature distance is less than the distance from each grid point to other grid points in the same image, they are considered a valid feature pair. If the number of valid feature pairs exceeds a given threshold (80), the RANSAC algorithm is used to fit the projective transformation matrices of the two images to obtain the final projective transformation matrix. Based on the final projective transformation matrix, it is determined whether there is an overlapping region between the two images. If there is an overlapping region, and the overlapping region is greater than the set threshold, the inspection image with poor clarity is deleted. In this scheme, the projective transformation parameter matrix of the image is fitted using the nearest neighbor algorithm and the RANSAC algorithm. By evaluating the fitting results, it is determined whether two power transmission inspection images constitute a duplicate, and duplicate images are deleted. This scheme makes reasonable use of the shooting order of the inspection images and only judges a few adjacent images, effectively avoiding the misidentification of images of different equipment with similar shapes (such as left, middle, and right phase insulators) as the same target and their removal by the deduplication algorithm. This solves the problem that existing defect identification systems will identify a large number of duplicate potential defects from photos, and the manual review of defect analysis is very labor-intensive.

[0097] Preferably,

[0098] The step of comparing adjacent inspection images pairwise using the nearest neighbor algorithm includes:

[0099] The adjacent inspection images are divided into squares by a pre-set resolution, and a set of 256-dimensional features are extracted from each square by mean pooling, projection layer and L2 normalization.

[0100] The feature points of the neighboring images that are closest to each grid point are found by random projection hashing. The same method is used to find the grid points in the same image that are spatially distant from a given threshold and have the closest feature distance. The feature distances of the grid points to themselves are recorded to obtain the comparison results.

[0101] Understandably, the finely tuned model is used to infer from the image. The image is divided into squares at a pre-specified resolution (e.g., 16*16 or 32*32), and a set of 256-dimensional features is extracted from each square using mean pooling, projection layers, and L2 normalization. The Random Projective Hash (LSH) method is used to find the feature points of the nearest neighboring images to each square's feature distance (an approximate algorithm). Similarly, within the same image, the grid point with the nearest feature distance among grid points whose spatial distance exceeds a given threshold is found, and this feature distance is recorded. The Euclidean distance between the feature vectors extracted from two grid points is the feature distance between them; the coordinate distance between two grid points on the same image is the spatial distance. The method for finding the closest feature distance using random projective hashing can be found in the reference https: / / arxiv.org / pdf / 1806.00588v1.pdf. Specifically, after L2 normalizing all feature vectors of the target image, M dimensions are randomly selected each time, and the indices (0 to M-1) of the maximum values ​​of the feature vectors in these M dimensions are recorded. This selection is repeated N times until 16MN is approximately equal to the number of features. The N indices are divided into k groups (1-4 depending on the actual situation), and the feature vectors of the target image are stored in k buckets consisting of N / k indices. Then, for each feature vector of the source image, its N indices are obtained using the same method, and the target image feature vectors located in the same bucket are queried from the k buckets. On average, each feature vector can retrieve 16k potentially matching feature vectors. The Euclidean distances between these retrieved feature vectors and the target image are calculated sequentially, and the closest matching feature vector is selected.

[0102] Preferably,

[0103] The process of fitting the projective transformation matrices of the two images using the RANSAC algorithm to obtain the final projective transformation matrix includes:

[0104] Four pairs of valid features are randomly selected, and their projective transformation matrices are calculated.

[0105] Transform the coordinates of all valid feature pairs in the original image to the target image using a projective transformation matrix, and calculate the distance between valid feature pairs in the target image;

[0106] Arrange the distances of the effective feature pairs in ascending order, take the top 50% of the distances, and calculate their mean distance.

[0107] Repeat the above steps N times, select the target image with the smallest mean distance, take the 50% feature pairs with the smallest distance under the target image, and adjust the target image based on these feature pairs using the least square distance to obtain the final projective transformation matrix.

[0108] Understandably, the process begins by randomly selecting four pairs of valid feature pairs and calculating their projective transformation matrices. Then, all valid feature pairs are transformed from their source image coordinates to the target image using these transformation matrices, and their distances to corresponding grid points are calculated. These distances are then sorted, and the smallest 50% are taken. Their mean is calculated, and this process is repeated N times (N can be 30-50). The transformation matrix with the smallest mean is found, and the 50% of feature pairs with the smallest distances under this transformation matrix are selected. Based on these feature pairs, the projective transformation matrix is ​​adjusted using the least squares distance to obtain the final transformation matrix.

[0109] Preferably,

[0110] The step of determining whether two images have overlapping regions based on the final projective transformation matrix includes:

[0111] In the final projective transformation matrix, if the average coordinate error of the 50% feature pairs with the smallest distance does not exceed X pixels, and the scaling ratio of the projective transformation is between [1 / K, K], then it is determined that there is an overlapping region between the two images.

[0112] Understandably, if, under the effect of the obtained transformation matrix, the average coordinate error of the 50% feature pairs with the smallest distance does not exceed 5 pixels, and the scaling ratio of the projective transformation is between [1 / K, K] (K is a pre-set constant, usually taken as 2-5), then it is considered that there is an overlapping area between the two images.

[0113] Preferably,

[0114] If the overlapping area is greater than a set threshold, a patrol image with poor clarity will be deleted, including:

[0115] After determining whether there is an overlapping area between the nth image and the (n-1)th image, if there is an overlapping area between the nth image and the (n-1)th image, and the area of ​​the overlapping area exceeds the first given threshold, then the image with poor clarity is deleted.

[0116] If the area of ​​the overlapping region between the (n-1)th image and the nth image or the (n-2)th image exceeds the second given threshold, then the (n-1)th image is deleted.

[0117] Understandably, after determining the nth image and the (n-1)th image, if the nth image and the (n-1)th image have an overlapping area, and the area of ​​the overlapping area exceeds a given threshold (60%), then the two images are considered to be duplicates, and the image with lower clarity is deleted; if the area of ​​the overlapping area between the (n-1)th image and the nth image or the (n-2)th image exceeds a given threshold (80%), then the (n-1)th image is deleted.

[0118] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0119] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0120] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0121] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0122] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0123] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0124] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0125] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0126] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for building a defect deduplication model for transmission line inspection images, characterized in that, The method includes: Acquire general images and power transmission inspection images, preprocess the general images and power transmission inspection images to obtain slices for training; The slices used for training are input into the built MAE model for training to obtain the pre-trained model; After performing random cropping, random rotation, and random projective transformation on the power transmission inspection images, projective transformation images of multiple power transmission inspection images are obtained. Among the multiple projective transformation images, two projective transformation images with the same number of grid points at the same coordinates greater than a preset threshold are selected. Random color jittering, graying, blurring, and erasing operations are performed on two projective transformation images to obtain two slices with preset resolutions and the coordinates of the same grid points in the two slices are recorded. The coordinates of two slices and their common grid points are input into the pre-trained model, and the pre-trained model is trained using the improved MOCO v3 training method to obtain the final model. The improvements to the MOCO v3 training method include: The original global self-attention module is replaced by a local self-attention module with a window width of 16. The local self-attention module includes 12 modules, and each local self-attention module is set to scroll every X intervals. The scrolling setting includes: performing scrolling operations with step sizes of -8 and 8 on the feature maps it has acquired. The improvements to the MOCO v3 training method also include: The feature vectors of the same grid points in two slices are obtained by using bilinear interpolation. Comparative learning is performed on a grid point basis, and the projection head and prediction head parameters are kept consistent with the image-level parameters during the learning process. The positive samples in the contrastive learning are features extracted from grid points on another projective transformation image corresponding to the same position of itself, and the negative samples are features from other grid points on the original power transmission inspection image that are more than 24 pixels away from itself, as well as features from other samples. The method for obtaining the feature values ​​of the same grid points in two slices includes: feeding the two slices into the ViT network respectively, obtaining the feature maps of the slices, and using bilinear interpolation to obtain the feature vectors of the corresponding positions of the same grid points in the two slices from the feature maps.

2. The method according to claim 1, characterized in that, The preprocessing of general images and power transmission inspection images to obtain slices for training includes: The general image is scaled to a side length of no more than 800, and the power transmission inspection image is generated in a cropped and scaled manner to 800 800 slices, and 3-5 slices are generated from each original power transmission inspection image; Images are randomly selected according to a 7:3 ratio of general images to power transmission inspection images. The randomly selected images are then cropped according to a preset cropping method, and the cropped images are scaled to 256. 256, obtain the slices used for training the MAE model.

3. The method according to claim 2, characterized in that, After performing random cropping, random rotation, and random projective transformation operations on the power transmission inspection images, the resulting projective transformation images of multiple power transmission inspection images include: The power transmission inspection images are cropped according to a preset cropping method, and the coordinates of the four vertices of the cropped area (top left, bottom left, top right, and bottom right) are recorded. Rotate the cropping area around the center point at a random angle, and record the coordinates of the top left and top right vertices of the cropping area after rotation; Randomly move the coordinates of the top left and top right vertices to obtain the coordinates of the four vertices of the clipping region after the movement, with a side length of 512. The projective transformation coordinates of the four vertices of a 512 square are used to perform a projective transformation on the original power transmission inspection image according to the projective transformation coordinates of the four vertices, resulting in a projective transformation image of the power transmission inspection image. The above steps are repeated to generate multiple projective transformation images for training the improved MOCO v3 method.

4. A method for defect deduplication of transmission inspection images, characterized by, The method is based on the final model obtained by the defect deduplication model construction method for power transmission inspection images according to any one of claims 1-3, and the method includes: All the captured inspection images of the towers are input into the final model, and adjacent inspection images are compared pairwise using the nearest neighbor algorithm according to the order of shooting. Based on the comparison results, the projective transformation matrices of two adjacent inspection images are estimated to determine whether there is an overlapping region. This estimation of the projective transformation matrices of two adjacent inspection images to determine whether there is an overlapping region includes: Obtain effective feature pairs from two adjacent inspection images; the effective feature pair is a pair of grid points from the two images, and the feature distance between them is less than the distance from each grid point to any other grid point in the same image; If the number of valid feature pairs exceeds a preset threshold, the RANSAC algorithm is used to fit the projective transformation matrices of the two images to obtain the final projective transformation matrix. Based on the final projective transformation matrix, it is determined whether there is an overlapping region between the two images. If there are overlapping areas, and the overlapping area is larger than the set threshold, then delete the inspection image with poor clarity.

5. The method according to claim 4, characterized in that, The step of comparing adjacent inspection images pairwise using the nearest neighbor algorithm includes: The adjacent inspection images are divided into squares by a pre-set resolution, and a set of 256-dimensional features are extracted from each square by mean pooling, projection layer and L2 normalization. The feature points of the nearest neighboring images to each grid point are found using the random projection hash method. The same method is used to find the grid points in the same image whose spatial distance exceeds a given threshold and whose feature distance is the nearest grid point. The feature distance to itself is recorded to obtain the comparison results.

6. The method according to claim 5, characterized in that, The process of fitting the projective transformation matrices of the two images using the RANSAC algorithm to obtain the final projective transformation matrix includes: Four pairs of valid features are randomly selected, and their projective transformation matrices are calculated. Transform the coordinates of all valid feature pairs in the original image to the target image using a projective transformation matrix, and calculate the distance between valid feature pairs in the target image; Arrange the distances of the effective feature pairs in ascending order, take the top 50% of the distances, and calculate their mean distance. Repeat the above steps N times, select the target image with the smallest mean distance, take the 50% feature pairs with the smallest distance under the target image, and adjust the target image based on these feature pairs using the least square distance to obtain the final projective transformation matrix.

7. The method according to claim 6, characterized in that, The step of determining whether two images have overlapping regions based on the final projective transformation matrix includes: In the final projective transformation matrix, if the average coordinate error of the 50% feature pairs with the smallest distance does not exceed X pixels, and the scaling ratio of the projective transformation is between [1 / K, K], then it is determined that there is an overlapping region between the two images.

8. The method according to claim 7, characterized in that, If the overlapping area is greater than a set threshold, a patrol image with poor clarity will be deleted, including: After determining whether there is an overlapping area between the nth image and the (n-1)th image, if there is an overlapping area between the nth image and the (n-1)th image, and the area of ​​the overlapping area exceeds the first given threshold, then the image with poor clarity is deleted. If the ratio of the area of the overlapping region of the (n-1)th picture with the n picture or with the (n-2)th picture to the (n-1)th picture exceeds a second given threshold, the (n-1)th picture is deleted.

Images