Column-mounted equipment metal point defect detection method and device based on image recognition

Abstract

The invention discloses a column equipment metal point defect detection method and device based on image recognition. The method comprises the following steps: collecting sample images of column equipment metal points to form a training set; establishing an improved YOLOv8 neural network model and a convolutional neural network model, and respectively training the models; obtaining video data of the to-be-detected pole-mounted equipment, and carrying out shot segmentation, key frame extraction and superimposed sliding segmentation on the video data to obtain a segmentation feature map; inputting the segmented feature map into a YOLOv8 neural network model to carry out on-column equipment identification; respectively inputting the on-column equipment feature map into a YOLOv8 neural network model and a convolutional neural network model, and respectively obtaining a first probability that the metal points have defects and a second probability that the metal points do not have defects; determining whether the metal points have defects according to the first probability and the second probability; the method can improve the accuracy of metal point defect identification.

Description

Technical Field

[0001] The present invention relates to the technical field of power transmission line defect detection, and in particular to a method and device for detecting metal point defects of pole-mounted equipment based on image recognition. Background Art

[0002] The exposed metal points at the connection of 10kV pole equipment and the aging and shedding of insulation are common problems in power systems. They not only reduce the electrical insulation performance of the line, but also make the line more susceptible to short circuit or grounding problems, increasing the risk of power outages. They also lead to power supply interruptions, increasing the difficulty and cost of maintenance. Therefore, early detection and diagnosis are crucial for the operation and maintenance of power systems.

[0003] The traditional pole-mounted equipment status detection method mainly relies on manual inspection. However, this method has problems such as large workload, low efficiency, and strong subjectivity. In order to solve these problems, a detection model based on deep learning has been born. For example, the patent text CN118037643A discloses a transmission line crimping fitting defect detection method based on deep learning, including the following steps: obtaining the transmission line crimping fitting X-ray historical image, constructing a fitting historical image data set; labeling the crimping fittings and the defect categories of the crimping fittings in the fitting historical image data set, and constructing a fitting historical image labeling data set; using the improved IC-YOLOv5 algorithm to train the fitting historical image labeling data set, and construct an X-ray image defect recognition model; obtaining the current transmission line crimping fitting X-ray image, inputting the current transmission line crimping fitting X-ray image into the X-ray image defect recognition model for defect detection, and obtaining the crimping fitting defect detection result. Patent text CN 118154980A discloses a method for segmenting and evaluating hardware corrosion, including: grading the degree of corrosion of hardware according to the color and roughness of the corrosion area, and constructing a hardware corrosion status grade table; collecting hardware images with power transmission lines, and preprocessing the hardware images according to the hardware corrosion status grade table to generate a first data set; constructing a corrosion recognition model according to an instance segmentation algorithm, and training the corrosion recognition model according to the first data set; the corrosion recognition model includes a backbone network, a region generation network, a feature aggregation module and a fully convolutional neural network; detecting hardware in the power inspection image according to the trained corrosion recognition model, obtaining the corrosion area and the corrosion status, and evaluating the degree of corrosion of the hardware according to the corrosion area and the corrosion status.

[0004] However, there are still some problems and challenges that need to be solved. First, the connection points of pole-mounted equipment may vary due to factors such as the shape of the device to which it belongs and the external environment. Therefore, the model needs to have a certain generalization ability to adapt to the detection needs in different situations, and the model provided by the above patent text may have insufficient generalization ability. Secondly, the quality and reliability of data are also one of the challenges faced by image recognition detection methods. Because image data may be affected by environmental factors, noise interference, etc., thereby affecting the accuracy of the detection results. In addition, the exposed part of the metal point at the connection of the pole-mounted equipment is extremely small, and the probability of missed detection and misidentification is relatively high. Summary of the invention

[0005] The present invention provides a method and device for detecting metal point defects of pole-mounted equipment based on image recognition, which can effectively improve the accuracy of metal point defect detection of pole-mounted equipment.

[0006] A method for detecting metal point defects of pole-mounted equipment based on image recognition, comprising:

[0007] Collect sample images of metal points of equipment on the pole and process them to form a training set;

[0008] Establish an improved YOLOv8 neural network model and a convolutional neural network model;

[0009] Based on the training set, the YOLOv8 neural network model and the convolutional neural network model are trained respectively;

[0010] Acquire video data of the pole-mounted device to be detected, perform shot segmentation, key frame extraction, and overlapping sliding segmentation on the video data, and obtain a segmentation feature map;

[0011] Input the segmentation feature map into the YOLOv8 neural network model to perform pole-mounted equipment recognition to obtain a pole-mounted equipment feature map;

[0012] Inputting the pole-mounted equipment feature map into the YOLOv8 neural network model and the convolutional neural network model respectively, and obtaining a first probability that the metal point has a defect and a second probability that the metal point does not have a defect respectively;

[0013] When the first probability is greater than a first preset value and the second probability is less than a second preset value, it is determined that the metal dot has a defect.

[0014] Furthermore, sample images of metal points of the equipment on the pole are collected and processed to form a training set, including:

[0015] Classifying and labeling the sample images according to the types of pole-mounted equipment to obtain a classification data set;

[0016] Mark the defects of metal points of each type of pole-mounted equipment to obtain a defect data set;

[0017] Rotating and interpolating sample images in the classification data set and the defect data set to obtain an expanded data set;

[0018] The expanded data set is divided to obtain a training set.

[0019] Furthermore, the video data is segmented into shots, including:

[0020] Acquire each frame of image in the video data;

[0021] Calculate the second-order difference between two adjacent frames of images to obtain a second-order difference sequence;

[0022] According to the second-order difference sequence, the second-order difference average value and the shot switching threshold are calculated in sequence;

[0023] The second-order difference average value is compared with the lens switching threshold. When the second-order difference average value is greater than the lens switching threshold, it is determined that the lens has been switched, and the image corresponding to the corresponding second-order difference sequence is determined as a lens image under an independent lens, and the next round of comparison calculation is entered until the calculation is completed to obtain the lens image under each independent lens.

[0024] Furthermore, the second-order difference is calculated by the following formula:

[0025] Δ 2 H n =|ΔH n+1 -ΔH n |;

[0026] ΔH n =k(n-1,n)·s(n-1,n)=k(n-1,n)·v(n-1,n)Δt;

[0027] Where ΔH n Represents the first-order difference between the nth frame image and the n-1th frame image, ΔH n+1 represents the first-order difference between the n+1th frame image and the nth frame image, Δ 2 H n represents the second-order difference between the nth frame image and the n-1th frame image, k(n-1,n) represents the motion feature parameter, s(n-1,n) represents the displacement from the nth frame image to the n+1th frame image, v(n-1,n) represents the average speed from the nth frame image to the n+1th frame image, and Δt represents the time from the nth frame image to the n+1th frame image;

[0028] The second-order difference average is calculated by the following formula:

[0029]

[0030] Among them, Avg(Δ 2 H(n)) represents the average value of the second-order difference of n frames of images;

[0031] The lens switching threshold is calculated by the following formula:

[0032] T=αAvg(Δ 2 H(n));

[0033] Wherein, T represents the shot switching threshold and α is a constant.

[0034] Furthermore, extracting key frames from the lens image includes:

[0035] Calculate the feature vector of each frame image under each independent lens, and use each feature vector as the initial cluster;

[0036] Calculating the Euclidean distance between the initial clusters, merging the two initial clusters with the smallest Euclidean distance, updating the initial clusters and performing the next round of Euclidean distance calculation and merging, until the number of updated clusters reaches a preset number, and obtaining the preset number of initial clusters;

[0037] Calculate the feature mean of each initial cluster according to the feature vector of each frame of the lens image in each initial cluster, and use the feature mean as the cluster center of the initial cluster;

[0038] Calculating the Euclidean distance between each frame of the lens image and the cluster center, assigning each frame of the lens image to the cluster center corresponding to the minimum Euclidean distance, and obtaining an updated cluster;

[0039] Recalculate the cluster center for the updated cluster, perform the next round of Euclidean distance calculation and the allocation of each frame lens image and update the cluster until each cluster no longer changes, and obtain the final cluster;

[0040] Calculating the mean and variance of the feature vectors in each final cluster, and determining a distance threshold based on the mean and variance;

[0041] The Euclidean distance between two frames of lens images in each final cluster is calculated, and if the Euclidean distance between the two frames of lens images is greater than the distance threshold, the corresponding two frames of lens images are determined to be key frames.

[0042] Furthermore, the key frames are overlapped and segmented to obtain a segmentation feature map, including:

[0043] Selecting a partial area in the key frame as an initial sliding window;

[0044] The sliding step length is set according to the size of the initial sliding window, wherein the sliding step length in the length direction is smaller than the length of the initial sliding window, and the sliding step length in the width direction is smaller than the width of the initial sliding window;

[0045] Starting from the initial sliding window, the key frame is segmented by sliding in sequence according to the sliding step size and the preset sliding direction, and the window obtained by each sliding is the segmentation feature map.

[0046] Furthermore, the improved YOLOv8 neural network model includes a backbone network, a neck network and a head network, and the neck network and the head network are connected through an attention mechanism module;

[0047] The backbone network is used to extract features at different stages, the neck network is used to perform multi-scale feature fusion on the extracted features, and the head network is used to output classification results.

[0048] Furthermore, the convolutional neural network model is a Faster R-CNN neural network model;

[0049] The Faster R-CNN neural network model includes a backbone feature extraction network, a candidate region proposal network, an interest region pooling module and a classification regression module; the backbone feature extraction module is used to extract backbone features, the candidate region proposal network is used to perform convolution calculations on the backbone features, generate different region proposal boxes and corresponding target probability scores, and sort the region proposal boxes according to the target probability scores, and select a preset number of region proposal boxes for output according to the sorting, the interest region pooling module is used to perform pooling operations on the region proposal boxes output by the candidate region proposal network to obtain a feature map, and the classification regression module is used to perform classification prediction and adjustment on the feature map, and output the classification result.

[0050] A metal point defect detection device for column-mounted equipment based on image recognition, comprising:

[0051] An acquisition module is used to acquire sample images of metal points of equipment on the pole and process them to form a training set;

[0052] Model building module, used to build improved YOLOv8 neural network model and convolutional neural network model;

[0053] A training module, which trains the YOLOv8 neural network model and the convolutional neural network model respectively based on the training set;

[0054] A data processing module is used to obtain video data of the pole-mounted device to be detected, perform shot segmentation, key frame extraction and overlapping sliding segmentation on the video data, and obtain a segmentation feature map;

[0055] The device identification module inputs the segmentation feature map into the YOLOv8 neural network model to perform pole-mounted device identification to obtain a pole-mounted device feature map;

[0056] A defect detection module, used to input the column equipment feature map into the YOLOv8 neural network model and the convolutional neural network model, respectively, to obtain a first probability that the metal point has a defect and a second probability that the metal point does not have a defect;

[0057] The positive and negative verification module is used to determine that the metal point has a defect when the first probability is greater than a first preset value and the second probability is less than a second preset value.

[0058] Furthermore, the acquisition module acquires sample images of metal points of the pole equipment and processes them to form a training set, including:

[0059] Classifying and labeling the sample images according to the types of pole-mounted equipment to obtain a classification data set;

[0060] Mark the defects of metal points of each type of pole-mounted equipment to obtain a defect data set;

[0061] Rotating and interpolating sample images in the classification data set and the defect data set to obtain an expanded data set;

[0062] The expanded data set is divided to obtain a training set.

[0063] Furthermore, the data processing module performs shot segmentation on the video data, including:

[0064] Acquire each frame of image in the video data;

[0065] Calculate the second-order difference between two adjacent frames of images to obtain a second-order difference sequence;

[0066] According to the second-order difference sequence, the second-order difference average value and the shot switching threshold are calculated in sequence;

[0067] The second-order difference average value is compared with the lens switching threshold. When the second-order difference average value is greater than the lens switching threshold, it is determined that the lens has been switched, and the image corresponding to the corresponding second-order difference sequence is determined as a lens image under an independent lens, and the next round of comparison calculation is entered until the calculation is completed to obtain the lens image under each independent lens.

[0068] Furthermore, the second-order difference is calculated by the following formula:

[0069] Δ 2 H n =|ΔH n+1 -ΔHn |;

[0070] ΔH n =k(n-1,n)·s(n-1,n)=k(n-1,n)·v(n-1,n)Δt;

[0071] Where ΔH n Represents the first-order difference between the nth frame image and the n-1th frame image, ΔH n+1 represents the first-order difference between the n+1th frame image and the nth frame image, Δ 2 H n represents the second-order difference between the nth frame image and the n-1th frame image, k(n-1,n) represents the motion feature parameter, s(n-1,n) represents the displacement from the nth frame image to the n+1th frame image, v(n-1,n) represents the average speed from the nth frame image to the n+1th frame image, and Δt represents the time from the nth frame image to the n+1th frame image;

[0072] The second-order difference average is calculated by the following formula:

[0073]

[0074] Among them, Avg(Δ 2 H(n)) represents the average value of the second-order difference of n frames of images;

[0075] The lens switching threshold is calculated by the following formula:

[0076] T=αAvg(Δ 2 H(n));

[0077] Wherein, T represents the shot switching threshold and α is a constant.

[0078] Furthermore, the data processing module extracts key frames from the lens image, including:

[0079] Calculate the feature vector of each frame image under each independent lens, and use each feature vector as the initial cluster;

[0080] Calculating the Euclidean distance between the initial clusters, merging the two initial clusters with the smallest Euclidean distance, updating the initial clusters and performing the next round of Euclidean distance calculation and merging, until the number of updated clusters reaches a preset number, and obtaining the preset number of initial clusters;

[0081] Calculate the feature mean of each initial cluster according to the feature vector of each frame of the lens image in each initial cluster, and use the feature mean as the cluster center of the initial cluster;

[0082] Calculating the Euclidean distance between each frame of the lens image and the cluster center, assigning each frame of the lens image to the cluster center corresponding to the minimum Euclidean distance, and obtaining an updated cluster;

[0083] Recalculate the cluster center for the updated cluster, perform the next round of Euclidean distance calculation and the allocation of each frame lens image and update the cluster until each cluster no longer changes, and obtain the final cluster;

[0084] Calculating the mean and variance of the feature vectors in each final cluster, and determining a distance threshold based on the mean and variance;

[0085] The Euclidean distance between two frames of lens images in each final cluster is calculated, and if the Euclidean distance between the two frames of lens images is greater than the distance threshold, the corresponding two frames of lens images are determined to be key frames.

[0086] Furthermore, the data processing module performs overlapping sliding segmentation on the key frames to obtain a segmentation feature map, including:

[0087] Selecting a partial area in the key frame as an initial sliding window;

[0088] The sliding step length is set according to the size of the initial sliding window, wherein the sliding step length in the length direction is smaller than the length of the initial sliding window, and the sliding step length in the width direction is smaller than the width of the initial sliding window;

[0089] Starting from the initial sliding window, the key frame is segmented by sliding in sequence according to the sliding step size and the preset sliding direction, and the window obtained by each sliding is the segmentation feature map.

[0090] Furthermore, the improved YOLOv8 neural network model includes a backbone network, a neck network and a head network, and the neck network and the head network are connected through an attention mechanism module;

[0091] The backbone network is used to extract features at different stages, the neck network is used to perform multi-scale feature fusion on the extracted features, and the head network is used to output classification results.

[0092] Furthermore, the convolutional neural network model is a Faster R-CNN neural network model;

[0093] The Faster R-CNN neural network model includes a backbone feature extraction network, a candidate region proposal network, an interest region pooling module and a classification regression module; the backbone feature extraction module is used to extract backbone features, the candidate region proposal network is used to perform convolution calculations on the backbone features, generate different region proposal boxes and corresponding target probability scores, and sort the region proposal boxes according to the target probability scores, and select a preset number of region proposal boxes for output according to the sorting, the interest region pooling module is used to perform pooling operations on the region proposal boxes output by the candidate region proposal network to obtain a feature map, and the classification regression module is used to perform classification prediction and adjustment on the feature map, and output the classification result.

[0094] A computer storage medium stores computer instructions, which are used to execute the above method when called.

[0095] The method and device for detecting metal point defects of column-mounted equipment based on image recognition provided by the present invention have at least the following beneficial effects:

[0096] (1) Perform shot segmentation, key frame extraction, and overlapping sliding segmentation on the video data of the pole-mounted equipment to be detected, thereby improving the quality and reliability of the image data to be identified, thereby improving the accuracy of subsequent identification;

[0097] (2) Improve the YOLOv8 neural network model and convolutional neural network model. YOLOv8.s is generated by adding the GAM attention mechanism and replacing the loss function EIoU on the basis of YOLOv8, which is more suitable for target recognition. The backbone network of the convolutional neural network is replaced, which effectively reduces the amount of calculation of the invalid part, improves the recognition rate, and improves the generalization ability of the model.

[0098] (3) Through the overlapping and sliding segmentation of the images, the target ratio in the original image is increased, the recognition accuracy is improved, and the missing of valid parts is avoided;

[0099] (4) Based on the improved YOLOv8 neural network model and convolutional neural network model, metal point defects are verified both positively and negatively, which avoids the problems of missed detection and false detection caused by a single algorithm and improves the credibility of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 A flow chart of an embodiment of a method for detecting metal point defects of pole-mounted equipment based on image recognition provided by the present invention.

[0101] Figure 2 A flowchart of an embodiment of forming a training set in the method for detecting metal point defects of pole-mounted equipment based on image recognition provided by the present invention.

[0102] Figure 3 A structural diagram of an embodiment of an improved YOLOv8 neural network model in the method for detecting metal point defects of pole-mounted equipment based on image recognition provided by the present invention.

[0103] Figure 4 The present invention provides a flowchart of an embodiment of performing shot segmentation on video data in the method for detecting metal point defects of pole-mounted equipment based on image recognition. Figure 5 The present invention provides a flowchart of an embodiment of extracting key frames from lens images in the method for detecting metal point defects of pole-mounted equipment based on image recognition. Figure 6 The present invention provides a flowchart of an embodiment of superimposing and sliding segmentation of key frames in the method for detecting metal point defects of pole-mounted equipment based on image recognition provided by the present invention. Figure 7 The present invention is a schematic diagram of an embodiment of a sliding window in a method for detecting metal point defects of pole-mounted equipment based on image recognition provided by the present invention. Figure 8 A schematic structural diagram of an embodiment of a metal point defect detection device for pole-mounted equipment based on image recognition provided by the present invention. DETAILED DESCRIPTION

[0104] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0105] refer to Figure 1 In some embodiments, a method for detecting metal point defects of pole-mounted equipment based on image recognition is provided, comprising:

[0106] S1, collect sample images of metal points of equipment on the column and process them to form a training set;

[0107] S2. Establish an improved YOLOv8 neural network model and a convolutional neural network model;

[0108] S3. Training the YOLOv8 neural network model and the convolutional neural network model respectively based on the training set;

[0109] S4, obtaining video data of the pole-mounted device to be detected, performing shot segmentation, key frame extraction, and overlapping sliding segmentation on the video data to obtain a segmentation feature map;

[0110] S5. Input the segmentation feature map into the YOLOv8 neural network model to perform pole-mounted equipment recognition to obtain a pole-mounted equipment feature map;

[0111] S6. Inputting the column equipment characteristic graph into the YOLOv8 neural network model and the convolutional neural network model respectively, and obtaining a first probability that the metal point has a defect and a second probability that the metal point does not have a defect respectively;

[0112] S7. When the first probability is greater than a first preset value and the second probability is less than a second preset value, it is determined that the metal dot has a defect.

[0113] Specifically, refer to Figure 2 In step S1, sample images of metal points of the equipment on the column are collected and processed to form a training set, and the following further includes:

[0114] S11, classifying and labeling the sample images according to the types of pole-mounted equipment to obtain a classification data set;

[0115] S12, marking defects of metal points of each type of pole-mounted equipment to obtain a defect data set;

[0116] S13, rotating and interpolating the sample images in the classification data set and the defect data set to obtain an expanded data set;

[0117] S14: Divide the expanded data set to obtain a training set.

[0118] In some embodiments, there are six types of pole-mounted equipment: fuse RDQ, lightning arrester BLQ, voltage transformer PT, wire clamp XJ, pole switch ZK, and pole transformer ZB. Take it as a classification data set, use the labelimg tool to annotate the image data set, annotate the image with a rectangular frame, and the defects of the metal points of the pole-mounted equipment include: the damaged wire label is DX, the lightning arrester connection point defect label is BLQ, the fuse connection point defect label is RDQ, the wire clamp protective cover defect label is XJ, and the U-shaped hanging ring label is GH, and a txt annotation file is generated. Select the annotated data set for rotation, image interpolation and other operations to achieve the purpose of expanding the data set, and at the same time divide the data set into a training set and a test set in a ratio of 8:2.

[0119] Since there are differences in production batches of 10kV pole-mounted equipment and the defect types of each pole-mounted equipment may not be the same, in order to avoid storing all images in only one data set, which would interfere with target recognition due to excessive and complex data set content, a defect data set corresponding to each device is carefully established in this embodiment.

[0120] Further, in step S2, the improved YOLOv8 neural network model includes a backbone network, a neck network and a head network, and the neck network and the head network are connected through an attention mechanism module;

[0121] The backbone network is used to extract features at different stages, the neck network is used to perform multi-scale feature fusion on the extracted features, and the head network is used to output classification results.

[0122] Further, refer to Figure 3 , the backbone network includes a first Conv module, a second Conv module, a first C2f module, a third Conv module, a second C2f module, a fourth Conv module, a third C2f module, a fifth Conv module, a fourth C2f module and an SPPF module connected in sequence;

[0123] The output of the first Conv module is connected to the input of the second Conv module, the output of the second Conv module is connected to the input of the first C2f module, the output of the first C2f module is connected to the input of the third Conv module, the output of the third Conv module is connected to the input of the second C2f module, the first output of the second C2f module is connected to the input of the fourth Conv module, the output of the fourth Conv module is connected to the input of the third C2f module, the first output of the third C2f module is connected to the input of the fifth Conv module, the output of the fifth Conv module is connected to the input of the fourth C2f module, and the output of the fourth C2f module is connected to the input of the SPPF module;

[0124] The Conv module includes a conv2d module, a BatchNorm2d module, and a SiLU module. The data input into the Conv module is input into the Conv2d module. The output of the Conv2d module is input into the Bat module. The output of the Bat module is input into the SiLU module. The output of the SiLU module is used as the output of the Conv module.

[0125] SiLU stands for SigmoidLinearUnit, also known as Swish. It is an adaptive activation function introduced by Google Brain in 2017. SiLU is an improved version of Sigmoid and ReLU. SiLU has the characteristics of no upper bound, lower bound, smoothness, and non-monotonicity. SiLU works better than ReLU on deep models and can be regarded as a smooth ReLU activation function.

[0126] SiLU(x) = x × sigmoid; (1)

[0127] Where sigmoid(x) is the standard sigmoid function, and its value is between 0 and 1. The characteristics of the Swish function include nonlinearity, continuous differentiation, and are defined in the range from negative infinity to positive infinity.

[0128] The neck network includes a sixth Conv module, a first Upsample module, a first Concat module, a fifth C2f module, a second Upsample module, a second Concat module, a sixth C2f module, a first GAM attention mechanism module, a seventh Conv module, a third Concat module, a seventh C2f module, a second GAM attention mechanism module, an eighth Conv module, a fourth Concat module, an eighth C2f module, and a third GAM attention mechanism module, wherein the first GAM attention mechanism module, the second GAM attention mechanism module, and the third GAM attention mechanism module are outputs of the neck network;

[0129] Specifically, the output of the SPPF module is connected to the input of the sixth Conv module, the first output of the sixth Conv module is connected to the input of the first Upsample module, the output of the first Upsample module is connected to the first input of the first Concat module, the second output of the third C2f module is connected to the second input of the first Concat module, the output of the first Concat module is connected to the input of the fifth C2f module, the first output of the fifth C2f module is connected to the input of the second Upsample module, the output of the second Upsample module is connected to the first input of the second Concat module, the second output of the second C2f module is connected to the second input of the second Concat module, the output of the second Concat module is connected to the output of the sixth C2f module, and the output of the sixth C2f module is connected to the first GAM attention machine The first output of the first GAM attention mechanism is connected to the input of the seventh Conv module, the output of the seventh Conv module is connected to the first input of the third Concat module, the second input of the third Concat module is connected to the second output of the fifth C2f module, the output of the third Concat module is connected to the input of the sixth C2f module, the output of the sixth C2f module is connected to the input of the second GAM attention mechanism, the output of the second GAM attention mechanism is connected to the input of the eighth Conv, the output of the eighth Conv module is connected to the first input of the fourth Concat module, the second input of the fourth Concat module is connected to the second output of the sixth Conv module, the first output of the fourth Concat module is connected to the input of the eighth C2f, and the output of the eighth C2f module is connected to the input of the third GAM attention mechanism.

[0130] Among them, the GAM attention mechanism can reduce information reduction and amplify global interaction representation to improve the performance of neural networks. This mechanism uses a channel-space attention mechanism. At the channel level, GAM first transforms the dimension of the input feature map, then inputs it into MIP to restore the original dimension, and finally outputs it through Sigmoid. At the spatial level, GAM is processed through a series of convolutions and then sent to the Sigmoid output, ultimately achieving the superposition of the attention effects at the two levels.

[0131] The head network includes a first Detect module, a second Detect module and a third Detect module, which are respectively connected to the first GAM attention mechanism, the second GAM attention mechanism and the third first GAM attention mechanism.

[0132] The traditional YOLOv8 model uses the CIoU loss function, setting the predicted box width to w, the predicted box height to h, and the real box width and height to w gt With h gt The center points of the predicted box and the real box are b and b respectively. gt , ρ is the center point b and the center point b gt The Euclidean distance between them, the width and height of the minimum bounding box is C w With C h , IoU is the intersection over union ratio. The CIoU loss function calculation formula is as follows:

[0133]

[0134] in:

[0135]

[0136]

[0137] However, CIoU can only reflect the difference between the width and length of the target box, but cannot reflect the difference between the width and length and its confidence, which hinders the optimization similarity of the model. To solve this problem, this embodiment uses EIoU to replace CIoU.

[0138] EIoU loss value L EIoU It consists of three parts: oU (Intersection over Union loss IoU loss value), L dis (the distance loss between the center point of the predicted box and the real box), L asp (Height and width loss value), the specific formula is as follows:

[0139]

[0140] EIoU takes into account the accuracy and position information of the target bounding box, allowing the model to better measure the overlap between the detection box and the real target, capture the details of the target more clearly, and achieve better results in defect detection of 10kV pole-mounted equipment.

[0141] Compared with the traditional YOLOv8, the improved YOLOv8 neural network model in this embodiment adds the GAM attention mechanism to the connection between the neck network and the head network, which can adaptively learn the feature map information of the channel and spatial dimensions, and refit and allocate the features, which helps to focus on small targets and increase the weight of small targets. In order to verify the effectiveness of the improved method, the data set was trained using the same parameters. The experimental results are shown in Table 1 below. The experiment shows that the improved YOLOv8 neural network model has a significantly improved mAP compared with the original YOLOv8 neural network. It can be seen that the improved YOLOv8 neural network model is more suitable for the detection of metal point defects in column-mounted equipment in this embodiment.

[0142]

[0143] Table 1

[0144] Furthermore, the convolutional neural network model is a FasterR-CNN neural network model.

[0145] The FasterR-CNN neural network model includes a backbone feature extraction network, a candidate region proposal network, an interest region pooling module and a classification regression module; the backbone feature extraction module is used to extract backbone features, the candidate region proposal network is used to perform convolution calculations on the backbone features, generate different region proposal boxes and corresponding target probability scores, and sort the region proposal boxes according to the target probability scores, and select a preset number of region proposal boxes for output according to the sorting, the interest region pooling module is used to perform pooling operations on the region proposal boxes output by the candidate region proposal network to obtain a feature map, and the classification regression module is used to perform classification prediction and adjustment on the feature map, and output the classification results.

[0146] In the existing Faster R-CNN neural network model, VGG16, as the backbone network of Faster R-CNN, has a network depth of 16 layers. Its structure is simple, but with the increase in the number of network layers, there will be problems of gradient explosion and gradient disappearance, resulting in a decrease in accuracy during recognition. Therefore, in this embodiment, EfficientNetV2_M is used to replace the original VGG16 backbone network. In addition, EfficientNetV2_M improves training efficiency by adopting an improved step-by-step learning method, which accelerates the learning process by gradually increasing the image size in the early stage of training. Subsequently, according to the change in image size, the regularization parameter is dynamically adjusted to maintain the accuracy of detection, thereby solving the problem of decreased accuracy caused by the increase in image size.

[0147] EfficientNetV2_M mainly uses two structures: Fused-MBConv and MBConv. EfficientNetV2_M uses a combination of Fused-MBConv and MBConv to avoid the result that EfficientNet only uses MBConv, which results in a slower training speed than Fused-MBConv.

[0148] Candidate Region Proposal Network (RPN): First, a 3x3 sliding window is used to perform convolution calculations on the feature map extracted in the previous step to further concentrate the feature information. Then it is divided into two branch tasks. One branch generates region proposal boxes of different proportions for each point on the feature map. The other branch calculates the probability score of the existence of the target in the generated region proposal box. Finally, the probability scores are sorted to obtain a specific number of region proposal boxes (Region of Interests.ROls) with larger probability scores.

[0149] Region of Interest Pooling Layer RO1: Traverse each region proposal box and pool all feature maps into 7x7 feature maps of the same size as the input for the next classification and regression.

[0150] Classification and Regression: Classification prediction (determine the category of the candidate area) and regression prediction (adjust the proposed box) are performed on the candidate area to obtain the precise position and target score of the final detection box.

[0151] In order to verify the effectiveness of the improved method, the dataset was trained using the same parameters. The experimental results are shown in Table 2. The experiment shows that the improved Faster R-CNN has improved performance in all aspects, proving that the improvement is effective.

[0152]

[0153] Table 2

[0154] Furthermore, in step S3, the YOLOv8 neural network model and the convolutional neural network model are trained respectively based on the training set to obtain the trained YOLOv8 neural network model and the convolutional neural network model.

[0155] Further, refer to Figure 4 In step S4, the video data is segmented into shots, including:

[0156] S41, acquiring each frame of image in the video data;

[0157] S42, calculating the second-order difference of two adjacent frames of images to obtain a second-order difference sequence;

[0158] S43, calculating the second-order difference average value and the shot switching threshold in sequence according to the second-order difference sequence;

[0159] S44, comparing the second-order difference average value with the lens switching threshold value. When the second-order difference average value is greater than the lens switching threshold value, it is determined that the lens has been switched, and the image corresponding to the corresponding second-order difference sequence is determined as a lens image under an independent lens, and the next round of comparison calculation is entered until the calculation is completed to obtain the lens image under each independent lens.

[0160] Further, in step S42, the second-order difference is calculated by the following formula:

[0161] Δ 2 H n =|ΔH n+1 -ΔH n |; (6)

[0162] ΔH n =k(n-1,n)·s(n-1,n)=k(n-1,n)·v(n-1,n)Δt; (7)

[0163] Where ΔH n Represents the first-order difference between the nth frame image and the n-1th frame image, ΔH n+1 represents the first-order difference between the n+1th frame image and the nth frame image, Δ 2 H n represents the second-order difference between the nth frame image and the n-1th frame image, k(n-1,n) represents the motion feature parameter, s(n-1,n) represents the displacement of the nth frame image from the n+1th frame image, v(n-1,n) represents the average speed of the nth frame image from the n+1th frame image, and Δt represents the time it takes for the nth frame image to move to the n+1th frame image.

[0164] The obtained second-order difference sequence is {Δ 2 H2,Δ 2 H3…Δ 2 H n-1}.

[0165] Further, in step S43, the second-order difference average is calculated by the following formula:

[0166]

[0167] Among them, Avg(Δ 2 H(n)) represents the average value of the second-order difference of n frames of images;

[0168] The lens switching threshold is calculated by the following formula:

[0169] T=αAvg(Δ 2 H(n)); (9)

[0170] Wherein, T represents the lens switching threshold, and α is a constant, with a value between 3.5 and 4.5.

[0171] Furthermore, in step S44, for example, when the second-order difference average value of m frames of images is greater than the corresponding lens switching threshold, it is determined that the lens has switched at this time, and the m frames of images are determined as lens images under an independent lens, and the m+1th frame image is used as the starting frame of the next lens, and the next round of comparison calculation is entered until the calculation is completed to obtain the lens image under each independent lens. Lens segmentation is used for scenes with sudden changes during drone shooting.

[0172] Further, refer to Figure 5 In step S4, key frame extraction is performed on the lens image, including:

[0173] S45, calculating the feature vector of each frame of lens image under each independent lens, and taking each feature vector as an initial cluster;

[0174] S46, calculating the Euclidean distance between the initial clusters, merging the two initial clusters with the smallest Euclidean distance, updating the initial clusters and performing the next round of Euclidean distance calculation and merging, until the number of updated clusters reaches a preset number, and obtaining the preset number of initial clusters;

[0175] S47, calculating the feature mean of each initial cluster according to the feature vector of each frame of the lens image in each initial cluster, and taking the feature mean as the cluster center of the initial cluster;

[0176] S48, calculating the Euclidean distance between each frame of the lens image and the cluster center, assigning each frame of the lens image to the cluster center corresponding to the minimum Euclidean distance, and obtaining an updated cluster;

[0177] S49, recalculating the cluster center for the updated cluster, performing the next round of Euclidean distance calculation and the allocation of each frame lens image and updating the cluster, until each cluster no longer changes, and obtaining the final cluster;

[0178] S410, calculating the mean and variance of the feature vectors in each final cluster, and determining a distance threshold according to the mean and variance;

[0179] S411, calculating the Euclidean distance between two frames of lens images in each final cluster, and if the Euclidean distance between two frames of lens images is greater than the distance threshold, determining the corresponding two frames of lens images as key frames.

[0180] In step S45, the feature vector F of the lens image is defined as:

[0181] F={H j |j=1,2,3...,Q}; (10)

[0182] Among them, H j represents the information entropy of the j-th image block. Assuming that the image is a 256-level grayscale image, the image is divided into Q blocks, and the information entropy H of each image block is:

[0183]

[0184] Information entropy is a measure of the random variable x. i The average amount of information in the i ) is a random variable x i Probability of occurrence.

[0185] Furthermore, in step S46, a hierarchical clustering method is used to obtain initial clusters: the Euclidean distance between the initial clusters is calculated, and the two initial clusters with the smallest Euclidean distance are merged, the initial clusters are updated and the next round of Euclidean distance calculation and merging is performed until the number of updated clusters reaches a preset number, and a preset number of initial clusters are obtained.

[0186] Among them, the Euclidean distance between two frames of images is calculated by the following formula:

[0187]

[0188] Wherein, d(Fi, Fj) represents the Euclidean distance between the i-th frame image and the j-th frame image, Fi represents the feature vector of the i-th frame image, and Fj represents the feature vector of the j-th frame image.

[0189] Furthermore, in steps S47-S49, after the initial clustering is obtained, K-Means clustering is used for optimization: the feature mean of each initial cluster is calculated according to the feature vector of each frame of the lens image in each initial cluster, and the feature mean is used as the cluster center of the initial cluster; the Euclidean distance between each frame of the lens image and the cluster center is calculated, and each frame of the lens image is assigned to the cluster center corresponding to the minimum Euclidean distance to obtain an updated cluster; the cluster center of the updated cluster is recalculated, and the next round of Euclidean distance calculation and allocation of each frame of the lens image are performed and the clustering is updated until each cluster no longer changes, and the final cluster is obtained.

[0190] The feature mean is calculated using the following formula:

[0191]

[0192] Where W represents the feature mean, N represents the number of feature vectors, and Fi represents the feature vector.

[0193] Furthermore, in step S410, the feature mean is calculated by formula (13), and the variance is calculated by the following formula:

[0194]

[0195] Where S represents the variance.

[0196] The distance threshold is

[0197] In step S411, the Euclidean distance between two frames of lens images in each final cluster is calculated. If the Euclidean distance between two frames of lens images is greater than the distance threshold, it is considered that the difference between frames is large, and the corresponding two frames of lens images are determined to be key frames.

[0198] Further, refer to Figure 6 In step S4, the key frames are overlapped and segmented to obtain a segmentation feature map, including:

[0199] S412, selecting a partial area in the key frame as an initial sliding window;

[0200] S413: setting a sliding step length according to the size of the initial sliding window, wherein the sliding step length in the length direction is smaller than the length of the initial sliding window, and the sliding step length in the width direction is smaller than the width of the initial sliding window;

[0201] S414, starting from the initial sliding window, sliding in sequence according to the sliding step and the preset sliding direction to segment the key frame, and the window obtained by each sliding is the segmentation feature map.

[0202] Specifically, to prevent the drone from being damaged, the distance between the drone and the target object is limited to 1 meter, which results in a smaller target being photographed. In the prior art, the image sent back by the drone is first divided into equal parts, and the equal division algorithm is as follows:

[0203] S = a*b; (15)

[0204]

[0205] Among them, S represents the area of ​​each block cut out, n represents the number of blocks divided, a represents the length of the original image, and b represents the width of the original image.

[0206] This method enlarges the proportion of the recognition target in the original image and improves the recognition accuracy. However, when dividing the image into 9 and 16 equal parts, the experiment found that the larger the n, the higher the recognition accuracy. When n is too large, the pixels of the cut image are too low and it becomes blurred and difficult to recognize. Therefore, we choose n = 4 for equal division, but only equal division will cause some devices in the image to be chopped up and unable to be recognized by the neural network.

[0207] In order to avoid this situation, this embodiment proposes a method of overlapping sliding segmentation processing, which uses the overlapping sliding segmentation method to identify the type of pole-mounted equipment. First, a partial area in the key frame is selected as the initial sliding window; the sliding step length is set according to the size of the initial sliding window, the sliding step length in the length direction is less than the length of the initial sliding window, and the sliding step length in the width direction is less than the width of the initial sliding window. For example, the sliding step length in the length direction can be set to half the length of the initial sliding window, and the sliding step length in the width direction can be set to half the width of the initial sliding window. Starting from the initial sliding window, the key frame is segmented by sliding in sequence according to the sliding step length and the preset sliding direction. The window obtained by each sliding is the segmentation feature map.

[0208] refer to Figure 7 , select 1 / 4 of the original image as the sliding window, that is, the window length is a / 2 and the width is b / 2. The left and right sliding step is a / 4 of the window length, and the up and down sliding step is b / 4 of the window width. Slide from the upper left corner to the lower right corner, that is, start from window No. 1 and end at window No. 9. This method will overlap with the previous segmented image each time. While enlarging the size ratio of the recognition target in the original image, each sliding step is only 1 / 2 of the window length, which avoids the problem of the device being cut into pieces during direct segmentation.

[0209] Furthermore, in step S5, the segmentation feature map is input into the YOLOv8 neural network model to perform pole-mounted device identification to obtain a pole-mounted device feature map, that is, the pole-mounted device is first identified through the YOLOv8 neural network model.

[0210] Further, in step S6, the characteristic map of the pole-mounted equipment is respectively input into the YOLOv8 neural network model and the convolutional neural network model to obtain a first probability that the metal point has defects and a second probability that the metal point does not have defects, respectively; that is, positive and negative verification is performed through the YOLOv8 neural network model and the convolutional neural network model, the YOLOv8 neural network model outputs the probability that the metal points of the pole-mounted equipment have defects, and the convolutional neural network model outputs the probability that the metal points of the pole-mounted equipment do not have defects.

[0211] Further, in step S7, if the first probability that the metal point has a defect output by the YOLOv8 neural network model is greater than a first preset value, and the second probability that the metal point does not have a defect output by the convolutional neural network model is less than a second preset value, it can be determined that the metal point has a defect.

[0212] refer to Figure 8 In some embodiments, a device for detecting metal point defects of a column-mounted device based on image recognition is provided, comprising:

[0213] The acquisition module 201 is used to acquire sample images of metal points of the pole equipment and process them to form a training set;

[0214] A model building module 202 is used to build an improved YOLOv8 neural network model and a convolutional neural network model;

[0215] A training module 203 is used to train the YOLOv8 neural network model and the convolutional neural network model based on the training set;

[0216] The data processing module 204 is used to collect video data of the pole-mounted device to be detected, perform shot segmentation, key frame extraction and overlapping sliding segmentation on the video data, and obtain a segmentation feature map;

[0217] The device identification module 205 inputs the segmentation feature map into the YOLOv8 neural network model to perform pole-mounted device identification to obtain a pole-mounted device feature map;

[0218] The defect detection module 206 is used to input the pole-mounted equipment characteristic map into the YOLOv8 neural network model and the convolutional neural network model respectively, and obtain a first probability that the metal point has a defect and a second probability that the metal point does not have a defect respectively;

[0219] The positive and negative verification module 207 is used to determine that the metal point has a defect when the first probability is greater than a first preset value and the second probability is less than a second preset value.

[0220] Furthermore, the acquisition module acquires sample images of metal points of the pole equipment and processes them to form a training set, including:

[0221] Classifying and labeling the sample images according to the types of pole-mounted equipment to obtain a classification data set;

[0222] Mark the defects of metal points of each type of pole-mounted equipment to obtain a defect data set;

[0223] Rotating and interpolating sample images in the classification data set and the defect data set to obtain an expanded data set;

[0224] The expanded data set is divided to obtain a training set.

[0225] Furthermore, the data processing module performs shot segmentation on the video data, including:

[0226] Acquire each frame of image in the video data;

[0227] Calculate the second-order difference between two adjacent frames of images to obtain a second-order difference sequence;

[0228] According to the second-order difference sequence, the second-order difference average value and the shot switching threshold are calculated in sequence;

[0229] The second-order difference average value is compared with the lens switching threshold. When the second-order difference average value is greater than the lens switching threshold, it is determined that the lens has been switched, and the image corresponding to the corresponding second-order difference sequence is determined as a lens image under an independent lens, and the next round of comparison calculation is entered until the calculation is completed to obtain the lens image under each independent lens.

[0230] Furthermore, the second-order difference is calculated by the following formula:

[0231] Δ 2 H n =|ΔH n+1 -ΔH n |;

[0232] ΔH n =k(n-1,n)·s(n-1,n)=k(n-1,n)·v(n-1,n)Δt;

[0233] Where ΔH n Represents the first-order difference between the nth frame image and the n-1th frame image, ΔH n+1 represents the first-order difference between the n+1th frame image and the nth frame image, Δ 2 H nrepresents the second-order difference between the nth frame image and the n-1th frame image, k(n-1,n) represents the motion feature parameter, s(n-1,n) represents the displacement from the nth frame image to the n+1th frame image, v(n-1,n) represents the average speed from the nth frame image to the n+1th frame image, and Δt represents the time from the nth frame image to the n+1th frame image;

[0234] The second-order difference average is calculated by the following formula:

[0235]

[0236] Among them, Avg(Δ 2 H(n)) represents the average value of the second-order difference of n frames of images;

[0237] The lens switching threshold is calculated by the following formula:

[0238] T=αAvg(Δ 2 H(n));

[0239] Wherein, T represents the shot switching threshold and α is a constant.

[0240] Furthermore, the data processing module extracts key frames from the lens image, including:

[0241] Calculate the feature vector of each frame image under each independent lens, and use each feature vector as the initial cluster;

[0242] Calculating the Euclidean distance between the initial clusters, merging the two initial clusters with the smallest Euclidean distance, updating the initial clusters and performing the next round of Euclidean distance calculation and merging, until the number of updated clusters reaches a preset number, and obtaining the preset number of initial clusters;

[0243] Calculate the feature mean of each initial cluster according to the feature vector of each frame of the lens image in each initial cluster, and use the feature mean as the cluster center of the initial cluster;

[0244] Calculating the Euclidean distance between each frame of the lens image and the cluster center, assigning each frame of the lens image to the cluster center corresponding to the minimum Euclidean distance, and obtaining an updated cluster;

[0245] Recalculate the cluster center for the updated cluster, perform the next round of Euclidean distance calculation and the allocation of each frame lens image and update the cluster until each cluster no longer changes, and obtain the final cluster;

[0246] Calculating the mean and variance of the feature vectors in each final cluster, and determining a distance threshold based on the mean and variance;

[0247] The Euclidean distance between two frames of lens images in each final cluster is calculated, and if the Euclidean distance between the two frames of lens images is greater than the distance threshold, the corresponding two frames of lens images are determined to be key frames.

[0248] Furthermore, the data processing module performs overlapping sliding segmentation on the key frames to obtain a segmentation feature map, including:

[0249] Selecting a partial area in the key frame as an initial sliding window;

[0250] The sliding step length is set according to the size of the initial sliding window, wherein the sliding step length in the length direction is smaller than the length of the initial sliding window, and the sliding step length in the width direction is smaller than the width of the initial sliding window;

[0251] Starting from the initial sliding window, the key frame is segmented by sliding in sequence according to the sliding step size and the preset sliding direction, and the window obtained by each sliding is the segmentation feature map.

[0252] Furthermore, the improved YOLOv8 neural network model includes a backbone network, a neck network and a head network, and the neck network and the head network are connected through an attention mechanism module;

[0253] The backbone network is used to extract features at different stages, the neck network is used to perform multi-scale feature fusion on the extracted features, and the head network is used to output classification results.

[0254] Furthermore, the convolutional neural network model is a Faster R-CNN neural network model;

[0255] The Faster R-CNN neural network model includes a backbone feature extraction network, a candidate region proposal network, an interest region pooling module and a classification regression module; the backbone feature extraction module is used to extract backbone features, the candidate region proposal network is used to perform convolution calculations on the backbone features, generate different region proposal boxes and corresponding target probability scores, and sort the region proposal boxes according to the target probability scores, and select a preset number of region proposal boxes for output according to the sorting, the interest region pooling module is used to perform pooling operations on the region proposal boxes output by the candidate region proposal network to obtain a feature map, and the classification regression module is used to perform classification prediction and adjustment on the feature map, and output the classification result.

[0256] In some embodiments, a computer storage medium is also provided, wherein the computer storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the above method.

[0257] The above-mentioned embodiments provide a method and device for detecting metal point defects of pole-mounted equipment based on image recognition, which have at least the following beneficial effects:

[0258] (1) Perform shot segmentation, key frame extraction, and overlapping sliding segmentation on the video data of the pole-mounted equipment to be detected, thereby improving the quality and reliability of the image data to be identified, thereby improving the accuracy of subsequent identification;

[0259] (2) Improve the YOLOv8 neural network model and convolutional neural network model. YOLOv8.s is generated by adding the GAM attention mechanism and replacing the loss function EIoU on the basis of YOLOv8, which is more suitable for target recognition. The backbone network of the convolutional neural network is replaced, which effectively reduces the amount of calculation of the invalid part, improves the recognition rate, and improves the generalization ability of the model.

[0260] (3) Through the overlapping and sliding segmentation of the images, the target ratio in the original image is increased, the recognition accuracy is improved, and the missing of valid parts is avoided;

[0261] (4) Based on the improved YOLOv8 neural network model and convolutional neural network model, metal point defects are verified both positively and negatively, which avoids the problems of missed detection and false detection caused by a single algorithm and improves the credibility of recognition.

[0262] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for detecting metal point defects of column-mounted equipment based on image recognition, characterized in that: include: Collect sample images of metal points of equipment on the pole and process them to form a training set; Establish an improved YOLOv8 neural network model and a convolutional neural network model; Based on the training set, the YOLOv8 neural network model and the convolutional neural network model are trained respectively; Acquire video data of the pole-mounted device to be detected, perform shot segmentation, key frame extraction, and overlapping sliding segmentation on the video data, and obtain a segmentation feature map; Input the segmentation feature map into the YOLOv8 neural network model to perform pole-mounted equipment recognition to obtain a pole-mounted equipment feature map; Inputting the pole-mounted equipment feature map into the YOLOv8 neural network model and the convolutional neural network model respectively, and obtaining a first probability that the metal point has a defect and a second probability that the metal point does not have a defect respectively; When the first probability is greater than a first preset value and the second probability is less than a second preset value, it is determined that the metal dot has a defect.

2. The method according to claim 1, characterized in that Collect sample images of metal points on the pole equipment and process them to form a training set, including: Classifying and labeling the sample images according to the types of pole-mounted equipment to obtain a classification data set; Mark the defects of metal points of each type of pole-mounted equipment to obtain a defect data set; Rotating and interpolating sample images in the classification data set and the defect data set to obtain an expanded data set; The expanded data set is divided to obtain a training set.

3. The method according to claim 1, characterized in that Segmenting the video data, including: Acquire each frame of image in the video data; Calculate the second-order difference between two adjacent frames of images to obtain a second-order difference sequence; According to the second-order difference sequence, the second-order difference average value and the shot switching threshold are calculated in sequence; The second-order difference average value is compared with the lens switching threshold. When the second-order difference average value is greater than the lens switching threshold, it is determined that the lens has been switched, and the image corresponding to the corresponding second-order difference sequence is determined as a lens image under an independent lens, and the next round of comparison calculation is entered until the calculation is completed to obtain the lens image under each independent lens.

4. The method according to claim 3, characterized in that The second order difference is calculated by the following formula: Δ 2 H n =|ΔH n+1 -ΔH n |; ΔH n =k(n-1,n)·s(n-1,n)=k(n-1,n)·v(n-1,n)Δt; Where ΔH n Represents the first-order difference between the nth frame image and the n-1th frame image, ΔH n+1 represents the first-order difference between the n+1th frame image and the nth frame image, Δ 2 H n represents the second-order difference between the nth frame image and the n-1th frame image, k(n-1,n) represents the motion feature parameter, s(n-1,n) represents the displacement from the nth frame image to the n+1th frame image, v(n-1,n) represents the average speed from the nth frame image to the n+1th frame image, and Δt represents the time from the nth frame image to the n+1th frame image; The second-order difference average is calculated by the following formula: Among them, Avg(Δ 2 H(n)) represents the average value of the second-order difference of n frames of images; The lens switching threshold is calculated by the following formula: T=αAvg(Δ 2 H(n)); Wherein, T represents the shot switching threshold and α is a constant.

5. The method according to claim 3, characterized in that: Extracting key frames from the lens image includes: Calculate the feature vector of each frame image under each independent lens, and use each feature vector as the initial cluster; Calculating the Euclidean distance between the initial clusters, merging the two initial clusters with the smallest Euclidean distance, updating the initial clusters and performing the next round of Euclidean distance calculation and merging, until the number of updated clusters reaches a preset number, and obtaining the preset number of initial clusters; Calculate the feature mean of each initial cluster according to the feature vector of each frame of the lens image in each initial cluster, and use the feature mean as the cluster center of the initial cluster; Calculating the Euclidean distance between each frame of the lens image and the cluster center, assigning each frame of the lens image to the cluster center corresponding to the minimum Euclidean distance, and obtaining an updated cluster; Recalculate the cluster center for the updated cluster, perform the next round of Euclidean distance calculation and the allocation of each frame of lens image and update the cluster until each cluster no longer changes, and obtain the final cluster; Calculating the mean and variance of the feature vectors in each final cluster, and determining a distance threshold based on the mean and variance; The Euclidean distance between two frames of lens images in each final cluster is calculated, and if the Euclidean distance between the two frames of lens images is greater than the distance threshold, the corresponding two frames of lens images are determined to be key frames.

6. The method according to any one of claims 1 to 5, characterized in that: Perform overlapping sliding segmentation on the key frames to obtain the segmentation feature map, including: Selecting a partial area in the key frame as an initial sliding window; The sliding step length is set according to the size of the initial sliding window, wherein the sliding step length in the length direction is smaller than the length of the initial sliding window, and the sliding step length in the width direction is smaller than the width of the initial sliding window; Starting from the initial sliding window, the key frame is segmented by sliding in sequence according to the sliding step size and the preset sliding direction, and the window obtained by each sliding is the segmentation feature map.

7. The method according to claim 1, characterized in that The improved YOLOv8 neural network model includes a backbone network, a neck network and a head network, and the neck network and the head network are connected through an attention mechanism module; The backbone network is used to extract features at different stages, the neck network is used to perform multi-scale feature fusion on the extracted features, and the head network is used to output classification results.

8. The method according to claim 1, characterized in that The convolutional neural network model is a Faster R-CNN neural network model; The Faster R-CNN neural network model includes a backbone feature extraction network, a candidate region proposal network, an interest region pooling module and a classification regression module; the backbone feature extraction module is used to extract backbone features, the candidate region proposal network is used to perform convolution calculations on the backbone features, generate different region proposal boxes and corresponding target probability scores, and sort the region proposal boxes according to the target probability scores, and select a preset number of region proposal boxes for output according to the sorting, the interest region pooling module is used to perform pooling operations on the region proposal boxes output by the candidate region proposal network to obtain a feature map, and the classification regression module is used to perform classification prediction and adjustment on the feature map, and output the classification result.

9. A metal point defect detection device for column equipment based on image recognition, characterized in that: include: An acquisition module is used to acquire sample images of metal points of equipment on the pole and process them to form a training set; Model building module, used to build improved YOLOv8 neural network model and convolutional neural network model; A training module, which trains the YOLOv8 neural network model and the convolutional neural network model respectively based on the training set; A data processing module is used to obtain video data of the pole-mounted device to be detected, perform shot segmentation, key frame extraction and overlapping sliding segmentation on the video data, and obtain a segmentation feature map; The device identification module inputs the segmentation feature map into the YOLOv8 neural network model to perform pole-mounted device identification to obtain a pole-mounted device feature map; A defect detection module, used to input the column equipment feature map into the YOLOv8 neural network model and the convolutional neural network model, respectively, to obtain a first probability that the metal point has a defect and a second probability that the metal point does not have a defect; The positive and negative verification module is used to determine that the metal point has a defect when the first probability is greater than a first preset value and the second probability is less than a second preset value.

10. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the method according to any one of claims 1 to 8.

Images