A method and system for detecting a power transmission line fitting by using a SAM model to amplify a sample

Abstract

This invention relates to a method and system for detecting transmission line fittings using SAM model augmentation samples. The method includes: sorting images of the types of fittings to be detected based on UAV imagery; labeling the fittings in the images; segmenting the fitting samples from the images according to the corresponding labeling files; pasting the data-augmented fitting samples into the images to construct a training dataset with a balanced distribution of fitting sample sizes; and iteratively training the target detection model using the training data to obtain a high-precision multi-category fitting detection model. This invention combines multiple data augmentation techniques to generate fitting samples, amplifying the original small sample sizes of fittings, and ultimately generating a data-trained target detection model with a balanced distribution of fitting sample sizes, achieving high-precision multi-category fitting detection.

Description

Technical Field

[0001] This invention relates to the field of power transmission line maintenance, and in particular to a method and system for detecting power transmission line fittings based on SAM model amplification samples. Background Technology

[0002] Hardware fittings are widely used metal accessories on power transmission lines, used to support, fix, and connect bare conductors, insulators, and other components. Because power transmission lines operate outdoors for extended periods, they are susceptible to various weather conditions and line load factors, making various hardware fittings on power transmission lines prone to corrosion, damage, deformation, and other faults. Therefore, regular operation and maintenance of the hardware fixed to power transmission lines is crucial.

[0003] With the increasing number of power transmission lines, the contradiction between the growing demand for operation and maintenance of hardware and the limited number of inspection personnel is becoming increasingly prominent, making the need for automated and intelligent fault diagnosis technologies for hardware more and more urgent. Currently, fault diagnosis technology for power transmission line hardware mainly relies on hardware images obtained by drone aerial photography, supported by computer vision and image processing technologies, to achieve the detection, location, and fault diagnosis of hardware targets.

[0004] For the detection and localization of hardware fittings, the current mainstream technology relies on deep learning models to autonomously extract multi-level and multi-angle features, and uses the model's powerful learning and expression capabilities to locate the hardware fitting target.

[0005] Deep learning-based object detection models can theoretically detect multiple types of hardware simultaneously, but their performance is severely limited by the distribution of different types of hardware samples in the training data. In real-world scenarios, different hardware functions and are present in varying numbers on power transmission lines, leading to an imbalanced distribution of different types of hardware samples in the acquired image data. For example, the frequency of hardware such as equalizing rings, vibration dampers, and U-shaped hanging rings is much higher than that of pre-twisted suspension clamps, wedge-shaped tension clamps, and parallel groove clamps. Training the model with data exhibiting a highly imbalanced hardware sample distribution causes the model to tend to correctly detect hardware types with larger sample sizes, while hardware types with smaller sample sizes are essentially undetectable. Therefore, it is difficult to significantly improve the performance of multi-type hardware detection.

[0006] To balance the sample size distribution of different hardware types in the training data, in many cases, the only solution is to manually extract hardware samples with small sample sizes from images captured by drones using Photoshop, and then paste them into other images to expand the sample size for that hardware type. This manual Photoshop method is labor-intensive and inefficient; furthermore, the hardware samples generated by simply cutting out and pasting images do not allow the model to fully learn the features of the hardware, making their use for training datasets ineffective.

[0007] With the rise of artificial intelligence and computer vision technology, more and more tedious manual operations have been replaced, freeing up our hands. In recent years, multimodal artificial intelligence technology combining semantics and vision has developed rapidly. Among them, the SAM (SegmentAnything Model) model has the function of segmenting objects with only the bounding box information of foreground objects in an image. Therefore, how to achieve high-precision multi-class detection of metal fittings is a worthy research topic. Summary of the Invention

[0008] To address the aforementioned issues, this invention employs an advanced SAM model for automatic trimming of hardware targets and combines various data augmentation techniques to generate hardware samples, amplifying hardware samples with limited original sample sizes. Ultimately, it generates a data training target detection model with a balanced distribution of hardware sample sizes, achieving high-precision multi-class detection of hardware.

[0009] The specific technical solution of the present invention is as follows:

[0010] A method for detecting transmission line fittings based on SAM model amplification samples, comprising:

[0011] Step (1) Based on UAV imagery, sort out the images of the types of hardware to be inspected and label the hardware in the images;

[0012] Step (2) Based on the annotation file corresponding to the image, segment the hardware samples in the image;

[0013] The image and the hardware box location information in the corresponding label file are used as input to the SAM model. The SAM model first uses an image encoder and a cue semantic encoder to encode the image and hardware box location information into feature vectors, respectively. Then, a lightweight mask decoder is used to decode the two sets of feature vectors and output a mask image.

[0014] By combining the mask image and the original image, the pixel information of all foreground regions on the corresponding mask image can be extracted from the original image, thus enabling automatic cropping of hardware targets.

[0015] Step (3) After data augmentation, the hardware samples are pasted into the image to construct a training dataset with a balanced distribution of hardware sample size;

[0016] Three projection transformation methods—scaling, rotation, and deformation—were used to perform random image processing on the hardware sample images.

[0017] Step (4) Use the training data to iteratively train the target detection model to obtain a high-precision multi-type hardware detection model.

[0018] Furthermore, in step (2), the sample size of the hardware is calculated, and this value is used to limit the number of samples to be cut for each type of hardware. The specific formula is as follows:

[0019]

[0020] Where N is the number of hardware types to be inspected; E i S represents the amplified sample size of the i-th type of hardware; i S represents the original sample size of the i-th type of hardware; Δ It is the maximum value of the original sample size of various types of hardware, i.e., S Δ =Max(S1,S2,…S N σ is the amplification coefficient, which directly affects the number of images in the amplified training dataset; E is calculated using this formula. i The numerical values ​​are used, on the one hand, to control the amplification amount of various types of hardware samples in the subsequent process, and on the other hand, to control the amount of samples during the trimming stage using the formula Min(E). i ,S i Determine the cutting quantity for each type of hardware sample.

[0021] Furthermore, in step (1), the metal objects in the image are outlined and labeled using a labeling tool. The information in the label outline includes the category name and location information.

[0022] The category name uses the first letters of the pinyin of the name of the framed hardware; the location information includes the coordinates of the upper left and lower right corners of the annotation box; after an image is annotated, all annotation information in the image is saved in the xml tag file in VOC data format.

[0023] Furthermore, in step (3), the projection transformation of the hardware sample image is as follows:

[0024] The overall image processing formula is expressed as follows:

[0025]

[0026] Where x and y represent the x-coordinate and y-coordinate of a pixel before the image projection transformation, and x′ and y′ represent the new x-coordinate and y-coordinate of the pixel after the projection transformation. The projection transformation process converts the color information of all pixels (x, y) in the image to (x′, y′); A s A r A d Representing the scaling matrix, rotation matrix, and deformation matrix respectively, they can be expressed as:

[0027]

[0028]

[0029]

[0030] Where w and h represent the scaling factors of the image along the width and length directions, respectively; θ is the image rotation angle; ω and These represent the degree of deformation of the image along the x-axis and y-axis, respectively.

[0031] Furthermore, in step (3), in order to obtain a hardware sample image with a random and reasonable shape, before each projection transformation process, w, h, θ, ω, All five parameters are randomly selected within a certain range: w and h range from [0.5, 1.2], θ, ω, and... The angle range is between -15 degrees and 15 degrees.

[0032] Furthermore, in step (3), the color conversion of the hardware sample image is as follows:

[0033] First, convert the image from the RGB color space to the HSV color space. While keeping the hue (H) component unchanged, randomly adjust the saturation (S) and brightness (V) components. The formulas for randomly adjusting the S and V components of each pixel in the image are as follows:

[0034]

[0035] In the formula, S ′ and V ′ α1 and α2 represent the randomly adjusted saturation and brightness, respectively, and are random values ​​within the range [0.75, 1.25]. α1 and α2 are randomly selected once before color conversion for each image. After color conversion, the image is converted from the HSV color space back to the RGB color space.

[0036] Saturation indicates how close a color is to the spectral color; high saturation results in a deep and vibrant color. Brightness indicates the lightness or darkness of a color. Randomly adjusting the color saturation and brightness of an image can generate hardware samples with different color characteristics without changing the actual color of the hardware sample.

[0037] Furthermore, in step (3), noise is added to the hardware sample image as follows:

[0038] For the hardware sample image to be processed, firstly, the R, G, and B channel components of each pixel in the image are normalized to the range of [0, 1]. Secondly, a Gaussian noise array with a mean of 0, a standard deviation of 0.1, and the same dimension as the image is randomly generated. Finally, the image is added to the Gaussian noise array to obtain a noisy image, and the pixel components of the processed image are inversely normalized by multiplying by 255 to restore them to the original range.

[0039] Further, in step (3), a training dataset with a balanced distribution of metal fitting sample sizes is constructed as follows:

[0040] Traverse all the images obtained in step (1), and after data augmentation, randomly paste the hardware sample images into random positions in the images to construct a training dataset with a balanced distribution of hardware sample size.

[0041] During the traversal process, the amplified sample size of each type of hardware obtained in step (2) is sorted in descending order, and the hardware sample images with the largest amplified sample size are pasted in sequence. For the i-th type of hardware, for each hardware of this type pasted, E i Subtract 1 from the value of E i Only when the value is 0 will the next type of hardware sample image be processed sequentially;

[0042] When performing data augmentation on the hardware sample image, a projection transformation is performed first, followed by color conversion. Then, a random number in the range (0, 1) is generated. If the random number is greater than 0.7, noise is added to the image.

[0043] When pasting the hardware sample image into the image, take a random integer Num in the range [1, 3] and perform Num data augmentation on the hardware sample image to obtain Num processed hardware sample images;

[0044] These sample images are pasted one by one to random positions in the image in a pixel-overlay manner, and the bounding rectangle information of the pasted hardware sample is written into the corresponding XML file. Each time the image is pasted, the position is checked for conflict by combining the information of the existing annotation box in the XML file. If the position conflicts, the position information is randomly re-selected.

[0045] Further, in step (4), the target detection model is trained using the training dataset with a balanced distribution of the obtained hardware sample size to obtain a high-precision multi-type hardware detection model.

[0046] The YOLOX model was used as the detection model to determine the types of hardware that the model needed to train and learn.

[0047] The YOLOX model is trained using an adaptive moment estimation optimizer as the network's training optimization strategy; the learning rate is dynamically adjusted starting from 0.0003, and multiplied by 0.9 after each round of updates;

[0048] In the first 25 rounds, the weight parameters of the backbone network and the feature fusion network are frozen, and only the network layers that are not frozen are fine-tuned to participate in the gradient update in backpropagation; in the last 25 rounds, the model as a whole is trained.

[0049] The model loss is calculated after each round of training. When the loss value does not decrease for 5 consecutive rounds, training is stopped, and the final multi-type hardware detection model is obtained.

[0050] The present invention also relates to a transmission line fitting detection system for SAM model amplification samples, characterized in that: it includes a memory, a processor, and a computer program on the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0051] This invention proposes an automatic amplification method for hardware samples, which generates data with a balanced distribution of hardware sample volume to train a target detection model, thereby achieving high-precision multi-type detection of hardware.

[0052] This invention proposes an automatic cropping method for hardware samples based on the advanced segmentation model SAM, which automatically crops hardware samples in the image according to the annotation box information of the existing annotation file.

[0053] For the cropped hardware sample images, three random image transformation methods—projection transformation, color conversion, and noise addition—were designed for data augmentation to provide the model with higher quality training data, enabling the model to learn more general features of different types of hardware.

[0054] This invention achieves high-precision multi-type hardware inspection by balancing the sample size distribution. Compared with manual PS (photocopying) techniques, this method eliminates the need for manual trimming of hardware samples, significantly reducing manpower and material resources and greatly improving operational efficiency. Furthermore, it enhances the feature representation of hardware samples before pasting them using data augmentation algorithms, providing higher-quality training data for the model and enabling it to learn more universal features across different types of hardware. Our method streamlines the hardware sample expansion, balancing sample distribution, and data augmentation processes, providing a data foundation for training multi-type hardware inspection models. Attached Figure Description

[0055] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0056] Figure 2 This is a data flow diagram according to an embodiment of the present invention;

[0057] Figure 3 This is a system block diagram of an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] Unless otherwise defined, the technical or scientific terms used in the embodiments of this application shall have the ordinary meaning understood by one of ordinary skill in the art. The terms "first," "second," and similar terms used in this embodiment do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two elements. Terms such as "upper," "lower," "left," "right," "horizontal," and "vertical" are used only relative to the orientation of the components in the accompanying drawings. These directional terms are relative concepts used for relative description and clarification, and they may change accordingly depending on the orientation of the components in the accompanying drawings.

[0060] like Figure 1 As shown in this embodiment, a method for detecting transmission line fittings using SAM model amplification samples includes:

[0061] Step (1) Based on UAV imagery, sort out images containing the types of hardware to be detected and label the hardware in the images.

[0062] In the early stages, image data was collected by inspecting power transmission lines using drones. However, since the images captured by the drones contained a large number of images that did not include the hardware items to be inspected, it was necessary to manually sort the images to select those that contained the types of hardware items to be inspected.

[0063] Using annotation tools, the metal objects in the image are outlined and labeled. The label information includes the category name and location information. The category name uses the first letters of the pinyin of the outlined metal object's name; the location information includes the coordinates of the top-left and bottom-right corners of the label, namely Xmin (X-coordinate of the top-left corner), Ymin (Y-coordinate of the top-left corner), Xmax (X-coordinate of the bottom-right corner), and Ymax (Y-coordinate of the bottom-right corner). After an image is annotated, all annotation information in that image is saved in VOC data format into an XML tag file.

[0064] Step (2) Based on the annotation file corresponding to the image, the SAM model is used to segment the hardware samples in the image.

[0065] The SAM model is an advanced image segmentation model based on Foundation Model 3, distributed by Facebook AI in 2020. This model utilizes traditional computer vision techniques and deep learning algorithms, trained on a massive dataset containing 11 million images and 1.1 billion segmentation masks, demonstrating excellent generalization performance and thus applicability across various fields. The SAM model can accurately segment any foreground object in an image based on simple input cues (such as coordinates, bounding boxes, or masks). Therefore, this embodiment uses the SAM model to replace manual cropping, automatically cropping the hardware from the image based on the bounding box information in the corresponding annotation file.

[0066] The flowchart below shows the automatic cropping and sample acquisition of hardware targets using the SAM model. The image and the hardware bounding box location information from the corresponding XML tag file are used as input to the SAM model. The SAM model first uses an image encoder and a cue semantic encoder to encode the image and hardware bounding box location information into feature vectors, respectively. Then, a lightweight mask decoder decodes the two sets of feature vectors, outputting a mask image. The mask image is a binary grayscale image consisting only of 0s and 255s, where 0 represents the foreground (black) and 255 represents the background (white). By combining the mask image and the original image, the pixel information of all foreground regions corresponding to the mask image is extracted from the original image, thus achieving automatic cropping of the hardware target.

[0067] Create a separate folder for each type of hardware to store sample images. Save the hardware samples cropped from the SAM model as PNG images with transparent backgrounds and store them in the corresponding folder. For example... Figure 2 As shown.

[0068] Since balancing the sample size distribution of hardware primarily involves amplifying hardware types with initially small sample sizes, only a small number of samples need to be cropped for amplification for hardware types with already large sample sizes. Not all hardware samples in all images need to be cropped. To balance the hardware sample size distribution and effectively reduce overall cropping time, this embodiment designs a formula for calculating the amplified sample size of hardware, and uses this value to limit the cropping quantity for each type of hardware sample. The specific formula is as follows:

[0069]

[0070] Where N is the number of hardware types to be inspected; E i S represents the amplified sample size of the i-th type of hardware; i S represents the original sample size of the i-th type of hardware; Δ It is the maximum value of the original sample size of various types of hardware, i.e., S Δ=Max(S1,S2,…S N σ is the amplification coefficient, which directly affects the number of images in the amplified training dataset; in this embodiment, it is set to 0.1. E is calculated using this formula. i The numerical values ​​are used, on the one hand, to control the amplification amount of various types of hardware samples in the subsequent process, and on the other hand, to control the amount of samples during the trimming stage using the formula Min(E). i ,S i Determine the cutting quantity for each type of hardware sample.

[0071] Step (3) In order to enhance the feature representation of the hardware samples and enable the model to learn more general features of different types of hardware, this embodiment designs three random image transformation methods—projection transformation, color conversion, and noise addition—to augment the hardware sample images obtained in step (2), providing the model with higher quality training data. Then, the hardware samples are randomly pasted into random positions in the image to construct a training dataset with a balanced distribution of hardware sample size.

[0072] Step 3-1: Projection Transformation of Hardware Sample Image

[0073] Training the model using hardware samples of different scales and angles allows the model to learn more general hardware features. To generate hardware samples with different features from the same sample, this invention employs three projection transformation methods—scaling, rotation, and deformation—to perform random image processing on the hardware sample images. The overall image processing formula can be expressed as:

[0074]

[0075] Where x and y represent the x-coordinate and y-coordinate of a pixel point before the image projection transformation, x ′ and y ′ This represents the new x and y coordinates of the pixel after projection transformation. The projection transformation process converts the color information of all pixels (x, y) in the image to (x, y). ′ y ′ (A) s A r A d These represent the scaling matrix, rotation matrix, and deformation matrix, respectively, and can be expressed as:

[0076]

[0077]

[0078]

[0079] Where w and h represent the scaling factors of the image along the width and length directions, respectively; θ is the image rotation angle; ω and These represent the degree of image deformation along the x-axis and y-axis, respectively. In this embodiment, to obtain randomly shaped and reasonable hardware sample images, before each projection transformation process, w, h, θ, ω, All five parameters are randomly selected within a certain range: w and h range from [0.5, 1.2], θ, ω, and... The angle range is between -15 degrees and 15 degrees.

[0080] Step 3-2: Color Conversion of Hardware Sample Image

[0081] In reality, metal fittings are typically silver-gray. Directly transforming the red (R), green (G), and blue (B) components of an RGB image easily generates metal fitting colors that don't exist in real-world scenarios. Therefore, this patent first converts the image from the RGB color space to the HSV color space, and while keeping the hue (H) component unchanged, randomly adjusts the saturation (S) and brightness (V) components. The formulas for randomly adjusting the S and V components of pixels in the image are as follows:

[0082]

[0083] In the formula, S ′ and V ′ α1 and α2 represent the randomly adjusted saturation and brightness, respectively, and are random values ​​within the range [0.75, 1.25]. α1 and α2 are randomly selected once before color conversion for each image. After color conversion, the image is converted from the HSV color space back to the RGB color space.

[0084] Saturation indicates how close a color is to the spectral color; high saturation results in a deep and vibrant color. Brightness indicates the lightness or darkness of a color. Randomly adjusting the color saturation and brightness of an image can generate hardware samples with different color characteristics without altering the actual color of the hardware sample.

[0085] Step 3-3: Add noise to the hardware sample image

[0086] In practical applications, images may be affected by various kinds of interference and noise. If the model is only trained on high-quality images, it is easy to overfit. Therefore, this patent randomly adds Gaussian noise to the hardware sample images to improve the robustness and anti-interference ability of the trained detection model.

[0087] For the hardware sample image to be processed, firstly, the R, G, and B channel components of each pixel in the image are normalized to the range of [0, 1]. Secondly, a Gaussian noise array with a mean of 0, a standard deviation of 0.1, and the same dimension as the image is randomly generated. Finally, the image is added to the Gaussian noise array to obtain a noisy image, and the pixel components of the processed image are inversely normalized by multiplying by 255 to restore them to the original range.

[0088] Steps 3-4: Construct a training dataset with balanced sample size distribution for metal fittings

[0089] Traverse all images obtained in step (1), and after data augmentation, randomly paste the hardware sample images into random positions in the images to construct a training dataset with a balanced distribution of hardware sample size.

[0090] During the traversal process, the amplified sample size of each type of hardware obtained in step (2) is sorted in descending order, and the hardware sample images with the largest amplified sample size are pasted in order.

[0091] For the i-th type of hardware, for each hardware of that type pasted, E i Subtract 1 from the value of E i Only when the value is 0 will the next type of hardware sample image be processed sequentially.

[0092] When performing data augmentation on hardware sample images, a projection transformation is performed first, followed by color conversion. Then, a random number within the range (0, 1) is generated. If this random number is greater than 0.7, noise is added to the image.

[0093] When pasting the hardware sample images into the image, a random integer Num within the range [1, 3] is used to perform Num data augmentation operations on the hardware sample images, resulting in Num processed hardware sample images. These sample images are then pasted one by one into random positions within the image, with each image being pixel-overlapped. The bounding rectangle information of the pasted hardware sample is written to the corresponding XML file. Each time the image is pasted, the position is checked against the existing bounding box information in the XML file to determine if there is a conflict. If a conflict is found, the position information is randomly recalculated.

[0094] Step (4) uses the training dataset with balanced distribution of metal fitting sample size obtained in steps (3)-(4) to train the target detection model and obtain a high-precision multi-type metal fitting detection model.

[0095] In this embodiment, the YOLOX model with high detection performance is used as the detection model. The types of hardware that the model needs to train and learn include equalizing rings, pre-twisted suspension clamps, vibration dampers, bag suspension clamps, connecting plates, U-shaped hanging rings, hanging plates, adjusting plates, shielding rings, spacers, compression tension clamps, parallel groove clamps, and wedge tension clamps, totaling 13 types.

[0096] The YOLOX model was trained using an adaptive moment estimation optimizer as the network's training strategy, for a total of 50 epochs. The learning rate was dynamically adjusted starting from 0.0003, and multiplied by 0.9 after each epoch. For the first 25 epochs, the weight parameters of the backbone network and feature fusion network were frozen, and only the unfrozen network layers were fine-tuned for gradient updates during backpropagation. The last 25 epochs trained the entire model. The model loss was calculated after each training epoch, and training was stopped when the loss value did not decrease for five consecutive epochs, resulting in the final multi-type metalware detection model.

[0097] In this embodiment, the trained YOLOX model is deployed on a computing platform that needs to detect transmission line fittings. Transmission line inspection videos collected by drones are transmitted to the computing platform via the network. The platform then activates the improved YOLOX model to perform frame-by-frame detection of the video. Each frame is first scaled to 416×416 pixels and preprocessed using normalization. The detected fittings include 13 categories: equalizing rings, pre-stretched suspension clamps, vibration dampers, bag-lift suspension clamps, connecting plates, U-shaped hanging rings, hanging plates, adjusting plates, shielding rings, spacers, compression tension clamps, parallel groove clamps, and wedge-shaped tension clamps. If a fitting is detected in the image, its coordinates and category are recorded. After the video detection is completed, a visualization report is generated for power industry personnel to further analyze and process the line.

[0098] like Figure 3 As shown in the figure, this embodiment provides a transmission line fitting detection system for SAM model amplification samples, including a memory, a processor, and a computer program on the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above method.

[0099] The processors mentioned above can be general-purpose processors, including central processing units, network processors, etc.; they can also be digital signal processors, application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0100] Optionally, embodiments of this application also provide a storage medium storing instructions that, when run on a computer, cause the computer to perform the methods described in the above embodiments.

[0101] Optionally, embodiments of this application also provide a chip for executing instructions, the chip being used to execute the methods of the embodiments shown above.

[0102] This application also provides a program product, which includes a computer program stored in a storage medium. At least one processor can read the computer program from the storage medium, and when the at least one processor executes the computer program, it can implement the method of the above embodiments.

[0103] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting transmission line fittings based on SAM model amplification samples, characterized in that: include: Step (1) Based on UAV imagery, sort out the images of the types of hardware to be inspected and label the hardware in the images; Step (2) Based on the annotation file corresponding to the image, segment out the hardware samples in the image; The image and the hardware bounding box location information in the corresponding label file are used as input to the SAM model. The SAM model first uses an image encoder and a cue semantic encoder to encode the image and hardware bounding box location information into feature vectors, respectively. Then, a lightweight mask decoder is used to decode the two sets of feature vectors and output a mask image. By combining the mask image and the original image, the pixel information of all foreground regions on the corresponding mask image can be extracted from the original image, thus enabling automatic cropping of hardware targets. Step (3) After data augmentation, the hardware samples are pasted into the image to construct a training dataset with a balanced distribution of hardware sample size; Three projection transformation methods—scaling, rotation, and deformation—were used to perform random image processing on the hardware sample images. Construct a training dataset with a balanced sample size distribution for metal fittings as follows: Traverse all the images obtained in step (1), and after data augmentation, randomly paste the hardware sample images into random positions in the images to construct a training dataset with a balanced distribution of hardware sample size. During the traversal process, the amplified sample size of each type of hardware obtained in step (2) is sorted in descending order, and the hardware sample images with the largest amplified sample size are pasted in sequence. For the i-th type of hardware, for each hardware of that type pasted, then Subtract 1 from the value, when When =0, the next type of hardware sample image will be processed sequentially; When performing data augmentation on the hardware sample image, a projection transformation is performed first, followed by color conversion. Then, a random number in the range (0, 1) is generated. If the random number is greater than 0.7, noise is added to the image. When pasting the hardware sample image into the image, take a random integer Num in the range [1, 3] and perform Num data augmentation on the hardware sample image to obtain Num processed hardware sample images; These sample images are pasted one by one to random positions in the image in a pixel-overlay manner, and the bounding rectangle information of the pasted hardware sample is written into the corresponding XML file. Each time the image is pasted, the position conflict is checked by combining the information of the existing annotation box in the XML file. If the position conflict is found, the position information is randomly re-selected. Step (4) Use the training data to iteratively train the target detection model to obtain a high-precision multi-type hardware detection model.

2. The method according to claim 1, characterized in that: In step (2), the sample size for expanding the hardware is calculated, and this value is used to limit the number of samples to be cut for each type of hardware. The specific formula is as follows: ； Where N is the number of hardware types to be inspected; This represents the amplified sample size for the i-th type of hardware. This represents the original sample size of the i-th type of hardware. It is the maximum value of the original sample size of various types of hardware, that is ; The amplification coefficient directly affects the number of images in the amplified training dataset; it is calculated using this formula. The numerical values ​​are used, on the one hand, to control the amplification amount of various types of hardware samples in the subsequent process, and on the other hand, to implement formulas during the trimming stage. Determine the cutting quantity for each type of hardware sample.

3. The method according to claim 1, characterized in that: In step (1), the hardware target in the image is outlined and labeled using the labeling tool. The information in the labeling box includes the category name and location information. The category name uses the first letters of the pinyin of the name of the framed hardware; the location information includes the coordinates of the upper left and lower right corners of the annotation box; after an image is annotated, all the annotation information in the image is saved in the xml tag file in VOC data format.

4. The method according to claim 1, characterized in that: In step (3), the projection transformation of the hardware sample image is as follows: The overall image processing formula is expressed as follows: Where x and y represent the x-coordinate and y-coordinate of a pixel point before the image projection transformation. and This represents the new x and y coordinates of the pixel after projection transformation. The projection transformation process converts the color information of all pixels (x, y) in the image to (x, y). , ) place; , , Representing the scaling matrix, rotation matrix, and deformation matrix respectively, they can be expressed as: Where w and h represent the scaling factors of the image along the width and length directions, respectively; The rotation angle of the image; and These represent the degree of deformation of the image along the x-axis and y-axis, respectively.

5. The method according to claim 4, characterized in that: In step (3), in order to obtain random and reasonable shape metal fitting sample images, before each projection transformation process, w, h, , , All five parameters are randomly selected within a certain range: the values ​​of w and h are [0.5, 1.2]. , and The angle range is between -15 degrees and 15 degrees.

6. The method according to claim 1, characterized in that: In step (3), the color conversion of the hardware sample image is as follows: First, convert the image from the RGB color space to the HSV color space. While keeping the hue (H) component unchanged, randomly adjust the saturation (S) and brightness (V) components. The formulas for randomly adjusting the S and V components of each pixel in the image are as follows: In the formula, and These represent the saturation and brightness after random adjustments, respectively. and The values ​​are random values ​​within the range [0.75, 1.25], and are used before color conversion for each image. and Perform a random value selection, and after color conversion, convert the image from the HSV color space back to the RGB color space; Saturation indicates how close a color is to the spectral color; high saturation results in a deep and vibrant color. Brightness indicates the lightness or darkness of a color. Randomly adjusting the color saturation and brightness of an image generates hardware samples with different color characteristics without altering the actual color of the hardware sample.

7. The method according to claim 1, characterized in that: In step (3), noise is added to the hardware sample image as follows: For the hardware sample image to be processed, firstly, the R, G, and B channel components of each pixel in the image are normalized to the range of [0, 1]. Secondly, a Gaussian noise array with a mean of 0, a standard deviation of 0.1, and the same dimension as the image is randomly generated. Finally, the image is added to the Gaussian noise array to obtain a noisy image, and the pixel components of the processed image are inversely normalized by multiplying by 255 to restore them to the original range.

8. The method according to claim 1, characterized in that: Step (4) Use the training dataset with balanced distribution of the obtained hardware sample size to train the target detection model and obtain a high-precision multi-type hardware detection model; The YOLOX model was used as the detection model to determine the types of hardware that the model needed to train and learn. The YOLOX model is trained using an adaptive moment estimation optimizer as the network's training optimization strategy. The learning rate is dynamically adjusted starting from 0.0003, and after each round of updates, the learning rate is multiplied by 0.

9. In the first 25 rounds, the weight parameters of the backbone network and the feature fusion network are frozen, and only the network layers that are not frozen are fine-tuned to participate in the gradient update in backpropagation; in the last 25 rounds, the model as a whole is trained. The model loss is calculated after each round of training. When the loss value does not decrease for 5 consecutive rounds, training is stopped, and the final multi-type hardware detection model is obtained.

9. A transmission line fitting testing system based on SAM model amplification samples, characterized in that: It includes a memory, a processor, and a computer program on the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method of any one of claims 1 to 8.

Images