Insulator defect detection method, system and equipment based on improved SAM segmentation model

Abstract

The invention discloses an insulator defect detection method based on an improved SAM segmentation model. The method comprises the steps of firstly collecting a sample image and a to-be-detected image of an insulator; then, performing image annotation on the sample images, and training a YOLOv5 model and improving an SAM segmentation model by using the sample images subjected to image annotation; inputting a to-be-detected image into the trained YOLOv5 model for target detection, identifying each object in the image and correspondingly generating a bounding box, extracting bounding box information of each object, performing format conversion on the bounding box information, inputting the bounding box information into the trained improved SAM segmentation model to perform zero sample segmentation, outputting a segmentation result, and finally performing zero sample segmentation on the basis of the segmentation result. And determining the defect position and type of the insulator. According to the method, the YOLOv5 and the improved SAM segmentation model are integrated, so that an object can be accurately segmented, the defect detection accuracy is improved, and the method can adapt to newly appearing defect types.

Description

Technical Field

[0001] The present invention belongs to the technical field of insulator defect detection, and in particular relates to an insulator defect detection method, system, equipment and medium based on an improved SAM segmentation model. Background Art

[0002] Power line insulators are used to isolate electrical conductors from their supporting structures, ensuring that current does not flow to the ground or other unintended paths. They not only ensure the effective transmission of electricity, but also improve the safety of the system, playing a key role in the power transmission system. However, due to long-term exposure to harsh environmental conditions, power line insulators are prone to various defects, which may be caused by physical damage, chemical corrosion or other external factors. If not detected and handled in time, they may cause power outages, equipment damage, and even safety hazards to the public and maintenance personnel.

[0003] Traditional power line insulator inspection mainly relies on manual inspection on the ground, which is time-consuming and labor-intensive, and may miss defects that are high or difficult to observe. With the development of drone technology, aerial inspection of power lines has become possible. Drones can capture high-resolution aerial images, providing a new perspective for comprehensive inspection of insulators. However, how to accurately detect insulator defects from complex backgrounds has also become a new challenge. Summary of the invention

[0004] The purpose of the present invention is to provide an insulator defect detection method, system, device and medium based on an improved SAM segmentation model, which can not only improve the accuracy of defect detection, but also ensure adaptation to new defect types, in order to solve the above problems existing in the prior art.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows:

[0006] In a first aspect, the present invention provides an insulator defect detection method based on an improved SAM segmentation model, the insulator defect detection method comprising the following steps:

[0007] S1, collecting sample images of insulators and images to be tested;

[0008] S2. Annotate the sample images, and use the annotated sample images to train the YOLOv5 model and the improved SAM segmentation model;

[0009] S3. Input the image to be detected into the trained YOLOv5 model for target detection, identify each object in the image and generate a corresponding bounding box, then extract the bounding box information of each object, convert its format and input it into the trained improved SAM segmentation model to perform zero-sample segmentation, output the segmentation result, and finally determine the defect location and type of the insulator based on the segmentation result.

[0010] The bounding box information output by the YOLOv5 model includes the center point coordinates of the bounding box, the length and width of the bounding box. The bounding box information output by the YOLOv5 model is converted into a coordinate format acceptable to the improved SAM segmentation model based on the following formula:

[0011]

[0012]

[0013]

[0014]

[0015] In the above formula, x a ,y a x is the horizontal and vertical coordinates of the upper left corner of the rectangular box of the input improved SAM segmentation model; b ,y b x are the horizontal and vertical coordinates of the lower right corner of the rectangular box of the input improved SAM segmentation model; center ,y center L are the horizontal and vertical coordinates of the center point of the bounding box output by the YOLOv5 model; width , L height They respectively represent the length and width data of the rectangular box input into the improved SAM segmentation model.

[0016] The improved SAM segmentation model includes an image encoder, a hint encoder and a mask decoder. The mask decoder replaces the original MLP module with an improved MLP module. The improved MLP module includes a height branch, a width branch and a channel information branch. The height branch is used to first perform a height channel rearrangement operation on the input image X, and then use a linear layer with a weight of WH to encode the height information to obtain an output XH of the height branch; the width branch is used to first perform a width channel rearrangement operation on the input image X, and then use a linear layer with a weight of WW to encode the width information to obtain an output XW of the width branch; the channel information branch is used to encode the channel information of the input image X using a linear layer with a weight of WC to obtain an output XC of the channel branch; the output XH of the height branch, the output XW of the width branch, and the output X of the channel branch are first weighted summed and then information fused through a linear layer with a weight of W' to obtain an output Y of the improved MLP module; the output Y of the improved MLP module is formulated as follows:

[0017] Y=W′(α1*XH+α2*XW+α3*XC);

[0018] In the above formula, α1, α2, and α3 are weight coefficients used to control the relative importance of height branch, width branch, and channel branch, respectively.

[0019] The training improved SAM segmentation model refers to training the mask decoder and freezing other modules besides it.

[0020] In a second aspect, the present invention provides an insulator defect detection system based on an improved SAM segmentation model, the insulator defect detection system comprising an image acquisition module, a model training module, and a target detection module;

[0021] The image acquisition module is used to acquire sample images and images to be detected of the insulator;

[0022] The model training module is used to annotate sample images, and train the YOLOv5 model and the improved SAM segmentation model using the annotated insulator sample images;

[0023] The target detection module is used to input the image to be detected into the trained YOLOv5 model for target detection, identify each object in the image and generate a corresponding bounding box, then extract the bounding box information of each object, convert the format of the bounding box into the trained improved SAM segmentation model, perform zero-sample segmentation, output the segmentation result, and finally determine the defect position and type of the insulator based on the segmentation result.

[0024] The bounding box information output by the YOLOv5 model includes the center point coordinates of the bounding box, the length and width of the bounding box;

[0025] The target detection module converts the bounding box information output by the YOLOv5 model into a coordinate format acceptable to the improved SAM segmentation model based on the following formula:

[0026]

[0027]

[0028]

[0029]

[0030] In the above formula, x a ,y a x is the horizontal and vertical coordinates of the upper left corner of the rectangular box of the input improved SAM segmentation model; b ,y b x are the horizontal and vertical coordinates of the lower right corner of the rectangular box of the input improved SAM segmentation model; center ,y center L are the horizontal and vertical coordinates of the center point of the bounding box output by the YOLOv5 model; width , L height Respectively represent the length and width data of the bounding box output by the YOLOv5 model.

[0031] The improved SAM segmentation model includes an image encoder, a hint encoder and a mask decoder. The mask decoder replaces the original MLP module with an improved MLP module. The improved MLP module includes a height branch, a width branch and a channel information branch. The height branch is used to first perform a height channel rearrangement operation on the input image X, and then use a linear layer with a weight of WH to encode the height information to obtain an output XH of the height branch; the width branch is used to first perform a width channel rearrangement operation on the input image X, and then use a linear layer with a weight of WW to encode the width information to obtain an output XW of the width branch; the channel information branch is used to encode the channel information of the input image X using a linear layer with a weight of WC to obtain an output XC of the channel branch; the output XH of the height branch, the output XW of the width branch, and the output X of the channel branch are first weighted summed and then information fused through a linear layer with a weight of W' to obtain an output Y of the improved MLP module; the output Y of the improved MLP module is formulated as follows:

[0032] Y=W′(α1*XH+α2*XW+α3*XC);

[0033] In the above formula, α1, α2, and α3 are weight coefficients used to control the relative importance of height branch, width branch, and channel branch, respectively.

[0034] The training improved SAM segmentation model refers to training the mask decoder and freezing other modules besides it.

[0035] In a third aspect, the present invention provides an insulator defect detection device based on an improved SAM segmentation model, the insulator defect detection device comprising a memory and a processor;

[0036] The memory is used to store computer program code and transmit the computer program code to the processor;

[0037] The processor is configured to execute the aforementioned method according to the instructions in the computer program code.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the aforementioned method when executed by a processor.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. The present invention discloses an insulator defect detection method based on an improved SAM segmentation model. The method first collects a sample image and an image to be detected of an insulator; then annotates the sample image, and uses the annotated sample image to train a YOLOv5 model and an improved SAM segmentation model; then the image to be detected is input into the trained YOLOv5 model for target detection, each object in the image is identified and a corresponding bounding box is generated, and then the bounding box information of each object is extracted, and the bounding box information is converted into a format and input into the trained improved SAM segmentation model to perform zero-sample segmentation, output the segmentation result, and finally determine the defect position and type of the insulator based on the segmentation result; the design uses YOLOv5 to draw bounding box prompts, which are used as preliminary annotations of objects in the image, and these bounding box prompts are used as inputs of the improved SAM segmentation model to further accurately segment objects and improve the accuracy of defect detection; in addition, since the improved SAM segmentation model can perform image segmentation tasks on unseen categories, it can ensure that it adapts to new defect types. Therefore, the present invention not only has high defect detection accuracy, but also can adapt to new defect types.

[0041] 2. In the insulator defect detection method based on the improved SAM segmentation model of the present invention, the original MLP module is replaced by the improved MLP module, the improved MLP module first performs weighted summation on the outputs of the three branches, and then performs information fusion on the outputs of the three branches through a linear layer with a weight of W', and uses weight coefficients to assign weights to different branches during weighted summation, and these weight coefficients can be obtained through training and learning, thereby further improving the accuracy of defect detection. Therefore, the present invention can further improve the accuracy of defect detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The present invention is a flowchart of the method.

[0043] Figure 2 The structure diagram of the system of the present invention is shown in FIG.

[0044] Figure 3 It is a structural block diagram of the device described in the present invention. DETAILED DESCRIPTION

[0045] The present invention is further described in detail below in conjunction with specific implementations and drawings.

[0046] Embodiment 1:

[0047] See also Figure 1 , an insulator defect detection method based on an improved SAM segmentation model, is performed in the following steps:

[0048] S1, collecting sample images of insulators and images to be tested;

[0049] S2, preprocessing and image annotation of the sample image; the preprocessing includes adjusting the image size, normalizing, and removing noise;

[0050] S3, using the sample image obtained in step S2 to train the YOLOv5 model and the improved SAM segmentation model respectively to adjust the model parameters; the improved SAM segmentation model includes an image encoder, a prompt encoder and a mask decoder, and the mask decoder uses an improved MLP module to replace the original MLP module; when training the improved SAM segmentation model, first use the ordinary SAM segmentation model to participate in the training, and after the training, replace the original MLP module with the improved MLP module, and then retrain the mask decoder, and freeze the other modules; the improved MLP module includes a height branch, a width branch and a channel information branch, and the height branch is used to first perform a height channel rearrangement operation on the input image X, and then use A linear layer with a weight of WH encodes the height information to obtain the output XH of the height branch; the width branch is used to first perform a width channel rearrangement operation on the input image X, and then use a linear layer with a weight of WW to encode the width information to obtain the output XW of the width branch, and the channel information branch is used to encode the channel information of the input image X using a linear layer with a weight of WC to obtain the output XC of the channel branch; the output XH of the height branch, the output XW of the width branch, and the output X of the channel branch are first weighted summed and then information fused through a linear layer with a weight of W' to obtain the output Y of the improved MLP module; the output Y of the improved MLP module is formulated as follows:

[0051] Y=W′(α1*XH+α2*XW+α3*XC);

[0052] In the above formula, α1, α2, and α3 are weight coefficients used to control the relative importance of height branch, width branch, and channel branch respectively;

[0053] S4, input the image to be detected into the trained YOLOv5 model for target detection, identify each object in the image and generate a corresponding bounding box, then extract the bounding box information of each object, convert the format and input it into the trained improved SAM segmentation model to perform zero-sample segmentation, and output the segmentation result; the bounding box information output by the YOLOv5 model includes the center point coordinates of the bounding box, the length and width of the bounding box, and the bounding box information output by the YOLOv5 model is converted into a coordinate format acceptable to the improved SAM segmentation model based on the following formula:

[0054]

[0055]

[0056]

[0057]

[0058] In the above formula, x a ,y a x is the horizontal and vertical coordinates of the upper left corner of the rectangular box of the input improved SAM segmentation model; b ,y b x are the horizontal and vertical coordinates of the lower right corner of the rectangular box of the input improved SAM segmentation model; center ,y center L are the horizontal and vertical coordinates of the center point of the bounding box output by the YOLOv5 model; width , L height Respectively represent the length and width data of the bounding box output by the YOLOv5 model;

[0059] S5. Based on the segmentation results, determine the defect location and type of the insulator.

[0060] In order to verify the segmentation effect of the insulator defect detection method of the present invention, the accuracy of the ordinary SAM and the YOLOv5+ improved SAM trained in Example 1 in detecting different defect types is tested. The test results are shown in Tables 1 and 2:

[0061] Table 1 Test results when the defect type is crack

[0062] Model DSC value Accuracy Recall F1 score Normal SAM 0.78 0.75 0.80 0.77 YOLOv5+Improved SAM 0.85 0.80 0.88 0.84

[0063] Table 2 Test results when the defect type is fracture

[0064] Model DSC value Accuracy Recall F1 score Normal SAM 0.83 0.79 0.85 0.82 YOLOv5+ Improved SAM 0.90 0.88 0.93 0.90

[0065] As can be seen from Tables 1 and 2, for both defect types, the DSC value of YOLOv5+Improved SAM is higher, indicating that its segmentation results have a higher overlap with the real defect area; it also performs well in precision and recall, which means that it can not only accurately identify more real defects, but also reduce misjudgments; it is also ahead of the pure SAM in F1 score, indicating a more balanced and consistent performance; in summary, YOLOv5+Improved SAM is better than the pure SAM in all evaluation indicators, which shows that YOLOv5+Improved SAM is more accurate in positioning and segmentation details.

[0066] Embodiment 2:

[0067] See also Figure 2 , an insulator defect detection system based on an improved SAM segmentation model, comprising an image acquisition module, a model training module, and a target detection module; the image acquisition module is used to acquire sample images of insulators and images to be detected; the model training module is used to annotate sample images, and use the annotated insulator sample images to train a YOLOv5 model and an improved SAM segmentation model respectively; wherein the improved SAM segmentation model comprises an image encoder, a hint encoder, and a mask decoder, the mask decoder uses an improved MLP module to replace the original MLP module, the improved MLP module comprises a height branch, a width branch, and a channel information branch, the height branch is used to first perform height channel re-encoding on the input image X The height information is encoded by a linear layer with a weight of WH, and the output XH of the height branch is obtained; the width branch is used to first perform a width channel rearrangement operation on the input image X, and then a linear layer with a weight of WW is used to encode the width information to obtain the output XW of the width branch, and the channel information branch is used to encode the channel information of the input image X by a linear layer with a weight of WC to obtain the output XC of the channel branch; the output XH of the height branch, the output XW of the width branch, and the output X of the channel branch are first weighted summed and then information fused through a linear layer with a weight of W' to obtain the output Y of the improved MLP module; the output Y of the improved MLP module is formulated as follows:

[0068] Y=W′(α1*XH+α2*WW+α3*XC);

[0069] In the above formula, α1, α2, and α3 are weight coefficients used to control the relative importance of height branch, width branch, and channel branch respectively;

[0070] The target detection module is used to input the image to be detected into the trained YOLOv5 model for target detection, identify each object in the image and generate a corresponding bounding box, then extract the bounding box information of each object, convert the format of the bounding box into the trained improved SAM segmentation model to perform zero-sample segmentation, output the segmentation result, and finally determine the defect position and type of the insulator based on the segmentation result; wherein the format conversion is performed according to the following formula:

[0071]

[0072]

[0073]

[0074]

[0075] In the above formula, x a ,y a x is the horizontal and vertical coordinates of the upper left corner of the rectangular box of the input improved SAM segmentation model; b ,y b x are the horizontal and vertical coordinates of the lower right corner of the rectangular box of the input improved SAM segmentation model; center ,y center L are the horizontal and vertical coordinates of the center point of the bounding box output by the YOLOv5 model; width , L height Respectively represent the length and width data of the bounding box output by the YOLOv5 model.

[0076] Embodiment 3:

[0077] See also Figure 3 , an insulator defect detection device based on an improved SAM segmentation model, comprising a memory and a processor; the memory is used to store computer program code and transmit the computer program code to the processor; the processor is used to execute the method described in Example 1 according to the instructions in the computer program code.

[0078] Embodiment 4:

[0079] A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described in Embodiment 1 when executed by a processor.

[0080] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0081] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0082] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0083] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0084] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0085] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An insulator defect detection method based on an improved SAM segmentation model, characterized in that: The insulator defect detection method comprises the following steps: S1, collecting sample images of insulators and images to be tested; S2. Annotate the sample images, and use the annotated sample images to train the YOLOv5 model and the improved SAM segmentation model; S3. Input the image to be detected into the trained YOLOv5 model for target detection, identify each object in the image and generate a corresponding bounding box, then extract the bounding box information of each object, convert its format and input it into the trained improved SAM segmentation model to perform zero-sample segmentation, output the segmentation result, and finally determine the defect location and type of the insulator based on the segmentation result.

2. The insulator defect detection method based on the improved SAM segmentation model according to claim 1, characterized in that: The bounding box information output by the YOLOv5 model includes the center point coordinates of the bounding box, the length and width of the bounding box. The bounding box information output by the YOLOv5 model is converted into a coordinate format acceptable to the improved SAM segmentation model based on the following formula: In the above formula, x a ,y a x is the horizontal and vertical coordinates of the upper left corner of the rectangular box of the input improved SAM segmentation model; b ,y b x are the horizontal and vertical coordinates of the lower right corner of the rectangular box of the input improved SAM segmentation model; center ,y center L are the horizontal and vertical coordinates of the center point of the bounding box output by the YOLOv5 model; width , L height Respectively represent the length and width data of the bounding box output by the YOLOv5 model.

3. The insulator defect detection method based on the improved SAM segmentation model according to claim 1, characterized in that: The improved SAM segmentation model includes an image encoder, a hint encoder and a mask decoder. The mask decoder replaces the original MLP module with an improved MLP module. The improved MLP module includes a height branch, a width branch and a channel information branch. The height branch is used to first perform a height channel rearrangement operation on the input image X, and then use a linear layer with a weight of WH to encode the height information to obtain an output XH of the height branch; the width branch is used to first perform a width channel rearrangement operation on the input image X, and then use a linear layer with a weight of WW to encode the width information to obtain an output XW of the width branch; the channel information branch is used to encode the channel information of the input image X using a linear layer with a weight of WC to obtain an output XC of the channel branch; the output XH of the height branch, the output XW of the width branch, and the output X of the channel branch are first weighted summed and then information fused through a linear layer with a weight of W' to obtain an output Y of the improved MLP module; the output Y of the improved MLP module is formulated as follows: Y=W′(α1*XH+α2*XW+α3*XC); In the above formula, α1, α2, and α3 are weight coefficients used to control the relative importance of height branch, width branch, and channel branch, respectively.

4. The insulator defect detection method based on the improved SAM segmentation model according to claim 3 is characterized in that: The training improved SAM segmentation model refers to training the mask decoder and freezing other modules besides it.

5. The insulator defect detection system based on the improved SAM segmentation model is characterized by: The insulator defect detection system includes an image acquisition module, a model training module, and a target detection module; The image acquisition module is used to acquire sample images and images to be detected of the insulator; The model training module is used to annotate sample images, and train the YOLOv5 model and the improved SAM segmentation model using the annotated insulator sample images; The target detection module is used to input the image to be detected into the trained YOLOv5 model for target detection, identify each object in the image and generate a corresponding bounding box, then extract the bounding box information of each object, convert the format of the bounding box into the trained improved SAM segmentation model, perform zero-sample segmentation, output the segmentation result, and finally determine the defect position and type of the insulator based on the segmentation result.

6. The insulator defect detection system based on the improved SAM segmentation model according to claim 5, characterized in that: The bounding box information output by the YOLOv5 model includes the center point coordinates of the bounding box, the length and width of the bounding box; The object detection module converts the bounding box information output by the YOLOv5 model into a coordinate format acceptable to the improved SAM segmentation model based on the following formula: In the above formula, x a ,y a x is the horizontal and vertical coordinates of the upper left corner of the rectangular box of the input improved SAM segmentation model; b ,y b x are the horizontal and vertical coordinates of the lower right corner of the rectangular box of the input improved SAM segmentation model; center ,y center L are the horizontal and vertical coordinates of the center point of the bounding box output by the YOLOv5 model; width , L height Respectively represent the length and width data of the bounding box output by the YOLOv5 model.

7. The insulator defect detection system based on the improved SAM segmentation model according to claim 5, characterized in that: The improved SAM segmentation model includes an image encoder, a hint encoder and a mask decoder. The mask decoder replaces the original MLP module with an improved MLP module. The improved MLP module includes a height branch, a width branch and a channel information branch. The height branch is used to first perform a height channel rearrangement operation on the input image X, and then use a linear layer with a weight of WH to encode the height information to obtain an output XH of the height branch; the width branch is used to first perform a width channel rearrangement operation on the input image X, and then use a linear layer with a weight of WW to encode the width information to obtain an output XW of the width branch; the channel information branch is used to encode the channel information of the input image X using a linear layer with a weight of WC to obtain an output XC of the channel branch; the output XH of the height branch, the output XW of the width branch, and the output X of the channel branch are first weighted summed and then information fused through a linear layer with a weight of W' to obtain an output Y of the improved MLP module; the output Y of the improved MLP module is formulated as follows: Y=W′(α1*XH+α2*XW+α3*XC); In the above formula, α1, α2, and α3 are weight coefficients used to control the relative importance of height branch, width branch, and channel branch, respectively.

8. The insulator defect detection system based on the improved SAM segmentation model according to claim 7, characterized in that: The training improved SAM segmentation model refers to training the mask decoder and freezing other modules besides it.

9. Insulator defect detection equipment based on improved SAM segmentation model, characterized in that: The insulator defect detection device includes a memory and a processor; The memory is used to store computer program code and transmit the computer program code to the processor; The processor is configured to execute the method according to claims 1-4 according to the instructions in the computer program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to claims 1-4 is implemented.

Images