A substation defect semi-automatic labeling method based on SAM

Abstract

The application discloses a kind of based on SAM substation equipment defect semi-automatic labeling method, applied to semi-automatic labeling technical field.The present application includes: from substation facility and work site collection defect image data, and data cleaning is carried out to data;Semi-automatic labeling model is established based on SAM;Defect image data is manually labeled, and the model is preliminarily trained using the defect image data of artificial labeling;Unlabeled defect image data is sent into the model, and the labeling result of semi-automatic labeling model is obtained;The labeling result of semi-automatic labeling model is manually corrected;According to the fine tuning of semi-automatic labeling model according to artificial correction result;Semi-automatic labeling model carries out automatic labeling to remaining defect image data.The present application provides a semi-automatic labeling technology for substation defect data, solves the problem of mass data labeling difficulty, and improves substation inspection efficiency.

Description

Technical Field

[0001] This invention relates to the field of semi-automatic labeling technology, and more specifically to a semi-automatic labeling method for defects in power equipment based on SAM. Background Technology

[0002] With the widespread application of video image-based power line inspection technology, the amount of video image data related to substation equipment is growing exponentially, necessitating the annotation of related defect data. However, traditional manual substation defect data annotation is inefficient and cannot meet the demand. Existing semi-automatic annotation tools are suitable for general scenarios, but their annotation performance is poor in complex power scenarios; moreover, the initial learning cycle is long, requiring a large amount of manual annotation to achieve relatively stable results. How to design a truly usable semi-automatic annotation technology is a challenge that those skilled in the art need to solve.

[0003] Therefore, proposing a semi-automatic defect labeling method for power equipment based on SAM to solve the difficulties of existing technologies is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the present invention provides a semi-automatic annotation method for substation equipment defects based on SAM, which provides a semi-automatic annotation technology for substation defect data, solves the problem of difficulty in annotating massive amounts of data, and improves the efficiency of substation inspection.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A semi-automatic defect labeling method for substation equipment based on SAM includes the following steps:

[0007] S1. Collect defect image data from substation facilities and work sites, and perform data cleaning on the defect image data;

[0008] S2. Establish a semi-automatic annotation model based on SAM;

[0009] S3. Manually annotate the defective image data, and use the manually annotated defective image data to conduct preliminary training on the semi-automatic annotation model;

[0010] S4. Input the defect image data that has not been manually annotated into the semi-automatic annotation model and obtain the annotation results of the semi-automatic annotation model;

[0011] S5. Manually correct the annotation results of the semi-automatic annotation model;

[0012] S6. Fine-tune the semi-automatic annotation model based on the results of manual correction;

[0013] S7. Use the semi-automatic annotation model obtained in S6 to automatically annotate the remaining defect image data.

[0014] Optionally, in step S1, defect image data collected from substation facilities and work sites may include, but is not limited to, metal corrosion, dial damage, and breather oil seal damage; during the data cleaning process, damaged or incorrectly collected defect image data may be removed.

[0015] Optionally, the SAM model in step S2 includes multiple ViT blocks, each block containing one MHSA layer and one FFN layer. The calculation formula for the MHSA layer is as follows:

[0016]

[0017] Where Q represents the query matrix, K is the key matrix, V is the value matrix, and d k It is the dimension of keys and queries, K T It is the dot product of the bond matrix K.

[0018] Optionally, an Adapter module is added after the MHSA layer of each ViT block. The Adapter module includes a downsampling network, a ReLU layer, and an upsampling network.

[0019] Optionally, in step S3, YOLO is used as the image annotation format; the YOLO data annotation box contains 5 fields, namely the annotation box category name, the x-coordinate of the normalized annotation box center point, the y-coordinate of the normalized annotation box center point, the normalized annotation box width, and the normalized annotation box height; a decoder is added to the model's output layer, which can convert the model's output into YOLO format annotation boxes.

[0020] Optionally, the unlabeled data in step S4 does not overlap with the data in step S3; after the model reads the unlabeled data, it outputs the labeling results in YOLO format; a confidence scoring mechanism is added to the model, which provides a confidence score for each label box.

[0021] Optionally, in step S6, the fine-tuning process is performed on the Adapter module, and the parameters of the rest of the SAM structure are frozen; to ensure the stability of the fine-tuning process, a learning rate decay and early stopping strategy are introduced.

[0022] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a semi-automatic defect labeling method for substation equipment based on SAM, which has the following beneficial effects:

[0023] (1) This invention provides a semi-automated annotation technology for substation defect data, which solves the problem of difficulty in annotating massive amounts of data and improves the efficiency of substation inspection.

[0024] (2) The semi-automatic annotation technology of the present invention has good generalization performance. It can achieve automatic annotation with only a small amount of manually annotated data. It can be used for semi-automatic annotation of defective data with few samples and difficult to obtain, and has practical application value.

[0025] (3) The semi-automatic annotation technology of the present invention has fewer steps, is simple to use, and has less hardware requirements for training and deployment.

[0026] (4) Compared with similar semi-automatic annotation technologies, the semi-automatic annotation technology provided by this invention has high accuracy, good robustness, and can be applied to substation defect images of various resolutions and environments, with strong generalization performance. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0028] Figure 1 A flowchart of a method for semi-automatic defect labeling of substation equipment based on SAM provided by the present invention;

[0029] Figure 2 This is a schematic diagram of the structure of a semi-automatic annotation model for defects in power equipment based on SAM according to the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] This invention provides a semi-automatic defect annotation method for substation equipment based on SAM (Self-Annotated Model). The SAM-based model is first trained using a portion of manually annotated data. Subsequently, the model generates annotations for a small amount of additional data, which are then manually corrected, and the model is fine-tuned based on the correction results. Finally, the model automatically annotates the remaining massive amount of data. The specific technical solution is as follows:

[0032] Reference Figure 1As shown, this invention discloses a semi-automatic defect labeling method for substation equipment based on SAM, comprising the following steps:

[0033] S1. Collect defect image data from substation facilities and work sites, and perform data cleaning on the defect image data;

[0034] S2. Establish a semi-automatic annotation model based on SAM;

[0035] S3. Manually annotate the defective image data, and use the manually annotated defective image data to conduct preliminary training on the semi-automatic annotation model;

[0036] S4. Input the defect image data that has not been manually annotated into the semi-automatic annotation model and obtain the annotation results of the semi-automatic annotation model;

[0037] S5. Manually correct the annotation results of the semi-automatic annotation model;

[0038] S6. Fine-tune the semi-automatic annotation model based on the results of manual correction;

[0039] S7. Use the semi-automatic annotation model obtained in S6 to automatically annotate the remaining defect image data.

[0040] Furthermore, the defect image data in step S1 is collected from the substation site and is real image data, including various defect types such as metal corrosion, dial damage, and breather oil seal damage. During the data cleaning process, damaged or incorrectly collected defect image data will be excluded.

[0041] Furthermore, the SAM model in step S2 stands for Segment Anything Model, which is an image segmentation and detection model based on the Transformer architecture. The SAM model consists of multiple ViT blocks, each containing a Multi-headed Self-attention (MHSA) layer and a Feed-forward Network (FFN) layer, as shown below. Figure 2 As shown. The formula for calculating MHSA is:

[0042]

[0043] Where Q represents the query matrix, K is the key matrix, V is the value matrix, and d k It is the dimension of keys and queries, K T It is the dot product of the bond matrix K.

[0044] Specifically, Q represents the query matrix, used to match it with the key matrix K to determine how to weight the value matrix V. K is the key matrix, and the degree of matching between it and the query matrix Q determines the weights of the value matrix V. V is the value matrix, representing the actual information content. When calculating the attention weights, the dot product of the query matrix Q and the key matrix K is scaled by a scaling factor. Normalization, where d k The dimensions are the key and the query, ensuring the dot product size is appropriate. Finally, the softmax function is used to ensure the sum of all weights is 1. The resulting weights are multiplied by the value matrix V to output the attention result.

[0045] To accommodate the characteristics of substation defect data, this design incorporates an Adapter module after the MHSA of each ViT block. The Adapter module consists of a downsampling network, a ReLU layer, and an upsampling network, as shown below. Figure 2 As shown. This design allows the model to perform more refined processing of specific defect data while maintaining its original image perception capabilities.

[0046] Furthermore, in step S3, the data annotators use YOLO as the image annotation format; the YOLO data annotation box contains 5 fields, namely the annotation box category name, the x-coordinate of the normalized annotation box center point, the y-coordinate of the normalized annotation box center point, the normalized annotation box width, and the normalized annotation box height; in order for the SAM model to better process YOLO format data, a specific decoder is added to the model's output layer, which converts the model's output into YOLO format annotation boxes.

[0047] Furthermore, the unlabeled data in step S4 does not overlap with the data in step S3. After reading this unlabeled data, the model outputs the labeling results in YOLO format. To improve the accuracy of the labeling, a confidence scoring mechanism is added to the model. This mechanism provides a confidence score for each bounding box, thereby helping data labelers to make corrections more quickly.

[0048] Specifically, in step S5, data labelers label the unlabeled data using the YOLO format. The corrected data is then fed into the model. To further improve the model's robustness, a data augmentation strategy is introduced, including random pruning, rotation, and flipping, enabling the model to be trained with a wider range of data variations.

[0049] Furthermore, in step S6, the fine-tuning process is performed only on the Adapter module, and the parameters of the rest of the SAM structure are frozen. This strategy ensures that the model does not lose its original image perception capabilities during fine-tuning, but only optimizes specific defective data. In addition, to ensure the stability of the fine-tuning process, learning rate decay and early stopping strategies are introduced.

[0050] Specifically, in step S7, the model automatically labels the remaining data. In this step, the model performs a complete traversal of the entire dataset to ensure that all data is labeled. To improve the accuracy of the labeling, a post-processing step is introduced, including non-maximum suppression (NMS) and threshold filtering, thereby ensuring the accuracy and robustness of the labeling results.

[0051] In actual industrial production, the semi-automated defect labeling technology for substation equipment based on SAM (Self-Augmented Model) first needs to be integrated with the existing inspection system. During the inspection process, image data collected by drones or other equipment is transmitted to the central processing system in real time. This data first undergoes preliminary screening and cleaning, and then is fed into the SAM model for labeling. The labeling results are displayed on the operator's interface, allowing the operator to quickly correct the labeling results. The corrected data can not only be used to update the model, but also provide important reference for subsequent maintenance and repair work.

[0052] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0053] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semi-automatic defect labeling method for substation equipment based on SAM, characterized in that, Includes the following steps: S1. Collect defect image data from substation facilities and work sites, and perform data cleaning on the defect image data; S2. Establish a semi-automatic annotation model based on SAM; S3. Manually annotate the defective image data, and use the manually annotated defective image data to conduct preliminary training on the semi-automatic annotation model; S4. Input the defect image data that has not been manually annotated into the semi-automatic annotation model and obtain the annotation results of the semi-automatic annotation model; S5. Manually correct the annotation results of the semi-automatic annotation model; S6. Fine-tune the semi-automatic annotation model based on the results of manual correction; S7. Use the semi-automatic annotation model obtained in S6 to automatically annotate the remaining defect image data; The SAM model in step S2 includes multiple ViT blocks, each block containing an MHSA layer and an FFN layer. The calculation formula for the MHSA layer is as follows: Where Q represents the query matrix, K is the key matrix, and V is the value matrix. It's the dimension of keys and queries. It is the dot product of the key matrix K; An Adapter module was added after the MHSA layer of each ViT block. The Adapter module includes a downsampling network, a ReLU layer and an upsampling network. In step S6, the fine-tuning process is performed on the Adapter module, and the parameters of the rest of the SAM structure are frozen. To ensure the stability of the fine-tuning process, a learning rate decay and early stopping strategy are introduced.

2. The semi-automatic defect labeling method for substation equipment based on SAM according to claim 1, characterized in that, In step S1, defect image data is collected from substation facilities and work sites, including but not limited to metal corrosion, dial damage, and breather oil seal damage. During the data cleaning process, damaged or incorrectly collected defect image data is removed.

3. The semi-automatic defect labeling method for substation equipment based on SAM according to claim 1, characterized in that, In step S3, YOLO is used as the image annotation format; the YOLO data annotation box contains 5 fields, namely the annotation box category name, the x-coordinate of the normalized annotation box center point, the y-coordinate of the normalized annotation box center point, the normalized annotation box width, and the normalized annotation box height; a decoder is added to the output layer of the model to convert the model output into YOLO format annotation boxes.

4. The semi-automatic defect labeling method for substation equipment based on SAM according to claim 3, characterized in that, The unlabeled data in step S4 does not overlap with the data in step S3; after reading the unlabeled data, the model outputs the labeling results in YOLO format; a confidence scoring mechanism is added to the model, which provides a confidence score for each bounding box.

Images