Substation defect detection method, device and equipment for small sample scenarios and medium

Abstract

This invention discloses a method, apparatus, equipment, and medium for substation defect detection in small-sample scenarios, relating to the field of substation defect detection technology. The method includes: a regression network spatially expanding and grid-sampling candidate prediction boxes, embedding prototype features using category prototype vectors and an attention mechanism, and constructing a mask-stacking region propagation mechanism to output target box prediction results; a target classification network filtering candidate defect category sets based on candidate prediction boxes and category prototype vectors, and generating candidate defect category confidence scores through multi-source embedding fusion by constructing intra-class response embeddings, inter-class ranking embeddings, and background semantic embeddings; and introducing a text-image alignment network to assist in scoring the semantic consistency of candidate prediction boxes within a preset interval, obtaining calibrated candidate defect category confidence scores. This method can improve defect detection accuracy under limited sample conditions.

Description

Technical Field

[0001] This invention relates to the field of substation defect detection technology, and in particular to a method, apparatus, equipment and medium for substation defect detection in a small sample scenario. Background Technology

[0002] With the rapid adoption of intelligent inspection technology in substations, deep learning-based computer vision algorithms have been widely applied in power equipment defect detection tasks. Currently, mainstream target detection methods such as YOLO, Faster R-CNN, and DETR have been extensively studied for identifying and locating various defect targets in substation scenarios. These methods generally rely on large-scale, high-quality labeled data.

[0003] However, in actual substation inspection tasks, some defect events occur very infrequently, such as loose screws, discolored and corroded breathers, and peeling equipment coatings. Only a very small number (single-digit to a dozen or so) of labeled defect images can be collected, thus constituting a typical small-sample detection scenario. Under these conditions, mainstream target detection methods have the following typical limitations:

[0004] (1) The boundaries between different defect categories are difficult to define accurately, which makes the confidence distribution of the model unclear when outputting the category probability. This makes it easy for inter-class confusion to occur, which may lead to false detection or false negative.

[0005] (2) Because small sample targets often exhibit weak features in images, the model has difficulty obtaining sufficient discriminative support for bounding box regression, which leads to a significant decrease in localization accuracy.

[0006] (3) When the number of samples is limited, deep feature extraction networks are unable to learn representative category discrimination features. The distribution of small sample categories in the feature space is easily affected by the overfitting of the backbone network, ultimately resulting in insufficient feature expression ability.

[0007] Although related technologies have promoted the development of small-sample defect detection to some extent, there are still technical difficulties such as low detection accuracy, weak category differentiation ability, and poor model generalization when facing challenges such as the scarcity of image samples and similar features between categories in real power inspections. Summary of the Invention

[0008] This invention aims to at least partially address one of the technical problems in related technologies. Therefore, the objective of this invention is to provide a method, apparatus, device, and medium for substation defect detection in small sample scenarios, thereby improving defect detection accuracy under limited sample conditions.

[0009] To achieve the above objectives, a first aspect of the present invention proposes a substation defect detection method for small sample scenarios, comprising:

[0010] The substation inspection images to be detected are input into the backbone network and the Region Proposal Network (RPN) respectively, and the full-image features and candidate prediction boxes of the substation inspection images are extracted.

[0011] The full-image features and candidate prediction boxes of the substation inspection image are input into a trained regression network that integrates prototype attention and mask propagation for processing. The regression network is used to spatially expand and grid sample the candidate prediction boxes, embed prototype features by combining category prototype vectors and attention mechanisms, and construct a region propagation mechanism based on binary masks to output the target box prediction results.

[0012] The full-image features and candidate prediction boxes of the substation inspection image are input into the trained target classification network for processing. The target classification network is used to filter the candidate defect category set based on the similarity between the candidate prediction box and the category prototype vector, and generate the candidate defect category confidence of the candidate prediction box by constructing intra-class response embedding, inter-class ranking embedding and background semantic embedding, and using a multi-source embedding fusion method.

[0013] A text-image alignment network is introduced to assist in scoring the semantic consistency of candidate predicted boxes whose candidate defect category confidence scores are within a preset range, thereby obtaining the calibrated candidate defect category confidence scores.

[0014] The category prototype vector is constructed during model training and dynamically updated based on a momentum update mechanism.

[0015] In addition, the substation defect detection method for small sample scenarios in the above embodiments of the present invention may also have the following additional technical features:

[0016] According to one embodiment of the present invention, prototype feature embedding is performed by combining the category prototype vector with an attention mechanism, including:

[0017] Using the full-image features and the expanded candidate prediction boxes, candidate region features within each expanded candidate prediction box are extracted through region alignment operations.

[0018] The query vector and key vector in the attention mechanism are used to perform a linear transformation on the candidate region features and category prototype vector to obtain a low-dimensional embedding representation.

[0019] Calculate the attention response relationship between each position of the extended candidate prediction box and the low-dimensional embedding representation to obtain the foreground category response map and the background category response map;

[0020] The foreground region embedding features generated based on the foreground category response map are concatenated with the background embedding features generated based on the background category response map to obtain region embedding features, which are then used as input for the region propagation mechanism.

[0021] According to one embodiment of the present invention, a region propagation mechanism based on binary masks and mask stacking is constructed to output target bounding box prediction results, including:

[0022] An input mask is constructed based on the positional relationship between the sampling points in the grid sampling and the candidate prediction box; the input mask is used to characterize whether the sampling points in the grid sampling fall into the candidate prediction box;

[0023] The input mask is used as the initial input to the region propagation mechanism and fused with the region embedding features to generate a mask response map layer by layer through a multi-layer propagation network.

[0024] The final mask response map output by the multilayer propagation network is decoded into the normalized coordinates of the target box through coordinate integration.

[0025] According to one embodiment of the present invention, the step of introducing a graph-text alignment network to assist in scoring the semantic consistency of candidate predicted boxes whose candidate defect category confidence scores are within a preset range, thereby obtaining calibrated candidate defect category confidence scores, includes:

[0026] Extract the region visual feature vectors of candidate predicted boxes whose confidence scores for candidate defect categories fall within a preset range, and calculate the similarity between these vectors and the category text feature matrix generated based on the natural language description of each defect category to obtain a semantic similarity score.

[0027] The semantic similarity score is weighted and fused with the candidate defect category confidence score of the corresponding candidate prediction box to obtain the calibrated candidate defect category confidence score.

[0028] According to an embodiment of the present invention, the process of constructing category prototype vectors during model training includes:

[0029] Based on the number of sample images in the sample set, the sample type of the sample set is determined; the sample type is either a basic category or a small sample category.

[0030] Extract the average semantic features of the defect regions of the support set sample images in the sample set for each defect category, and select the appropriate aggregation method to perform feature aggregation on the average semantic features according to the sample type to which the support set sample images belong.

[0031] If the sample type is a small sample category, then the average semantic features are aggregated using the mean aggregation method.

[0032] If the sample type belongs to the basic category, then the average semantic features are aggregated using the K-Means clustering algorithm based on cosine distance.

[0033] According to one embodiment of the present invention, the method further includes:

[0034] After each training iteration, a momentum update method is used to update the prototype vector of the category that best matches the query set sample image in the defect category to which the query set sample image belongs.

[0035] According to one embodiment of the present invention, during model training, the loss value of the target classification network is calculated by fusing Focal Loss with a classification loss function based on positive and negative sample co-optimization.

[0036] To achieve the above objectives, a second aspect of the present invention provides a substation defect detection device for small sample scenarios, comprising:

[0037] The data extraction module is used to input the substation inspection images to be detected into the backbone network and the RPN network respectively, and extract the full-image features and candidate prediction boxes of the substation inspection images.

[0038] The regression network processing module inputs the full-image features and candidate prediction boxes of the substation inspection image into the trained regression network that integrates prototype attention and mask propagation for processing. The regression network is used to spatially expand and grid sample the candidate prediction boxes, embed prototype features by combining category prototype vectors and attention mechanisms, and construct a region propagation mechanism based on binary masks to stack masks and output the target box prediction results.

[0039] The target classification network processing module inputs the full-image features and candidate prediction boxes of the substation inspection image into the trained target classification network for processing. The target classification network is used to filter and obtain a set of candidate defect categories based on the similarity between the candidate prediction boxes and the category prototype vectors. For each candidate defect category, intra-class response embedding, inter-class ranking embedding, and background semantic embedding are constructed respectively, and the candidate defect category confidence of the candidate prediction box is generated by multi-source embedding fusion.

[0040] The image-text alignment network processing module introduces an image-text alignment network to assist in scoring the semantic consistency of candidate prediction boxes whose candidate defect category confidence scores are within a preset range, thereby obtaining the calibrated candidate defect category confidence scores.

[0041] The category prototype vector is constructed during model training and dynamically updated based on a momentum update mechanism.

[0042] To achieve the above objectives, a third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the substation defect detection method for the small sample scenario described above.

[0043] To achieve the above objectives, a fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the steps of the substation defect detection method for the small sample scenario described above.

[0044] The substation defect detection method, apparatus, equipment, and medium for small sample scenarios in this invention improves the accuracy of candidate box localization by integrating category prototype attention and mask propagation mechanisms into the regression network, making candidate region features closer to the discriminative features of the target category. During the classification stage, multi-source embeddings such as intra-class response, inter-class ranking, and background semantics are combined to effectively enhance class separability and suppress background interference. A text-image alignment network is introduced to calibrate semantic consistency for low-confidence candidate boxes, reducing missed and false detections in small sample scenarios. Furthermore, the category prototype vectors used in the regression and target classification networks are optimized through momentum updates during model training, improving the model's generalization ability and stability in small sample scenarios, thereby significantly improving the accuracy and reliability of substation defect detection even with scarce labeled samples. Attached Figure Description

[0045] Figure 1 This is a diagram illustrating the overall framework of a small-sample defect detection algorithm in one embodiment.

[0046] Figure 2 This is a flowchart illustrating a substation defect detection method for a small sample scenario in one embodiment.

[0047] Figure 3 This is a schematic diagram of data processing for a regression network in one embodiment;

[0048] Figure 4 This is a schematic diagram of data processing for a text-image alignment network in one embodiment;

[0049] Figure 5 This is an example diagram showing the update of the category prototype vector in one embodiment;

[0050] Figure 6 This is a schematic diagram of the substation defect detection device in a small sample scenario of one embodiment. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] The implementation details of the technical solutions in the embodiments of this application are described in detail below.

[0053] Figure 1 The overall framework diagram of the small sample defect detection algorithm is shown below. Based on this diagram and different embodiments, the small sample defect detection will be described in detail below.

[0054] In one embodiment, such as Figure 2 The diagram illustrates a flowchart of a substation defect detection method for a small sample scenario. This method may include the following steps:

[0055] Step S101: Input the substation inspection images to be detected into the backbone network and RPN network respectively, and extract the full-image features and candidate prediction boxes of the substation inspection images.

[0056] refer to Figure 1 During the model inference phase, the input image is the substation inspection image to be detected. This image can originate from on-site images captured by inspection robots, drones, or fixed monitoring equipment. The substation inspection image is input into the visual backbone network DINOv2 ViT (i.e., the self-supervised visual feature extraction model DINOv2 based on the Vision Transformer architecture) for forward propagation. Patch token features are extracted in the intermediate layers and, after necessary mapping or rearrangement, multi-channel full-image features are obtained.

[0057]

[0058] in, This indicates the input image, which here is the substation inspection image to be inspected; This represents the feature extraction process of the backbone network; This represents the full-image features of the output; Representing feature dimension, Indicates the height of the feature map. Indicates the width of the feature map. Full-map features. It preserves the image's positional information in the spatial dimension and carries multi-level semantic descriptions in the channel dimension, which can be directly used for subsequent candidate region alignment and prototype embedding operations.

[0059] Meanwhile, the substation inspection images are input into the RPN network to obtain candidate prediction boxes for the substation inspection images. All the candidate prediction boxes constitute a candidate prediction box set. .

[0060] Step S102: Input the full-image features and candidate prediction boxes of the substation inspection image into the trained regression network that fuses prototype attention and mask propagation for processing.

[0061] Extract full-image features and candidate prediction boxes As input to the regression network, it is used to perform subsequent processing such as candidate box expansion and grid sampling to achieve accurate localization and discrimination. The regression network is a detection model that combines category prototype vectors with an attention mechanism for feature enhancement and uses a mask propagation mechanism to progressively strengthen foreground region features. Figure 3 As shown, Figure 3 This diagram illustrates the data processing of a regression network. The following section combines... Figure 3 The data processing flow of regression networks is explained in detail.

[0062] Regression network predicts candidate boxes Spatial expansion and grid sampling operations are performed to preserve more contextual information and enhance the feature representation capability of candidate regions. Specifically, the candidate predicted bounding box set...

[0063] Each candidate prediction box From the top left corner and the coordinates of the bottom right corner It means that, among them, and These represent the x-coordinate and y-coordinate of the image coordinate plane, respectively. During spatial expansion, the candidate prediction boxes are first converted into representations of the center point and width / height, and a minimum expansion amount is introduced. :

[0064]

[0065] in, This is an expansion factor, with a value range of 0.05 to 0.2. and They represent the first The width and height of each candidate box. The expanded width and height are respectively... and Based on this, a corresponding set of expanded candidate prediction boxes is generated. .

[0066] For each expanded candidate prediction box Divide it in the horizontal and vertical directions For a uniform grid area ( (This represents the number of divisions in the horizontal and vertical directions), and grid sampling is performed to construct the corresponding set of coordinate sampling points. Based on this, a binary mask is constructed to indicate the sampling location within the candidate region.

[0067] Regression networks will expand candidate prediction boxes Combined with class prototype vectors, an attention mechanism is used to embed prototype features, thereby strengthening the semantic association between candidate regions and their corresponding categories in the feature space. After prototype feature embedding, a region propagation mechanism based on the aforementioned binary mask is constructed. This mechanism uses the mask to guide feature flow in the feature space, allowing foreground region features to be gradually enhanced during multi-layer propagation, while background interference is suppressed. The regression network outputs the target bounding box prediction result based on the features processed by the region propagation mechanism, mainly including defect location information.

[0068] It should be noted that the category prototype vector is a high-dimensional vector that can represent the overall semantic features of the defect category in the feature space. It can be understood as the feature "anchor" or "center" of the category. In the inference process of the regression network and the subsequent target classification network, it serves as a reference for category features, guiding the downstream detector to focus on the discriminative direction, thereby reducing feature distribution bias and enhancing generalization ability under small sample conditions. The category prototype vector is constructed during the training phase, where it is dynamically updated through a momentum update mechanism, allowing the category vector to be continuously optimized. During the inference phase, the category prototype vector remains fixed to ensure the stability of the localization process.

[0069] In one embodiment, reference Figure 3 As shown, the process of embedding prototype features in a regression network by combining category prototype vectors and attention mechanisms includes the following steps:

[0070] Step S201: Using the full-image features and the expanded candidate prediction boxes, extract the regional image features within each expanded candidate prediction box through a region alignment operation.

[0071] Utilizing full-image features With the expanded candidate prediction box set Region feature alignment (RoIAlign) is used to align features across the entire image. Image features within the expanded candidate prediction bounding box are extracted through mid-sample extraction. RoIAlign, in particular, can accurately sample candidate regions on the feature map and ensure spatial consistency of features.

[0072] The extracted candidate region features are denoted as ,in Indicates the number of candidate boxes. For feature dimension, This represents the spatial resolution (e.g., 14×14) within each candidate box after pooling. These region image features will serve as input for subsequent prototype feature embedding and mask propagation mechanisms.

[0073] Step S202: Using the query vector and key vector in the attention mechanism, a linear transformation is performed on the candidate region features and category prototype vector to obtain a low-dimensional embedding representation.

[0074] To introduce asymmetric matching capabilities, linear mappings are performed between candidate region features and category prototype vectors. During the linear mapping process, the category prototype vectors are used... As a reference for category features, Indicates category The set of prototype sample vectors. For each category, the mean of all its prototype sample vectors is taken and normalized to obtain a single normalized prototype sample vector for that category. The category prototype set is formed by sequentially stacking these normalized prototype sample vectors. ,in, For the number of categories, For feature dimensions.

[0075] Based on this, we can obtain the following through linear mapping:

[0076]

[0077]

[0078] in, This represents the set of query vectors obtained by the feature mapping of candidate regions, used to represent the feature representation of different spatial locations within each candidate region in the embedding space; This represents the set of key vectors mapped from the set of category prototypes, used to represent the feature index of each category prototype in the embedding space; These are the learnable linear mapping matrices, Embedding dimension for low-dimensional attention. Indicates will Flatten it into a one-dimensional position sequence.

[0079] Understandably, the category prototype sample vector can be further divided into foreground category prototype sample vector and background category prototype sample vector. If a linear mapping is performed between the candidate region features and the foreground category prototype sample vector, the result is a low-dimensional embedding representation of the foreground; if a linear mapping is performed between the candidate region features and the background category prototype sample vector, the result is a low-dimensional embedding representation of the background.

[0080] Step S203: Calculate the attention response relationship between each position of the extended candidate prediction box and the low-dimensional embedding representation to obtain the foreground category response map and the background category response map.

[0081] This explanation uses the foreground category as an example. For the foreground category, the attention similarity between each location within the candidate region and all foreground category prototypes is calculated:

[0082]

[0083] in, The candidate prediction box number. Candidate prediction box grid location numbering, Category number, Represents the vector dot product. Scaling factor It is the foreground category prototype sample vector and the linear mapping matrix Obtained.

[0084] Organize the attention similarity set of all positions with the foreground category prototype into a tensor form, denoted as . Each element Corresponding tensor The value at the corresponding position in the middle.

[0085] right Softmax normalization is performed along the category dimension to obtain the foreground category response map. ,in, In the formula This represents the attention similarity matrix before normalization. This indicates normalization along the category dimension.

[0086] The calculation method for the background category response map is the same as that for the foreground, except that its category prototype comes from the feature set of the background region.

[0087] Step S204: The foreground region embedding feature generated based on the foreground category response map is concatenated with the background embedding feature generated based on the background category response map to obtain the region embedding feature, which is used as the input of the region propagation mechanism.

[0088] Response maps for each foreground category Response map with background category Input to the foreground embedding transformation module Background Embedded Transformation Module ,get:

[0089]

[0090]

[0091] in, Foreground embedding features, Embed features for the background. This represents the number of candidate predicted boxes. The number of grid cells divided for each candidate box. To embed the feature dimension, the transformation module and It can include operations such as linear mapping and positional encoding.

[0092] Finally, the foreground embedding and background embedding are concatenated along the feature channel dimension to obtain the region embedding feature:

[0093]

[0094] Among them, region embedding features It serves as the initial input for the regional propagation mechanism, and is used for subsequent feature information interaction and updates.

[0095] In one embodiment, reference Figure 3 As shown, the processing flow of the target bounding box prediction result output by the region propagation mechanism based on binary mask stacking in the regression network includes the following steps:

[0096] Step S301: Construct an input mask based on the positional relationship between sampling points in the grid sampling and candidate prediction boxes.

[0097] Based on each expanded candidate prediction box Divided uniform coordinate sampling point set Combined with candidate prediction boxes The location coordinates are used to construct an input mask for the region propagation mechanism.

[0098] In practical applications, the coordinates of each sampling point in the coordinate sampling point set can be represented as:

[0099]

[0100]

[0101] in, Indicates the first The coordinates of the top left corner of each expanded candidate prediction box. and These represent the expanded width and height of the expanded candidate prediction box, respectively. , Number of grid divisions.

[0102] Input mask It can be defined as a binary matrix indicating whether a sampling point is located inside the corresponding candidate prediction box:

[0103]

[0104] In this way, the generated input mask corresponds to the set of coordinate sampling points, which is used to clearly indicate whether the sampling points belong to the valid area of ​​the candidate prediction box, thereby effectively distinguishing the foreground and background areas and providing initial spatial guidance for the region propagation mechanism.

[0105] Step S302: The input mask is used as the initial input of the region propagation mechanism and fused with the region embedding features to generate the mask response map layer by layer through a multi-layer propagation network.

[0106] A multi-stage Region Propagation Network (RPNET) was designed within the regression network. The input mask serves as the initial input to the region propagation mechanism, along with the region embedding features. The fusion process involves encoding the binary information of the input mask into a feature representation, and then concatenating it with the region embedding features along the channel dimension.

[0107] The fused feature input is fed into RPNET, whose network structure consists of several lightweight residual blocks (ResBlock) and convolutional layers, combined with the sigmoid activation function and a temperature factor. This enables iterative generation of layer-by-layer mask response maps.

[0108] The propagation process is formally represented as follows:

[0109] Initialize the 0th layer mask and the 0th layer mask With region embedding features Concatenate the data to obtain the propagation input for layer 1. Continue this process for the next layer... layer( The propagation input of ) can be represented as:

[0110]

[0111] in, This is the input mask corresponding to the candidate prediction box. This represents the number of candidate predicted boxes. The number of grid cells divided for each candidate box. Indicates the first Layer propagation input.

[0112] Each layer of the propagation network Receive the fused input and generate the next layer mask response diagram:

[0113]

[0114] in, Indicates the first A lightweight propagation module for the layer. Indicates the first The output of the lightweight propagation module, This is a sigmoid activation function. Through iterative propagation, foreground features are gradually enhanced, while background interference is gradually suppressed.

[0115] Step S303: The final mask response map output by the multilayer propagation network is decoded into the normalized coordinates of the target box through coordinate integration.

[0116] The final mask response map output by the multilayer propagation network is decoded to obtain the normalized coordinates of the target box. The decoding method involves using coordinate integration to perform a weighted summation of the mask response map based on the spatial distribution of response intensity in the mask response map, thereby calculating the normalized coordinate values ​​of the target box's center point and size.

[0117] Step S103: Input the full-image features and candidate prediction boxes of the substation inspection image into the trained target classification network for processing.

[0118] In object classification networks, the set of candidate predicted boxes is calculated. Each candidate prediction box The similarity between the candidate predicted bounding box and the category prototype vector is the similarity between the candidate predicted bounding box and the category prototype vector. The features are compared with the category prototype vectors. Specifically, this is done by combining the candidate predicted box set. For each candidate predicted bounding box, the RoIAlign operation is performed on the full-image features to extract candidate region features of a fixed size. ,in This represents the number of candidate predicted boxes. This represents the spatial resolution after the RoIAlign operation. For the... Average pooling is performed on the spatial dimensions of each candidate box to obtain the global features of the region. .

[0119] It should be noted that the global features used here are the patch token features natively output by the visual backbone network DINOv2 ViT. ,in For feature dimension, This represents the spatial resolution of the feature map. It is a two-dimensional matrix, and it is a sequence of tokens. The aforementioned global features... It is obtained by rearranging the patch token features, and its value is equivalent to... ,Right now and These are different expressions of the same feature.

[0120] Based on this, the global features of each candidate prediction box are... With the category prototype vector (here the category prototype vector is) Perform a dot product operation to obtain a preliminary category matching score, and then select the highest score from the results. Categories, select a set of candidate categories. As the first Candidate defect categories for each candidate prediction box.

[0121] At the same time, the full image features Perform dot product matching along the channel dimension with the category prototype vector to obtain the category response map at the full-image scale. , used to express the response intensity of each spatial location to each category of prototype, where Indicates the number of categories. Indicates the number of background prototypes.

[0122] After obtaining the candidate category set, in order to capture the difference in category distribution between the inside and outside of the region, a three-way semantic embedding is constructed for each candidate category based on the response information of the entire image scale, namely, intra-class distribution embedding. Inter-class sorting embedding and background semantic embedding .

[0123] For intra-class distributed embedding The response channels of candidate defect categories are extracted, and positional distance encoding is performed on the spatial region corresponding to the candidate prediction box (e.g., relative coordinates or distance-to-center encoding are added to each spatial location). This is then projected into a fixed-dimensional representation through a linear projection or convolutional layer to obtain an embedding that characterizes the category distribution within the region. .

[0124] For inter-class sorting embedding For each candidate defect category Extract the corresponding response map channel. This is combined with other category channels to form inter-class ranking features. Specifically, for other category channels besides the current category... The channels are sorted according to their category response intensity to obtain the sorting results. ,in, This indicates a sorting operation for the channel response strengths. Indicates the candidate category The ranking feature matrix is ​​then mapped to a uniform dimension using linear interpolation upsampling. Then, a feedforward network is used to generate semantically ordered embeddings. This is used to reflect the semantic priority and suppression relationship of candidate prediction boxes relative to other categories.

[0125] For background semantic embedding Global features of candidate prediction boxes Dot product matching is performed with the background category prototype vector to obtain the background semantic embedding. This background semantic embedding is essentially a background distribution map, which is used to characterize the background distribution information of the candidate region.

[0126] The three paths are embedded in the channel dimension and spliced ​​together to construct a fusion feature. The details are as follows:

[0127]

[0128] in, , , These represent the number of channels embedded in the three paths, and These represent the height and width, respectively.

[0129] Fusion features Input to binary classification network To obtain candidate defect categories Confidence score for the candidate predicted bounding box:

[0130]

[0131] in, This indicates that the candidate prediction box belongs to the category. The confidence score.

[0132] For the background category, the background ranking embedding feature is also constructed. and embedded with background semantics After concatenation, the data is input into a background binary classification network to obtain a background discrimination score. :

[0133]

[0134] in, As a background discrimination score, Background binary classification network.

[0135] Final summary The set of scores for the foreground candidate categories of each candidate box. Background score The confidence level of candidate defect categories is determined by splicing together:

[0136]

[0137] Step S104: Introduce a text-image alignment network to assist in scoring the semantic consistency of candidate prediction boxes whose candidate defect category confidence scores are within a preset range, and obtain the calibrated candidate defect category confidence scores.

[0138] In some substation defect detection tasks, especially for small sample categories, the detection model often has low confidence in the target category during the inference phase, and may even be misclassified as other similar-looking categories. To improve the semantic consistency judgment ability of candidate targets, this embodiment introduces a graph-text alignment network. Utilizing a cross-modal feature matching mechanism, the visual features of the candidate prediction box and the semantic features of the corresponding defect category text description are mapped to the same embedding space, and the semantic similarity score between the two is calculated. This provides effective semantic supplementation in the discrimination process of low-confidence samples, achieving cross-modal calibration of confidence.

[0139] Here, candidate predicted boxes whose confidence levels for candidate defect categories fall within a preset range are selected. The preset range is used to select candidate predicted boxes that require further semantic calibration. It is usually defined as candidate predicted boxes whose confidence levels fall within a range (e.g., 0.2 to 0.7). This indicates that the candidate predicted boxes within this range are neither extremely high nor extremely low in confidence, and have a large degree of uncertainty, making them suitable for introducing semantic similarity for auxiliary judgment.

[0140] Subsequently, the visual features of these candidate predicted boxes and the textual features of the defect categories are input into an image-text alignment network, and an additional semantic consistency score is obtained through similarity calculation. This score serves as supplementary information to the original classification confidence of the detection model. By fusing the two, a calibrated candidate defect category confidence is obtained, thereby improving the discrimination accuracy in cases of low confidence and small sample sizes.

[0141] It should be noted that, in order to improve calibration efficiency and stability, specific categories such as background and invalid categories can be excluded to prevent them from participating in the semantic calibration process.

[0142] In one embodiment, Figure 4 The diagram illustrates the data processing of a text-image alignment network. The data processing flow of a text-image alignment network may include the following steps:

[0143] Step S401: Extract the region visual feature vector of the candidate prediction box whose confidence level of the candidate defect category is within the preset range, and calculate the similarity with the category text feature matrix generated based on the natural language description of each defect category to obtain the semantic similarity score.

[0144] For candidate predicted bounding boxes whose confidence scores for candidate defect categories fall within a preset range, region feature extraction is performed. Specifically, for candidate predicted bounding boxes within the preset range, a feature extractor matching the visual branch of the image-text model is used to crop and encode the candidate regions. The candidate regions are then pooled into fixed-size image patches using the ROIAlign operation and input into the visual encoder `encode_image` to obtain the region visual feature vectors for the inference stage. .

[0145] Meanwhile, natural language descriptions of each defect category are encoded based on pre-trained image-text alignment models (such as CLIP) to obtain normalized category text features:

[0146]

[0147] in, These represent text feature vectors for different defect categories. This represents the total number of categories. Category descriptions can be constructed using cue words. For example, for a defect category identified as "hxq_gjtps", "the silicone barrel of the respirator is damaged" can be used as input text, and the corresponding feature vector can be extracted using the text encoder encode_text in the image-text alignment model.

[0148] Subsequently, the region visual feature vector After normalizing the text features of each category, a similarity score is calculated to obtain the semantic similarity score:

[0149]

[0150] in, Indicates category The text feature vector, Represents the visual feature vector of the region transpose, Indicates the first Text feature vectors of each category, This is the temperature scaling factor. Indicates the first The candidate prediction box and the first Semantic similarity scores between categories.

[0151] Step S402: The semantic similarity score and the confidence score of the candidate defect category of the corresponding candidate prediction box are weighted and fused to obtain the calibrated confidence score of the candidate defect category.

[0152] For candidate boxes that fall within the threshold range, the confidence scores of subsequent defect categories (i.e., the original confidence scores) output by the fusion detection model are used. and semantic similarity score The calibrated confidence scores for candidate defect categories are obtained as follows:

[0153]

[0154] in, This is the fusion coefficient, typically set between 0.5 and 0.9; This is the semantic similarity score corresponding to the predicted defect category.

[0155] It should be noted that, Figure 4 The interval judgment in the process is used to filter predicted candidate boxes that are within the threshold interval. If the confidence of the candidate defect category of the candidate box is within the threshold interval, the original confidence of the predicted candidate box and the semantic similarity score are weighted and fused to obtain the calibrated candidate defect category confidence. If the confidence of the candidate defect category of the candidate box is not within the threshold interval, the original confidence remains unchanged.

[0156] The cross-modal semantic calibration method described above can effectively improve the accuracy of class determination of low-confidence candidate boxes in small-sample defect detection, and enhance the robustness and generalization ability of the model in rare defect category recognition tasks.

[0157] It is understood that the above embodiments are based on the trained regression network and target classification network, combined with the image-text alignment network, to obtain the detection results of small sample defects. In practical applications, the training process of the regression network and target classification network can refer to the relevant content of the above embodiments. Figure 1 During the model training phase, the input image can be a support set and a query set based on the sample set. The support set is used for prototype feature construction, and the query set is used as input for model training.

[0158] During model training, the regression network employs a mechanism that integrates prototype attention and mask propagation. It combines dynamic category prototype vectors with candidate region features to generate an attention response map, and uses mask propagation to achieve fine-grained localization of candidate boxes, outputting the target box prediction results during model training. The target classification network generates semantic priorities for candidate categories based on inter-class ranking strategies. It also extracts three feature paths: intra-class distribution embeddings, inter-class ranking embeddings, and background semantic embeddings. These three embeddings are then fused to calculate the defect category confidence of the candidate boxes, outputting the candidate defect category confidence during model training. The loss values ​​of the regression network and the target classification network during model training are calculated separately, and the networks are optimized based on these loss values ​​to ultimately obtain the regression network and target classification network used in the model inference stage.

[0159] It should be noted that there are the following differences between the model training phase and the model inference phase:

[0160] (1) During the model training phase, it is necessary to construct the category prototype vector for each defect category and dynamically update the category prototype vector;

[0161] (2) There is a difference between the expanded candidate prediction boxes in the model training phase and the expanded candidate prediction boxes in the model inference phase. In the model training phase, expanded boxes are constructed using the minimum expansion amount. That is, the expanded box The dimensions are: And calculate the expanded box. With real frame The bounding box is used as the final expanded candidate prediction box set. However, during the model inference phase, since there are no true bounding boxes... An expanded box constructed based on the minimum expansion amount can be used. Directly used as the final expanded candidate prediction box set .

[0162] (3) Loss calculation for regression network and target classification network.

[0163] The differences (1) and (3) mentioned above will be explained in detail below through different embodiments to clearly illustrate the training process of regression network and target classification network.

[0164] In one embodiment, the process of constructing a category prototype vector includes the following steps:

[0165] Step S501: Determine the sample type of the sample set based on the number of sample images in the sample set.

[0166] During model training, a sample set needs to be constructed. Image data containing the operating status of various equipment is acquired through the substation inspection system. These sample images can be obtained through drone inspections, fixed camera captures, or handheld inspection devices. The acquired sample images cover several typical defect types, including but not limited to the following seven categories: damaged meter casing, damaged breather silicone tank, abnormal oil level seal, discolored breather silicone, blurred meter dial, suspended debris, and damaged insulators.

[0167] To facilitate subsequent training of the defect detection model, each acquired sample image undergoes manual or semi-automatic annotation. The annotation includes the precise location of the defect region in the image (such as a bounding box or mask) and the category label to which the defect belongs. For example, when an insulator of a power transmission equipment is damaged, the annotator will select the defect region in the image and classify the annotated defect into the corresponding defect category.

[0168] By using the above method, a sample dataset covering multiple defect types can be constructed, and each category of sample images can be guaranteed to have clear defect annotation information, thus providing a reliable data foundation for subsequent small sample training.

[0169] In practical applications, to enhance feature representation under small sample conditions, the constructed sample sets for each defect category are divided into support sets and query sets according to the category. The construction of the category prototype vectors here depends on the support set.

[0170] To simulate the problem of insufficient sample categories in real substation inspections and to support the training and evaluation of small sample detection algorithms, the types of all sample sets are divided based on the number of sample instances of each defect category in the sample set.

[0171] The number of sample instances for each defect category in the statistical sample set is determined. Based on a preset threshold, categories with at least 50 samples are classified as base classes, and categories with fewer than 50 samples are classified as novel classes. Base classes include common defects such as discolored silicone rubber on breather units, blurred meter dials, suspended debris, and damaged insulators; novel classes include less common defects such as damaged meter housings, damaged silicone rubber tanks on breather units, and abnormal oil seals indicating oil level.

[0172] Step S502: Extract the average semantic features of the defect regions of the support set sample images in the sample set of each defect category, and select the appropriate aggregation method to perform feature aggregation on the average semantic features according to the sample type to which the support set sample images belong.

[0173] In the support set, the average semantic features of the labeled defect regions in each support set sample image are extracted. Specifically, the support set sample images are forward-propagated through the DINOv2 (a self-supervised visual feature extraction model) visual backbone network with frozen parameters to extract the patch token features of the intermediate layers. These features represent the semantic representation of each region after the image is divided into several regions, as follows:

[0174]

[0175] In the above formula, where, This represents the input image, which here is a sample image from the support set. This represents the feature extraction process of the backbone network; This represents the full-image features of the output; Representing feature dimension, Indicates the height of the feature map. Indicates the width of the feature map.

[0176] Generate a corresponding binary mask based on the marked defect area (border or mask information). In this mask matrix, the patch positions belonging to the defect area are assigned a value of 1, and the remaining positions are assigned a value of 0. For example, when the defect area is located in the 3×3 patch area in the middle of the image, the corresponding 9 positions in the mask matrix are 1, and the remaining positions are 0.

[0177] Generated binary mask With features The average semantic features of the defect region in the support set sample images are calculated by multiplying element-wise at spatial locations and then performing a weighted average for non-zero locations. .

[0178] Different feature aggregation strategies are adopted for different sample types to aggregate the average semantic features.

[0179] For small sample classes, directly apply the class... All Average semantic features of each region The arithmetic mean is calculated to obtain the prototype vector for each class. Its mathematical expression is:

[0180]

[0181] in, For category prototype vector, For category The number of regions.

[0182] For the basic categories, the K-Means clustering algorithm based on cosine distance is used, with a preset number of clusters. , for category Clustering is performed on the average semantic features to obtain... There are 1 cluster center. Each cluster center From clusters The mean of all eigenvectors in the given data is obtained as follows:

[0183]

[0184] in, This represents the preset total number of clusters. The cluster index, with values ​​ranging from 1 to... , indicating the first Clusters, Indicates the first The number of feature vectors in each cluster Indicates the first Cluster center vectors, Indicates category The average semantic features.

[0185] Final Category The set of category prototype vectors is represented as:

[0186]

[0187] in, Indicates category The set of prototype vectors.

[0188] The set of prototype vectors for all categories is This is used as input for subsequent models.

[0189] In one embodiment, after each training iteration, a momentum update mechanism is used to optimize the class prototype vectors to improve their representation quality and adaptability. For example... Figure 5 As shown, Figure 5 An example graph showing the update of the category prototype vector is provided. Figure 5 Different colors in the image correspond to different defect category prototype vectors. Specifically, for the query set sample images... ,in, This represents the visual feature vectors extracted from candidate regions of the query set sample images using the same method during the training phase. Indicates the first in the query set The category label of each query set sample image is first determined by its corresponding defect category. The complete set of prototype vectors In the middle, calculate the relationship with the sample The cosine similarity is used to select the prototype vector with the highest similarity. As the most matching prototype, the corresponding expression is:

[0190]

[0191] in, The index value obtained through the above expression represents the index of the class prototype vector that is most similar to the sample features in that class. After selecting the best-matching class prototype vector, it is updated according to the momentum update formula:

[0192]

[0193] in, This is the momentum coefficient, which controls the balance between historical prototype information and current sample features. A larger one... The value can maintain the stability of the prototype vector, while a smaller value... The value helps the prototype vector to quickly adapt to the new sample feature distribution.

[0194] By using the momentum update method described above, the category prototype vector can continuously integrate new sample information during the iteration process, gradually optimize the representation effect, and thus improve the detection performance in small sample scenarios.

[0195] In one embodiment, the loss value of the target classification network is used to measure the accuracy of candidate boxes in class prediction and background discrimination. The loss value of the target classification network is calculated by fusing the focal loss and the classification loss function based on positive and negative sample co-optimization, in order to address the problems of sample imbalance and insufficient learning of hard samples in multi-class defect detection tasks.

[0196] To strengthen the training weights of difficult-to-separate samples, a positive sample loss term from Focal Loss is introduced to reduce the focus on easily separable samples and enhance the training weights of difficult-to-separate samples. For the th For each query set of sample images, the positive sample loss term is defined as:

[0197]

[0198] in, Indicates the first Positive sample loss for a query set of sample images. For the first The target category probability of a sample image in a query set. This is a modulating factor used to reduce focus on easily distinguishable samples.

[0199] In the negative sample part, to improve the model's ability to distinguish non-target categories, the following negative sample information enhancement term is designed:

[0200]

[0201] in, Indicates the first Negative sample loss for each query set of sample images. Indicates the first The query set sample images for the first query set sample image Predicted probability of class This is a confidence enhancement factor for negative samples, used to highlight the impact of high-confidence misclassified categories on the total loss.

[0202] To avoid the background category from overly dominating the loss calculation process in multi-class detection tasks, a background category weighting factor is introduced. Define the sample loss weight coefficient as follows:

[0203]

[0204] in, Indicates the first Weight coefficients of sample images in a query set. As the background category weight factor, Indicates the first The true class labels of the sample images in the query set. The weights of the loss terms for positive and negative samples are adjusted using this coefficient:

[0205]

[0206] Finally, the weighted positive and negative loss terms are fused to obtain the sample-level classification loss:

[0207]

[0208] in, and These are the weighting coefficients for the positive and negative sample loss terms. The classification losses of all samples are summed and averaged to obtain the overall loss value of the target classification network in this training batch, and the network parameters are optimized and updated accordingly.

[0209] Through the above design, the classification loss function can significantly improve the model's ability to identify minority class defects and difficult-to-classify samples in small sample scenarios with highly imbalanced sample numbers and a high proportion of difficult samples, thereby improving the overall accuracy and robustness of substation defect detection.

[0210] In practical applications, the loss value of the regression network is used to measure the deviation between the predicted bounding box and the ground truth bounding box in terms of position, size, etc. For the loss calculation of the regression network, a hierarchical loss supervision mechanism is used during the training phase, that is, two types of metrics, mask loss and coordinate regression loss, are calculated at each layer of the region propagation network. The mask loss includes binary cross-entropy loss and Dice similarity loss. Binary cross-entropy loss measures the pixel-level classification error between the predicted mask and the ground truth mask, while Dice similarity loss reflects the degree of region overlap between the predicted mask and the ground truth mask. Its calculation method is as follows:

[0211]

[0212] in, Indicates the regression network number Binary cross-entropy loss of the layer, Indicates the regression network number Dice similarity loss of layers, For the first Layer prediction mask, This is the corresponding output mask. Output Mask The generation rules are based on the coordinate sampling point set. Each sampling point With real frame (i.e., the marked defect area) is defined to determine whether the sampling point is located within the corresponding ground truth bounding box, specifically:

[0213]

[0214] Coordinate regression loss includes L1 distance loss Compared with generalized IoU loss The L1 distance loss measures the numerical difference between predicted and ground truth coordinates, while the generalized IoU loss considers both the overlap and relative positional relationship between the candidate and ground truth bounding boxes. Their calculation methods are as follows:

[0215]

[0216]

[0217] in, Indicates the regression network number L1 distance loss of layer Indicates the regression network number The generalized IoU loss of the layer, and They represent the first The predicted normalized coordinates and the true normalized coordinates of the layer. This represents the inverse mapping function from normalized coordinates to the actual candidate box coordinates.

[0218] The final total loss of the regression network is the weighted sum of the losses from each layer:

[0219]

[0220] in, This represents the total number of layers in the regression network. This is the loss weighting coefficient.

[0221] In each training round, the loss values ​​of the regression network and the target classification network are used as optimization objectives, respectively. Gradient backpropagation and parameter update strategies are employed to independently optimize the weight parameters of the regression network and the target classification network. After sufficient training, the regression network can accurately predict the spatial location and size of defective targets, while the target classification network can accurately distinguish different defect categories and effectively suppress background interference.

[0222] After training, the trained regression network and target classification network showed significant advantages in small sample detection scenarios. Even under conditions where the number of defect category samples was extremely limited, they could still maintain high detection accuracy and robustness. Thus, the optimized regression network and target classification network were applied to the substation inspection images to be detected, so as to realize the location and category recognition of defect targets in the images, thereby completing the defect detection task.

[0223] To verify the effectiveness of the substation defect detection method for small-sample scenarios proposed in this invention, the substation defect image test set published by the State Grid in 2022 was used for performance evaluation. This test set includes three typical small-sample defect types: meter casing damage, breather silicone barrel damage, and oil seal abnormalities. The evaluation metric adopted is mAP@50 (mean Average Precision at IoU=0.5), which is widely used in the field of target detection. mAP@50 represents the detection performance index obtained by averaging the average precision for each category when the intersection-union ratio (IoU) of the predicted and ground truth bounding boxes is 0.5.

[0224] Under two small sample settings, 5-shot and 10-shot, where 5-shot means that only 5 labeled training images are provided for each defect category and 10-shot means that only 10 labeled training images are provided for each defect category, the method of this invention is compared with existing mainstream small sample object detection models FSCE (Few-Shot Object Detection via Contrastive Proposal Encoding), VFA (Few-Shot Object Detection via Variational Feature Aggregation) and DEViT (Detect Everything with Few Examples).

[0225] Table 1 Performance Comparison of Different Detection Methods

[0226]

[0227] Experimental results show that, in the 10-shot task, the method of the present invention improves the mAP@50 index by 16.18%, 8.33%, and 4.92% respectively compared with FSCE, VFA, and DEViT; it also shows significant performance advantages in the 5-shot task, with a stable improvement in average detection accuracy, fully demonstrating the high accuracy and good generalization ability of the method of the present invention in the small sample defect detection task of substation.

[0228] In addition, to further verify the effectiveness of the regression network (module 1) containing prototype momentum update and visual fusion and the cross-modal semantic calibration mechanism (module 2) in this invention, a systematic ablation experiment was conducted, and the experimental results are shown in Table 2.

[0229] Table 2 Ablation Experiment Results

[0230]

[0231] Experimental results show that Module 1, by introducing similarity guidance between category prototypes and region features, effectively enhances the target perception capability in the region candidate generation stage, making the semantics of the generated candidate regions closer to the real target regions. This improves detection accuracy by approximately 2.2% and 3.47% in 5-shot and 10-shot scenarios, respectively. Module 2, by introducing a semantic alignment mechanism between text category priors and visual features, further improves detection performance, achieving improvements of approximately 1.16% and 1.1% in 5-shot and 10-shot scenarios, respectively. Regarding overall inference efficiency, the inference time for Module 1 is 0.132 seconds, and after adding Module 2, the inference time is 0.167 seconds, both meeting the State Grid standard requirement that the single-image inference time should not exceed 0.3 seconds.

[0232] In summary, the experimental results fully verify the superior performance and practical feasibility of the method of the present invention in the field of small sample defect detection in substations. It not only ensures a significant improvement in detection accuracy but also meets the efficiency requirements of real-time inference.

[0233] In the above embodiments, a regression network fusing prototype attention and mask propagation is introduced into the substation defect detection process. This effectively combines category prototype vectors and attention mechanisms during the spatial expansion and grid sampling of candidate prediction boxes, making the feature representation of the candidate region closer to the discriminative features of the target category. This results in high localization accuracy even with small sample sizes. Simultaneously, the region propagation mechanism using mask stacking enhances the feature correlation between the candidate box and its boundary regions, further improving the robustness and accuracy of target regression.

[0234] In the classification stage, intra-class response embeddings, inter-class ranking embeddings, and background semantic embeddings are constructed based on the category prototype vectors. The confidence of candidate defect categories is generated by multi-source embedding fusion. This not only enhances the separability between different categories under limited sample conditions, but also suppresses the risk of misclassification caused by background noise.

[0235] Furthermore, when processing candidate predicted boxes in low-confidence intervals, an image-text alignment network is introduced to calibrate the original confidence level using cross-modal semantic consistency scoring. This effectively compensates for the lack of semantic judgment ability when visual information is insufficient and reduces the false positives and false negatives in small sample categories.

[0236] By dynamically optimizing the class prototype vectors using a momentum update mechanism during training, the prototype representation can continuously integrate new sample feature information while maintaining stability, thereby improving the generalization ability and adaptability of the prototype in small-sample scenarios. In summary, this invention can significantly improve the localization accuracy and classification reliability of substation defect detection even when labeled samples are scarce.

[0237] In one embodiment, a substation defect detection device for a small sample scenario is provided, with reference to... Figure 6 As shown, the substation defect detection device 500 for this small sample scenario may include: a data extraction module 501, a regression network processing module 502, a target classification network processing module 503, an image-text alignment network processing module 504, and a model training module 505. Among them,

[0238] The data extraction module 501 is used to input the substation inspection images to be detected into the backbone network and the RPN network respectively, and extract the full-image features and candidate prediction boxes of the substation inspection images.

[0239] The regression network processing module 502 is used to input the full-image features and candidate prediction boxes of the substation inspection image into the trained regression network that integrates prototype attention and mask propagation for processing. The regression network is used to spatially expand and grid sample the candidate prediction boxes, embed prototype features by combining category prototype vectors and attention mechanism, and construct a region propagation mechanism based on binary mask stacking to output the target box prediction result.

[0240] The target classification network processing module 503 is used to input the full-image features and candidate prediction boxes of the substation inspection image into the trained target classification network for processing. The target classification network is used to filter the candidate defect category set based on the similarity between the candidate prediction box and the category prototype vector, and generate the candidate defect category confidence of the candidate prediction box by constructing intra-class response embedding, inter-class ranking embedding and background semantic embedding, and adopting a multi-source embedding fusion method.

[0241] The image-text alignment network processing module 504 is used to introduce the image-text alignment network to assist in scoring the semantic consistency of candidate prediction boxes whose candidate defect category confidence scores are within a preset range, and to obtain the calibrated candidate defect category confidence scores.

[0242] Among them, the category prototype vector is constructed during model training and dynamically updated based on the momentum update mechanism.

[0243] In one embodiment, the regression network processing module 502 is specifically used to extract candidate region features within each expanded candidate prediction box by using full-image features and expanded expanded candidate prediction boxes through region alignment operations.

[0244] By using the query vector and key vector in the attention mechanism, a linear transformation is performed on the candidate region features and the category prototype vector to obtain a low-dimensional embedding representation;

[0245] Calculate the attention response relationship between each position of the extended candidate prediction box and the low-dimensional embedding representation to obtain the foreground category response map and the background category response map;

[0246] The foreground region embedding features generated based on the foreground category response map are concatenated with the background embedding features generated based on the background category response map to obtain region embedding features, which are then used as input for the region propagation mechanism.

[0247] In one embodiment, the regression network processing module 502 is specifically used to construct an input mask based on the positional relationship between the sampling points in the grid sampling and the candidate prediction box; the input mask is used to characterize whether the sampling points in the grid sampling fall into the candidate prediction box;

[0248] The input mask is used as the initial input to the region propagation mechanism and fused with the region embedding features. The mask response map is generated layer by layer through a multi-layer propagation network.

[0249] The final mask response map output by the multilayer propagation network is decoded into the normalized coordinates of the target box through coordinate integration.

[0250] In one embodiment, the image-text alignment network processing module 504 is specifically used to extract the region visual feature vector of the candidate prediction box whose candidate defect category confidence is within a preset range, and to perform similarity calculation with the category text feature matrix generated based on the natural language description of each defect category to obtain a semantic similarity score.

[0251] The semantic similarity score is weighted and fused with the confidence score of the candidate defect category of the corresponding candidate prediction box to obtain the calibrated confidence score of the candidate defect category.

[0252] In one embodiment, the substation defect detection device 500 for small sample scenarios further includes a model training module 505 for determining the sample type of the sample set based on the number of sample images in the sample set; the sample type is either the basic category or the small sample category.

[0253] Extract the average semantic features of the defect regions of the support set sample images in the sample set for each defect category, and select the appropriate aggregation method to perform feature aggregation on the average semantic features according to the sample type to which the support set sample images belong.

[0254] If the sample type is a small sample category, then the mean aggregation method is used to aggregate the average semantic features;

[0255] If the sample type belongs to the basic category, then the K-Means clustering algorithm based on cosine distance is used to aggregate the average semantic features.

[0256] In one embodiment, the model training module 505 is specifically used to update the prototype vector of the category that best matches the query set sample image in the defect category to which the query set sample image belongs, using a momentum update method after each training iteration.

[0257] In one embodiment, the model training module 505 is specifically used to calculate the loss value of the target classification network during model training by fusing Focal Loss with a classification loss function based on positive and negative sample co-optimization.

[0258] Specific limitations regarding the substation defect detection device 500 for small sample scenarios can be found in the limitations of the substation defect detection method for small sample scenarios described above, and will not be repeated here. Each module in the aforementioned substation defect detection device 500 for small sample scenarios can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0259] In one embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement a substation defect detection method for a small sample scenario.

[0260] In one embodiment, a computer storage medium is provided on which a computer program is stored, which, when executed by a processor, implements a substation defect detection method for a small sample scenario.

[0261] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0262] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0263] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0264] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0265] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for detecting defects in substations in a small sample scenario, characterized in that, include: The substation inspection images to be detected are input into the backbone network and the RPN network respectively, and the full-image features and candidate prediction boxes of the substation inspection images are extracted. The full-image features and candidate prediction boxes of the substation inspection image are input into a trained regression network that integrates prototype attention and mask propagation for processing. The regression network is used to spatially expand and grid sample the candidate prediction boxes, embed prototype features by combining category prototype vectors and attention mechanisms, and construct a region propagation mechanism based on binary masks to output the target box prediction results. The full-image features and candidate prediction boxes of the substation inspection image are input into the trained target classification network for processing; The target classification network is used to filter a set of candidate defect categories based on the similarity between the candidate prediction box and the category prototype vector, and to generate the candidate defect category confidence of the candidate prediction box by constructing intra-class response embedding, inter-class ranking embedding and background semantic embedding, and using a multi-source embedding fusion method. A text-image alignment network is introduced to assist in scoring the semantic consistency of candidate predicted boxes whose candidate defect category confidence scores are within a preset range, thereby obtaining the calibrated candidate defect category confidence scores. The category prototype vector is constructed during model training and dynamically updated based on a momentum update mechanism.

2. The substation defect detection method for small sample scenarios according to claim 1, characterized in that, Prototype feature embedding is performed by combining category prototype vectors with an attention mechanism, including: Using the full-image features and the expanded candidate prediction boxes, candidate region features within each expanded candidate prediction box are extracted through region alignment operations. The query vector and key vector in the attention mechanism are used to perform a linear transformation on the candidate region features and category prototype vector to obtain a low-dimensional embedding representation. Calculate the attention response relationship between each position of the extended candidate prediction box and the low-dimensional embedding representation to obtain the foreground category response map and the background category response map; The foreground region embedding features generated based on the foreground category response map are concatenated with the background embedding features generated based on the background category response map to obtain region embedding features, which are then used as input for the region propagation mechanism.

3. The substation defect detection method for small sample scenarios according to claim 2, characterized in that, A region propagation mechanism based on binary masks and mask stacking is constructed to output target bounding box prediction results, including: An input mask is constructed based on the positional relationship between the sampling points in the grid sampling and the candidate prediction box; the input mask is used to characterize whether the sampling points in the grid sampling fall into the candidate prediction box; The input mask is used as the initial input to the region propagation mechanism and fused with the region embedding features to generate a mask response map layer by layer through a multi-layer propagation network. The final mask response map output by the multilayer propagation network is decoded into the normalized coordinates of the target box through coordinate integration.

4. The substation defect detection method for small sample scenarios according to claim 1, characterized in that, The introduced image-text alignment network assists in scoring the semantic consistency of candidate predicted boxes whose candidate defect category confidence scores fall within a preset range, resulting in calibrated candidate defect category confidence scores, including: Extract the region visual feature vectors of candidate predicted boxes whose confidence scores for candidate defect categories fall within a preset range, and calculate the similarity between these vectors and the category text feature matrix generated based on the natural language description of each defect category to obtain a semantic similarity score. The semantic similarity score is weighted and fused with the candidate defect category confidence score of the corresponding candidate prediction box to obtain the calibrated candidate defect category confidence score.

5. The substation defect detection method for small sample scenarios according to claim 1, characterized in that, The process of constructing category prototype vectors during model training includes: Based on the number of sample images in the sample set, the sample type of the sample set is determined; the sample type is either a basic category or a small sample category. Extract the average semantic features of the defect regions of the support set sample images in the sample set for each defect category, and select the appropriate aggregation method to perform feature aggregation on the average semantic features according to the sample type to which the support set sample images belong. If the sample type is a small sample category, then the average semantic features are aggregated using the mean aggregation method. If the sample type belongs to the basic category, then the average semantic features are aggregated using the K-Means clustering algorithm based on cosine distance.

6. The substation defect detection method for small sample scenarios according to claim 5, characterized in that, The method further includes: After each training iteration, a momentum update method is used to update the prototype vector of the category that best matches the query set sample image in the defect category to which the query set sample image belongs.

7. The substation defect detection method for small sample scenarios according to claim 1, characterized in that, During model training, the loss value of the target classification network is calculated by fusing Focal Loss with a classification loss function based on positive and negative sample co-optimization.

8. A substation defect detection device for small sample scenarios, characterized in that, include: The data extraction module is used to input the substation inspection images to be detected into the backbone network and the RPN network respectively, and extract the full-image features and candidate prediction boxes of the substation inspection images. The regression network processing module inputs the full-image features and candidate prediction boxes of the substation inspection image into the trained regression network that integrates prototype attention and mask propagation for processing. The regression network is used to spatially expand and grid sample the candidate prediction boxes, embed prototype features by combining category prototype vectors and attention mechanisms, and construct a region propagation mechanism based on binary masks to stack masks and output the target box prediction results. The target classification network processing module inputs the full-image features and candidate prediction boxes of the substation inspection image into the trained target classification network for processing; The target classification network is used to filter and obtain a set of candidate defect categories based on the similarity between the candidate prediction box and the category prototype vector; for each candidate defect category, intra-class response embedding, inter-class ranking embedding and background semantic embedding are constructed respectively, and the candidate defect category confidence of the candidate prediction box is generated by multi-source embedding fusion. The image-text alignment network processing module introduces an image-text alignment network to assist in scoring the semantic consistency of candidate prediction boxes whose candidate defect category confidence scores are within a preset range, thereby obtaining the calibrated candidate defect category confidence scores. The category prototype vector is constructed during model training and dynamically updated based on a momentum update mechanism.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the substation defect detection method for small sample scenarios as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the substation defect detection method for small sample scenarios as described in any one of claims 1 to 7.

Images