Unmanned aerial vehicle electric power inspection image segmentation method and system

Abstract

The invention discloses an unmanned aerial vehicle electric power inspection image segmentation method and system, and relates to the technical field of image segmentation. By means of a labeled sample image set, positive and negative information is fully fused in a confidence map cooperation module to carry out confidence modeling, so that a cooperation confidence map can accurately reflect the difference between a target and a background; the negative sample distribution data effectively depicts the features of background interference, then high-discrimination positive and negative points are screened out in the point selection module, accurate prompt is provided for segmentation, iterative optimization is performed through the noise sensing and refining module, and therefore the segmentation efficiency is improved under the condition that only few labeled samples are needed. According to the method, high-precision, high-robustness and training-free segmentation of the target component in the electric power inspection image is realized, the influence of complex backgrounds and similar interferents is effectively resisted, the method has high robustness, the technical problems of low segmentation precision and insufficient robustness under the condition of few labeled samples in the prior art are fully solved, and the requirements of electric power inspection on automation and precision are met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, in particular to a UAV power inspection image segmentation method and system. BACKGROUND

[0002] With the deep integration of UAV technology and computer vision technology, using UAVs to carry high-definition imaging equipment to carry out automatic inspection of power line facilities has become a key means to ensure the safe and stable operation of the power grid. By implementing accurate semantic segmentation on the inspection image, defects and potential risks of key components such as power transmission lines, insulators, and towers can be automatically identified, greatly improving the inspection efficiency and fault positioning accuracy, and providing strong support for the intelligent operation and maintenance of the power system.

[0003] In the field of power inspection image segmentation, traditional deep learning methods highly depend on large-scale and high-quality labeled training data. However, the power inspection scene has its own characteristics. It is extremely costly and time-consuming to obtain labeled samples covering all equipment types and defect types, especially rare fault samples are extremely scarce. This data-dependent characteristic seriously restricts the application of traditional methods in actual power inspection scenarios, making accurate segmentation under the condition of few samples or even zero samples a difficult problem to be solved in the industry.

[0004] In recent years, the emergence of large visual models has provided a new solution to this problem. After pre-training on a large amount of general data, such models have strong zero-shot segmentation capabilities and can segment unknown images with only human prompts. In theory, they are suitable for power inspection scenarios where labeled data is scarce. However, existing prompt-based large visual models still have significant limitations when directly applied to UAV power inspection image segmentation. For example, existing methods focus on indicating what needs to be segmented, and lack effective integration and deep utilization of prompts such as what does not need to be segmented. In complex background and numerous interference (such as insulators and background trees, bolts and similar texture background) inspection images, it is difficult to accurately distinguish targets and interference. At the same time, existing training-free application solutions fail to build a unified framework to effectively convert the domain knowledge contained in few-shot labeled samples into prompt information input into the model, resulting in semantic understanding bias when the model faces highly specialized conditions such as complex background, varying target size, and large differences in lighting conditions, leading to false segmentation of non-target objects, and unsatisfactory segmentation accuracy, robustness, and generalization ability. SUMMARY

[0005] The present application provides a UAV power inspection image segmentation method and system, which solves the technical problem of how to achieve high-precision and training-free segmentation of power inspection images under the condition of few labeled samples.

[0006] The unmanned aerial vehicle power inspection image segmentation method provided in the first aspect of the application comprises:

[0007] An inspection image set of a power line is acquired and preprocessed to obtain an image to be segmented and a labeled sample image set;

[0008] A preset power inspection segmentation model is acquired, wherein the preset power inspection segmentation model comprises a confidence map cooperation module, a point selection module and a noise perception refinement module connected in sequence;

[0009] The labeled sample image set and the image to be segmented are input into the confidence map cooperation module for feature fusion and confidence modeling to obtain a cooperative confidence map and negative sample distribution data;

[0010] The cooperative confidence map and the negative sample distribution data are input into the point selection module for positive and negative point screening and clustering to obtain a coarse segmentation mask image;

[0011] The coarse segmentation mask image and the image to be segmented are input into the noise perception refinement module for iterative mask refinement to obtain a segmentation result.

[0012] Optionally, the feature fusion and confidence modeling specifically comprise:

[0013] Global feature extraction is performed on each labeled sample image in the labeled sample image set to obtain a corresponding first high-level semantic feature map;

[0014] A second high-level semantic feature map of the image to be segmented is extracted;

[0015] A pixel set mask of each labeled sample image is extracted;

[0016] A global positive confidence map is determined based on the first high-level semantic feature map, the second high-level semantic feature map and the pixel set mask;

[0017] A reverse mask is generated using the pixel set mask;

[0018] A global negative confidence map is determined based on the first high-level semantic feature map, the second high-level semantic feature map and the reverse mask;

[0019] Segmentation is performed on the image to be segmented to obtain a locally sensitive segmentation map;

[0020] The global positive confidence map is input into a preset global feature extraction model to output a global context feature map;

[0021] The locally sensitive segmentation map and the global context feature map are weighted and fused to obtain a cooperative confidence map;

[0022] The global negative confidence map is obtained by performing maximum likelihood estimation on the global negative confidence map and inputting the estimation result into a preset Gaussian probability density function.

[0023] Optionally, the determining the global positive confidence map based on the first high-level semantic feature map, the second high-level semantic feature map, and the pixel set mask comprises:

[0024] The spatial-level multiplication is performed on each first high-level semantic feature map and the associated pixel set mask to obtain a plurality of support target features.

[0025] The cosine similarity between the second high-level semantic feature map and each support target feature is calculated to obtain a plurality of local positive confidence maps.

[0026] The plurality of local positive confidence maps are aggregated to obtain a global positive confidence map.

[0027] Optionally, the determining the global negative confidence map based on the first high-level semantic feature map, the second high-level semantic feature map, and the reverse mask comprises:

[0028] The spatial-level multiplication is performed on each first high-level semantic feature map and the associated reverse mask to obtain a plurality of background region features.

[0029] The cosine similarity between the second high-level semantic feature map and each background region feature is calculated to obtain a plurality of local negative confidence maps.

[0030] The plurality of local negative confidence maps are aggregated to obtain a global negative confidence map.

[0031] Optionally, the positive and negative point screening and clustering specifically comprises:

[0032] Based on a preset positive point screening parameter, a high-confidence pixel screening number is determined.

[0033] From the collaborative confidence map, a high-confidence pixel of the high-confidence pixel screening number is selected as an initial positive point set.

[0034] The initial positive point set is clustered to obtain a target positive point set.

[0035] Based on a preset negative point screening parameter, a feature value screening interval is determined.

[0036] According to the feature value screening interval, a global negative confidence map pixel is selected from the negative sample distribution data as an initial negative point set.

[0037] According to the initial negative point set, a target negative point set is determined.

[0038] Label assignment is performed on the target positive point set and the target negative point set respectively, and the target positive point set and the target negative point set after label assignment are coupled to generate a label prompt point set;

[0039] The label prompt point set is input into a preset segmentation model to obtain a coarse segmentation mask image of the image to be segmented.

[0040] Optionally, the iterative mask refinement specifically includes:

[0041] The coarse segmentation mask image is eroded and expanded to obtain an initial optimized mask image;

[0042] Multiple connected regions in the initial optimized mask image are marked;

[0043] Average confidence scores of the connected regions are determined;

[0044] A connected region associated with a maximum value of the average confidence scores is selected as a target region;

[0045] A sub-optimized mask image associated with the target region is selected from the initial optimized mask image as a refined segmentation mask image;

[0046] The refined segmentation mask image is taken as a new coarse segmentation mask image, and the step of eroding and expanding the coarse segmentation mask image to obtain an initial optimized mask image is executed until a preset iteration number is reached, and then the output refined segmentation mask image is taken as a segmentation result.

[0047] Optionally, the determination of the average confidence scores of the connected regions includes:

[0048] A region mask corresponding to each connected region is extracted;

[0049] Each region mask and the second high-level semantic feature map are multiplied in a spatial level to obtain multiple region features;

[0050] The support target features are classified by category, and the classified support target features are averaged and pooled to obtain comprehensive support features of multiple power component categories;

[0051] The region features are associated with power component category matching corresponding comprehensive support features;

[0052] Similarities are calculated between the region features and the associated comprehensive support features to obtain confidence scores corresponding to the connected regions;

[0053] The confidence scores are used to calculate average confidence scores corresponding to the connected regions.

[0054] Optionally, the initial optimization mask image comprises a plurality of the sub-optimization mask images, and each of the sub-optimization mask images corresponds to one of the connected regions.

[0055] Optionally, the obtaining the inspection image set of the power line and preprocessing to obtain the image to be segmented and the labeled sample image set comprises:

[0056] obtaining the inspection image set of the power line;

[0057] sequentially performing geometric correction, light and color optimization, image blocking, image alignment and screening on the inspection image set to obtain the image to be segmented;

[0058] masking the inspection image set after light and color optimization;

[0059] screening the typical defect sample images from the inspection image set after masking to construct the labeled sample image set, and the labeled sample image set is a set of each type of power component containing a preset number of typical defect sample images.

[0060] The second aspect of the present application provides an unmanned aerial vehicle power inspection image segmentation system, comprising:

[0061] a data acquisition module for obtaining the inspection image set of the power line and preprocessing to obtain the image to be segmented and the labeled sample image set;

[0062] a model acquisition module for acquiring a preset power inspection segmentation model, wherein the preset power inspection segmentation model comprises a confidence map cooperation module, a point selection module and a noise perception refinement module connected in sequence;

[0063] a first processing module for inputting the labeled sample image set and the image to be segmented into the confidence map cooperation module for feature fusion and confidence modeling to obtain a cooperative confidence map and negative sample distribution data;

[0064] a second processing module for inputting the cooperative confidence map and the negative sample distribution data into the point selection module for positive and negative point screening and clustering to obtain a coarse segmentation mask image;

[0065] a third processing module for inputting the coarse segmentation mask image and the image to be segmented into the noise perception refinement module for iterative mask refinement to obtain a segmentation result.

[0066] From the above technical solutions, the present application has the following advantages:

[0067] The application realizes high-precision, high-robustness and training-free segmentation of target components in the power inspection image under the condition of only a few labeled samples, effectively resists the influence of complex background and similar interference objects, has high robustness, fully solves the technical problems of low segmentation precision and insufficient robustness of the prior art under a few labeled samples, and meets the needs of power inspection for automation and precision. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0069] Figure 1 A step flow chart of a UAV power inspection image segmentation method provided by the embodiment of the present application;

[0070] Figure 2 A structural block diagram of a UAV power inspection image segmentation system provided by the embodiment of the present application. DETAILED DESCRIPTION

[0071] The embodiment of the present application provides a UAV power inspection image segmentation method and system, which is used to solve the technical problem of how to realize high-precision and training-free segmentation of power inspection images under the condition of a few labeled samples.

[0072] In order to make the purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the embodiments described below are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0073] Feature fusion and confidence modeling

[0074] Refers to the process of integrating different features (such as local detail features, global structure features, target / background features, etc.) from different sources through weighted fusion, similarity calculation and other methods, and constructing a confidence model of quantifying target / background possibility. Specifically, it includes: fusing local sensitive segmentation map and global context feature map to balance details and structure, calculating the matching degree of target and background through positive / negative confidence map, and finally generating a collaborative confidence map that can comprehensively reflect the possibility of target region and negative sample distribution data, providing a quantitative basis for pixel screening.

[0075] Positive and negative point screening and clustering

[0076] Refers to the process of screening high-confidence target pixels (positive points) and background interference pixels (negative points) from the confidence map, and optimizing the distribution of point sets through clustering algorithm. Specifically, it includes: determining the screening quantity and interval based on the preset parameters, selecting high-confidence pixels from the collaborative confidence map as the initial positive point set, and selecting background interference pixels from the negative sample distribution as the initial negative point set; then, through K-Means clustering algorithm, the initial point set is aggregated to obtain a target positive point set and a target negative point set that can cover the target key area and the background interference area, providing accurate point hints for the segmentation model.

[0077] Iterative mask refinement

[0078] Refers to an iterative process of gradually optimizing the segmentation result based on the coarse segmentation mask through morphological operations, connected region screening and other means, and repeating the process to improve accuracy. Specifically, it includes: performing erosion and dilation operations on the coarse segmentation mask to remove noise and restore target structure, marking connected regions and calculating the average confidence of each region, and selecting the most reliable target region as the refined mask; repeat the above steps with the refined mask as the new input until the preset iteration number is reached, and finally output a high-precision segmentation mask, achieving gradual optimization from coarse segmentation to fine segmentation.

[0079] Please refer to Figure 1 , Figure 1 A step flowchart of a UAV power inspection image segmentation method provided by the embodiment of the present application.

[0080] The UAV power inspection image segmentation method provided by the present application comprises:

[0081] Step 101, obtaining the inspection image set of the power line and performing preprocessing to obtain the image to be segmented and the labeled sample image set.

[0082] Further, step 101 can include the following sub-steps:

[0083] S11, obtaining the inspection image set of the power line.

[0084] The inspection image set refers to a batch of images generated by the UAV cruising along the target power line to take pictures, containing multi-angle images of key power components such as power transmission lines, insulators, fittings, and towers, with complete metadata, and is the basis of raw data for all subsequent processing links.

[0085] The power component refers to the core structure and functional component in the power line, mainly including the power transmission line (power transmission carrier), insulator (insulation performance guarantee), fitting (connecting and fixing component), and tower (supporting line structure), which is the core target of the inspection image segmentation and defect identification.

[0086] In the embodiment of the present application, the UAV carrying a high-definition camera performs automatic cruising and shooting tasks along the target power line, the flight height is adjusted to 30-60 meters according to the voltage level of the power line, the flight speed is controlled at ≤5 meters / second to avoid image motion blur, and the shooting angle covers front view, side view, overhead view and oblique view, ensuring comprehensive capture of the appearance state of key power components such as power transmission lines, insulators, fittings, and towers. At the same time, the metadata of each image including GPS coordinates, shooting time, gimbal attitude, camera parameters and line voltage level are recorded synchronously, and finally the inspection image set containing multi-segment line, multi-type component and multi-angle image is formed.

[0087] It is worth mentioning that the camera parameters of the high-definition camera should be set to a resolution of not less than 4K, a long focal length lens, an aperture setting of f / 5.6 to f / 8, a shutter speed of not less than 1 / 1000 seconds, and an ISO control of within 400, and manual white balance or lock setting should be preferred to avoid color shift caused by light change.

[0088] The shooting time should be selected on sunny days to ensure uniform light and short shadows, and to avoid backlight or strong reflection affecting the imaging quality. At the same time, the metadata of each image including GPS coordinates, shooting time, gimbal attitude, device type, voltage level and camera parameters should be recorded for subsequent analysis and positioning.

[0089] S12, the inspection image set is sequentially subjected to geometric correction, illumination and color optimization, image blocking, image alignment and screening to obtain the image to be segmented.

[0090] Geometric correction refers to a process of eliminating perspective distortion of images caused by changes in shooting device attitude by using camera parameters and attitude data through algorithms, so that the shape and proportion of power components in the image are consistent with the actual situation.

[0091] Illumination and color optimization refers to a processing method of adjusting the brightness and contrast of the image through algorithms, unifying the color standard under different shooting conditions, and eliminating the influence of uneven light and color deviation on component recognition.

[0092] Image blocking refers to the processing step of cutting high-resolution inspection images into standardized image blocks that meet the model input requirements and recording the global coordinates of each block in the original image, facilitating subsequent result splicing.

[0093] Image alignment refers to the processing procedure of keeping the spatial position and lighting style of the image to be processed consistent with the labeled sample image through feature matching and transformation algorithm, thereby improving the accuracy of feature matching.

[0094] Image to be segmented refers to a single / standardized image that contains the power target component to be analyzed and meets the model processing requirements after pre-processing, which is the direct processing object of the segmentation task.

[0095] In the embodiment of the present application, based on the camera calibration parameters and gimbal attitude data in the inspection image metadata obtained by S11, geometric correction is performed on each image in the inspection image set through homography transformation algorithm to eliminate perspective distortion caused by the tilt and rotation of the unmanned aerial vehicle, ensuring the accuracy of the power component form and proportion; then the Retinex algorithm is used to decompose the image light and reflection components, and color mapping is performed in combination with gray card reference to complete light and color optimization, solve the problem of uneven light and color deviation in different shooting periods, and enhance the component detail recognition; subsequently, the corrected and optimized high-resolution image is cut into standardized image blocks with global coordinates according to fixed size to avoid affecting the processing effect due to image size exceeding the model input limit; then through SIFT feature matching and affine transformation, the spatial position and lighting style of each image block and the subsequent labeled sample image used for reference are aligned to ensure the reliability of feature matching; finally, according to the inspection task requirements (such as specific line segment, target component type), single / multi-piece images containing the target to be analyzed are selected from the aligned image blocks as the image to be segmented.

[0096] S13, mask labeling is performed on the light and color optimized inspection image set.

[0097] Mask labeling refers to a labeling method that outlines the contour by hand and assigns pixel label values (0 or 1) on the image to realize pixel-level differentiation between target and non-target regions. The generated mask can be directly used for model feature learning and segmentation reference.

[0098] Pixel-level binary mask refers to a single-channel image that is completely consistent with the size of the original image, containing only two pixel values of 0 and 1, where 1 corresponds to the target component / defect area, and 0 corresponds to the background and interference area. It is a concrete carrier of labeling information.

[0099] Key power component defects refer to abnormal states of key power components during operation, mainly including damage (incomplete component structure), rust (oxidation and corrosion of metal components), and missing (component falling off or not installed), which are the objects that need to be marked as key in mask labeling.

[0100] Interference refers to objects in the inspection image that are not related to the target power component and may affect the segmentation accuracy, such as branches around the line, sky background, non-target structure of the tower, etc. When labeling, it needs to be marked as a non-target area.

[0101] The labeling tool refers to a software tool for implementing image mask labeling, supporting functions such as manual contouring, pixel marking, and labeling information storage, and is suitable for the labeling needs of power inspection images.

[0102] In the embodiment of the application, based on the inspection image set after illumination and color optimization in S12, the accurate contours of key power components such as power transmission lines, insulators, fittings, and towers and their typical defects (such as damage, rust, and missing) in the image are manually outlined by technicians using a labeling tool. The pixels inside the contours are marked as target pixels (with a mark value of 1), and the pixels outside the contours and the interference (such as branches, sky, and non-target structures) are marked as non-target pixels (with a mark value of 0). A pixel-level binary mask consistent with the size of each optimized image is generated, achieving accurate differentiation between target and background areas and providing basic data with labeling information for subsequent construction of a labeled sample image set.

[0103] It should be noted that the preset number is preferably 3-5.

[0104] S14, selecting typical defect sample images from the mask-labeled inspection image set to construct a labeled sample image set, which is a collection of 3-5 typical defect sample images for each type of power component.

[0105] Typical defect sample images refer to images selected from mask-labeled images that have clear defect features, no severe occlusion, small background interference, and can represent the general form of a specific defect of a certain type of power component, with strong feature representativeness.

[0106] The labeled sample image set refers to a small sample image set constructed by classifying power component types and defect types, containing only 3-5 typical defect sample images for each type of power component, which is the core labeling data for model reference.

[0107] The power component type refers to the classification of core functional components in the power line, such as insulators, fittings, towers, and power transmission lines, which is one of the core bases for sample classification and selection.

[0108] The defect type refers to the classification of abnormal states of power components in operation, such as insulator damage, fitting rust, and tower cracks, which is another core basis for sample classification and selection.

[0109] The few-sample application requirement refers to a requirement that a model only needs a small number of labeled samples (3-5 samples per class) to complete feature learning and task adaptation. The requirement is suitable for the power inspection scene with high labeling cost and scarce samples.

[0110] In the embodiment of the present application, based on the inspection image set after mask labeling is completed by S13, the inspection image set is classified and arranged according to “power component type + defect type” (such as insulator-breakage, hardware-rust, tower-crack), and then for each combination, images with clear defect features, no serious occlusion, small background interference and representing the general form of the defect are selected as typical defect sample images. Then, 3-5 typical defect sample images are selected according to the category, and the labeled sample image set is formed to ensure that the set covers the core defect types of various power components and meets the few-sample application requirement, and to provide high-quality reference samples for feature fusion and confidence modeling of the model.

[0111] It is worth mentioning that each type of power component only contains 3-5 typical defect sample images, which greatly reduces the collection and labeling cost of labeled samples in the power inspection scene, avoids the problem of high human and time investment caused by massive sample requirements, and adapts to the actual situation of rare defect sample scarcity in power inspection. At the same time, 3-5 typical defect sample images can accurately cover the key features of the core defects of various power components, which can provide high-quality and representative reference for the model, avoid the influence of redundant samples on the processing efficiency of the model, perfectly meet the technical requirement of few-sample segmentation, and significantly improve the practicality under actual working conditions while ensuring the segmentation accuracy and robustness of the model.

[0112] Step 102, a pre-set power inspection segmentation model is obtained, and the pre-set power inspection segmentation model includes a confidence map cooperation module, a point selection module and a noise perception refinement module connected in sequence.

[0113] The pre-set power inspection segmentation model is a pre-constructed segmentation inference framework that does not need to be additionally trained for the power inspection scene, which is built based on the pre-training weight of a large vision model and includes multiple functional modules cooperating to complete the segmentation task.

[0114] The confidence map cooperation module is one of the core functional modules in the model, which is responsible for fusing the features of the labeled sample and the image to be segmented, and generating a cooperative confidence map (reflecting the probability that a pixel belongs to a target) and negative sample distribution data (characterizing background interference features) through confidence modeling.

[0115] The point selection module is a module that receives the output of the confidence map cooperation module, extracts high-confidence positive points (pixels that may belong to a target) from the cooperative confidence map and high-recognition negative points (pixels that may belong to the background) from the negative sample distribution through threshold filtering and clustering processing, and finally generates a coarse segmentation mask image.

[0116] The noise-aware refinement module refers to the end processing module of the model, taking the coarse segmentation mask image and the image to be segmented as input, optimizing the mask boundary through iteration, eliminating noise areas, improving segmentation accuracy, and outputting the final accurate segmentation result.

[0117] The free training inference framework refers to a model architecture that does not need to fine-tune parameters for specific tasks (such as power inspection segmentation) and can complete inference directly based on pre-trained weights, reducing dependence on large-scale labeled data and shortening the application landing cycle.

[0118] In the embodiment of the present application, the pre-installed power inspection segmentation model is a free training inference framework based on the pre-training weights of a large vision model, which does not need to fine-tune parameters for the power inspection scene, wherein the confidence map synergy module (CMSM) fuses the features extracted by SAM-ViT (i.e. the pre-installed segmentation model) and DINOv2 (i.e. the pre-installed global feature extraction model), combines support mask calculation to model the negative class distribution, and generates preliminary positive and negative region responses. This mechanism helps to distinguish target regions and interference regions, thereby generating high-quality positive and negative prompts to support subsequent accurate segmentation; the point selection module (PSM) sorts pixels according to the confidence score of the collaborative confidence map, and filters out high-discriminative positive and negative point prompts in combination with the negative confidence distribution to guide segmentation. This method can more accurately locate the key parts and potential defects of power equipment, improving the accuracy and robustness of segmentation; the noise-aware refinement module (NRM) uses the feature relationship between the generated initial segmentation result and the query image to refine the coarse segmentation mask, suppress noise and optimize the boundary, providing clearer and more accurate segmentation results, and providing reliable basis for subsequent fault diagnosis and operation and maintenance decision-making. The three modules are linked in sequence to form a complete segmentation inference link, adapting to the power inspection image segmentation requirements under the condition of few samples.

[0119] It should be noted that the SAM-ViT pre-installed segmentation model (Segment Anything Model with Vision Transformer, segmentation everything model based on vision transformer) is a general image segmentation model that can segment any target in an image according to the input prompt (such as point, box, text), with strong generalization ability, which can be used for accurate identification of the contours of power components in power inspection image segmentation.

[0120] The transformer architecture refers to a deep learning model structure based on a self-attention mechanism, which can effectively capture long-distance dependencies in input data, and can simultaneously focus on local details and global context in image feature extraction, thereby improving the integrity of feature expression.

[0121] The DINOv2 is a self-supervised learning visual feature extraction model, which can learn image feature representation with strong generalization ability without manual annotation data, and can be used for extracting global features of inspection images in the power inspection scene, thereby providing a basis for feature fusion and confidence modeling.

[0122] In step 103, the labeled sample image set and the image to be segmented are input into the confidence map collaborative module for feature fusion and confidence modeling, to obtain a collaborative confidence map and negative sample distribution data.

[0123] Further, the feature fusion and confidence modeling are specifically as follows:

[0124] A11, global feature extraction is performed on each labeled sample image in the labeled sample image set to obtain a corresponding first high-level semantic feature map.

[0125] The first high-level semantic feature map refers to a high-level semantic feature map extracted by the DINOv2 from the labeled sample image, which contains global component structure information and is a core carrier of support set features.

[0126] In the embodiment of the present application, the labeled sample image set is input into the pre-trained DINOv2 model to extract the first high-level semantic feature map containing global component structure information .

[0127] A12, a second high-level semantic feature map of the image to be segmented is extracted.

[0128] The second high-level semantic feature map refers to a high-level semantic feature map extracted by the DINOv2 from the image to be segmented, which contains segmented component information and is a core carrier of query features.

[0129] In the embodiment of the present application, the image to be segmented is input into the DINOv2 model to extract the second high-level semantic feature map containing segmented component information .

[0130] A13, a pixel set mask of each labeled sample image is extracted.

[0131] The pixel set mask index is a binary mask of the sample image. 1 marks the target power component and defect area, and 0 marks the background area. It is used to filter target features of the support set image.

[0132] In this embodiment of the invention, for each image in the labeled sample image set, its corresponding pixel-level mask is extracted. 1 represents the target power component and defect area, and 0 represents the background area.

[0133] A14. Determine the global positive confidence map based on the first high-level semantic feature map, the second high-level semantic feature map, and the pixel set mask.

[0134] Furthermore, A14 may include the following sub-steps:

[0135] A141. Spatial multiplication is performed using each first high-level semantic feature map and the associated pixel set mask to obtain multiple supporting target features.

[0136] Spatial multiplication refers to the operation of performing multiplication pixel by pixel in the spatial dimension of the feature map and the mask. The 0 / 1 values ​​of the mask are used to preserve the features of the target region in the feature map and filter the features of the background region.

[0137] Supported target features refer to the feature set obtained after spatial multiplication processing, which contains only the target power components and defect areas in the labeled sample images. Each feature corresponds to a pixel in the target area.

[0138] In this embodiment of the invention, for the first high-level semantic feature map of each labeled sample image obtained by A11, a pixel-by-pixel multiplication operation is performed with the corresponding pixel set mask extracted by A13 according to spatial position. Positions with a mask value of 1 retain the pixel features of the original feature map, while positions with a mask value of 0 have their features set to 0. This filters out interfering features from the background region, retaining only the effective features of the target power components and defect areas in the labeled samples, ultimately obtaining multiple target support features that correspond one-to-one with the labeled sample images. .

[0139] The above process can be encapsulated in the form of a formula, specifically:

[0140]

[0141] In the formula, Indicates the first The first labeled sample image There are support target feature vectors, where The number of pixels in the target area. For the first in the target area The index of a pixel; representing a feature vector of the i-th target pixel; representing a pixel set mask of the labeled sample image (1 represents a target region and 0 represents a background region); representing a feature vector of the i-th target pixel; representing a first high-level semantic feature map of the i-th labeled sample image; representing a spatial-level multiplication operation of the pixel set mask and the high-level semantic feature map; representing a feature vector excluding invalid pixels, ensuring that the retained features are pure and effective.

[0142] A142, calculating the cosine similarity between the second high-level semantic feature map and each support target feature, to obtain a plurality of local positive confidence maps.

[0143] The local positive confidence map refers to a two-dimensional matrix obtained by calculating the cosine similarity between the second high-level semantic feature map of the image to be segmented and a single support target feature vector. The higher the pixel value, the higher the possibility that the position belongs to the target region, and is used to locally depict target similarity.

[0144] The cosine similarity refers to an index for measuring the similarity of the directions of two vectors, with a value range of [-1, 1]. The closer the value is to 1, the more similar the directions of the vectors are. Here, it is used to quantify the semantic correlation degree of the features of the image to be segmented and the support target features.

[0145] In the embodiments of the present application, for the second high-level semantic feature map obtained by A12 and each support target feature obtained by A141, the cosine similarity between the second high-level semantic feature map and each support target feature vector is calculated one by one, and the corresponding local positive confidence map of the i-th sample is generated. the i-th local positive confidence map .

[0146] The above process is encapsulated in the form of a formula, specifically:

[0147]

[0148] In the formula, represents the transpose of the i-th support target feature vector of the i-th labeled sample image, represents the total number of pixels of the target region in the labeled sample image.​​​​​​​​​​​

[0149] A143. Aggregate multiple local positive confidence maps to obtain a global positive confidence map.

[0150] Aggregation refers to the process of combining multiple local positive confidence maps into a more globally representative result through average pooling. This is divided into two levels: first, pooling the data from a single sample... The local positive confidence plots are aggregated into the median positive confidence plot for that sample, and then all of them are... The intermediate positive confidence maps of the samples are aggregated into a global positive confidence map. Hierarchical integration eliminates local bias and enhances the overall feature expression of the target region.

[0151] The global positive confidence map refers to a single confidence map obtained after two-level aggregation. It integrates the target features of all samples in the labeled sample image set with the matching information of the image to be segmented, and is presented in the form of a two-dimensional matrix. Each pixel value reflects the overall confidence that the location belongs to the target region.

[0152] In this embodiment of the invention, considering that the labeled sample image set contains Zhang labeled sample images, and first for each sample Zhang's local positive confidence diagram Initial aggregation is performed using average pooling to obtain the median positive confidence plot of a single sample. Specifically:

[0153] ;

[0154] Again The positive confidence plot of the sample A second aggregation is performed to obtain the global positive confidence map. By integrating confidence information from multiple samples and local areas through two-level aggregation, the global positive confidence map can comprehensively reflect the matching degree between target features and the image to be segmented in all labeled samples, thereby improving the comprehensiveness and reliability of the confidence assessment of the target region.

[0155] A15. Generate an inverse mask using a pixel set mask.

[0156] An inverted mask is a mask obtained by inverting a pixel set mask, where 1 represents the background region and 0 represents the target region, and is used to filter background features.

[0157] In this embodiment of the invention, for the first Pixel set mask of the labeled sample image (1 represents the target, 0 represents the background), generate an inverse mask. (1 represents background, 0 represents target), used to extract background region features.

[0158] A16, determining a global negative confidence map based on the first high-level semantic feature map, the second high-level semantic feature map and the reverse mask.

[0159] Further, A16 can include the following sub-steps:

[0160] A161, performing spatial-level multiplication on each first high-level semantic feature map and the associated reverse mask to obtain a plurality of background region features.

[0161] The background region feature refers to a feature set obtained by performing spatial-level multiplication on the reverse mask and the first high-level semantic feature map, which only contains features of the background region in the labeled sample image, and is used to depict the background semantic information.

[0162] In the embodiment of the application, the first high-level semantic feature map of the i-th labeled sample image obtained by A11 is multiplied with the reverse mask generated by A15 to obtain the background region feature set of the i-th labeled sample image. The first high-level semantic feature map of the i-th labeled sample image is multiplied with the reverse mask generated by A15 to obtain the background region feature set of the i-th labeled sample image .

[0163] The above process is encapsulated into the form of a formula, specifically:

[0164]

[0165] In the formula, represents the total number of pixels of the image, represents the total number of pixels of the target region in the i-th labeled sample image, represents the total number of pixels of the background region in the i-th labeled sample image, represents the i-th background region feature vector of the i-th labeled sample image.

[0166] A162, calculating the cosine similarity between the second high-level semantic feature map and each background region feature to obtain a plurality of local negative confidence maps.

[0167] The local negative confidence map refers to a two-dimensional matrix obtained by calculating the cosine similarity between the second high-level semantic feature map of the image to be segmented and a single background region feature vector, and the higher the pixel value, the higher the possibility that the position belongs to the background region.

[0168] In the embodiment of the application, the cosine similarity between the second high-level semantic feature map obtained by A12 and each background region feature obtained by A161 is calculated to generate a local negative confidence map for each background region pixel.

[0169] The above process is encapsulated into the form of a formula, specifically: ​​

[0170]

[0171] In the formula, represents the transpose of the background area feature vector of the labeled sample image, represents the transpose of the local negative confidence map of the labeled sample image.

[0172] A163, aggregate the plurality of local negative confidence maps to obtain a global negative confidence map.

[0173] The global negative confidence map refers to a single confidence map obtained by aggregating a plurality of local negative confidence maps through average pooling, which comprehensively reflects the overall confidence distribution of the background area in the image to be segmented.

[0174] In the embodiment of the application, the plurality of local negative confidence maps obtained by A162 are aggregated into a global negative confidence map by using an average pooling algorithm, and a higher pixel value indicates that the position is more likely to be a background, providing a confidence reference for the background area for the subsequent point selection module.

[0175] A17, segmenting the image to be segmented to obtain a locally sensitive segmentation map.

[0176] The locally sensitive segmentation map refers to a segmentation map generated by the SAM model for the query image, which is good at capturing small local features of power components (such as bolt texture, component edge, etc.), and provides local detail reference for the segmentation task.

[0177] In the embodiment of the application, the pre-trained SAM model is called to segment the image to be segmented obtained by A12 to generate a locally sensitive segmentation map which is good at capturing small features of power components (such as bolt texture, etc.), and provides local detail information for feature fusion.

[0178] A18, inputting the global positive confidence map into a preset global feature extraction model to output a global context feature map.

[0179] The global context feature map refers to a feature map generated by DINOv2 for extracting the global positive confidence map, which is good at identifying the overall structure of the power component (such as insulator string, overall form of the tower, etc.), and provides global structure reference for the segmentation task.

[0180] In the embodiment of the application, the global positive confidence map obtained by A143 is input into DINOv2 to extract a global context feature map which is good at identifying the overall structure of the power component (such as insulator string, etc.), and provides global structure information for feature fusion.

[0181] A19, the local sensitive segmentation map and the global context feature map are fused by weighting to obtain a collaborative confidence map.

[0182] The collaborative confidence map is obtained by fusing the local sensitive segmentation map and the global context feature map by weighting, integrates local details and global structure information, and is used for accurately distinguishing the target region and the background region.

[0183] In the embodiment of the application, the local sensitive segmentation map and the global context feature map are fused by weighting to obtain a collaborative confidence map.

[0184] The above process is encapsulated in the form of a formula, specifically:

[0185]

[0186] In the formula, the weight coefficient is represented, and the sum is 1, the collaborative confidence map is represented, the Hadamard product (element-by-element multiplication) is represented, and through the fusion mode, the high-confidence region consistent with the two is amplified, and the accuracy of target positioning is enhanced.

[0187] A110, maximum likelihood estimation is performed on the global negative confidence map, and the estimation result is input into a preset Gaussian probability density function to obtain negative sample distribution data.

[0188] The negative sample distribution data is obtained by performing maximum likelihood estimation on the global negative confidence map and substituting it into the Gaussian probability density function, and is used for quantifying the probability that each pixel in the image to be segmented belongs to the background region.

[0189] In the embodiment of the application, for the global negative confidence map obtained in A163, the mean and variance of the background feature are calculated by maximum likelihood estimation, and then input into the preset Gaussian probability density function to obtain the negative sample distribution data .

[0190] It should be noted that in the process of calculating the cosine similarity between the second high-level semantic feature map of the image to be segmented and each background region feature to generate the local negative confidence map, the feature vector of the corresponding pixel in the image to be segmented is , which is the "query feature vector" for similarity calculation between the pixel in the image to be segmented and the background feature of the labeled sample. It can be understood that the feature vector of the corresponding pixel in the second high-level semantic feature map is , which indirectly reflects the semantic correlation degree of the pixel belonging to the background through matching, and is finally used for quantifying the background probability in the Gaussian probability density function.

[0191] The above process is encapsulated in the form of a formula, specifically:

[0192]

[0193] wherein, represents the background feature mean value, represents the background feature variance, represents the background feature vector of a certain pixel in the image to be segmented, represents the negative sample distribution (background probability).

[0194] Step 104, using the cooperative confidence map and the negative sample distribution data to input a point selection module to perform positive and negative point screening and clustering, and obtaining a coarse segmentation mask image.

[0195] Further, the positive and negative point screening and clustering are specifically:

[0196] A21, determining the high-confidence pixel screening number based on a preset positive point screening parameter.

[0197] The preset positive point screening parameter includes a positive point number parameter and a clustering scaling factor. The positive point number parameter refers to a parameter for controlling the final point number of the target positive point set, which is set according to scene factors such as the size of the power component (for example, 20-50), to ensure that the point set can cover the target key area. The clustering scaling factor refers to a parameter for adjusting the number of initial candidate points, which controls the number of high-confidence pixels screened from the cooperative confidence map, to balance the richness of candidate points and the subsequent clustering efficiency.

[0198] In the embodiment of the present application, the positive point number parameter is set to 20-50 (for example, according to the size of the power component) and the clustering scaling factor is set to 0.8 (for example, to control the number of candidate points). The number of high-confidence pixels screened from the cooperative confidence map is calculated as follows: This number is used to screen the pixels most likely to belong to the target core area from the cooperative confidence map subsequently.

[0199] A22, selecting the high-confidence pixels of the high-confidence pixel screening number from the cooperative confidence map as the initial positive point set.

[0200] The initial positive point set refers to the coordinate set of the first high-confidence pixels screened from the cooperative confidence map, which is a preliminary candidate set of the target positive points.

[0201] In the embodiment of the present application, the first high-confidence pixels (these pixels are high-probability candidates of the target core area) are selected from the cooperative confidence map obtained in A19, and their coordinates are recorded to form the initial positive point set .

[0202] ​​A23, clustering the initial positive point set to obtain a target positive point set.

[0203] The target positive point set refers to a set of coordinates of K-Means clustered points obtained by clustering the initial positive point set. The target positive point set refers to a set of coordinates of K-Means clustered points obtained by clustering the initial positive point set.

[0204] In an embodiment of the present application, the K-Means clustering algorithm is used to cluster the initial positive point set to obtain K target positive points (to avoid excessive concentration of points and ensure coverage of different regions of the target), which are the final target positive point set. .

[0205] The above process is encapsulated into a formula, which is as follows:

[0206]

[0207] In the formula, K-Means clustering function is represented by K-Means, and the target positive point set is represented by T.

[0208] A24, determining a feature value screening interval based on a preset negative point screening parameter.

[0209] The negative point screening parameter includes a confidence interval coefficient, a negative point number parameter, and a clustering scaling factor, which are used to guide the screening and clustering of background interference region pixels to ensure that the negative hint points can effectively suppress false segmentation.

[0210] In an embodiment of the present application, the confidence interval coefficient is set to 1.2 (for example). The confidence interval coefficient is set to 0.8 (for example), and the feature value screening interval is determined as , which is used to screen the pixels of the interference region in the background similar to the target. A25, selecting global negative confidence image pixels from the negative sample distribution data as an initial negative point set according to the feature value screening interval.

[0211] The feature value screening interval refers to an interval determined based on the background feature mean, variance, and confidence interval coefficient, which is used to screen the pixels of the interference region in the background similar to the target.

[0212] The initial negative point set refers to a set of pixel coordinates filtered from the negative sample distribution data, whose feature values are within the set interval, and is a preliminary candidate set of the background interference region.

[0213] In an embodiment of the present application, from the negative sample distribution data obtained in A110, the feature values of the pixels in the negative sample distribution data are filtered to obtain an initial negative point set whose feature values are within the interval

[0214] ​​​​The coordinates of the pixels within the interval (these pixels are interference areas in the background similar to the target) are recorded to form an initial negative point set. .

[0215] A26. Determine the target negative point set based on the initial negative point set.

[0216] The target negative point set refers to the set of candidate negative points obtained by K-Means clustering. A set of centroid coordinates is used to suppress interference regions in the background that are similar to the target, thereby reducing missegmentation.

[0217] In this embodiment of the invention, a parameter for the number of negative points is set. (and Similar (e.g., 20-50), clustering scaling factor (e.g., 0.8); if The number of pixels is less than or equal to Take directly As a candidate negative point set (denoted as Otherwise, select by random sampling. Composed of pixels ; after that Using the K-Means clustering algorithm, we obtain The centroids are used as the final target negative point set. .

[0218]

[0219] In the formula, This represents the target negative point set.

[0220] A27. Assign labels to the target positive point set and the target negative point set respectively, and couple the target positive point set and the target negative point set after label assignment to generate a label prompt point set.

[0221] The label cue point set is formed by merging the set of positive target points and the set of negative target points with labels. Each point contains pixel coordinates and a label (1 represents the target and 0 represents the background), which is used to guide the preset segmentation model to generate a coarse segmentation mask.

[0222] In this embodiment of the invention, the coordinates of each pixel in the target point set obtained by A23 are... Assign label "1" (representing the target) to each pixel coordinate in the target negative point set obtained from A26. Assign label "0" (representing background), then merge the two into a label cue point set. .

[0223] The above process can be encapsulated in the form of a formula, specifically:

[0224] .

[0225] A28, adopt the label prompt point set input preset segmentation model, obtain the rough segmentation mask image of the image to be segmented.

[0226] The rough segmentation mask image refers to a binary image generated by the preset segmentation model according to the label prompt point set, 1 marks the target segmentation area, and 0 marks the background area, which is the basis for subsequent fine segmentation.

[0227] In the embodiment of the application, the label prompt point set P generated by A27 is input into the pre-trained SAM model, and SAM generates a rough segmentation mask image of the image to be segmented according to the position of the point prompt and the label , the image is a binary image, 1 represents the target segmentation area, and 0 represents the background.

[0228] Step 105, adopt the rough segmentation mask image and the image to be segmented to input the noise perception refinement module for iterative mask refinement to obtain a segmentation result.

[0229] Further, the iterative mask refinement is specifically:

[0230] A31, erode and dilate the rough segmentation mask image to obtain an initial optimized mask image.

[0231] The morphological erosion operation refers to a morphological operation of image processing, which traverses the image through a structure element, removes the bright area (target area) or dark area (background area) in the image smaller than the structure element, which is used here to remove small noise blocks in the rough segmentation mask.

[0232] The morphological dilation operation refers to a morphological operation opposite to the erosion operation, which traverses the image through a structure element, expands the bright area (target area) in the image, which is used here to restore the target main structure area reduced after the erosion operation.

[0233] The initial optimized mask image refers to the mask image obtained after the erosion and dilation operations, which removes small noise blocks and retains the main structure area of the target.

[0234] In the embodiment of the application, first, the morphological erosion operation (using a 3x3 or 5x5 rectangular structure element) is used to remove small noise blocks (which are most likely to be mis-segmented areas) in the rough segmentation mask image obtained by A28 whose area is smaller than a set threshold (such as 50 pixels); then the morphological dilation operation (using the same structure element as the erosion operation) is used to restore the target main structure area (to avoid the target area being reduced due to erosion) to obtain the initial optimized mask image .

[0235] A32, mark a plurality of connected regions in the initial optimized mask image.

[0236] The connected region is a region in which pixel values are continuous (connected) in the initial optimization mask image, and each region corresponds to a sub-optimization mask image, which is used for subsequent region confidence analysis.

[0237] The initial optimization mask image includes a plurality of sub-optimization mask images, and each sub-optimization mask image corresponds to a connected region.

[0238] In the embodiment of the application, the initial optimization mask image obtained by A31 is subjected to connected region analysis, and all connected regions (denoted as , region number) are marked, and each connected region corresponds to a sub-optimization mask image, and the number of pixels of each connected region is calculated .

[0239] A33, determine the average confidence score of each connected region.

[0240] Further, A33 can include the following sub-steps:

[0241] A331, extract the region mask corresponding to each connected region.

[0242] The region mask is a binary mask containing only the pixels of a single connected region, which is used to extract the semantic features of the region.

[0243] In the embodiment of the application, for each connected region, the region mask corresponding thereto is extracted , which contains only the pixels of the corresponding connected region (1 represents inside the region and 0 represents outside the region).

[0244] A332, spatially multiply each region mask with the second high-level semantic feature map to obtain a plurality of region features.

[0245] The region feature is a feature obtained by spatially multiplying the connected region with the second high-level semantic feature map, which reflects the semantic attributes of the region.

[0246] In the embodiment of the application, each region mask is spatially multiplied with the second high-level semantic feature map obtained by A12 to extract a region feature set corresponding to the connected region .

[0247] The above process is encapsulated in the form of a formula, which is specifically:

[0248]

[0249] In the formula, denotes the region feature set of the th connected region, is the an index of the pixel; a region mask representing the first connected region; a number of pixels representing the first connected region.

[0250] A333, classifying the plurality of support target features, and performing average pooling on the classified plurality of support target features to obtain comprehensive support features of the plurality of power component categories.

[0251] The comprehensive support features are features obtained by performing average pooling on support target features of the same category, and represent typical semantic features of a certain category of power components, and are used for similarity matching of region features.

[0252] In the embodiment of the application, the support target features obtained by A141 are classified according to power component categories (such as insulators, towers, etc.) and defect types (such as normal, broken, etc.), and average pooling is performed on the support target features in each category to obtain comprehensive support features corresponding to the category , which are used for subsequent semantic similarity matching.

[0253] A334, according to the region feature, the power component category is matched to the corresponding comprehensive support feature.

[0254] In the embodiment of the application, the semantic attributes of the region feature are analyzed, the metadata (such as device type, inspection task) of the query image or the previous recognition result is combined to determine the power component category and the defect type to which the current connected region belongs, and then the corresponding comprehensive support feature is matched.

[0255] A335, the similarity between the region feature and the associated comprehensive support feature is calculated to obtain the confidence score corresponding to each connected region.

[0256] In the embodiment of the application, the cosine similarity between the region feature set and the matched comprehensive support feature is calculated, and the sum of all similarity values is obtained to obtain the confidence score of the connected region . If the support set contains multiple comprehensive support features, the similarity between the region feature and each comprehensive support feature is calculated, and the highest value or the weight fusion is taken as the final .

[0257] A336, the average confidence score corresponding to each connected region is calculated using the confidence score.

[0258] In the embodiment of the application, the average confidence score corresponding to each connected region is calculated using the confidence score.

[0259] The average confidence score is the ratio of the total confidence score of a connected region to the number of pixels in the region, and is used to measure the average confidence that the region belongs to the target.

[0260] The above process can be encapsulated in the form of a formula, specifically:

[0261]

[0262] In the formula, This represents the average confidence score.

[0263] A34. Select the connected region associated with the maximum value from multiple average confidence scores as the target region.

[0264] In this embodiment of the invention, the average confidence scores of all connected regions are compared, and the connected region with the highest score is selected as the target region. This region is the segmentation region that best matches the target features of the labeled sample.

[0265] A35. Select a sub-optimized mask image associated with the target region from the initial optimized mask image as the thinning segmentation mask image.

[0266] The refined segmentation mask image is the mask image corresponding to the connected region with the highest average confidence score selected from the initial optimized mask image, and is the preliminary optimized segmentation result.

[0267] In this embodiment of the invention, a sub-optimized mask image associated with the target region determined by A34 is selected from the initial optimized mask image and used as the refined segmentation mask image. This excludes missegmented regions with low confidence levels.

[0268] A36. Using the refined segmentation mask image as the new coarse segmentation mask image, jump to the step of eroding and dilating the coarse segmentation mask image to obtain the initial optimized mask image, until the preset number of iterations is reached, then the output refined segmentation mask image is used as the segmentation result.

[0269] Iterative optimization refers to using the refined segmentation mask image as a new coarse segmentation mask, repeatedly performing steps such as erosion, dilation, and connected component filtering to gradually improve the accuracy of the segmentation results until the preset number of iterations is reached.

[0270] In this embodiment of the invention, the refined segmentation mask image is used as a new coarse segmentation mask image, and steps A31 to A35 (i.e., erosion, dilation, connected region filtering, etc.) are repeated. When the number of iterations reaches a preset value (e.g., 2 times), the final output refined segmentation mask image is the segmentation result of the image to be segmented.

[0271] The confidence map cooperation module (CMSM) of the present application generates a cooperative confidence map by fusing multi-scale features, and models the negative sample distribution, effectively improving the target region recognition accuracy. Specifically, the support features of the foreground (target) and background are extracted from the visual concept level by combining support masks, the positive confidence map is calculated using cosine similarity, the local positive confidence map is aggregated to obtain the global positive confidence map through average pooling, and the background features are extracted through the reverse mask to calculate the negative confidence map, further enhancing the model's ability to distinguish targets and backgrounds. The point selection module (PSM) sorts pixels according to the confidence score of the cooperative confidence map, filters high-discriminative positive and negative points, clusters the initially filtered high-confidence pixels through the K-Means clustering algorithm to obtain positive and negative point sets composed of centroids, and merges the two to generate a labeled point set. After inputting the point set into a pre-set segmentation model (such as SAM), a coarse segmentation mask is generated, significantly improving the accuracy and robustness of segmentation. The noise perception refinement module (NRM) removes small noise blocks in the coarse segmentation mask through morphological erosion, restores the main structure area of the target through morphological dilation, performs connected region analysis on the processed mask, extracts the semantic features of each connected region and calculates its average confidence score, selects the highest-scoring connected region as the refined segmentation mask, and inputs the refined mask and the labeled point set into the model (such as SAM) to further optimize the segmentation boundary details, and finally outputs a high-precision target segmentation mask. The present application is aimed at power line inspection scenes, and realizes high-precision, few-sample segmentation of key components of power line facilities by constructing the above modules, which has the following advantages: fully fusing global semantic features (such as DINOv2) and spatial positioning ability (such as SAM) to generate high-quality positive and negative point hints, effectively guiding the model to accurately identify targets in complex inspection scenes such as uneven lighting and severe background interference, significantly improving segmentation robustness; without relying on a large amount of labeled data or model fine-tuning, the deployment is fast, suitable for few-sample or even zero-sample inspection tasks of new production lines or new devices, reducing data labeling cost and threshold; through the negative hint mechanism, background interference is actively excluded, and the boundary details are optimized through post-processing, which not only reduces the false detection rate but also improves the defect positioning accuracy, providing an efficient, reliable and explainable solution for automated circuit inspection. Compared with traditional deep learning methods that rely on a large amount of labeled data, the present application can still guarantee high-precision segmentation under few-sample conditions, and is efficient and adaptable in deployment, providing technical support for the intelligent upgrading of power line inspection.

[0272] Please refer to Figure 2 , Figure 2 The structure block diagram of the unmanned aerial vehicle power inspection image segmentation system provided by the embodiment of the present application is shown in the figure.

[0273] The unmanned aerial vehicle power inspection image segmentation system provided by the present application comprises:

[0274] The data acquisition module 201 is configured to acquire a set of inspection images of a power line and perform preprocessing to obtain a to-be-segmented image and a set of labeled sample images.

[0275] The model acquisition module 202 is configured to acquire a preset power inspection segmentation model, wherein the preset power inspection segmentation model comprises a confidence map cooperation module, a point selection module and a noise perception refinement module connected in sequence.

[0276] The first processing module 203 is configured to input the set of labeled sample images and the to-be-segmented image into the confidence map cooperation module to perform feature fusion and confidence modeling, so as to obtain a cooperative confidence map and negative sample distribution data.

[0277] The second processing module 204 is configured to input the cooperative confidence map and the negative sample distribution data into the point selection module to perform positive-negative point screening and clustering, so as to obtain a coarse segmentation mask image.

[0278] The third processing module 205 is configured to input the coarse segmentation mask image and the to-be-segmented image into the noise perception refinement module to perform iterative mask refinement, so as to obtain a segmentation result.

[0279] Further, the feature fusion and confidence modeling specifically comprises:

[0280] Global feature extraction is performed on each labeled sample image in the set of labeled sample images to obtain a corresponding first high-level semantic feature map;

[0281] A second high-level semantic feature map of the to-be-segmented image is extracted;

[0282] A pixel set mask of each labeled sample image is extracted;

[0283] Based on the first high-level semantic feature map, the second high-level semantic feature map and the pixel set mask, a global positive confidence map is determined;

[0284] A reverse mask is generated based on the pixel set mask;

[0285] Based on the first high-level semantic feature map, the second high-level semantic feature map and the reverse mask, a global negative confidence map is determined;

[0286] Segmentation is performed on the to-be-segmented image to obtain a locally sensitive segmentation map;

[0287] The global positive confidence map is input into a preset global feature extraction model to output a global context feature map;

[0288] The locally sensitive segmentation map and the global context feature map are weighted and fused to obtain a cooperative confidence map;

[0289] Maximum likelihood estimation is performed on the global negative confidence map, and the estimation result is input into a preset Gaussian probability density function to obtain negative sample distribution data.

[0290] Further, based on the first high-level semantic feature map, the second high-level semantic feature map and the pixel set mask, a global positive confidence map is determined, comprising:

[0291] A spatial level multiplication is performed on each first high-level semantic feature map and the associated pixel set mask to obtain a plurality of support target features;

[0292] Cosine similarity between the second high-level semantic feature map and each support target feature is calculated to obtain a plurality of local positive confidence maps;

[0293] The plurality of local positive confidence maps are aggregated to obtain the global positive confidence map.

[0294] Further, based on the first high-level semantic feature map, the second high-level semantic feature map and the reverse mask, a global negative confidence map is determined, comprising:

[0295] A spatial level multiplication is performed on each first high-level semantic feature map and the associated reverse mask to obtain a plurality of background region features;

[0296] Cosine similarity between the second high-level semantic feature map and each background region feature is calculated to obtain a plurality of local negative confidence maps;

[0297] The plurality of local negative confidence maps are aggregated to obtain the global negative confidence map.

[0298] Further, the positive and negative point screening and clustering are specifically:

[0299] Based on the preset positive point screening parameter, a high confidence pixel screening number is determined;

[0300] From the collaborative confidence map, a high confidence pixel of the high confidence pixel screening number is selected as an initial positive point set;

[0301] The initial positive point set is clustered to obtain a target positive point set;

[0302] Based on the preset negative point screening parameter, a feature value screening interval is determined;

[0303] According to the feature value screening interval, a pixel of the global negative confidence map is selected from the negative sample distribution data as an initial negative point set;

[0304] According to the initial negative point set, a target negative point set is determined;

[0305] The target positive point set and the target negative point set are respectively subjected to label allocation, and the target positive point set and the target negative point set after label allocation are coupled to generate a label prompt point set;

[0306] The label prompt point set is input into the preset segmentation model to obtain a coarse segmentation mask image of the image to be segmented.

[0307] Further, the iterative mask refinement specifically comprises:

[0308] eroding and dilating the coarse segmentation mask image to obtain an initial optimized mask image;

[0309] labeling a plurality of connected regions in the initial optimized mask image;

[0310] determining average confidence scores of the connected regions;

[0311] selecting a connected region associated with a maximum value from the plurality of average confidence scores as a target region;

[0312] selecting a sub-optimized mask image associated with the target region from the initial optimized mask image as a refined segmentation mask image;

[0313] taking the refined segmentation mask image as a new coarse segmentation mask image, and jumping to execute the step of eroding and dilating the coarse segmentation mask image to obtain the initial optimized mask image until a preset number of iterations is reached, and then outputting the refined segmentation mask image as a segmentation result.

[0314] Further, the determination of the average confidence scores of the connected regions comprises:

[0315] extracting region masks corresponding to the connected regions;

[0316] performing spatial-level multiplication on the region masks and the second high-level semantic feature map to obtain a plurality of region features;

[0317] performing category classification on the plurality of support target features, and performing average pooling on the classified plurality of support target features to obtain comprehensive support features of a plurality of power component categories;

[0318] matching corresponding comprehensive support features according to the region features and the power component category;

[0319] calculating similarity between the region features and the associated comprehensive support features to obtain confidence scores corresponding to the connected regions;

[0320] calculating average confidence scores corresponding to the connected regions using the confidence scores.

[0321] Further, the initial optimized mask image comprises a plurality of sub-optimized mask images, and each sub-optimized mask image corresponds to a connected region.

[0322] Further, the data acquisition module 201 comprises:

[0323] a patrol image set submodule for acquiring a set of patrol images of the power line;

[0324] The image to be segmented processing submodule is configured to sequentially perform geometric correction, light and color optimization, image blocking, image alignment and screening on the inspection image set to obtain the image to be segmented processing.

[0325] The mask labeling submodule is configured to perform mask labeling on the inspection image set after light and color optimization.

[0326] The labeled sample image set submodule is configured to screen typical defect sample images from the inspection image set after mask labeling to construct a labeled sample image set, and the labeled sample image set is a set of only 3-5 typical defect sample images of each type of power component.

[0327] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0328] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0329] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on a plurality of network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0330] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0331] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the entire or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0332] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for segmenting images of power line inspection using unmanned aerial vehicles (UAVs), characterized in that, include: The inspection image set of power lines is acquired and preprocessed to obtain the image to be segmented and the labeled sample image set; A pre-set power inspection segmentation model is obtained, which includes a confidence map collaboration module, a point selection module, and a noise perception refinement module connected in sequence. The labeled sample image set and the image to be segmented are input into the confidence map collaboration module for feature fusion and confidence modeling to obtain a collaborative confidence map and negative sample distribution data. The collaborative confidence map and the negative sample distribution data are input into the point selection module to perform positive and negative point filtering and clustering, resulting in a coarse segmentation mask image. The coarse segmentation mask image and the image to be segmented are input into the noise-aware thinning module for iterative mask thinning to obtain the segmentation result.

2. The UAV power line inspection image segmentation method according to claim 1, characterized in that, The feature fusion and confidence modeling are specifically as follows: Global feature extraction is performed on each labeled sample image in the labeled sample image set to obtain the corresponding first high-level semantic feature map; Extract the second high-level semantic feature map of the image to be segmented; Extract the pixel set mask of each of the labeled sample images; Based on the first high-level semantic feature map, the second high-level semantic feature map, and the pixel set mask, a global positive confidence map is determined. Generate an inverse mask using the pixel set mask; Based on the first high-level semantic feature map, the second high-level semantic feature map, and the inverse mask, a global negative confidence map is determined. The image to be segmented is segmented to obtain a local sensitive segmentation map; The global positive confidence map is input into a preset global feature extraction model, which outputs a global context feature map. The local sensitive segmentation map and the global context feature map are weighted and fused to obtain a collaborative confidence map; The global negative confidence map is subjected to maximum likelihood estimation, and the estimation result is input into a preset Gaussian probability density function to obtain negative sample distribution data.

3. The UAV power line inspection image segmentation method according to claim 2, characterized in that, The step of determining the global positive confidence map based on the first high-level semantic feature map, the second high-level semantic feature map, and the pixel set mask includes: Spatial-level multiplication is performed using each of the first high-level semantic feature maps and the associated pixel set mask to obtain multiple supporting target features; Calculate the cosine similarity between the second high-level semantic feature map and each of the supporting target features to obtain multiple local positive confidence maps; The multiple local positive confidence maps are aggregated to obtain a global positive confidence map.

4. The UAV power line inspection image segmentation method according to claim 2, characterized in that, The step of determining the global negative confidence map based on the first high-level semantic feature map, the second high-level semantic feature map, and the inverse mask includes: Spatial-level multiplication is performed using each of the first high-level semantic feature maps and the associated inverse mask to obtain multiple background region features; Calculate the cosine similarity between the second high-level semantic feature map and each of the background region features to obtain multiple local negative confidence maps; The multiple local negative confidence maps are aggregated to obtain the global negative confidence map.

5. The UAV power line inspection image segmentation method according to claim 1, characterized in that, The positive and negative point screening and clustering are specifically as follows: The number of high-confidence pixels to be selected is determined based on preset on-time filtering parameters; Select the specified number of high-confidence pixels from the collaborative confidence map as the initial positive point set; Cluster the initial set of punctuality points to obtain the target set of punctuality points; Based on preset negative point filtering parameters, determine the feature value filtering range; Based on the feature value filtering interval, global negative confidence map pixels are selected from the negative sample distribution data as the initial negative point set; Based on the initial set of negative points, determine the target set of negative points; Labels are assigned to the target positive point set and the target negative point set respectively, and the labeled target positive point set and the target negative point set are coupled to generate a label prompt point set; The preset segmentation model is input using the set of label prompts to obtain a coarse segmentation mask image of the image to be segmented.

6. The UAV power line inspection image segmentation method according to claim 3, characterized in that, The iterative mask refinement specifically includes: The coarsely segmented mask image is eroded and dilated to obtain an initial optimized mask image; Mark multiple connected regions within the initial optimized mask image; Determine the average confidence score for each of the connected regions; The connected region associated with the maximum value among multiple average confidence scores is selected as the target region; A sub-optimized mask image associated with the target region is selected from the initial optimized mask image as a refined segmentation mask image; The refined segmentation mask image is used as a new coarse segmentation mask image. The process of eroding and dilating the coarse segmentation mask image to obtain an initial optimized mask image is then repeated until a preset number of iterations is reached. Finally, the output refined segmentation mask image is used as the segmentation result.

7. The UAV power line inspection image segmentation method according to claim 6, characterized in that, Determining the average confidence score of each of the connected regions includes: Extract the region mask corresponding to each of the connected regions; Multiple region features are obtained by performing spatial multiplication between the region masks and the second high-level semantic feature map; The multiple supporting target features are classified into categories, and the classified supporting target features are averaged and pooled to obtain comprehensive supporting features for multiple power component categories. Based on the regional characteristics, the corresponding comprehensive support features are matched with the power component categories associated with those features. The similarity between the regional features and the associated comprehensive support features is calculated to obtain the confidence score corresponding to each connected region; The average confidence score corresponding to each connected region is calculated using the confidence score.

8. The UAV power line inspection image segmentation method according to claim 6, characterized in that, The initial optimized mask image includes multiple sub-optimized mask images, each sub-optimized mask image corresponding to a connected region.

9. The UAV power line inspection image segmentation method according to any one of claims 1-8, characterized in that, The process of acquiring and preprocessing a set of inspection images of power lines to obtain images to be segmented and a set of labeled sample images includes: Acquire a set of inspection images of power lines; The inspection image set is sequentially subjected to geometric correction, illumination and color optimization, image segmentation, image alignment and filtering to obtain the image to be segmented; Masking is applied to the inspection image set after lighting and color optimization. A set of labeled sample images is constructed by selecting typical defect sample images from the inspection image set after masking. The set of labeled sample images is a collection of typical defect sample images of various power components containing a preset number of such images.

10. An image segmentation system for unmanned aerial vehicle (UAV) power line inspection, characterized in that, include: The data acquisition module is used to acquire and preprocess the inspection image set of power lines to obtain the image to be segmented and the labeled sample image set. The model acquisition module is used to acquire a preset power inspection segmentation model, which includes a confidence map collaboration module, a point selection module, and a noise perception refinement module connected in sequence. The first processing module is used to input the labeled sample image set and the image to be segmented into the confidence map collaboration module to perform feature fusion and confidence modeling, so as to obtain a collaborative confidence map and negative sample distribution data. The second processing module is used to input the collaborative confidence map and the negative sample distribution data into the point selection module to perform positive and negative point filtering and clustering to obtain a coarse segmentation mask image. The third processing module is used to input the coarse segmentation mask image and the image to be segmented into the noise-aware thinning module for iterative mask thinning to obtain the segmentation result.

Images