A method and system for detecting pole abnormalities based on a multi-modal large model

Abstract

The application relates to a power pole anomaly detection method and system based on a multimodal large model, which comprises the following steps: acquiring a power pole image set to be detected; based on a pre-trained PixelLM multimodal large model, adopting a phased progressive detection strategy to perform instruction-based coarse screening processing and fine judgment on the power pole image set, obtaining inference text and a multi-level mask graph corresponding to each power pole image; performing mask post-processing on the multi-level mask graph to perform mask fusion, obtaining a fused mask graph reflecting the abnormal area of the power pole; scoring the words in the inference text to obtain word scores for distinguishing the normal and abnormal state degrees, and the fused mask graph jointly reflects the final anomaly detection result. Compared with the prior art, the application has the advantages of improving detection efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of utility pole anomaly detection technology, and in particular to a utility pole anomaly detection method and system based on a multimodal large model. Background Technology

[0002] In recent years, multimodal large models have emerged as a significant technological breakthrough in the field of artificial intelligence, demonstrating powerful capabilities in visual understanding tasks. Compared to traditional unimodal models, multimodal models can simultaneously process heterogeneous data from multiple sources, such as images and text. Through mechanisms like cross-modal semantic alignment and joint feature modeling, they achieve more complex and in-depth understanding and reasoning of information. Such multimodal large models, such as Contrastive Language-Image Pre-training (CLIP), Segment Anything Model (SAM), and Pixel-Level Language Model (PixelLM), have achieved significant success in various tasks including image question answering, image-text matching, text-guided image generation, and image segmentation. Their generalization ability and cross-task transfer potential also provide new ideas for rapid adaptation to downstream specific application tasks.

[0003] However, although existing multimodal large models have shown superior performance in theory and experiments, they still generally suffer from the problem of "strong versatility but weak specialization" when facing specific application tasks, especially in fine-grained and structurally specific professional fields. For the power pole anomaly detection scenario in the power inspection system that this invention focuses on, this problem is particularly prominent. Specifically: (1) In the specific scenario of power pole anomaly detection, the general multimodal model is more susceptible to target ambiguity, small-scale interference and semantic confusion. The generated mask image may contain a large amount of noise, artifacts or irrelevant areas. If there is no effective post-processing mechanism to optimize and filter the results, the stability and reliability of the detection results will be low. (2) Unlike conventional target detection tasks, power pole anomaly detection has inherent challenges such as no fixed pattern of anomaly categories and irregular structure of anomaly areas. This makes the existing general multimodal model insufficient in effectively dealing with these challenges, which can easily lead to missed detection of anomaly targets or incorrect segmentation of target areas. (3) At present, there is a lack of concise and efficient proprietary instructions for specific tasks such as power pole anomaly detection. This results in the existing model being insufficient in accurately understanding task requirements and simultaneously completing accurate semantic judgment and anomaly area segmentation, making it difficult to generate detection results that meet professional requirements and have high positioning accuracy.

[0004] Therefore, how to effectively optimize the application methods and processing flow of existing general multimodal large models so that they can efficiently and accurately complete the specific task of detecting abnormalities in power poles in the field of power inspection has become an urgent technical challenge to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for detecting utility pole anomalies based on a multimodal large model, which improves detection efficiency and accuracy.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A method for detecting anomalies in utility poles based on a multimodal large model, comprising:

[0008] Obtain the set of images of the utility poles to be detected;

[0009] Based on the pre-trained PixelLM multimodal large model, a phased progressive detection strategy is adopted to perform instruction-based coarse screening and fine judgment on the set of utility pole images, so as to obtain the inference text and multi-level mask image corresponding to each utility pole image.

[0010] The multi-level mask image is post-processed to perform mask fusion, resulting in a fused mask image that reflects the abnormal area of ​​the utility pole.

[0011] The words in the reasoning text are scored to obtain word scores that distinguish the degree of normal and abnormal states, and these scores, together with the fused mask image, reflect the final anomaly detection result.

[0012] Furthermore, the steps for performing the coarse screening process in the coarse screening stage include:

[0013] Construct coarse screening instructions and combine them with the utility pole image set. The images are input into a pre-trained PixelLM multimodal large model to determine whether a utility pole target exists in each image and to initially locate the target region, thus obtaining a filtered set of utility pole images. ,in express The nth image of the utility pole in the image. represent The first in Image of a utility pole.

[0014] Furthermore, the steps for making the detailed judgment in the detailed judgment stage include:

[0015] Construct a composite collaborative instruction, wherein the content of the composite collaborative instruction includes determining whether there is an anomaly on the utility pole, instructing the pre-trained PixelLM multimodal large model to detect and segment the main body of the utility pole and potential abnormal objects;

[0016] The filtered set of utility pole images obtained after coarse screening Each image Each image is processed sequentially with the composite collaborative instruction input into the pre-trained PixelLM multimodal large model, and each image is output. Corresponding reasoning text and multi-level mask diagram ,

[0017] The inference text mentioned above The content includes images Understanding the content, judging whether there are any anomalies, and describing the anomalies.

[0018] The multi-level mask image These are the segmentation masks output at different decoding stages. For hierarchical indexes, The total number of layers in the layer-by-layer decoder, the multi-level mask diagram This is a floating-point probability graph, where the value of each pixel represents the probability or confidence level of belonging to a specific target region.

[0019] Furthermore, the mask post-processing steps include:

[0020] The multi-level mask image is processed using bicubic interpolation. Perform resolution unification to obtain the resolution-normalized mask. , is represented as:

[0021] ,

[0022] In the formula, This represents a bicubic interpolation operation. For hierarchical indexes, This represents the total number of layers in the layer-by-layer decoder of the pre-trained PixelLM multimodal large model;

[0023] The resolution-normalized mask is processed using morphological opening operations. Etching and expansion processes are performed sequentially to suppress noise, resulting in a noise-suppressed mask. , is represented as:

[0024] ,

[0025] In the formula, This represents the structural element used when performing corrosion and expansion operations. This structural element defines the shape and size of the neighborhood where the operation operates. The symbol represents the erosion operation. The symbol represents the expansion operation;

[0026] The mask after noise suppression using the standard Gaussian kernel. Perform convolution operations to obtain an edge-smoothing mask. , is represented as:

[0027] ,

[0028] ,

[0029] In the formula, For Gaussian kernel, Represents pixel coordinates. Represents the convolution operation. This represents the standard deviation of the Gaussian function;

[0030] The edge smoothing mask Perform small-area filtering to obtain the filtered mask. ;

[0031] Masks after filtering at each level Multi-scale mask fusion is performed using a recursive fusion method to obtain the cumulative fused mask. As an image Multi-scale fusion mask;

[0032] For the cumulative fusion mask Adaptive threshold binarization is performed to obtain a binary mask that reflects the abnormal area of ​​the utility pole. This serves as the final merged mask image.

[0033] Furthermore, the obtained filtered mask The steps include:

[0034] Edge smoothing mask for anomalies Perform connectivity analysis to identify all independent connected components marked as anomalous, and calculate the independent connected component for each anomalous component. pixel area ;

[0035] Calculate the area of ​​each pixel The ratio of the total pixel area to the total pixel area is used to obtain the corresponding area percentage, and it is then determined whether it is less than a set area percentage threshold. If not, then the abnormal independent connected region will be... If it is considered a valid region, then the abnormal independent connected region will be treated as such. These are considered invalid regions and removed to obtain the filtered mask. The small-area filtering process is represented as follows:

[0036] ,

[0037] In the formula, This represents the total pixel area of ​​the image.

[0038] Furthermore, the operational expression for the recursive fusion method is:

[0039] ,

[0040] In the formula, This is the global fusion ratio coefficient, used to control the weighting of historical fusion results and current layer information fusion. Represents the current level The specific contribution weight coefficients are used to adjust the importance of different mask levels in the final fusion result. End of representative period The cumulative fusion mask obtained from the layers, Representative before Layer accumulation fusion mask.

[0041] Furthermore, the obtained binary mask The steps include:

[0042] For images Cumulative fusion mask Calculate the statistical distribution of all pixel values, and select the 75th percentile of the statistical distribution as the image. Adaptive binarization threshold , is represented as:

[0043] ,

[0044] In the formula, This represents the quantile calculation operation;

[0045] According to the adaptive binarization threshold For images Cumulative fusion mask Perform binarization to obtain the final binary mask. , is represented as:

[0046] ,

[0047] In the formula, Represents pixel coordinates.

[0048] Furthermore, the steps for calculating the vocabulary score include:

[0049] All the inference texts are divided into normal utility pole inference texts and abnormal utility pole inference texts, and normal utility pole inference text libraries and abnormal utility pole inference text libraries are constructed respectively.

[0050] All reasoning texts in the normal utility pole reasoning text library and the abnormal utility pole reasoning text library are preprocessed to extract a candidate word set. The preprocessing operations include word segmentation, stop word removal, and word form normalization.

[0051] For each candidate word in the candidate word set The frequencies of occurrence of this element in the normal utility pole inference text database and the abnormal utility pole inference text database were calculated separately and denoted as follows: , ;

[0052] Calculate each candidate word The vocabulary score, where the calculation expression is:

[0053] ,

[0054] In the formula, Candidate words The vocabulary score, whose value range is: ,reflect The degree of relative difference in frequency of occurrence between normal sample reasoning text and abnormal sample reasoning text. It is a preset positive number, and its function is to prevent the denominator from being zero.

[0055] Furthermore, the method also includes the following steps: evaluating the mask quality of the fused mask image based on a scoring mechanism of mask activation strength to obtain a corresponding overall quality score. The specific execution steps include:

[0056] For the merged mask image ,use sigmoid The function is uniformly mapped to the interval [0,1] to obtain the standardized mask activation value. , represented as:

[0057] ,

[0058] In the formula, represent sigmoid function;

[0059] The standardized mask activation values ​​are statistically analyzed. The value exceeds the preset threshold All pixels form a set of highly activated pixels. , represented as:

[0060] ,

[0061] Calculate the set of highly active pixels The average activation value of all pixels in the image is used as the fused mask. The overall quality score is expressed as:

[0062] ,

[0063] In the formula, Score the overall quality.

[0064] This invention also provides a utility pole anomaly detection system based on a multimodal large model, comprising:

[0065] Image acquisition module: used to acquire a set of images of the utility poles to be detected;

[0066] The phased detection module is used to perform instruction-based coarse screening and fine judgment on the set of utility pole images based on the pre-trained PixelLM multimodal large model, and to obtain the inference text and multi-level mask image corresponding to each utility pole image.

[0067] Post-processing module: used to perform mask post-processing on the multi-level mask image to perform mask fusion and obtain a fused mask image that reflects the abnormal area of ​​the utility pole;

[0068] The vocabulary scoring module is used to score the words in the reasoning text to obtain vocabulary scores that distinguish the degree of normal and abnormal states, and together with the fused mask image, it reflects the anomaly detection results.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] (1) Phased progressive detection improves detection efficiency and accuracy: This invention significantly improves the overall efficiency and positioning accuracy of utility pole anomaly detection by combining coarse screening to quickly filter out irrelevant images and fine judgment with a composite collaborative instruction. The coarse screening process uses simple instructions to quickly eliminate irrelevant images, effectively reducing the computational load; the fine judgment stage guides the model to simultaneously complete the semantic understanding and pixel-level segmentation of anomalies, ensuring the comprehensiveness and accuracy of the detection results.

[0071] (2) Multi-dimensional mask post-processing enhances mask quality and reliability: This invention effectively eliminates noise, isolated points, and artifacts that may exist in the original mask by integrating basic post-processing steps such as unified resolution adjustment, morphological operation denoising, Gaussian blur edge smoothing, and small-area connected region filtering. It also smooths the mask edges and filters small-area false alarm regions that have no practical significance. On this basis, through multi-scale mask weighted fusion, the feature information output by the model decoder at different levels is fully utilized and integrated, so that the final generated anomaly mask can not only retain the low-level details such as the edges and textures of the anomaly region, but also take into account the high-level global information such as the overall contour and semantics of the anomaly region, thereby significantly improving the accuracy of anomaly region localization and the overall quality and interpretability of the mask. Furthermore, by combining the adaptive threshold binarization method, the segmentation threshold is dynamically adjusted according to the actual distribution of the mask of each image, overcoming the problem of poor adaptability of fixed thresholds under different lighting, backgrounds, and anomaly types. This ensures that binary masks can be generated stably and accurately in various complex scenes, improving the robustness of the detection results.

[0072] (3) Vocabulary scoring to enhance semantic understanding and discrimination capabilities: This invention also performs vocabulary scoring, quantifying and mining keywords highly related to the abnormal state of utility poles by comparing and analyzing the reasoning texts of normal and abnormal samples. This enhances the model's deep understanding of abnormal semantics, provides reliable linguistic clues for automatic anomaly detection, and improves the accuracy of anomaly identification.

[0073] (4) An objective scoring mechanism based on mask activation intensity to achieve quantitative evaluation of mask quality: This invention proposes a scoring method based on the average activation intensity of high-activation regions in the mask, which achieves an objective quantitative evaluation of the mask quality and significance of the model output. This mechanism helps to automatically screen high-quality detection results, improves the reliability and consistency of the final detection, and reduces the cost of manual screening. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0075] Figure 2 This is a schematic diagram illustrating the phased, progressive detection effect of the present invention;

[0076] Figure 3 The following are example figures illustrating the performance of the present invention for four typical image types;

[0077] Figure 4 This is a schematic diagram illustrating the design and effect enhancement of the multi-dimensional mask post-processing method of the present invention. Detailed Implementation

[0078] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0079] Example 1.

[0080] This embodiment provides a method for detecting anomalies in utility poles based on a multimodal large model, such as... Figure 1 As shown, the method includes the following steps:

[0081] Step 1: First, obtain the original set of utility pole images, denoted as... , in, represent The nth image. Obtain the pre-trained PixelLM multimodal large model as the base model for subsequent optimization and applications.

[0082] Step 2: To effectively improve the processing speed and inference stability of the PixelLM multimodal large model in the utility pole anomaly detection task, this embodiment of the invention proposes a phased progressive detection strategy. This strategy divides the overall detection process into two main stages: a coarse screening stage and a fine judgment stage, as described in the reference... Figure 2 .

[0083] This embodiment selects the PixelLM multimodal large model as the basis for optimization and application, mainly based on the following key characteristics of this model:

[0084] It supports guiding the model to focus on specific target regions or semantic features in an image that are relevant to the task through natural language instructions. In anomaly detection, it can help the model focus on potential anomaly locations through task-customized text descriptions.

[0085] It possesses pixel-level reasoning capabilities, enabling it to directly generate semantic masks to accurately locate the spatial extent of abnormal regions. Moreover, this process does not rely on fixed abnormal category labels, thus providing the necessary versatility and flexibility for detecting diverse and difficult-to-predefine utility pole anomalies.

[0086] (1) Entering the coarse screening stage:

[0087] The PixelLM multimodal large model receives raw utility pole image sets. The task instruction is concise ("Detect the telephone pole in the diagram, please segment it."). The model then processes the image set according to this instruction. Each image in the process is processed quickly to determine whether a utility pole target exists within the image and to initially locate or focus on the target area. The core objective of this stage is to quickly eliminate image samples irrelevant to the utility pole anomaly detection task.

[0088] Original set of utility pole images After the initial screening stage, all image samples that were preliminarily identified as containing utility pole targets and retained constitute the filtered image set, denoted as . Assuming that after coarse sieving, the remaining... Zhang image samples, then ,in, Representative of the filtered image set The first in [Number of pictures]

[0089] (2) Entering the detailed judgment stage:

[0090] It should be noted that in the PixelLM multimodal large model architecture adopted in this invention, text instructions play a decisive role in guiding the model to effectively perform visual reasoning and anomaly region localization tasks. Specifically, in the fine-grained judgment stage of utility pole anomalies, the model needs to collaboratively complete two key sub-tasks: semantic-level anomaly interpretation and pixel-level segmentation of the anomaly region and the utility pole itself. Therefore, the construction method of the text instructions not only affects the model's attention allocation to potential anomaly regions in the image, but also directly relates to whether it can simultaneously and effectively achieve the above two sub-tasks.

[0091] Conventional single-purpose text instructions, such as questions that only require description of anomalies or instructions that focus solely on target segmentation, exhibit significant limitations when applied to such dual-task scenarios. If the instruction emphasizes semantic description, the model may generate relatively complete natural language inference results, but the quality of the generated image segmentation mask may be insufficient or incomplete; conversely, if the instruction overemphasizes image segmentation, the model may output a segmentation mask, but its accompanying semantic interpretation may be insufficient or inaccurate.

[0092] To overcome the above limitations, this invention proposes an optimized text instruction structure, specifically a composite collaborative instruction. A preferred embodiment uses the following English instruction format: "Is there anything unusual on the telephone pole? Detect anomalous objects and the telephone pole respectively in the diagram, please segment them." (The Chinese functional interpretation is: First, it queries whether there are any abnormalities on the telephone pole, then the instruction model detects and segments the abnormal objects and the telephone pole itself in the diagram.)

[0093] The advantages of the composite collaborative instruction are as follows: its first clause clearly defines the objectives of anomaly recognition and semantic judgment, guiding the model to perform state evaluation and text generation; its second clause clearly assigns the detection and pixel-level segmentation tasks for multiple targets (i.e., the utility pole itself and potential abnormal objects). This structured instruction design effectively enables the multimodal model to generate inference text with clear semantic references and simultaneously output high-quality image masks covering relevant target regions. Adopting this optimized instruction structure significantly improves the consistency of the model's multimodal information output in utility pole anomaly detection applications and enhances its overall quality and reliability in completing complex detection tasks.

[0094] Filtered image set Samples in Each input of the composite task command is processed in turn with the PixelLM multimodal large model.

[0095] The output of the PixelLM multimodal large model after processing each input sample mainly includes two parts:

[0096] a) Reasoning text The model is designed for the current input sample. The natural language description generated after completing the anomaly detection task. This text typically contains an understanding of the image content, a judgment on the presence of anomalies, and a description of the anomalies, such as judgmental or descriptive words or statements like "there is an anomaly," "no obvious anomaly," or "there is a black object above the utility pole."

[0097] b) Mask diagram of multi-level output The decoder of the model is for the current input sample. This refers to a series of segmentation masks output at different decoding stages within the system. Specifically, for the current input sample... , The first part of the model decoder represents the... The segmentation mask output by the layer. For hierarchical indexes, , This represents the total number of decoder layers in the PixelLM multimodal large model. The mask is represented as a floating-point probability map, where the value of each pixel represents the probability or confidence that it belongs to a specific target region.

[0098] Step 3: The multi-level mask diagram output by the decoder in this embodiment of the invention. Based on this, a multi-level masking post-processing operation is introduced to fully utilize the multi-level masking information, effectively improving the accuracy and quality of the abnormal region mask. The specific steps are as follows:

[0099] a) Unified resolution: The PixelLM model adopts a layer-by-layer decoder structure, generating corresponding mask outputs at different resolution levels (i.e., ...). ,in For hierarchical indexes, For sample index, , Because the details and semantic information exhibited by the abnormal regions of utility poles vary at different scales, the abnormal information captured by the multiple masks generated by the decoder at different levels is highly complementary and rich.

[0100] To facilitate the full utilization of this multi-layered, multi-scale information in subsequent masking processing, this invention first uses a unified upsampling method (preferably bicubic interpolation to better preserve image details) to mask the outputs of all layers. Adjust to the preset target output resolution:

[0101] ,

[0102] in, This represents a bicubic interpolation operation. Represents the original level mask The output mask is obtained after resolution normalization.

[0103] b) Noise suppression: During the actual inference process, the PixelLM model generates a mask... The presence of isolated noisy pixels leads to variations in the corresponding mask after uniform resolution. The same problem exists. It not only affects the final visualization of the mask, but may also lead to subsequent misreading or misjudgment of abnormal areas. To effectively suppress mask noise, this invention employs morphological opening operations on the mask after unifying resolution. An etching process is performed, followed by an expansion operation on the result of the etching. Specifically:

[0104] ,

[0105] in, This represents the structural element used when performing corrosion and expansion operations, which defines the shape and size of the neighborhood where the operation operates; The symbol represents the erosion operation. The symbol represents the expansion operation.

[0106] This morphological opening operation step can effectively remove small isolated spots and noise points inside the mask, while better preserving the main structural outline and shape features of the abnormal area, avoiding excessive damage to the overall mask.

[0107] c) Edge Smoothing: To improve or eliminate possible pseudo-edge effects in the mask, embodiments of the present invention employ Gaussian blurring on the mask. (of which is) Hierarchical index, For sample index, , The mask is then smoothed. Specifically, this process uses a standard Gaussian kernel to smooth the mask. Perform convolution operations to smooth the mask edges:

[0108] ,

[0109] ,

[0110] in, Represents pixel coordinates. Table of standard deviation of Gaussian function For Gaussian kernel, Represents the convolution operation. The representative uses the Gaussian kernel to match the mask. The edge-smoothing mask obtained after performing a convolution operation.

[0111] By performing this operation, the transition of the mask's boundary portions can be made more natural and smooth, thereby significantly suppressing the false edge effects that may be introduced by high-frequency noise and interpolation operations, further improving the visual quality of the mask and the accuracy of subsequent analysis.

[0112] d) Small Region Filtering: In anomaly detection tasks, the model may incorrectly label small regions as anomalies in the image background or non-critical areas. These small region labels typically lack actual semantic meaning and are often misjudgments or noise generated when the model captures subtle features, affecting detection accuracy and increasing the complexity of subsequent processing. To reduce such false positives, this embodiment of the invention introduces a small region filtering strategy based on connected component analysis. Specifically:

[0113] First, the anomaly mask after edge smoothing. Perform connectivity analysis to identify all independent connected regions that are marked as "anomalies".

[0114] Calculate each independent connected component (denoted as the i-th component). pixel area of ​​each region . This area With the total pixel area of ​​the image By comparison, the area proportion of the abnormal region can be obtained.

[0115] Set an area percentage threshold If the area of ​​a connected region is less than this threshold, the region is considered invalid and removed from the mask. Remove from the middle to obtain the final filtered mask. This filtering process can be represented as:

[0116] ,

[0117] in, represent The Middle The pixel area of ​​each independent connected abnormal region Represents the total pixel area of ​​the currently processed image. This represents the preset area percentage filtering threshold.

[0118] This filtering mechanism automatically ignores abnormal regions that are too small and may be noise, making the final output anomaly mask more focused on important anomaly regions with significant semantic meaning, thereby effectively reducing the false alarm rate and improving the practicality and reliability of the anomaly mask.

[0119] e) Multi-scale mask fusion: To fully utilize the multi-level, multi-scale information generated by the PixelLM model decoder at different resolution levels, this invention example obtains the sample... The output masks of each layer after the aforementioned optimization steps (resolution normalization, noise suppression, edge smoothing, and small region filtering) Subsequently, multi-scale mask fusion operations were introduced.

[0120] Specifically: the optimized mask for each layer (in, , Weighted fusion is performed for the total number of decoder layers. A preferred recursive fusion method is to perform weighted fusion for each layer. Iterative processing from 1 to L can be represented by the following formula:

[0121] ,

[0122] in, This is the global fusion ratio coefficient, used to control the weight of fusion between historical fusion results and current layer information; Represents the current level The specific contribution weight coefficients are used to adjust the importance of different levels of masks in the final fusion result; End of representative period The cumulative fusion mask obtained from the layers, Representative before Layer accumulation fusion mask, For hierarchical indexes, This represents the total number of layers in the decoder.

[0123] After completing all The final cumulative fusion mask obtained after iterative fusion at each level. That is, as a sample Multi-scale fusion mask output.

[0124] By employing the multi-scale mask fusion method described above, feature information from different levels of the decoder can be effectively integrated. This allows for the full utilization of detailed information such as edges and textures of abnormal regions contained in lower-level feature information, while also effectively preserving global information such as the overall contour and semantics of abnormal regions contained in higher-level feature information. The resulting fused mask exhibits significantly improved accuracy in detail representation and coherence in semantic information.

[0125] f) Adaptive Threshold Binarization: The original mask generated by the PixelLM multimodal large model is essentially a floating-point probability map, which represents the probability or confidence that each pixel in the image belongs to the target region (such as an anomaly on a utility pole). Directly using a fixed global threshold to binarize this floating-point probability map to obtain the final binary mask has certain limitations. Specifically, in different utility pole inspection image samples, due to differences in lighting conditions, shooting distance, anomaly type and degree, background complexity, etc., the response intensity and numerical distribution of the anomaly region in the mask image may vary significantly. Using a fixed threshold may lead to over-identification of anomaly regions in some samples (i.e., misjudging some background noise or irrelevant areas as anomalies), or under-identification (i.e., missing some real, relatively weak anomaly regions), thus affecting the accuracy of the final generated mask and the consistency of cross-sample results, thereby reducing the overall stability of the anomaly detection effect.

[0126] To overcome the shortcomings of the fixed threshold method mentioned above, this invention introduces an adaptive binarization strategy based on statistical quantiles. The core idea of ​​this strategy is to dynamically adjust the binarization threshold according to the actual distribution of abnormal response signals in the fused mask image of each image. The specific implementation steps are as follows:

[0127] First, regarding the samples currently to be processed... fusion mask image The statistical distribution of all pixel values ​​is calculated. Then, the 75th percentile of this distribution is selected as the adaptive binarization threshold for the current sample. :

[0128] ,

[0129] in, This represents the quantile calculation operation.

[0130] Furthermore, based on this threshold Fusion mask image of the sample Binarization is performed to generate the final binary mask. :

[0131] ,

[0132] in, Represents pixel coordinates.

[0133] By employing this adaptive threshold binarization method based on statistical quantiles, the binarization process becomes more adaptable to differences in the intensity and distribution of anomalous signals across different images. This method effectively avoids the problem of inconsistent binarization results (such as oversegmentation or undersegmentation) on some images that may be caused by improper selection of a fixed threshold, thereby significantly improving the stability and robustness of the mask segmentation process. This allows the model to more accurately and robustly identify and delineate anomalous regions on utility poles.

[0134] The above-mentioned masking post-processing identifies the locations of utility poles and abnormal areas in the image, which are then marked on the image.

[0135] Step 4: For each input sample of the PixelLM multimodal large model The generated reasoning text This invention establishes a vocabulary scoring method. This method aims to mine and quantify key words that exhibit significant distribution differences in the inference text between normal and abnormal utility pole image data. By comparing and analyzing the text output of these two datasets, and utilizing word frequency statistics and dissimilarity calculation, it identifies words with strong semantic discriminative power, thereby providing more reliable linguistic clues for determining abnormal states. The specific construction steps are as follows:

[0136] (1) Obtain a normal sample set containing only images of utility poles in normal condition, denoted as B; and simultaneously, obtain an abnormal sample set containing images of utility poles in various known abnormal conditions, denoted as C. Use the PixelLM multimodal model to perform inference on all images in these two sample sets, and collect the natural language description text generated by the model for each image to form a normal utility pole inference text library and an abnormal utility pole inference text library, respectively.

[0137] (2) Perform preprocessing operations on all texts in the two reasoning text libraries. These operations may include, but are not limited to, word segmentation, stop word removal, and word form normalization. After preprocessing, extract a candidate word set from the text.

[0138] (3) For reasoning texts Each candidate word The frequency of each occurrence in the normal telephone pole inference text database was statistically analyzed and denoted as . And its frequency of occurrence in the abnormal utility pole inference text database, denoted as .

[0139] (4) In order to measure each candidate word The ability or importance of distinguishing between normal and abnormal utility pole conditions is scored using the following formula, denoted as follows: :

[0140] ,

[0141] in, It is a preset, extremely small positive number, the purpose of which is to prevent the denominator from being zero. The range of values ​​is .

[0142] This score reflects vocabulary The relative difference in frequency of occurrence between normal and abnormal sample inference texts. The larger the absolute value of the score, the more significant the role of the word in distinguishing between normal and abnormal states. Specifically, a large positive score indicates that the word is more likely to appear in descriptions of normal states, while a large negative score (large absolute value) indicates that the word is more likely to appear in descriptions of abnormal states.

[0143] By using a word scoring mechanism, the reasoning text contains many sentences and possible situations, such as "There is a black object on the telephone pole". The word "black object" is scored. If the words "black object" appear frequently in the reasoning text of the abnormal image, then "black object" is more likely to be abnormal, thereby improving the accuracy of anomaly detection.

[0144] Step 5: During anomaly detection, multiple masks may be generated by different models or the same model under different parameter configurations. To objectively and effectively evaluate the quality of each mask, this embodiment of the invention proposes a scoring mechanism based on mask activation strength. This mechanism aims to quantify the concentration and salience of target responses in the mask, thereby quickly filtering out more reliable detection results for subsequent processing. The specific steps are as follows:

[0145] (1) Floating-point anomaly region mask map directly generated for multimodal large models (Its element value is) , representing pixel coordinates (mask value at the location), application sigmoid The function maps its values ​​uniformly to the interval [0,1]. This step aims to standardize the response scale between different masks, obtaining the standardized mask activation values. This process can be represented by the following formula:

[0146] ,

[0147] in, represent sigmoid function.

[0148] (2) Count the activation values ​​of all standardized masks. Exceeding the preset threshold The pixels form a set of highly activated pixels. :

[0149] ,

[0150] in, This represents the preset activation threshold.

[0151] (3) Calculate the set The average activation value of all pixels in the mask is used as the current mask. Overall quality rating:

[0152] ,

[0153] in, Representative set The total number of elements in the array.

[0154] This scoring mechanism effectively reflects the concentration and intensity of the mask's response to anomaly regions by statistically analyzing the average activation intensity of high-confidence response pixels in the mask. A higher-scoring mask typically indicates a strong and concentrated response in important anomaly regions, representing a more reliable detection result; conversely, a lower-scoring mask may have a scattered response or a weak signal. This mechanism helps to automatically select higher-quality mask outputs without manual intervention, thereby improving the stability and accuracy of the overall anomaly detection process.

[0155] To verify the beneficial effects of this invention, experimental verification was conducted. This invention uses a custom-designed test set for systematic evaluation of model performance and ablation experiment analysis. The test set contains 160 high-quality image samples, covering typical scenarios of utility pole targets in various environments and states, specifically divided into three categories: images without target utility poles, images with normal utility poles, and images with utility poles and anomalies (further subdivided into obvious anomalies and inconspicuous anomalies).

[0156] It should be noted that "not obvious abnormality" mainly refers to the abnormal area accounting for a small proportion (less than 3%) of the entire image.

[0157] To comprehensively evaluate model performance, four metrics—accuracy, precision, recall, and F1-score—are used for evaluation. These metrics reflect the model's overall classification accuracy, false positive control, false negative control, and overall detection capability, respectively. Accuracy measures the model's overall correctness in classifying all samples; precision focuses on the suppression of false positives in anomaly detection; recall emphasizes the model's coverage of anomaly samples; and the F1-score, as the harmonic mean of precision and recall, comprehensively assesses the model's balance between accuracy and coverage, reflecting the stability and reliability of its overall detection performance.

[0158] Referring to Table 1 below, the method of the present invention exhibits excellent performance across different test set categories:

[0159] 1. The model achieved extremely high accuracy (accuracy 1) on both samples without target utility poles and samples with normal utility poles, indicating that the method of the present invention can effectively identify situations with no target and normal targets, and has strong false alarm control capabilities.

[0160] 2. For samples containing abnormal utility poles, whether the abnormality is obvious or not, the model exhibits high detection performance. For example, the F1-score reaches 0.8947 on samples with obvious abnormalities and 0.8387 on samples with indistinct abnormalities, demonstrating that the method has good detection capability and stability for abnormalities of different degrees.

[0161] 3. On the overall test set, the method of the present invention also achieved good comprehensive performance indicators (such as accuracy of 0.8736 and F1-score of 0.8750), further verifying the overall effectiveness of the various measures described in the present invention in improving detection efficiency and accuracy.

[0162] Table 1 Overall performance test results

[0163]

[0164] The above results demonstrate that the present invention, through a phased, progressive detection strategy combined with effective multi-dimensional masking post-processing technology, exhibits excellent false alarm control capabilities in both target-free and normal target scenarios. Furthermore, it achieves superior overall detection performance on various abnormal utility pole samples, proving that the proposed method possesses strong anomaly sensitivity, stability, and a good balance between accuracy and coverage.

[0165] Reference Figure 3 The PixelLM multimodal large model can provide reasonable masks and language interpretations based on text instructions for different scenarios such as no utility poles, normal utility poles, obvious anomalies, and inconspicuous anomalies. It demonstrates strong multi-scenario adaptability and semantic understanding level, and verifies that the present invention effectively improves the efficiency and accuracy of anomaly detection through a phased progressive detection strategy, composite collaborative instructions, and a refined mask post-processing process.

[0166] To further verify the effectiveness of the multi-dimensional mask post-processing module in this invention, it was compared with the original PixelLM model without mask post-processing. The experiment focused on the model's ability to locate abnormal regions, control misjudgments of normal structures, and overall detection accuracy and consistency. A unified test set and text instructions were used to ensure comparability. Specific ablation experiment results are shown in Table 2 below. Figure 4 .

[0167] Table 2 Ablation experiment results of the multi-dimensional mask post-processing method

[0168]

[0169] Experiments show that the original model often suffers from problems such as discontinuous response regions and numerous noise fragments during mask generation. (Refer to...) Figure 4The anomaly detection mask, after multi-dimensional masking post-processing, exhibits more stable performance, clearer boundaries, and a more consistent overall structure. Referring to Table 2, the multi-dimensional masking post-processing strategy proposed in this invention effectively reduces the false alarm rate while maintaining strong anomaly detection capabilities, significantly enhancing the model's ability to discriminate anomaly regions. By integrating post-processing steps such as unified resolution, denoising, edge smoothing, small-region filtering, multi-scale fusion, and adaptive threshold binarization, the accuracy of anomaly region localization and the overall quality and reliability of the mask are effectively improved, thereby enhancing the robustness of the detection results.

[0170] Example 2

[0171] This embodiment provides a utility pole anomaly detection system based on a multimodal large model, including:

[0172] Image acquisition module: used to acquire a set of images of the utility poles to be detected;

[0173] The phased detection module is used to perform coarse screening and fine judgment on the set of utility pole images based on the pre-trained PixelLM multimodal large model, and to obtain the inference text and multi-level mask image corresponding to each utility pole image.

[0174] Post-processing module: used to perform mask post-processing on the multi-level mask image to perform mask fusion and obtain a fused mask image that reflects the abnormal area of ​​the utility pole;

[0175] The vocabulary scoring module is used to score the words in the reasoning text to obtain vocabulary scores that distinguish the degree of normal and abnormal states, and together with the fused mask image, it reflects the anomaly detection results.

[0176] The rest are as in Example 1.

[0177] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0179] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0182] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0183] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for detecting anomalies in utility poles based on a multimodal large model, characterized in that, include: Obtain the set of images of the utility poles to be detected; Based on the pre-trained PixelLM multimodal large model, a phased progressive detection strategy is adopted to perform instruction-based coarse screening and fine judgment on the set of utility pole images, so as to obtain the inference text and multi-level mask image corresponding to each utility pole image. The multi-level mask image is post-processed to perform mask fusion, resulting in a fused mask image that reflects the abnormal area of ​​the utility pole. The words in the reasoning text are scored to obtain word scores that distinguish the degree of normal and abnormal states, and these scores, together with the fused mask image, reflect the final anomaly detection result. The steps for making the fine judgment in the fine judgment stage include: Construct a composite collaborative instruction, wherein the content of the composite collaborative instruction includes determining whether there is an anomaly on the utility pole, instructing the pre-trained PixelLM multimodal large model to detect and segment the main body of the utility pole and potential abnormal objects; The filtered set of utility pole images obtained after coarse screening Each image Each image is processed sequentially with the composite collaborative instruction input into the pre-trained PixelLM multimodal large model, and each image is output. Corresponding reasoning text and multi-level mask diagram , The inference text mentioned above The content includes images Understanding the content, judging whether there are any anomalies, and describing the anomalies. The multi-level mask image These are the segmentation masks output at different decoding stages. For hierarchical indexes, The total number of layers in the layer-by-layer decoder, the multi-level mask diagram This is a floating-point probability graph, where the value of each pixel represents the probability or confidence level of belonging to a specific target region.

2. The method for detecting anomalies in utility poles based on a multimodal large model according to claim 1, characterized in that, The steps for performing the coarse screening process in the coarse screening stage include: Construct coarse screening instructions and combine them with the utility pole image set. The images are input into a pre-trained PixelLM multimodal large model to determine whether a utility pole target exists in each image and to initially locate the target region, thus obtaining a filtered set of utility pole images. ,in express The nth image of the utility pole in the image. represent The first in Image of a utility pole.

3. The method for detecting anomalies in utility poles based on a multimodal large model according to claim 1, characterized in that, The mask post-processing steps include: The multi-level mask image is processed using bicubic interpolation. Perform resolution unification to obtain the resolution-normalized mask. , is represented as: ， In the formula, This represents a bicubic interpolation operation. For hierarchical indexes, This represents the total number of layers in the layer-by-layer decoder of the pre-trained PixelLM multimodal large model; The resolution-normalized mask is processed using morphological opening operations. Etching and expansion processes are performed sequentially to suppress noise, resulting in a noise-suppressed mask. , is represented as: ， In the formula, This represents the structural element used when performing corrosion and expansion operations. This structural element defines the shape and size of the neighborhood where the operation operates. The symbol represents the erosion operation. The symbol represents the expansion operation; The mask after noise suppression using the standard Gaussian kernel. Perform convolution operations to obtain an edge-smoothing mask. , is represented as: ， ， In the formula, For Gaussian kernel, Represents pixel coordinates. Represents the convolution operation. This represents the standard deviation of the Gaussian function; The edge smoothing mask Perform small-area filtering to obtain the filtered mask. ; Masks after filtering at each level Multi-scale mask fusion is performed using a recursive fusion method to obtain the cumulative fused mask. As an image Multi-scale fusion mask; For the cumulative fusion mask Adaptive threshold binarization is performed to obtain a binary mask that reflects the abnormal area of ​​the utility pole. This serves as the final merged mask image.

4. The method for detecting anomalies in utility poles based on a multimodal large model according to claim 3, characterized in that, The obtained filtered mask The steps include: Edge smoothing mask for anomalies Perform connectivity analysis to identify all independent connected components marked as anomalous, and calculate the independent connected component for each anomalous component. pixel area ; Calculate the area of ​​each pixel The ratio of the total pixel area to the total pixel area is used to obtain the corresponding area percentage, and it is then determined whether it is less than a set area percentage threshold. If not, then the abnormal independent connected region will be... If it is considered a valid region, then the abnormal independent connected region will be treated as such. These are considered invalid regions and removed to obtain the filtered mask. The small-area filtering process is represented as follows: ， In the formula, This represents the total pixel area of ​​the image.

5. The method for detecting anomalies in utility poles based on a multimodal large model according to claim 3, characterized in that, The computational expression for the recursive fusion method is: ， In the formula, This is the global fusion ratio coefficient, used to control the weighting of historical fusion results and current layer information fusion. Represents the current level The specific contribution weight coefficients are used to adjust the importance of different levels of masks in the final fusion result. End of representative period The cumulative fusion mask obtained from the layers, Representative before Layer accumulation fusion mask.

6. The method for detecting anomalies in utility poles based on a multimodal large model according to claim 3, characterized in that, The obtained binary mask The steps include: For images Cumulative fusion mask Calculate the statistical distribution of all pixel values, and select the 75th percentile of the statistical distribution as the image. Adaptive binarization threshold , is represented as: ， In the formula, This represents the quantile calculation operation; According to the adaptive binarization threshold For images Cumulative fusion mask Perform binarization to obtain the final binary mask. , is represented as: ， In the formula, Represents pixel coordinates.

7. The method for detecting anomalies in utility poles based on a multimodal large model according to claim 1, characterized in that, The steps for calculating the vocabulary score include: All the inference texts are divided into normal utility pole inference texts and abnormal utility pole inference texts, and normal utility pole inference text libraries and abnormal utility pole inference text libraries are constructed respectively. All reasoning texts in the normal utility pole reasoning text library and the abnormal utility pole reasoning text library are preprocessed to extract a candidate word set. The preprocessing operations include word segmentation, stop word removal, and word form normalization. For each candidate word in the candidate word set The frequencies of occurrence of this element in the normal utility pole inference text database and the abnormal utility pole inference text database were calculated separately and denoted as follows: , ; Calculate each candidate word The vocabulary score, where the calculation expression is: ， In the formula, Candidate words The vocabulary score, whose value range is: ,reflect The degree of relative difference in frequency of occurrence between normal sample reasoning text and abnormal sample reasoning text. It is a preset positive number, and its function is to prevent the denominator from being zero.

8. The method for detecting anomalies in utility poles based on a multimodal large model according to claim 1, characterized in that, It also includes the following steps: The fused mask image is evaluated based on a scoring mechanism for mask activation strength to obtain a corresponding overall quality score. The specific steps include: For the merged mask image ,use sigmoid The function is uniformly mapped to the interval [0,1] to obtain the standardized mask activation value. , is represented as: ， In the formula, represent sigmoid function; The standardized mask activation values ​​are statistically analyzed. The value exceeds the preset threshold All pixels form a set of highly activated pixels. , is represented as: ， Calculate the set of highly active pixels The average activation value of all pixels in the image is used as the fused mask. The overall quality score is expressed as: ， In the formula, Score the overall quality.

9. A utility pole anomaly detection system based on a multimodal large model, characterized in that, include: Image acquisition module: used to acquire a set of images of the utility poles to be detected; The phased detection module is used to perform instruction-based coarse screening and fine judgment on the set of utility pole images based on the pre-trained PixelLM multimodal large model, and to obtain the inference text and multi-level mask image corresponding to each utility pole image. Post-processing module: used to perform mask post-processing on the multi-level mask image to perform mask fusion and obtain a fused mask image that reflects the abnormal area of ​​the utility pole; The vocabulary scoring module is used to score the words in the reasoning text to obtain vocabulary scores that distinguish the degree of normal and abnormal states, and together with the fused mask image, it reflects the anomaly detection results. The steps for making the fine judgment in the fine judgment stage include: Construct a composite collaborative instruction, wherein the content of the composite collaborative instruction includes determining whether there is an anomaly on the utility pole, instructing the pre-trained PixelLM multimodal large model to detect and segment the main body of the utility pole and potential abnormal objects; The filtered set of utility pole images obtained after coarse screening Each image Each image is processed sequentially with the composite collaborative instruction input into the pre-trained PixelLM multimodal large model, and each image is output. Corresponding reasoning text and multi-level mask diagram , The inference text mentioned above The content includes images Understanding the content, judging whether there are any anomalies, and describing the anomalies. The multi-level mask image These are the segmentation masks output at different decoding stages. For hierarchical indexes, The total number of layers in the layer-by-layer decoder, the multi-level mask diagram This is a floating-point probability graph, where the value of each pixel represents the probability or confidence level of belonging to a specific target region.

Images