A method and system for image annotation with multi-level association of power transmission and transformation equipment

Abstract

The present invention discloses a method and system for multi-level associated image annotation of power transmission and transformation equipment. The method includes: identifying sample images including defects of power transmission and transformation equipment based on an image quality recognition standard, and identifying the sample images that meet the image quality standard; based on the correspondence between the equipment ledger and the equipment location information of the power transmission and transformation equipment in the sample images, annotating the basic attributes of the sample images to attribute tags, where the basic attributes include the acquisition terminal that provides the sample images; establishing multi-level tags for the power transmission and transformation equipment, and respectively annotating each level of tags in the multi-level tags of the power transmission and transformation equipment in the sample images; each level of tag has a subordinate relationship with its corresponding upper-level tag: starting from the highest-level tag, annotating down to the next lower-level tag in sequence to generate multi-level tags.

Description

Technical Field

[0001] The present invention relates to the technical field of power equipment monitoring, and more specifically, to a method and system for annotating images of multi-level associations of power transmission and transformation equipment. Background Art

[0002] At present, online intelligent inspection systems have been applied in substations, centralized control stations, converter stations and other scenarios. At the same time, channel monitoring and drone inspections have been widely used on transmission lines. According to the typical primary defects listed in the State Grid Corporation's enterprise standard Q / GDW1906-2013 "Classification Standard for Defects of Primary Transmission and Transformation Equipment", more than 50% of the defects are visual defects. Among them, the artificial intelligence method based on image and single-frame video target detection has also achieved certain results, which can identify the status, defects and risks of 25 sub-categories of substation equipment in 3 categories, including: meter blur, dial damage, shell damage, insulator cracks, insulator rupture, oil stains on the surface of components, oil stains on the ground, metal rust, silicone barrel damage, abnormal box door closure, hanging suspended objects, bird nests, doors, windows, walls and floors damaged, cover damage, frame ladder unlocked, surface dirt, crossing the line, not wearing a helmet, not wearing work clothes, smoking, abnormal meter readings, abnormal oil level of respirator oil seal, silicone discoloration, pressure plate closing, pressure plate separation. It can identify defects in 9 categories and 54 subcategories of power transmission equipment, including: rust of tower materials above the flat top, rust of tower materials below the flat top, missing bolts, foreign objects in the tower body, missing tower materials, and tower material deformation. The images of power transmission and transformation equipment have the following problems:

[0003] In the existing technology, due to the uneven image quality of power transmission and transformation equipment defect images collected from different channels, the image quality used for intelligent diagnosis of power transmission and transformation equipment defect images is uneven. There are a large number of image samples with low image pixels, blurred images, re-shot images and secondary processing, which have a great impact on the training effect of the intelligent diagnosis model.

[0004] The features of images (single-frame videos) acquired by different acquisition terminals in the prior art vary greatly. Indiscriminate labeling of images acquired by different acquisition terminals will lead to poor model training results. The reason is that there are large differences in resolution, viewing angle, and environment between different terminals, which will dilute the target features during model training. For example, the viewing angle of patrol robots is mostly horizontal or upward, while the viewing angle of drones is mostly downward. The features of the same power transmission and transformation equipment are very different, resulting in a decrease in the accuracy of the defect recognition model.

[0005] The appearance of power transmission and transformation equipment components with different voltage levels and different models varies greatly. Using the same type of power transmission and transformation equipment component labels to represent equipment components with different voltage levels and different models results in poor model training effects. The reason is that the image features of equipment components with different voltage levels and different models vary greatly, and the model features are diluted. For example, there are obvious differences in the surface fouling of glass insulators, porcelain insulators, and composite insulators, and they are all labeled as "surface fouling", resulting in a decrease in the accuracy of the defect recognition model.

[0006] The defects, personnel safety risks, and equipment states of different power transmission and transformation equipment components vary greatly. Using the same type of defect labels to represent the defects of different power transmission and transformation equipment components results in poor model training effects. The reason is that the image features of different power transmission and transformation equipment components vary greatly, and the target features will be diluted during model training. For example, there are obvious differences between the oil stains on the surface of the transformer tank, the oil stains on the surface of the riser, and the oil stains on the surface of the bushing, and they are all labeled as "oil stains on the component surface", resulting in a decrease in the accuracy of the defect recognition model.

[0007] The severity of defects in different power transmission and transformation equipment components is different. For the images with abnormal defects identified, their severity and corresponding handling measures are different. For example, the lack of zinc layer on the steel foot starting to rust is a general defect, but the deposition of rust on the surface is a serious defect. If the same label is used to represent defects of different severity levels, it will cause confusion in the defect handling measures of power transmission and transformation equipment.

[0008] The shapes of the annotation frames for different defects vary greatly. For example, the annotation frames for insulators, bushings, etc. are mostly long and narrow; while the annotation frames for types such as metal corrosion are mostly those with fewer pixels; and the annotation frames for abnormal meters are mostly square. Different algorithms have differences in the detection speed for different annotation frames. If the same algorithm is used to learn different types of annotation frames, it will reduce the detection speed and effect of the algorithm model.

[0009] Therefore, a technology is needed to achieve the technology of multi-level associated image annotation for power transmission and transformation equipment. Summary of the Invention

[0010] The technical solution of the present invention provides a method and system for multi-level associated image annotation of power transmission and transformation equipment to solve the problem of how to perform multi-level associated image annotation on power transmission and transformation equipment.

[0011] To solve the above problems, the present invention provides a method for multi-level associated image annotation of power transmission and transformation equipment, and the method includes:

[0012] Based on the image quality recognition standard, identify the sample images including the defects of power transmission and transformation equipment, and identify the sample images that meet the image quality standard;

[0013] Based on the correspondence between the equipment ledger and the equipment location information of the power transmission and transformation equipment in the sample image, label the basic attributes of the sample image to the attribute tags, where the basic attributes include the acquisition terminal that provides the sample image;

[0014] Establish multi-level tags for the power transmission and transformation equipment, and label each level of tags in the multi-level tags of the power transmission and transformation equipment in the sample image respectively:

[0015] Start labeling from the highest-level tag and sequentially label to the next lower-level tag to generate multi-level tags.

[0016] Preferably, it further includes:

[0017] Retrieve based on the attribute of any level of the multi-level tags to obtain a customized sample image set;

[0018] Determine the power transmission and transformation equipment image defect recognition algorithm model corresponding to the customized sample image set, and recognize the defects of the power transmission and transformation equipment in the customized sample image set based on the power transmission and transformation equipment image defect recognition algorithm model.

[0019] Preferably, the multi-level tags of the power transmission and transformation equipment include: equipment level tags, scene level tags, sub-scene level tags, and defect severity level tags; there is a subordinate relationship between the lower-level tags and the upper-level tags corresponding to the lower-level tags.

[0020] Preferably, the recognition of the sample image including the defects of the power transmission and transformation equipment based on the image quality recognition standard includes: sample image clarity recognition, sample image brightness exposure recognition, and sample image recognition degree recognition;

[0021] Evaluate the clarity of the sample image based on the energy gradient function, and select the sample images that reach the clarity threshold, where the energy gradient function is:

[0022] E(f) = ∑ y ∑ x (|f(x + 1, y) - f(x, y)| 2 + |f(x, y + 1) - f(x, y)| 2

[0023] E(f) is the energy gradient function. The larger the value of the energy gradient function, the clearer the sample image; conversely, the more blurred the sample image;

[0024] x is the pixel coordinate in the horizontal direction of the sample image;

[0025] y is the pixel coordinate in the vertical direction of the sample image;

[0026] f(x, y) is the gray value of the pixel coordinate (x, y).

[0027] Preferably, the recognition of the sample images including the defects of power transmission and transformation equipment based on the image quality recognition standard includes:

[0028] Detecting underexposed or overexposed image samples based on luminance offset detection, and the exposure calculation function is:

[0029]

[0030] N = W * H

[0031] W is the width of the sample image;

[0032] H is the height of the sample image;

[0033]

[0034] M is the average deviation from the average gray level of 128;

[0035] Hist is the histogram of gray levels;

[0036]

[0037]

[0038] Preferably, the recognition of the sample images including the defects of power transmission and transformation equipment based on the image quality recognition standard includes:

[0039] Detecting difficult-to-identify image samples through a noise detection algorithm, and the noise level calculation function is:

[0040]

[0041]

[0042] Av is the average value of the sample image area;

[0043] R(x, y) is the pixel value of the pixel coordinate (x, y);

[0044] VAR is the function for calculating variance;

[0045] Noise is the noise level of the sample image; the larger the Noise value, the more difficult it is to identify the sample image.

[0046] Preferably, the acquisition terminal includes: a robot, a drone, a fixed camera, and a handheld terminal.

[0047] Preferably, the multi-level tags are output in a structured text format. On the other hand of the present invention, the present invention provides an image annotation system for multi-level association of power transmission and transformation equipment, and the system includes:

[0048] An initial unit for identifying sample images including defects of power transmission and transformation equipment based on image quality recognition criteria, and identifying sample images that meet the image quality criteria;

[0049] A first annotation unit for annotating the basic attributes of the sample image to an attribute label based on the correspondence between the equipment ledger and equipment location information of the power transmission and transformation equipment in the sample image, where the basic attributes include the acquisition terminal that provides the sample image;

[0050] A second annotation unit for establishing multi-level labels for power transmission and transformation equipment, and respectively annotating each level of label in the multi-level labels of the power transmission and transformation equipment in the sample image: starting from the highest-level label and sequentially annotating to the next lower-level label to generate multi-level labels.

[0051] Preferably, it further includes an identification unit for:

[0052] Retrieving based on the attribute of any level of label in the multi-level labels to obtain a customized sample image set;

[0053] Determining an image defect recognition algorithm model for power transmission and transformation equipment corresponding to the customized sample image set, and identifying defects of power transmission and transformation equipment in the customized sample image set based on the image defect recognition algorithm model for power transmission and transformation equipment.

[0054] Preferably, the multi-level labels of the power transmission and transformation equipment include: equipment level labels, scene level labels, sub-scene level labels, and defect severity level labels; there is a subordinate relationship between the lower-level labels and the upper-level labels corresponding to the lower-level labels.

[0055] Preferably, the initial unit for identifying sample images including defects of power transmission and transformation equipment based on image quality recognition criteria includes: sample image clarity recognition, sample image brightness exposure recognition, and sample image recognizability recognition.

[0056] Evaluating the clarity of the sample image based on the energy gradient function, and selecting the sample images that reach the clarity threshold, where the energy gradient function is:

[0057] E(f) = ∑ y ∑ x (|f(x + 1, y) - f(x, y)| 2 +|f(x, y + 1) - f(x, y)| 2

[0058] E(f) is the energy gradient function, the larger the value of the energy gradient function, the clearer the sample image; conversely, the more blurred the sample image;

[0059] x is the pixel coordinate in the horizontal direction of the sample image;

[0060] y is the pixel coordinate in the vertical direction of the sample image;

[0061] f(x, y) is the gray value of the pixel coordinate (x, y).

[0062] Preferably, the initial unit is used to identify a sample image including a defect of a power transmission and transformation equipment based on an image quality identification standard, including:

[0063] Detecting underexposed or overexposed image samples based on a brightness offset detection, and the exposure calculation function is:

[0064]

[0065] N = W * H

[0066] W is the width of the sample image;

[0067] H is the height of the sample image;

[0068]

[0069] M is the average deviation from the average gray value of 128;

[0070] Hist is the histogram of gray levels;

[0071]

[0072]

[0073] Preferably, the initial unit is used to identify a sample image including a defect of a power transmission and transformation equipment based on an image quality identification standard, including:

[0074] Detecting an indistinguishable image sample through a noise detection algorithm, and the noise level calculation function is:

[0075]

[0076]

[0077] Av is the average value of the sample image area;

[0078] R(x, y) is the pixel value of the pixel coordinate (x, y);

[0079] VAR is the function for calculating variance;

[0080] Noise is the noise level of the sample image; the larger the Noise value, the more indistinguishable the sample image.

[0081] Preferably, the acquisition terminal includes: a robot, a drone, a fixed camera, and a handheld terminal.

[0082] Preferably, the multi-level tags are output in a structured text format. The present invention provides a method and system for image annotation with multi-level association of power transmission and transformation equipment. The method includes: identifying sample images including defects of power transmission and transformation equipment based on an image quality recognition standard, and selecting sample images that meet the image quality standard; based on the correspondence between the equipment ledger and equipment location information of the power transmission and transformation equipment in the sample images, labeling the basic attributes of the sample images to attribute tags, where the basic attributes include the acquisition terminal that provides the sample images; establishing multi-level tags for the power transmission and transformation equipment, and respectively labeling each level of tags in the multi-level tags of the power transmission and transformation equipment in the sample images; and each level of tag has a subordinate relationship with its corresponding superior tag. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The exemplary embodiments of the present invention can be more fully understood by referring to the following drawings:

[0084] Figure 1 FIG. is a flowchart of a method for image annotation with multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention;

[0085] Figure 2 FIG. is a flowchart of a method for image annotation with multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention;

[0086] Figure 3 FIG. is a framework structure diagram of a system for image annotation with multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention;

[0087] Figure 4 FIG. is a hierarchical annotation module of a system for image annotation with multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention;

[0088] Figure 5 FIG. is a schematic diagram of a labeling frame for image annotation with multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention;

[0089] Figure 6 FIG. is a schematic diagram of a keyword retrieval module of a system for image annotation with multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention; and

[0090] Figure 7 FIG. is a structure diagram of a system for image annotation with multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0091] Reference is now made to the accompanying drawings to describe exemplary embodiments of the present invention. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to disclose the present invention in detail and completely, and to fully convey the scope of the present invention to those skilled in the art. The terms in the exemplary embodiments shown in the drawings are not intended to limit the present invention. In the drawings, the same units / components are denoted by the same reference numerals.

[0092] Unless otherwise specified, the terms used herein (including scientific and technical terms) have the ordinary meaning understood by those skilled in the art. Additionally, it can be understood that terms defined in commonly used dictionaries should be construed to have a meaning consistent with the context of their relevant fields, and should not be construed as having an idealized or overly formal meaning.

[0093] Figure 1 A flowchart of an image annotation method for multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention. The present invention provides an image annotation method for multi-level association of power transmission and transformation equipment, and provides a basic solution for identifying defects in power transmission and transformation images by establishing an image annotation method for multi-level association of power transmission and transformation equipment.

[0094] As Figure 1 shown, the present invention provides an image annotation method for multi-level association of power transmission and transformation equipment, and the method includes:

[0095] Step 101: Identify sample images including power transmission and transformation equipment defects based on an image quality recognition standard, and identify sample images that meet the image quality standard;

[0096] Preferably, identifying sample images including power transmission and transformation equipment defects based on an image quality recognition standard includes:

[0097] Evaluating the clarity of the sample images based on an energy gradient function, and selecting sample images that reach the clarity threshold, where the energy gradient function is:

[0098] E(f) = ∑ y ∑ x (|f(x + 1, y) - f(x, y)| 2 + |f(x, y + 1) - f(x, y)| 2

[0099] E(f) is the energy gradient function. The larger the value of the energy gradient function, the clearer the sample image; conversely, the more blurred the sample image;

[0100] x is the pixel coordinate in the horizontal direction of the sample image;

[0101] y is the pixel coordinate in the vertical direction of the sample image;

[0102] f(x, y) is the gray value of the pixel coordinates (x, y).

[0103] Preferably, the sample images including the defects of power transmission and transformation equipment are identified based on the image quality recognition criteria, including:

[0104] Based on the brightness offset detection, the under-exposed or over-exposed images are detected, and the exposure calculation function is:

[0105]

[0106] N = W * H

[0107] W is the width of the sample image;

[0108] H is the height of the sample image;

[0109]

[0110] M is the average deviation from the average gray value of 128;

[0111] Hist is the histogram of gray levels;

[0112]

[0113]

[0114] Preferably, the sample images including the defects of power transmission and transformation equipment are identified based on the image quality recognition criteria, including:

[0115] The hard-to-identify images are detected through the noise detection algorithm, and the noise level calculation function is:

[0116]

[0117]

[0118] Av is the average value of the sample image area;

[0119] R(x, y) is the pixel value of the pixel coordinates (x, y);

[0120] VAR is the function for calculating variance;

[0121] Noise is the noise level of the sample image; the larger the Noise value, the harder it is to identify the sample image.

[0122] The present invention automatically discriminates the image quality of sample image (single-frame video) files: the image resolution is not lower than 1920*1080, and images that do not meet the resolution will be automatically excluded; the file should be the original file taken and stored by the inspection image acquisition device without secondary processing, and images with insufficient exposure, backlight shooting, low contrast, and difficult to identify will be automatically excluded.

[0123] Step 102: Based on the correspondence between the equipment ledger and the equipment point information of the power transmission and transformation equipment in the sample image, label the basic attributes of the sample image to the attribute tags, and the basic attributes include the acquisition terminal that provides the sample image.

[0124] The present invention selects the file attributes of the sample image (single-frame video): the acquisition terminals are divided into robots, drones, fixed cameras, and handheld terminals, and their perspectives and image qualities are not uniform. The information of the image acquisition terminal needs to be used as the basic attribute for sample annotation.

[0125] Step 103: Establish multi-level tags for the power transmission and transformation equipment, and label each level of tags in the multi-level tags of the power transmission and transformation equipment in the sample image respectively; each level of tag has a subordinate relationship with its corresponding superior tag. Preferably, the multi-level tags of the power transmission and transformation equipment include: equipment level tags, scene level tags, sub-scene level tags, and defect severity level tags.

[0126] The present invention performs hierarchical classification annotation on the sample image of the sample: the tags are divided into equipment, components, scenes, sub-scenes, and their severity levels, a total of four levels of tags. There is a subordinate relationship between the lower-level tags and the upper-level tags.

[0127] Among them, the equipment components can be associated with the equipment ledger and the equipment ID through point settings.

[0128] Preferably, it further includes: retrieving based on the attribute of any level of tag in the multi-level tags to obtain a customized sample image set; determining the power transmission and transformation equipment image defect recognition algorithm model corresponding to the customized sample image set, and recognizing the defects of the power transmission and transformation equipment in the customized sample image set based on the power transmission and transformation equipment image defect recognition algorithm model.

[0129] The sample image of the present invention can be retrieved according to the hierarchical classification of equipment component defects, providing different samples for different power transmission and transformation equipment defect recognition algorithms, and training the defect recognition model in a targeted manner.

[0130] Preferably, the acquisition terminals include: robots, drones, fixed cameras, and handheld terminals.

[0131] The present invention screens fuzzy sample images based on an energy gradient function; screens images that are difficult to identify through a target detection algorithm for transmission and transformation equipment components; detects underexposed or overexposed images through brightness offset detection; and pushes the screened image samples that do not meet the image quality requirements to manual secondary review to improve the quality of the sample images stored in the library.

[0132] The present invention provides a defect annotation system for transmission and transformation equipment, including a function for importing equipment account books, a function for automatically identifying the attributes of image capture terminals, a function for classifying and labeling equipment components at different levels, and a function for automatically matching the severity levels of defects, which greatly improves the annotation efficiency of sample annotators and the multi-level attributes of samples.

[0133] The sample images after annotation and storage in the present invention can be retrieved through classification keywords at different levels to obtain a customized sample set that returns results according to the retrieval keywords, which is used for model training of targeted image defect recognition algorithms for transmission and transformation equipment, and differentially improves the speed and accuracy of various defect recognition algorithms.

[0134] The present invention discloses a method for multi-level associated image annotation of transmission and transformation equipment, a method for hierarchical classification annotation of samples for deep learning algorithm training for transmission and transformation defect image recognition, belonging to the technical field of power equipment monitoring.

[0135] As Figure 2 shown, the method for multi-level associated image annotation of transmission and transformation equipment of the present invention includes the following steps: First, automatically judge the image quality of the transmitted and collected image defect samples of transmission and transformation equipment; Second, for the screened image defect samples of transmission and transformation equipment, the reviewers conduct manual review of the image quality; Third, in combination with the equipment account book and point information, the system automatically annotates the sample attributes of the reviewed image samples; Fourth, in the multi-level associated annotation system of transmission and transformation equipment, the annotators conduct hierarchical annotation of the image samples; Fifth, for the annotated samples, the reviewers conduct sample quality review of the samples annotated by the annotators; Sixth, generate a customized sample set for model training through retrieval of annotation label keywords.

[0136] Among them, the present invention provides an automatic image quality discrimination function for the problem of uneven image quality.

[0137] First, evaluate the sample clarity based on the energy gradient function, and automatically eliminate the samples with poor clarity. Among them, the energy gradient function is as follows:

[0138] E(f) = ∑ y ∑ x (|f(x + 1, y) - f(x, y)| 2 + |f(x, y + 1) - f(x, y)| 2

[0139] E(f) is the energy gradient function. The larger the value of the energy gradient function, the clearer the image; conversely, the more blurred the image.

[0140] x is the pixel coordinate in the horizontal direction of the image;

[0141] y is the pixel coordinate in the vertical direction of the image;

[0142] f(x, y) is the gray value of the pixel coordinate (x, y);

[0143] Second, detect underexposed or overexposed images through brightness offset detection. The exposure is calculated as follows:

[0144]

[0145] N = W * H

[0146] W is the width of the image;

[0147] H is the height of the image;

[0148]

[0149] M is the average deviation from the average gray value of 128;

[0150] Hist is the histogram of gray levels;

[0151]

[0152]

[0153] Third, detect images that are difficult to recognize through a noise detection algorithm. The noise level is calculated as follows:

[0154]

[0155]

[0156] Av is the average value of the image area;

[0157] R(x, y) is the pixel value of the pixel coordinate (x, y);

[0158] VAR is the function for calculating variance;

[0159] Noise is the image noise level;

[0160] The larger the value of Noise, the more difficult it is to recognize the image.

[0161] Through the algorithms listed above, automatically eliminate the input samples with poor image quality.

[0162] Among them, image sample annotation software such as labelimg or labelme for deep learning usually cannot obtain basic information about image objects. For example, for public datasets such as COCO and PASCAL, there is a lack of description of the pictures themselves, such as the shooting location, shooting equipment, etc. The understanding and annotation of pictures are completely based on the common sense of the annotators. However, due to the variety of power equipment and the inconsistent reporting forms of line and substation terminal defect pictures, annotators without long-term on-site work experience will not be able to effectively distinguish the sources of samples. The image samples of the acquisition terminals vary greatly. Through the account and point information, the present invention automatically obtains information of different acquisition terminals as attributes for image annotation.

[0163] The mapping table of the equipment component inspection points and the acquisition terminal deployment of the present invention is as follows:

[0164] Table 1. Mapping Table of Equipment Component Inspection Points and Acquisition Terminal Deployment (Example)

[0165]

[0166]

[0167]

[0168] The equipment point information table is shown as follows:

[0169] Table 2. Equipment Point Information Table (Example)

[0170]

[0171]

[0172]

[0173] Table 3 Attribute Automatic Association List

[0174]

[0175] Through the mapping table (such as Table 1) and the equipment point information (such as Table 2), the present invention automatically generates attributes (such as Table 3) in the annotation file to realize the classification of the sources of sample images. The form of the automatically generated attribute output is shown in Table 4.

[0176] Table 4 Automatically Generated Attribute xml Output Format

[0177]

[0178]

[0179] Among them, the appearance differences of power transmission and transformation equipment components with different voltage levels and different models are significant. Using the same type of power transmission and transformation equipment component labels to represent equipment components with different voltage levels and different models results in poor model training effects. The reason is that the image features of equipment components with different voltage levels and different models vary greatly, diluting the model features; the defects, personnel safety risks, and equipment states of different power transmission and transformation equipment components also vary greatly. Using the same type of defect label to represent the defects of different power transmission and transformation equipment components leads to poor model training effects. The reason is that the image features of different power transmission and transformation equipment components vary greatly, diluting the target features during model training; the severity of defects of different power transmission and transformation equipment components is different. The defect severity level association table is shown in Table 5 below:

[0180] Table 5 Defect and Severity Level Association Table

[0181]

[0182]

[0183]

[0184] Therefore, this patent provides a multi-level association annotation method, as Figure 4 shown, to achieve hierarchical and graded association annotation, and to distinguish equipment, components, defects, and defect severity levels. The output file format of the multi-level association annotation is as follows:

[0185] Table 5 Multi-level Annotation Output Format and XML Format Example

[0186]

[0187]

[0188] The multi-level labels of the present invention are output in a structured text format. In addition to the XML format output illustrated in the examples of the present invention, they can also be output in formats such as HTML, TXT, and JASON.

[0189] As Figure 5 shown, the annotation boxes are divided into three levels, namely the equipment annotation box, the component annotation box, and the box within the scene annotation box. The boxes are automatically recognized as the next-level labels. The annotation image should represent the output result and the target range in the form of a marked box. The target range recording method is to save the vertex coordinates of the upper left and lower right corners of the rectangle. If it is a polygon annotation box, the recording method is all the vertex coordinates generated during the annotation process. The marked box should meet the annotation requirements of the corresponding equipment, components, and inspection scenarios.

[0190] a) For the annotation of components, the area of the annotation box should be minimized on the basis of covering the target component.

[0191] b) For the annotation of defects that do not depend on the surrounding environment, the annotation shall refer to the requirements for component annotation.

[0192] c) For the annotation of defects that depend on the surrounding environment, when the annotation box covers the target, it shall cover the necessary surrounding environment part.

[0193] d) The output label is output in a three - level form. The first level shall label the equipment, the second level shall label the component, and the third level shall label the fault type.

[0194] If the annotation of the fault is a whole - component fault, the component annotation box can be reused to output the fault type. If it is a local component fault, the fault area needs to be framed on the component to output the fault name. There are also some faults that can appear at any position, and the fault area can be directly annotated, such as foreign objects.

[0195] The final multi - level label output form is shown in Table 6:

[0196] Table 6 Multi - level annotation output format and XML format example

[0197]

[0198]

[0199]

[0200] Among them, for the images with identified abnormal defects, their severities and corresponding handling measures are different; the shapes of the annotation boxes for different defects vary greatly, with different proportions and sizes of small boxes, long boxes, square boxes, etc. There are differences in the detection speeds of different algorithms for different annotation boxes. If the same algorithm is used to learn different types of annotation boxes, it will reduce the detection speed and effect of the algorithm model. Therefore, as provided by the present invention, Figure 6 as shown, through label keyword screening, sample customization is realized for training a customized defect recognition model for power transmission and transformation equipment.

[0201] The present invention stores hierarchical labels in the form of XML (Extensible Markup Language), and retrieves and counts the attributes at all levels in the XML through the methods of parse, findall, and find.

[0202] By obtaining all the XML file lists in the storage directory, fast operations are performed on all the stored XML files;

[0203] The parse function is used for parsing XML files. Since XML files are in text format (string), the computer cannot directly read them as structured text. Therefore, the parse function is used to point the pointer to the XML file to be parsed and construct the entire XML structured text between the root nodes.

[0204] The findall function returns all matching elements that meet the search keyword at once. By using the find function in the returned list, corresponding elements can be operated on.

[0205] The graph function visualizes the hierarchical statistical situation through the interface, such as Figure 6 shown.

[0206] Figure 7 It is a structural diagram of an image annotation system for multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention. As Figure 7 shown, the present invention provides an image annotation system for multi-level association of power transmission and transformation equipment, and the system includes:

[0207] An initial unit 701, configured to identify sample images including power transmission and transformation equipment defects based on an image quality recognition standard, and identify sample images that meet the image quality standard.

[0208] Preferably, the initial unit is configured to identify sample images including power transmission and transformation equipment defects based on an image quality recognition standard, including:

[0209] Evaluating the clarity of the sample image based on an energy gradient function, and selecting sample images that reach the clarity threshold. Among them, the energy gradient function is:

[0210] E(f) = ∑ y ∑ x (|f(x + 1, y) - f(x, y)| 2 + |f(x, y + 1) - f(x, y)| 2

[0211] E(f) is the energy gradient function. The larger the value of the energy gradient function, the clearer the sample image; on the contrary, the more blurred the sample image;

[0212] x is the pixel coordinate in the horizontal direction of the sample image;

[0213] y is the pixel coordinate in the vertical direction of the sample image;

[0214] f(x, y) is the gray value of the pixel coordinate (x, y).

[0215] Preferably, the initial unit is configured to identify sample images including power transmission and transformation equipment defects based on an image quality recognition standard, including:

[0216] Detect underexposed or overexposed image samples based on luminance offset, and the exposure calculation function is:

[0217]

[0218] N = W * H

[0219] W is the width of the sample image;

[0220] H is the height of the sample image;

[0221]

[0222] M is the average deviation from the average gray level of 128;

[0223] Hist is the histogram of gray levels;

[0224]

[0225]

[0226] Preferably, the initial unit is used to identify sample images including defects of power transmission and transformation equipment based on image quality recognition criteria, including:

[0227] Detect hard-to-identify image samples through a noise detection algorithm, and the noise level calculation function is:

[0228]

[0229]

[0230] Av is the average value of the sample image area;

[0231] R(x, y) is the pixel value of the pixel coordinate (x, y);

[0232] VAR is the function for calculating variance;

[0233] Noise is the noise level of the sample image; the larger the Noise value, the harder it is to identify the sample image.

[0234] The first annotation unit 702 is used to annotate the basic attributes of the sample image to the attribute label based on the correspondence between the equipment ledger and the equipment location information of the power transmission and transformation equipment in the sample image.

[0235] The second annotation unit 703 is used to establish multi-level tags for power transmission and transformation equipment, and annotate each level of tags in the multi-level tags of the power transmission and transformation equipment in the sample image; each level of tag has a subordinate relationship with its corresponding superior tag. Preferably, the multi-level tags of the power transmission and transformation equipment include: equipment level tags, scene level tags, sub-scene level tags, and defect severity level tags.

[0236] Preferably, the system further includes an identification unit, which is used to: retrieve based on any level of tag attributes of the multi-level tags to obtain a customized sample image set;

[0237] Determine an image defect recognition algorithm model for power transmission and transformation equipment corresponding to the customized sample image set, and recognize the defects of the power transmission and transformation equipment in the customized sample image set based on the image defect recognition algorithm model for power transmission and transformation equipment.

[0238] An image annotation system 700 with multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention corresponds to an image annotation method 100 with multi-level association of power transmission and transformation equipment according to a preferred embodiment of the present invention, and will not be elaborated here.

[0239] The present invention has been described by referring to a few embodiments. However, as is well known to those skilled in the art, as defined by the appended patent claims, other embodiments equivalent to those disclosed above of the present invention equally fall within the scope of the present invention.

[0240] Generally, all terms used in the claims are interpreted according to their ordinary meanings in the technical field, unless otherwise clearly defined therein. All references to "a / the / this [device, component, etc.]" are to be interpreted openly as at least one instance of the device, component, etc., unless otherwise clearly stated. The steps of any method disclosed herein need not be run in the exact order disclosed, unless clearly stated.

Claims

1. A method for multi - level associated image annotation of power transmission and transformation equipment, the method comprising: Identifying sample images including power transmission and transformation equipment defects based on image quality recognition criteria, and identifying sample images that meet the image quality standards, including: Identifying sample images including power transmission and transformation equipment defects based on image quality recognition criteria, including: Evaluating the clarity of sample images based on an energy gradient function, and selecting sample images that reach the clarity threshold. Among them, the energy gradient function is: E(f) = ∑ y ∑ x (|f(x + 1, y) - f(x, y)| 2 + |f(x, y + 1) - f(x, y)| 2 E(f) is the energy gradient function. The larger the value of the energy gradient function, the clearer the sample image; conversely, the more blurred the sample image; x is the pixel coordinate in the horizontal direction of the sample image; y is the pixel coordinate in the vertical direction of the sample image; f(x, y) is the gray value of the pixel coordinate (x, y); Identifying sample images including power transmission and transformation equipment defects based on image quality recognition criteria, including: Detecting under-exposed or over-exposed images based on brightness offset detection. The exposure calculation function is: N = W * H W is the width of the sample image; H is the height of the sample image; M is the average deviation from the average gray value of 128; Hist is the histogram of gray levels; Identifying sample images including power transmission and transformation equipment defects based on image quality recognition criteria, including: Detecting images that are not easily recognizable through a noise detection algorithm. The noise level calculation function is: Av is the average value of the sample image area; R(x, y) is the pixel value of the pixel coordinate (x, y); VAR is the function for calculating variance; Noise is the noise level of the sample image; the larger the value of Noise, the less recognizable the sample image; Based on the correspondence between the equipment ledger and equipment location information of the power transmission and transformation equipment in the sample images that meet the image quality standards, labeling the basic attributes of the sample images to attribute tags. The basic attributes include the acquisition terminal that provides the sample images; Establishing multi-level tags for power transmission and transformation equipment, and respectively labeling each level of tags in the multi-level tags of the power transmission and transformation equipment in the sample images that meet the image quality standards: Labeling starts from the highest-level tag and proceeds to the next lower-level tag in sequence to generate multi-level tags; the multi-level tags of the power transmission and transformation equipment include: equipment level tags, component level tags, defect level tags, sub-defect level tags, and defect severity level tags; there is a subordinate relationship between lower-level tags and the corresponding upper-level tags; Retrieving based on the attribute of any level of the multi-level tags to obtain a customized sample image set; Determining a power transmission and transformation equipment image defect recognition algorithm model corresponding to the customized sample image set, and identifying the defects of the power transmission and transformation equipment in the customized sample image set based on the power transmission and transformation equipment image defect recognition algorithm model.

2. The method according to claim 1, wherein identifying the sample image including power transmission and transformation equipment defects based on the image quality recognition standard comprises: Sample image clarity recognition, sample image brightness exposure recognition, sample image recognition degree recognition.

3. The method according to claim 1, wherein the multi - level labels are output in a structured text format.

4. An image annotation system for multi - level associated power transmission and transformation equipment, the system comprising: An initial unit for identifying sample images including power transmission and transformation equipment defects based on image quality recognition criteria, and identifying sample images that meet the image quality standards, including: Identifying sample images including power transmission and transformation equipment defects based on image quality recognition criteria, including: Evaluate the sharpness of the sample images based on the energy gradient function, and select the sample images that reach the sharpness threshold. Among them, the energy gradient function is: E(f) = ∑ y ∑ x (|f(x + 1, y) - f(x, y)| 2 + |f(x, y + 1) - f(x, y)| 2 E(f) is the energy gradient function. The larger the value of the energy gradient function, the sharper the sample image; on the contrary, the more blurred the sample image; x is the pixel coordinate in the horizontal direction of the sample image; y is the pixel coordinate in the vertical direction of the sample image; f(x, y) is the grayscale value of the pixel coordinate (x, y); Identify the sample images including the defects of power transmission and transformation equipment based on the image quality identification standard, including: Detect under-exposed or over-exposed images based on brightness offset detection. The exposure calculation function is: N = W * H W is the width of the sample image; H is the height of the sample image; M is the average deviation from the average gray level of 128; Hist is the histogram of grayscale; Identify the sample images including the defects of power transmission and transformation equipment based on the image quality identification standard, including: Detect images that are not easily recognizable through the noise detection algorithm. The noise level calculation function is: Av is the average value of the sample image area; R(x, y) is the pixel value of the pixel coordinate (x, y); VAR is the function for calculating variance; Noise is the noise level of the sample image; the larger the value of Noise, the less recognizable the sample image; The first annotation unit is used to annotate the basic attributes of the sample image to the attribute label based on the correspondence between the equipment ledger and the equipment point information of the power transmission and transformation equipment in the sample image that meets the image quality standard. The basic attributes include the acquisition terminal that provides the sample image; The second annotation unit is used to establish multi-level labels for the power transmission and transformation equipment, and annotate each level of label in the multi-level labels of the power transmission and transformation equipment in the sample image that meets the image quality standard: start from the highest-level label and annotate down to the next-level label in turn to generate multi-level labels; the multi-level labels of the power transmission and transformation equipment include: equipment level label, component level label, defect level label, sub-defect level label, and defect severity level label; there is a subordinate relationship between the lower-level label and the upper-level label corresponding to the lower-level label; The recognition unit is used to: retrieve based on any level of label attribute of the multi-level labels to obtain a customized sample image set; Determine the power transmission and transformation equipment image defect recognition algorithm model corresponding to the customized sample image set, and identify the defects of the power transmission and transformation equipment in the customized sample image set based on the power transmission and transformation equipment image defect recognition algorithm model.

5. The system according to claim 4, wherein the initial unit is configured to identify a sample image including a defect of a power transmission and transformation device based on an image quality recognition criterion, including: Sample image sharpness recognition, sample image brightness exposure recognition, sample image recognition degree recognition.

6. The system according to claim 4, wherein the multi-level tags are output in a structured text format.

Images