Multi-person collaborative annotation system and audit management method for inspection images of overhead transmission lines

Abstract

The present invention relates to a multi-person collaborative annotation system and an audit management method for inspection images of overhead transmission lines. The method of the present invention can automatically audit the quality of image annotation. For special labels in the field of power inspection, the workload of professional personnel for auditing is greatly reduced. The method of the present invention eliminates the professional personnel auditing step. For developers, automatic auditing improves work efficiency and reduces the workload. In addition, based on the special environment of overhead transmission lines, the present invention solves the problems of frame covering and reverse labeling, and identifies overlapping targets to realize the judgment of annotation quality. Finally, the final annotation result is derived by a label integration algorithm. The present invention is strict in measuring the quality of labels, which is manifested in that this algorithm evaluates all label data, reduces abnormal labels, and thus improves the quality of label data.

Description

Technical Field

[0001] The present invention relates to the technical field of inspection image data annotation for overhead transmission lines, and particularly relates to a multi-person collaborative annotation system and an audit management method for inspection images of overhead transmission lines. Background Art

[0002] Overhead transmission lines are an important part of the national power engineering distribution network, and the importance of their maintenance is related to the stability of the country's people's livelihood. Drones have become the first choice for power inspection due to their unique performance advantages. To ensure the efficiency and quality of the drone image inspection technology, professionals have started training high-precision target recognition models for transmission line components.

[0003] The training of target recognition models in such professional fields as overhead transmission lines requires a large amount of data for training. Therefore, it is particularly important to construct a high-quality data set. Dividing and labeling the required objects on the pictures is what people often call "tagging". At present, the production of data sets is mainly completed through crowdsourcing platforms and professional annotation platforms. However, the annotators often do not understand the objects in such professional fields as overhead transmission lines, resulting in low annotation quality. To solve this problem, the current mainstream solution is that after the annotator finishes the annotation, one or more professional auditors review each annotation result. For this reason, a large amount of professional human cost needs to be invested in the review link. Multi-person collaborative image annotation review is mostly a voting mechanism. For the same object on a picture, if most people make the same annotation for it, it is considered a correct annotation. Using the voting mechanism generally requires at least three people, and the manpower cost for producing the same data set is twice as much. There will be multiple annotations for the same object in multi-person collaborative annotation, and these annotations need to be effectively utilized to obtain more accurate annotations.

[0004] In the prior art, such as CN108932724A, a system automatic review method based on multi-person collaborative image annotation, application number: 201810550862.X, designed a system automatic review method based on multi-person collaborative image annotation to automatically review the overall image annotation quality. This invention uses a voting mechanism to discard the object labels of a few people, which may lead to discarding high-quality labels in the case of a small number of annotators; and this invention uses all label pixel points as the weight to analyze and match the object area, but it cannot solve the common bounding box phenomenon in the context of multi-person annotation of transmission lines.

[0005] Therefore, the present invention proposes AIOU and DIOU to solve the bounding box phenomenon matching problem from the perspective of labels, and proposes an integration algorithm to make the final annotation result close to the target object we want to annotate, enhancing the quality of the annotation data, accelerating the time spent on automatic review, reducing the manual review cost, and improving the annotation review efficiency. Summary of the Invention

[0006] The object of the present invention is to solve the problems of high labor cost in the current multi-person collaborative annotation and review and low annotation quality of the overhead transmission line image dataset, and to provide a multi-person collaborative annotation system and review management method for overhead transmission line inspection images, which can preliminarily review the image annotation quality and save a large amount of labor costs.

[0007] To achieve the above object, the technical solution of the present invention is: a multi-person collaborative annotation system and review management method for overhead transmission line inspection images. First, match the annotation data of multiple persons to obtain an object area annotation box, judge its category and quantity, solve the problems of frame covering and incorrect labeling based on the special environment of overhead transmission lines, and identify overlapping targets to achieve a preliminary judgment of annotation quality. Finally, the final annotation result is derived by the Gaussian integration algorithm. The method specifically includes the following steps:

[0008] Step 1, make a dataset for multi-person image annotation;

[0009] Step 2, design a label area matching method: first randomly select a label from all labels and define the range of its bounding box as an object, then continuously randomly select labels from the remaining labels, calculate the IOU value with the existing object label and take the maximum IOU value. If this IOU is greater than the set threshold, it is considered that this label belongs to the label of the corresponding area range, otherwise it is regarded as a new area until all labels have their own areas;

[0010] Step 3, perform label area matching using the label area matching method in Step 2;

[0011] Step 4, label object matching:

[0012] First, propose AIOU and DIOU:

[0013] AIOU = |θ i - θ ∩ |

[0014]

[0015]

[0016] where θ ∩ is the aspect ratio angle of the overlapping part of the bounding box, W ∩ is the width value of the overlapping part of the bounding box, H ∩ is the height value of the overlapping part of the bounding box, θ i is the aspect ratio angle of the i-th bounding box, W i is the width value of the i-th bounding box, H i is the height value of the i-th bounding box, d is the distance between the centers of the two bounding boxes, S ∩ is the area of the overlapping part of the bounding box;

[0017] Next, perform label-object matching. The process is as follows: Randomly select one of the object regions, count all the label categories in this object region, and determine whether all the label categories in this object region are unique categories. If different category labels appear, compare its AIOU with other labels of objects of this category on the image to see if it is greater than the set threshold. If so, it is judged as a scene where the category label is reversed, and this label is removed; if not, it is judged that there is an overlapping occlusion of heterogeneous objects at this time, and calculate whether the DIOU is greater than the threshold. If so, it is classified as a new object; if all the label categories in this object region are unique categories, count whether the annotator in this region has a unique label. If all the labels in this region come from different annotators, end the label-object matching; if all the labels in this region come from the same annotator, it is judged that there is an overlapping phenomenon of homogeneous objects at this time, and calculate whether the DIOU is greater than the threshold. If so, it is classified as a new object;

[0018] Step 5: Based on the labels after label matching, propose an object label integration algorithm to output the final annotation result; for the object label integration algorithm, first obtain all the labels of the same object, and set its coordinates as:

[0019] BBOX i (X i ,Y i ,W i ,H i )

[0020] Taking the upper left corner of the image as the coordinate origin, with the right direction as the positive Y-axis coordinate axis and the downward direction as the positive X-axis coordinate axis, where X i is the x coordinate value of the upper left corner of the i-th annotation box, Y i is the y coordinate value of the upper left corner of the i-th annotation box, W i is the width value of the i-th annotation box, and H i is the height value of the i-th annotation box;

[0021] After obtaining the coordinates of the upper left corner and the lower right corner of each annotation box, dynamically generate the weights of the coordinates, and calculate the final coordinates BBOX(X, Y, W, H) of this object through the weights. The coordinate calculation formula is as follows:

[0022]

[0023] where represents the weight of the x coordinate value of the i-th annotation box, represents the weight of the y coordinate value of the i-th annotation box, represents the weight of the height value h of the i-th annotation box, represents the weight of the width value w of the i-th annotation box;

[0024] Step 6: After the dataset is integrated through annotation review and clustering, the annotation file format is output as needed to obtain the output dataset.

[0025] In an embodiment of the present invention, the value range of IOU is from 0 to 1, where 0 means no overlap and 1 means complete overlap; the specific calculation method of the IOU value is as follows: First, calculate the intersection of the two labels, that is, the area they jointly cover; then, calculate the union of the two labels, that is, the total area they cover; finally, divide the intersection by the union to obtain the IOU value; the IOU calculation formula is:

[0026]

[0027] where A∩B is the overlapping area of the two object bounding boxes, and A∪B is the total area of the two object bounding boxes.

[0028] In an embodiment of the present invention, the calculation formula of the weight is as follows:

[0029]

[0030] where u (x，y，h，w) and σ (x，y，h，w) are the mean and variance of (x, y, h, w) respectively, and the calculation formulas of the mean and variance are as follows:

[0031]

[0032]

[0033] Compared with the prior art, the present invention has the following beneficial effects: By using the overlapping annotation box area to match and cluster the target objects, the present invention eliminates low-quality annotation information, uses the aspect ratio and center distance of the annotation box and the overlapping area to solve the problem of reverse annotation in special environments, reduces manual review, improves the annotation quality, applies the Gaussian distribution model to the image coordinate calculation to integrate the labels, makes the final annotation result close to the target object we want to annotate, enhances the quality of the annotation data, accelerates the time spent on automatic review, reduces the manual review cost, and improves the annotation review efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is the flowchart of the multi-person collaborative annotation system and review management method for overhead transmission line inspection images of the present invention;

[0035] Figure 2 is the label area matching diagram;

[0036] Figure 3 is the flowchart of label area matching;

[0037] Figure 4 is the IOU schematic diagram;

[0038] Figure 5 Schematic diagrams of AIOU and DIOU;

[0039] Figure 6 Flow chart for matching labeled objects;

[0040] Figure 7 Label coordinate system. Specific implementation manners

[0041] The technical solution of the present invention will be specifically described below in conjunction with the accompanying drawings.

[0042] As Figure 1 shown, for an overhead transmission line inspection image multi-person collaborative annotation system and review management method of the present invention, first, the annotation data of multiple persons are matched to obtain an object area annotation box, and the category and quantity thereof are judged. Based on the special environment of the overhead transmission line, the phenomena of frame covering and wrong marking are solved, and overlapping targets are identified to achieve a preliminary judgment of the annotation quality. Finally, the final annotation result is derived by the Gaussian integration algorithm. The specific implementation steps are as follows:

[0043] Step 1: Make a data set for multi-person image annotation.

[0044] Step 2: The present invention designs a label area matching method. First, a label is randomly selected from all the labels, and the range enclosed by it is defined as an object. Then, labels are continuously randomly taken from the remaining labels, and the IOU value is calculated with the existing object label and the maximum IOU value is taken. If this IOU is greater than the set threshold, it is considered that this label belongs to the label of this area; otherwise, it is regarded as a new area until all labels have their own areas. As Figure 2 shown is the label area matching diagram.

[0045] The flow chart of label area matching is as Figure 3 shown.

[0046] The IOU (Intersection over Union) calculation of labels is a technique for detecting the overlapping degree between two objects. It measures the overlapping degree between two objects by calculating the intersection and union between them. The value range of IOU is from 0 to 1, where 0 means no overlap and 1 means complete overlap. For the specific calculation process of the IOU value, first calculate the intersection of the two labels, that is, the area they jointly cover. Then, calculate the union of the two labels, that is, the total area they cover. Finally, divide the intersection by the union to obtain the IOU value. The IOU calculation formula is:

[0047]

[0048] where \(A\cap B\) is the overlapping area of the two object bounding boxes, and \(A\cup B\) is the total area of the two object bounding boxes, as Figure 4 shown.

[0049] Step 4: Considering the special situation of overhead transmission lines, non-professional annotators are prone to reverse annotation during annotation. Moreover, components such as insulator dampers in overhead transmission lines usually appear in pairs, and annotators are very likely to have the phenomenon of bounding box nesting during annotation. To improve the quality of the annotation dataset, AIOU and DIOU are proposed as Figure 5 shown.

[0050] \(AIOU = |\theta\) o -\theta ∩ |\)

[0051]

[0052]

[0053] where \(\theta\) ∩ is the aspect ratio angle of the overlapping part of the annotation box, \(W\) ∩ is the width value of the overlapping part of the annotation box, \(H\) ∩ is the height value of the overlapping part of the annotation box, \(\theta\) i is the aspect ratio angle of the \(i\)-th annotation box, \(W\) i is the width value of the \(i\)-th annotation box, \(H\) i is the height value of the \(i\)-th annotation box; \(d\) is the distance between the centers of the two annotation boxes, and \(S\) ∩ is the area of the overlapping part of the annotation box;

[0054] After the region matching is completed, label object matching is performed, as Figure 6 shown. Randomly select one of the object regions, count all the label categories of this object region, and determine whether it is a unique category. If different category labels appear, compare the AIOU with the set threshold with other labels of objects of this category on this image. If so, it is determined that it is a category reversal scene, and this label is removed. If it is less, it is determined that there is an overlapping occlusion of different-class objects at this time, and calculate whether the DIOU is greater than the threshold. If so, it is classified as a new object.

[0055] If all the labels of this object region are of a unique category, count whether the annotator of this region has a unique label. If all the labels of this region come from different annotators, end the label object matching; if all the labels of this region come from the same annotator, it is determined that there is an overlapping phenomenon of the same-class objects at this time, and calculate whether the DIOU is greater than the threshold. If so, it is classified as a new object.

[0056] Step 5: For the tags matched by the tags, propose a tag integration algorithm to output the final annotation result. To reduce labor costs and improve the accuracy of annotation, an object tag integration algorithm is proposed. First, obtain all the tags of the same object, and set its coordinates as

[0057] BBOX i (X i , Y i , W i , H i )

[0058] Taking the upper left corner of the image as the coordinate origin, with the right direction as the positive Y-axis coordinate axis and the downward direction as the positive X-axis coordinate axis, where X i is the x coordinate value of the upper left corner of the i-th annotation box, Y i is the y coordinate value of the upper left corner of the i-th annotation box, W i is the width value of the i-th annotation box, H i is the height value of the i-th annotation box. Therefore, the four corner coordinates of the annotation box on the plane image are as Figure 7 shown.

[0059] After obtaining the coordinates of the upper left corner and the lower right corner of each annotation box, dynamically generate the weights of the coordinates, and calculate the final coordinates of the object through the weights. The coordinate calculation formula is as follows:

[0060]

[0061] where represents the weight of the x coordinate value of the i-th annotation box, represents the weight of the y coordinate value of the i-th annotation box, represents the weight of the height value h of the i-th annotation box, represents the weight of the width value w of the i-th annotation box. After obtaining the coordinates of the upper left corner and the lower right corner of each annotation box, dynamically generate the weights of the coordinates. The weight calculation formula is as follows:

[0062]

[0063] where u (x，y，h，w) and σ (x，y，h，w) are the mean and variance of (x, y, h, w) respectively. The mean and variance calculation formulas are as follows:

[0064] Mean

[0065] Variance

[0066] Therefore, output the final coordinates of the object:

[0067] BBOX(X, Y, W, H)

[0068]

[0069] Step 6: After the dataset is integrated through annotation review and clustering, output the annotation file format as required to obtain the dataset.

[0070] The above are the preferred embodiments of the present invention. All changes made according to the technical solution of the present invention, when the functions and effects produced do not exceed the scope of the technical solution of the present invention, shall fall within the protection scope of the present invention.

Claims

1. An inspection image multi - person collaborative annotation system and audit management method for overhead transmission lines, characterized in that, First, match the annotation data of multiple people to obtain an annotation box for an object area, judge its category and quantity, solve the problems of frame covering and wrong annotation based on the special environment of overhead transmission lines, identify overlapping targets, achieve a preliminary judgment of annotation quality, and finally derive the final annotation result by the Gaussian integration algorithm; The method specifically includes the following steps: Step 1: Make a dataset for multi-person image annotation; Step 2: Design a label area matching method: First, randomly select a label from all the labels, and define the range it encloses as an object. Then, continuously randomly select labels from the remaining labels, calculate the IOU value with the existing object labels and take the maximum IOU value. If this IOU is greater than the set threshold, it is considered that this label belongs to the label of the corresponding area range; otherwise, it is regarded as a new area until all labels have their own areas; Step 3: Use the label area matching method in Step 2 to perform label area matching; Step 4: Label object matching: First, propose AIOU and DIOU: AIOU = |θ i -θ ∩ | where θ ∩ is the aspect ratio angle of the overlapping part of the bounding box, W ∩ is the width value of the overlapping part of the bounding box, H ∩ is the height value of the overlapping part of the bounding box, θ i is the aspect ratio angle of the i-th bounding box, W i is the width value of the i-th bounding box, H i is the height value of the i-th bounding box, d is the distance between the centers of the two bounding boxes, S ∩ is the area of the overlapping part of the bounding box; Secondly, perform label object matching. The process is as follows: Randomly select one of the object areas, count all the label categories of this object area, and judge whether all the label categories of this object area are unique categories. If different category labels appear, compare its AIOU with other labels of this category object on the image to see if it is greater than the set threshold. If so, it is judged as a wrong annotation of the category and this label is removed; if not, it is judged that there is an overlapping occlusion phenomenon of heterogeneous targets at this time, and calculate whether DIOU is greater than the threshold. If so, it is classified as a new object; if all the label categories of this object area are unique categories, count whether the annotators in this area have unique labels. If all the labels in this area come from different annotators, end the label object matching; if all the labels in this area come from the same annotator, it is judged that there is an overlapping phenomenon of the same category targets at this time, and calculate whether DIOU is greater than the threshold. If so, it is classified as a new object; Step 5: Based on the labels after label matching, propose an object label integration algorithm to output the final annotation result; for the object label integration algorithm, first obtain all the labels of the same object, and set its coordinates as: BBOX i (X i ,Y i ,W i ,H i ) Taking the upper left corner of the image as the coordinate origin, the positive direction of the Y-axis is to the right, and the positive direction of the X-axis is downward, where X i is the x-coordinate value of the upper left corner of the i-th annotation box, Y i is the y-coordinate value of the upper left corner of the i-th annotation box, W i is the width value of the i-th annotation box, H i is the height value of the i-th annotation box; After obtaining the coordinates of the upper left corner and the lower right corner of each annotation box, dynamically generate the weights of the coordinates, and calculate the final coordinates BBOX(X, Y, W, H) of the object through weight calculation. The coordinate calculation formula is as follows: wherein represents the weight of the x coordinate value of the i-th annotation box, represents the weight of the y coordinate value of the i-th annotation box, represents the weight of the height value h of the i-th annotation box, represents the weight of the width value w of the i-th annotation box; Step 6: After the dataset is clustered and integrated through annotation review, output the annotation file format as required to obtain the output dataset.

2. The overhead transmission line inspection image multi-person collaborative annotation system and audit management method according to claim 1, wherein, The value range of IOU is from 0 to 1, where 0 means no overlap and 1 means complete overlap; the specific calculation method of the IOU value is: First, calculate the intersection of the two labels, that is, the area they jointly cover; then, calculate the union of the two labels, that is, the total area they cover; finally, divide the intersection by the union to obtain the IOU value; the IOU calculation formula is: Where A∩B is the overlapping area of the two object bounding boxes, and A∪B is the total area of the two object bounding boxes.

3. The overhead transmission line inspection image multi-person collaborative annotation system and audit management method according to claim 1, characterized in that, The calculation formula of the weight is as follows: where u (x,y,h,w) and σ (x,y,h,w) are the mean and variance of (x, y, h, w) respectively, and the calculation formulas for the mean and variance are as follows:

Images