Multi-type high-voltage tower three-granularity attitude estimation system and method

Abstract

The invention belongs to the technical field of image processing, and discloses a multi-type high-voltage tower three-granularity attitude estimation system and method.The system comprises an attitude estimation model, a PAF-based key point connection module and a high-voltage power transmission tower attitude generation module; the attitude estimation model is used for carrying out key point identification under three specified granularities on the to-be-detected high-voltage power transmission tower image, and a key point identification result comprises key point position information and key point position direction information; the key point connection module based on the PAF is used for constructing key point connection according to the key points and the position direction information under each granularity; and the high-voltage power transmission tower attitude generation module is used for forming the attitude of the high-voltage power transmission tower according to the connecting lines obtained under different granularities. The method not only can effectively estimate the attitude of the high-voltage power transmission tower, but also has good generalization ability.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, relates to posture estimation and safety detection of high-voltage electric towers, and in particular to a three-granularity posture estimation system and method for multiple types of high-voltage electric towers. Background Art

[0002] High-voltage transmission towers play a vital role in the transmission of electricity in modern society and are important infrastructure. The safety status of high-voltage transmission towers can have a significant impact on large areas, so they need to be frequently inspected for safety. These inspections are usually carried out using drone-assisted visual image inspection methods. Generally speaking, towers are designed to accommodate transmission lines, lightning protection lines, and provide support structures, so the key nodes of their safety inspections are often strongly related to the posture structure of the towers. For example, the distribution of key parts such as transmission lines, insulators, and dampers is related to the tower posture design. In view of this, the safety inspection of high-voltage towers does not require drones to take pictures and check each key node separately. Instead, all key nodes can be predicted from the panoramic image of the tower taken by the drone, and then the connection between the key nodes can be established to construct the tower posture. Finally, the predicted key node images can be intercepted for safety inspection. Therefore, it can be regarded as a multi-object posture estimation task; the focus is on estimating the posture of different parts of the high-voltage tower.

[0003] Traditional attitude estimation models, whether following a top-down or bottom-up framework, or using heatmap soft regression or direct coordinate point regression, specifically name all the key nodes of the object. A dedicated heatmap channel is used to determine specific key nodes during the prediction process. However, in the attitude estimation task of high-voltage towers, on the one hand, the various types of towers lead to heterogeneity problems in this task, which challenges the generalization ability of attitude estimation models; on the other hand, how attitude estimation models can make full use of the panoramic images of high-voltage towers taken by drones to solve the confusion problem of key nodes is also a challenge.

[0004] At present, the traditional methods for attitude estimation and key point safety inspection of high-voltage transmission towers in China mainly use single tower images taken by drones for attitude estimation, and each type of key nodes needs to be labeled separately to complete the tower attitude estimation. Gianfranco F et al. proposed a method for attitude estimation of high-voltage towers. By individually marking each specific type of key point and predicting the corresponding tower type, they completed the attitude estimation of five types of towers (see Gianfranco F, Dall'Asta Elisa, Fabrizio D, et al. Quality Assessment of DSMs Produced from UAV Flights Georeferenced with On-Board RTK Positioning [J]. Remote Sensing, 2018, 10 (2): 311. DOI: 10.3390 / rs10020311). Based on the proposed matching principle, Xu L et al. implemented a multi-class object pose estimation method. This method requires specific definition of the key point type of each type of object in the data set, and completes the pose estimation of multi-type objects through data training (see Xu L, Jin S, Zeng W, et al. Pose for Everything: Towards Category-Agnostic Pose Estimation [J]. Springer, Cham, 2022. DOI: 10.48550 / arXiv.2207.10387).

[0005] However, the above-mentioned existing technologies all have the following problems: (1) Due to the traditional attitude estimation method, only the position and category of the key nodes are marked separately during the model training process; and in the process of estimating the tower attitude with the predicted key nodes, the connection between the key nodes is not considered; this makes these traditional attitude estimation models only able to predict a single simple object attitude; when faced with multi-type and significantly heterogeneous high-voltage towers, the generalization ability of the model is insufficient and it is difficult to handle the problem of tower heterogeneity; (2) The traditional attitude estimation method does not fully consider the problem of difficulty in encoding the granularity information in the tower images taken by drones. The key points of high-voltage towers are dense and similar, and the model can easily confuse similar key nodes. Summary of the invention

[0006] The purpose of the present invention is to address the deficiencies in the above-mentioned prior art and provide a three-granularity attitude estimation system for multi-type high-voltage towers. The system can detect key nodes at three different granularities, namely the trunk, lateral support and overall structural details of the tower image taken by a drone, to solve the problem of key node confusion. The system also combines Part Affinity Fields (PAF) to provide position and direction information for the connection between key points, so that the model can overcome the difficulties of insufficient generalization ability of the model and difficulty in handling heterogeneous towers when predicting the attitudes of multi-type tower structures.

[0007] Another object of the present invention is to provide a training method for the above-mentioned three-granularity posture estimation system for multiple types of high-voltage towers.

[0008] The third object of the present invention is to provide a three-granularity posture estimation method for multiple types of high-voltage towers.

[0009] The inventive idea of ​​the present invention is: for the images of power towers taken by drones, the key nodes are distinguished and labeled with three granularities according to the trunk, lateral brackets and overall structural details of the power tower structure, and the feature pyramid structure of HRNET with an enhanced neck structure is used to realize the classification of key points with three granularities; on this basis, in order to solve the problem of insufficient generalization ability of traditional attitude estimation methods in the face of heterogeneous power towers, the method uses PAF (partial affinity field) to provide position and direction information for the connection between the classified key points, thereby realizing the attitude estimation of multiple types of towers; finally, the three-granularity Gaussian heat map of the key nodes and the PAF connected by the key points are combined to generate target structured information.

[0010] In view of the first invention objective, the present invention provides a three-granularity posture estimation system for multiple types of high-voltage towers, including:

[0011] A posture estimation model is used to identify key points at three specified granularities for the image of the high-voltage transmission tower to be tested; the posture estimation model uses an improved HRNER model; the key point recognition result includes key point position information and key point position direction information;

[0012] A key point connection module based on PAF is used to construct key point connections according to key points and position direction information at each granularity; the key point connection module based on PAF includes a connection line construction unit, a connection line score calculation unit and a connection line screening unit;

[0013] A connection line construction unit constructs a number of connection point pairs consisting of a starting point and an end point according to the identified key points, and the line between the starting point and the end point is the connection line;

[0014] The connection line score calculation unit estimates the position between the start point and the end point of any connection point pair, and multiplies the PAF vector of each position with the unit direction vector of the corresponding connection line to obtain the PAF score of the position, and then averages the PAF scores of all estimated positions to obtain the connection line score;

[0015] A connection line screening unit is used to screen out connection lines whose scores exceed a set threshold;

[0016] The high-voltage transmission tower posture generation module is used to construct the posture of the high-voltage transmission tower according to the connection lines obtained at different granularities.

[0017] The above-mentioned three-granularity posture estimation system for multiple types of high-voltage transmission towers, the three granularities are a first granularity based on the main structure of the high-voltage transmission tower, a second granularity based on the main and lateral support structure of the high-voltage transmission tower, and a third granularity based on the overall structure of the high-voltage transmission tower.

[0018] The above-mentioned three-granularity posture estimation system for multi-type high-voltage transmission towers, the improved HRNET model consists of four stages, the first stage is divided into four layers, including 1, 1, 4 and 3 convolution modules respectively; the second stage is divided into three layers, including 1, 4 and 3 convolution modules respectively; the third stage is divided into two layers, including 4 and 3 convolution modules respectively; the fourth stage contains 3 convolution modules; the corresponding layers of the four stages are aligned, and the output of the previous layer of each stage is used as the input of the next layer of all stages; all convolution modules are ResNet residual modules; the output features of the four stages are fused through the feature pyramid to obtain the final output of the model, and the feature pyramid includes three layers, and the output of the previous layer is also used as the input of the next layer.

[0019] The output features of the four stages are C1, C2, C3 and C4 respectively; the feature pyramid includes three layers:

[0020] In the first layer, the feature C4 is upsampled to obtain the feature C4′; the feature C3 and the feature C4′ are concatenated to obtain the feature C34; the feature C34 is convolved to obtain the first output feature;

[0021] In the second layer, the feature C34 is concatenated with the first output feature to obtain feature C34-3; the feature C34-3 is upsampled to obtain feature C3′; the feature C3′ is concatenated with feature C2 to obtain feature C23; the feature C23 is convolved to obtain the second output feature;

[0022] In the third layer, the feature C23 is concatenated with the second output feature to obtain feature C23-2; the feature C23-2 is upsampled to obtain feature C2′; the feature C2′ is concatenated with feature C1 to obtain feature C12; and the feature C12 is convolved to obtain the third output feature.

[0023] The first output feature corresponds to the key point recognition result at the first granularity; the second output feature corresponds to the key point recognition result at the second granularity; the third output feature corresponds to the key point recognition result at the third granularity. The key point recognition result includes key point position information and key point position direction information, and also includes the probability of each key point.

[0024] The above-mentioned PAF-based key point connection module is used to obtain the connection lines serving as the skeleton of the high-voltage transmission tower.

[0025] For the connection line score calculation unit, assume that the candidate connection line is P1P2, the starting point is P1(x1,y1), the end point is P2(x2,y2), and the unit direction vector is L P1P2 , interpolate N points between the line segments P1P2, assuming that the interpolation point is P n (x n ,y n ), and the pose estimation model predicts P n PAF vector of a point Pn for Then P n The PAF score calculation formula for a point on line segment P1P2 is as follows:

[0026]

[0027]

[0028] Then, the connection line score is calculated as follows:

[0029]

[0030] In the above-mentioned three-granularity posture estimation system for multiple types of high-voltage transmission towers, the high-voltage transmission tower posture generation module retains the connection line with the highest score for the connection lines related to the same key point at different granularities.

[0031] In view of the second invention objective, the present invention also provides a training method for the above-mentioned multi-type high-voltage transmission tower three-granularity attitude estimation system, comprising the following steps:

[0032] S1 builds a three-granularity key point dataset of high-voltage transmission towers; this step includes the following sub-steps:

[0033] S11 obtains a number of high-voltage transmission tower image samples;

[0034] S12 respectively marks key points of each high-voltage transmission tower image sample according to the three set granularities to obtain labels of corresponding granularities; and uses all high-voltage transmission tower image samples and the labels of the three corresponding granularities to form a three-granularity key point data set of the high-voltage transmission tower;

[0035] S2 uses the three-granularity key point dataset of high-voltage transmission towers to train the posture estimation model and identify the key points of each sample granularity;

[0036] S3 calculates the loss at the corresponding granularity based on the labels of the samples at each granularity and the key points determined in step S2, and then performs weighted addition of the losses at different granularities to obtain the model loss;

[0037] S4 optimizes the parameters of the posture estimation model according to the model loss, and then returns to step S2;

[0038] Repeat steps S2-S4 until the pose estimation model converges.

[0039] The above step S2 includes the following sub-steps:

[0040] S21 inputs the original image of the sample in the three-granularity key point data set of the high-voltage transmission tower into the posture estimation model, and outputs the candidate key points of each sample at each granularity;

[0041] S22 generates a heat map corresponding to each candidate key point at each granularity of the sample;

[0042] S23 filters the heat map of each candidate key point, and extracts the area with the peak value from the filtered heat map as the candidate connection point, thereby obtaining a plurality of candidate starting points and end points;

[0043] S24 constructs a plurality of connection point pairs consisting of a starting point and an end point, where a line between the starting point and the end point is a connection line, and determines a PAF vector;

[0044] S25 estimates the position between any starting point and end point, and calculates the PAF vector of each interpolated position, and then multiplies it with the unit direction vector of the corresponding connecting line to obtain the PAF score of the position, and then averages the PAF scores of all estimated positions to obtain the connecting line score;

[0045] S26 selects the connection lines whose scores exceed the set threshold, and then determines a number of key points.

[0046] In the above step S24, the PAF vector calculation formula is as follows:

[0047]

[0048]

[0049]

[0050] Where the coordinates (x, y) represent points in the vector field; x1, y1, x2, and y2 represent the horizontal and vertical coordinates of the starting and ending points of the connection; aligning the connection direction with the X-axis direction and taking the starting point as the origin yields the transformed coordinates (x', y'); θ represents the angle of the connection; L represents the length of the connection, and W represents the width range of the connection, which requires additional configuration.

[0051] In the above step S25, the PAF vector of each estimated position is calculated by the above PAF vector formula.

[0052] In step S3, the model loss is expressed as:

[0053]

[0054] Among them, λ i represents the loss coefficient of the i-th particle size; l i represents the loss associated with the i-th granularity.

[0055] For the third invention purpose, the present invention also provides a three-granularity posture estimation method based on multiple types of high-voltage transmission towers, using the above system to perform the following steps including the following steps:

[0056] L1 inputs the image of the high-voltage transmission tower to be tested into the posture estimation model to identify and obtain key points at three specified granularities;

[0057] L2 builds key point connections based on the key points at each granularity through a key point connection module based on PAF; this step includes the following sub-steps:

[0058] L21 constructs a unit through connecting lines. According to the identified key points, it constructs several connecting point pairs consisting of a starting point and an end point. The connecting line between the starting point and the end point is the connecting line.

[0059] L22 estimates the position between the start point and the end point of any connection point pair through the connection line score calculation unit, and multiplies the PAF vector of each interpolation position with the unit direction vector of the corresponding connection line to obtain the PAF score of the position, and then averages the PAF scores of all estimated positions to obtain the connection line score;

[0060] L23 uses a connection line screening unit to screen out connection lines with scores exceeding a set threshold;

[0061] L3 uses the high-voltage transmission tower posture generation module to construct the posture of the high-voltage transmission tower based on the connection lines obtained at different granularities.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] 1) The posture estimation model proposed in the present invention can realize the key point identification of the high-voltage transmission tower at three granularities, and then screen and connect the key points at different granularities through the key point connection module based on PAF, and then obtain the posture estimation of the high-voltage transmission tower through the high-voltage transmission tower posture generation module; the present invention can not only effectively estimate the posture of the high-voltage transmission tower, but also has good generalization ability;

[0064] 2) The attitude estimation model proposed in the present invention classifies and labels key nodes at three granularities according to the main trunk, lateral support, and overall structural details of the tower structure, which can more clearly label each key point in the complex structure, thereby solving the problem of confusion of similar key points;

[0065] 3) The feature pyramid structure of HRNET with enhanced neck structure proposed in the present invention takes into account the obvious progressive relationship between the three-granularity outputs, and uses the feature pyramid structure to fuse feature information to achieve accurate identification of key points of three granularities;

[0066] 4) This method proposes to use PAF to provide positional direction information for the connections between all key points of classification, thereby avoiding the position encoding of key points and having better generalization ability when facing heterogeneous towers. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Schematic diagram of the principle of a three-granularity posture estimation system for multiple types of high-voltage towers provided by an embodiment of the present invention;

[0068] Figure 2 This is a schematic diagram of the improved HRNET model structure;

[0069] Figure 3 It is a schematic diagram of the feature pyramid structure;

[0070] Figure 4 This is a schematic diagram of the training process of the three-granularity posture estimation system for multiple types of high-voltage towers;

[0071] Figure 5 Schematic diagram of labels of different high-voltage transmission towers at three granularities; (a) corresponds to the original high-voltage transmission tower image, (b) corresponds to the label at the first granularity, (c) corresponds to the label at the second granularity, and (d) corresponds to the label at the third granularity;

[0072] Figure 6 A schematic diagram of the key point connection process using PAF vector calculation and construction;

[0073] Figure 7The heat map of the key nodes of the high-voltage tower identified by the embodiment of the present invention and the connection diagram of the key nodes obtained by the PAF vector; wherein (a) corresponds to the first granularity, (b) corresponds to the second granularity, (c) and (d) correspond to the third granularity; (c) and (d) both correspond to the third granularity. There are many key points on the trunk line. In order to facilitate display and prevent overlap, only one diagram is used to display the edge key points;

[0074] Figure 8 This is a diagram showing the posture estimation effect of different types of high-voltage transmission towers according to an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The following will clearly and completely describe the technical solutions of various embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0076] Example

[0077] (I) Three-granularity attitude estimation system for multi-type high-voltage towers

[0078] The current posture estimation and key point safety inspection methods for high-voltage transmission towers directly perform posture estimation using tower images taken by drones, and each type of key nodes needs to be labeled separately, without considering the status of granular information and paying attention to the connection between key nodes. As a result, the existing posture estimation model has insufficient generalization ability, is difficult to handle the heterogeneous problems of high-voltage towers, is difficult to classify and encode, and is prone to confusion in predicting key nodes. The present invention provides a three-granularity posture estimation system for multiple types of high-voltage towers.

[0079] In this embodiment, the image of the high-voltage transmission tower is collected by a drone. Figure 1 As shown, the drone is equipped with an image sensor (such as a camera), and the image sensor is installed on a camera gimbal to ensure the stability of the image sensor. When the drone automatically patrols the line, the image sensor (camera) takes a distant view of the power tower, and the storage card stores the captured image. Finally, the signal base station transmits the captured distant view of the high-voltage transmission tower to the computer.

[0080] In a specific implementation, the three-granularity posture estimation system for multiple types of high-voltage towers provided in this embodiment includes a posture estimation model, a key point connection module based on PAF, and a high-voltage transmission tower posture generation module.

[0081] (1) Posture Estimation Model

[0082] Since the present invention focuses on solving the problems of confusion of similar key nodes in the attitude estimation model, poor model generalization ability, and difficulty in meeting the heterogeneous conditions of power towers, the improvement of the attitude estimation model is also a focus of the present invention.

[0083] The posture estimation model is used to identify key points of the high-voltage transmission tower images under three specified granularities.

[0084] The posture estimation model uses the improved HRNER model. Figure 2 As shown in the figure, the improved HRNET model consists of four stages. The first stage is divided into four layers, including 1 first convolution module, 1 second convolution module, 4 third convolution modules set in sequence, and 3 fourth convolution modules set in sequence; the second stage is divided into three layers, including 1 fifth convolution module, 4 sixth convolution modules set in sequence, and 3 seventh convolution modules set in sequence; the third stage is divided into two layers, including 4 eighth convolution modules set in sequence and 3 ninth convolution modules set in sequence; the fourth stage contains 3 tenth convolution modules set in sequence. The corresponding layers of the four stages are aligned, and the output of the previous layer of each stage is used as the input of the next layer of all stages. The output of the first convolution module is used as the input of the second and fifth convolution modules at the same time; the output of the second and fifth convolution modules is used as the input of 4 third convolution modules, 4 sixth convolution modules, and 4 eighth convolution modules at the same time; the output of 4 third convolution modules, 4 sixth convolution modules, and 4 eighth convolution modules is used as the input of 3 fourth convolution modules, 3 seventh convolution modules, 3 ninth convolution modules, and 3 tenth convolution modules at the same time. The structure of each convolution module is the same, and they are all ResNet residual modules. The last four output feature layers of different sizes are set to 48, 96, 192, and 384 channels respectively.

[0085] The output features of the four stages are fused through the feature pyramid to obtain the final output of the model. The feature pyramid consists of three layers, and the output of the previous layer is also used as the input of the next layer. Figure 3 As shown, the output features of the four stages are C1, C2, C3 and C4 respectively; the feature pyramid includes three layers, and the output channel sizes of the three layers are 15, 15 and 92 respectively.

[0086] The specific settings of the three layers are:

[0087] In the first layer, the feature C4 is upsampled to obtain the feature C4′; the feature C3 and the feature C4′ are concatenated to obtain the feature C34; the feature C34 is convolved to obtain the first output feature;

[0088] In the second layer, the feature C34 is concatenated with the first output feature to obtain feature C34-3; the feature C34-3 is upsampled to obtain feature C3′; the feature C3′ is concatenated with feature C2 to obtain feature C23; the feature C23 is convolved to obtain the second output feature;

[0089] In the third layer, the feature C23 is concatenated with the second output feature to obtain feature C23-2; the feature C23-2 is upsampled to obtain feature C2′; the feature C2′ is concatenated with feature C1 to obtain feature C12; and the feature C12 is convolved to obtain the third output feature.

[0090] The first output feature corresponds to the key point recognition result at the first granularity; the second output feature corresponds to the key point recognition result at the second granularity; the third output feature corresponds to the key point recognition result at the third granularity. The key point recognition result includes the key point position information and the position direction information of the key point, as well as the probability of each key point.

[0091] In this embodiment, in the improved HRNET model, the image input size is set to 1024×1024, and the corresponding three granularity sizes are 64×64, 128×128, and 256×256, respectively.

[0092] (2) Key point connection module based on PAF

[0093] The key point connection module based on PAF is used to build key point connections according to the key points and position direction information at each granularity.

[0094] The key point connection module based on PAF includes a connection line construction unit, a connection line score calculation unit and a connection line screening unit.

[0095] The connection line construction unit constructs a number of connection point pairs consisting of a starting point and an end point based on the identified key points, and the line between the starting point and the end point is the connection line.

[0096] The connection line score calculation unit estimates the position between the starting point and the end point of any connection point pair, and multiplies the PAF vector of each position with the unit direction vector of the corresponding connection line to obtain the PAF score of the position, and then averages the PAF scores of all estimated positions to obtain the connection line score. In a specific implementation, all candidate starting points and end points are iterated, and an interpolation method (such as integral interpolation) is used to estimate the position between the starting point and the end point. The PAF vector of the interpolation position is multiplied with the unit direction vector of the connection line from the starting point to the end point to obtain the PAF score of the position, and then the PAF scores of all estimated positions are averaged to obtain the score of the connection line.

[0097] Assume that the candidate connecting line is P1P2, the starting point is P1(x1,y1), the end point is P2(x2,y2), and the unit direction vector is L P1P2 , interpolate N points (10 in this embodiment) between the line segments P1P2, assuming that the interpolation point is P n (x n ,y n ), and the pose estimation model predicts P n PAF vector of a point Pn for Then P n The PAF score calculation formula for a point on line segment P1P2 is as follows:

[0098]

[0099]

[0100] Then, the connection line score is calculated as follows:

[0101]

[0102] The connection line screening unit screens out the connection lines whose scores exceed the set threshold. If the score exceeds the threshold, it is predicted as a tower skeleton connection; otherwise, it is discarded.

[0103] The high-voltage transmission tower posture generation module is used to construct the posture of the high-voltage transmission tower based on the connection lines obtained at different granularities. For the connection lines related to the same key point at different granularities, the connection line with the highest score is retained.

[0104] The working principle of the three-granularity posture estimation system for multiple types of high-voltage transmission towers provided in this embodiment is: first, the posture estimation model is trained, and then the three-granularity key nodes of the high-voltage tower are accurately identified using the trained posture estimation model. Finally, the posture estimation of the high-voltage tower is completed based on the identified key nodes combined with the connection between the key points provided by PAF to provide position and direction information, and the key point images are captured for safety inspection.

[0105] (II) Training of a three-granularity attitude estimation system for multiple types of high-voltage towers

[0106] This embodiment also provides a training method for the above-mentioned multi-type high-voltage transmission tower three-granularity posture estimation system, such as Figure 4 As shown, the following steps are included:

[0107] S1 builds a three-granularity key point dataset of high-voltage transmission towers; this step includes the following sub-steps:

[0108] S11 obtains several high-voltage transmission tower image samples.

[0109] The drone given above can be used to collect several high-voltage transmission tower image samples.

[0110] S12 respectively marks key points of each high-voltage transmission tower image sample according to the three set granularities to obtain labels of corresponding granularities; and uses all high-voltage transmission tower image samples and the labels of the corresponding three granularities to form a three-granularity key point data set of the high-voltage transmission tower.

[0111] The three granularities are a first granularity based on the main structure of the high-voltage transmission tower, a second granularity based on the main and transverse support structure of the high-voltage transmission tower, and a third granularity based on the overall structure of the high-voltage transmission tower.

[0112] like Figure 5 As shown in the figure, the label making process at different granularities is as follows: (1) extract several key points in the longitudinal and transverse directions of the high-voltage transmission tower and connect them to obtain the corresponding labels of the first granularity; (2) extract several key points in the longitudinal and transverse directions of the high-voltage transmission tower, and at the same time extract several key points on each transverse support of the high-voltage transmission tower (for example, several key points around the edge of the transverse support), and connect the key points to obtain the corresponding labels of the second granularity; (3) based on the overall structure of the high-voltage transmission tower, extract several key points on the longitudinal and transverse supports of the high-voltage transmission tower (for example, several key points around the edge of the transverse support, several key points around the edge of the longitudinal support, and the intersection key points of the transverse support and the longitudinal support), and connect the key points to obtain the corresponding labels of the third granularity. There is no need to encode the position of the key points for labels at different granularities.

[0113] S2 uses the three-granularity key point dataset of high-voltage transmission towers to train the posture estimation model and identify the key points of each sample granularity.

[0114] This step includes the following sub-steps:

[0115] S21 inputs the original image of the sample in the three-granularity key point data set of the high-voltage transmission tower into the posture estimation model, and outputs the candidate key points of each sample at each granularity.

[0116] The identified candidate key point information includes position information, probability and position direction information (ie, PAF vector).

[0117] S22 generates a heat map corresponding to each candidate key point at each sample granularity.

[0118] During training, the key point heat map is represented as:

[0119]

[0120] in, is the true value of the key point, σ is the standard deviation, which is used to control the size of the bright area (area with large value); Represents the position value at the heat map (x, y). When the position is close to (x, y), its value tends to be close to 1, indicating high confidence. On the contrary, when the position deviates from (x, y), its value tends to be close to 0, indicating low confidence.

[0121] In this embodiment, for three different particle sizes, the standard deviations can be 2, 2, and 1.5 respectively.

[0122] S23 filters the heat map of each candidate key point, and extracts the area with a peak value from the filtered heat map as a candidate connection point, thereby obtaining a plurality of candidate starting points and end points.

[0123] In this step, the heat map generated by the above-mentioned specified key points is Gaussian filtered to obtain their probability distribution map (ie, Gaussian heat map); then, the area with peak value is extracted from the filtered Gaussian heat map as the candidate connection point, thereby obtaining multiple candidate starting points and end points.

[0124] S24 constructs a number of connection point pairs consisting of a starting point and an end point, where the line between the starting point and the end point is the connection line, and determines the PAF vector.

[0125] The PAF consists of two vectors for each pair of related key points: one is the x-direction component and its unit vector, and the other is the y-direction component and its unit vector. The PAF vector calculation unit calculates the PAF vector of any start and end point according to the following formula:

[0126]

[0127]

[0128]

[0129] In the formula, the coordinates (x, y) represent the points in the vector field; x1, y1, x2, and y2 represent the horizontal and vertical coordinates of the starting and ending points of the connection; aligning the connection direction with the X-axis direction and taking the starting point as the origin will give the transformed coordinates (x', y'); θ represents the angle of the connection line; L represents the connection length, and W represents the connection width range, which requires additional configuration.

[0130] In this embodiment, for three different particle sizes, W can be 2.5, 3 and 2 respectively.

[0131] S25 estimates the position between any starting point and end point, and calculates the PAF vector of each interpolated position, which is then multiplied by the unit direction vector of the corresponding connecting line to obtain the PAF score of the position, and then the PAF scores of all estimated positions are averaged to obtain the connecting line score.

[0132] In this step, for any starting point and end point, integral interpolation is used to interpolate and estimate the position between the starting point and the end point.

[0133] Then, the PAF vector of each interpolation position is calculated according to the PAF vector formula given above; the PAF vector of the interpolation position is then multiplied by the directional unit vector of the connecting line from the starting point to the end point to obtain the PAF score of the position (the calculation formula is shown above), and then the PAF scores of all estimated positions are averaged to obtain the score of each connecting line.

[0134] S26 selects the connection lines whose scores exceed the set threshold, and then determines a number of key points.

[0135] If the score of the connection line exceeds the set threshold, it is predicted as a tower skeleton connection; otherwise, it is discarded. The key points at both ends of the connection line are the selected key points.

[0136] S3 calculates the loss at the corresponding granularity based on the labels of the samples at each granularity and the key points determined in step S2, and then performs weighted addition of the losses at different granularities to obtain the model loss.

[0137] The model loss is expressed as:

[0138]

[0139] Among them, λ i represents the loss coefficient of the i-th particle size, and the weights of the first particle size, the second particle size, and the third particle size are 0.2, 0.4, and 1 respectively; l i Represents the loss associated with the i-th granularity, which is calculated by adding the probability loss (calculated by the key point label position and the predicted position) and the PAF vector loss (calculated by the PAF vector of the interpolated position calculated in step S25 and the PAF vector predicted by the posture estimation model).

[0140] All losses in this example are evaluated using the MSE loss function, while both heatmaps and PAF help generate the output and participate in calculating the loss.

[0141] S4 optimizes the parameters of the posture estimation model according to the model loss, and then returns to step S2.

[0142] According to the model loss, the parameters of the posture estimation model are optimized through the gradient optimization algorithm.

[0143] Repeat steps S2-S4 until the pose estimation model converges.

[0144] Whether the posture estimation model has converged can be determined by the number of training iterations, for example, the iteration threshold is set to 200. When the number of training iterations reaches 200, it can be considered that the posture estimation model has converged.

[0145] (III) Test of three-granularity attitude estimation system for multi-type high-voltage towers

[0146] This embodiment also provides a three-granularity posture estimation method based on multiple types of high-voltage transmission towers, using the above system to perform the following steps:

[0147] L1 inputs the image of the high-voltage transmission tower to be tested into the posture estimation model to identify and obtain key points at three specified granularities.

[0148] In this step, the input to be tested is fed into the trained posture estimation model, and key points at three granularities are output; the key point information includes position information and PAF vector.

[0149] The posture estimation model also outputs the PAF vector of each pixel at different granularities.

[0150] In this embodiment, the key points at different granularities are identified as follows: Figure 7 As shown in the figure, it can be seen that the above posture estimation model can accurately estimate the key points of different high-voltage transmission towers.

[0151] L2 builds key point connections based on the key points at each granularity through a key point connection module based on PAF; this step includes the following sub-steps:

[0152] L21 constructs a unit through connecting lines. According to the identified key points, it constructs several connecting point pairs consisting of starting points and end points. The connecting line between the starting point and the end point is the connecting line.

[0153] L22 estimates the position between the start and end points of any connection point pair through the connection line score calculation unit, and multiplies the PAF vector of each interpolation position with the unit direction vector of the corresponding connection line to obtain the PAF score of the position, and then averages the PAF scores of all estimated positions to obtain the connection line score.

[0154] In this step, for the start point and end point of any connection point pair, integral interpolation is used to interpolate and estimate the position between the start point and the end point.

[0155] Then, according to the formula given above, the PAF vector of the interpolated position is multiplied by the unit vector of the connecting line from the starting point to the end point to obtain the PAF score of the position, and then the PAF scores of all estimated positions are averaged to obtain the score of each connecting line. The obtained PAF connection is as follows Figure 7 The X-axis component and Y-axis component of PAF are shown in the figure.

[0156] L23 uses a connection line screening unit to screen out connection lines whose scores exceed a set threshold.

[0157] If the score of a connection line exceeds the set threshold, it is predicted as a tower skeleton connection; otherwise, it is discarded.

[0158] L3 uses the high-voltage transmission tower posture generation module to construct the posture of the high-voltage transmission tower based on the connection lines obtained at different granularities.

[0159] In this step, the connection lines at different granularities are combined together; for the connection lines related to the same key point at different granularities, the connection line with the highest score is retained to obtain the posture of the high-voltage transmission tower, such as Figure 8 shown.

[0160] In summary, the three-granularity attitude estimation system and method for multiple types of high-voltage towers provided by the present invention have the following advantages:

[0161] (1) The key nodes of the traditional attitude estimation model generally need to be classified according to different physical objects (such as insulators, dampers, transmission line connection points, etc. of the tower), without considering the existence of similar key points in the complex structure. Therefore, it is easy to confuse the key points when facing a complex tower structure with similar key points. The attitude estimation model proposed in the present invention classifies and labels the key nodes at three granularities according to the trunk, lateral support, and overall structural details of the tower structure, which can more clearly label each key point in the complex structure, thereby solving the problem of confusion of similar key points.

[0162] (2) Compared with the present invention, the traditional posture estimation model does not have an enhanced neck structure when using HRNET as the backbone of the key point recognition network, and cannot take into account the obvious progressive relationship between the three-granularity outputs, which makes the key point recognition effect worse;

[0163] (3) The traditional attitude estimation model directly connects the predicted key point type and the corresponding position code to achieve the attitude estimation of the tower. This has the disadvantages of complex and difficult position coding and poor generalization ability when facing heterogeneous towers. This method proposes to use PAF to provide position direction information for the connection between all key points of the classification, thereby avoiding the position coding of key nodes and having better generalization ability when facing heterogeneous towers.

[0164] Those skilled in the art will appreciate that the embodiments herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A three-granularity attitude estimation system for multi-type high-voltage transmission towers, characterized in that: include: The pose estimation model is used to identify key points of the high-voltage transmission tower image under test at three specified granularities; The posture estimation model uses an improved HRNER model; the key point recognition result includes key point position information and key point position direction information; A key point connection module based on PAF is used to construct key point connections according to key points and position direction information at each granularity; the key point connection module based on PAF includes a connection line construction unit, a connection line score calculation unit and a connection line screening unit; A connection line construction unit constructs a number of connection point pairs consisting of a starting point and an end point according to the identified key points, and the line between the starting point and the end point is the connection line; The connection line score calculation unit estimates the position between the start point and the end point of any connection point pair, and multiplies the PAF vector of each position with the unit direction vector of the corresponding connection line to obtain the PAF score of the position, and then averages the PAF scores of all estimated positions to obtain the connection line score; A connection line screening unit is used to screen out connection lines whose scores exceed a set threshold; The high-voltage transmission tower posture generation module is used to construct the posture of the high-voltage transmission tower according to the connection lines obtained at different granularities.

2. The three-granularity posture estimation system for multi-type high-voltage transmission towers according to claim 1 is characterized in that: The three granularities are a first granularity based on the main structure of the high-voltage transmission tower, a second granularity based on the main and transverse support structures of the high-voltage transmission tower, and a third granularity based on the overall structure of the high-voltage transmission tower.

3. The three-granularity posture estimation system for multi-type high-voltage transmission towers according to claim 1 is characterized in that: The improved HRNET model consists of four stages. The first stage is divided into four layers, including 1, 1, 4 and 3 convolution modules respectively; the second stage is divided into three layers, including 1, 4 and 3 convolution modules respectively; the third stage is divided into two layers, including 4 and 3 convolution modules respectively; the fourth stage includes 3 convolution modules; the output of the previous layer of each stage is used as the input of the next layer of all stages; the output features of the four stages are fused through a feature pyramid to obtain the final output of the model. The feature pyramid includes three layers, and the output of the previous layer is also used as the input of the next layer.

4. The three-granularity posture estimation system for multiple types of high-voltage transmission towers according to claim 3 is characterized in that: The output features of the four stages are C1, C2, C3 and C4 respectively; the feature pyramid includes three layers: In the first layer, the feature C4 is upsampled to obtain feature C4′; the feature C3 and feature C4′ are concatenated to obtain feature C34; The feature C34 is convolved to obtain a first output feature; In the second layer, the feature C34 is concatenated with the first output feature to obtain feature C34_3; the feature C34_3 is upsampled to obtain feature C3′; the feature C3′ is concatenated with feature C2 to obtain feature C23; the feature C23 is convolved to obtain the second output feature; In the third layer, the feature C23 is concatenated with the second output feature to obtain feature C23_2; the feature C23_2 is upsampled to obtain feature C2′; the feature C2′ is concatenated with feature C1 to obtain feature C12; and the feature C12 is convolved to obtain the third output feature.

5. The three-granularity posture estimation system for multiple types of high-voltage transmission towers according to claim 1 is characterized in that: For the connection line score calculation unit, assume that the candidate connection line segment is P1P2, the starting point is P1(x1,y1), the end point is P2(x2,y2), and the unit vector is L P1P2 , interpolate N points between the line segments P1P2, assuming that the interpolation point is P n (x n ,y n ), and the pose estimation model predicts P n PAF vector of a point Pn for Then P n The PAF score calculation formula for a point on line segment P1P2 is as follows: Then, the connection line score is calculated as follows:

6. The three-granularity posture estimation system for multiple types of high-voltage transmission towers according to claim 1 is characterized in that: The high-voltage transmission tower posture generation module retains the connection line with the highest score for the connection lines related to the same key point at different granularities.

7. The training method for a three-granularity posture estimation system for multiple types of high-voltage transmission towers according to any one of claims 1 to 6, characterized in that: The following steps are involved: S1 builds a three-granularity key point dataset of high-voltage transmission towers; this step includes the following sub-steps: S11 obtains a number of high-voltage transmission tower image samples; S12 respectively marks key points of each high-voltage transmission tower image sample according to the three set granularities to obtain labels of corresponding granularities; and uses all high-voltage transmission tower image samples and the labels of the three corresponding granularities to form a three-granularity key point data set of the high-voltage transmission tower; S2 uses the three-granularity key point dataset of high-voltage transmission towers to train the posture estimation model and identify the key points of each sample granularity; S3 calculates the loss at the corresponding granularity based on the labels of the samples at each granularity and the key points determined in step S2, and then performs weighted addition of the losses at different granularities to obtain the model loss; S4 optimizes the parameters of the posture estimation model according to the model loss, and then returns to step S2; Repeat steps S2-S4 until the pose estimation model converges.

8. The training method for a three-granularity posture estimation system for multiple types of high-voltage transmission towers according to claim 7 is characterized in that: Step S2 includes the following sub-steps: S21 inputs the original image of the sample in the three-granularity key point data set of the high-voltage transmission tower into the posture estimation model, and outputs the candidate key points of each sample at each granularity; S22 generates a heat map corresponding to each candidate key point at each granularity of the sample; S23 filters the heat map of each candidate key point, and extracts the area with the peak value from the filtered heat map as the candidate connection point, thereby obtaining a plurality of candidate starting points and end points; S24 constructs a plurality of connection point pairs consisting of a starting point and an end point, where a line between the starting point and the end point is a connection line, and determines a PAF vector; S25 estimates the position between any starting point and end point, and calculates the PAF vector of each interpolated position, and then multiplies it with the unit direction vector of the corresponding connecting line to obtain the PAF score of the position, and then averages the PAF scores of all estimated positions to obtain the connecting line score; S26 selects the connection lines whose scores exceed the set threshold, and then determines a number of key points.

9. The training method for a three-granularity posture estimation system for multiple types of high-voltage transmission towers according to claim 8 is characterized in that: In step S3, the model loss is expressed as: Among them, λ i represents the loss coefficient of the i-th particle size; l i represents the loss associated with the i-th granularity.

10. A three-granularity attitude estimation method based on multi-type high-voltage transmission towers, characterized in that: Using the system according to any one of claims 1 to 6 is carried out according to the following steps: L1 inputs the image of the high-voltage transmission tower to be tested into the posture estimation model to identify and obtain key points at three specified granularities; L2 builds key point connections based on the key points at each granularity through a key point connection module based on PAF; this step includes the following sub-steps: L21 constructs a unit through connecting lines. According to the identified key points, it constructs several connecting point pairs consisting of a starting point and an end point. The connecting line between the starting point and the end point is the connecting line. L22 estimates the position between the start point and the end point of any connection point pair through the connection line score calculation unit, and multiplies the PAF vector of each interpolation position with the unit direction vector of the corresponding connection line to obtain the PAF score of the position, and then averages the PAF scores of all estimated positions to obtain the connection line score; L23 uses a connection line screening unit to screen out connection lines with scores exceeding a set threshold; L3 uses the high-voltage transmission tower posture generation module to construct the posture of the high-voltage transmission tower based on the connection lines obtained at different granularities.

Images