Method for automatically generating three-dimensional model of power transmission line construction equipment

Abstract

A method for automatically generating a three-dimensional model of power transmission line construction equipment comprises the following steps: performing target detection on a to-be-detected image based on an SSD network model and generating candidate boxes, and screening the candidate boxes by using a non-maximum suppression method to obtain a to-be-reconstructed image; the method comprises the following steps of: performing feature point extraction and matching on any two to-be-reconstructed images by using an SURF algorithm and an AKAZE algorithm to obtain a first feature point matching set and a second feature point matching set, and finally performing three-dimensional reconstruction on the power transmission line construction equipment based on the first feature point matching set and the second feature point matching set to generate a three-dimensional model of the power transmission line construction equipment. According to the method, on the basis of extracting the main feature points by using the SURF algorithm, the edge feature points are extracted by using the AKAZE algorithm, so that the extracted feature points are more comprehensive, and the subsequent three-dimensional model reconstruction precision is further improved.

Description

Technical Field

[0001] The invention belongs to the technical field of automatic generation of three-dimensional models, and in particular relates to a method for automatically generating a three-dimensional model of power transmission line construction equipment. Background Art

[0002] With the rapid development of digital technology, the construction of digital twin power grids is of great significance to the construction of new power systems, and the construction of equipment three-dimensional models is a key link in realizing digital twin power grids. As an important component of the power system, the construction of transmission lines is of key significance to the operation and stability of the entire system. There are many types of transmission line construction equipment and the construction site environment is complex, which poses severe challenges to the recognition of construction equipment images. At the same time, some construction equipment has weak texture and single surface color, which affects the extraction of feature points in the subsequent three-dimensional reconstruction process, resulting in poor visualization of the generated three-dimensional model. Therefore, how to accurately identify transmission line construction equipment and improve the visualization of subsequent three-dimensional generated models is a problem worth exploring.

[0003] The paper "Identification and Fault Detection of Anti-bird Thorn Components in Transmission Lines Based on Deep Convolutional Neural Network" selects the faulty anti-bird thorn area in the power inspection image based on the YOLOv3 target detection network, and realizes the fault detection based on the Resnet152 feature extraction network. The paper "Improved SSD Photovoltaic Module Hot Spot Defect Detection Method" improves the SSD algorithm by replacing the backbone neural network and adding an attention mechanism, and realizes the rapid detection of photovoltaic modules based on the improved SSD algorithm. The above documents show that introducing deep learning into image target detection can better capture the complex features in the image, which is conducive to extracting equipment information, and thus achieves better performance in the recognition and detection tasks. However, the deep neural network in the above documents has the problems of complex network structure and long calculation time, and too many network layers will increase the possibility of feature information loss, making it difficult to ensure the accuracy of target detection. At present, there are existing documents that introduce lightweight networks into deep neural networks to solve the problem of many network model parameters. The paper "Lightweight Yolov4 object detection algorithm integrating ECA mechanism" proposes a lightweight Yolov4 object detection network based on EfficientNet-B0, and introduces the Soft-NMS algorithm in the final detection processing stage to reduce the missed detection rate in dense areas. However, the above paper still uses a constant NMS threshold to screen candidate boxes in the detection box processing, which has certain limitations when facing images with objects of different density.

[0004] In the field of 3D reconstruction, 3D reconstruction technology based on the SFM principle has attracted extensive research by relevant scholars due to its advantages such as low cost and high efficiency. The document "Research on Optimization of 3D Reconstruction Method of Urban Cable Tunnels Based on SFM" constructs a 3D model of a cable tunnel with complete internal texture information based on the SFM 3D reconstruction method. The document "Fast SIFT Feature Matching Algorithm Based on Geometric Transformation" is based on the SFM 3D reconstruction method and combines the SIFT algorithm to complete feature extraction, feature matching and other operations to achieve 3D model reconstruction. However, the above documents are not comprehensive enough in feature point extraction and ignore the feature information of the edge of the image, resulting in poor 3D reconstruction accuracy. Summary of the invention

[0005] The purpose of the present invention is to address the above-mentioned problems existing in the prior art and to provide a method for automatically generating a three-dimensional model of power transmission line construction equipment by using the AKAZE algorithm to extract edge feature points based on the extraction of main feature points using the SURF algorithm, so that the extracted feature points are more comprehensive, thereby improving the accuracy of three-dimensional model reconstruction.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] A method for automatically generating a three-dimensional model of a power transmission line construction equipment, the method comprising the following steps:

[0008] S1. Acquire the image to be detected of the power transmission line construction equipment;

[0009] S2, first perform target detection on the image to be detected based on the SSD network model and generate candidate boxes, then use the non-maximum suppression method to screen the candidate boxes to obtain the image to be reconstructed;

[0010] S3, using the SURF algorithm to extract and match feature points of any two images to be reconstructed to obtain a first feature point matching set, and using the AKAZE algorithm to extract and match feature points of any two images to be reconstructed to obtain a second feature point matching set;

[0011] S4. Perform three-dimensional reconstruction of the transmission line construction equipment based on the first feature point matching set and the second feature point matching set to generate a three-dimensional model of the transmission line construction equipment.

[0012] In step S2, the SSD network model is an improved SSD network model. Specifically, the improved SSD network model is as follows: the ShufflenetV2 lightweight network is first used as the backbone network of the SSD algorithm to perform preliminary feature extraction on the image to be detected, and the features obtained by the preliminary feature extraction are downsampled using the feature extraction layer to obtain a multi-scale feature map, and then the multi-scale feature map is feature fused using a bidirectional weighted pyramid network to obtain an initial feature map, and finally the initial feature map is used to perform target detection and generate candidate boxes using the output layer, and the last layer of the ShufflenetV2 lightweight network is replaced by a convolutional layer from the original fully connected layer, and the feature extraction layer consists of four convolutional layers.

[0013] The bidirectional weighted pyramid network is an improved bidirectional weighted pyramid network. The improved bidirectional weighted pyramid network is specifically as follows: the input multi-scale feature map is subjected to feature fusion through a top-down path and a bottom-up path to obtain a new feature layer, and the new feature layer is subjected to a dilated convolution to obtain an initial feature map; wherein the upsampling method of the top-down path is replaced by deconvolution instead of traditional linear interpolation; the top-down path is specifically as follows:

[0014]

[0015] In the above formula, represents the intermediate transition feature of the i-th layer on the top-down path; w1 and w2 are the input weights of the i-th layer and the i+1-th layer on the top-down path respectively; are the inputs of the i-th layer and the i+1-th layer on the top-down path respectively; ε represents the hyperparameter used to prevent the gradient from disappearing;

[0016] The bottom-up path is specifically:

[0017]

[0018] In the above formula, represents the last output feature of the i-th layer on the bottom-up path; w2′ represents the output weight of the intermediate transition feature of the current layer; w3′ represents the output weight of the i-1-th layer on the bottom-up path; w1′ represents the input weight of the i-th layer; represents the last output feature of the i-1th layer; DC represents the hole convolution operation; Conv represents the convolution operation.

[0019] An improved CBAM is introduced before the last layer of the ShufflenetV2 lightweight network, between the last layer of the ShufflenetV2 lightweight network and the first convolution layer of the feature extraction layer, and between two adjacent convolution layers in the feature extraction layer; the improved CBAM includes a first branch and a second branch connected in series, the first branch includes a first channel attention module and a first spatial attention module, the feature map adjusted by the first channel attention module is the input of the first spatial attention module, and the output of the first spatial attention module is the output of the first branch, the second branch includes a second spatial attention module and a second channel attention module, the feature map adjusted by the second spatial attention module is the input of the second channel attention module, and the output of the second channel attention module is the output of the second branch, the outputs of the first branch and the second branch are combined to obtain the output of the improved CBAM; the combination formula is:

[0020] F′=F′ c +F s ;

[0021] In the above formula, F′, F′ c 、F s They represent the feature maps output by the improved CBAM, the second branch, and the first branch, respectively.

[0022] In step S2, the non-maximum suppression method is an improved non-maximum suppression method, and the specific steps are:

[0023] A1. For a set of candidate boxes of the same type, calculate the confidence of all candidate boxes in the set;

[0024] A2. Select the candidate box with the highest confidence as the best candidate box, calculate the intersection-over-union ratio between the best candidate box and the remaining candidate boxes in the set, and update the confidence of the remaining candidate boxes based on the intersection-over-union ratio calculation result:

[0025]

[0026] In the above formula, b m represents the best candidate box; b i is the i-th candidate box among the remaining candidate boxes; s i 、s i ′ represents the confidence of the i-th candidate box in the remaining candidate boxes before and after the update; σ is the continuous penalty coefficient; IoU(b m , b i ) represents the candidate box b m With candidate box b i The intersection and union ratio between them; b m ∩b i Represents the candidate box b m With candidate box bi The area of ​​the overlapping part; b m ∪b i Represents the candidate box b m With candidate box b i The combined area; N t is a fixed threshold value set;

[0027] A3. Adjust the confidence of the remaining candidate boxes again through linear interpolation:

[0028]

[0029] In the above formula, is the confidence of the i-th candidate box in the remaining candidate boxes after further adjustment; α is the scale parameter;

[0030] A4, groups the remaining candidate boxes into a new set, returns to A2 and starts a new round of calculation until the calculation stop condition is reached;

[0031] A5. For the best candidate box obtained in each round, select the best candidate box with the largest confidence as the final candidate box of this type of candidate box.

[0032] In step S3, the feature point extraction and matching of the two images to be reconstructed using the SURF algorithm is specifically as follows:

[0033] B1. Preprocess the two images to be reconstructed, then detect the feature points in the two images to be reconstructed by finding the local maximum of the Hessian matrix, and establish feature descriptions of the two images to be reconstructed based on the rBRIEF feature descriptor;

[0034] B2. Perform feature point matching based on the Hamming distance to obtain a first feature point matching set, specifically: calculate the Hamming distance between the feature point P1 in one image to be reconstructed and all the feature points in another image to be reconstructed, and assume that the feature point in the other image to be reconstructed that has the closest Hamming distance to the feature point P1 is P2, and the feature point with the second closest Hamming distance is P3. If the ratio of the closest Hamming distance to the second closest Hamming distance is less than a preset threshold, it is considered that the feature point P1 in one image to be reconstructed matches the feature point P2 in the other image to be reconstructed successfully; the expression of the Hamming distance is:

[0035]

[0036] In the above formula, D ham (u,v) represents the Hamming distance between u and v; u and v are the feature descriptor vectors of one image to be reconstructed and another image to be reconstructed respectively; v i 、u iare respectively the i-th component in the feature descriptor vector of one image to be reconstructed and another image to be reconstructed; n is the vector dimension of the feature descriptor.

[0037] In step S3, the feature point extraction and matching of the two images to be reconstructed using the AKAZE algorithm is specifically as follows:

[0038] C1. Preprocess the two images to be reconstructed first, then detect the feature points in the two images to be reconstructed by finding the local maximum of the Hessian matrix, and establish the feature description of the two images to be reconstructed based on the FREAK feature descriptor;

[0039] C2. Feature point matching is performed based on cosine similarity to obtain a second feature point matching set, specifically: the cosine similarity between the feature point P1 in one image to be reconstructed and all the feature points in another image to be reconstructed is calculated, and the feature point with the closest cosine similarity to the feature point P1 in the other image to be reconstructed is set to P2, and the feature point with the second closest cosine similarity is set to P3. If the ratio of the closest cosine similarity to the second closest cosine similarity is less than a preset threshold, it is considered that the feature point P1 in one image to be reconstructed matches the feature point P2 in the other image to be reconstructed successfully; the expression of the cosine similarity is:

[0040]

[0041] In the above formula, S(R a ,R b ) represents R a With R b The cosine similarity of R a , R b are the feature descriptor vectors of one image to be reconstructed and another image to be reconstructed respectively; R ai , R bi are respectively the i-th component of the feature descriptor vector of one image to be reconstructed and another image to be reconstructed; n is the vector dimension of the feature descriptor.

[0042] The preprocessing is Gaussian filtering.

[0043] Step S3 also includes using the PROSAC algorithm to eliminate erroneous matching points in the first feature point matching set and the second feature point matching set.

[0044] Step S4 is specifically as follows: obtaining the internal and external parameters of the camera, mapping the first feature point matching set and the second feature point matching set from the two-dimensional coordinate system to the three-dimensional coordinate system to form a three-dimensional sparse point cloud, using the PMVS algorithm to densify the three-dimensional sparse point cloud, completing the surface reconstruction of the transmission line construction equipment, and obtaining the three-dimensional model of the transmission line construction equipment.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. In a method for automatically generating a three-dimensional model of power transmission line construction equipment of the present invention, target detection is first performed on an image to be detected based on an SSD network model and a candidate frame is generated. Then, the candidate frame is screened using a non-maximum suppression method to obtain an image to be reconstructed. Then, the SURF algorithm and the AKAZE algorithm are used to extract and match feature points of any two images to be reconstructed, respectively, to obtain a first feature point matching set and a second feature point matching set. Finally, three-dimensional reconstruction of the power transmission line construction equipment is performed based on the first feature point matching set and the second feature point matching set to generate a three-dimensional model of the power transmission line construction equipment. In addition to using the SURF algorithm to extract main feature points, the above design also uses the AKAZE algorithm to extract edge feature points. The extracted feature points are more comprehensive, thereby improving the accuracy of subsequent three-dimensional model reconstruction.

[0047] 2. In a method for automatically generating a three-dimensional model of transmission line construction equipment in the present invention, the backbone network of the SSD network model is replaced with the ShufflenetV2 lightweight network, which greatly reduces the model parameters and thus improves the model detection speed.

[0048] 3. In a method for automatically generating a three-dimensional model of transmission line construction equipment of the present invention, an improved CBAM is introduced. The improved CBAM adds a second branch on the basis of the traditional CBAM. The second branch is composed of a front spatial attention module and a rear channel attention module. By serially connecting the newly added second branch with the original first branch, the interaction between the spatial and channel attentions can be enhanced, and the accuracy of the feature layer extracted by the backbone network can be improved.

[0049] 4. The present invention provides a method for automatically generating a three-dimensional model of transmission line construction equipment. In the SURF algorithm, an rBRIEF feature descriptor is established and feature matching is completed through the Hamming distance. In the AKAZE algorithm, a FREAK feature descriptor is established and feature matching is completed through the cosine similarity. Compared with the traditional Euclidean distance, the above design completes feature matching based on the Hamming distance and cosine similarity, and the quality and accuracy of feature matching are higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 The present invention is a flowchart of the method.

[0051] Figure 2 This is a network structure diagram of the improved SSD network model in the present invention.

[0052] Figure 3 This is a comparison chart of the average accuracy means of the improved SSD network model in the present invention and the traditional SSD model during the training process.

[0053] Figure 4 This is a comparison chart of the detection performance of the improved SSD network model in the present invention and the traditional SSD model for different transmission line construction equipment.

[0054] Figure 5 A comparison chart of the number of point clouds obtained by the method proposed in the present invention and solution 1. DETAILED DESCRIPTION

[0055] The present invention is further described in detail below in conjunction with specific implementations and drawings.

[0056] The present invention proposes a method for automatically generating a three-dimensional model of power transmission line construction equipment. First, the construction equipment is detected in the power transmission line construction equipment image to be detected based on the improved SSD network model and a candidate frame is generated. Then, the extracted candidate frame is screened using the improved non-maximum suppression method to obtain the image to be reconstructed. Then, based on the incremental SFM principle, two feature point extraction and matching operations are performed on the reconstructed image to finally complete the three-dimensional reconstruction of the power transmission line construction equipment. The method proposed in the present invention can quickly and accurately identify and locate the power transmission line construction equipment from the image, effectively support the subsequent three-dimensional automatic generation of the model, and fully extract feature information in the three-dimensional reconstruction process, thereby improving the robustness of the three-dimensional reconstruction.

[0057] Example:

[0058] See also Figure 1 A method for automatically generating a three-dimensional model of a power transmission line construction equipment is performed in the following steps:

[0059] S1. Acquire the image to be detected of the power transmission line construction equipment;

[0060] S2, based on the SSD network model, perform target detection on the image to be detected and generate candidate boxes;

[0061] The SSD network model is an improved SSD network model, and its network structure is as follows: Figure 2 As shown in the figure, the target detection process using the improved SSD network model is as follows: first, the ShufflenetV2 lightweight network is used as the backbone network of the SSD algorithm to perform preliminary feature extraction on the image to be detected, and the features obtained by the preliminary feature extraction are downsampled using the feature extraction layer to obtain a multi-scale feature map, and then the multi-scale feature map is subjected to feature fusion using a bidirectional weighted pyramid network to obtain an initial feature map, and finally the initial feature map is subjected to target detection using the output layer and a candidate box is generated;

[0062] The ShufflenetV2 lightweight network uses a deep separable convolution module to extract image features, so that the model parameters and calculation amount are smaller; the deep separable convolution module is composed of two convolution parts: deep convolution and point-by-point convolution. The deep convolution uses a channel-by-channel convolution kernel to complete the convolution operation. Each convolution kernel only participates in the channel corresponding to it and extracts feature maps respectively, thereby effectively reducing the model parameters; the point-by-point convolution fuses the independent feature maps extracted by the deep convolution through a convolution kernel of size 1x1; the last layer of the ShufflenetV2 lightweight network is replaced by a convolution layer from the original fully connected layer, and then a feature extraction layer consisting of four convolution layers is added, so that six multi-scale feature maps of different scales can be extracted in the end and input into the bidirectional weighted pyramid network;

[0063] The improved CBAM is introduced before the last layer of the ShufflenetV2 lightweight network, between the last layer of the ShufflenetV2 lightweight network and the first convolution layer of the feature extraction layer, and between two adjacent convolution layers in the feature extraction layer. The improved CBAM is to improve the traditional CBAM by adding a new serial branch on the basis of the traditional CBAM. The two branches connected in series can make the network pay more attention to important channels and ignore unimportant channels, thereby improving the feature extraction ability of the network and increasing the accuracy and efficiency of feature extraction. The improved CBAM includes a first branch (i.e., traditional CBAM) and a second branch connected in series. The first branch includes a first channel attention module and a first spatial attention module. The feature map adjusted by the first channel attention module is the input of the first spatial attention module, and the output of the first spatial attention module is the output of the first branch. The second branch includes a second spatial attention module and a second channel attention module. The feature map adjusted by the second spatial attention module is the input of the second channel attention module. The output of the second channel attention module is the output of the second branch. The outputs of the first branch and the second branch are combined to obtain the output of the improved CBAM. The combination formula is:

[0064] F′=F c ′+F s ;

[0065] In the above formula, F′, F′ c 、F s They represent the feature maps output by the improved CBAM, the second branch, and the first branch respectively;

[0066] The bidirectional weighted pyramid network (BiFPN network) is an improved bidirectional weighted pyramid network. The improved BiFPN network is specifically as follows: weighted feature fusion is performed on the input multi-scale feature map through a top-down path and a bottom-up path to obtain a new feature layer, and a hole convolution operation is performed on the new feature layer to obtain an initial feature map; the hole convolution operation refers to adding a 3*3 hole convolution after the weighted feature fusion obtains the new feature layer. The hole convolution forms a hole by inserting 0 into the ordinary convolution kernel. Under the premise of not changing the size of the input feature map, the receptive field of the convolution kernel is increased, and the feature extraction range is improved, thereby further strengthening the feature representation; the upsampling method of the top-down path is replaced by deconvolution from the traditional linear interpolation. The deconvolution contains learnable parameters and can be optimized through learning. Therefore, compared with the traditional upsampling method through linear interpolation, the deconvolution can restore a more complex spatial structure, and at the same time, restore the edge features through learning parameters to improve the accuracy of the model; the top-down path is specifically as follows:

[0067]

[0068] In the above formula, represents the intermediate transition feature of the i-th layer on the top-down path; w1 and w2 are the input weights of the i-th layer and the i+1-th layer on the top-down path respectively; are the inputs of the i-th layer and the i+1-th layer on the top-down path respectively; ε represents the hyperparameter used to prevent the gradient from disappearing;

[0069] The bottom-up path is specifically:

[0070]

[0071] In the above formula, represents the last output feature of the i-th layer on the bottom-up path; w2′ represents the output weight of the intermediate transition feature of the current layer; w3′ represents the output weight of the i-1-th layer on the bottom-up path; w1′ represents the input weight of the i-th layer; Represents the last output feature of the i-1th layer; DC represents the hole convolution operation; Conv represents the convolution operation;

[0072] S3. The candidate boxes are screened by using the non-maximum suppression method to obtain the image to be reconstructed. The traditional non-maximum suppression method (NMS algorithm) is improved by introducing a dynamic threshold into the NMS algorithm. The specific steps of improving the NMS algorithm are as follows:

[0073] A1. For a set of candidate boxes of the same type, calculate the confidence of all candidate boxes in the set;

[0074] A2. Select the candidate box with the highest confidence as the best candidate box, calculate the intersection-over-union ratio between the best candidate box and the remaining candidate boxes in the set, and update the confidence of the remaining candidate boxes based on the intersection-over-union ratio calculation result:

[0075]

[0076] In the above formula, b m represents the best candidate box; b i is the i-th candidate box among the remaining candidate boxes; s i 、s i ′ represents the confidence of the i-th candidate box in the remaining candidate boxes before and after the update; σ is the continuous penalty coefficient; IoU(b m , b i ) represents the candidate box b m With candidate box b i The intersection and union ratio between them; b m ∩b i Represents the candidate box b m With candidate box b i The area of ​​the overlapping part; b m ∪b i Represents the candidate box b m With candidate box b i The combined area; N t is a fixed threshold value set;

[0077] A3. Adjust the confidence of the remaining candidate boxes again through linear interpolation:

[0078]

[0079] In the above formula, is the confidence of the i-th candidate box in the remaining candidate boxes after further adjustment; α is the scale parameter;

[0080] A4, the remaining candidate boxes are grouped into a new set, and the process returns to A2 to start a new round of calculation until a stop calculation condition is met; the stop calculation condition may be that the number of remaining candidate boxes is 0;

[0081] A5. For the best candidate box in each round, select the best candidate box with the largest confidence as the final candidate box of this type of candidate box;

[0082] S4. First, use the SURF algorithm to extract and match feature points of any two images to be reconstructed to obtain a first feature point matching set, and use the AKAZE algorithm to extract and match feature points of any two images to be reconstructed to obtain a second feature point matching set; since the SURF algorithm alone has a limited degree of image detail retention, the feature matching result is poor, and it is impossible to extract sufficient and effective feature points to support three-dimensional point cloud reconstruction in complex scenes, the method proposed in the present invention also uses the AKAZE algorithm for extraction. The AKAZE algorithm uses nonlinear diffusion filtering to construct a scale space, which can retain more edge features of the image; the main feature points and edge feature points are extracted using the SURF algorithm and the AKAZE algorithm respectively, and more feature points are extracted, so that the subsequent three-dimensional reconstruction accuracy is also higher ; In the SURF algorithm, the rBRIEF feature descriptor is established and the feature matching is completed by the Hamming distance. The rBRIEF feature descriptor is a binary string with less calculation and compact storage. At the same time, the calculation efficiency of the Hamming distance in matching binary characters is higher than that of the traditional Euclidean distance; In the AKAZE algorithm, the FREAK feature descriptor is established and the feature matching is completed by the cosine similarity. The FREAK feature descriptor has rotation invariance and better matching effect. At the same time, the cosine similarity is used to calculate the similarity of the FREAK feature descriptor in the spatial direction. Compared with the Euclidean distance, the matching quality and accuracy are higher; After the feature matching is completed, the PROSAC algorithm is used to eliminate the wrong matching points in the first feature point matching set and the second feature point matching set to further improve the matching quality;

[0083] The specific steps of extracting and matching feature points of two images to be reconstructed using the SURF algorithm are as follows:

[0084] B1. Perform Gaussian filtering on both images to be reconstructed to reduce the impact of noise. The formula for Gaussian filtering is:

[0085]

[0086] In the above formula, L(x, y, σ) is the pixel function of the processed image; f(x, y) is the pixel function in the original image; x and y are the horizontal and vertical coordinates of the pixel respectively; represents the convolution operation; σ is the width parameter of the Gaussian function; G(σ) is a binary Gaussian function;

[0087] After Gaussian filtering, the Hessian matrix H(x,y,σ) of the image pixels is constructed:

[0088]

[0089] In the above formula, L xy (x,y,σ),L xx(x,y,σ),L yy (x,y,σ),L yx (x, y, σ) are the second-order partial derivatives of the Gaussian function at the point (x, y) about the x and y directions, the second-order partial derivative about the x direction, the second-order partial derivative about the y direction, and the second-order partial derivatives about the y and x directions respectively;

[0090] The feature points in the image to be reconstructed are detected by finding the local maximum of the Hessian matrix. After obtaining the position of the feature points, the feature description of the image to be reconstructed is established through the rBRIEF feature descriptor to speed up the subsequent feature matching.

[0091] B2. Perform feature point matching based on the Hamming distance to obtain a first feature point matching set, specifically: calculate the Hamming distance between the feature point P1 in one image to be reconstructed and all the feature points in another image to be reconstructed, and assume that the feature point in the other image to be reconstructed that has the closest Hamming distance to the feature point P1 is P2, and the feature point with the second closest Hamming distance is P3. If the ratio of the closest Hamming distance to the second closest Hamming distance is less than a preset threshold, it is considered that the feature point P1 in one image to be reconstructed matches the feature point P2 in the other image to be reconstructed successfully; the expression of the Hamming distance is:

[0092]

[0093] In the above formula, D ham (u,v) represents the Hamming distance between u and v; u and v are the feature descriptor vectors of one image to be reconstructed and another image to be reconstructed respectively; v i 、u i are the i-th component in the feature descriptor vector of one image to be reconstructed and another image to be reconstructed respectively; n is the vector dimension of the feature descriptor;

[0094] The specific steps of using the AKAZE algorithm to extract and match feature points of two images to be reconstructed are:

[0095] C1. Gaussian filtering is performed on both images to be reconstructed. The Gaussian filtering process is the same as the previous one and will not be described here. After Gaussian filtering, the Hessian matrix L of the image pixel points is constructed. Hessian :

[0096] L Hessian =σ 2 (L xx L yy -L xy 2 );

[0097] In the above formula, L xx , L yy , Lxy Respectively represent the second-order lateral derivative, second-order longitudinal derivative, and second-order cross derivative of the image at point (x, y);

[0098] The feature points in the image to be reconstructed are detected by finding the local maximum of the Hessian matrix. After obtaining the position of the feature points, sub-pixel precise positioning is performed, and the second-order Taylor expansion is used at the feature points to interpolate and obtain the precise positioning result. Finally, the feature description of the image to be reconstructed is established based on the FREAK feature descriptor.

[0099] C2. Feature point matching is performed based on cosine similarity to obtain a second feature point matching set, specifically: the cosine similarity between the feature point P1 in one image to be reconstructed and all the feature points in another image to be reconstructed is calculated, and the feature point with the closest cosine similarity to the feature point P1 in the other image to be reconstructed is set to P2, and the feature point with the second closest cosine similarity is set to P3. If the ratio of the closest cosine similarity to the second closest cosine similarity is less than a preset threshold, it is considered that the feature point P1 in one image to be reconstructed matches the feature point P2 in the other image to be reconstructed successfully; the expression of the cosine similarity is:

[0100]

[0101] In the above formula, S(R a ,R b ) represents R a With R b The cosine similarity of R a , R b are the feature descriptor vectors of one image to be reconstructed and another image to be reconstructed respectively; R ai , R bi are the i-th component of the feature descriptor vector of one image to be reconstructed and another image to be reconstructed respectively; n is the vector dimension of the feature descriptor;

[0102] S5, first obtain the internal and external parameters of the camera, map the first feature point matching set and the second feature point matching set from the two-dimensional coordinate system to the three-dimensional coordinate system, and form a three-dimensional sparse point cloud; the position (x, y) of the feature point in the image is converted into the coordinate (x) in the three-dimensional space w ,Y w ,Z w ), the transformation relationship between the two-dimensional coordinate system and the three-dimensional space coordinate system can be expressed as:

[0103]

[0104] In the above formula, f x 、f y 、c x 、c yThey represent the fixed parameters in the camera's internal parameter matrix M1, which do not change with the object's position; R and T represent the rotation matrix and translation matrix in the camera's external parameter matrix M2, respectively;

[0105] Then, the PMVS algorithm is used to densify the three-dimensional sparse point cloud, complete the surface reconstruction of the transmission line construction equipment, and obtain the three-dimensional model of the transmission line construction equipment.

[0106] Performance Verification:

[0107] 1. The improved SSD network model in the present invention is compared with the traditional SSD model in terms of average accuracy during training and detection performance of different transmission line construction equipment.

[0108] Obtain the average precision mean (map) comparison chart during training as follows Figure 3 As shown by Figure 3 It can be seen that the map value of the improved SSD network model of the present invention can reach a stable convergence state within a lower number of iterations compared to the traditional SSD model. The reason is that the improved SSD network model uses the lightweight network ShufflenetV2 as the backbone network, which effectively reduces the complexity of the model, reduces the amount of calculation, and improves the detection speed. At the same time, after reaching a stable convergence state, the map value of the improved SSD network model is also higher than that of the traditional SSD model, which verifies that the improved SSD network model has higher accuracy.

[0109] The comparison chart of the detection performance of different transmission line construction equipment is obtained as follows: Figure 4As shown in the figure; five transmission line construction equipment, namely crane, excavator, transport truck, pile driver and dump truck, are selected for detection. The evaluation indicators of detection performance are average precision P, recall rate R and F1 value; as can be seen from Figure (a), the average precision of the improved SSD network model is improved compared with the SSD algorithm. The average precision of the improved SSD network model is 98.83%, and the average precision of the SSD algorithm is 98.41%. This is because the traditional SSD model may lose information in the multi-scale feature extraction process, which leads to a decrease in detection accuracy. In contrast, the improved SSD network model introduces an improved BiFPN network to fuse image features and improve The accuracy of device detection is improved; as can be seen from Figure (b), the average recall rate of the improved SSD network model is also improved compared with the SSD algorithm. The average recall rate of the improved SSD network model is 86.95%, and the average recall rate of the SSD model is 83.62%. This is because the use of a constant NMS threshold results in missed detections in the screening of detection frames, and missed detection frames will lead to a decrease in the recall rate. In contrast, the improved SSD network model introduces a dynamic threshold and improves the adaptability of the threshold; as can be seen from Figure (c), the improved SSD network model still performs better than the traditional SSD model in terms of F1 value, which also verifies the effectiveness of the improved SSD network model.

[0110] 2. The method proposed in the present invention is compared with Scheme 1 in terms of the number of reconstructed point clouds. Scheme 1 only extracts feature points through the SURF algorithm and uses the Euclidean distance for feature matching. The other steps are the same as the method proposed in the present invention.

[0111] The comparison of the number of point clouds obtained is shown in the figure below. Figure 5 As shown by Figure 5 It can be seen that under the same reprojection error, the method proposed in the present invention reconstructs more point clouds, indicating that the reconstruction accuracy is higher; this is because the SURF algorithm has a weak feature extraction capability at the edge of the image, resulting in errors in feature matching and low reconstruction accuracy; in contrast, the method proposed in the present invention further uses the AKAZE algorithm to extract edge point feature information on the basis of the SURF algorithm, thereby enhancing the feature extraction capability, and performs feature matching through two different similarity matching methods, thereby generating more point clouds and improving reconstruction accuracy.

Claims

1. A method for automatically generating a three-dimensional model of a power transmission line construction equipment, characterized in that: The automatic generation method comprises the following steps: S1. Acquire the image to be detected of the power transmission line construction equipment; S2, first perform target detection on the image to be detected based on the SSD network model and generate candidate boxes, then use the non-maximum suppression method to screen the candidate boxes to obtain the image to be reconstructed; S3, using the SURF algorithm to extract and match feature points of any two images to be reconstructed to obtain a first feature point matching set, and using the AKAZE algorithm to extract and match feature points of any two images to be reconstructed to obtain a second feature point matching set; S4. Perform three-dimensional reconstruction of the transmission line construction equipment based on the first feature point matching set and the second feature point matching set to generate a three-dimensional model of the transmission line construction equipment.

2. The method for automatically generating a three-dimensional model of a power transmission line construction equipment according to claim 1, characterized in that: In step S2, the SSD network model is an improved SSD network model. Specifically, the improved SSD network model is as follows: the ShufflenetV2 lightweight network is first used as the backbone network of the SSD algorithm to perform preliminary feature extraction on the image to be detected, and the features obtained by the preliminary feature extraction are downsampled using the feature extraction layer to obtain a multi-scale feature map, and then the multi-scale feature map is feature fused using a bidirectional weighted pyramid network to obtain an initial feature map, and finally the initial feature map is used to perform target detection and generate candidate boxes using the output layer, and the last layer of the ShufflenetV2 lightweight network is replaced by a convolutional layer from the original fully connected layer, and the feature extraction layer consists of four convolutional layers.

3. The method for automatically generating a three-dimensional model of a power transmission line construction equipment according to claim 2, characterized in that: The bidirectional weighted pyramid network is an improved bidirectional weighted pyramid network. The improved bidirectional weighted pyramid network is specifically as follows: the input multi-scale feature map is subjected to feature fusion through a top-down path and a bottom-up path to obtain a new feature layer, and the new feature layer is subjected to a dilated convolution to obtain an initial feature map; wherein the upsampling method of the top-down path is replaced by deconvolution instead of traditional linear interpolation; the top-down path is specifically as follows: In the above formula, P i td represents the intermediate transition feature of the i-th layer on the top-down path; w1 and w2 are the input weights of the i-th layer and the i+1-th layer on the top-down path respectively; P i in , are the inputs of the i-th layer and the i+1-th layer on the top-down path respectively; ε represents the hyperparameter used to prevent the gradient from disappearing; The bottom-up path is specifically: In the above formula, P i out represents the last output feature of the i-th layer on the bottom-up path; w2′ represents the output weight of the intermediate transition feature of the current layer; w3′ represents the output weight of the i-1-th layer on the bottom-up path; w1′ represents the input weight of the i-th layer; represents the last output feature of the i-1th layer; DC represents the hole convolution operation; Conv represents the convolution operation.

4. The method for automatically generating a three-dimensional model of a power transmission line construction equipment according to claim 2, characterized in that: An improved CBAM is introduced before the last layer of the ShufflenetV2 lightweight network, between the last layer of the ShufflenetV2 lightweight network and the first convolution layer of the feature extraction layer, and between two adjacent convolution layers in the feature extraction layer; the improved CBAM includes a first branch and a second branch connected in series, the first branch includes a first channel attention module and a first spatial attention module, the feature map adjusted by the first channel attention module is the input of the first spatial attention module, and the output of the first spatial attention module is the output of the first branch, the second branch includes a second spatial attention module and a second channel attention module, the feature map adjusted by the second spatial attention module is the input of the second channel attention module, and the output of the second channel attention module is the output of the second branch, the outputs of the first branch and the second branch are combined to obtain the output of the improved CBAM; the combination formula is: F′=F c ′+F s ; In the above formula, F′, F c ′、F s They represent the feature maps output by the improved CBAM, the second branch, and the first branch, respectively.

5. The method for automatically generating a three-dimensional model of a power transmission line construction equipment according to claim 1, characterized in that: In step S2, the non-maximum suppression method is an improved non-maximum suppression method, and the specific steps are: A1. For a set of candidate boxes of the same type, calculate the confidence of all candidate boxes in the set; A2. Select the candidate box with the highest confidence as the best candidate box, calculate the intersection-over-union ratio between the best candidate box and the remaining candidate boxes in the set, and update the confidence of the remaining candidate boxes based on the intersection-over-union ratio calculation result: In the above formula, b m represents the best candidate box; b i is the i-th candidate box among the remaining candidate boxes; s i 、s i ′ represents the confidence of the i-th candidate box in the remaining candidate boxes before and after the update; σ is the continuous penalty coefficient; IoU(b m , b i ) represents the candidate box b m With candidate box b i The intersection and union ratio between them; b m ∩b i Represents the candidate box b m With candidate box b i The area of ​​the overlapping part; b m ∪b i Represents the candidate box b m With candidate box b i The combined area; N t is a fixed threshold value set; A3. Adjust the confidence of the remaining candidate boxes again through linear interpolation: In the above formula, is the confidence of the i-th candidate box in the remaining candidate boxes after further adjustment; α is the scale parameter; A4, groups the remaining candidate boxes into a new set, returns to A2 and starts a new round of calculation until the calculation stop condition is reached; A5. For the best candidate box in each round, select the best candidate box with the largest confidence as the final candidate box of this type of candidate box.

6. The method for automatically generating a three-dimensional model of power transmission line construction equipment according to claim 1, characterized in that: In step S3, the feature point extraction and matching of the two images to be reconstructed using the SURF algorithm is specifically as follows: B1. Preprocess the two images to be reconstructed, then detect the feature points in the two images to be reconstructed by finding the local maximum of the Hessian matrix, and establish feature descriptions of the two images to be reconstructed based on the rBRIEF feature descriptor; B2. Perform feature point matching based on the Hamming distance to obtain a first feature point matching set, specifically: calculate the Hamming distance between the feature point P1 in one image to be reconstructed and all the feature points in another image to be reconstructed, and assume that the feature point in the other image to be reconstructed that has the closest Hamming distance to the feature point P1 is P2, and the feature point with the second closest Hamming distance is P3. If the ratio of the closest Hamming distance to the second closest Hamming distance is less than a preset threshold, it is considered that the feature point P1 in one image to be reconstructed matches the feature point P2 in the other image to be reconstructed successfully; the expression of the Hamming distance is: In the above formula, D ham (u,v) represents the Hamming distance between u and v; u and v are the feature descriptor vectors of one image to be reconstructed and another image to be reconstructed respectively; v i 、u i are respectively the i-th component in the feature descriptor vector of one image to be reconstructed and another image to be reconstructed; n is the vector dimension of the feature descriptor.

7. The method for automatically generating a three-dimensional model of power transmission line construction equipment according to claim 1, characterized in that: In step S3, the feature point extraction and matching of the two images to be reconstructed using the AKAZE algorithm is specifically as follows: C1. Preprocess the two images to be reconstructed first, then detect the feature points in the two images to be reconstructed by finding the local maximum of the Hessian matrix, and establish the feature description of the two images to be reconstructed based on the FREAK feature descriptor; C2. Feature point matching is performed based on cosine similarity to obtain a second feature point matching set, specifically: the cosine similarity between the feature point P1 in one image to be reconstructed and all the feature points in another image to be reconstructed is calculated, and the feature point with the closest cosine similarity to the feature point P1 in the other image to be reconstructed is set to P2, and the feature point with the second closest cosine similarity is set to P3. If the ratio of the closest cosine similarity to the second closest cosine similarity is less than a preset threshold, it is considered that the feature point P1 in one image to be reconstructed matches the feature point P2 in the other image to be reconstructed successfully; the expression of the cosine similarity is: In the above formula, S(R a ,R b ) represents R a With R b The cosine similarity of R a , R b are the feature descriptor vectors of one image to be reconstructed and another image to be reconstructed respectively; R ai , R bi are respectively the i-th component of the feature descriptor vector of one image to be reconstructed and another image to be reconstructed; n is the vector dimension of the feature descriptor.

8. A method for automatically generating a three-dimensional model of power transmission line construction equipment according to claim 6 or 7, characterized in that: The preprocessing is Gaussian filtering.

9. The method for automatically generating a three-dimensional model of power transmission line construction equipment according to claim 1, characterized in that: Step S3 also includes using the PROSAC algorithm to eliminate erroneous matching points in the first feature point matching set and the second feature point matching set.

10. The method for automatically generating a three-dimensional model of power transmission line construction equipment according to claim 1, characterized in that: Step S4 is specifically as follows: obtaining the internal and external parameters of the camera, mapping the first feature point matching set and the second feature point matching set from the two-dimensional coordinate system to the three-dimensional coordinate system to form a three-dimensional sparse point cloud, using the PMVS algorithm to densify the three-dimensional sparse point cloud, completing the surface reconstruction of the transmission line construction equipment, and obtaining the three-dimensional model of the transmission line construction equipment.

Images