A three-dimensional reconstruction method and system for substation construction site

Abstract

The present invention provides a three-dimensional reconstruction method and system for a substation construction site, the method comprising acquiring an image of the substation construction site; extracting ORB feature points of each frame of the image, and extracting a polygonal area corresponding to a steel bar structure in the image according to a trained word bag tree model, eliminating feature points inside the polygonal area, and calculating a feature representation of the current image; performing feature matching on two adjacent frames of the image to obtain a camera pose; utilizing a projection relationship to generate a point cloud from pixels of a key frame image, and fusing the point cloud to form a three-dimensional model of the substation construction site. The present invention trains a word bag tree model based on a deep neural network, and matches complex and interlaced steel bar structures such as steel bars in the image to trained polygonal areas, eliminates feature points inside, and performs feature representation of the image, which facilitates accurate matching of image features, and is suitable for a three-dimensional reconstruction method at a substation site, and realizes high-precision three-dimensional reconstruction of the scene.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional modeling, and in particular to a three-dimensional reconstruction method and system for a transformer substation construction site. Background Art

[0002] Drone inspections are now a viable option for substation construction progress checks. Using drones and computer vision to reconstruct the site in 3D and manage progress can reduce costs, speed up inspections, and facilitate decision-making by managers.

[0003] The ultimate goal of lean management of substation construction sites is to use the images captured by drones and apply algorithms to generate high-quality 3D models of all buildings on the construction site. In order to maximize the quality of the model, it is necessary to design a corresponding 3D reconstruction algorithm based on the substation site video taken by drones to obtain a high-quality 3D model of the substation site.

[0004] Among them, processing repeated and staggered strip structures is a difficult point in existing 3D reconstruction algorithms. Currently, the feature points extracted from this structure are prone to mismatching, which leads to poor results in the 3D reconstructed model. Summary of the invention

[0005] The present invention provides a three-dimensional reconstruction method and system for a substation construction site, which are used to solve the problem that the three-dimensional modeling effect of the existing substation construction site is not good.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides a three-dimensional reconstruction method for a substation construction site, the method comprising the following steps:

[0008] Acquire images of substation construction sites;

[0009] Extract the ORB feature points of each frame image, and extract the polygonal area corresponding to the steel structure in the image according to the trained word bag tree model, remove the feature points inside the polygonal area, and calculate the feature representation of the current image;

[0010] Perform feature matching on two adjacent frames of images, locate the cameras that took the two frames of images based on the paired feature points, and obtain the camera pose;

[0011] Constructing a key frame set, and when the number of images in the key frame set reaches a preset threshold, performing a local bundle adjustment method on the common view feature points of the key frames in the set to optimize the camera posture;

[0012] According to the optimized camera posture, the key frame image pixels are generated into point clouds using the projection relationship, and the point clouds are fused to form a three-dimensional model of the substation construction site.

[0013] Furthermore, the acquisition of substation construction site images is specifically as follows:

[0014] Break up the live video captured by the drone into single-frame images.

[0015] Furthermore, the training of the bag-of-words tree model is specifically as follows:

[0016] Acquire a live video taken by a drone, and break the live video into single-frame images;

[0017] Set the training configuration and train the bag-of-words tree model based on the ORB (Oriented Fast and Rotated Brief, feature extraction algorithm) feature points. During the training process, the U-net network is used for image training, and the steel structure under construction is segmented to obtain the corresponding polygonal area.

[0018] Furthermore, the cameras used to capture the two frames of images are positioned using a bundle adjustment method.

[0019] Furthermore, the process of constructing the key frame set is specifically as follows:

[0020] The first frame image is added to the key frame set, the feature representation of each subsequent frame image is compared with the feature representation of the first frame, and the Hamming distance is calculated. If the Hamming distance exceeds a preset threshold, the current image is used as a candidate frame, otherwise the current image is added to the key frame set;

[0021] The polygonal area of ​​the candidate frame is calculated. If the ratio of the polygonal area to the current image area exceeds a ratio threshold, the current candidate frame is not added to the key frame set; otherwise, the current candidate frame is added to the key frame set.

[0022] Furthermore, the method further comprises the steps of:

[0023] When a loop is detected, a global bundle adjustment is performed on the common view feature points of all key frames in the loop to optimize the camera pose and common view feature points.

[0024] Furthermore, the specific process of fusing the point clouds to form a three-dimensional model of the substation construction site is as follows:

[0025] The point cloud is placed in an octree structure, and the surface is represented by implicit function representation. The implicit surfaces corresponding to the points under the same voxel grid are fused in a weighted manner to obtain a three-dimensional model of the substation construction site.

[0026] A second aspect of the present invention provides a three-dimensional reconstruction system for a substation construction site, the system comprising:

[0027] An image acquisition unit, used to acquire images of the substation construction site;

[0028] An image processing unit is used to extract ORB feature points of each frame of the image, and extract the polygonal area corresponding to the steel structure in the image according to the trained word bag tree model, remove the feature points inside the polygonal area, and calculate the feature representation of the current image;

[0029] A positioning unit is used to perform feature matching on two adjacent frames of images, locate the shooting cameras of the two frames of images according to the paired feature points, and obtain the camera posture;

[0030] A first optimization unit is used to construct a key frame set, and when the number of images in the key frame set reaches a preset threshold, a local bundle adjustment method is performed on the common view feature points of the key frames in the set to optimize the camera posture;

[0031] The modeling unit generates point clouds from the key frame image pixels according to the optimized camera posture and utilizes the projection relationship, and fuses the point clouds to form a three-dimensional model of the substation construction site.

[0032] Furthermore, the system also includes a model training unit, which obtains the on-site video taken by the drone and breaks up the on-site video into single-frame pictures; trains a bag-of-words tree model based on ORB feature points based on a preset configuration, uses a U-net network for picture training during the training process, and segments the steel structure part under construction to obtain the corresponding polygonal area.

[0033] Furthermore, the system also includes a second optimization unit, which, after detecting the loop, performs a global bundle adjustment method on the common view feature points of all key frames in the loop to optimize the camera posture and the common view feature points.

[0034] The three-dimensional reconstruction system of the second aspect of the present invention can implement the method of the first aspect and the various implementation modes of the first aspect, and achieve the same effect.

[0035] The effects provided in the content of the invention are only the effects of the embodiments, not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0036] The present invention is based on a deep neural network training bag-of-words tree model, and matches the complex and interlaced steel bar structures such as steel bars in the image to the trained polygonal areas, removes the internal feature points, and performs feature representation of the image, which facilitates accurate matching of image features. The present invention is suitable for the 3D reconstruction method of the substation site and realizes high-precision 3D reconstruction of the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 is a schematic diagram of a flow chart of an embodiment of the method of the present invention;

[0039] Figure 2 It is a flow chart of one implementation mode of the method of the present invention;

[0040] Figure 3 It is a schematic diagram of the structure of the system embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to clearly illustrate the technical features of the present solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The disclosure below provides many different embodiments or examples for realizing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below. In addition, the present invention may repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed. It should be noted that the components illustrated in the accompanying drawings are not necessarily drawn to scale. The present invention omits the description of known components and processing techniques and processes to avoid unnecessary limitations on the present invention.

[0042] like Figure 1 As shown, an embodiment of the present invention provides a three-dimensional reconstruction method for a substation construction site, the method comprising the following steps:

[0043] S1, obtain the image of the substation construction site;

[0044] S2, extracting the ORB feature points of each frame image, and extracting the polygonal area corresponding to the steel structure in the image according to the trained word bag tree model, removing the feature points inside the polygonal area, and calculating the feature representation of the current image;

[0045] S2, feature matching is performed on two adjacent frames of images, and the shooting cameras of the two frames of images are located according to the paired feature points to obtain the camera pose;

[0046] S4, constructing a key frame set, and when the number of images in the key frame set reaches a preset threshold, performing a local bundle adjustment method on the common view feature points of the key frames in the set to optimize the camera posture;

[0047] S5, according to the optimized camera posture, using the projection relationship, generating point clouds from the key frame image pixels, and fusing the point clouds to form a three-dimensional model of the substation construction site.

[0048] like Figure 2 As shown, the bag-of-words tree model is trained first, and the specific process is as follows:

[0049] The historical data of drone videos of the substation construction site were collected and broken into frames. The bag-of-words tree model based on ORB feature points was trained with a 6-layer, 10-branch configuration. The substation construction site pictures collected contained repeated and staggered strip structures. The U-net network was used to train the relevant pictures, segment the steel structure under construction, and obtain its corresponding polygonal area φ.

[0050] In step S1, a live video shot by a drone is obtained and the video is broken up into single-frame images.

[0051] In step S2, ORB feature points are extracted for each frame of the image, and the polygonal area φ is extracted using the trained network, and the feature points inside the polygonal area φ are removed. Based on the leaf nodes of the bag-of-words tree, the Hamming distance between the remaining ORB feature points in the current image and the leaf nodes is calculated to obtain the feature representation of the current image in the bag-of-words tree model.

[0052] In step S3, feature matching is performed on the two frames of images, and the camera that took the two images is positioned by the bundle adjustment method based on the paired feature points to obtain the camera posture.

[0053] In step S4, the process of constructing a key frame set is specifically as follows:

[0054] The first frame image ψ is added to the key frame set, the feature representation of each subsequent frame image is compared with the feature representation of the first frame, and the Hamming distance is calculated. If the Hamming distance exceeds the preset threshold τ1, the current image is used as the candidate frame K, otherwise the current image is added to the key frame set;

[0055] For the candidate frame K, the polygon area of ​​the candidate frame is calculated. If the ratio of the polygon area to the current image area exceeds the ratio threshold τ2, the current candidate frame is not added to the key frame set, otherwise the current candidate frame is added to the key frame set. Avoid the influence of repeated and staggered strip structures on key frames.

[0056] Every time 20 key frames are updated in the key frame set, the local bundle adjustment method is performed on the common view feature points of the 20 key frames in the key frame set to optimize the camera posture to reduce the error.

[0057] In step S5, the point clouds are fused and the point clouds with overlapping key frames are spliced ​​to form a three-dimensional model of the substation construction site. The specific process is:

[0058] The point cloud is placed in an octree structure, and the surface is represented by implicit function representation. The implicit surfaces corresponding to the points under the same voxel grid are fused in a weighted manner to obtain a three-dimensional model of the substation construction site.

[0059] In one implementation of the embodiment of the present invention, the method further includes the steps of:

[0060] When a loop is detected, a global bundle adjustment is performed on the common view feature points of all key frames in the loop to optimize the camera pose and common view feature points.

[0061] like Figure 3 As shown, an embodiment of the present invention also provides a three-dimensional reconstruction system for a substation construction site, the system comprising an image acquisition unit 1, an image processing unit 2, a positioning unit 3, a first optimization unit 4, a second optimization unit 5, a modeling unit 6 and a model training unit 7.

[0062] The image acquisition unit 1 is used to acquire images of the substation construction site; the image processing unit 2 is used to extract the ORB feature points of each frame of the image, and extract the polygonal area corresponding to the steel structure in the image according to the trained word bag tree model, remove the feature points inside the polygonal area, and calculate the feature representation of the current image; the positioning unit 3 is used to perform feature matching on two adjacent frames of images, locate the shooting cameras of the two frames of images according to the paired feature points, and obtain the camera posture; the first optimization unit 2 is used to construct a key frame set, and when the number of images in the key frame set reaches a preset threshold, the local bundle adjustment method is performed on the common view feature points of the key frames in the set to optimize the camera posture; the modeling unit 6 generates a point cloud from the key frame image pixels according to the optimized camera posture using the projection relationship, and fuses the point cloud to form a three-dimensional model of the substation construction site.

[0063] The model training unit 7 obtains the on-site video taken by the drone, breaks up the on-site video into single-frame images; trains the bag-of-words tree model based on ORB feature points based on a preset configuration, uses a U-net network for image training during the training process, and segments the steel structure part under construction to obtain the corresponding polygonal area.

[0064] After detecting the loop, the second optimization unit 5 performs a global bundle adjustment on the common view feature points of all key frames in the loop to optimize the camera posture and the common view feature points.

[0065] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A three-dimensional reconstruction method for a substation construction site, characterized in that: The method comprises the following steps: Acquire images of substation construction sites; Extract the ORB feature points of each frame image, and extract the polygonal area corresponding to the steel structure in the image according to the trained word bag tree model, remove the feature points inside the polygonal area, and calculate the feature representation of the current image; Perform feature matching on two adjacent frames of images, locate the cameras that took the two frames of images based on the paired feature points, and obtain the camera pose; Constructing a key frame set, and when the number of images in the key frame set reaches a preset threshold, performing a local bundle adjustment method on the common view feature points of the key frames in the set to optimize the camera posture; The process of constructing a key frame set is specifically as follows: The first frame image is added to the key frame set, the feature representation of each subsequent frame image is compared with the feature representation of the first frame, and the Hamming distance is calculated. If the Hamming distance exceeds a preset threshold, the current image is used as a candidate frame, otherwise the current image is added to the key frame set; Calculating the polygonal area of ​​the candidate frame, if the ratio of the polygonal area to the current image area exceeds a ratio threshold, the current candidate frame is not added to the key frame set, otherwise the current candidate frame is added to the key frame set; According to the optimized camera pose, the key frame image pixels are generated into a point cloud by using the projection relationship, and the point cloud is fused to form a three-dimensional model of the substation construction site; The specific process of fusing the point clouds to form a three-dimensional model of the substation construction site is as follows: The point cloud is placed in an octree structure, and the surface is represented by an implicit function representation. The implicit surface corresponding to the points under the same voxel grid is fused by a weighted method to obtain a three-dimensional model of the substation construction site. The method further comprises the steps of: When a loop is detected, a global bundle adjustment is performed on the common view feature points of all key frames in the loop to optimize the camera pose and common view feature points.

2. The three-dimensional reconstruction method of the substation construction site according to claim 1 is characterized in that: The acquisition of substation construction site images is as follows: Break up the live video captured by the drone into single-frame images.

3. The three-dimensional reconstruction method of the substation construction site according to claim 1 is characterized in that: The training of the bag-of-words tree model is specifically as follows: Acquire a live video taken by a drone, and break the live video into single-frame images; Set the training configuration and train the bag-of-words tree model based on ORB feature points based on the configuration. During the training process, the U-net network is used for image training and the steel structure under construction is segmented to obtain the corresponding polygonal area.

4. The three-dimensional reconstruction method of the substation construction site according to claim 1 is characterized in that: The cameras shooting the two frames of images are positioned by using the bundle adjustment method.

5. A three-dimensional reconstruction system for a substation construction site, characterized in that: The system comprises: An image acquisition unit, used to acquire images of the substation construction site; An image processing unit is used to extract ORB feature points of each frame of the image, and extract the polygonal area corresponding to the steel structure in the image according to the trained word bag tree model, remove the feature points inside the polygonal area, and calculate the feature representation of the current image; A positioning unit is used to perform feature matching on two adjacent frames of images, locate the shooting cameras of the two frames of images according to the paired feature points, and obtain the camera posture; A first optimization unit is used to construct a key frame set, and when the number of images in the key frame set reaches a preset threshold, a local bundle adjustment method is performed on the common view feature points of the key frames in the set to optimize the camera posture; The process of constructing a key frame set is specifically as follows: The first frame image is added to the key frame set, the feature representation of each subsequent frame image is compared with the feature representation of the first frame, and the Hamming distance is calculated. If the Hamming distance exceeds a preset threshold, the current image is used as a candidate frame, otherwise the current image is added to the key frame set; Calculating the polygonal area of ​​the candidate frame, if the ratio of the polygonal area to the current image area exceeds a ratio threshold, the current candidate frame is not added to the key frame set, otherwise the current candidate frame is added to the key frame set; The modeling unit generates point clouds from the key frame image pixels according to the optimized camera pose and utilizes the projection relationship, and fuses the point clouds to form a three-dimensional model of the substation construction site; The specific process of fusing the point clouds to form a three-dimensional model of the substation construction site is as follows: The point cloud is placed in an octree structure, and the surface is represented by an implicit function representation. The implicit surface corresponding to the points under the same voxel grid is fused by a weighted method to obtain a three-dimensional model of the substation construction site. The system also includes a second optimization unit, which performs a global bundle adjustment method on the common view feature points of all key frames in the loop after detecting the loop, so as to optimize the camera posture and the common view feature points.

6. The three-dimensional reconstruction system of the substation construction site according to claim 5 is characterized in that: The system also includes a model training unit, which obtains the on-site video taken by the drone and breaks the on-site video into single-frame pictures; trains a bag-of-words tree model based on ORB feature points based on a preset configuration, uses a U-net network for picture training during the training process, and segments the steel structure part under construction to obtain the corresponding polygonal area.

Images