Binocular vision reconstruction method for high spatial resolution 3D shape of underground buildings

Abstract

The binocular vision reconstruction method of the high spatial resolution three-dimensional shape of underground buildings disclosed in the present invention uses a binocular camera to record images of caverns, detects and matches the image feature points obtained by the binocular camera, corrects the matching errors, and obtains a feature point coordinate matching set; builds a fully connected residual neural network model, and trains the fully connected residual neural network model according to the feature point coordinate matching set; inputs the pixel coordinates in the binocular camera image into the trained fully connected residual neural network model to obtain the image full pixel matching coordinate set, and establishes a three-dimensional coordinate point cloud model of the cavern through the calibration parameters of the binocular camera, and reconstructs the high spatial resolution three-dimensional shape of the cavern. The present invention solves the technical problem that the current use of total stations for shape monitoring can only obtain specific point coordinates at a certain density, which is not conducive to the visualization of underground structures.

Description

Technical Field

[0001] The invention belongs to the technical field of hydropower and pumped storage power station construction, and specifically relates to a binocular vision reconstruction method for high spatial resolution three-dimensional appearance of underground buildings. Background Art

[0002] In the clean energy industry, hydropower and pumped storage projects have become one of the key tasks of the industry. As the core construction content of hydropower and pumped storage projects, underground buildings composed of water diversion shafts, tailwater tunnels, curtain irrigation and drainage corridors, underground powerhouses, main transformer rooms, tailwater surge chambers, etc., timely grasping the morphological characteristics of underground buildings with caverns as the core is of great significance for guiding construction operations, improving the support measures of underground building morphology during the construction process, and ensuring construction safety.

[0003] The existing method for monitoring the morphology of underground structures in hydropower and pumped storage power stations mainly uses total stations for measurement, that is, measurement control points are placed on the structures, mapping points are arranged in a total station manner, the three-dimensional coordinates of the discrete points of the fragments are obtained, and finally the underground structure plan is drawn in combination with professional mapping software. This type of method has the advantage of high accuracy, but it can only obtain the coordinates of specific points at a certain density, which is not conducive to the visualization of underground structures. In addition, its three-dimensional morphology recognition process requires the use of field sketches to distinguish different points, and the operation process is complicated, time-consuming and labor-intensive. In recent years, underground structure morphology monitoring technology with laser as the core has been gradually developed, that is, by reconstructing the spatial structure in the form of laser point cloud, it has achieved certain results, but this method currently mainly uses single-point measurement. Scanning hundreds of thousands or even millions of point cloud data is time-consuming and labor-intensive, the testing cost is high, and in most cases, special materials are required to mark the underground structures, which is difficult to promote on a large scale.

[0004] Based on the above shortcomings, considering the advantages of current image pixel resolution that can reach millions or even hundreds of millions, and the ability of neural networks to describe nonlinearity, a binocular vision reconstruction method for high spatial resolution three-dimensional morphology of underground structures is proposed to achieve accurate and efficient reconstruction of the morphology of underground structures. Summary of the invention

[0005] The purpose of the present invention is to provide a binocular vision reconstruction method for high spatial resolution three-dimensional morphology of underground buildings, which solves the technical problem that the current use of total stations for morphology monitoring can only obtain specific point coordinates at a certain density, which is not conducive to the visualization of underground structures.

[0006] The technical solution adopted by the present invention is a binocular vision reconstruction method of high spatial resolution three-dimensional appearance of underground buildings, comprising the following steps:

[0007] S1: Use a binocular camera to record images of the cave, detect and match the image feature points obtained by the binocular camera, correct the matching error, and obtain a feature point coordinate matching set;

[0008] S2: Build a fully connected residual neural network model and train it according to the feature point coordinate matching set;

[0009] S3: Input the pixel coordinates in the binocular camera image into the trained fully connected residual neural network model to obtain the image full pixel matching coordinate set, and establish the three-dimensional coordinate point cloud model of the cave through the binocular camera calibration parameters to reconstruct the high spatial resolution three-dimensional morphology of the cave.

[0010] The present invention is also characterized in that:

[0011] S1 specifically includes: using a binocular camera to record images of the cave and detecting the image feature points obtained by the binocular camera; using the pixel brightness and gradient of the feature points to construct a pixel descriptor vector, and by proposing a descriptor vector similarity measurement function, matching the feature points of the binocular camera image; correcting the matching error of the feature points based on the error correction function, and obtaining the coordinate matching set of the feature points of the binocular camera image.

[0012] The specific detection in S1 is:

[0013] The Gaussian function is used to convolve the image obtained by the binocular camera to obtain the multi-scale space of the image, and the multi-scale difference space of the image is obtained through the adjacent scale images. Centered on the pixel, the brightness of the 5×5 neighborhood of the pixel and the 5×5 pixels of the upper and lower scales are compared to select the pixel with the largest brightness as the image feature point, and by traversing all the pixels of the image, a set of image feature points is formed.

[0014] The pixel descriptor vector constructed in S1 is specifically:

[0015] Select the 16×16 area with feature points and calculate the pixels in each 4×4 interval in the area. , , , , , , and The gradient direction histogram of 8 directions, make a cumulative value for each gradient direction, and get a feature point. Each feature point is composed of It is composed of seed points, each of which has vector information in 8 directions, generating feature information of 128 feature points in total, which is the descriptor vector of a feature point;

[0016] Among them, the gradient of the feature point and direction The calculation formula is:

[0017]

[0018]

[0019] Where: L The difference space where each feature point is located.

[0020] In S1, the feature points of the binocular camera image are matched as follows:

[0021] The matching between feature points is achieved by calculating the cosine distance between the descriptor vectors of the feature points in the binocular camera image. cos(d) Calculated by the following formula:

[0022]

[0023] Where: is the descriptor vector of the feature points of the left view image, is the descriptor vector of the feature points of the right view image, k 1 is the dimension of the descriptor vector; the cosine distance of the descriptor vectors of the feature points of the left and right view images of the binocular camera is calculated by traversal. The more similar the descriptor vectors are, the larger the cosine distance is; the feature point pairs with larger cosine distances are selected as the matching results of the feature points.

[0024] In S1, the matching error of the feature points is corrected based on the error correction function as follows:

[0025] Taking the matched binocular camera image feature points as the core, the zero-mean normalized cross-correlation function is used to calculate the neighborhood correlation of the feature points, and the matching of the feature points is further corrected:

[0026]

[0027] Where: is the neighborhood correlation, and Left and right camera images respectively (x,y) The image brightness value at M is the feature point neighborhood size, and are the mean brightness values ​​of image pixels in the neighborhood respectively;

[0028] By calculating the similarity of feature point matching pairs one by one, the feature point matching pairs with greater similarity are selected as the final binocular camera image feature point coordinate matching set.

[0029] S2 is specifically as follows: building a fully connected residual neural network model, based on the binocular camera image feature point coordinate matching set, taking the left camera image feature point coordinates in the matching set as input, and the corresponding right camera feature point coordinates in the matching set as output, training the fully connected residual neural network model, and obtaining a neural network model for pixel matching of left and right view images in the unmarked cave.

[0030] In S2, during the training process of the fully connected residual neural network model, the fully connected residual neural network model is optimally solved by calculating the damage function based on the Langmuir distance:

[0031]

[0032] Where: is the matching coordinate obtained by the fully connected residual network, is the real coordinate in the feature point coordinate matching set, k 2 is the coordinate dimension, namely the horizontal and vertical directions.

[0033] S3 is specifically as follows: taking the pixel coordinates in the left camera image as input, obtaining the corresponding pixel coordinate positions of the left camera image pixels in the right camera image through the trained fully connected residual neural network model, obtaining the image full pixel matching coordinate set, and establishing the three-dimensional coordinate point cloud model of the cave through the binocular camera calibration parameters, and reconstructing the high spatial resolution three-dimensional morphology of the cave.

[0034] The three-dimensional coordinate point cloud model of the cave is established in S3 as follows:

[0035] Suppose the first i The pixel coordinates are described as and , let the physical space coordinates of the cave structure corresponding to the pixel be ;

[0036] Based on the spatial position of the binocular camera, we can get , and through the triangular relationship we can get:

[0037]

[0038] Where: f and B is the calibration parameter, f represents the focal length of the camera, B Represents the basis moment between cameras;

[0039] According to the above relationship, the three-dimensional coordinates of the cave space corresponding to pixel i can be calculated as:

[0040]

[0041] The obtained three-dimensional space point cloud coordinates are imported into the physical simulation software, and the point cloud data are converted into a surface entity to obtain the three-dimensional structure of the cave.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] In terms of cave morphology reconstruction, a cave morphology construction method based on video images is proposed, which takes advantage of the high spatial resolution of images and can quickly achieve high spatial resolution reconstruction of cave morphology; in terms of binocular perspective image pixel matching, the neural network model is trained using matched feature points to achieve pixel matching of left and right perspective images in the case of unmarked caves; in terms of neural network construction, a fully connected residual neural network model is proposed to solve the problem that the existing fully connected neural network cannot achieve identity transformation and the model cannot be fully trained; secondly, the model training adopts a loss function based on Langmuir distance to improve the training accuracy and efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic flow chart of a binocular vision reconstruction method of a high spatial resolution three-dimensional appearance of an underground building according to the present invention;

[0045] Figure 2 It is a schematic diagram of the process of image feature detection in the binocular vision reconstruction method of the high spatial resolution three-dimensional appearance of underground buildings of the present invention;

[0046] Figure 3 It is a schematic diagram of constructing image feature point descriptors in the binocular vision reconstruction method of the high spatial resolution three-dimensional appearance of underground buildings of the present invention;

[0047] Figure 4 It is a structural schematic diagram of a fully connected residual neural network model in the binocular vision reconstruction method of the high spatial resolution three-dimensional appearance of underground buildings of the present invention;

[0048] Figure 5 It is a schematic diagram of the spatial position and coordinates of binocular cameras in the binocular vision reconstruction method of the high spatial resolution three-dimensional appearance of underground buildings of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] Example 1

[0051] like Figure 1 As shown, the binocular vision reconstruction method of the high spatial resolution three-dimensional appearance of underground buildings disclosed in this embodiment includes the following steps:

[0052] S1: Use a binocular camera to record images of the cave, detect and match the image feature points obtained by the binocular camera, correct the matching error, and obtain a feature point coordinate matching set;

[0053] S2: Build a fully connected residual neural network model and train it according to the feature point coordinate matching set;

[0054] S3: Input the pixel coordinates in the binocular camera image into the trained fully connected residual neural network model to obtain the image full pixel matching coordinate set, and establish the three-dimensional coordinate point cloud model of the cave through the binocular camera calibration parameters to reconstruct the high spatial resolution three-dimensional morphology of the cave.

[0055] Example 2

[0056] On the basis of Example 1, S1 specifically comprises: using a binocular camera to record images of the cave, and detecting feature points of the image acquired by the binocular camera; constructing a pixel descriptor vector using the pixel brightness and gradient of the feature points, and matching the feature points of the binocular camera image by proposing a descriptor vector similarity measurement function; correcting the matching error of the feature points based on the error correction function, and obtaining a coordinate matching set of the feature points of the binocular camera image.

[0057] Example 3

[0058] On the basis of Example 2, the present invention compares Harris, SIFT, SURF and other image feature detection algorithms from the perspectives of computational efficiency, accuracy and illumination robustness, and selects the SIFT feature detection algorithm to detect the feature points of the cave image. The detection in S1 is specifically as follows:

[0059] The Gaussian function is used to convolve the image obtained by the binocular camera to obtain the multi-scale space of the image, and the multi-scale difference space of the image is obtained through the adjacent scale images. In order to improve the detection efficiency and accuracy, the pixel is centered and the brightness of the 5×5 neighborhood of the pixel and the 5×5 pixels of the upper and lower scales is compared to select the pixel with the largest brightness as the image feature point, such as Figure 2 As shown, by traversing all pixels of the image, a set of image feature points is formed.

[0060] Example 4

[0061] Based on Example 3, the pixel descriptor vector constructed in S1 is specifically:

[0062] Select the 16×16 area with feature points and calculate the pixels in each 4×4 interval in the area. , , , , , , and The gradient direction histogram of 8 directions, make a cumulative value for each gradient direction, and get a feature point. Each feature point is composed of The feature information of 128 feature points is generated, which is the descriptor vector of a feature point. Figure 3 As shown;

[0063] Among them, the gradient of the feature point and direction The calculation formula is:

[0064]

[0065]

[0066] Where: L The difference space where each feature point is located.

[0067] Furthermore, in order to achieve the matching between the feature points of the left and right camera images, a feature point descriptor vector measurement method based on the cosine function is proposed in the present invention. The matching of the feature points of the binocular camera images in S1 is specifically as follows:

[0068] The matching between feature points is achieved by calculating the cosine distance between the descriptor vectors of the feature points in the binocular camera image. cos(d) Calculated by the following formula:

[0069]

[0070] Where: is the descriptor vector of the feature points of the left view image, is the descriptor vector of the feature points of the right view image, k 1 is the dimension of the descriptor vector; the cosine distance of the descriptor vectors of the feature points of the left and right view images of the binocular camera is calculated by traversal. The more similar the descriptor vectors are, the larger the cosine distance is; the feature point pairs with larger cosine distances are selected as the matching results of the feature points.

[0071] Example 5

[0072] On the basis of Example 4, in order to further ensure the accuracy of feature point matching, the matching feature points of the left and right camera images are taken as the core, and the neighborhood correlation is calculated through the correlation function, and the matching of the feature points is further corrected through the correlation function. In the present invention, considering the influencing factors such as illumination and calculation efficiency, a zero-mean normalized cross-correlation function is adopted, and the matching error of the feature points is corrected based on the error correction function in S1 specifically as follows:

[0073] Taking the matched binocular camera image feature points as the core, the zero-mean normalized cross-correlation function is used to calculate the neighborhood correlation of the feature points, and the matching of the feature points is further corrected:

[0074]

[0075] Where: is the neighborhood correlation, and Left and right camera images respectively (x,y) The image brightness value at M is the feature point neighborhood size, and are the mean brightness values ​​of image pixels in the neighborhood respectively;

[0076] By calculating the similarity of feature point matching pairs one by one, the feature point matching pairs with greater similarity are selected as the final binocular camera image feature point coordinate matching set.

[0077] Example 6

[0078] On the basis of Example 1, considering the need for model training accuracy and model interpretability, a fully connected residual neural network model was built, such as Figure 4 As shown in Figure 1, the model is mainly composed of an input layer, a hidden layer and an output layer. The input layer and the output layer are the corresponding coordinates in the matching set of the left and right camera image feature point coordinates of the matched pixel coordinates. In the hidden layer, it is mainly composed of a fully connected network module and a residual network module. The fully connected network is mainly used to describe the nonlinear relationship between the left and right camera pixel points. The residual module is built based on the fully connected network. The purpose is to solve the problem that the fully connected network cannot achieve the same mapping and the gradient disappears during the training process. S2 is specifically: build a fully connected residual neural network model, based on the binocular camera image feature point coordinate matching set, with the coordinates of the left camera image feature points in the matching set as input, and the corresponding right camera feature point coordinates in the matching set as output. The fully connected residual neural network model is trained to obtain a neural network model for pixel matching of left and right view images in the case of no marking in the cave.

[0079] Example 7

[0080] On the basis of Example 6, in S2, during the training process of the fully connected residual neural network model, the fully connected residual neural network model is optimally solved by calculating the damage function based on the Langmuir distance:

[0081]

[0082] Where: is the matching coordinate obtained by the fully connected residual network, is the real coordinate in the feature point coordinate matching set, k 2 is the coordinate dimension, namely the horizontal and vertical directions.

[0083] Example 8

[0084] On the basis of Example 1, S3 is specifically as follows: taking the pixel coordinates in the left camera image as input, obtaining the pixel coordinate positions corresponding to the pixels of the left camera image in the right camera image through the trained fully connected residual neural network model, achieving complete matching of the pixels of the left and right camera images, and obtaining the full pixel matching coordinate set of the image, and obtaining the camera focal length f, object distance and other camera internal and external parameters through the binocular camera calibration parameters, combining the left and right camera image pixel matching results, establishing a three-dimensional coordinate point cloud model of the cave, and reconstructing the high spatial resolution three-dimensional morphology of the cave.

[0085] Furthermore, the three-dimensional coordinate point cloud model of the cavern is established in S3 as follows:

[0086] Suppose the first i The pixel coordinates are described as and , let the physical space coordinates of the cave structure corresponding to the pixel be In the present invention, it is considered that the two cameras are on the same horizontal plane, such as Figure 5 shown.

[0087] Based on the spatial position of the binocular camera, we can get , and through the triangular relationship we can get:

[0088]

[0089] Where: f and B is the calibration parameter, f represents the focal length of the camera, B Represents the basis moment between cameras;

[0090] According to the above relationship, the pixel i The corresponding three-dimensional coordinates of the cave space can be calculated as:

[0091]

[0092] At this point, the spatial position of any pixel in the cave images of the left and right cameras can be calculated, that is, the three-dimensional spatial point cloud coordinates of the cave structure can be obtained.

[0093] The obtained three-dimensional space point cloud coordinates are imported into the physical simulation software, and the point cloud data is converted into a surface entity, so that the three-dimensional structure of the cave is obtained. Specifically, the obtained three-dimensional space point cloud coordinates are imported into the software COMSOL, and the Model Builder module of COMSOL is used to convert the point cloud data into a surface entity, so that the three-dimensional structure of the cave is obtained.

[0094] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0095] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0096] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A binocular vision reconstruction method for high spatial resolution three-dimensional morphology of underground buildings, characterized in that: The following steps are involved: S1: Use a binocular camera to record images of the cave, detect and match the image feature points obtained by the binocular camera, correct the matching error, and obtain a feature point coordinate matching set; S2: Build a fully connected residual neural network model and train it according to the feature point coordinate matching set; S3: Input the pixel coordinates in the binocular camera image into the trained fully connected residual neural network model to obtain the image full pixel matching coordinate set, and establish the three-dimensional coordinate point cloud model of the cave through the binocular camera calibration parameters to reconstruct the high spatial resolution three-dimensional morphology of the cave; The S1 specifically includes: using a binocular camera to record an image of the cave, detecting the image feature points acquired by the binocular camera; constructing a pixel descriptor vector using the pixel brightness and gradient of the feature points, and matching the feature points of the binocular camera image by proposing a descriptor vector similarity measurement function; correcting the matching error of the feature points based on the error correction function, and obtaining a coordinate matching set of the binocular camera image feature points; The detection in S1 is specifically: The image acquired by the binocular camera is convolved with the Gaussian function to obtain the multi-scale space of the image, and the multi-scale difference space of the image is obtained through the adjacent scale images; Taking the pixel as the center, by comparing the brightness of the 5×5 neighborhood of the pixel and the 5×5 pixels above and below, the pixel with the largest brightness is selected as the image feature point, and by traversing all the pixels of the image, an image feature point set is formed; The pixel descriptor vector constructed in S1 is specifically: Select the 16×16 area with feature points and calculate the pixels in each 4×4 interval in the area. , , , , , , and The gradient direction histogram of 8 directions, make a cumulative value for each gradient direction, and get a feature point. Each feature point is composed of It is composed of seed points, each of which has vector information in 8 directions, generating feature information of 128 feature points in total, which is the descriptor vector of a feature point; Among them, the gradient of the feature point and direction The calculation formula is: Where: L The difference space where each feature point is located; The S3 is specifically as follows: taking the pixel coordinates in the left camera image as input, obtaining the pixel coordinate positions corresponding to the pixels in the left camera image in the right camera image through the trained fully connected residual neural network model, obtaining the image full pixel matching coordinate set, and establishing a three-dimensional coordinate point cloud model of the cave through the calibration parameters of the binocular camera, and reconstructing the high spatial resolution three-dimensional morphology of the cave; The three-dimensional coordinate point cloud model of the cave is established in S3 as follows: Suppose the first i The pixel coordinates are described as and , let the physical space coordinates of the cave structure corresponding to the pixel be ; Based on the spatial position of the binocular camera, we can get , and through the triangular relationship we can get: Where: f and B is the calibration parameter, f represents the focal length of the camera, B represents the basis moment between cameras, According to the above relationship, the pixel i The corresponding three-dimensional coordinates of the cave space can be calculated as: The obtained three-dimensional space point cloud coordinates are imported into the physical simulation software, and the point cloud data are converted into a surface entity to obtain the three-dimensional structure of the cave.

2. The binocular vision reconstruction method of high spatial resolution three-dimensional appearance of underground buildings according to claim 1 is characterized in that: The matching of the feature points of the binocular camera image in S1 is specifically as follows: The matching between feature points is achieved by calculating the cosine distance between the descriptor vectors of the feature points in the binocular camera image. cos(d) Calculated by the following formula: Where: is the descriptor vector of the feature points of the left view image, is the descriptor vector of the feature points of the right view image, k 1 is the dimension of the descriptor vector; the cosine distance of the descriptor vectors of the feature points of the left and right view images of the binocular camera is calculated by traversal. The more similar the descriptor vectors are, the larger the cosine distance is; the feature point pairs with larger cosine distances are selected as the matching results of the feature points.

3. The binocular vision reconstruction method of high spatial resolution three-dimensional appearance of underground buildings according to claim 2 is characterized in that: The correction of the matching error of the feature points based on the error correction function in S1 is specifically as follows: Taking the matched binocular camera image feature points as the core, the zero-mean normalized cross-correlation function is used to calculate the neighborhood correlation of the feature points, and the matching of the feature points is further corrected: Where: is the neighborhood correlation, and Left and right camera images respectively (x,y) The image brightness value at M is the feature point neighborhood size, and are the mean brightness values ​​of image pixels in the neighborhood respectively; By calculating the similarity of feature point matching pairs one by one, the feature point matching pairs with greater similarity are selected as the final binocular camera image feature point coordinate matching set.

4. The binocular vision reconstruction method of high spatial resolution three-dimensional appearance of underground buildings according to claim 3 is characterized in that: The S2 specifically includes: building a fully connected residual neural network model, based on the binocular camera image feature point coordinate matching set, taking the left camera image feature point coordinates in the matching set as input, and the corresponding right camera feature point coordinates in the matching set as output, training the fully connected residual neural network model, and obtaining a neural network model for pixel matching of left and right perspective images in an unmarked cave.

5. The binocular vision reconstruction method of high spatial resolution three-dimensional appearance of underground buildings according to claim 4 is characterized in that: In S2, during the training process of the fully connected residual neural network model, the fully connected residual neural network model is optimally solved by calculating the damage function based on the Langmuir distance: Where: is the matching coordinate obtained by the fully connected residual network, is the real coordinate in the feature point coordinate matching set, k 2 is the coordinate dimension, namely the horizontal and vertical directions.

Images