A neighborhood image stereo matching reconstruction method based on sparse point input

Abstract

The application discloses a kind of neighborhood image stereo matching reconstruction methods based on sparse point input, applied to three-dimensional reconstruction technical field, comprising: the network model including adaptive propagation module, differentiable transform module, neighborhood image map module is constructed;With sparse point as the input of network model, source image is iteratively regressed training, and the training result is optimized and compared by loss function, until the training frequency reaches the set threshold or the value of loss function reaches the set range;The input to be measured set is input into the network model trained, and three-dimensional point cloud is obtained, and fusion matching reconstruction is carried out.The application uses sparse point as reference, applies neighborhood image map, not only greatly reduces the operation amount of feature extraction network, and effectively solves the redundancy of depth information, improves the matching network precision on the basis of improving efficiency, while the optimization of loss function removes the parameter redundancy, so that the robustness of the trained network is higher, and the reconstruction precision is higher.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, and particularly relates to a neighborhood image stereo matching reconstruction method based on sparse point input. BACKGROUND

[0002] Three-dimensional reconstruction is widely used in military measurement, unmanned driving system, intelligent robot, medical diagnosis, digital city modeling, motion sensing entertainment and the like. Neighborhood image stereo matching is an end-to-end learning method belonging to three-dimensional reconstruction technology. The method introduces multi-scale matching of iterative neighborhood images and adaptively propagates through sparse point initialization, and has the characteristics of fast matching speed and high stability. In addition, in order to improve the effect of subsequent point cloud fusion, it is necessary to optimize the matched depth map. In order to obtain effective depth information in better dense reconstruction, in the existing multi-view stereo matching reconstruction method, a feature extraction network is mostly used to extract the features of the image as input and construct cost aggregation to obtain a dense depth map, but the calculation process is time-consuming and the accuracy is low.

[0003] Chinese patent publication No. CN113066168A, entitled "Multi-view stereo network three-dimensional reconstruction method and system", first acquires reference images and multiple actual shooting images of a target object to be reconstructed; then, a multi-scale feature extraction module is used to extract features of the multiple actual shooting images; then, through a regularization operation, a fixed depth interval is used to divide the depth map in a rough stage, and an adaptive interval is determined by using the uncertainty of depth prediction in a refinement stage; finally, according to the two-stage depth prediction interval, a final depth map with the same resolution as the reference image is inferred by using a cascaded depth architecture, and then the depth map is filtered and fused into a dense three-dimensional point cloud and reconstructed; the method uses a feature network to extract a feature map, which not only occupies a large amount of memory and takes a long time to train, but also has low reconstruction accuracy.

[0004] Therefore, how to provide a neighborhood image stereo matching reconstruction method based on sparse point input with less time consumption and high accuracy is a problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the present application provides a neighborhood image stereo matching reconstruction method based on sparse point input. The obtained depth map is smoother and more complete, has better reconstruction effect for subsequent point cloud fusion, is more consistent with the details of the object observation, and the present application uses a neighborhood image patch learning method, which has a simple implementation process, less processing time and higher reconstruction accuracy.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] A neighborhood image stereo matching reconstruction method based on sparse point input, comprising:

[0008] Step (1): a network model comprising an adaptive propagation module, a differentiable transform module and a neighborhood image mapping module is constructed; the differentiable transform module twists the source image features, which are input to the neighborhood image mapping module together with the sparse points; the neighborhood image mapping module takes the sparse points as a reference, performs patch mapping on the feature points through the image information of the neighborhood frame to obtain an initial depth map, and returns to the adaptive propagation module; the adaptive propagation module analyzes the returned initial depth map and inputs it to the differentiable transform module for twisting.

[0009] Step (2): the source image and the sparse points are iteratively trained as the input of the network model, and the training results are optimized and compared through a loss function until the number of training reaches a set threshold or the value of the loss function reaches a set range.

[0010] Step (3): the to-be-tested set is input into the trained network model to obtain a three-dimensional point cloud for fusion matching reconstruction.

[0011] Optionally, in step (1), the adaptive propagation module is composed of a convolution layer and a regularization layer, and is used for adaptive propagation in a manually built neural network; the adaptive propagation module collects the regression depth map from the same surface; in the process of collecting the regression depth map, the adaptive propagation module learns a two-dimensional offset to fix the grid; the convolution is used to obtain the feature point information of the regression depth map and perform a regularization operation to reduce the complexity; finally, the adaptive propagation module is integrated with the source view and input to the differentiable transform module.

[0012] Optionally, in step (1), the differentiable transform module is composed of four convolution layers and an S-shaped activation function, and is used for twisting the source image features; the second convolution, the third convolution and the fourth convolution network are used to calculate the three-dimensional pixel points corresponding to the feature point information of the regression depth map from the calibrated camera intrinsic parameters in the source view; the first convolution is used to calculate the two-dimensional pixel points corresponding to the feature point information of the regression depth map, and the S-shaped activation function is used to make the network training stable; finally, the feature points are integrated and the obtained sparse points are mapped to the neighborhood image mapping module.

[0013] Optionally, in step (1), the neighborhood image mapping module is composed of six convolutional layers, two regularization layers and two S-type activation functions, and the process is completed in two ways with the same operation, aiming to better iteration of information, and convolution one and convolution two simply straighten the obtained depth information; convolution three and convolution four perform neighborhood mapping with the obtained sparse points, and each time the patching is performed by similar comparison of sparse points and depth values, if the error range of similar values is small, it is considered that the patching is successful, otherwise, the similar points are removed; convolution five and convolution six integrate the patching operation and remove redundant information, and multiply the outputs of convolution three and convolution four, and finally integrate them together through splicing operation; the regularization layer and the activation function are used to keep the network training process in a balanced state, and finally the initial depth map is obtained.

[0014] Optionally, the expression of the regularization layer function is as follows:

[0015]

[0016] Wherein, X and y are training samples and corresponding labels respectively; ω is a weight coefficient vector; J(.) is an objective function; Ω(ω) is a penalty term; α is a parameter for controlling the strength of regularization and thus controlling the complexity of the model, reducing overfitting.

[0017] Optionally, the expression of the S-type activation function is as follows:

[0018]

[0019] Optionally, in step (2), the sparse points are useful feature points randomly collected from the DTU dataset composed of source images, depth map true values and camera internal and external parameters.

[0020] Optionally, in step (2), the loss function is composed of photometric similarity loss and robust regression loss.

[0021] Optionally, the expression of the photometric similarity loss is as follows:

[0022]

[0023]

[0024] L bs =(1-β)L Smooth1 +βL Smooth2 ;

[0025] Wherein, is the normal vector gradient perpendicular to the x plane, is the normal vector gradient perpendicular to the y plane, and both are continuous, x and y are pixel points of the image; β is a similarity coefficient; L Smooth1 and LSmooth2 L1 is a similarity loss of sparse points and L2 is a similarity loss two; bs L is a photometric similarity loss.

[0026] Optionally, the robust regression loss expression is as follows:

[0027]

[0028] L converges to 0 when δ approaches 0; and L converges to (x-y) when δ approaches infinity. H L converges to 0 when δ approaches 0; and L converges to (x-y) when δ approaches infinity. H L converges to 0 when δ approaches 0; and L converges to (x-y) when δ approaches infinity. 2 L converges to 0 when δ approaches 0; and L converges to (x-y) when δ approaches infinity. H The slope of the mean absolute error segment of L is δ; x and y are both pixel points of an image; δ is a function parameter; L is a robust regression loss; and the parameter β is 1 by default. H L converges to 0 when δ approaches 0; and L converges to (x-y) when δ approaches infinity.

[0029] Compared with the prior art, the application provides a neighborhood image stereo matching reconstruction method based on sparse point input. By using sparse points as input of the training network, the operation amount of the feature extraction network is greatly reduced. The three-dimensional points corresponding to the sparse points are projected back to the actual meeting position in the three-dimensional world, the robustness of the network is enhanced, the completeness of the point cloud matching reconstruction is high, and the effect is better. The operation of the neighborhood image mapping is used in the backbone network, the method of constructing the cost volume in the original multi-view stereo matching is changed, the redundancy of the depth information is effectively solved, the accuracy of the matching network is improved on the basis of improving the efficiency. The splicing operation is used on the neighborhood mapping network, the effective depth information is mixed, the network has stronger calculation loss ability for two different depth images, the robustness of the trained network is higher, and the reconstruction accuracy is higher. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, brief descriptions will be given to the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of the provided drawings.

[0031] Figure 1 FIG. 1 is a method flowchart of the present application.

[0032] Figure 2 FIG. 2 is a network structure diagram of the present application.

[0033] Figure 3The structural schematic diagram of the adaptive propagation module of the present application.

[0034] Figure 4 The structural schematic diagram of the micro-variable module of the present application.

[0035] Figure 5 The structural schematic diagram of the neighborhood image mapping module of the present application. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0037] Embodiment 1

[0038] Embodiment 1 of the present application discloses a neighborhood image stereo matching reconstruction method based on sparse point input, as shown in the figure, which comprises the following steps: Figure 1

[0039] Step (1): constructing a network model comprising an adaptive propagation module, a micro-variable module and a neighborhood image mapping module; the micro-variable module twists the source image features, which are input to the neighborhood image mapping module together with the sparse points; the neighborhood image mapping module takes the sparse points as a reference, performs patch mapping on the feature points through the image information of the neighborhood frame, obtains an initial depth map, and returns to the adaptive propagation module; the adaptive propagation module analyzes the initial depth map in stages and inputs to the micro-variable module for twisting.

[0040] As shown in the figure, the adaptive propagation module is composed of a convolution layer and a regularization layer, the convolution kernel size is 3x3, the step is 1, and the padding is 2, which is used for analyzing the initial depth map in stages, taking the union of the comparison and optimized results, and inputting to the micro-variable module, specifically: for adaptive propagation in the artificially built neural network, the adaptive propagation module collects the regression depth map from the same surface, learns two-dimensional offset to fix the grid in the process of collecting the regression depth map, obtains the feature point information of the regression depth map through convolution and performs regularization operation to reduce the complexity, and finally integrates with the source view and inputs to the micro-variable module. Figure 3

[0041] As shown in the figure, the micro-variable module is composed of a convolution layer and a regularization layer, the convolution kernel size is 3x3, the step is 1, and the padding is 2, which is used for twisting the source image features, and the twisted source image features are input to the neighborhood image mapping module. Figure 4 ​​As shown, the differentiable transform module is composed of four convolutional layers and an S-shaped activation function, the first convolutional layer is a two-dimensional convolution with a convolution kernel size of 3*3, a step of 1 and a padding of 2, the second, third and fourth convolutional layers are three-dimensional convolutions with a convolution kernel size of 1*1*1, a step of 1 and no padding, which are used to twist the input source image features and map them to the neighborhood image mapping module, specifically: the differentiable transform module is composed of four convolutional layers and an S-shaped activation function, which is used to twist the source image features, and the second, third and fourth convolutional layers are used to calculate the three-dimensional pixel points corresponding to the feature point information of the regression depth map through the calibrated camera parameters in the source view; the first convolutional layer is used to calculate the two-dimensional pixel points corresponding to the feature point information of the regression depth map, and the S-shaped activation function is used to make the network training tend to be stable, and finally the sparse points obtained through the integration of the feature points are mapped to the neighborhood image mapping module.

[0042] As shown in the figure, Figure 5 As shown, the neighborhood image mapping module is composed of six convolutional layers, two regularization layers and two S-shaped activation functions, the first and second convolutional layers are two-dimensional convolutions with a convolution kernel size of 3*3, a step of 1 and a padding of 2, the third, fourth, fifth and sixth convolutional layers are three-dimensional convolutions with a convolution kernel size of 1*1*1, a step of 1 and no padding, which are used to take the sparse points as a reference, repair the feature points through the image information of the neighborhood frame, obtain the initial depth map, effectively solve the redundancy of depth information, and improve the matching network precision on the basis of improving the efficiency, specifically: it is completed in two ways and the operation is the same, and the purpose is to better iterate the information, the first and second convolutional layers simply straighten the depth information obtained; the third and fourth convolutional layers perform neighborhood mapping with the sparse points obtained, and each time the repair is compared with the sparse points and the depth values, if the error range of the similar values is small, the repair is considered to be successful, otherwise the similar points are removed; the fifth and sixth convolutional layers integrate the repair operation and remove the redundant information, and multiply the outputs of the third and fourth convolutional layers, and finally integrate them together through the splicing operation; the regularization layer and the activation function are used to make the network training process in a balanced state, and finally obtain the initial depth map.

[0043] The dimension of the network splicing is 1, and the pooling domain of the max pooling is 2*2.

[0044] The expression of the regularization layer function is as follows:

[0045]

[0046] Wherein, X and y are the training samples and the corresponding labels respectively; ω is the weight coefficient vector; J(.) is the objective function; Ω(ω) is the penalty term; α is used to control the strength of regularization and thus control the model complexity and reduce overfitting parameters.

[0047] The expression of the S-type activation function is as follows:

[0048]

[0049] Step (2): the source image and sparse points are taken as the input of the network model for iterative regression training, an initial depth map is obtained, and the initial depth map is optimized and compared through a loss function, the union of the comparison and optimization results is taken to obtain a more smooth dense depth map, and then a better matching reconstruction effect is achieved, until the training times reach a set threshold or the value of the loss function reaches a set range, the training is completed, and the model parameters are saved.

[0050] In the formula, the training time threshold is set to 16, and the upper limit of the number of pictures input to the network each time is mainly determined according to the performance of the computer graphics processor. Generally, the number of pictures input to the network each time is in the range of 1-4, which can make the network training more stable, the training result better, and the network quickly fitted. The number of sparse points selected is 0.05xHxW, H is the image height, and W is the image width. The selection method is a random sampling method. In addition, the coordinates of the four corner points are set according to the maximum depth value and the minimum depth value in the given camera parameters, and a little random noise is added in the selection process to enhance the stability of the image. In the training process, the learning rate of the parameter is 0.001, which can ensure the network to be quickly fitted without causing the network to be over-fitted. The adaptive matrix estimation algorithm is selected as the algorithm of the parameter optimizer, and the main advantage of the adaptive matrix estimation algorithm is that after bias correction, the learning rate of each iteration has a certain range, so that the parameters are relatively stable. The threshold of the loss function is set to about 0.0003, and the training of the entire network can be considered to be basically completed when the threshold is less than 0.0003.

[0051] The sparse points are useful feature points randomly collected from a DTU data set composed of a source image, a depth map frame value, and camera internal and external parameters.

[0052] The source image is composed of 124 different objects or scenes, each object is photographed from 49 different angles, and each angle has 7 different brightness, so there are 343 pictures in each object or scene folder, and there are also training image sets corresponding to the depth map frame values of the source images. The resolution of each image is 1600x1200. Among them, the data set does not have sparse points, so the sparse points are directly selected from the ground frame value in the present application, the number of selected points is 0.05xHxW, H is the image height, and W is the image width. The selection method is a random sampling method. In addition, the coordinates of the four corner points are set according to the maximum depth value and the minimum depth value in the given camera parameters, and a little random noise is added in the selection process to enhance the stability of the image.

[0053] The loss function is composed of photometric similarity loss and robust regression loss.

[0054] The photometric similarity loss expression is as follows:

[0055]

[0056]

[0057] L bs =(1-β)L Smooth1 +βL Smooth2 ;

[0058] wherein, is a normal vector gradient perpendicular to the x plane, is a normal vector gradient perpendicular to the y plane, and both are continuous, x and y are both pixels of the image; β is a similarity coefficient; L Smooth1 and L Smooth2 are similarity loss one and similarity loss two of sparse points; L bs is photometric similarity loss.

[0059] The robust regression loss expression is as follows:

[0060]

[0061] wherein, when δ tends to 0, L H converges to 0; when δ tends to infinity, L H converges to (x-y) 2 ; the slope of the average absolute error segment of L H is δ; x and y are both pixels of the image; δ is a function parameter; L H is robust regression loss; the parameter β is 1 by default.

[0062] According to the above two loss functions, the overall loss function with weights can be constructed as follows:

[0063] L=λ1L bs +λ2L δ ;

[0064] wherein, the weights are assigned as follows through a large number of comparisons and multiple tests: λ1=0.4, λ2=0.45.

[0065] Step (3): input the to-be-tested set (which can be Tank&Temples dataset and ETH3D dataset) into the trained network model to obtain a three-dimensional point cloud, perform fusion, and use MeshLab software to view the matching reconstruction effect after fusion.

[0066] As Figure 2As shown, it is a network overall structure diagram of the application.

[0067] The embodiment of the application constructs a neighborhood image stereo matching network with sparse point input, uses sparse points and source images as input, directly obtains a depth map and fuses into a point cloud, and no longer passes through other steps, thereby avoiding manual design of a stereo matching algorithm in a traditional method. By calculating a related index of a depth map obtained by the prior art, the feasibility and superiority of the method are further verified. The related index comparison between the prior art and the method proposed in the application is shown in Table 1.

[0068] Table 1: Related index comparison between the prior art and the method proposed in the application

[0069]

[0070] The above method is randomly taken by a data set, and is more widely applied. In addition, the image mapping network is an important implementation function of the application.

[0071] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0072] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the application. Various modifications of the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for stereo reconstruction of neighborhood images based on sparse point input, characterized in that, include: Step (1): Construct a network model including an adaptive propagation module, a differentiable transformation module, and a neighborhood image mapping module; The differentiable transformation module twists the features of the source image and inputs them together with the sparse points into the neighborhood image mapping module. The neighborhood image mapping module uses the sparse points as a reference and modifies and maps the feature points using the image information of the neighborhood frames to obtain an initial depth map, which is then returned to the adaptive propagation module. The adaptive propagation module performs a stage analysis on the returned initial depth map and inputs it into the differentiable transformation module to perform the twisting. Step (2): Use the source image and sparse points as input to the network model for iterative regression training, and optimize and compare the training results through the loss function until the number of training times reaches a set threshold or the value of the loss function reaches a set range. Step (3): Input the test set into the trained network model to obtain the 3D point cloud, and perform fusion matching reconstruction; In step (1), the neighborhood image mapping module consists of six convolutional layers, two regularization layers, and two sigmoid activation functions. This process is completed in two separate paths with identical operations, aiming to improve information iteration. Convolution 1 and Convolution 2 simply straighten the obtained depth information; Convolution 3 and Convolution 4 perform neighborhood mapping with the obtained sparse points. Each patching is performed by comparing the sparse points with the depth values. If the error range of the similarity values ​​is small, the patching is considered successful; otherwise, the patching is considered unsuccessful and the similar points are removed; Convolution 5 and Convolution 6 integrate the patching operations, remove redundant information, multiply the outputs of Convolution 3 and Convolution 4, and finally integrate them together through a splicing operation; The regularization layer and the activation function are used to keep the network training process in a balanced state and finally obtain the initial depth map.

2. The neighborhood image stereo matching reconstruction method based on sparse point input according to claim 1, characterized in that, In step (1), the adaptive propagation module consists of a convolutional layer and a regularization layer, which is used to adaptively propagate in the artificially constructed neural network. The adaptive propagation module collects regression depth maps from the same surface. During the process of collecting regression depth maps, the adaptive propagation module learns two-dimensional offsets to fix the grid, obtains feature point information of the regression depth map through convolution and performs regularization operation to reduce complexity, and finally integrates it with the source view and inputs it to the differentiable transformation module.

3. The neighborhood image stereo matching reconstruction method based on sparse point input according to claim 2, characterized in that, In step (1), the differentiable transformation module consists of four convolutional layers and a sigmoid activation function, which is used to twist the features of the source image. Through convolutional layers 2, 3, and 4, the three-dimensional pixel points corresponding to the feature point information of the regression depth map are calculated based on the camera intrinsic parameters calibrated in the source view. Convolutional layer 1 calculates the two-dimensional pixel points corresponding to the feature point information of the regression depth map, and together they pass through the sigmoid activation function to stabilize the training of the network. Finally, the feature points are integrated and the obtained sparse points are mapped to the neighborhood image mapping module.

4. The neighborhood image stereo matching reconstruction method based on sparse point input according to claim 1, characterized in that, The expression for the regularization layer function is as follows: in, and These are the training samples and their corresponding labels; This is a vector of weight coefficients; The objective function is... This is a penalty item; To control the strength of regularization, and thus the model complexity, parameters are used to reduce overfitting.

5. The neighborhood image stereo matching reconstruction method based on sparse point input according to claim 1, characterized in that, The expression for the sigmoid activation function is as follows: 。 6. The neighborhood image stereo matching reconstruction method based on sparse point input according to claim 1, characterized in that, In step (2), the sparse points are useful feature points obtained by randomly collecting DTU datasets. The DTU dataset consists of source images, ground truth depth maps, and camera intrinsic and extrinsic parameters.

7. The neighborhood image stereo matching reconstruction method based on sparse point input according to claim 5, characterized in that, In step (2), the loss function consists of photometric similarity loss and robust regression loss.

8. The neighborhood image stereo matching reconstruction method based on sparse point input according to claim 7, characterized in that, The expression for the photometric similarity loss is as follows: in, To be perpendicular to The gradient of the normal vector of the plane. To be perpendicular to The gradient of the normal vector of the plane, and all of them are continuous. and Pixels are also part of an image; The similarity coefficient; and For sparse points, there are two similarity losses: loss one and loss two. This represents the loss of photometric similarity.

9. The neighborhood image stereo matching reconstruction method based on sparse point input according to claim 7, characterized in that, The robust regression loss expression is as follows: Among them, when When it approaches 0, Converges to 0; when As it approaches infinity, converges to ; The slope of the mean absolute error segment is ; and Pixels are also part of an image; For function parameters; For robust regression loss; parameters The default value is 1.

Images