Space target three-dimensional reconstruction method based on epipolar line mapping constraint dense search

Abstract

The invention relates to the technical field of computer vision and image processing, in particular to a space target three-dimensional reconstruction method based on epipolar line mapping constraint dense search. The processing can reduce the dependence on transformation matrix projection parameters in the weak texture region matching process. A kernel line coordinate system is established, orientation processing is carried out on the relative direction of a matched feature image pair, a new inter-pixel mapping relation is obtained, the inter-pixel mapping relation is used as dense search constraint in a triangulation network, and dense search has a better search effect in a weak texture area. And integrating the kernel line mapping search result and the existing epipolar constraint relationship, and finally performing multi-view reconstruction to obtain a target physical model. According to the method, dense point searching between matched image pairs can be effectively carried out, pixel point level searching precision can be achieved, and three-dimensional reconstruction of a space target is achieved.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision and image processing, and in particular to a three-dimensional reconstruction method of a space target based on epipolar mapping constrained dense search. Background Art

[0002] With the advancement of space science and technology, relying on two-dimensional imaging to extract the features of space targets is no longer sufficient to meet the requirements of complex space missions. Understanding the three-dimensional structure and shape of the target is crucial to achieving detailed perception. Three-dimensional reconstruction is a technology that processes and analyzes the two-dimensional information of an object to restore the real physical information of the object and construct a three-dimensional model. Three-dimensional reconstruction technology is an important part of three-dimensional imaging and an important research area in the field of electronic imaging. It helps to accurately quantify and analyze various attributes and obtain accurate physical structure information of space targets. It plays an important role in detecting, identifying and real-time monitoring of space targets. At the same time, the implementation of three-dimensional reconstruction can greatly improve the extraction and understanding of objective information related to the object, thereby laying the foundation for more complex visual tasks. It has been applied in different disciplines, including medical imaging, remote sensing and industrial detection.

[0003] However, the existing technology is obviously scarce when reconstructing the planar weak texture area of ​​the space target. This is because the planar weak texture area lacks a structure with strong geometric changes, which limits the extraction method of geometric features, and the gradient change of the area is not obvious, making it difficult to find enough feature points for reconstruction. At the same time, in the specific reconstruction stage, due to insufficient collection of feature points in the weak texture area, the feature point matching performance is reduced, which easily generates holes in the point cloud model generation process. Summary of the invention

[0004] Aiming at the problem of sparsity of feature matching points in the reconstruction process and the problem of holes in the reconstructed point cloud in weak texture areas, the present invention proposes a three-dimensional reconstruction method for spatial objects based on kernel line mapping constrained dense search.

[0005] In order to achieve the above object, the technical solution specifically adopted by the present invention is as follows:

[0006] A method for three-dimensional reconstruction of space objects based on dense search of epipolar mapping constraints includes the following steps:

[0007] During the sparse reconstruction phase:

[0008] Establish visual connections between multiple images to facilitate image feature extraction and subsequent matching; then construct a nonlinear scale space and extract robust feature points by fitting the normalized local maximum.

[0009] Feature matching of robust feature point descriptors is performed; specifically, the feature point descriptors are used to represent the block difference information of the area block where the feature points of the reference image are located, and the descriptor representation information is constructed for each pixel point in the neighborhood of the feature points of the image to be matched; for each feature point on the reference image, the area centered on the corresponding feature point on the image to be matched is traversed, and according to the number of identical values ​​of the reference image feature point descriptors and the pixel point descriptors of the image to be matched at the same position, the pixel point in the area with the largest value is selected as the new feature point corresponding to the image feature point to achieve feature matching.

[0010] The stable feature point pairs that have completed feature matching are triangulated by a triangulation algorithm to generate a sparse point cloud;

[0011] During the intensive reconstruction phase:

[0012] Establish a kernel coordinate system, perform orientation processing on the relative directions of the matching feature image pairs, and obtain the kernel line mapping relationship, that is, the mapping relationship between pixels;

[0013] Based on the stable feature points that complete feature matching, a triangulation network is constructed, and then the kernel line mapping relationship and the epipolar line constraint of the epipolar geometry are integrated to search for dense points within the triangle;

[0014] Normalized cross-correlation functions were calculated to verify the accuracy of dense points;

[0015] Determine the spatial coordinates of dense points that meet the accuracy standard and obtain a complete point cloud model of the target.

[0016] Furthermore, for each candidate matching point, the normalized cross correlation (NCC) coefficient is used to evaluate the similarity between two image patches, thereby facilitating the matching process.

[0017] Furthermore, based on the sparse matching of corresponding points in the left and right images and the inherent parameters of the camera, a kernel coordinate system is established to align the relative orientations of the left and right images, and the kernel orientation coordinates corresponding to the original coordinates of any image pixel are obtained. Then, the color values ​​of the epipolar image points are obtained from the original image through grayscale resampling, and all the pixels of the epipolar image are traversed to finally generate an image pair with associated mapping, and the kernel line mapping relationship, that is, the mapping relationship between pixels, is obtained.

[0018] Furthermore, the constructed kernel-line mapping relationship is used as a dense point search constraint in the triangulation network.

[0019] Furthermore, when searching for dense points within a triangle, the minimum circumscribed rectangle is found based on the stable point of the triangle, and the pixels in the defined area are traversed. The elements that meet the conditions are stored in the set as points to be searched until all pixels in the area are searched.

[0020] Furthermore, for color images, the grayscale mean of each channel is calculated separately, and the cross-correlation coefficient of all color channels is calculated separately, and the cross-correlation coefficient is used to evaluate and filter out mismatched points.

[0021] Furthermore, the spatial coordinates of dense points that meet the accuracy standard are determined through multi-view geometry method to obtain a complete point cloud model.

[0022] In the dense point search part after the triangulation network is constructed based on sparse stable feature points, the present invention introduces the epipolar mapping constraint by establishing the epipolar coordinate system, combines the epipolar constraint of the epipolar geometry, performs one-dimensional search for the best matching point of the points to be matched in the triangle to obtain dense matching points, and obtains the target physical model through multi-view reconstruction, thereby obtaining a more complete three-dimensional reconstructed point cloud model, which has the following characteristics and beneficial effects:

[0023] Additional matching constraints are introduced in the dense matching part of the reconstruction, which can reduce the dependence on the transformation matrix projection parameters in the matching process of weakly textured regions.

[0024] By establishing a kernel line coordinate system, the relative directions of the matching feature image pairs are oriented to obtain a new mapping relationship between pixels. Then the constructed kernel line mapping is used as a dense search constraint in the triangulation network, which makes the dense search have a better search effect in weak texture areas.

[0025] Finally, the kernel line mapping search results and the existing epipolar line constraint relationship are integrated to obtain the target physical model through multi-view reconstruction, and then a more complete 3D reconstructed point cloud model can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0027] Figure 1 This is an overall flow chart of a method for three-dimensional reconstruction of a space target based on an epipolar mapping constrained dense search according to an embodiment of the present invention;

[0028] Figure 2 A diagram describing the relative orientation of a triangle in a kernel coordinate system in an embodiment of the present invention;

[0029] Figure 3 A three-dimensional reconstructed space target simulation image in an embodiment of the present invention;

[0030] Figure 4 A schematic diagram of a feature matching effect in an embodiment of the present invention;

[0031] Figure 5 A point cloud model rendering of a spatial target reconstructed in an embodiment of the present invention; DETAILED DESCRIPTION

[0032] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0033] Example 1

[0034] Taking two images in the three-dimensional reconstruction of sequential images as an example, first establish a visual connection between the two images, construct a nonlinear scale space, and extract robust feature points by fitting the normalized local maximum; then, perform feature matching of the robust feature point descriptors to obtain robust feature points that complete feature matching. Then, based on the sparse matching of corresponding points in the left and right images and the inherent parameters of the camera, establish a kernel coordinate system to align the relative orientation of the left and right images, thereby obtaining accurate directional elements. The left image in the image pair is defined in the kernel coordinate system by determining the vector coordinates corresponding to the X, Y, and Z axes in the left camera coordinate system. The origin of the left camera coordinate system is designated as the origin of the kernel coordinate system, and the translation vector t obtained by the relative orientation is the coordinate of the vector in the left camera coordinate system. The coordinates of the axis vector of the kernel coordinate system in the left camera coordinate system are given by Find out.

[0035] The average value Z' of the Z-axis direction vectors of the left and right cameras is calculated by the following formula:

[0036]

[0037] Where Z1 = (0, 0, 1) and Z r =(r 13 ,r 23 ,r 33 ) are the Z-axis vectors of the left camera and the right camera respectively. 13 ,r 23 ,r 33 is the element of the third column of the rotation matrix R obtained by relative orientation. Then the coordinate of the Y-axis vector of the kernel coordinate system in the left camera coordinate system is Y=Z′X. And the coordinate of the Z-axis vector of the kernel line coordinate system in the left camera coordinate system is Z=XY. A similar method to the image on the right can be used to construct the vector coordinates in the kernel line coordinate system. After constructing the relative orientation of the kernel coordinates, the kernel orientation coordinates corresponding to the original coordinates of any image pixel can be obtained.

[0038] Figure 2The relative orientation diagram of the kernel line map constructed in the kernel coordinate system is shown. Among them, p1(x1,y1), p2(x2,y2) and p3(x3,y3) are the three vertices of the triangle, and p(x,y) is a point in the triangle. The corresponding x-axis disparities of the corresponding points are d x1 , d x2 and d x3 . p 12 (x 12 ,y) and p 13 (x 13 ,y) is the intersection of the triangle's y pixel row and the two sides and. Point p 12 and p 13 The parallax d x12 and d x13 It is obtained by linear interpolation along the y-axis. Specifically:

[0039]

[0040] The coordinates of any corresponding initial matching point can be obtained based on the disparity. After constructing the relative orientation mapping of all points under the kernel coordinates, the kernel orientation coordinates corresponding to the original coordinates of any image pixel can be obtained. Then, the color value of the epipolar image point is obtained from the original image through grayscale resampling. All the pixel points of the epipolar image are traversed, and finally an image pair with an associated mapping is generated to obtain a new kernel line mapping relationship.

[0041] The constructed kernel line mapping relationship is used as a dense point search constraint in the triangulation network. The construction of the triangulation network is based on the robust feature point construction that completes the feature matching. Then, the above kernel line mapping and the epipolar line constraint of the epipolar geometry are integrated to perform a one-dimensional search for the best matching point in the triangle to obtain dense matching points. In the one-dimensional search, the minimum circumscribed rectangle is found based on the stable point of the triangle, and the pixels of the defined area are traversed. The elements that meet the conditions are stored in the set as the points to be searched until all the pixels in the area are searched. Then, an affine transformation is used to project a group of pixels into the corresponding triangle, and the above constraints are used to determine whether a point has the same name. For two groups of points, it is necessary to find a rotation matrix and a translation matrix that minimizes the sum of the squares of the distances between the point pairs. The goal is to determine the translation matrix that minimizes the objective function for the measurement point set. These calculations are used to optimize the matching point pairs to achieve pixel-level accuracy. Specifically, for any point to be matched p1(x,y), according to the disparity of the point, its rough matching point in the right image is estimated to be p2(x+d x ,y+d y ). With this point as the center, on the scanning line in the right figure [x+d x -r,x+d x+r] pixel range for one-dimensional search and matching. For color images, the grayscale mean of each channel is calculated separately, and the cross-correlation coefficient of all color channels is calculated separately. Finally, in the established triangulation network, this step is used to traverse the triangles in the network to generate the corresponding group of dense matching points.

[0042] Verify the normalized cross-correlation function to verify the accuracy of dense points; in the verification part of correlation matching, for each candidate matching point, the normalized cross-correlation (NCC) coefficient is used to evaluate the similarity between the two image blocks. If the correlation coefficient of the point is not less than t (t is generally taken as 0.7 to avoid the situation of no corresponding points caused by occlusion and ensure the correctness of matching), and the difference between the correlation coefficient and the second largest correlation coefficient is greater than a certain interval e (e is generally taken as 0.01 to ensure that the same-name points have a certain significance and improve the confidence), it is considered to be the best matching point, otherwise the point is eliminated, thereby promoting the matching process.

[0043] Through conventional multi-view Figure 3 The angle measurement method is used to obtain the spatial coordinates of dense points and obtain a complete point cloud model of the target. Specifically, triangulation is used to calculate the coordinates of spatial points based on the principle of triangle similarity. Combined with the internal and external parameters of the camera, the three-dimensional spatial coordinates of each point are obtained from the dense matching point pairs through the triangulation formula. First, the pixel coordinates of the matching points are normalized to remove the scale factor in the homogeneous coordinates. Then the coordinates are solved by a set of simultaneous linear equations, and the least squares method is used to solve the above set of simultaneous equations to obtain the coordinates of the spatial points. The three-dimensional coordinates of all matching point pairs are stored in coordinate order to form a point cloud coordinate data set. This data set is represented by a simple array structure, and each element contains the coordinates of a point.

[0044] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for three-dimensional reconstruction of space objects based on dense search of epipolar mapping constraints, characterized by: The steps include: During the sparse reconstruction phase: Establish visual connections between multiple images, construct a nonlinear scale space, and extract robust feature points by fitting the normalized local maximum; Perform feature matching of robust feature point descriptors; The stable feature point pairs that have completed feature matching are triangulated through a triangulation network to generate a sparse point cloud; During the intensive reconstruction phase: Establish a kernel coordinate system, perform orientation processing on the relative directions of the aforementioned multiple image matching pairs, and obtain the kernel line mapping relationship; Based on the stable feature points that complete feature matching, a triangulation network is constructed, and then the kernel line mapping relationship and the epipolar line constraint of the epipolar geometry are integrated to search for dense points within the triangle; Normalized cross-correlation functions were calculated to verify the accuracy of dense points; Determine the spatial coordinates of dense points that meet the accuracy standard and obtain a complete point cloud model of the target.

2. The method for three-dimensional reconstruction of space objects based on epipolar mapping constrained dense search according to claim 1, characterized in that: In the feature matching stage, for each candidate matching point, the normalized cross correlation (NCC) coefficient is used to evaluate the similarity between two image patches, thereby facilitating the matching process.

3. The method for three-dimensional reconstruction of space objects based on epipolar mapping constrained dense search according to claim 1, characterized in that: According to the sparse matching of corresponding points in the left and right images and the inherent parameters of the camera, a kernel coordinate system is established to align the relative orientations of the left and right images, and the kernel orientation coordinates corresponding to the original coordinates of any image pixel are obtained. Then, the color values ​​of the epipolar image points are obtained from the original image through grayscale resampling, and all the pixels of the epipolar image are traversed to finally generate an image pair with associated mapping and obtain the kernel line mapping relationship.

4. The method for three-dimensional reconstruction of space objects based on epipolar mapping constrained dense search according to claim 1, characterized in that: The constructed kernel-line mapping relationship is used as a dense point search constraint in the triangulation network.

5. The method for three-dimensional reconstruction of space objects based on epipolar mapping constrained dense search according to claim 1, characterized in that: When searching for dense points in a triangle, the minimum circumscribed rectangle is found based on the stable point of the triangle, and the pixels in the defined area are traversed. The elements that meet the conditions are stored in the set as points to be searched until all pixels in the area are searched.

6. The method for three-dimensional reconstruction of space objects based on epipolar mapping constrained dense search according to claim 1, characterized in that: For color images, the grayscale mean of each channel should be calculated separately, and the cross-correlation coefficient of all color channels should be calculated jointly. The cross-correlation coefficient can be used to evaluate and filter out mismatched points.

7. The method for three-dimensional reconstruction of space objects based on epipolar mapping constrained dense search according to claim 1, characterized in that: The spatial coordinates of dense points that meet the accuracy standard are determined through multi-view geometry method to obtain a complete point cloud model.

8. The method for three-dimensional reconstruction of space objects based on epipolar mapping constrained dense search as claimed in claim 1, characterized in that: The feature matching of the robust feature point descriptor is performed through the following steps: The area block where the feature point of the reference image is located uses the feature point descriptor to represent the block difference information, and each pixel point in the neighborhood of the feature point of the image to be matched constructs a descriptor representation information; by traversing the area centered on the corresponding feature point on the image to be matched for each feature point on the reference image, according to the number of identical values ​​of the reference image feature point descriptor and the pixel point descriptor of the image to be matched at the same position, the pixel point in the area with the largest value is selected as the new feature point corresponding to the image feature point to achieve feature matching.

Images