Rapid fine orientation method for three-dimensional construction of massive digital disordered images

Abstract

According to the mass digital disordered image three-dimensional construction rapid fine orientation method, in order to solve the problems that a traditional orientation method is low in efficiency and not easy to succeed, the matching frequency is reduced and the matching efficiency is improved by rapidly obtaining the approximate relation between the images, then the robust image relation is introduced into the orientation solution process, and the orientation accuracy is improved. And disordered image orientation based on the image relationship is realized. And an image vocabulary tree construction method is established, and matching time consumption is reduced. An image relation determination method based on global hash is provided, and an image relation undirected graph is constructed. In order to solve the problem that an error exists in an initial image relation, a weak correlation image relation removing method based on three-view matching constraint is provided, and a more stable image relation is obtained. In order to solve the problem of time consumption of incremental SFM orientation, an orientation strategy based on an image relation is provided, pose parameters of images and three-dimensional coordinates of matching points are globally solved, and especially when the number of digital disordered images is huge, the orientation precision and efficiency advantages are obvious.

Description

Technical Field

[0001] The present application relates to a method for three-dimensional orientation of disordered images, and in particular to a method for rapid and precise orientation of three-dimensional construction of massive digital disordered images, belonging to the technical field of disordered image orientation. Background Art

[0002] Three-dimensional reconstruction from digital images is low-cost and highly realistic, and has been widely used in various fields, including smart city construction, disaster emergency response, and the display and preservation of cultural relics. Simultaneously, with the continuous innovation and development of sensor technology and related processing techniques, image acquisition methods have gradually diversified. The widespread use of sensors such as mobile phones, digital cameras, standard SLR cameras, and portable drones has enabled people to obtain image information of interest anytime and anywhere through a variety of devices. Without specialized training or setup requirements, a large number of multi-view images of the captured object can be obtained. These images lack physical information or even camera calibration information. They can be captured by a variety of cameras under different lighting conditions, resulting in a large number of images and a rich collection of information. Research on three-dimensional reconstruction of digital images from large numbers and sources involves the intersection of computer vision, image science, pattern recognition, and image processing, and is of great value.

[0003] Furthermore, 3D reconstruction of disordered images also has broad application prospects. Acquiring 3D models can play a crucial role in various fields, including industrial production, medical research, and military training surveys. For example, in the construction of digital cities, 3D city models are beneficial for urban management and design, emergency response systems, environmental protection, and the deployment of communications facilities, contributing significantly to the development of smart cities. Building digital museums and digitally displaying cultural relics are effective approaches to cultural relic preservation. Applying 3D image reconstruction technology to the digitization of cultural relics, geometrically reconstructing 3D models of cultural relics from 2D disordered images, effectively enables digital display and thus proper preservation.

[0004] It is usually difficult to plan the route of a drone before takeoff, so the images obtained are disordered and messy, and may be obtained from multiple batches of flights. Therefore, it is very important to use these large numbers of disordered images for efficient and robust 3D reconstruction.

[0005] Therefore, 3D reconstruction of large numbers of disordered images has significant application value. A crucial step is efficiently acquiring the image's exterior orientation elements to restore its position and pose, i.e., orienting the disordered image. A method for orienting disordered images based on image relationships is urgently needed. This method improves matching efficiency by rapidly acquiring image relationships and then incorporates these relationships into the orientation process, achieving image-relationship-based orientation for disordered images.

[0006] Traditional multi-view 3D reconstruction is used for the orientation of disordered images. Due to the large number of images and unknown relationships, traditional methods are inefficient and difficult to succeed. Therefore, the technical difficulty lies in how to quickly obtain image relationships before matching, improve image matching efficiency, and bring robust image relationships into the image orientation process to achieve the purpose of rapid orientation of disordered images.

[0007] The problems that need to be solved in the existing technology of 3D orientation of massive disordered images and the key technical difficulties of this application include:

[0008] (1) The current image acquisition methods are becoming increasingly diverse. People can obtain image information of interest anytime and anywhere through various devices, and obtain a large number of multi-view images of the photographed objects. These images do not have physical information or even camera calibration information. They can be taken by various cameras under different lighting conditions. They are numerous and contain a large amount of information. Existing technologies are very difficult to perform three-dimensional reconstruction on digital images that are large in number and from a wide range of sources. The quality of three-dimensional reconstruction for disordered images is poor. The most critical step is how to efficiently obtain the external orientation elements of the image to restore the position and posture of the image, that is, the orientation of the disordered image. There is an urgent need for a disordered image orientation method based on image relations. By quickly obtaining image relations to improve matching efficiency, and introducing them into the orientation process, disordered image orientation based on image relations is realized.

[0009] (2) The existing technology cannot solve the orientation problem for disordered images well. The traditional multi-view image orientation method is inefficient and difficult to succeed. There is a lack of an orientation method for disordered images based on image relationships. It is impossible to improve the matching efficiency by quickly obtaining image relationships. There is a lack of fast orientation for disordered images based on image relationships. The existing technology addresses the problem of low efficiency of exhaustive matching of disordered images. There is a lack of an image relationship determination method based on global hashing. There are errors in the initial image relationships obtained. There is a lack of a weakly correlated image relationship elimination method based on three-view matching constraints, and it is impossible to obtain robust image relationships. The existing technology's incremental SFM orientation is relatively time-consuming, lacks an orientation strategy based on image relationships, and lacks three-view matching elements as constraints to globally solve the image pose parameters and the three-dimensional coordinates of the matching points. The orientation speed of disordered images is slow and the accuracy does not meet the requirements.

[0010] (3) Traditional orientation methods are inefficient and have low success rates. The matching efficiency between images is low, the robustness is poor, and the exhaustive matching efficiency of disordered images is low. The approximate relationship between images cannot be restored, and the matching takes a long time. There is a lack of a global hash-based image relationship determination method, which makes it impossible to build image relationships better and faster, and the stability is poor. The initial image relationship obtained by the existing technology is not robust enough, and it is impossible to effectively eliminate weakly associated image relationships and obtain a more robust image relationship undirected graph. The existing incremental SFM orientation is relatively time-consuming, lacks an orientation strategy based on image relationships, and cannot achieve fast and robust orientation based on image relationships. Especially when the number of digital disordered images is huge, the accuracy and efficiency of orientation drop rapidly. Summary of the Invention

[0011] This application addresses the low efficiency and difficulty of traditional orientation methods for unordered multi-view images. By developing an unordered image orientation method based on image relationship construction, this method reduces the number of matches by quickly acquiring approximate relationships between images, thereby improving matching efficiency. Robust image relationships are then introduced into the orientation solution process, enabling unordered image orientation based on image relationships. To address the low efficiency of exhaustive matching for unordered images, a method for constructing an image vocabulary tree is established. The quality of the constructed visual vocabulary is comprehensively considered. Based on an improved TF-IDF model, approximate relationships between images in an image set are determined, restoring the approximate relationships and reducing matching time. A global hashing-based image relationship determination method is proposed to construct an undirected image relationship graph. To address the issue of errors in the initial image relationships, a method for eliminating weakly associated image relationships based on three-view matching constraints is proposed to obtain more robust image relationships. To address the time-consuming nature of incremental SFM orientation, an image relationship-based orientation strategy is proposed to globally compute image pose parameters and the three-dimensional coordinates of matching points. This advantage in orientation accuracy and efficiency is particularly evident when the number of digital unordered images is large.

[0012] To achieve the above technical effects, the technical solutions adopted in this application are as follows:

[0013] A fast and precise orientation method for 3D construction of massive digital disordered images uses a global solution algorithm based on three-view matching elements to complete disordered image orientation based on image relationships. The technical framework is as follows:

[0014] Step 1: Estimate the relative orientation of each image pair using a five-point method based on a random sampling consensus architecture.<i,j> The relative rotation matrix R between i，j and the relative displacement direction vector t i，j , and calculate the number of inliers in the image. If the number of inliers used for calculation is less than a fixed critical value, it is determined that the relative pose of the image is not accurate enough and the image pair is discarded;

[0015] Step 2: According to the listed relative rotation matrix R i，j With the global rotation matrix R i and R j The relationship between , each image pair lists 9 equations, where R I The global rotation matrix of any image i is solved by the least squares method to obtain an estimate, and the orthogonality is satisfied by the SVD decomposition algorithm.

[0016] Step 3: For each three-view matching point in the three-view matching element, according to its object space coordinate X P The coordinates of the image plane list the equation of the point reprojection error, and solve the local translation vector that minimizes the reprojection error ρ and the coordinates X of the object space point P P The value of

[0017] Step 4: The three-view matching points are judged based on whether the reprojection error exceeds the critical value, and the three-view matching points that exceed the critical value are deleted. At the same time, it is judged whether the number of three-view matching points reaches the critical value for the three-view matching element. If it is less than the fixed critical value, the three-view matching element is deleted. After deletion, a depth-first search is performed to ensure that the undirected graph has connectivity, and isolated points in the vertex set are deleted. Finally, the unordered image relationship graph is updated;

[0018] Step 5: Leveraging the superior robustness of three-view matching between multi-view stereo pairs, a linear programming method is used to find the optimal solution for the global consistency position of the image projection center in the global coordinate system that minimizes the Euclidean distance.

[0019] Step 6: Use computational visual triangulation to solve the equations through iterative linear least squares to iteratively solve the spatial coordinates of all multi-view matching points, that is, the encrypted points, to restore the 3D structure of the scene;

[0020] Step 7: Perform a global bundle adjustment to globally optimize the pose parameters and point positions to obtain the final result.

[0021] Preferably, the present application reduces the number of matches by quickly acquiring the approximate relationship between images, and then introduces the robust image relationship into the orientation solution process to achieve disordered image orientation based on image relationship:

[0022] 1) Establish a global hash-based image relationship determination method. By constructing a multi-table hash for feature points, the image code of the image is obtained. Then, based on the bucketing method of hierarchical hashing, multiple groups of LSH hash functions are used to quickly determine the approximate relationship between images and construct an undirected image relationship graph.

[0023] 2) To address the problem that the initial image relationships are not robust enough, a method for eliminating weakly correlated image relationships based on three-view matching constraints is established. Based on the results obtained from fast matching, the image relationships are screened and updated to obtain robust image relationships.

[0024] 3) To address the time-consuming problem of incremental SFM orientation, a global solution method is adopted to establish an orientation strategy based on image relations. With three-view matching elements as constraints, the image pose parameters and the three-dimensional coordinates of the matching points are globally solved according to the rapidly acquired robust image relations, thus achieving fast and robust orientation based on image relations.

[0025] Preferably, the specific steps of the global hash improved disordered image relationship construction method are:

[0026] Step 1: Based on the LSH algorithm, all feature points extracted from all images in the image set are mapped into m (m = 8)-bit binary codes through L (L = 6) LSH hash functions. Then, a short-coded multi-table hash is constructed based on the binary codes of the feature points, so that similar feature points fall into the same hash bucket.

[0027] Step 2: For an image, the hash table formed by bucketing all the feature points on the image obtained in step 1 according to the hash algorithm is used as a global description of the image, indicating the different classifications of SIFT feature points on the image and the number of each category. If two images are exactly the same, then the hash tables obtained must also be completely consistent, that is, the hash lists obtained by approximate images must also be approximate. The image hash list obtained by bucketing the feature points through the short coding hash transformation is used as the image code describing the image; the number of buckets in the hash table is the number of bits of the image code, and then the number of feature points contained in each bucket in the hash list is calculated, and it is used as the value of the image code bit corresponding to the hash bucket in the image code to obtain the image code of each image; after constructing a multi-table hash with L different function mappings, each image obtains L image codes to describe it;

[0028] Step 3: Based on the feature point bucketing in the hierarchical hashing algorithm, the image code of the image is divided by using the short-coded LSH hash function. First, the feature code of the image is normalized, and then the normalized image code is encoded and bucketed using the LSH function. This allows similar images to be grouped into the same bucket, achieving differentiation between different images. Multiple LSH functions (L = 6) are used for bucketing to obtain various bucketing results.

[0029] Step 4: Images that fall into the same hash bucket are approximate images. For the L groups of image codes in this application, L functions are used to bucket them, and L*L, or 36 groups of results, are obtained. The number of times they fall into the same bucket is used as the weight of the approximate image, and an image relationship list with a weight w is obtained (0≤w≤36). The larger the weight w, the higher the similarity between the images.

[0030] Preferably, weakly associated image relationships are eliminated based on three-view matching: the present application obtains an initial image relationship undirected graph G(V, E), and based on the results obtained by fast matching, eliminates weakly associated image relationships therein based on the constraints of three-view matching, obtains robust image relationships, and orients the image set;

[0031] According to the results of fast matching, based on the constraints of three-view matching, the weakly associated parts in the image relationship graph are eliminated, including the following steps:

[0032] Step 1: When obtaining the results of image matching, first calculate and judge the number of matching points obtained by matching between any image pairs, and delete the image relationships with fewer matching point pairs than a fixed critical value (50), that is, delete the corresponding edges in the image relationship undirected graph and delete the image pairs with smaller overlap;

[0033] Step 2: For disordered multi-view image sets, matching constraints based on three views can better ensure the robustness of 3D reconstruction. Using the three views as constraints, we can filter and eliminate unstable parts in the image relationships.

[0034] Among them, the three-view matching element of the image refers to: if the three edges connecting the three images i, j, and k<i,j> ,<i,k> ,<j,k> exist, and there are enough three-view matching points, then images i, j, k constitute a three-view matching element;

[0035] In this process, we first search for three-view matching elements in the image relationship undirected graph G. When the three edges connecting any three images i, j, and k are<i,j> ,<i,k> ,<j,k> If all of them exist in the set E, and the number of three-view matching points on the image is greater than a fixed critical value, then it is determined that there is a correlation between the three images, which is recorded as t<i,j,k> At this point, all three-view matching elements in the image set are obtained, that is, a three-view set T is formed; then, a new undirected graph G'=(V', E') is constructed from the initial vertex set V and the edges contained in the three-view set T. Matching point pairs that do not satisfy multi-view visibility are deleted, and then image relationships that do not satisfy the three-view constraint are deleted;

[0036] Finally, based on the fact that the undirected graph obtained by continuously deleting image relationships is not necessarily connected and has some redundant vertices, i.e., images, a depth-first traversal search algorithm is used to obtain the maximum connected component of the undirected graph, so that all images in the image relationship have related images.

[0037] The specific process is: assume that all vertices in the undirected graph (G'=(V', E')) have not been visited at the beginning of the traversal. Then, starting from a vertex v, visit the unvisited vertices adjacent to it in turn, and traverse the entire undirected graph in a depth-first manner until all vertices connected to v have been visited. At this time, there will be some vertices in the graph that have not been visited. Select one of the unvisited vertices as the source point of the new traversal and repeat the above traversal process until all vertices have been visited.

[0038] Get several connected components in the image relationship undirected graph, and select the undirected graph with the largest number of images as the final image relationship graph G final =(V final , E final ), if any image in the three views is not in the set of vertices V final If the three views are in the same file, then the three views will be eliminated.

[0039] Preferably, the process of the orientation part of multi-view 3D reconstruction includes three steps: first, matching all possible related image pairs to obtain the relative position relationship between the image pairs; second, calculating the absolute orientation of all images in a global unified coordinate system, including the pose at the time of image capture and the center coordinates of the camera, and then calculating the 3D coordinates of all spatial points; third, using the bundle adjustment method, calculating the calculation result that minimizes the reprojection error, thereby optimizing the calculation result of the previous step;

[0040] The system sequentially completes the relationship recovery, fast matching, and elimination of weakly correlated image relationships between multi-view images, and obtains a stable image relationship graph G = (V, E) between disordered images. The absolute orientation of all images in the global coordinate system is calculated, that is, the absolute rotation of the image and the center coordinates of the camera are solved.

[0041] A global solution optimization method that does not rely on an iterative framework is used to achieve the orientation of disordered images.

[0042] Preferably, the global solution based on the three-view matching element is as follows: the global solution is based on the three-view matching, and the three-view matching element of the image is obtained.<i,j,k> An image relationship graph G = (V, E) between the disordered images is constructed. The solution process revolves around the image relationship graph, including: estimating the global rotation matrix, local translation estimation based on three-view matching, globally consistent position estimation, and solving the spatial point coordinates.

[0043] Preferably, a global rotation matrix is ​​estimated: for each image pair in the relationship graph G = (V, E)<i,j> , first use the relative orientation method to calculate the relative rotation matrix R between them i，j and the relative displacement direction vector t i，j ;

[0044] This application uses the five-point relative orientation method under the random sampling consistent architecture to estimate the relative rotation matrix R between each image pair i，j and the relative displacement direction vector t i，j , assuming that point x on image i and point x' on image j are a pair of points with the same name, then the following relationship exists between the two points:

[0045] xE i，j x′=0 Formula 1

[0046] E i，j =[t i，j ]×R i，j Formula 2

[0047] Where E is the essential matrix, which is a 3×3 square matrix with a rank of 2 and a degree of freedom of 5. Only 5 points of the same name are needed to linearly solve the essential matrix. The essential matrix is ​​composed of the translation vector t i，j and the relative rotation matrix R i，j Composition, decompose the rotation and translation of the two images from the essential matrix;

[0048] The number of inliers in the image is calculated. If the number of inliers used for calculation is less than a fixed critical value, the relative pose of the image is considered to be inaccurate, that is, the image correlation of the image pair is poor, and it is removed.

[0049] Assume that the global rotation matrices of image i and image j in the global coordinate system are R i and R j , R i and R j are all orthogonal matrices, then they satisfy the following conditions:

[0050]

[0051] According to the above conditions, the equation is:

[0052]

[0053] Where r i m，m The global rotation matrix is ​​R j The value of the mth row and nth column (m=1, 2, 3), (n=1, 2, 3), is the relative rotation matrix R i，j The mth row of is the nth column of the global rotation matrix R;

[0054] Each image pair<i,j> All of them can be listed as 9 equations through formula 4, let R I As the unit matrix, the least squares method is used to solve the global rotation matrix of any image i to obtain the estimated value Among them, the orthogonality is constrained by the SVD decomposition algorithm, and the closest rotation matrix under the Frobenius norm is obtained by projection transformation. First, Perform SVD decomposition to obtain the product of three matrices, as shown in Equation 5:

[0055]

[0056] Among them, U i and V i are all 3×3 unitary matrices, ∑ i is a non-negative real diagonal matrix. Perform forced orthogonalization to obtain;

[0057] R i =U i V i H , i, j∈V Equation 6

[0058] Calculate the global rotation matrix R of any image in the image set in the global coordinate system.

[0059] Preferably, local translation estimation based on three-view matching: for each three-view matching element in the undirected graph G, obtain the global rotation matrix R of the three images i , R j , R k , will be based on the three-view matching element<i,j,k> , the local translation vector of the image projection center in the local coordinate system Solve the problem and remove the three-view matching points with large errors according to the reprojection errors of the three-view matching points.

[0060] Assume that in the image relationship undirected graph, the three-view matching element<i,j,k> The set of spatial points in the object space corresponding to all three-view matching points on the three images is G P , where the spatial coordinates of a point P in the object space are X P , its image coordinates on image i are Then the reprojection error of the image point is calculated as:

[0061]

[0062] Where, is the mth row (m=1, 2, 3) of the global rotation matrix R; is the local translation vector The mth row of , solves the local translation vector that minimizes the reprojection error ρ and the coordinates X of the object space point P P The value of , that is:

[0063]

[0064] Convert to L ∞ Solve the linear programming problem under the norm:

[0065] minη

[0066]

[0067] t1=(0,0,0) T Formula 9

[0068] That is Under the condition of , the reprojection error of the three-view matching point is listed as an equation to minimize the value of the reprojection error and solve the local translation vector

[0069] The accuracy of the three-view matching points is judged based on whether the calculated reprojection error exceeds the critical value, and the three-view matching points that exceed the critical value are deleted. Then the three-view matching elements of the image are judged again. After deleting some three-view matching points, whether the number of three-view matching points between the image pairs still reaches the critical value, if it is less than the fixed critical value, the three-view matching element needs to be deleted, and after deletion, a depth-first search is performed on the new image relationship to ensure that the undirected graph has connectivity, delete isolated points in the vertex set, and finally update the unordered image relationship graph.

[0070] Preferably, globally consistent position estimation: Get the relative rotation matrix R of each image pair in the edge set E of the undirected graph i，j , relative displacement direction vector t i，j , and the local translation vector and the global rotation matrix R i ,Next, based on the above results, the global consistency position of the image projection center is estimated;

[0071] For any image pair in the relationship graph G = (V, E)<i,j> The position should satisfy the following equation:

[0072]

[0073] Among them, λ i，jis the scale factor, and the position T of the projection center of all images in the global coordinate system is calculated using the least squares algorithm according to Equation 10. i ;

[0074] First, assume that the scale factor λ of any two images i，j Are not less than 1, and use the minimum Euclidean distance algorithm to transform the equation into L ∞ Solve it using linear programming under the norm:

[0075]

[0076] The global optimal solution of the projection center is calculated using linear programming, and the constraints of three-view matching are used to replace the binocular stereo image pair for solution;

[0077] by To replace the relative position translation vector t in the stereo pair i，j , the local scale coefficient in the three-view matching element is positive To replace λ in the stereo pair i，j , the equation listed in formula 11 is rewritten as:

[0078]

[0079] The linear programming method is used to solve the position of the projection center of the image in the global coordinate system according to the above formula, and the optimal solution of the globally consistent position is obtained. The result obtained based on the three-view matching constraint has better robustness than the stereo image pair.

[0080] Preferably, the spatial coordinates of all multi-view matching points, i.e., encrypted points, are solved to restore the three-dimensional structure of the scene:

[0081] Assume that an object point X in space is visible on n images, then the following equation is listed for each visible image:

[0082] x i =P i X, i = 1, 2, 3…n Formula 13

[0083] Where x i and X represent the homogeneous coordinates of point X on the image and in space respectively. The specific formula is:

[0084] x i =w i (u i , v i , l) T , X=(X,Y,Z,l) T Formula 14

[0085] Where (u i , vi ) is the coordinate of the image point on image i, w i is the scale coefficient, the matrix P i represents the camera projection matrix corresponding to image i, where K is the camera calibration matrix;

[0086] P i =K i R i [I|-C i ] i=1,2,3...n Formula 15

[0087] Formula 14 can be expressed as:

[0088]

[0089] Among them, P i k Represents the matrix P i The kth row (k=1, 2, 3) of , the formula is converted to:

[0090]

[0091] List n equations (17), first use least squares to solve the initial value, then use iterative linear least squares trigonometric method to iteratively solve the equation, continuously improve the accuracy of the result, and obtain the optimal solution.

[0092] Compared with the existing technology, the innovation and advantages of this application are:

[0093] (1) This application addresses the problem that traditional orientation methods for unordered multi-view images are inefficient and difficult to succeed. A method for orientation of unordered images based on image relationship construction has been developed. By quickly obtaining the approximate relationship between images, the number of matches is reduced, thereby improving the matching efficiency. Then, robust image relationships are introduced into the orientation solution process to achieve orientation of unordered images based on image relationships. To address the problem of low efficiency of exhaustive matching of unordered images, a method for constructing an image vocabulary tree is established, and the quality of the constructed visual vocabulary is comprehensively considered. The approximate relationship between images in the image set is determined based on the improved TF-IDF model. This method can restore the approximate relationship of images to a certain extent and reduce the matching time. A method for determining image relationships based on global hashing is proposed to construct an undirected graph of image relationships. To address the problem of errors in the initial image relationships obtained, a method for eliminating weakly associated image relationships based on three-view matching constraints is proposed to obtain more robust image relationships. To address the problem that incremental SFM orientation is relatively time-consuming, an orientation strategy based on image relationships is proposed to globally solve the image pose parameters and the three-dimensional coordinates of the matching points. Especially when the number of digital unordered images is huge, the orientation accuracy and efficiency advantages are obvious.

[0094] (2) Based on in-depth research on hash algorithms in the field of image retrieval and a summary of the computational process of the CasHash algorithm, this application proposes a method for determining image relationships based on global hashing. By constructing a multi-table hash for feature points, the image code of the image is obtained. Then, drawing on the bucketing idea of ​​hierarchical hashing, multiple groups of LSH hash functions are used to quickly determine the approximate relationship between images, thereby constructing an undirected image relationship graph. Experimental results show that this method can construct image relationships better and faster than the image vocabulary tree method, and is more stable.

[0095] (3) To address the problem that the initial image relationships obtained are not robust enough, this application proposes a method for eliminating weakly correlated image relationships based on three-view matching constraints. Based on the results obtained from fast matching, the image relationships are screened and updated to obtain robust image relationships. Experimental results show that this method can effectively eliminate weakly correlated image relationships and obtain a more robust undirected image relationship graph with fast orientation speed and high accuracy.

[0096] (4) To address the time-consuming nature of incremental SFM orientation, this application adopts the concept of global solution and proposes an orientation strategy based on image relationships. Using three-view matching elements as constraints, this strategy globally solves the image pose parameters and the three-dimensional coordinates of the matching points based on rapidly acquired robust image relationships, achieving rapid and robust orientation based on image relationships. Experimental results show that this orientation method can achieve rapid orientation of disordered images with accuracy that meets requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 This is a schematic diagram of obtaining image codes by constructing disordered image relationships improved by global hashing.

[0098] Figure 2 It is a schematic diagram of three-view matching element.

[0099] Figure 3 This is a schematic diagram of the global solution of camera position deviation based on three-view matching elements.

[0100] Figure 4 This is an example of the feature point extraction results in the experiment.

[0101] Figure 5 This is a schematic diagram of the exhaustive matching image relationship in the experiment.

[0102] Figure 6 This is a schematic diagram of the results of the first set of experimental image relationship construction.

[0103] Figure 7 It is a schematic diagram of the accuracy analysis of image relationship construction.

[0104] Figure 8 This is an analysis diagram of the efficiency of image relationship construction.

[0105] Figure 9 It is a result map of image relationship constructed separately by multi-source image sets.

[0106] Figure 10 It is the result diagram of constructing the image relationship of multi-source data.

[0107] Figure 11 This is a comparative analysis result diagram of directional methods. DETAILED DESCRIPTION

[0108] The following further describes the technical solution of the method for rapid and precise orientation of three-dimensional construction of massive digital disordered images provided by this application in conjunction with the accompanying drawings, so that those skilled in the art can better understand this application and implement it.

[0109] Three-dimensional reconstruction using digital images is low-cost and highly realistic, and has been widely used in various fields such as smart city construction, disaster emergency response, and the display and protection of cultural relics. This application addresses the problem that traditional orientation methods for disordered multi-view images are inefficient and difficult to succeed. It studies an orientation method for disordered images based on image relationships. This method reduces the number of matches by quickly acquiring approximate relationships between images, thereby improving matching efficiency. It then introduces robust image relationships into the orientation solution process, achieving orientation for disordered images based on image relationships.

[0110] (1) To address the low efficiency of exhaustive matching of unordered images, we studied the construction method of an image vocabulary tree. We comprehensively considered the quality of the constructed visual vocabulary and determined the approximate relationship between images in an image set based on an improved TF-IDF model. Experimental results show that this method can restore the approximate relationship between images to a certain extent and reduce the matching time.

[0111] (2) Based on an in-depth study of hash algorithms in the field of image retrieval and a summary of the computational process of the CasHash algorithm, a method for determining image relationships based on global hashing is proposed. By constructing a multi-table hash for feature points, the image code of the image is obtained. Then, drawing on the bucketing idea of ​​hierarchical hashing, multiple groups of LSH hash functions are used to quickly determine the approximate relationship between images, thereby constructing an undirected image relationship graph. Experimental results show that this method can construct image relationships better, faster, and more stable than the image vocabulary tree method.

[0112] (3) To address the problem that the initial image relationships are not robust enough, a method for eliminating weakly correlated image relationships based on three-view matching constraints is proposed. Based on the results obtained from fast matching, the image relationships are screened and updated to obtain robust image relationships. Experimental results show that this method can effectively eliminate weakly correlated image relationships and obtain a more robust undirected graph of image relationships.

[0113] (4) To address the time-consuming nature of incremental SFM orientation, we propose an orientation strategy based on image relationships, employing a global solution approach. Using three-view matching elements as constraints, we globally solve the image pose parameters and the three-dimensional coordinates of the matching points based on rapidly acquired robust image relationships, achieving rapid and robust orientation based on image relationships. Experimental results demonstrate that this orientation method can achieve rapid orientation of disordered images with accuracy that meets requirements.

[0114] 1. 3D Construction of Disordered Image Relationships

[0115] (1) Global hashing improved disordered image relationship construction

[0116] The CasHash algorithm only accelerates the matching of two images, that is, an image pair, and targets the matching between feature points. Although it has a significant advantage in speed and can effectively increase the rate of image matching, it does not reduce the number of image matches required. For matching between large data and disordered images, pairwise matching is still required. In the matching process, there are still matches between unrelated image pairs, which is time-consuming.

[0117] To solve the problem of redundant matching times between image pairs, this application establishes a global hash algorithm for image sets based on the CasHash algorithm to restore the relative positions between images, reduce the number of matching times between images, and improve efficiency.

[0118] The specific steps of the global hash-modified disordered image relationship construction method are as follows:

[0119] Step 1: Based on the LSH algorithm, all feature points extracted from all images in the image set are mapped into m (m = 8)-bit binary codes through L (L = 6) LSH hash functions. Then, a short-coded multi-table hash is constructed based on the binary codes of the feature points, so that similar feature points fall into the same hash bucket.

[0120] Step 2: For an image, the hash table formed by bucketing all the feature points on the image obtained in step 1 according to the hash algorithm is used as a global description of the image, indicating the different categories of SIFT feature points on the image and the number of each category. If two images are exactly the same, then the hash tables obtained must be exactly the same, that is, the hash tables obtained for approximate images must also be approximate. The image hash table obtained by bucketing the feature points through the short code hash transform is used as the image code describing the image;

[0121] like Figure 1As shown in the figure, the number of buckets in the hash table is the number of bits of the image code. Then, the number of feature points contained in each bucket in the hash table is calculated and used as the value of the image code bit corresponding to the hash bucket in the image code to obtain the image code of each image.

[0122] After constructing a multi-table hash with L different function mappings, each image is described by L image codes;

[0123] Step 3: Based on the feature point bucketing in the hierarchical hashing algorithm, the image code of the image is divided by using the short-coded LSH hash function. First, the feature code of the image is normalized, and then the normalized image code is encoded and bucketed using the LSH function. This allows similar images to be grouped into the same bucket, achieving differentiation between different images. Multiple LSH functions (L = 6) are used for bucketing to obtain various bucketing results.

[0124] Step 4: Images that fall into the same hash bucket are approximate images. For the L sets of image codes in this application, L functions are used to bucket them, resulting in L*L, or 36 sets of results. The number of times they fall into the same bucket is used as the weight of the approximate image, and a list of image relationships with weights w is obtained (0≤w≤36). The larger the weight w, the higher the similarity between the images.

[0125] (2) Determining the relationship between images

[0126] Based on the global hash algorithm, a weighted image relationship list is obtained to determine whether the images are similar. For the images to be matched with the target image, they are first sorted according to the size of the weight, and the weight critical value weight is set as the upper limit. The images with weights greater than the critical value weight are taken as the approximate images of the target image, while the images with weights less than the critical value weight have their approximate relationship with the target image deleted and are not used as image pairs to be matched, thereby reducing the number of image matching times and improving the speed of disordered image matching.

[0127] (3) Constructing an initial image relationship undirected graph

[0128] An undirected graph with no directionality in the edges is used to describe the relationship between images in an image set. In this graph, a vertex represents each image in the image set, and an edge is used to describe whether the images are related. Image pairs with a certain overlapping relationship constitute the set of edges in the undirected graph. An undirected graph of image relationships G = (v, E) is constructed, where V is the set of all vertices and E is the set of all edges. When matching, only related image pairs are matched, thereby limiting the scope of image matching.

[0129] To address the practical problem of low efficiency of exhaustive matching of disordered images, this application restores the relative positional relationship between images in the constructed image set, reduces the number of matches between images (eliminating matches between irrelevant images), and achieves fast matching of disordered images.

[0130] First, an image vocabulary tree is constructed based on the feature vectors extracted from the image. For the inverted file composed of image IDs generated by the vocabulary tree, the correspondence between image vocabulary and image is calculated according to the TF-IDF model that comprehensively considers the quality of visual vocabulary. The visual word frequency vector of each image is obtained, and the initial association relationship between images is determined. Then, an image relationship construction method based on global hashing is established, and a global hashing algorithm is implemented to determine the approximate relationship between images. The image code is constructed through the multi-table hash constructed by feature points, the approximate relationship between images is determined, and an initial unordered image relationship graph is constructed.

[0131] 2. Eliminating weakly correlated image relationships based on three-view matching

[0132] Since not all SIFT feature points extracted from an image are image matching points, and non-image matching points account for a large proportion of the extracted feature points, the image relationships calculated and analyzed based on the extracted feature points have certain errors. Based on the results obtained from the initial image relationship undirected graph G(V, E), the application eliminates weakly associated image relationships based on the constraints of three-view matching, obtains robust image relationships, and orients the image set.

[0133] According to the results of fast matching, based on the constraints of three-view matching, the weakly associated parts in the image relationship graph are eliminated, including the following steps:

[0134] Step 1: When obtaining the results of image matching, first calculate and judge the number of matching points obtained by matching between any image pairs, and delete the image relationships with fewer matching point pairs than a fixed critical value (50), that is, delete the corresponding edges in the image relationship undirected graph and delete the image pairs with smaller overlap;

[0135] Step 2: For disordered multi-view image sets, matching constraints based on three views can better ensure the robustness of 3D reconstruction. Using the three views as constraints, we can filter and eliminate unstable parts in the image relationships.

[0136] Among them, the three-view matching element of the image refers to: if the three edges connecting the three images i, j, and k<i,j> ,<i,k> ,<j,k> If all exist and there are enough three-view matching points, then images i, j, k constitute a three-view matching element, such as Figure 2 As shown;

[0137] In this process, we first search for three-view matching elements in the image relationship undirected graph G. When the three edges connecting any three images i, j, and k are<i,j> ,<i,k> ,<j,k> If all of them exist in the set E, and the number of three-view matching points on the image is greater than a fixed critical value, then it is determined that there is a correlation between the three images, which is recorded as t<i,j,k> At this point, all three-view matching elements in the image set are obtained, that is, a three-view set T is formed; then, a new undirected graph G'=(V', E') is constructed from the initial vertex set V and the edges contained in the three-view set T. The matching point pairs that do not meet the multi-view visibility are deleted, and then the image relationships that do not meet the three-view constraint are deleted;

[0138] Finally, the undirected graph obtained by continuously deleting image relationships is not necessarily connected, and there are some redundant vertices, namely images. A depth-first traversal search algorithm is used to obtain the maximum connected component of the undirected graph, so that all images in the image relationship have related images.

[0139] The specific process is: assume that all vertices in the undirected graph (G'=(V', E')) have not been visited at the beginning of the traversal. Then, starting from a vertex v, visit the unvisited vertices adjacent to it in turn, and traverse the entire undirected graph in a depth-first manner until all vertices connected to v have been visited. At this time, there will be some vertices in the graph that have not been visited. Select one of the unvisited vertices as the source point of the new traversal and repeat the above traversal process until all vertices have been visited.

[0140] Get several connected components in the image relationship undirected graph, and select the undirected graph with the largest number of images as the final image relationship graph G final =(V final , E final ), if any image in the three views is not in the set of vertices V final If the three views are in the same file, then the three views will be eliminated.

[0141] Based on the above steps, an undirected image relationship graph is obtained to describe the association of all images in the image set. It is composed of three-view matching elements of the image, has no redundancy, has high precision and strong robustness, and is directly used to complete the directional part of the disordered image.

[0142] 3. Orientation based on three-dimensional image relationships

[0143] The orientation process for multi-view 3D reconstruction consists of three steps: first, matching all potentially relevant image pairs to determine their relative positional relationships; second, calculating the absolute orientation of all images in a global unified coordinate system, including the pose at the time of image capture and the camera's center coordinates, and then calculating the 3D coordinates of all spatial points; third, using bundle adjustment, calculating the result that minimizes the reprojection error, thereby optimizing the previous calculation result.

[0144] The relationship recovery, fast matching and elimination of weakly correlated image relationships between multi-view images are completed in sequence, and the image relationship graph G = (V, E) between disordered images with stability is obtained. The absolute orientation of all images in the global coordinate system is calculated, that is, the absolute rotation of the image and the center coordinates of the camera are solved.

[0145] How to calculate the absolute orientation of all images in a unified global coordinate system, that is, to solve the absolute rotation of the image and the center coordinates of the camera, is a very important link in the orientation process. Although the incremental SFM algorithm has the advantages of low image requirements and better reconstruction effect, this incremental algorithm requires continuous iterative operations and has low computational efficiency. It takes a huge amount of time to reconstruct a large number of disordered images. Therefore, this application will use a global solution optimization method that does not rely on an iterative framework to complete the orientation of disordered images. This method has high computational efficiency and accuracy.

[0146] (1) Global solution based on three-view matching elements

[0147] The global solution is based on three-view matching, and the three-view matching elements of the image are obtained.<i,j,k> An image relationship graph G=(V, E) between the disordered images is constructed, and the solving process revolves around the image relationship graph.

[0148] 1. Estimate the global rotation matrix

[0149] For each image pair in the relationship graph G = (V, E)<i,j> , first use the relative orientation method to calculate the relative rotation matrix R between them i，j and the relative displacement direction vector t i，j ;

[0150] This application uses the five-point relative orientation method under the random sampling consistent architecture to estimate the relative rotation matrix R between each image pair i，j and the relative displacement direction vector t i，j , assuming that point x on image i and point x' on image j are a pair of points with the same name, then the following relationship exists between the two points:

[0151] xE i，j x′=0 Formula 1

[0152] Ei，j =[t i，j ]×R i，j Formula 2

[0153] Where E is the essential matrix, which is a 3×3 square matrix with a rank of 2 and a degree of freedom of 5. Only 5 points of the same name are needed to linearly solve the essential matrix. The essential matrix is ​​composed of the translation vector t i，j and the relative rotation matrix R i，j Composition, decompose the rotation and translation of the two images from the essential matrix;

[0154] The number of inliers in the image is calculated. If the number of inliers used for calculation is less than a fixed critical value, the relative pose of the image is considered to be inaccurate, that is, the image correlation of the image pair is poor, and it is removed.

[0155] Assume that the global rotation matrices of image i and image j in the global coordinate system are R i and R j , R i and R j are all orthogonal matrices, then they satisfy the following conditions:

[0156]

[0157] According to the above conditions, the equation is:

[0158]

[0159] Where r i m，m The global rotation matrix is ​​R j The value of the mth row and nth column (m=1, 2, 3), (n=1, 2, 3), is the relative rotation matrix R i，j The mth row of is the nth column of the global rotation matrix R;

[0160] Each image pair<i,j> All of them can be listed as 9 equations through formula 4, let R I As the unit matrix, the least squares method is used to solve the global rotation matrix of any image i to obtain the estimated value Among them, the orthogonality is constrained by the SVD decomposition algorithm, and the closest rotation matrix under the Frobenius norm is obtained by projection transformation. First, Perform SVD decomposition to obtain the product of three matrices, as shown in Equation 5:

[0161]

[0162] Among them, U i and V iare all 3×3 unitary matrices, ∑ i is a non-negative real diagonal matrix. Perform forced orthogonalization to obtain;

[0163] R i =U i V i H , i, j∈V Equation 6

[0164] Calculate the global rotation matrix R of any image in the image set in the global coordinate system.

[0165] 2. Local translation estimation based on three-view matching

[0166] For each three-view matching element in the undirected graph G, obtain the global rotation matrix R of the three images i , R j , R k , will be based on the three-view matching element<i,j,k> , the local translation vector of the image projection center in the local coordinate system Solve the problem and remove the three-view matching points with large errors according to the reprojection errors of the three-view matching points.

[0167] Assume that in the image relationship undirected graph, the three-view matching element<i,j,k> The set of spatial points in the object space corresponding to all three-view matching points on the three images is G P , where the spatial coordinates of a point P in the object space are X P , its image coordinates on image i are Then the reprojection error of the image point is calculated as:

[0168]

[0169] Where, is the mth row (m=1, 2, 3) of the global rotation matrix R; is the local translation vector The mth row of , solves the local translation vector that minimizes the reprojection error ρ and the coordinates X of the object space point P P The value of , that is:

[0170]

[0171] Convert to L ∞ Solve the linear programming problem under the norm:

[0172] minη

[0173]

[0174] t1=(0,0,0) T Formula 9

[0175] That is Under the condition of , the reprojection error of the three-view matching point is listed as an equation to minimize the value of the reprojection error and solve the local translation vector

[0176] The accuracy of the three-view matching points is judged based on whether the calculated reprojection error exceeds the critical value, and the three-view matching points that exceed the critical value are deleted. Then the three-view matching elements of the image are judged again. After deleting some three-view matching points, whether the number of three-view matching points between the image pairs still reaches the critical value, if it is less than the fixed critical value, the three-view matching element needs to be deleted, and after deletion, a depth-first search is performed on the new image relationship to ensure that the undirected graph has connectivity, delete isolated points in the vertex set, and finally update the unordered image relationship graph.

[0177] 3. Globally consistent position estimation

[0178] Get the relative rotation matrix R of each image pair in the edge set E of the undirected graph i，j , relative displacement direction vector t i，j , and the local translation vector and the global rotation matrix R i ,Next, based on the above results, the globally consistent position of the image projection center is estimated.

[0179] For any image pair in the relationship graph G = (V, E)<i,j> The position should satisfy the following equation:

[0180]

[0181] Among them, λ i，j is the scale factor, and the position T of the projection center of all images in the global coordinate system is calculated using the least squares algorithm according to Equation 10. i ;

[0182] However, the calculation result is not robust and is not the optimal solution in multi-view geometry. Due to the influence of noise, Equation 10 is not strictly valid for all image pairs and there will be a certain degree of deviation, such as Figure 3 Therefore, the least square method cannot be used directly to solve the problem.

[0183] This application first assumes that the scale coefficient λ of any two images i，j Are not less than 1, and use the minimum Euclidean distance algorithm to transform the equation into L ∞ Solve it using linear programming under the norm:

[0184]

[0185] The global optimal solution of the projection center is calculated using linear programming, and the constraints of three-view matching are used to replace the binocular stereo image pair for solution;

[0186] by To replace the relative position translation vector t in the stereo pair i，j , the local scale coefficient in the three-view matching element is positive To replace λ in the stereo pair i，j , the equation listed in formula 11 is rewritten as:

[0187]

[0188] The linear programming method is used to solve the position of the projection center of the image in the global coordinate system according to the above formula, and the optimal solution of the globally consistent position is obtained. The result obtained based on the three-view matching constraint has better robustness than the stereo image pair.

[0189] 4. Solve the coordinates of spatial points

[0190] Solve the spatial coordinates of all multi-view matching points, i.e., encrypted points, and restore the three-dimensional structure of the scene.

[0191] Assume that an object point X in space is visible on n images, then the following equation is listed for each visible image:

[0192] x i =P i X, i = 1, 2, 3...n Formula 13

[0193] Where x i and X represent the homogeneous coordinates of point X on the image and in space respectively. The specific formula is:

[0194] x i =w i (u i ,y i , 1) T , X=(X,Y,Z,1) T Formula 14

[0195] Where (u i , v i ) is the coordinate of the image point on image i, w i is the scale coefficient, the matrix P i represents the camera projection matrix corresponding to image i, where K is the camera calibration matrix;

[0196] P i =K i R i[I|-C i ] i=1, 2, 3...n Formula 15

[0197] Formula 14 can be expressed as:

[0198]

[0199] Among them, P i k Represents the matrix P i The kth row (k=1, 2, 3) of , the formula is converted to:

[0200]

[0201] List n equations (17), first use least squares to solve the initial value, then use iterative linear least squares trigonometric method to iteratively solve the equation, continuously improve the accuracy of the result, and obtain the optimal solution.

[0202] (2) Directed Architecture Based on 3D Image Relationships

[0203] A global solution algorithm based on three-view matching elements is used to complete the orientation of disordered images based on image relationships. The technical framework is:

[0204] Step 1: Estimate the relative orientation of each image pair using a five-point method based on a random sampling consensus architecture.<i,j> The relative rotation matrix R between i，j and the relative displacement direction vector t i，j , and calculate the number of inliers in the image. If the number of inliers used for calculation is less than a fixed critical value, it is determined that the relative pose of the image is not accurate enough and the image pair is discarded;

[0205] Step 2: According to the listed relative rotation matrix R i，j With the global rotation matrix R i and R j The relationship between , each image pair lists 9 equations, where R I The global rotation matrix of any image i is solved by the least squares method to obtain an estimate, and the orthogonality is satisfied by the SVD decomposition algorithm.

[0206] Step 3: For each three-view matching point in the three-view matching element, according to its object space coordinate X P The coordinates of the image plane list the equation of the point reprojection error, and solve the local translation vector that minimizes the reprojection error ρ and the coordinates X of the object space point P P The value of

[0207] Step 4: The three-view matching points are judged based on whether the reprojection error exceeds the critical value, and the three-view matching points that exceed the critical value are deleted. At the same time, it is judged whether the number of three-view matching points reaches the critical value for the three-view matching element. If it is less than the fixed critical value, the three-view matching element is deleted. After deletion, a depth-first search is performed to ensure that the undirected graph has connectivity, and isolated points in the vertex set are deleted. Finally, the unordered image relationship graph is updated;

[0208] Step 5: Leveraging the superior robustness of three-view matching between multi-view stereo pairs, a linear programming method is used to find the optimal solution for the global consistency position of the image projection center in the global coordinate system that minimizes the Euclidean distance.

[0209] Step 6: Use computational visual triangulation to solve the equations through iterative linear least squares to iteratively solve the spatial coordinates of all multi-view matching points, that is, the encrypted points, to restore the 3D structure of the scene;

[0210] Step 7: Finally, perform a global bundle adjustment to optimize the pose parameters and point positions to obtain the final result.

[0211] IV. Experiment and Analysis

[0212] (I) Experimental Overview

[0213] In order to verify the feasibility of the disordered image orientation method based on image relationship construction proposed in this application in solving practical problems, this application uses image data obtained by drones to conduct multiple experiments. The experimental process is as follows:

[0214] Process 1, Feature Extraction: This application first uses the SIFT feature extraction algorithm to perform a one-time SIFT feature extraction on all selected images, obtaining a SIFT feature description file for each image. This is the first step of this application's global algorithm and also the data foundation for the experimental content of this application, so it will not be described in detail.

[0215] Process 2, Image Relationship Construction: Two image relationship construction methods, based on image vocabulary trees and global hashing, were compared with the traditional exhaustive matching method for obtaining image relationships. The experimental results were analyzed in terms of algorithm time consumption and result accuracy. After obtaining the initial image relationships, the final matching results and image relationships were obtained by performing rapid matching between image pairs and eliminating weakly associated image relationships. Finally, the experimental results of constructing image relationships from multi-source data (aerial and close-range imagery) were analyzed.

[0216] Process 3, orientation based on image relationships: After obtaining the finalized image relationships, a global solution method based on three-view matching constraints is used to orient the images. The feasibility of this method is verified by experiments on actual data. On this basis, this method is compared and analyzed with the classic SFM algorithm and the commercial photogrammetry software PhotoScan in terms of time efficiency and orientation accuracy.

[0217] (2) Image relationship construction experiment

[0218] Obtaining approximate relationships between images in photogrammetry not only helps avoid the time-consuming problem of exhaustive matching methods, but also plays an important role in image stitching and the generation of subsequent products. Therefore, to address the practical problem of recovering and constructing image relationships, this paper describes a method based on image vocabulary tree construction and a method based on global hashing. Both methods are implemented and evaluated in terms of efficiency and accuracy.

[0219] This application selects 40 images and 100 images from the dataset as two experimental groups to conduct experiments. First, feature extraction is performed on each group of images according to the SIFT feature extraction method. Figure 4 ,The number of feature points extracted from each image is about 8000.

[0220] 1. Accuracy Analysis of Image Relationship Construction Methods

[0221] After extracting the SIFT feature points of all images in the image set, the approximate relationship between the images in the image set is constructed based on the image vocabulary tree method and the global hash algorithm, thereby reducing the number of image matching and improving the matching efficiency.

[0222] The constructed image relationship is represented by a matrix-like diagram, where both the horizontal and vertical axes represent the image IDs. If there is a certain correlation between image I and image J and a match is required, a small grid is used to represent the corresponding coordinate (I, J) in the diagram. If there is no correlation, the corresponding coordinate is empty, i.e., there is no small grid. Since the correlation between image I and image J is equivalent to the correlation between image J and image I, the image relationship diagram is only an upper triangular structure. When performing exhaustive matching on unordered images, the image relationship diagram is as follows: Figure 5 shown.

[0223] Since the image relationship obtained by screening according to the number of matching point pairs after exhaustive matching can be said to be the most accurate description of the image relationship, each group of experiments in this application takes the image association relationship obtained after exhaustive matching as the true value, and compares and judges the accuracy of the image relationship based on the image vocabulary tree method and the image relationship based on the global hash algorithm.

[0224] The experimental results obtained from the two groups of experiments in this application are as follows: Figure 6 As shown in the figure, the results of each set of experiments include the following four parts: Figure (a) shows the image relationship used for exhaustive matching, that is, each pair of images needs to be matched in sequence, such as Figure 5 The full upper triangle graph shown in FIG; FIG (b) shows the result obtained after exhaustive matching, that is, the true value of the image relationship required for the experiment of this application, so as to be used to compare and analyze the accuracy of the image relationship construction of this application; FIG (c) shows the experimental results obtained based on the image vocabulary tree method; FIG (d) is the experimental result obtained based on the global hash algorithm proposed in this application.

[0225] according to Figure 6 The experimental results shown show that both the image vocabulary tree-based method and the global hashing-based algorithm can restore the approximate relationship between images to a certain extent. Compared with the exhaustive matching method, they can significantly reduce the number of matches, thereby achieving the goal of improving image matching efficiency. To more accurately compare the accuracy of the two algorithms, this application uses quantitative values, namely recall and precision, to quantitatively analyze the accuracy of the image relationships constructed by the two methods.

[0226] When both precision and recall are required, F1 (or F-score) is used as a comprehensive measure. The image relationship lists obtained from the two sets of experiments are calculated to obtain the accuracy results of the two sets of experiments, such as Figure 7 As shown:

[0227] according to Figure 7 The results listed show that both methods can achieve good results when the number of images is small and the image relationships are relatively simple, with the global hashing algorithm producing slightly better results. When the number of images is large and the image relationships are relatively complex, the global hashing algorithm proposed in this application can obtain a more complete image relationship while maintaining a similar precision, and its F-score is also higher than that of the image vocabulary tree method. Therefore, the global hashing algorithm can achieve relatively better results in terms of accuracy.

[0228] 2. Efficiency Analysis of Image Relationship Construction Methods

[0229] Comparative analysis of the efficiency of different methods. By calculating the time taken for each experimental process and sorting it out, we can get Figure 8 .

[0230] First, the efficiency of constructing image relationships based on the global hashing algorithm is significantly higher than that based on the image vocabulary tree, with the time basically within 10. Second, when constructing image relationships based on the global hashing algorithm, the time used basically does not change significantly with the increase in the number of images, while the method based on the image vocabulary tree will significantly increase the time consumption of the algorithm as the number of images increases, making the algorithm less efficient.

[0231] Furthermore, in the process of constructing image relationships in this application, the construction speed can be accelerated by reducing the number of feature points. For example, when constructing image relationships, 500 feature points can be extracted from each image to complete the image relationship construction, which will greatly increase the speed of obtaining image relationships. However, in the algorithm of this application, since feature points are still used for orientation after the image relationship is constructed, the algorithm must ensure consistency. Therefore, in order to ensure orientation accuracy, the number of feature points is not reduced here.

[0232] 3. Elimination of weakly correlated image relationships

[0233] Before conducting the orientation experiment, we need to quickly match image pairs based on the constructed undirected graph of the initial relationship between the disordered images. We use the CasHash algorithm to achieve fast matching between image pairs. Then, based on the matching results, we remove weakly correlated image relationships.

[0234] This application uses 100 images as an example to conduct an experiment on eliminating weakly correlated image relationships. The image relationships before and after elimination are compared as follows: (a) is the image relationship before elimination, and (b) is the image relationship after the weakly correlated image relationship is eliminated.

[0235] Through the method of eliminating weakly correlated image relationships in this application, incorrectly associated image pairs can be effectively deleted, and the result obtained is close to the true value of the image relationship. Therefore, it can be concluded that the method for eliminating weakly correlated image relationships proposed in this application can effectively improve the robustness of the acquired image relationships.

[0236] 4. Experimental Analysis of Multi-source Data Image Relationship Construction

[0237] The experimental data of this application are aerial images acquired by drones. However, in order to prove that the method of constructing image relationships proposed in this application can effectively distinguish image data from different sources and still obtain good image relationships when there are a large number of images, 79 close-up images taken by mobile phones in area A and 136 aerial images taken by drones in area B are used as a global dataset, and then the algorithm based on global hashing proposed in this application is used to construct image relationships.

[0238] First, the image relationships of the two image sets are constructed respectively, and the results after the weak correlation image relationships are eliminated are as follows: Figure 9The experimental result (a) proves that the image relationship construction method proposed in this application is still feasible for close-up images.

[0239] At the same time, according to the results, we can find that the image relationships constructed from the image set taken by mobile phones have no obvious rules and are somewhat random, while the image relationships constructed from the image set obtained by drones have certain regularities. This is because drones have certain flight paths (heading, sideways) when taking images.

[0240] This application uses the two data as a global data set, constructs the image relationship of this global data set, and obtains the result after eliminating the weakly correlated image relationship as follows Figure 10 As shown:

[0241] according to Figure 10 In the image relationship result constructed by merging the two image sets, the image relationships of the two parts are independent of each other. Each part can intuitively see the relative relationship between the images and analyze the shooting pattern of the images. At the same time, the upper right corner of the figure is empty, and the intersection of the two parts is also unrelated. It can be concluded that there is no intersection between the image relationships obtained from the two parts. Therefore, it is concluded that the image relationship construction method described in this application can well distinguish different image data and still obtain good results when the data volume is large.

[0242] (3) Orientation Experiments Based on Image Relationships

[0243] 1. Analysis of global solution experimental results

[0244] How to efficiently obtain the exterior orientation elements of an image to restore the position and posture of the image is an important part of the three-dimensional reconstruction process. This application uses a global solution method based on three-view matching constraints to complete the orientation of the image. According to the acquired image relationship diagram, the relative orientation algorithm is used to obtain the relative rotation matrix and relative displacement direction vector of each image pair, and then the rotation average algorithm is used to calculate the global rotation matrix of each image, and the optimal translation vector in the local coordinate system is estimated based on the three-view matching constraint. Then, the global optimal solution of the camera center position is calculated by minimizing the Euclidean distance, and the three-dimensional coordinates of the point are calculated using triangulation. Finally, a global bundle adjustment is performed for optimization.

[0245] The results are obtained by orienting 100 images based on the constructed image relationships using the method of this application. The final root mean square error (RMSE) obtained from the orientation is 0.508093. Based on these experimental results, we can conclude that the global solution method based on the three-view matching constraint can use the constructed undirected graph of image relationships to achieve orientation of unordered images. The orientation results are good and can roughly represent the shape and outline of the ground objects with acceptable accuracy.

[0246] 2. Global Analysis of Orientation Algorithms Based on Image Relationships

[0247] By comparing the experimental results of the orientation algorithm constructed by the image relationship of the present application with the classic incremental SFM algorithm and the photogrammetry commercial software PhotoScan, and performing calculations, the results of the algorithm of the present application are compared and analyzed.

[0248] Comparing the experimental results, we can see that the results obtained by the proposed method are basically consistent with those obtained by the classic incremental SFM algorithm and the software PhotoScan in terms of image distribution and the quality of sparse point clouds, and can show the global structure of the ground objects. This application analyzes and calculates the time consumption of each part of each method and the accuracy of the orientation results, and obtains Figure 11 .

[0249] Here, for the incremental SFM algorithm, since exhaustive matching in the full sense is too time-consuming, in order to quickly obtain experimental results, enumeration matching assisted by ANN (approximate nearest neighbor) algorithm for feature point search is used instead of complete exhaustive matching for time comparison.

[0250] Based on the analysis of the results, it is concluded that: First, the image relationship-based orientation method of the present application is significantly more efficient than the other two methods in the image matching process, and the orientation speed is also greatly improved compared to the classic incremental SFM algorithm; secondly, since there will be a small amount of error in the image relationship construction process, the accuracy of the final result obtained is higher than that of the other two methods, and the overall accuracy is acceptable, which can achieve orientation of disordered images.

Claims

1. A method for rapid and precise orientation of 3D construction of massive digital disordered images, characterized by: A global solution algorithm based on three-view matching elements is used to complete the orientation of disordered images based on image relationships. The technical framework is: Step 1: Estimate the relative orientation of each image pair using a five-point method based on a random sampling consensus architecture.<i,j> The relative rotation matrix R between i，j and the relative displacement direction vector t i，j , and calculate the number of inliers in the image. If the number of inliers used for calculation is less than a fixed critical value, it is determined that the relative pose of the image is not accurate enough and the image pair is discarded; Step 2: According to the listed relative rotation matrix R i，j With the global rotation matrix R i and R j The relationship between , each image pair lists 9 equations, where R I The global rotation matrix of any image i is solved by the least squares method to obtain an estimate, and the orthogonality is satisfied by the SVD decomposition algorithm. Step 3: For each three-view matching point in the three-view matching element, according to its object space coordinate X P The coordinates of the image plane list the equation of the point reprojection error, and solve the local translation vector that minimizes the reprojection error ρ and the coordinates X of the object space point P P The value of Step 4: The three-view matching points are judged based on whether the reprojection error exceeds the critical value, and the three-view matching points that exceed the critical value are deleted. At the same time, it is judged whether the number of three-view matching points reaches the critical value for the three-view matching element. If it is less than the fixed critical value, the three-view matching element is deleted. After deletion, a depth-first search is performed to ensure that the undirected graph has connectivity, and isolated points in the vertex set are deleted. Finally, the unordered image relationship graph is updated; Step 5: Leveraging the superior robustness of three-view matching between multi-view stereo pairs, a linear programming method is used to find the optimal solution for the global consistency position of the image projection center in the global coordinate system that minimizes the Euclidean distance. Step 6: Use computational visual triangulation to solve the equations through iterative linear least squares to iteratively solve the spatial coordinates of all multi-view matching points, that is, the encrypted points, to restore the 3D structure of the scene; Step 7: Perform a global bundle adjustment to globally optimize the pose parameters and point positions to obtain the final result.

2. The method for rapid and precise orientation of massive digital disordered images in 3D construction according to claim 1, characterized in that: This application reduces the number of matches by quickly acquiring the approximate relationship between images, and then introduces robust image relationships into the orientation solution process to achieve disordered image orientation based on image relationships: 1) Establish a global hash-based image relationship determination method. By constructing a multi-table hash for feature points, the image code of the image is obtained. Then, based on the bucketing method of hierarchical hashing, multiple groups of LSH hash functions are used to quickly determine the approximate relationship between images and construct an undirected image relationship graph. 2) To address the problem that the initial image relationships are not robust enough, a method for eliminating weakly correlated image relationships based on three-view matching constraints is established. Based on the results obtained from fast matching, the image relationships are screened and updated to obtain robust image relationships. 3) To address the time-consuming problem of incremental SFM orientation, a global solution method is adopted to establish an orientation strategy based on image relations. With three-view matching elements as constraints, the image pose parameters and the three-dimensional coordinates of the matching points are globally solved according to the rapidly acquired robust image relations, thus achieving fast and robust orientation based on image relations.

3. The method for rapid and precise orientation of massive digital disordered images in 3D construction according to claim 1, characterized in that: The specific steps of the global hash-modified disordered image relationship construction method are as follows: Step 1: Based on the LSH algorithm, all feature points extracted from all images in the image set are mapped into m (m = 8)-bit binary codes through L (L = 6) LSH hash functions. Then, a short-coded multi-table hash is constructed based on the binary codes of the feature points, so that similar feature points fall into the same hash bucket. Step 2: For an image, the hash table formed by bucketing all the feature points on the image obtained in step 1 according to the hash algorithm is used as a global description of the image, indicating the different classifications of SIFT feature points on the image and the number of each category. If two images are exactly the same, then the hash tables obtained must also be completely consistent, that is, the hash lists obtained by approximate images must also be approximate. The image hash list obtained by bucketing the feature points through the short coding hash transformation is used as the image code describing the image; the number of buckets in the hash table is the number of bits of the image code, and then the number of feature points contained in each bucket in the hash list is calculated, and it is used as the value of the image code bit corresponding to the hash bucket in the image code to obtain the image code of each image; after constructing a multi-table hash with L different function mappings, each image obtains L image codes to describe it; Step 3: Based on the feature point bucketing in the hierarchical hashing algorithm, the image code of the image is divided by using the short-coded LSH hash function. First, the feature code of the image is normalized, and then the normalized image code is encoded and bucketed using the LSH function. This allows similar images to be grouped into the same bucket, achieving differentiation between different images. Multiple LSH functions (L = 6) are used for bucketing to obtain various bucketing results. Step 4: Images that fall into the same hash bucket are approximate images. For the L groups of image codes in this application, L functions are used to bucket them, and L*L, or 36 groups of results, are obtained. The number of times they fall into the same bucket is used as the weight of the approximate image, and an image relationship list with a weight w is obtained (0≤w≤36). The larger the weight w, the higher the similarity between the images.

4. The method for rapid and precise orientation of massive digital disordered images in 3D construction according to claim 1, characterized in that: Eliminate weakly correlated image relationships based on three-view matching: This application obtains the initial image relationship undirected graph G(V, E), and based on the results obtained by fast matching, eliminates weakly correlated image relationships based on the constraints of three-view matching, obtains robust image relationships, and orients the image set; According to the results of fast matching, based on the constraints of three-view matching, the weakly associated parts in the image relationship graph are eliminated, including the following steps: Step 1: When obtaining the results of image matching, first calculate and judge the number of matching points obtained by matching between any image pairs, and delete the image relationships with fewer matching point pairs than a fixed critical value (50), that is, delete the corresponding edges in the image relationship undirected graph and delete the image pairs with smaller overlap; Step 2: For disordered multi-view image sets, matching constraints based on three views can better ensure the robustness of 3D reconstruction. Using the three views as constraints, we can filter and eliminate unstable parts in the image relationships. Among them, the three-view matching element of the image refers to: if the three edges connecting the three images i, j, and k<i,j> ,<i,k> ,<j,k> exist, and there are enough three-view matching points, then images i, j, k constitute a three-view matching element; In this process, we first search for three-view matching elements in the image relationship undirected graph G. When the three edges connecting any three images i, j, and k are<i,j> ,<i,k> ,<j,k> If all of them exist in the set E, and the number of three-view matching points on the image is greater than a fixed critical value, then it is determined that there is a correlation between the three images, which is recorded as t<i,j,k> At this point, all three-view matching elements in the image set are obtained, that is, a three-view set T is formed; then, a new undirected graph G'=(V', E') is constructed from the initial vertex set V and the edges contained in the three-view set T. The matching point pairs that do not meet the multi-view visibility are deleted, and then the image relationships that do not meet the three-view constraint are deleted; Finally, based on the fact that the undirected graph obtained by continuously deleting image relationships is not necessarily connected and has some redundant vertices, i.e., images, a depth-first traversal search algorithm is used to obtain the maximum connected component of the undirected graph, so that all images in the image relationship have related images. The specific process is: assume that all vertices in the undirected graph (G'=(V', E')) have not been visited at the beginning of the traversal. Then, starting from a vertex v, visit the unvisited vertices adjacent to it in turn, and traverse the entire undirected graph in a depth-first manner until all vertices connected to v have been visited. At this time, there will be some vertices in the graph that have not been visited. Select one of the unvisited vertices as the source point of the new traversal and repeat the above traversal process until all vertices have been visited. Get several connected components in the image relationship undirected graph, and select the undirected graph with the largest number of images as the final image relationship graph G final =(V final , E final ), if any image in the three views is not in the set of vertices V final If the three views are in the same file, then the three views will be eliminated.

5. The method for rapid and precise orientation of massive digital disordered images in 3D construction according to claim 1, characterized in that: The orientation process for multi-view 3D reconstruction consists of three steps: first, matching all potentially relevant image pairs to determine their relative positional relationships; second, calculating the absolute orientation of all images in a global unified coordinate system, including the pose at the time of image capture and the camera's center coordinates, and then calculating the 3D coordinates of all spatial points; third, using bundle adjustment, calculating the result that minimizes the reprojection error, thereby optimizing the previous calculation result. The system sequentially completes the relationship recovery, fast matching, and elimination of weakly correlated image relationships between multi-view images, and obtains a stable image relationship graph G = (V, E) between disordered images. The absolute orientation of all images in the global coordinate system is calculated, that is, the absolute rotation of the image and the center coordinates of the camera are solved. A global solution optimization method that does not rely on an iterative framework is used to achieve the orientation of disordered images.

6. The method for rapid and precise orientation of massive digital disordered images in 3D construction according to claim 1, characterized in that: Global solution based on three-view matching element: The global solution is based on three-view matching, and the three-view matching element of the image is obtained.<i,j,k> An image relationship graph G = (V, E) between the disordered images is constructed. The solution process revolves around the image relationship graph, including: estimating the global rotation matrix, local translation estimation based on three-view matching, globally consistent position estimation, and solving the spatial point coordinates.

7. The method for rapid and precise orientation of massive digital disordered images in 3D construction according to claim 1, characterized in that: Estimating the global rotation matrix: For each image pair in the relationship graph G = (V, E)<i,j> , first use the relative orientation method to calculate the relative rotation matrix R between them i，j and the relative displacement direction vector t i，j ; This application uses the five-point relative orientation method under the random sampling consistent architecture to estimate the relative rotation matrix R between each image pair i，j and the relative displacement direction vector t i，j , assuming that point x on image i and point x' on image j are a pair of points with the same name, then the following relationship exists between the two points: xE i，j x' = 0 Equation 1 E i，j =[t i，j ]×R i，j Formula 2 Where E is the essential matrix, which is a 3×3 square matrix with a rank of 2 and a degree of freedom of 5. Only 5 points of the same name are needed to linearly solve the essential matrix. The essential matrix is ​​composed of the translation vector t i，j and the relative rotation matrix R i，j Composition, decompose the rotation and translation of the two images from the essential matrix; The number of inliers in the image is calculated. If the number of inliers used for calculation is less than a fixed critical value, the relative pose of the image is considered to be inaccurate, that is, the image correlation of the image pair is poor, and it is removed. Assume that the global rotation matrices of image i and image j in the global coordinate system are R i and R j , R i and R j are all orthogonal matrices, then they satisfy the following conditions: According to the above conditions, the equation is: Where, The global rotation matrix is ​​R j The value of the mth row and nth column (m=1, 2, 3), (n=1, 2, 3), is the relative rotation matrix R i，j The mth row of is the nth column of the global rotation matrix R; Each image pair<i,j> All of them can be listed as 9 equations through formula 4, let R I As the unit matrix, the least squares method is used to solve the global rotation matrix of any image i to obtain the estimated value Among them, the orthogonality is constrained by the SVD decomposition algorithm, and the closest rotation matrix under the Frobenius norm is obtained by projection transformation. First, Perform SVD decomposition to obtain the product of three matrices, as shown in Equation 5: Among them, U i and V i are all 3×3 unitary matrices, ∑ i is a non-negative real diagonal matrix. Perform forced orthogonalization to obtain; Calculate the global rotation matrix R of any image in the image set in the global coordinate system.

8. The method for rapid and precise orientation of massive digital disordered images in three dimensions according to claim 1, characterized in that: Local translation estimation based on three-view matching: For each three-view matching element in the undirected graph G, obtain the global rotation matrix R of the three images i , R j , R k , will be based on the three-view matching element<i,j,k> , the local translation vector of the image projection center in the local coordinate system Solve the problem and remove the three-view matching points with large errors according to the reprojection errors of the three-view matching points. Assume that in the image relationship undirected graph, the three-view matching element<i,j,k> The set of spatial points in the object space corresponding to all three-view matching points on the three images is G P , where the spatial coordinates of a point P in the object space are X P , its image coordinates on image i are Then the reprojection error of the image point is calculated as: Where, is the mth row (m=1, 2, 3) of the global rotation matrix R; is the local translation vector The mth row of , solves the local translation vector that minimizes the reprojection error ρ and the coordinates X of the object space point P P The value of , that is: Convert to L ∞ Solve the linear programming problem under the norm: That is Under the condition of , the reprojection error of the three-view matching point is listed as an equation to minimize the value of the reprojection error and solve the local translation vector The accuracy of the three-view matching points is judged based on whether the calculated reprojection error exceeds the critical value, and the three-view matching points that exceed the critical value are deleted. Then the three-view matching elements of the image are judged again. After deleting some three-view matching points, whether the number of three-view matching points between the image pairs still reaches the critical value, if it is less than the fixed critical value, the three-view matching element needs to be deleted, and after deletion, a depth-first search is performed on the new image relationship to ensure that the undirected graph has connectivity, delete isolated points in the vertex set, and finally update the unordered image relationship graph.

9. The method for rapid and precise orientation of massive digital disordered images in three dimensions according to claim 1, characterized in that: Globally consistent position estimation: Get the relative rotation matrix R of each image pair in the edge set E of the undirected graph i，j , relative displacement direction vector t i，j , and the local translation vector and the global rotation matrix R i ,Next, based on the above results, the global consistency position of the image projection center is estimated; For any image pair in the relationship graph G = (V, E)<i,j> The position should satisfy the following equation: Among them, λ i，j is the scale factor, and the position T of the projection center of all images in the global coordinate system is calculated using the least squares algorithm according to Equation 10. i ; First, assume that the scale factor λ of any two images i，j Are not less than 1, and use the minimum Euclidean distance algorithm to transform the equation into L ∞ Solve it using linear programming under the norm: The global optimal solution of the projection center is calculated using linear programming, and the constraints of three-view matching are used to replace the binocular stereo image pair for solution; by To replace the relative position translation vector t in the stereo pair i，j , the local scale coefficient in the three-view matching element is positive To replace λ in the stereo pair i，j , the equation listed in formula 11 is rewritten as: The linear programming method is used to solve the position of the projection center of the image in the global coordinate system according to the above formula, and the optimal solution of the globally consistent position is obtained. The result obtained based on the three-view matching constraint has better robustness than the stereo image pair.

10. The method for rapid and precise orientation of 3D construction of massive digital disordered images according to claim 1, characterized in that: Solve the spatial coordinates of all multi-view matching points, i.e., encrypted points, and restore the three-dimensional structure of the scene: Assume that an object point X in space is visible on n images, then the following equation is listed for each visible image: x i =P i X, i = 1, 2, 3...n Formula 13 Where x i and X represent the homogeneous coordinates of point X on the image and in space respectively. The specific formula is: x i = w i (u i , v i , 1) T 、X = (X, Y, Z, 1) T Equation 14 Where (u i , v i ) is the coordinate of the image point on image i, w i is the scale coefficient, the matrix P i represents the camera projection matrix corresponding to image i, where K is the camera calibration matrix; P i = K i R i [I|-C i where i = 1, 2, 3…n Equation 15 Formula 14 can be expressed as: in, Represents the matrix P i The kth row (k=1, 2, 3) of , the formula is converted to: List n equations (17), first use least squares to solve the initial value, then use iterative linear least squares trigonometric method to iteratively solve the equation, continuously improve the accuracy of the result, and obtain the optimal solution.

Images