3D Vector and Multi-View Image Registration Method and Device Based on Point Set Pyramid

Abstract

The present invention provides a three-dimensional vector and multi-view image registration method and device based on a point set pyramid, including preprocessing the multi-view image and the three-dimensional vector to complete rough registration; using edge template matching based on the point set pyramid to achieve the matching of heterogeneous feature points; through false match elimination, multi-view forward intersection, precise alignment of coordinate systems, and vector back-projection and occlusion removal, precise registration between the three-dimensional vector and the multi-view image is completed. The present invention converts the three-dimensional heterogeneous point matching problem into a two-dimensional heterogeneous image matching problem, and combines multi-view imaging geometry to accurately locate a large number of feature points, and completes the precise alignment of the coordinate system, thereby realizing the precise registration between the three-dimensional vector and the multi-view image. In addition, the present invention is also compatible with the registration of two-dimensional vectors and images.

Description

Technical Field

[0001] The present invention belongs to the field of photogrammetry and remote sensing, and relates to a technical solution for realizing high-precision registration of three-dimensional vectors and multi-view images by using multi-view imaging geometry and heterologous image matching. Background Art

[0002] In large-scale manufacturing, three-dimensional vector models such as CAD (Computer Aided Design) or BIM (Building Information Modeling) are widely used to guide processing and production. Subsequently, measurement technologies such as three-dimensional laser scanning or vision-based three-dimensional reconstruction are used for model reconstruction, and then after registration, it is used for deviation analysis. Among them, the vision-based measurement method can provide richer texture information, can accurately photograph small objects, and has the advantages of low cost and high efficiency, so it is also one of the most commonly used measurement methods. In the fields of urban planning and geological exploration, etc., GIS (Geographic Information System) vectors and remote sensing images are usually registered to achieve data fusion, so as to provide strong data support for decision-making.

[0003] The core challenge of deviation analysis and data fusion lies in how to register two different types of data with high precision. Traditional registration methods often rely on manually selecting a small number of points for coordinate system conversion. This method not only easily introduces accidental errors and systematic errors and is difficult to meet the high-precision requirements, but also the workload will increase significantly when dealing with massive data. The difficulty of such problems lies in the data type difference between the three-dimensional vector model and the point cloud model, which belongs to the problem of three-dimensional heterologous point matching. However, there is still a great difficulty in accurately matching three-dimensional heterologous points. One is that it is difficult to design a good descriptor to accurately describe and match these points; the other is that the positioning accuracy of three-dimensional feature points (usually corner points and edge points) in the LiDAR scan point cloud and the dense point cloud of three-dimensional reconstruction is limited, and even if the corner point is matched, additional errors will be introduced in the registration.

[0004] The existing patent document CN104732577B provides a method for extracting building textures based on a UAV low-altitude aerial survey system, which involves research and improvement in registration. However, its disadvantage is that it is necessary to match the straight lines of the building contour (vector) with the straight lines in the image rather than corner points, and the registration of the building contour line and the image requires manual intervention, with low automation. Summary of the Invention

[0005] Aiming at the problem of low registration accuracy of traditional three-dimensional vectors and multi-view images, the present invention proposes a technical solution for registering three-dimensional vectors and multi-view images based on a point set pyramid.

[0006] The technical solution proposed by the present invention for the above technical problems is a three-dimensional vector and multi-view image registration method based on a point set pyramid, which performs the following processes:

[0007] Preprocess the multi-view image and the three-dimensional vector to complete rough registration;

[0008] Use edge template matching based on a point set pyramid to achieve the matching of heterogeneous feature points. The implementation method includes: in the vector back-projection image, take the vector intersection point as the vector feature point, and take the corresponding pixel as the center to extract a preset-size area as the template; in the edge feature image, take the pixel at the same position as the vector feature point as the center to extract a preset-size area as the search area; respectively extract the coordinates of the valid pixels from the template and the search area to form their respective point sets, where the valid pixels represent the pixels corresponding to the vector lines and edge features; record the extracted point sets as the point sets at the main scale, and select the scaling factor between adjacent scales to construct a point set pyramid; then perform global main scale rough search and local multi-scale fine matching, and fit a parabolic equation based on the coordinates and the corresponding average nearest neighbor distance in the neighborhood around the matching points under the obtained optimal scale combination to obtain the best matching points of the vector feature points, and complete the heterogeneous matching of the vector feature points in the vector back-projection image and the pixels at the same position in the edge feature image;

[0009] After removing false matches, multi-view forward intersection, precise alignment of coordinate systems, and vector back-projection and occlusion, complete the precise registration between the three-dimensional vector and the multi-view image.

[0010] Moreover, the implementation method includes the following steps:

[0011] Step 1, preprocess the multi-view image, including obtaining the accurate internal and external orientation elements and distortion coefficients of each image with different viewpoints, and then perform distortion correction and edge extraction on each image with different viewpoints;

[0012] Step 2, preprocess the three-dimensional vector, including converting the three-dimensional vector to the photogrammetric result coordinate system through rough alignment of the coordinate system, and then performing back-projection and occlusion processing to obtain the rough registration result between the three-dimensional vector and the multi-view image;

[0013] Step 3, perform heterogeneous matching of the vector feature points in the vector back-projection image and the pixels at the same position in the edge feature image based on a point set pyramid;

[0014] Step 4, perform RANSAC false match removal based on binocular forward intersection on the matching point set of each vector feature point, and then perform multi-view forward intersection on the vector feature points using the inlier set obtained by false match removal to obtain the optimal object space coordinates of the vector feature points;

[0015] Step 5: Perform fine alignment of the coordinate system using the vector feature points and the corresponding optimal object points.

[0016] Step 6: Back-project the three-dimensional vectors after fine alignment of the coordinate system onto the images of the corresponding viewpoints and perform occlusion processing to complete the precise registration of the three-dimensional vectors and the multi-view images.

[0017] Moreover, in Step 1, the following implementation method is used for preprocessing the multi-view images:

[0018] Obtain the internal and external parameters of the multi-view images, including performing aerial triangulation on the acquired multi-view images to solve the precise internal and external orientation elements and lens distortion coefficients of each image;

[0019] Perform distortion correction on each image according to the lens distortion coefficients and the distortion model;

[0020] Perform edge extraction on the images after distortion correction using the Canny operator to obtain the edge feature images.

[0021] Moreover, in Step 2, the following implementation method is used for preprocessing the three-dimensional vectors:

[0022] Select control points to solve the seven-parameter transformation coefficients and perform rough alignment of the coordinate system to transform the three-dimensional vectors into the photogrammetric result coordinate system;

[0023] Back-project the three-dimensional vectors after rough alignment of the coordinate system according to the internal and external parameters of the multi-view images to obtain the three-dimensional vector images of the multi-view images in this view, and perform occlusion removal to remove the occluded lines in the vector model.

[0024] Moreover, in Step 3, the following implementation method is used for heterologous matching of the vector back-projection map and the edge feature images based on the point set pyramid:

[0025] Step 3.1: Extract the template and the search area, including in the vector back-projection map, taking the vector intersection points as vector feature points, and taking the corresponding pixels as the center to extract a preset-size area as the template; in the edge feature image, taking the pixels at the same position as the vector feature points as the center to extract a preset-size area as the search area;

[0026] Step 3.2: Extract the point sets, including respectively extracting the coordinates of the valid pixels from the template and the search area to form their respective point sets, where the valid pixels represent the pixels corresponding to the vector lines and the edge features;

[0027] Step 3.3: Construct the point set pyramid, including recording the point sets extracted in Step 3.2 as the point sets at the main scale, and selecting the scaling factor between adjacent scales to construct the point set pyramid;

[0028] Step 3.4, global coarse search for the main scale, including performing a striding search on the template point set and the search area point set under the main scale, and combining clustering analysis to judge the reliability of the matching. Take the reliable clustering center to construct a buffer as the optimal search area;

[0029] Step 3.5, local multi-scale fine matching, including performing pixel-by-pixel fine matching between multiple scales on the optimal search area and the point set pyramid corresponding to the template. Then, divide the minimum point set distance obtained for each scale combination by the scaling ratio of the optimal search area point set relative to the main scale to obtain the normalized minimum distance in each scale combination. The template scale and the optimal search area scale corresponding to the minimum value in the normalized minimum distances are the optimal scale combination;

[0030] Step 3.6, parabolic surface fitting for precise positioning, including taking the coordinates and the corresponding average nearest neighbor distances in the neighborhood around the matching points obtained in Step 3.5 under the optimal scale combination to fit a parabolic surface equation, and obtaining the coordinates corresponding to the lowest point of the parabolic surface as the best matching point to be solved. Then, convert the coordinates of this point to the image coordinate system of the main scale to obtain the best matching point of the vector feature point.

[0031] Moreover, the global coarse search for the main scale in Step 3.4 is implemented in the following way:

[0032] Perform a striding search, and use the average nearest neighbor distance as the similarity measurement method during the search;

[0033] Perform clustering analysis on the 4 points with the minimum average nearest neighbor distance to judge the reliability of the matching, including according to the set clustering distance threshold. If 3 or more of these 4 points are clustered into one category, it is considered that the matching is reliable, and take the clustering center to construct a buffer as the optimal search area.

[0034] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the three-dimensional vector and multi-view image registration method based on the point set pyramid as described above.

[0035] On the other hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the three-dimensional vector and multi-view image registration method based on the point set pyramid as described above.

[0036] On the other hand, the present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the three-dimensional vector and multi-view image registration method based on the point set pyramid as described above.

[0037] The present invention proposes a three-dimensional vector and multi-view image registration method based on a point set pyramid. This method converts the three-dimensional heterogeneous point matching problem into a two-dimensional heterogeneous image matching problem, and combines multi-view imaging geometry to accurately locate a large number of feature points to complete the precise alignment of coordinate systems, thereby realizing the precise registration between the three-dimensional vector and the multi-view image. Compared with the prior art, the advantage of the present invention is that it uses the internal and external parameters of the multi-view image for vector back-projection to obtain the vector back-projection map corresponding to the multi-view image, and uses the pyramid strategy for template matching of the vector intersection points and the image edge detection results, with a high degree of automation and no need for manual intervention. Specifically, the present invention has the following advantages:

[0038] It has a wide range of applications. In addition to supporting the precise registration between the three-dimensional vector and the multi-view image, it is also compatible with two-dimensional vectors and large-sized objects. It has a high degree of automation, and a large number of homologous points are automatically extracted through heterogeneous matching, eliminating the need for a large number of manual point selections. It has a high registration accuracy. The coordinate system is precisely aligned through a large number of points, avoiding accidental errors and systematic errors that exist when only a small number of points are used. Brief Description of the Drawings

[0039] Figure 1 It is the flowchart of the three-dimensional vector and multi-view image registration in the embodiment of the present invention.

[0040] Figure 2 It is the schematic diagram of the edge template matching based on the point set pyramid in the embodiment of the present invention.

[0041] Figure 3 is the left and right rolling curtain diagram of the precise registration effect in the embodiment of the present invention.

[0042] Figure 4 is the up and down rolling curtain diagram of the precise registration effect in the embodiment of the present invention.

[0043] Figure 5 is the schematic diagram of the physical structure of the electronic device provided in the embodiment of the present invention. Detailed Embodiment

[0044] The technical solution of the present invention will be specifically described below in conjunction with the drawings and embodiments.

[0045] The embodiment of the present invention provides a three-dimensional vector and multi-view image registration method based on a point set pyramid, and the core includes three processes: First, preprocess the multi-view image and the three-dimensional vector to complete rough registration; then, use the edge template matching based on the point set pyramid to realize the matching of heterogeneous feature points; finally, through false match elimination, multi-view forward intersection, precise alignment of the coordinate system, and vector back-projection and occlusion removal, the precise registration between the three-dimensional vector and the multi-view image is completed.

[0046] As Figure 1 shown, the process of the embodiment of the present invention includes the following steps:

[0047] Step 1: Preprocess the multi-view images to obtain the accurate interior and exterior orientation elements and distortion coefficients for each image, and then perform distortion correction and edge extraction on each image.

[0048] The implementation method further preferably adopted in the embodiment includes the following sub-steps:

[0049] Step 1.1: Perform aerial triangulation on the obtained multi-view images to solve the accurate interior and exterior orientation elements and lens distortion coefficients corresponding to each image with different viewing angles. . Among them, are the parameters describing the radial distortion model, are the parameters describing the tangential distortion model.

[0050] Step 1.2: Perform distortion correction on each image according to the lens distortion coefficients and the following distortion model.

[0051]

[0052] In the formula, represents the ideal image point coordinates with the principal point of the image as the origin of the coordinate system; represents the image point coordinates with distortion; represents the radial length, .

[0053] Step 1.3: Use the Canny operator to perform edge extraction on each distortion-corrected image to obtain the corresponding edge feature image.

[0054] Step 2: Preprocess the 3D vector, including transforming the 3D vector to the photogrammetric result coordinate system through rough alignment of the coordinate system, and then performing back-projection and occlusion processing to obtain the rough registration result of the 3D vector and the multi-view images.

[0055] The implementation method further preferably adopted in the embodiment includes the following sub-steps:

[0056] Step 2.1: Select a small number of control points to solve the seven-parameter transformation coefficients and perform rough alignment of the coordinate system to transform the 3D vector to the photogrammetric result coordinate system. Specifically, it is recommended to select 5-9 control points in this step. The seven-parameter transformation includes three translation parameters, three rotation parameters, and one scaling parameter, and its model is shown as follows:

[0057]

[0058] In the formula, represents the coordinates of the 3D vector in the world coordinate system; represents the coordinates of the 3D vector in its own model coordinate system (hereinafter referred to as the 3D vector coordinate system); is the rotation matrix from the three-dimensional vector to the world coordinate system, are respectively the angles of rotation of the three-dimensional vector coordinate system around the axes of the world coordinate system, and correspond to three rotation parameters; are respectively the rotation matrices calculated by ; represents the offset between the origin of the three-dimensional vector coordinate system and the origin of the world coordinate system, which consists of three translation parameters; corresponds to the scaling parameter.

[0059] Step 2.2: Back-project the three-dimensional vector after the rough alignment of the coordinate system according to the internal and external parameters of the multi-view images obtained in Step 1.1, and preferably use the Z-Buffer algorithm to perform hidden line removal to remove the occluded lines in the vector model, so as to obtain the corresponding results for each image with different viewing angles, that is, a series of vector back-projection images.

[0060] The implementation method of the Z-Buffer algorithm is a prior art and will not be elaborated in the present invention. The back-projection is realized by the collinearity condition equation, which is specifically as follows:

[0061]

[0062] In the formula, represents the ideal image point coordinates with the principal point of the image as the origin of the coordinate system; the principal point coordinates and the focal length are the internal orientation elements of the image; is the coordinate of the object point in the world coordinate system; is the coordinate of the camera station in the world coordinate system, and is also the 3 external orientation line elements corresponding to this image; are the 3 external orientation angular elements of the image consisting of 9 direction cosines.

[0063] Step 3: As Figure 2 shown, use edge template matching based on the point set pyramid to match the heterogeneous feature points in the vector back-projection image and the edge feature image.

[0064] The implementation method further preferably adopted in the embodiment includes the following sub-steps:

[0065] Step 3.1: Extract the template and the search area. In the vector back-projection image, take the vector intersection point as the feature point, and take the corresponding pixel as the center to extract a region of size as the template, and then the template binary image can be obtained; in the edge feature image, take the pixel corresponding to the feature point as the center to extract a region of size as the search area, and then the search area edge binary image can be obtained. It is preferably recommended to set = 1.5 × , = 1.5 × , , The empirical threshold of is 300

[0066] Step 3.2, extract the point sets. Respectively extract the coordinates of the valid pixels from the template and the search area to form their respective point sets, where the valid pixels represent the pixels corresponding to the vector lines and edge features.

[0067] Step 3.3, construct the point set pyramid. Denote the point set extracted in Step 3.2 as the point set at the main scale and set as the scaling factor between adjacent scales to construct the point set pyramid. The scale of the k -th layer of the point set pyramid is , and the point set coordinates therein are proportional to the point set coordinates corresponding to the main scale , and the ratio is . The point set coordinates are saved using floating-point numbers, avoiding the errors of quantization and resampling.

[0068] Step 3.4, global coarse search at the main scale.

[0069] Perform a strided search on the template point set and the search area point set at the main scale , and combine cluster analysis to judge the reliability of the match. When performing the strided search, use the average nearest neighbor distance as the measurement method for the similarity between the two, and the specific formula is as follows:

[0070]

[0071] In the formula, the average nearest neighbor distance represents the average of the distances between each point in the two point sets , and the nearest point in the other point set, and can measure the average similarity between the two point sets; the forward distance represents the average distance from the points in the point set to the nearest point in the point set , and the backward distance represents the average distance from the points in the point set to the nearest point in the point set ; represents the minimum Euclidean distance from the point in the point set to the point set , represents the point in the point set The minimum Euclidean distance to the point set In Equation (6), m , n are the numbers of points in point sets A and B, respectively.

[0072] Subsequently, clustering analysis is performed on the 4 points with the minimum average nearest neighbor distance to judge the reliability of the match. According to the set clustering distance threshold, if 3 or more of these 4 points are clustered into one category, the match is considered reliable, and the buffer zone is constructed with the clustering center as the optimal search area; in other cases, it is considered that there is no area with a high similarity to the template point set in the search area, and the matching of this point is skipped.

[0073] Step 3.5, local multi-scale fine matching. Perform pixel-by-pixel fine matching between the optimal search area and the point set pyramid corresponding to the template at multiple scales, and then divide the minimum point set distance obtained for each scale combination by the scaling ratio of the optimal search area point set relative to the main scale to obtain the normalized minimum distance in each scale combination. The template scale and the optimal search area scale corresponding to the minimum value in the normalized minimum distance are used as the optimal scale combination, and the search area coordinates corresponding to this minimum value are the matching points of the feature point at this scale combination.

[0074] Step 3.6, paraboloid fitting for precise positioning. Take the 9 coordinates in the 3×3 neighborhood around the matching point obtained in Step 3.5 and the corresponding average nearest neighbor distance to fit the following paraboloid equation to obtain the coordinates of the lowest point of the paraboloid , which is also the best matching point to be solved; then convert the point coordinates from the optimal search area scale to the main scale in the image coordinate system to obtain the best matching point of the vector feature point.

[0075]

[0076] In the formula, a , b , c are the coefficients of the paraboloid equation to be fitted; ( x , y ) are the image point coordinates; z is the average nearest neighbor distance value;

[0077] Step 4, perform RANSAC outlier rejection based on binocular forward intersection on the set of matching points of each vector feature point, and then perform multi-view forward intersection on the inlier set obtained by outlier rejection of the vector feature point to obtain the optimal object space coordinates of the vector feature point;

[0078] The specific implementation method for processing each vector feature point preferably adopted in the embodiment further includes the following sub-steps:

[0079] Step 4.1, randomly select two matching points to perform binocular forward intersection to solve the object space coordinates of the intersection point. Initialize an inlier set and add these two matching points to the inlier set.

[0080] Step 4.2, perform back-projection using the obtained object space coordinates and calculate the reprojection error on the images corresponding to the remaining matching points. The calculation method of the reprojection error is prior art and will not be elaborated in this invention.

[0081] Step 4.3, include the matching points with reprojection error less than a preset threshold (preferably 0.8 pixels) into the inlier set.

[0082] Step 4.4, repeat Steps 4.1 to 4.3. Each iteration generates a corresponding inlier set until after reaching the maximum number of iterations, select the inlier set with the largest number of inliers as the effective matching set for multi-view forward intersection. The maximum number of iterations is preferably 200 and can be appropriately increased when the matching quality is poor.

[0083] Step 4.5, perform multi-view forward intersection on the inlier set obtained after removing false matches to obtain the optimal object space coordinates of the feature point.

[0084] The specific implementation of multi-view forward intersection is prior art and will not be elaborated in this invention.

[0085] Step 5, use the obtained large number of vector feature points and the corresponding optimal object space points to perform precise alignment of the coordinate systems.

[0086] In this step, substitute the vector feature points and the corresponding optimal object space points obtained after being processed by Step 4 into the seven-parameter transformation model described in Equation (2) to find the least squares solution, and further obtain the precise coordinate transformation parameters between the model coordinate system and the photogrammetric result coordinate system, and convert the three-dimensional vectors into the photogrammetric result coordinate system to complete the precise alignment.

[0087] In specific implementation, hundreds to thousands of vector feature points may be generated. In this step, it is preferably recommended to select no less than 100 vector points for precise alignment. The principle of coordinate system precise alignment in Step 5 is the same as that in Step 2.1, both of which are to solve the seven-parameter transformation coefficients. However, in Step 5, a large number of points are used for solving, which to a certain extent eliminates the systematic error existing in solving with only a small number of points in Step 2.1.

[0088] Step 6, back-project the three-dimensional vectors after coordinate system precise alignment onto the images of the corresponding viewpoints, and also preferably use the Z-Buffer algorithm for hidden surface removal to complete the precise registration of the three-dimensional vectors and multi-view images.

[0089] The principles of back-projection and hidden surface removal in Step 6 are the same as those in Step 2.2, except that the three-dimensional vectors participating in the back-projection have been precisely aligned.

[0090] The precise registration of 3D vectors and multi-view images can be achieved by completing the above steps.

[0091] For the convenience of understanding the technical effects of the present invention, a vector back-projection image is matched with the corresponding UAV edge feature image. The comparison of the matching effects of the present invention and the traditional template matching method is shown in Table 1:

[0092] Table 1 Matching results of vector back-projection image and 30 multi-view images

[0093]

[0094] As can be seen from Table 1, the method of the present invention has achieved a higher matching accuracy rate, which is significantly better than the traditional template matching method.

[0095] Through testing, it can be confirmed that the used vector feature points have all found accurate matching points on the edge feature image.

[0096] To prove the technical effects of the present invention, experiments were conducted using UAV close-range photography images and existing target CAD vector models. After the precise registration of the images and vectors based on the present invention, the effect of the superposition display of the vector back-projection image and the multi-view image is as Figure 3 、 Figure 4 shown. It can be seen that there is almost no misalignment at the seam between the image and the vector back-projection image, and the registration effect is good.

[0097] In specific implementation, the above process can be automatically run by computer software technology.

[0098] Figure 5 An example of the physical structure diagram of an electronic device is shown in Figure 5 as shown. The electronic device may include: a processor, a communications interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus. The processor can call the logical instructions in the memory to execute the method for registering 3D vectors and multi-view images based on the point set pyramid.

[0099] In addition, when the logical instructions in the above-mentioned memory are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0100] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional vector and multi-view image registration method based on the point set pyramid provided by the above-mentioned various methods.

[0101] In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the three-dimensional vector and multi-view image registration method based on the point set pyramid provided by the above-mentioned various methods.

[0102] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0103] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional vector and multi-view image registration method based on a point set pyramid, characterized in that: Perform the following process, Preprocess the multi-view images and 3D vectors to complete rough registration; Use edge template matching based on point set pyramids to match heterogeneous feature points. The implementation method includes: in the vector back-projection image, take the vector intersection point as the vector feature point, and take the corresponding pixel as the center to extract a preset-sized area as the template; in the edge feature image, take the pixel at the same position as the vector feature point as the center to extract a preset-sized area as the search area; respectively extract the coordinates of the valid pixels from the template and the search area to form their respective point sets, where the valid pixels represent the pixels corresponding to the vector lines and edge features; record the extracted point sets as the point sets at the main scale, and select the scaling factor between adjacent scales to construct a point set pyramid; then perform global main scale rough search and local multi-scale fine matching, and fit a parabolic equation based on the coordinates and the corresponding average nearest neighbor distance in the neighborhood around the matching points under the obtained optimal scale combination to obtain the best matching points of the vector feature points, and complete the heterogeneous matching of the vector feature points in the vector back-projection image and the pixels at the same position in the edge feature image; After removing false matches, multi-view forward intersection, precise alignment of coordinate systems, and vector back-projection and occlusion, complete the precise registration between the 3D vectors and the multi-view images.

2. A three-dimensional vector and multi-view image registration method based on a point set pyramid according to claim 1, characterized in that: The implementation method includes the following steps, Step 1, preprocess the multi-view images, including obtaining the accurate internal and external orientation elements and distortion coefficients of each image with different viewing angles, and then performing distortion correction and edge extraction on each image with different viewing angles; Step 2, preprocess the 3D vectors, including transforming the 3D vectors to the photogrammetric result coordinate system through rough alignment of the coordinate system, and then performing back-projection and occlusion processing to obtain the rough registration result of the 3D vectors and the multi-view images; Step 3, perform heterogeneous matching of the vector feature points in the vector back-projection image and the pixels at the same position in the edge feature image based on the point set pyramid; Step 4, perform RANSAC false match removal based on binocular forward intersection on the set of matching points of each vector feature point, and then perform multi-view forward intersection on the inlier set obtained by false match removal of the vector feature points to obtain the optimal object space coordinates of the vector feature points; Step 5, use the vector feature points and the corresponding optimal object space points to perform precise alignment of the coordinate system; Step 6, back-project the 3D vectors after precise alignment of the coordinate system onto the images of the corresponding viewing angles, and perform occlusion processing to complete the precise registration between the 3D vectors and the multi-view images.

3. A three-dimensional vector and multi-view image registration method based on a point set pyramid according to claim 2, characterized in that: In Step 1, the following implementation method is used to preprocess the multi-view images, Obtain the internal and external parameters of the multi-view images, including performing aerial triangulation processing on the obtained multi-view images to solve the accurate internal and external orientation elements and lens distortion coefficients of each image; Perform distortion correction on each image according to the lens distortion coefficients and distortion models; Use the Canny operator to perform edge extraction on the distorted-corrected images to obtain the edge feature images.

4. A three-dimensional vector and multi-view image registration method based on a point set pyramid according to claim 2, characterized in that: In Step 2, the following implementation method is used to preprocess the 3D vectors, Select control points to solve the seven-parameter transformation coefficients, and perform rough alignment of the coordinate system to transform the 3D vectors to the photogrammetric result coordinate system; The three-dimensional vectors after rough alignment of the coordinate system are back-projected according to the internal and external parameters of the multi-view images to obtain the three-dimensional vector image of the multi-view images in this view, and hidden lines in the vector model are removed by hidden line removal.

5. A three-dimensional vector and multi-view image registration method based on a point set pyramid according to claim 2 or 3 or 4, characterized in that: In step 3, the following implementation method is adopted for heterologous matching of the vector back-projection map and the edge feature image based on the point set pyramid. Step 3.1: Extract the template and the search area. Specifically, in the vector back-projection map, take the vector intersection points as vector feature points, and take the corresponding pixels as the center to extract a preset-size area as the template; in the edge feature image, take the pixels at the same position as the vector feature points as the center to extract a preset-size area as the search area. Step 3.2: Extract the point sets, including extracting the coordinates of valid pixels from the template and the search area respectively to form their own point sets, where valid pixels represent the pixels corresponding to the vector lines and edge features. Step 3.3: Construct the point set pyramid, including recording the point sets extracted in step 3.2 as the point sets at the main scale, and selecting the scaling factor between adjacent scales to construct the point set pyramid. Step 3.4: Global main scale rough search, including performing a striding search on the template point set and the search area point set at the main scale, and combining clustering analysis to judge the reliability of the matching. Take the reliable clustering center to construct a buffer as the optimal search area. Step 3.5: Local multi-scale fine matching, including performing pixel-by-pixel fine matching between multiple scales of the optimal search area and the point set pyramid corresponding to the template. Then, divide the minimum point set distance obtained by each scale combination by the scaling ratio of the optimal search area point set relative to the main scale to obtain the normalized minimum distance in each scale combination. The template scale and the optimal search area scale corresponding to the minimum value in the normalized minimum distance are the optimal scale combination. Step 3.6: Parabolic surface fitting for precise positioning, including taking the coordinates and the corresponding average nearest neighbor distances in the neighborhood around the matching points obtained in step 3.5 under the optimal scale combination to fit the parabolic surface equation, obtaining the coordinates corresponding to the lowest point of the parabolic surface as the best matching point to be solved. Then, convert the coordinates of this point to the image coordinate system of the main scale to obtain the best matching point of the vector feature points.

6. The three-dimensional vector and multi-view image registration method based on a point set pyramid according to claim 5, characterized in that: In step 3.4, the following implementation method is adopted for the global main scale rough search. Perform a striding search, and use the average nearest neighbor distance as the similarity measurement method during the search. Perform clustering analysis on the 4 points with the smallest average nearest neighbor distance to judge the reliability of the matching, including according to the set clustering distance threshold. If 3 or more of these 4 points are clustered into one category, it is considered that the matching is reliable. Take the clustering center to construct a buffer as the optimal search area.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the three-dimensional vector and multi-view image registration method based on the point set pyramid according to any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium storing a computer program thereon, characterized in that: When the computer program is executed by the processor, it implements the three-dimensional vector and multi-view image registration method based on the point set pyramid according to any one of claims 1 to 6.

9. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by the processor, it implements the three-dimensional vector and multi-view image registration method based on the point set pyramid according to any one of claims 1 to 6.

Images