Three-dimensional vector and multi-view image registration method and device based on point set pyramid

Abstract

The invention provides a point set pyramid-based three-dimensional vector and multi-view image registration method and device, and the method comprises the steps: carrying out the preprocessing of a multi-view image and a three-dimensional vector, and completing the coarse registration; matching of different-source feature points is achieved through edge template matching based on a point set pyramid; and accurate registration between the three-dimensional vector and the multi-view image is completed through mismatching elimination, multi-view forward intersection, coordinate system precise alignment and vector back projection and blanking. According to the method, a three-dimensional different-source point matching problem is converted into a two-dimensional different-source image matching problem, a large number of feature points are accurately positioned in combination with multi-view imaging geometry, coordinate system precise alignment is completed, and therefore precise registration between a three-dimensional vector and a multi-view image is achieved. In addition, the method is also compatible with registration of a two-dimensional vector and an image.

Description

Technical Field

[0001] The invention belongs to the field of photogrammetry and remote sensing, and relates to a technical solution for realizing high-precision registration of three-dimensional vectors and multi-view images by using multi-view imaging geometry and heterogeneous image matching. Background Art

[0002] In large-scale manufacturing industries, 3D vector models such as CAD (Computer Aided Design) or BIM (Building Information Modeling) are widely used to guide processing and production. Subsequently, the model is reconstructed using measurement technologies such as 3D laser scanning or vision-based 3D reconstruction, and then used for deviation analysis after registration. Among them, vision-based measurement methods can provide richer texture information, can accurately capture small objects, and have the advantages of low cost and high efficiency, so it is also one of the most commonly used measurement methods. In fields such as urban planning and geological exploration, GIS (Geographic Information System) vectors are usually registered with remote sensing images to achieve data fusion, thereby providing strong data support for decision-making.

[0003] The core challenge of deviation analysis and data fusion lies in how to align two different types of data with high precision. Traditional alignment methods often rely on manually selecting a small number of points for coordinate system transformation. This method is not only prone to introducing accidental errors and systematic errors, and is difficult to meet high-precision requirements, but also significantly increases the workload when processing massive data. The difficulty of this type of problem lies in the difference in data types between the three-dimensional vector model and the point cloud model, which belongs to the problem of three-dimensional heterogeneous point matching. However, there are still great difficulties in accurately matching three-dimensional heterogeneous points. First, it is difficult to design a good descriptor to accurately describe and match these points; second, the positioning accuracy of three-dimensional feature points (usually corner points and edge points) in the LiDAR scan point cloud and the three-dimensional reconstructed dense point cloud is limited. Even after matching the corner point, additional errors will be introduced in the alignment.

[0004] The existing patent document CN104732577B provides a building texture extraction method based on a UAV low-altitude aerial survey system, which involves research and improvement in registration. However, its disadvantage is that it is necessary to match the building outline (vector) straight line with the straight line in the image instead of the corner point, and the registration of the building outline and the image requires manual intervention, and the degree of automation is low. Summary of the invention

[0005] Aiming at the problem of low accuracy of traditional three-dimensional vector and multi-view image registration, the present invention proposes a three-dimensional vector and multi-view image registration technical solution based on point set pyramid.

[0006] The technical solution proposed by the present invention for the above technical problem is a three-dimensional vector and multi-view image registration method based on a point set pyramid, which performs the following process: Preprocess multi-view images and 3D vectors to complete rough registration; The matching of heterogeneous feature points is realized by edge template matching based on point set pyramid. The implementation method includes taking the vector intersection point as the vector feature point in the vector back projection map, taking the corresponding pixel as the center, and extracting a preset size area as the template; in the edge feature image, taking the pixel at the same position of the vector feature point as the center, extracting a preset size area as the search area; extracting the coordinates of valid pixels from the template and the search area to form respective point sets, wherein the valid pixels represent the pixels corresponding to the vector line and the edge feature; recording the extracted point set as the point set under the main scale, and selecting the scaling factor between adjacent scales to construct a point set pyramid; then performing a global main scale coarse search and a local multi-scale fine matching, fitting the parabola equation based on the coordinates in the neighborhood around the matching point under the obtained optimal scale combination and the corresponding average nearest neighbor distance, obtaining the best matching point of the vector feature point, and completing the heterogeneous matching of the vector feature point in the vector back projection map and the pixel at the same position in the edge feature image; After false matching elimination, multi-view forward intersection, precise alignment of coordinate systems, and vector back-projection and hidden removal, accurate registration between the three-dimensional vector and the multi-view image is completed.

[0007] Furthermore, the implementation method includes the following steps, Step 1: preprocess the multi-view images, including obtaining the precise internal and external orientation elements and distortion coefficients of each image at different viewing angles, and then performing distortion correction and edge extraction on each image at different viewing angles; Step 2, preprocessing the three-dimensional vector, including converting the three-dimensional vector into the coordinate system of the photogrammetry result through rough alignment of the coordinate system, and then performing back-projection and hidden removal processing to obtain the rough registration result of the three-dimensional vector and the multi-view image; Step 3, performing heterogeneous matching on the vector feature points in the vector back-projection image and the pixels at the same position in the edge feature image based on the point set pyramid; Step 4, perform RANSAC false matching elimination based on double-image forward intersection on the matching point set of each vector feature point, and then perform multi-view forward intersection on the inner point set obtained by false matching elimination on the vector feature point to obtain the optimal object coordinates of the vector feature point; Step 5: Use the vector feature points and the corresponding optimal object points to precisely align the coordinate system; Step 6, back-project the three-dimensional vector after the coordinate system is precisely aligned onto the image of the corresponding viewing angle, and perform hidden processing to complete the precise registration of the three-dimensional vector and the multi-view image.

[0008] Furthermore, the preprocessing of the multi-view images in step 1 is implemented in the following manner: Obtain the internal and external parameters of the multi-view images, including performing aerial triangulation on the acquired multi-view images to obtain the precise internal and external orientation elements and lens distortion coefficients of each image

[0009] Perform distortion correction on each image according to the lens distortion coefficient and distortion model; The Canny operator is used to extract the edge of the distortion-corrected image to obtain the edge feature image.

[0010] Moreover, the preprocessing of the three-dimensional vector in step 2 is implemented in the following manner: Select control points to solve the seven-parameter transformation coefficients, perform rough alignment of the coordinate system, and transform the three-dimensional vector into the coordinate system of the photogrammetry results; The three-dimensional vector after the rough alignment of the coordinate system is back-projected according to the internal and external parameters of the multi-view image to obtain the three-dimensional vector image of the multi-view image in the view, and the blocked lines in the vector model are removed by hiding.

[0011] Moreover, in step 3, the heterogeneous matching of the vector back-projection image and the edge feature image based on the point set pyramid is implemented in the following way: Step 3.1, extracting the template and the search area, including taking the vector intersection point as the vector feature point in the vector back projection image, taking the corresponding pixel as the center, and extracting the area of ​​preset size as the template; in the edge feature image, taking the pixel at the same position of the vector feature point as the center, extracting the area of ​​preset size as the search area; Step 3.2, extracting point sets, including extracting coordinates of valid pixels from the template and the search area to form respective point sets, wherein valid pixels represent pixels corresponding to vector lines and edge features; Step 3.3, constructing a point set pyramid, including recording the point set extracted in step 3.2 as the point set under the main scale, and selecting the scaling factor between adjacent scales to construct the point set pyramid; Step 3.4, global main scale coarse search, including stride search of the template point set and search area point set under the main scale, and combining cluster analysis to judge the reliability of matching, taking reliable cluster centers to construct a buffer zone as the optimal search area; Step 3.5, local multi-scale fine matching, including multi-scale and multi-scale pixel-by-pixel fine matching of the point set pyramid corresponding to the optimal search area and the template, and then dividing the minimum point set distance obtained in each scale combination by the scaling ratio of the optimal search area point set used relative to the main scale to obtain the normalized minimum distance in each scale combination. The template scale and optimal search area scale corresponding to the minimum value in the normalized minimum distance are the optimal scale combination; Step 3.6, parabola fitting and precise positioning, includes taking the coordinates in the neighborhood around the matching point under the optimal scale combination obtained in step 3.5 and the corresponding average nearest neighbor distance to fit the parabola equation, and obtaining the coordinates corresponding to the lowest point of the parabola as the best matching point to be solved; then converting the coordinates of the point to the image coordinate system of the main scale to obtain the best matching point of the vector feature point.

[0012] Moreover, the global main scale coarse search in step 3.4 is implemented as follows: Perform a stride search and use the average nearest neighbor distance as the similarity measure during the search; Cluster analysis is performed on the four points with the smallest average nearest neighbor distance to determine the reliability of the match, including according to the set cluster distance threshold. If three or more of the four points are clustered into one category, the match is considered reliable, and the cluster center is used to construct a buffer zone as the optimal search area.

[0013] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the three-dimensional vector and multi-view image registration method based on a point set pyramid as described above is implemented.

[0014] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the three-dimensional vector and multi-view image registration method based on a point set pyramid as described above.

[0015] On the other hand, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the three-dimensional vector and multi-view image registration method based on a point set pyramid as described above.

[0016] The present invention proposes a three-dimensional vector and multi-view image registration method based on a point set pyramid. The method converts the three-dimensional heterogeneous point matching problem into a two-dimensional heterogeneous image matching problem, and accurately locates a large number of feature points in combination with multi-view imaging geometry to complete the precise alignment of the coordinate system, thereby achieving accurate registration between the three-dimensional vector and the multi-view image. Compared with the prior art, the advantage of the present invention is that it uses the internal and external parameters of the multi-view image to perform vector back projection, obtains the vector back projection map corresponding to the multi-view image, and uses the pyramid strategy to perform template matching on the vector intersection point and the image edge detection result, with a high degree of automation and no need for manual intervention. Specifically, the present invention has the following advantages: It has a wide range of applications. In addition to supporting accurate registration between 3D vectors and multi-view images, it is also compatible with 2D vectors and large-sized objects. It has a high degree of automation and can automatically extract a large number of points with the same name through heterogeneous matching, eliminating the need for manual point selection. It has high registration accuracy and can precisely align the coordinate system through a large number of points, avoiding accidental errors and systematic errors that exist when using only a small number of points. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 The figure is a flow chart of the three-dimensional vector and multi-view image registration according to an embodiment of the present invention.

[0017] Figure 2 Schematic diagram of edge template matching based on point set pyramid according to an embodiment of the present invention.

[0018] FIG. 3 is a diagram of left and right rolling blinds showing the precise registration effect according to an embodiment of the present invention.

[0019] FIG. 4 is a diagram of upper and lower rolling shutters showing the precise registration effect according to an embodiment of the present invention.

[0020] FIG. 5 is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0022] The embodiment of the present invention provides a three-dimensional vector and multi-view image registration method based on a point set pyramid, the core of which includes three processes: first, preprocessing the multi-view image and the three-dimensional vector to complete the rough registration; then, using the edge template matching based on the point set pyramid to achieve the matching of heterogeneous feature points; finally, through false matching elimination, multi-view front intersection, coordinate system precise alignment and vector back projection and blanking, the precise registration between the three-dimensional vector and the multi-view image is completed.

[0023] like Figure 1 As shown, the process of the embodiment of the present invention includes the following steps: Step 1: pre-process the multi-view images to obtain the precise internal and external orientation elements and distortion coefficients of each image, and then perform distortion correction and edge extraction on each image.

[0024] The implementation method further preferably adopted in the embodiment includes the following sub-steps: Step 1.1: Perform aerial triangulation on the acquired multi-view images to obtain the precise internal and external orientation elements and lens distortion coefficients for each image at different viewing angles. .in, are the parameters describing the radial distortion model, are the parameters describing the tangential distortion model.

[0025] Step 1.2, perform distortion correction on each image according to the lens distortion coefficient and the following distortion model.

[0026]

[0027] In the formula, represents the coordinates of the ideal image point with the image principal point as the origin of the coordinate system; It indicates the coordinates of the image points with distortion; represents the radial length, .

[0028] Step 1.3: Use the Canny operator to extract the edge of each distortion-corrected image to obtain the corresponding edge feature image.

[0029] Step 2, preprocessing the three-dimensional vector, including converting the three-dimensional vector into the coordinate system of the photogrammetry result through rough alignment of the coordinate system, and then obtaining the rough alignment result of the three-dimensional vector and the multi-view image through back projection and hidden removal processing.

[0030] The implementation method further preferably adopted in the embodiment includes the following sub-steps: Step 2.1, select a small number of control points to solve the seven-parameter transformation coefficients, and perform rough alignment of the coordinate system to transform the three-dimensional vector into the coordinate system of the photogrammetry results. In specific implementation, it is recommended to select 5-9 control points in this step. The seven-parameter transformation includes three translation parameters, three rotation parameters, and one scaling parameter. Its model is as follows:

[0031] In the formula, Represents the coordinates of a three-dimensional vector in the world coordinate system; Represents the coordinates of a three-dimensional vector in its own model coordinate system (hereinafter referred to as the three-dimensional vector coordinate system); is the rotation matrix for transforming a 3D vector to the world coordinate system, They are the three-dimensional vector coordinate system around the world coordinate system The angle of axis rotation, and corresponding three rotation parameters; They are respectively The calculated rotation matrix; Represents the offset between the origin of the three-dimensional vector coordinate system and the origin of the world coordinate system, which consists of three translation parameters; Corresponding scaling parameters.

[0032] Step 2.2, back-project the three-dimensional vector after the rough alignment of the coordinate system according to the internal and external parameters of the multi-view image obtained in step 1.1, and preferably use the Z-Buffer algorithm to hide and remove the blocked lines in the vector model to obtain the corresponding results for each image with different viewing angles, that is, a series of vector back-projection images.

[0033] The implementation of the Z-Buffer algorithm is a prior art and will not be described in detail in the present invention. The back projection is implemented through the collinear condition equation, which is specifically as follows:

[0034] In the formula, represents the ideal image point coordinates with the image principal point as the origin of the coordinate system; the image principal point coordinates and focal length is the internal orientation element of the image; Object point Coordinates in the world coordinate system; For photography site The coordinates in the world coordinate system are also the three exterior orientation line elements corresponding to the image; The three exterior azimuth elements of the image The 9 direction cosines.

[0035] Step 3, such as Figure 2 As shown, the heterogeneous feature points in the vector back projection image and the edge feature image are matched by using edge template matching based on point set pyramid.

[0036] The implementation method further preferably adopted in the embodiment includes the following sub-steps: Step 3.1, extract the template and search area. In the vector back projection map, take the vector intersection as the feature point and extract the corresponding pixel as the center. The template binary image can be obtained by taking the area of ​​​​the size of the template as the template; in the edge feature image, the pixel corresponding to the feature point is taken as the center to extract The area of ​​the search area can be used as the search area to obtain the binary image of the edge of the search area. =1.5× , =1.5× , , The experience threshold is 300 180.

[0037] Step 3.2, extracting point sets. The coordinates of valid pixels are extracted from the template and the search area to form their respective point sets, where valid pixels represent pixels corresponding to vector lines and edge features.

[0038] Step 3.3: Construct a point set pyramid. The point set extracted in step 3.2 is recorded as the main scale The following point set, and set As the scaling factor between adjacent scales, the point set pyramid is constructed. k The scale of the layer is , where the point set coordinates and the main scale The corresponding point set coordinates are proportional to each other, and the ratio is The point set coordinates are saved as floating point numbers to avoid quantization and resampling errors.

[0039] Step 3.4, global main scale coarse search.

[0040] Main scale The template point set and the search area point set are used for stride search, and the reliability of the match is judged by cluster analysis. The average nearest neighbor distance is used for stride search. As a measure of the similarity between the two, the specific formula is as follows:

[0041] In the formula, the average nearest neighbor distance Represents two point sets , The average distance between each point in the set and the nearest point in the other point set can be used to measure the average similarity between the two point sets; forward distance Represents a point set Point-to-point set in Average distance between the closest points, backward distance Represents a point set Point-to-point set in The average distance between the closest points; Represents a point set Points in Arrival Set The minimum Euclidean distance, Represents a point set Points in Arrival Set The minimum Euclidean distance. In formula (6), m , n are the number of concentrated points at points A and B respectively.

[0042] Then, cluster analysis is performed on the four points with the smallest average nearest neighbor distance to determine the reliability of the match. According to the set cluster distance threshold, if three or more of the four points are clustered into one category, the match is considered reliable, and the cluster center is taken to build a buffer zone as the optimal search area. In other cases, it is considered that there is no area in the search area with high similarity to the template point set, and the point is skipped for matching.

[0043] Step 3.5: Local multi-scale fine matching. Perform pixel-by-pixel fine matching between the optimal search area and the point set pyramid corresponding to the template, and then divide the minimum point set distance obtained by each scale combination by the scaling ratio of the optimal search area point set relative to the main scale. After that, the normalized minimum distance in each scale combination is obtained. The template scale and the optimal search area scale corresponding to the minimum value in the normalized minimum distance are taken as the optimal scale combination, and the search area coordinates corresponding to the minimum value are the matching points of the feature points under this scale combination.

[0044] Step 3.6, parabola fitting precision positioning. Take the 9 coordinates in the 3×3 neighborhood around the matching point under the optimal scale combination obtained in step 3.5 and the corresponding average nearest neighbor distance to fit the following parabola equation to obtain the coordinates corresponding to the lowest point of the parabola , which is the best matching point to be solved; then convert the coordinates of the point from the optimal search area scale to the main scale The best matching point of the vector feature point can be obtained in the image coordinate system.

[0045]

[0046] In the formula, a , b , c are the coefficients of the parabola equation to be fitted; ( x , y ) is the image point coordinate; z is the average nearest neighbor distance value; Step 4, perform RANSAC false matching elimination based on double-image forward intersection on the matching point set of each vector feature point, and then perform multi-view forward intersection on the inner point set obtained by false matching elimination on the vector feature point to obtain the optimal object coordinates of the vector feature point; The specific implementation method for processing each vector feature point further preferably adopted in the embodiment includes the following sub-steps: Step 4.1, randomly select two matching points to perform forward intersection of the two images and solve the object coordinates of the intersection points. Initialize an inner point set and add the two matching points to the inner point set.

[0047] Step 4.2, use the obtained object coordinates to perform back projection and calculate the reprojection error on the image corresponding to the remaining matching points. The reprojection error calculation method is a prior art and will not be described in detail in the present invention.

[0048] In step 4.3, matching points whose reprojection error is less than a preset threshold (preferably 0.8 pixels) are included in the inliers.

[0049] Step 4.4, repeat steps 4.1 to 4.3, and generate corresponding inner point sets each time until the maximum number of iterations is met. Then, the inner point set with the largest number of inner points is selected as the valid matching set for multi-view forward intersection. The maximum number of iterations is recommended to be 200, which can be increased appropriately when the matching quality is poor.

[0050] Step 4.5, perform multi-view forward intersection on the internal point set obtained by eliminating false matches to obtain the optimal object coordinates of the feature point.

[0051] The specific implementation of multi-view forward intersection is a prior art and will not be described in detail in the present invention.

[0052] Step 5: Use the large number of vector feature points obtained to precisely align the coordinate system with the corresponding optimal object space points.

[0053] In this step, the vector feature points and the corresponding optimal object points obtained through step 4 are substituted into the seven-parameter transformation model described in formula (2) to obtain the least squares solution, and then the precise coordinate system conversion parameters between the model coordinate system and the photogrammetry result coordinate system are obtained, and the three-dimensional vector is converted into the photogrammetry result coordinate system to complete the precise alignment.

[0054] In the specific implementation, hundreds to thousands of vector feature points may be generated. In this step, it is preferably recommended to select no less than 100 vector points for fine alignment. The principle of fine alignment of the coordinate system in step 5 is the same as that in step 2.1, both of which are to solve the seven-parameter transformation coefficients, but a large number of points are used for solving in step 5, which to a certain extent eliminates the system error existing in step 2.1 when only a small number of points are used for solving.

[0055] Step 6, back-project the three-dimensional vector after the coordinate system is precisely aligned onto the image of the corresponding perspective, and it is also preferably recommended to use the Z-Buffer algorithm for hidden processing to complete the precise alignment of the three-dimensional vector and the multi-view image.

[0056] The back-projection and hidden-elimination principles in step 6 are the same as those in step 2.2, except that the three-dimensional vectors involved in the back-projection are precisely aligned.

[0057] After completing the above steps, accurate registration of 3D vectors and multi-view images can be achieved.

[0058] In order to facilitate understanding of the technical effect of the present invention, a vector back projection image is matched with the corresponding drone edge feature image. The matching effect of the present invention and the effect of the traditional template matching method are compared as shown in Table 1: Table 1 Matching results of vector back projection map and 30 multi-view images

[0059] As can be seen from Table 1, the method of the present invention achieves a higher matching accuracy, which is significantly better than the traditional template matching method.

[0060] The test confirmed that the vector feature points used have found accurate matching points on the edge feature images.

[0061] In order to prove the technical effect of the present invention, an experiment was conducted using a drone to approach a photographic image and an existing target CAD vector model. After the image and vector were precisely registered based on the present invention, the effect of the vector back projection image and the multi-view image superposition display is as follows: Figure 3 , Figure 4 As shown in the figure, it can be seen that there is almost no misalignment at the joint of the image and the vector back projection image, and the registration effect is good.

[0062] During specific implementation, the above process can be automatically operated using computer software technology.

[0063] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The processor may call the logic instructions in the memory to execute the three-dimensional vector and multi-view image registration method based on the point set pyramid.

[0064] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0065] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the three-dimensional vector and multi-view image registration method based on point set pyramid provided by the above methods.

[0066] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the three-dimensional vector and multi-view image registration method based on a point set pyramid provided by the above methods.

[0067] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0068] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional vector and multi-view image registration method based on point set pyramid, characterized by: Carry out the following process, Preprocess multi-view images and 3D vectors to complete rough registration; The matching of heterogeneous feature points is realized by edge template matching based on point set pyramid. The implementation method includes taking the vector intersection point as the vector feature point in the vector back projection map, taking the corresponding pixel as the center, and extracting a preset size area as the template; in the edge feature image, taking the pixel at the same position of the vector feature point as the center, extracting a preset size area as the search area; extracting the coordinates of valid pixels from the template and the search area to form respective point sets, wherein the valid pixels represent the pixels corresponding to the vector line and the edge feature; recording the extracted point set as the point set under the main scale, and selecting the scaling factor between adjacent scales to construct a point set pyramid; then performing a global main scale coarse search and a local multi-scale fine matching, fitting the parabola equation based on the coordinates in the neighborhood around the matching point under the obtained optimal scale combination and the corresponding average nearest neighbor distance, obtaining the best matching point of the vector feature point, and completing the heterogeneous matching of the vector feature point in the vector back projection map and the pixel at the same position in the edge feature image; After false matching elimination, multi-view forward intersection, precise alignment of coordinate systems, and vector back-projection and hidden removal, accurate registration between the three-dimensional vector and the multi-view image is completed.

2. The method for aligning three-dimensional vectors and multi-view images based on a point set pyramid as claimed in claim 1, characterized in that: The implementation method includes the following steps: Step 1: preprocess the multi-view images, including obtaining the precise internal and external orientation elements and distortion coefficients of each image at different viewing angles, and then performing distortion correction and edge extraction on each image at different viewing angles; Step 2, preprocessing the three-dimensional vector, including converting the three-dimensional vector into the coordinate system of the photogrammetry result through rough alignment of the coordinate system, and then performing back-projection and hidden removal processing to obtain the rough registration result of the three-dimensional vector and the multi-view image; Step 3, performing heterogeneous matching on the vector feature points in the vector back-projection image and the pixels at the same position in the edge feature image based on the point set pyramid; Step 4, perform RANSAC false matching elimination based on double-image forward intersection on the matching point set of each vector feature point, and then perform multi-view forward intersection on the inner point set obtained by false matching elimination on the vector feature point to obtain the optimal object space coordinates of the vector feature point; Step 5: Use the vector feature points and the corresponding optimal object points to precisely align the coordinate system; Step 6, back-project the three-dimensional vector after the coordinate system is precisely aligned onto the image of the corresponding viewing angle, and perform hidden processing to complete the precise registration of the three-dimensional vector and the multi-view image.

3. The method for aligning three-dimensional vectors and multi-view images based on a point set pyramid as claimed in claim 2, characterized in that: In step 1, the preprocessing of multi-view images is implemented in the following way: Obtain the internal and external parameters of the multi-view images, including performing aerial triangulation on the acquired multi-view images to obtain the precise internal and external orientation elements and lens distortion coefficients of each image Perform distortion correction on each image according to the lens distortion coefficient and distortion model; The Canny operator is used to extract the edge of the distortion-corrected image to obtain the edge feature image.

4. The method for aligning three-dimensional vectors and multi-view images based on a point set pyramid as claimed in claim 2, characterized in that: In step 2, the three-dimensional vector is preprocessed in the following manner: Select control points to solve the seven-parameter transformation coefficients, perform rough alignment of the coordinate system, and transform the three-dimensional vector into the coordinate system of the photogrammetry results; The three-dimensional vector after the rough alignment of the coordinate system is back-projected according to the internal and external parameters of the multi-view image to obtain the three-dimensional vector image of the multi-view image in the view, and the blocked lines in the vector model are removed by hiding.

5. The method for aligning three-dimensional vectors and multi-view images based on a point set pyramid as claimed in claim 2, 3 or 4, characterized in that: In step 3, the heterogeneous matching of the vector back projection image and the edge feature image based on the point set pyramid is implemented in the following way: Step 3.1, extracting the template and the search area, including taking the vector intersection point as the vector feature point in the vector back projection image, taking the corresponding pixel as the center, and extracting the area of ​​preset size as the template; in the edge feature image, taking the pixel at the same position of the vector feature point as the center, extracting the area of ​​preset size as the search area; Step 3.2, extracting point sets, including extracting coordinates of valid pixels from the template and the search area to form respective point sets, wherein valid pixels represent pixels corresponding to vector lines and edge features; Step 3.3, constructing a point set pyramid, including recording the point set extracted in step 3.2 as the point set under the main scale, and selecting the scaling factor between adjacent scales to construct the point set pyramid; Step 3.4, global main scale coarse search, including stride search of the template point set and search area point set under the main scale, and combining cluster analysis to judge the reliability of matching, taking reliable cluster centers to construct a buffer zone as the optimal search area; Step 3.5, local multi-scale fine matching, including multi-scale and multi-scale pixel-by-pixel fine matching of the point set pyramid corresponding to the optimal search area and the template, and then dividing the minimum point set distance obtained in each scale combination by the scaling ratio of the optimal search area point set used relative to the main scale to obtain the normalized minimum distance in each scale combination. The template scale and optimal search area scale corresponding to the minimum value in the normalized minimum distance are the optimal scale combination; Step 3.6, parabola fitting and precise positioning, includes taking the coordinates in the neighborhood around the matching point under the optimal scale combination obtained in step 3.5 and the corresponding average nearest neighbor distance to fit the parabola equation, and obtaining the coordinates corresponding to the lowest point of the parabola as the best matching point to be solved; then converting the coordinates of the point to the image coordinate system of the main scale to obtain the best matching point of the vector feature point.

6. The method for aligning three-dimensional vectors and multi-view images based on a point set pyramid as claimed in claim 5, characterized in that: The global main scale coarse search in step 3.4 is implemented as follows: Perform a stride search and use the average nearest neighbor distance as the similarity measure during the search; Cluster analysis is performed on the four points with the smallest average nearest neighbor distance to determine the reliability of the match, including according to the set cluster distance threshold. If three or more of the four points are clustered into one category, the match is considered reliable, and the cluster center is used to construct a buffer zone as the optimal search area.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the three-dimensional vector and multi-view image registration method based on point set pyramid as described in any one of claims 1 to 6 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for aligning three-dimensional vectors and multi-view images based on a point set pyramid as claimed in any one of claims 1 to 6 is implemented.

9. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the method for aligning three-dimensional vectors and multi-view images based on a point set pyramid as claimed in any one of claims 1 to 6 is implemented.

Images