A method and system for constructing a sparse three-dimensional model with multi-view perception

Abstract

The present invention relates to the technical field of power equipment, and specifically to a method and system for constructing a sparse three-dimensional model with multi-perspective perception, the method comprising collecting target image data, and extracting time information, coordinate information and elevation information from the target image data; dividing the extracted target image data into regions based on the coordinate point information and the elevation information to obtain a global image set and multiple bounding boxes; processing the registration of image data within a single bounding box; processing the matching between neighborhood bounding boxes, and splicing the neighborhood bounding boxes to obtain an initial sparse three-dimensional model; optimizing the obtained initial sparse three-dimensional model to obtain a sparse three-dimensional model with multi-perspective perception. The present invention can construct an initial sparse model based on a large number of visible light images that currently exist and are being generated. Visible light images can be obtained through fixed and mobile acquisition devices, and are not limited to the same type of acquisition devices.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power equipment, and in particular to a method and system for constructing a sparse three-dimensional model with multi-view perception. Background Art

[0002] Power equipment inspection is the main task of power operation and maintenance personnel. The introduction of drone inspection has improved efficiency and the working environment of operation and maintenance personnel. With the deepening of digitalization and intelligence, the development of virtual operation and digital twin technology has put forward requirements for the coverage of physical digital models.

[0003] At present, in addition to design drawings, the modeling of power equipment and scenes generally uses laser point cloud to collect point cloud data for modeling. The laser point cloud modeling method has high accuracy, but the cycle is long, the processing requirements are high, and the cost is high. At the same time, when electric drones are used as vehicles to carry high-load and high-energy-consuming equipment such as laser point clouds, the flight time is shorter due to battery life limitations, and the fixed camera angles and shooting scenes installed in the power scene are single.

[0004] For situations where high-precision measurement is not required and there is a need to quickly generate an initial sparse model of a physical digital model, the sparse model is the basis of the fine model, and the sparse model enables the preliminary determination of the three-dimensional structure of the scene. Summary of the invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the problem in the prior art that a sparse three-dimensional model cannot be constructed based on visible light images.

[0006] In order to solve the above technical problems, the present invention provides a method for constructing a sparse three-dimensional model with multi-view perception, comprising:

[0007] Collecting target image data, and extracting time information, coordinate information and elevation information from the target image data;

[0008] Divide the extracted target image data into regions based on the coordinate point information and the elevation information to obtain a global image set and a plurality of bounding boxes;

[0009] Processing registration of image data within a single bounding box;

[0010] Process the matching between the neighborhood bounding boxes and splice the neighborhood bounding boxes to obtain the initial sparse 3D model;

[0011] The obtained initial sparse three-dimensional model is optimized to obtain a multi-view perceived sparse three-dimensional model.

[0012] In one embodiment of the present invention, the extracted target image data is divided into regions based on the coordinate point information and the elevation information to obtain a global image set and a plurality of bounding boxes, including:

[0013] Divide the target image data into regions from low to high based on the elevation range according to the image coordinate point coverage area;

[0014] The image data corresponding to the coordinate discrete points outside the region are grouped as the global image set region;

[0015] The remaining spatial regions not including the global image set are grouped into a square bounding box with a side length of 20m to 50m in a three-dimensional coordinate system.

[0016] In one embodiment of the present invention, processing the registration of image data within a single bounding box includes:

[0017] For images taken at different times within the same single bounding box, a sequence of images under coordinate constraints is constructed according to the time sequence;

[0018] The images of the same time series whose coordinates are in each bounding box are matched to obtain the feature points of the image set and the three-dimensional spatial structure represented by the feature points.

[0019] In one embodiment of the present invention, when matching images of the same time series whose coordinates are within each bounding box, a SIMATIC registration algorithm is used to obtain a matching relationship between feature points in the images.

[0020] In one embodiment of the present invention, the step of matching the image order by the sift registration algorithm comprises:

[0021] In the same bounding box, a center point is obtained according to the distribution of coordinate points, a three-dimensional k-nearest coordinate point image of the center point is calculated, and the images are saved as a comprehensive image sorting;

[0022] Perform sift registration on the first two three-dimensional k-nearest-neighbor coordinate point images, and use the center points of the coordinate points of the first two three-dimensional k-nearest-neighbor coordinate point images as new center points;

[0023] Based on the comprehensive image sorting, the third closest coordinate point is determined, and the corresponding image is used as an increment to match the previous matching result until all images in the bounding box are matched, thereby obtaining a local matching model of the bounding box.

[0024] In one embodiment of the present invention, processing the matching between the neighborhood bounding boxes and splicing the neighborhood bounding boxes to obtain an initial sparse three-dimensional model includes:

[0025] When the coordinate point is located on the boundary of the bounding box or on an adjacent surface, the image with the coordinate point located on the boundary is matched with the image with the closest coordinate position of the adjacent bounding box, so as to realize the splicing of two or more local matching models corresponding to the adjacent bounding boxes;

[0026] When the coordinate point position is not on the boundary of the bounding box or the adjacent surface, when the bounding boxes are aligned, the coordinate point corresponding to the image closest to the coplanar surface is selected for matching to achieve the splicing of two or more local matching models corresponding to adjacent bounding boxes.

[0027] In one embodiment of the present invention, after all the local matching models are spliced ​​together, a spliced ​​initial sparse three-dimensional model is obtained.

[0028] In one embodiment of the present invention, when the local matching models are spliced, the local matching models are geometrically transformed, and the feature point set in the local matching models to be spliced ​​is subjected to homography matrix estimation based on the RANSAC algorithm to obtain the translation and rotation matrix in the geometric transformation.

[0029] The present invention further provides a multi-view perception sparse three-dimensional model construction system, which is applied to the multi-view perception sparse three-dimensional model construction method described in any one of the above embodiments, comprising:

[0030] An image data extraction module, used to collect target image data and extract time information, coordinate information and elevation information from the target image data;

[0031] A partitioning module, used for partitioning the extracted target image data into regions based on the coordinate point information and the elevation information to obtain a global image set and a plurality of bounding boxes;

[0032] A bounding box processing module, used to process the registration of image data within a single bounding box;

[0033] The bounding box processing module is used to process the matching between neighboring bounding boxes;

[0034] The local matching model stitching module is used to stitch the neighborhood bounding boxes to obtain an initial sparse 3D model;

[0035] The sparse three-dimensional model optimization module is used to optimize the initial sparse three-dimensional model to obtain a sparse three-dimensional model with multi-view perception.

[0036] In one embodiment of the present invention, it also includes:

[0037] The feature point extraction module is used to extract feature points in the local matching model and optimize the quality of the feature point set.

[0038] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0039] The present invention can build an initial sparse model based on a large number of visible light images that currently exist and are being generated. The visible light images can be acquired through fixed and mobile acquisition devices and are not limited to the same type of acquisition devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0041] Figure 1 A flowchart of a method for constructing a sparse three-dimensional model with multi-view perception provided by an embodiment of the present invention;

[0042] Figure 2 A functional module diagram of a multi-view perception sparse three-dimensional model construction system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0044] In order to overcome the problems existing in the prior art, the present invention proposes a method and system for constructing a sparse three-dimensional model with multi-view perception. Figure 1 As shown, Figure 1 A flowchart of a method for constructing a sparse three-dimensional model with multi-view perception provided by an embodiment of the present invention, the method comprising:

[0045] S101: Acquire target image data, and extract time information, coordinate information, and elevation information from the target image data.

[0046] S102: Divide the extracted target image data into regions based on the coordinate point information and the elevation information to obtain a global image set and a plurality of bounding boxes.

[0047] In some possible implementations, in step S102, the extracted target image data is divided into regions based on the coordinate point information and the elevation information to obtain a global image set and a plurality of bounding boxes, including:

[0048] Divide the target image data into regions from low to high based on the elevation range according to the image coordinate point coverage area;

[0049] The image data corresponding to the coordinate discrete points outside the region are grouped as the global image set region;

[0050] The remaining spatial regions not including the global image set are grouped into a square bounding box with a side length of 20m to 50m in a three-dimensional coordinate system.

[0051] Exemplarily, the time, coordinate and elevation information in the collected image data are extracted, and the image data is divided into regions from low to high according to the coverage area of ​​the image coordinate points, and the images corresponding to the discrete coordinate points outside the region can be grouped into a global image set, and the discrete points can be determined by clustering the coordinate points. The global image set can be extracted and saved separately, and then used for the optimization of the initial sparse three-dimensional model. Excluding the remaining spatial areas of the global image set, the division unit uses a square bounding box with a side length of 20-50m in the three-dimensional coordinate system.

[0052] S103: Processing the registration of image data within a single bounding box.

[0053] In some possible implementations, processing the registration of image data within a single bounding box in step S103 includes:

[0054] For images taken at different times within the same single bounding box, a sequence of images under coordinate constraints is constructed according to the time sequence;

[0055] The images of the same time series whose coordinates are in each bounding box are matched to obtain the feature points of the image set and the three-dimensional spatial structure represented by the feature points.

[0056] In some possible implementations, when matching images of the same time series whose coordinates are within each bounding box, a SIMATIC registration algorithm is used to obtain a matching relationship between feature points in the images.

[0057] Exemplarily, for images taken at different times in the same bounding box, sequential images under coordinate constraints are constructed before and after time, and the images of the same time series with coordinates in each bounding box are matched. The matching adopts an incremental sift registration algorithm to obtain the feature points of the image set and the three-dimensional space structure represented by the feature points. The physical correspondence of the feature points includes but is not limited to vertices, corner points, intersections, etc. The extracted feature digital form is a vector or histogram, etc. At this time, the three-dimensional space structure is still very sparse and needs to be processed in subsequent links. Different time series images also use the incremental method to optimize and supplement the model to enhance the robustness of the model.

[0058] In some possible implementations, the step of matching the image order by the sift registration algorithm includes:

[0059] In the same bounding box, a center point is obtained according to the distribution of coordinate points, a three-dimensional k-nearest coordinate point image of the center point is calculated, and the images are saved as a comprehensive image sorting;

[0060] Perform sift registration on the first two three-dimensional k-nearest-neighbor coordinate point images, and use the center points of the coordinate points of the first two three-dimensional k-nearest-neighbor coordinate point images as new center points;

[0061] Based on the comprehensive image sorting, the third closest coordinate point is determined, and the corresponding image is used as an increment to match the previous matching result until all images in the bounding box are matched, thereby obtaining a local matching model of the bounding box.

[0062] Exemplarily, the registration within the bounding box can use Sift registration to obtain the matching relationship of significant feature points in the image. The method for determining the order of matching images is to obtain the center point according to the distribution of coordinate points in the same bounding box, calculate the three-dimensional k nearest coordinate point images of the center point, and save them as comprehensive image sorting, and perform Sift registration on the first two images. Then, the center of the two image coordinate points is used as the new center point, and based on the comprehensive image sorting, the third nearest coordinate point is determined, and the corresponding image is used as an increment to match the previous matching result, and so on, until all images in the bounding box are matched, and the same operation is performed on the remaining bounding boxes to obtain local matching models of different bounding boxes.

[0063] S104: Process the matching between the neighborhood bounding boxes and splice the neighborhood bounding boxes to obtain an initial sparse three-dimensional model.

[0064] In some possible implementations, processing the matching between the neighborhood bounding boxes in step S104 and splicing the neighborhood bounding boxes to obtain an initial sparse three-dimensional model includes:

[0065] When the coordinate point is located on the boundary of the bounding box or on an adjacent surface, the image with the coordinate point located on the boundary is matched with the image with the closest coordinate position of the adjacent bounding box, so as to realize the splicing of two or more local matching models corresponding to the adjacent bounding boxes;

[0066] When the coordinate point position is not on the boundary of the bounding box or the adjacent surface, when the bounding boxes are aligned, the coordinate point corresponding to the image closest to the coplanar surface is selected for matching to achieve the splicing of two or more local matching models corresponding to adjacent bounding boxes.

[0067] In some possible implementations, when the local matching models are spliced, the local matching models are geometrically transformed, and the feature point set in the local matching models to be spliced ​​is subjected to homography matrix estimation based on the RANSAC algorithm to obtain a translation and rotation matrix in the geometric transformation.

[0068] For example, when the coordinate point is exactly on the boundary of the bounding box or on an adjacent surface, the neighborhood attributes of the bounding box are restricted, and the image with the coordinate point at the boundary is matched with the image with the coordinate position closest to the adjacent bounding box, thereby achieving the splicing of two or more local matching models corresponding to the adjacent bounding boxes.

[0069] It should be noted that in the matching process, since the local matching model comes from multiple images within the bounding box, the geometric transformation of the local model is required when the local model is spliced. The feature point set in the model to be spliced ​​is estimated based on the RANSAC (random sampling consensus) algorithm to obtain the translation and rotation matrix in the geometric transformation.

[0070] When there is no corresponding coordinate point image on the adjacent bounding box boundary or adjacent surface, when the bounding boxes are aligned, the corresponding image of the coordinate point closest to the coplanar surface is selected for matching, and the connection relationship of the adjacent local matching models is constructed. Subsequently, the local matching models are also spliced, which should also satisfy the translation or rotation relationship and record the corresponding translation rotation matrix. The translation rotation matrix can realize the recording of the direction change of the mobile acquisition device.

[0071] In some possible implementations, after all the local matching models are spliced ​​together, a spliced ​​initial sparse three-dimensional model is obtained.

[0072] Exemplarily, local matching model stitching is to eliminate seams, lighting differences and visual discontinuities between adjacent models, and use image fusion technology, such as multi-band blending, Poisson fusion and other methods, to perform operations on all local matching models to obtain a stitched initial sparse three-dimensional model.

[0073] S105: Optimizing the obtained initial sparse three-dimensional model to obtain a multi-view perceived sparse three-dimensional model.

[0074] Furthermore, for power scenes, there are a lot of constraints of standards and specifications, such as the constraints between the voltage level and tower height of the transmission tower, the installation position and size of the tower number and other signs on the transmission tower, the height information of the base of the substation equipment, etc. By manually setting certain unified standard features, the camera parameters can be solved through these features and the analytical relationship between the corresponding feature points extracted from the image. The obtained camera parameters will be used for the reprojection of the 3D model to optimize the accuracy of the generated model.

[0075] The optimization of the sparse three-dimensional model with multi-view perception uses reprojection to improve the accuracy of existing feature points. The method is to extract feature points, reproject the three-dimensional feature points into the eight-neighborhood of the bounding box where they are located, remove the six bounding boxes adjacent to the surface, and connect the remaining points to four bounding boxes. The translation and rotation matrices between the bounding boxes adjacent to the surface are known, and any combination can be selected to realize the translation and rotation matrix relationship between the bounding boxes connected by the points, so that the randomly extracted feature points can be projected into the image corresponding to the four bounding boxes connected by the points. For the case where there are four bounding boxes connected by the points and the projected image of the feature points, the average value of the error between the positions of all projected image points corresponding to multiple feature points and the actual image point positions is calculated to minimize the feature points. For the case where there are no four bounding boxes connected by the points and the projected image of the feature points, all bounding boxes connected by the points are calculated (no more than four).

[0076] Furthermore, the registration between bounding boxes can also be adjusted based on the different scales of the same target in the image, and unified according to the relationship between the bounding box division areas and the scaling of the same target scale.

[0077] The present invention also provides a multi-perspective perception sparse three-dimensional model construction system, which is applied to the multi-perspective perception sparse three-dimensional model construction method described in any of the above embodiments. Since the principle of solving the problem by the multi-perspective perception sparse three-dimensional model construction system in the embodiment of the present application is similar to the multi-perspective perception sparse three-dimensional model construction method in the above embodiment of the present application, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be repeated.

[0078] like Figure 2 As shown, Figure 2 A functional module diagram of a multi-view perception sparse three-dimensional model construction system 100 provided by the present invention includes:

[0079] The image data extraction module 110 is used to collect target image data and extract time information, coordinate information and elevation information from the target image data;

[0080] A partitioning module 120 is used to divide the extracted target image data into regions based on the coordinate point information and the elevation information to obtain a global image set and a plurality of bounding boxes;

[0081] A bounding box processing module 130, for processing the registration of image data within a single bounding box;

[0082] An inter-bounding box processing module 140 is used to process the matching between neighboring bounding boxes;

[0083] The local matching model stitching module 150 is used to stitch the neighborhood bounding boxes to obtain an initial sparse three-dimensional model;

[0084] The sparse 3D model optimization module 160 is used to optimize the initial sparse 3D model to obtain a multi-view perception sparse 3D model.

[0085] In some possible implementations, the following further includes:

[0086] The feature point extraction module 170 is used to extract feature points in the local matching model and optimize the quality of the feature point set.

[0087] In summary, the present invention discloses a method and system for constructing a sparse three-dimensional model with multi-perspective perception, which can construct an initial sparse three-dimensional model based on a large number of visible light images that currently exist and are being generated. The visible light images can be acquired through fixed and mobile acquisition devices, and are not limited to the same type of acquisition devices.

[0088] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0089] The present application is described with reference to flowcharts and / or block diagrams of methods according to embodiments of the present application.

[0090] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A method for constructing a sparse three-dimensional model with multi-view perception, characterized in that: include: Collecting target image data, and extracting time information, coordinate information and elevation information from the target image data; Divide the extracted target image data into regions based on the coordinate information and the elevation information to obtain a global image set and a plurality of bounding boxes; Processing registration of image data within a single bounding box; Process the matching between the neighborhood bounding boxes and splice the neighborhood bounding boxes to obtain the initial sparse 3D model; Optimizing the obtained initial sparse three-dimensional model to obtain a multi-view perceived sparse three-dimensional model; Among them, processing the registration of image data within a single bounding box includes: For images taken at different times within the same single bounding box, a sequence of images under coordinate constraints is constructed according to the time sequence; Match the images of the same time series whose coordinates are in each bounding box to obtain the feature points of the image set and the three-dimensional space structure represented by the feature points; Process the matching between the neighborhood bounding boxes and splice the neighborhood bounding boxes to obtain the initial sparse 3D model, including: When the coordinate point is located on the boundary of the bounding box or on an adjacent surface, the image with the coordinate point located on the boundary is matched with the image with the closest coordinate position of the adjacent bounding box, so as to realize the splicing of two or more local matching models corresponding to the adjacent bounding boxes; When the coordinate point position is not on the boundary of the bounding box or the adjacent surface, when the bounding boxes are aligned, the coordinate point corresponding to the image closest to the coplanar surface is selected for matching to achieve the splicing of two or more local matching models corresponding to adjacent bounding boxes.

2. The method for constructing a sparse three-dimensional model with multi-view perception according to claim 1, characterized in that: The extracted target image data is divided into regions based on the coordinate information and the elevation information to obtain a global image set and a plurality of bounding boxes, including: Divide the target image data into regions from low to high based on the elevation range according to the image coordinate point coverage area; The image data corresponding to the coordinate discrete points outside the region are grouped as the global image set region; The remaining spatial regions not including the global image set are grouped into a square bounding box with a side length of 20m to 50m in a three-dimensional coordinate system.

3. The method for constructing a sparse three-dimensional model with multi-view perception according to claim 1, characterized in that: When matching images of the same time series whose coordinates are in each bounding box, the SIFT registration algorithm is used to obtain the matching relationship of feature points in the image.

4. The method for constructing a sparse three-dimensional model with multi-view perception according to claim 3, characterized in that: The steps of matching the image order by the SIFT registration algorithm include: In the same bounding box, a center point is obtained according to the distribution of coordinate points, a three-dimensional k-nearest coordinate point image of the center point is calculated, and the images are saved as a comprehensive image sorting; Perform SIFT registration on the first two three-dimensional k-nearest-neighbor coordinate point images, and use the center points of the coordinate points of the first two three-dimensional k-nearest-neighbor coordinate point images as new center points; Based on the comprehensive image sorting, the third closest coordinate point is determined, and the corresponding image is used as an increment to match the previous matching result until all images in the bounding box are matched, thereby obtaining a local matching model of the bounding box.

5. The method for constructing a sparse three-dimensional model with multi-view perception according to claim 1, characterized in that: After all the local matching models are spliced ​​together, a spliced ​​initial sparse three-dimensional model is obtained.

6. The method for constructing a sparse 3D model with multi-view perception according to claim 1, characterized in that When the local matching models are spliced, the local matching models are geometrically transformed, and the feature point set in the local matching model to be spliced ​​is subjected to homography matrix estimation based on the RANSAC algorithm to obtain the translation and rotation matrix in the geometric transformation.

7. A multi-view perception sparse three-dimensional model construction system, applied to the multi-view perception sparse three-dimensional model construction method according to any one of claims 1 to 6, characterized in that: include: An image data extraction module, used to collect target image data and extract time information, coordinate information and elevation information from the target image data; A partitioning module, used for partitioning the extracted target image data into regions based on the coordinate information and the elevation information to obtain a global image set and a plurality of bounding boxes; A bounding box processing module, used to process the registration of image data within a single bounding box; The bounding box processing module is used to process the matching between neighboring bounding boxes; The local matching model stitching module is used to stitch the neighborhood bounding boxes to obtain an initial sparse 3D model; A sparse three-dimensional model optimization module, used to optimize the obtained initial sparse three-dimensional model to obtain a sparse three-dimensional model with multi-view perception; Among them, processing the registration of image data within a single bounding box includes: For images taken at different times within the same single bounding box, a sequence of images under coordinate constraints is constructed according to the time sequence; Match the images of the same time series whose coordinates are in each bounding box to obtain the feature points of the image set and the three-dimensional space structure represented by the feature points; Process the matching between the neighborhood bounding boxes and splice the neighborhood bounding boxes to obtain the initial sparse 3D model, including: When the coordinate point is located on the boundary of the bounding box or on an adjacent surface, the image with the coordinate point located on the boundary is matched with the image with the closest coordinate position of the adjacent bounding box, so as to realize the splicing of two or more local matching models corresponding to the adjacent bounding boxes; When the coordinate point position is not on the boundary of the bounding box or the adjacent surface, when the bounding boxes are aligned, the coordinate point corresponding to the image closest to the coplanar surface is selected for matching to achieve the splicing of two or more local matching models corresponding to adjacent bounding boxes.

8. The multi-view perception sparse 3D model construction system according to claim 7, characterized in that: Also includes: The feature point extraction module is used to extract feature points in the local matching model and optimize the quality of the feature point set.

Images