High-fidelity modeling method

Through multi-camera acquisition of images and using SfM algorithm and GPS data optimization, the problem of low mesh model quality in the existing modeling methods is solved, and the efficiency improvement of the high-fidelity 3D model generation and modeling process is achieved.

CN118918280APending Publication Date: 2024-11-08SUZHOU BOTTLER ROBOT CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410924351.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-11
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The mesh models generated by existing building modeling methods are of low quality, low fidelity and slow modeling process, and no effective solution is available.

Method used

Multiple cameras are used to collect images, calculate camera poses and sparse point clouds, extract feature points to estimate the essence matrix, decompose to obtain the RT matrix, calculate the projection matrix and reconstruct the three-dimensional point cloud, generate data points through Gaussian distribution parameters, perform interleaving optimization and Render to get a high-fidelity 3D model.

Benefits of technology

High-fidelity 3D model generation is realized, which improves the efficiency and quality of the modeling process, is suitable for a variety of modeling situations, and improves the applicability of the modeling method through SfM algorithm and GPS data optimization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918280A_ABST
    Figure CN118918280A_ABST
Patent Text Reader

Abstract

The invention provides a high-fidelity modeling method, which comprises the following steps of: providing a plurality of cameras for acquiring a plurality of images; calculating a camera pose and a sparse point cloud, extracting feature points in a plurality of images, performing matching, and estimating an essential matrix according to the matched feature points; decomposing the essential matrix to obtain an RT matrix, and calculating a projection matrix corresponding to the image according to the participating RT matrix in the camera; reconstructing a three-dimensional point cloud according to the projection matrix and the matched feature points; the Gaussian point cloud is initialized through the three-dimensional point cloud, and parameters of Gaussian distribution are determined; generating data points through random numbers of multivariate normal distribution according to parameters of Gaussian distribution, and calculating Gaussian function values of the data points in a 3D space through a camera pose and an image; and performing staggered optimization, density control, rendering and comparison on the Gaussian function value to obtain a 3D model of the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application belongs to the technical field of modeling, and in particular, relates to a high-fidelity modeling method. Background Art

[0002] In the prior art, mainstream building modeling generally uses modeling software such as ContextCapture to generate mesh models.

[0003] In the above modeling process, the generated mesh model is of low quality and low fidelity, and the modeling process is slow.

[0004] To address the above problems, no effective solution has been proposed yet.

[0005] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present invention and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because these solutions are described in the background technology section of the present invention. Summary of the invention

[0006] The purpose of this application is to provide a high-fidelity modeling method to solve the above problems.

[0007] The present application provides a high-fidelity modeling method, comprising the following steps:

[0008] Providing multiple cameras for collecting multiple images;

[0009] Calculating camera pose and sparse point cloud, extracting feature points from a plurality of the images, and estimating an essential matrix based on the feature points;

[0010] Decomposing the essential matrix to obtain an RT matrix, and calculating a projection matrix corresponding to the image according to the RT matrix in the camera;

[0011] Reconstructing the sparse point cloud according to the projection matrix and the matching feature points to obtain a three-dimensional point cloud;

[0012] Initializing the Gaussian point cloud with the three-dimensional point cloud to determine the parameters of the Gaussian distribution;

[0013] Generate data points using random numbers of multivariate normal distribution according to the parameters of the Gaussian distribution, and calculate Gaussian function values ​​of the data points in 3D space using the camera pose and the image;

[0014] The Gaussian function values ​​in the 3D space are subjected to interlaced optimization and density control as well as rendering and comparison to obtain a 3D model of the image.

[0015] Preferably, the camera is arranged on a drone for collecting images.

[0016] Preferably, in the step of "calculating camera pose and sparse point cloud, and extracting feature points in the image", a Structure from Motion (SfM) algorithm is used for calculation and extraction.

[0017] Preferably, feature points are extracted from the plurality of images, and the feature points of the plurality of images are matched.

[0018] Preferably, in the step of "estimating the essential matrix", the essential matrix is ​​estimated by using a 5-point method or / and an 8-point method based on the feature points matched by the plurality of images.

[0019] Preferably, the parameters of the Gaussian distribution include a mean vector μ and a covariance matrix Σ.

[0020] Preferably, the point cloud coordinates of the image are modified according to GPS data when the camera is shooting, so as to adjust the 3D model.

[0021] Preferably, the adjusted 3D model is loaded into a corresponding geographical location of a GIS platform.

[0022] Preferably, the UAV has a control platform, and the planned route is extracted according to the control platform, and images are collected periodically.

[0023] Preferably, in "performing interlaced optimization and density control on the Gaussian function value", anisotropic covariance is optimized to accurately represent the scene.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention estimates the essential matrix according to the matching feature points, camera pose and sparse point cloud, decomposes the essential matrix to obtain the RT matrix, calculates the projection matrix corresponding to the image according to the RT matrix in the camera, and obtains a high-fidelity 3D model according to the projection matrix, the feature points and the 3D Gaussian modeling algorithm.

[0026] 1. The present invention adopts the Structure from Motion (SfM) algorithm for calculation and extraction. By using the SfM algorithm for calculation and extraction, feature points of multiple images can be stably extracted and feature points can be accurately matched.

[0027] 2. The present invention estimates the essential matrix based on feature points that match multiple images, and selects a 5-point method or an 8-point method according to time, computing resources, and costs, thereby improving the applicability of the modeling method of the present application.

[0028] 3. The present invention can effectively optimize the modeling process in the present application by changing the sparse point cloud at any time according to the GPS data when the camera is shooting, so that the modeling method of the present application can be applied to modeling in various situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 It is a flow chart of a high-fidelity modeling method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0032] In the description of the present invention, it should be noted that the orientations or positional relationships indicated by the terms "upper", "middle", "lower", "inner", "outer", "front", "back", etc. are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or component referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. The terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The following describes an implementation method of the present invention based on its overall structure.

[0033] Reference Figure 1 As shown, the embodiment of the present application provides a high-fidelity modeling method, the following steps:

[0034] Providing multiple cameras for capturing images at multiple angles;

[0035] Calculate the camera pose and sparse point cloud, extract feature points from multiple images, and estimate the essential matrix based on the feature points;

[0036] Decompose the essential matrix to obtain the RT matrix, and calculate the projection matrix corresponding to the image based on the RT matrix in the camera;

[0037] Reconstruct the sparse point cloud according to the projection matrix and the matching feature points to obtain a three-dimensional point cloud;

[0038] Initialize the Gaussian point cloud with the three-dimensional point cloud to determine the parameters of the Gaussian distribution;

[0039] Generate data points using random numbers from a multivariate normal distribution according to the parameters of the Gaussian distribution, and calculate the Gaussian function value of the data points in 3D space using the camera pose and image.

[0040] Gaussian function values ​​in 3D space are subjected to interlaced optimization and density control as well as rendering and comparison to obtain a 3D model of the image.

[0041] In order to perform high-quality modeling, those skilled in the art can achieve it through the above process. Specifically, those skilled in the art can set up multiple cameras to collect images. Preferably, the camera is set on a drone to collect images. That is, the camera can be set on the drone. By controlling the drone, the camera can be controlled to shoot.

[0042] Furthermore, after the multiple cameras have captured multiple images from multiple angles, the camera poses and sparse point clouds may be calculated. Preferably, feature points are extracted from the multiple images, and the feature points of the multiple images are matched.

[0043] That is, those skilled in the art can extract feature points from the collected multiple images, match the feature points extracted from the multiple images, and estimate the essential matrix based on the feature points that can be matched in the multiple images.

[0044] Preferably, in the step of "calculating camera pose and sparse point cloud, and extracting feature points in the image", the Structure from Motion (SfM) algorithm is used for calculation and extraction. By using the SfM algorithm for calculation and extraction, feature points of multiple images can be stably extracted and feature points can be accurately matched.

[0045] More preferably, in the step of "estimating the essential matrix", the essential matrix is ​​estimated by using a 5-point method or / and an 8-point method based on feature points that match a plurality of images.

[0046] It is understandable that the 5-point method requires at least five matching point pairs to calculate the essential matrix. Since it only requires fewer point pairs, it can accurately estimate the essential matrix when computing resources are limited or fast computing is required.

[0047] The 8-point method requires at least eight matching point pairs to calculate the essential matrix. In more precise cases, RANSAC (Random Sample Consensus) is usually used to exclude outliers to improve accuracy.

[0048] Through the above process, those skilled in the art can choose to use the 5-point method or the 8-point method to estimate the essential matrix according to time, computing resources and cost, thereby improving the applicability of the modeling method of the present application.

[0049] After obtaining the essential matrix, those skilled in the art can decompose the essential matrix. After decomposing the essential matrix, the R (rotation) T (displacement) matrix can be obtained, that is, the RT matrix can be obtained. According to the above implementation, those skilled in the art can combine the RT matrix with the intrinsic parameters of the camera and perform calculations to obtain the projection matrix corresponding to the multiple images. Then, based on the projection matrix and the above matching feature points, the triangulation method can be used to reconstruct the image.

[0050] In the above process, the technicians in this field combine the RT matrix with the intrinsic parameters of the camera to calculate the projection matrix corresponding to the multiple images. Specifically, assuming that two cameras are provided, and the relative camera pose between the two cameras has been calculated in the above process. Moreover, both cameras capture the same 3D point, we can use the following formula to perform triangulation:

[0051] Suppose the projection matrix of camera 1 is P1, the projection matrix of camera 2 is P2, the coordinate of the 3D point is X, and its projection on camera 1 and camera 2 is x1 and x2 respectively, then: x1 = P1*X x2 = P2*X. It is worth noting that x1 and x2 are the projection coordinates of the 3D point in image 1 and image 2 captured by the two cameras, respectively, and P1 and P2 are the projection matrices of the two cameras.

[0052] Specifically, the projection matrix P is composed of the camera's intrinsic parameter matrix (describing the camera's focal length, principal point, etc.) and the extrinsic parameter matrix (describing the camera's position and posture). That is, with known camera intrinsic parameters and camera posture, those skilled in the art can construct the projection matrix.

[0053] When multiple corresponding points are collected, the least square method can be used to solve the optimal three-dimensional point coordinate X. Then, the projection matrix of the image collected by the camera and the reconstruction of the three-dimensional point cloud can be realized.

[0054] In the above process, a person skilled in the art can obtain a reconstructed three-dimensional point cloud. Furthermore, the Gaussian point cloud can be initialized by the three-dimensional point cloud, and then the parameters of the Gaussian distribution can be determined. Preferably, the parameters of the Gaussian distribution include a mean vector μ and a covariance matrix Σ.

[0055] Data points are generated by random numbers of multivariate normal distribution according to the parameters of the Gaussian distribution, and Gaussian function values ​​of the data points in 3D space are calculated by the camera posture and the image; the Gaussian function values ​​are interlaced optimized and density controlled to obtain a 3D model of the image.

[0056] In the above process of interleaving optimization and density control of the Gaussian function values, preferably, the anisotropic covariance can be optimized to accurately represent the scene. Further, the result map can be rendered and compared with the training view.

[0057] That is, after obtaining the Gaussian function value of the data point in 3D space, the Gaussian function value can be iteratively optimized. Specifically, iterative optimization can perform staggered optimization and density control, especially optimizing anisotropic covariance to accurately represent the scene. In addition, the parameters of the 3D Gaussian can be adjusted based on continuous iterative rendering and comparison of the result image with the training view. Then, the optimization of the 3D model can be completed.

[0058] Preferably, the point cloud coordinates of the image are modified according to the GPS data when the camera is shooting, so as to adjust the 3D model. Specifically, during the image acquisition process, the drone will record the coordinates when the image is acquired. The coordinates when the image is acquired and the coordinates calculated by the SfM algorithm are used to calculate the transformation matrix E. The transformed point cloud data can be obtained by multiplying the position parameters (XY Z) in the Gaussian point cloud by the transformation matrix E.

[0059] That is, the sparse point cloud can be modified at any time according to the GPS data when the camera is shooting. Therefore, the modeling process in the present application can be effectively optimized, so that the modeling method of the present application can be applied to modeling in various situations.

[0060] Preferably, the adjusted 3D model is loaded into the corresponding geographical location of the GIS platform, so that the 3D model can be effectively displayed.

[0061] In the above process, those skilled in the art can set up the collection of images of the drone carrying the camera entering the factory. Preferably, the drone has a control platform, and the control platform extracts the planned route and periodically collects images. This can realize the periodic automatic modeling and information update of the modeling object.

[0062] Although different specific embodiments are mentioned in the content of this application, this application is not limited to the situations described in the industry standards or embodiments, etc. Some industry standards or slightly modified implementation plans based on the implementation of the custom methods or embodiments can also achieve the same, equivalent or similar, or predictable implementation effects after deformation of the above embodiments. The embodiments of data acquisition, processing, output, judgment methods, etc. after these modifications or deformations can still fall within the scope of the optional implementation plans of this application.

[0063] Although the present application has been described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present application without departing from the spirit of the present application. It is intended that the attached embodiments include these modifications and changes without departing from the present application.

Claims

1. A high-fidelity modeling method, characterized in that: The following steps are involved: Providing multiple cameras for capturing images at multiple angles; Calculating the camera pose and sparse point cloud, extracting and matching feature points in a plurality of the images, and estimating the intrinsic matrix based on the matched feature points; Decomposing the essential matrix to obtain an RT matrix, and calculating a projection matrix corresponding to the image according to the RT matrix in the camera; Reconstructing the sparse point cloud according to the projection matrix and the matching feature points to obtain a three-dimensional point cloud; Initializing the Gaussian point cloud with the three-dimensional point cloud to determine the parameters of the Gaussian distribution; Generate data points using random numbers of multivariate normal distribution according to the parameters of the Gaussian distribution, and calculate Gaussian function values ​​of the data points in 3D space using the camera pose and the image; The Gaussian function values ​​in the 3D space are subjected to interlaced optimization and density control as well as rendering and comparison to obtain a 3D model of the image.

2. The high-fidelity modeling method according to claim 1, characterized in that: The camera is arranged on the UAV and is used for collecting images.

3. The high-fidelity modeling method according to claim 1, characterized in that: In the step of "calculating camera pose and sparse point cloud, and extracting feature points in the image", the Structure from Motion (SfM) algorithm is used for calculation and extraction.

4. The high-fidelity modeling method according to claim 1, characterized in that: Feature points are extracted from the plurality of images, and the feature points of the plurality of images are matched.

5. The high-fidelity modeling method according to claim 4, characterized in that: In the step of “estimating the essential matrix”, the essential matrix is ​​estimated by using a 5-point method or / and an 8-point method based on the feature points matched by the plurality of images.

6. The high-fidelity modeling method according to claim 1, characterized in that: The parameters of the Gaussian distribution include a mean vector μ and a covariance matrix Σ.

7. The high-fidelity modeling method according to claim 1, characterized in that: The point cloud coordinates of the image are modified according to the GPS data when the camera is shooting, so as to adjust the 3D model.

8. The high-fidelity modeling method according to claim 7, characterized in that: The adjusted 3D model is loaded into the corresponding geographic location of the GIS platform.

9. The high-fidelity modeling method according to claim 2, characterized in that: The unmanned aerial vehicle has a control platform, and a planned route is extracted according to the control platform, and images are collected periodically.

10. The high-fidelity modeling method according to claim 1, characterized in that: In "interleaved optimization and density control of the Gaussian function value", the anisotropic covariance is optimized to accurately represent the scene.