Image-based non-rigid body target three-dimensional model inversion method
The image-based method addresses the challenge of non-rigid 3D reconstruction by calibrating cameras and using advanced algorithms to stitch images, resulting in accurate geometric modeling of non-rigid targets.
Patent Information
- Application Number
- CN202411138365.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-12-27
AI Technical Summary
Existing 3D reconstruction methods struggle with non-rigid bodies due to their complex movements and structural changes, making accurate reconstruction difficult.
An image-based method using multiple aerial cameras with calibrated internal and external parameters, synchronized image capture, and advanced algorithms like SFM and MVS to generate and encrypt point clouds, then map color and texture features for non-rigid target modeling.
Enables accurate reconstruction of non-rigid targets by calibrating camera parameters, identifying ground targets, and stitching images to form a three-dimensional model with geometric parameters, providing a basis for non-rigid object research.
Smart Images
Figure CN119206127A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of modeling, and in particular relates to an image-based three-dimensional model inversion method for non-rigid targets. Background Art
[0002] Since rigid body motion is simple, early 3D reconstruction research objects were all based on rigid body motion. After years of development, rigid body reconstruction research has become mature. However, most movements in real life are flexible and belong to non-rigid body motion. During the movement of non-rigid bodies, the structure will change, and its movement is more complex than that of rigid bodies, so the reconstruction will be more difficult. Summary of the invention
[0003] In order to overcome the shortcomings of the prior art, the present invention provides an image-based three-dimensional model inversion method for non-rigid targets, which uses multiple aerial cameras with accurately calibrated internal reference files to shoot ground positioning cooperative targets from different angles to complete the external parameter accurate calibration of aerial cameras, and continuously collects images of non-rigid targets based on the principle of aerial intersection measurement, and obtains non-rigid target images at different angles at the same time through a synchronous trigger device; completes image matching and splicing through multiple target features such as non-rigid targets and ground control points, and then obtains three-dimensional point cloud data, and combines the color and texture attributes of the image itself to map the point cloud data, and then obtains three-dimensional model data with texture and color. Finally, the frame-by-frame model can be associated with the time axis to form an inversion of the three-dimensional transformation process of the non-rigid target, thereby realizing the measurement of geometric parameters such as size, volume, and shape of the non-rigid target.
[0004] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0005] Step 1: Calibration of the intrinsic parameters of the optical measurement equipment;
[0006] Step 2: Calibration of external parameters of optical measurement equipment;
[0007] Step 3: Automatic identification of ground cooperative targets;
[0008] Step 4: spatial intersection solution;
[0009] Step 5: 3D model restoration;
[0010] The intersecting image feature points are used as control points for solution to achieve matching, orientation and stitching between multi-angle images, the image coordinate system is converted into the object coordinate system, each pixel is converted into spatial point cloud data, the SFM algorithm is used to generate a sparse point cloud, the MVS algorithm is used to encrypt the point cloud, and then the similar point clouds are encapsulated into triangular faces; mapping is performed based on the color and texture features of the image itself, and finally a three-dimensional model of each frame of the image is obtained.
[0011] Preferably, the device internal parameters include the principal distance f and the image principal point coordinates X 0 , Y 0 .
[0012] Preferably, in step 2, the external parameters of the optical measurement device are calculated based on the principle of resection using a cooperative target with known coordinates.
[0013] Preferably, in step 3, a coded cooperative target is used to achieve automatic recognition and invariance to rotation, translation and scale.
[0014] Preferably, in step 4, forward intersection is performed based on known external parameter coefficients of the measuring stations, and the coordinate values Xs, Ys, and Zs of the feature points of the forward target are solved by utilizing the baseline distance between the measuring stations, the angles of their respective intersections with the forward target, the principles of forward intersection and beam adjustment.
[0015] Preferably, the step 5 is specifically:
[0016] (1) Image feature point detection and matching; (2) Spatially oriented camera model restoration; 3) SFM algorithm generates sparse point cloud; (4) MVS algorithm encrypts point cloud; (5) Reconstructs point cloud into a mesh model.
[0017] Step 5-1: Image feature point detection and matching;
[0018] Use SIFT or SURF feature point detection operators to detect points in the image that are different from the surrounding pixels, and then store them in a list;
[0019] By matching the feature points stored in the two images according to a certain algorithm, it is possible to determine whether the points on different images are the same point;
[0020] Step 5-2: spatial orientation restoration of camera model;
[0021] a) Establishment of camera model;
[0022] Calibrate the camera parameters and calculate the pixel position and camera posture based on the image; the camera adopts the pinhole camera model, and the camera imaging includes four coordinate systems: pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system;
[0023] b) spatial orientation;
[0024] Through direct linear transformation or robust method based on RANSAC, epipolar geometry describes the geometric relationship between two images of the same scene, matching points are obtained through feature matching, and the basic matrix is derived according to the epipolar constraints;
[0025] Step 5-3: SFM method generates sparse point cloud;
[0026] Step 5-4: MVS algorithm encrypts point cloud;
[0027] Step 5-5: Encapsulate as a mesh model.
[0028] Preferably, the SFM method adopts one of the following three methods:
[0029] a) Triangulation: Each point in the image corresponds to a spatial projection. The intersection of the spatial projections of the same point in different images is the coordinate point of the point in 3D space. The coordinates of the matching points in 3D space are restored by triangulation using camera parameters and matching points.
[0030] b) PnP problem - direct linear transformation method: Given 3D points and corresponding 2D points, solving the camera's internal and external parameters requires at least 6 pairs of 3D-2D corresponding points;
[0031] c) Bundle adjustment: Nonlinear optimization of the 3D point cloud position and camera parameters is performed simultaneously. The position and camera attitude of several images are calculated first, and then new images are added and optimized until all images are optimized.
[0032] Preferably, the MVS algorithm includes: a voxel-based method, a spatial patch-based method, and a depth map fusion-based method.
[0033] The beneficial effects of the present invention are as follows:
[0034] The method of the present invention is to perform internal parameter calibration, external parameter solution, image enhancement, texture feature matching, three-dimensional model restoration, etc. on multi-eye images of non-rigid target expansion process testing, so as to provide a measurement basis for non-rigid object research. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a schematic diagram of the method of the present invention.
[0036] Figure numerals: 1-high-speed camera; 2-UAV platform; 3-test site; 4-ground positioning cooperation target; 5-non-rigid target; 6-UAV wireless signal trigger terminal. DETAILED DESCRIPTION
[0037] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0038] The technical solution of the present invention is:
[0039] 1) Accurate calibration of the internal parameters of optical measurement equipment.
[0040] The accurate calibration of the intrinsic parameters of optical measurement equipment is the basis of spatial multi-eye intersection. The accurate calibration of the intrinsic parameters of the camera effectively solves the problems of image distortion, the spatial mapping conversion relationship between the image space and the object space, etc.
[0041] 2) Accurate calibration of external parameters of optical measurement equipment.
[0042] Most conventional camera external parameters use GPS or RTK combined with IMU (code disk) to achieve their own positioning posture. This method is effective for ground optical measurement equipment. For aerial cameras, the spatial positioning posture accuracy is low (affected by load and hovering). The use of cooperative targets with known coordinates to calculate the external parameters of optical measurement equipment based on the principle of rear intersection can meet the function of real-time external parameter calculation, and the external parameters are relatively accurate.
[0043] 3) Automatic identification of ground cooperative targets.
[0044] The deployment site is mostly an outdoor environment, the arrangement of targets is quite random, and the cooperative targets seen by multiple stations and multiple postures are different. In order to effectively improve false recognition and high matching items, the use of coded cooperative targets can effectively achieve automatic recognition and invariance to rotation, translation and scale.
[0045] 4) Spatial intersection solution.
[0046] Spatial intersection is mainly based on the known external parameter coefficients of the measuring station for forward intersection. By using the baseline distance between the measuring stations and the angle of their respective intersection with the target in front, and the principle of forward intersection and beam adjustment, the coordinate values Xs, Ys, and Zs of the feature points of the target in front can be solved.
[0047] 5) Three-dimensional model restoration.
[0048] The intersecting image feature points are used as control points for solution to achieve matching, orientation and stitching between multi-angle images, the image coordinate system is converted into the object coordinate system, each pixel is converted into spatial point cloud data, the SFM (Structure from motion) algorithm is used to generate a sparse point cloud, the MVS (Multi view Stereo) algorithm is used to encrypt the point cloud, and then the similar point clouds are encapsulated into triangular faces; mapping is performed based on the color and texture features of the image itself, and finally a three-dimensional model of each frame of the image is obtained.
[0049] The functions of the various parts of the present invention are as follows:
[0050] 1) UAV platform: used to carry high-speed cameras. The UAV platform includes a gimbal and a control system to adjust the shooting position and angle of the high-speed camera in the air.
[0051] 2) High-speed camera: responsible for capturing the changing process of non-rigid targets in the test field, caching the data and transmitting it to the ground control terminal.
[0052] 3) Wireless communication and control module: transmits the signal transmitted by the wireless signal end to the control system via optical fiber signal, and receives and converts wireless signals.
[0053] 4) Camera calibration: Determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image by establishing a geometric model of camera imaging.
[0054] 5) 3D model restoration: By taking pictures of a scene from different perspectives and using the principle of forward intersection, a 3D mesh model is produced through image preprocessing, feature point enhancement, image matching and stitching, spatial orientation, SFM algorithm to generate sparse point cloud, MVS algorithm to encrypt point cloud, and point cloud reconstruction into a mesh model.
[0055] Example:
[0056] An image-based three-dimensional model inversion method for non-rigid targets mainly includes the calibration and solution of internal and external parameters of image sensors, extraction, matching, and splicing of image features, conversion and unification between image and object coordinate systems, and restoration algorithm of three-dimensional models.
[0057] Before the experiment begins, the internal parameters of the high-speed camera are calibrated, mainly including the principal distance f and the image principal point coordinate X 0 , Y 0 ; Secondly, multiple coded markers are arranged around the maximum diffusion range of the non-rigid body as object control points, and RTK is used to obtain the object coordinates of the control points; then the drone is mounted with a high-speed camera and hovered to the predetermined position according to the design idea and adjusted to the optimal angle and field of view. Some time before the non-rigid body changes, the synchronous trigger device controls multiple cameras to enter the same-frequency image acquisition state.
[0058] At the beginning of the test, the image is captured until the end of the test, data collection is completed, and the image is downloaded to the terminal. Finally, the application software completes the data processing and model synthesis; the restoration of the three-dimensional model is the key point of this method, which is emphasized here.
[0059] The main technical contents involved in the 3D model restoration algorithm include: (1) image feature point detection and matching; (2) spatial orientation restoration of camera model; 3) SFM algorithm to generate sparse point cloud; (4) MVS algorithm to encrypt point cloud; (5) reconstruction of point cloud into mesh model.
[0060] (1) Image feature point detection and matching
[0061] Feature point detection is the first step of 3D reconstruction. The points in the image that are different from the surrounding pixels are detected by SIFT or SURF feature point detection operators, and then stored in a list; the matching principle is to match the feature points stored in the two images according to a certain algorithm to determine whether the points on different images are the same point. Two commonly used concepts: scale invariance (the same feature point can be detected at different scales) and rotation invariance (the feature point can still be correctly matched after rotation).
[0062] (2) Spatial directional restoration camera model
[0063] a) Camera model establishment
[0064] The expression of the camera is the basis of 3D reconstruction. The camera parameters must be calibrated to calculate the position of the pixel points and the camera posture according to the image. Usually the camera adopts the pinhole camera model, and the camera imaging has four coordinate systems (pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system).
[0065] b) Spatial orientation
[0066] Through direct linear transformation method (eight-point method, least squares method) or robust method based on RANSAC, epipolar geometry describes the geometric relationship between two images of the same scene, matching points are obtained through feature matching, and the fundamental matrix (F) is derived according to the epipolar constraints.
[0067] (3) SFM method to generate sparse point cloud
[0068] SFM, also known as structure from motion, is a commonly used sparse reconstruction method that restores three-dimensional scenes through collected unordered images.
[0069] a) Triangulation: Each point in a photo corresponds to a spatial projection. The intersection of the spatial projections of the same point in different photos is the coordinate point of the point in 3D space. The coordinates of the matching points in 3D space are restored by triangulation using camera parameters and matching points.
[0070] b) PnP problem - direct linear transformation method: Given a 3D point and its corresponding 2D point, at least 6 pairs of 3D-2D corresponding points are required to solve the camera's internal and external parameters.
[0071] c) Bundle Adjustment: A method for nonlinear optimization of 3D point cloud positions and camera parameters simultaneously (first calculate the positions and camera attitudes of several images, then add new images and optimize until all images are optimized). The main optimization methods for bundle adjustment are the steepest descent method, Newton's method, and LM (Levenberg-Marquardt) method (combining Newton's method with the steepest descent method), which have the best effect.
[0072] (4) MVS algorithm encrypted point cloud
[0073] Since the restored 3D point cloud data is too complex, the MVS algorithm is used to encrypt the point cloud data. Commonly used methods in the MVS algorithm include: voxel-based method, spatial patch method (PMVS), and depth map fusion method.
[0074] (5) Encapsulation as a grid model
[0075] There are two main methods for representing three-dimensional grid models: boundary representation method and space division method.
[0076] 1. Boundary representation: describing a three-dimensional object as a set of surfaces
[0077] a. Polyhedron representation method, which represents the surface of an object as a set of polygons that close the object space, among which triangles and quadrilaterals are the most commonly used.
[0078] b. Surface representation: The boundary is represented as a set of parametric or non-parametric surfaces.
[0079] c. Parametric surface: z = f(x, y), used to represent regular objects in the scene.
[0080] d. Implicit surface: Through the zero level set of the function f(x, y, z), it can be used to describe more complex objects.
[0081] 2. Space division method: Divide the space inside the object into small, non-overlapping continuous entities to describe the shape of the object.
[0082] a. Voxel construction method: Generate new objects by performing combination operations such as addition, subtraction, union and intersection on some elements
[0083] b. Octree method: Use a hierarchical tree structure to construct a tree structure for the object to be represented. The tree nodes correspond to a specific area in the space. Starting from the root node, the node containing the object is evenly divided into eight child nodes until the iteration meets the termination condition.
[0084] c. Binary space method: Similar to the octree method, both methods divide the space gradually, but the binary method divides the space into two parts, and the space distribution can be adjusted at any time.
[0085] The most common three-dimensional networks are triangular meshes, whose basic structure is vertices, faces, and edges.
[0086] In summary, the present invention provides an image-based three-dimensional model inversion method for non-rigid targets, which can have the functions of internal parameter calibration and external parameter solution of aerial optical measurement equipment, automatically identify ground cooperative targets, import and match the coordinates of ground control points (cooperative targets), and then synthesize aerial multi-eye images into a three-dimensional model. The model data is fitted with color, texture, and standard shape, and the model generation format is universal, and the chromatographic deviation relative to the model is given. It provides a basis for the establishment, analysis, editing, and measurement of relevant dimensional information of non-rigid target three-dimensional models.
Claims
1. An image-based three-dimensional model inversion method for non-rigid targets, characterized in that: The steps include: Step 1: Calibration of the internal parameters of the optical measurement equipment; Step 2: Calibration of external parameters of optical measurement equipment; Step 3: Automatic identification of ground cooperative targets; Step 4: spatial intersection solution; Step 5: 3D model restoration; The intersecting image feature points are used as control points for solution to achieve matching, orientation and stitching between multi-angle images, the image coordinate system is converted into the object coordinate system, each pixel is converted into spatial point cloud data, the SFM algorithm is used to generate a sparse point cloud, the MVS algorithm is used to encrypt the point cloud, and then the similar point clouds are encapsulated into triangular faces; mapping is performed based on the color and texture features of the image itself, and finally a three-dimensional model of each frame of the image is obtained.
2. The image-based non-rigid target three-dimensional model inversion method according to claim 1, characterized in that: The device internal parameters include the principal distance f and the image principal point coordinates X0 and Y0.
3. The image-based non-rigid target three-dimensional model inversion method according to claim 2, characterized in that: In step 2, the external parameters of the optical measurement device are calculated based on the principle of resection using a cooperative target with known coordinates.
4. The image-based non-rigid target three-dimensional model inversion method according to claim 3, characterized in that: In step 3, a coding cooperative target is used to achieve automatic recognition and invariance of rotation, translation and scale.
5. The image-based non-rigid target three-dimensional model inversion method according to claim 4, characterized in that: In step 4, forward intersection is performed based on known external parameter coefficients of the measuring stations, and the coordinate values Xs, Ys, and Zs of the feature points of the forward target are solved by using the baseline distance between the measuring stations, the angles of their respective intersections with the forward target, and the principles of forward intersection and beam adjustment.
6. The image-based non-rigid target three-dimensional model inversion method according to claim 5, characterized in that: The step 5 is specifically as follows: Step 5-1: Image feature point detection and matching; Use SIFT or SURF feature point detection operators to detect points in the image that are different from the surrounding pixels, and then store them in a list; By matching the feature points stored in the two images according to a certain algorithm, it is possible to determine whether the points on different images are the same point; Step 5-2: spatial orientation restoration of camera model; a) Establishment of camera model; Calibrate the camera parameters and calculate the pixel position and camera posture based on the image; the camera adopts the pinhole camera model, and the camera imaging includes four coordinate systems: pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system; b) spatial orientation; Through direct linear transformation or robust method based on RANSAC, epipolar geometry describes the geometric relationship between two images of the same scene, matching points are obtained through feature matching, and the basic matrix is derived according to the epipolar constraints; Step 5-3: SFM method generates sparse point cloud; Step 5-4: MVS algorithm encrypts point cloud; Step 5-5: Encapsulate as a mesh model.
7. The image-based non-rigid target three-dimensional model inversion method according to claim 6, characterized in that: The SFM method uses one of the following three methods: a) Triangulation: Each point in the image corresponds to a spatial projection. The intersection of the spatial projections of the same point in different images is the coordinate point of the point in 3D space. The coordinates of the matching points in 3D space are restored by triangulation using camera parameters and matching points. b) PnP problem - direct linear transformation method: to solve the internal and external parameters of the camera with known 3D points and corresponding 2D points, at least 6 pairs of 3D-2D corresponding points are required; c) Bundle adjustment: Nonlinear optimization of the 3D point cloud position and camera parameters is performed simultaneously. The position and camera attitude of several images are calculated first, and then new images are added and optimized until all images are optimized.
8. The image-based non-rigid target three-dimensional model inversion method according to claim 6, characterized in that: The MVS algorithm includes: a voxel-based method, a spatial patch-based method, and a depth map fusion method.