A three-dimensional image matching optimization method, medium and system

Abstract

The application discloses a three-dimensional image matching optimization method, medium and system, the method comprising: acquiring a three-dimensional image frame, and performing pose optimization on the three-dimensional image frame to obtain a motion trajectory graph; estimating an absolute pose based on the motion trajectory graph; measuring a relative pose through a depth sensor; constructing a target optimization function based on the relative pose and the absolute pose; and fusing a preset penalty factor into the target optimization function, performing inter-frame matching optimization on the image, and performing fusion network construction. By adding a penalty factor to the optimization function, inter-frame optimization is performed on the three-dimensional image, cumulative errors are eliminated, and in the case of not increasing additional calculation cost, different constraint pairs can be well checked and screened, the accuracy and stability of the pose optimization are ensured, and fast and accurate optimization is realized.

Description

[0001] This application is a divisional application of application number 202110625611.5, the parent application which was filed on June 4, 2021, and is entitled "An Online Matching Optimization Method and a 3D Scanning System Combining Geometry and Texture". Technical Field

[0002] This invention belongs to the field of image recognition, and more specifically, relates to a three-dimensional image matching optimization method, medium, and system. Background Technology

[0003] In recent years, 3D scanning, as a rapid 3D digitization technology, has been increasingly used in various fields, including reverse engineering, industrial inspection, computer vision, CG production, etc. Especially in the rapidly developing fields of 3D printing and intelligent manufacturing, 3D scanning, as a front-end 3D digitization and 3D vision sensing technology, has become an important link in the industrial chain. At the same time, various applications have put forward higher requirements for 3D scanning in terms of cost, practicality, accuracy and reliability.

[0004] Traditional 3D scanning optimization methods optimize all frames together, resulting in a large data processing volume and low optimization efficiency. Furthermore, traditional methods lack effective penalty mechanisms, failing to adequately filter out incorrect matches or results with low confidence levels. Summary of the Invention

[0005] This invention provides a three-dimensional image matching optimization method and a three-dimensional scanning system that can improve scanning efficiency.

[0006] The technical solution adopted by this invention to solve its technical problem is: to provide an online matching optimization method combining geometry and texture, the method comprising:

[0007] Acquire three-dimensional image frames and perform pose optimization on the three-dimensional image frames to obtain a motion trajectory map;

[0008] The absolute pose is estimated based on the motion trajectory diagram.

[0009] Relative pose obtained by depth sensor measurement;

[0010] Construct a target optimization function based on the relative pose and the absolute pose;

[0011] A preset penalty factor is incorporated into the target optimization function to perform inter-frame matching optimization on the image and then perform fusion mesh construction.

[0012] Furthermore, incorporating the preset penalty factor into the objective optimization function is as follows:

[0013] E2=∑i,j ρ(e 2 (p i ,p j ;∑ i,j ,T i,j ));

[0014] Here, the estimated absolute pose is taken as a node, p i Representing nodes i and p j Represents node j; T i,j Represents the relative pose between node i and node j, ∑ i,j This indicates that all constraint pairs are summed; e 2 (p i ,p j ;∑ i,j ,T i,j )=e(p i ,p j ;T i,j ) T ∑ i,j -1 e(p i ,p j ;T i,j ), e(p i ,p j ;T i,j ) = T i,j -p i -1 p j ρ is the penalty factor for integration.

[0015] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0016] The present invention also provides a three-dimensional scanning system for an online matching optimization method, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0017] The three-dimensional image matching optimization method, medium, and system of this invention optimize the three-dimensional image frame by adding a penalty factor to the objective function, thereby eliminating accumulated errors. Without increasing additional computational costs, it can perform good verification and screening of different constraint pairs, ensuring the accuracy and stability of pose optimization and achieving fast and accurate optimization. Attached Figure Description

[0018] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0019] Figure 1This is a flowchart of a three-dimensional image matching optimization method according to one embodiment of the present invention;

[0020] Figure 2 This is a typical optical path diagram of a 3D scanning system;

[0021] Figure 3 This is a schematic diagram showing the process details of online matching optimization combining geometry and texture in one embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the effect after the network is integrated. Detailed Implementation

[0023] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0024] Please refer to Figure 1 This is a flowchart of a three-dimensional image matching optimization method according to one embodiment of the present invention, the method comprising:

[0025] S1. Obtain a one-to-one corresponding depth-texture image pair, wherein the depth-texture image pair includes a depth image acquired by a depth sensor and a texture image acquired by a camera device.

[0026] For details, please refer to Figure 2 This diagram illustrates a typical optical path of a 3D scanning system. Two optical paths exist: Beam A is a structured light, which, after penetrating a specific coded pattern as white light, is projected onto the object being measured. Beam B is a texture illumination light, which, also as white light, is directly projected onto the object. Simultaneously with beam B's projection, the camera activates its image capture function, with its exposure time strictly synchronized with the beam projection time pulse. It should be noted that while beam A is being projected once, the camera also captures a single image of the object projected by beam A. Immediately afterward, beam B is projected, and the camera captures a single image of the object projected by beam B. This constitutes a single cycle of the measurement process. By repeatedly performing this process at a certain repetition frequency, while the relative positions and angles of the 3D scanning device and the object being measured continuously change, continuous measurement of the object's 3D scanning structure can be achieved.

[0027] Optionally, in one embodiment, the aforementioned three-dimensional scanning device is applied in a continuous rapid measurement mode. In this mode, beams A and B are projected alternately to complete the measurement of the object being measured. The beams emitted by the three-dimensional scanning device are output in the form of high-power short pulses, which provides a good foundation for subsequent high-precision measurements. It should be noted that in this embodiment, the instantaneous power of beam A can reach the kilowatt level, with a pulse width in the hundreds of microseconds range; the instantaneous power of beam B is in the hundreds of watts range, with a pulse width in the hundreds of microseconds range; the time difference between beams A and B and the camera exposure time for both are in the hundreds of microseconds range.

[0028] S2. Using a progressive refinement strategy, feature matching is performed between the depth texture image corresponding to the current frame and the depth texture image corresponding to the sample frame to estimate the initial pose of the depth sensor.

[0029] Specifically, in step S2, estimating the preliminary pose of the depth sensor includes:

[0030] S21. For each image frame that needs to be matched in the depth texture image, obtain a sample frame that is compatible with the image frame.

[0031] S22. For each of the image frames and sample frames, extract the corresponding image feature data, and perform image feature matching between the image frames and the corresponding sample frames to obtain multiple initial feature pairs;

[0032] S23. Select an initial transformation matrix from the plurality of initial feature pairs, and estimate the preliminary pose of the depth sensor based on the initial transformation matrix.

[0033] Specifically, this application considers extracting SIFT features from captured RGB images and performing feature matching between the current frame and sample frames based on the extracted SIFT features. It should be noted that SIFT is a widely used feature detector and descriptor, significantly superior to other features in terms of the detail and stability of feature point description. During the SIFT matching process, features are extracted from image frames F... i Find the nearest neighbor to obtain frame F j The best candidate match for each keypoint. This brute-force matching method can obtain frame F. j With frame F iThe initial N feature pairs between the two are represented by vectors (U;V). These feature pairs contain both correct data (Inliers) and outliers. In order to filter out the correct data from these matched feature pairs, this application uses the RANSAC algorithm to filter the effective sample data from the sample dataset containing outliers. The idea of ​​the RANSAC algorithm is: randomly select a set of RANSAC samples from N and calculate the transformation matrix (r;t). Based on (r;t), calculate the number of consistent points that satisfy the preset error metric function (see formula (1) below), that is, the number of inliers f, as shown in formula (2) below. This process is repeated cyclically to obtain the consistent set with the largest f. Then, the optimal transformation matrix is ​​calculated from the consistent set.

[0034]

[0035]

[0036] Among them, I(U i V i (,r,t) represents the i-th matching point pair (U i V i If the preset threshold values ​​d and θ can be satisfied under the current constraints (r; t), then I = 1; otherwise, I = 0. Pi N Qi Representing three-dimensional point P respectively i Q i The unit normal vector. N is the total number of matched point pairs. f(r,t) is the number of interior points.

[0037] S3. Combine geometric constraints and texture constraints to optimize the preliminary pose estimated in step S2 to obtain the refined inter-frame motion estimate.

[0038] Specifically, in step S3, the optimization of the preliminary pose estimated in step S2 by combining geometric constraints and texture constraints to obtain a refined inter-frame motion estimate includes:

[0039] S31. Using the initial transformation matrix as the optimization objective, construct the initial optimization function E1 according to the following formula:

[0040]

[0041] Where G is the geometric constraint, L is the texture constraint, ω is the confidence level of the texture constraint, κi,j is the set of matching point pairs, p is the 3D point of image frame i, q is the corresponding point of 3D point p in image frame j, and m is the preset total number of matching point pairs.

[0042] Specifically, combining geometric and optical constraints, the minimization objective in the current embodiment includes two parts: one is the distance between the tangent plane of each target point and its corresponding source point, and the other is the gradient error between each target point and its corresponding source point. The two will be assigned different weights w according to the actual application.

[0043] In the current frame F i With sample frame F j The corresponding matching point set κ i,j In the above, assume that p = (p x ,p y ,p z ,1)T is the source point cloud, q=(p x ,p y ,p z ,1) T For the target point cloud corresponding to p, n = (n x ,n y ,n z 1) T is the unit normal vector, g p Let g be the gradient value of the source point cloud p. q Let q be the gradient value of the target point cloud, and m be the number of matching point pairs. When iteratively optimizing the above formula (3), the goal of each iteration is to find the optimal (r) opt ;t opt ), where (r opt ;t opt The following equation must be satisfied:

[0044]

[0045] S32. The optimization objective is iteratively optimized using a nonlinear optimization method, and when the preset iteration termination condition is reached, the refined inter-frame motion estimate is obtained based on the optimal transformation matrix output by the last iteration.

[0046] Specifically, in order to solve the objective function constructed above, this embodiment defines the initial transformation matrix as a vector with six parameters: that is, ξ = (α, β, γ, a, b, c). Then the initial transformation matrix can be linearly represented as:

[0047]

[0048] Among them, T k This is the transform estimate from the last iteration, currently using the Gauss-Newton method (J... r T J r +λI)ξ=-J r T r Solve for the parameter ξ and apply the parameter ξ to T. k To update T, where r is the residual and Jr It is a Jacobian matrix.

[0049] In one embodiment, the preset iteration termination condition can be reaching a preset maximum number of iterations, etc., and different embodiments can be flexibly adjusted according to the actual application scenario.

[0050] In the above embodiments, geometric and optical constraints are integrated, texture information is fully utilized, and a method is proposed to calculate and solve the texture image to obtain feature values ​​that are insensitive to illumination and have strong anti-interference ability to replace the unprocessed pixel intensity. This makes the system more adaptable to optical changes and the registration results more robust.

[0051] S4. The data obtained through inter-frame motion estimation is segmented to obtain multiple data segments, and the pose in each data segment is optimized; wherein each data segment includes multiple image frames.

[0052] S5. For each data segment, select a key frame from the multiple image frames included in the data segment, and combine the key frames and loop closure information to perform joint optimization between segments.

[0053] Optionally, keyframe extraction needs to satisfy at least one of the following conditions:

[0054] (1) There is at least one keyframe in every N image frames, so that global information can be expressed through the keyframe.

[0055] (2) When the current image frame can match the previous image frame, but the current image frame cannot match the preset reference key frame, the previous image frame of the current image frame will be added to the preset key frame set to ensure the continuity of trajectory tracking.

[0056] (3) Although the current image frame can match the previous image frame and the current image frame can also match the preset reference key frame, the overlap rate between the current image frame and the preset reference key frame is not high enough. In this case, the current image frame needs to be added to the preset key frame set to ensure that there is overlap between adjacent key frames.

[0057] In one embodiment, the absolute pose estimation of the depth sensor accumulates significant pose errors over time. Furthermore, after implementing the local optimization measures in step S4, the pose information between segments lacks global consistency, and accumulated errors persist. To overcome these problems, this embodiment utilizes loop closure information and each keyframe to perform joint optimization between segments. It should be noted that loop closure information is typically calculated directly based on the image or features. In one embodiment, to obtain accurate loop closure information, this application employs an inter-frame matching method, matching adjacent keyframes pairwise. A corresponding loop is formed when a match is successful and the overlap rate reaches a set threshold.

[0058] In addition, since keyframes run through the entire tracking process and can fully reflect the global picture, in order to improve the efficiency of pose optimization, the global optimization in this embodiment does not involve all frames. Instead, one frame is selected from each data segment to represent that data segment. This image frame is collectively referred to as the keyframe. Then, the loop closure information is combined to perform global optimization. At this time, most of the accumulated errors can be quickly eliminated through global optimization.

[0059] In the above embodiments, based on the segmented multi-mode optimization strategy, the problem can be modeled at different levels of abstraction, achieving fast and accurate optimization.

[0060] S6. For each data segment, fix the pose of the key frame in the corresponding data segment, and optimize the pose of other image frames in the data segment to obtain a globally consistent and smooth motion trajectory map.

[0061] Specifically, the poses of keyframes have been updated through optimization of the global pose graph. However, to obtain a globally consistent and smooth motion trajectory, the poses within a local area also need to be updated. Therefore, this embodiment adopts a layered approach, not optimizing all image frames simultaneously, but fixing the poses of each keyframe segment and only optimizing the poses of other image frames within the segment.

[0062] S7. Combining the relative pose measured by the depth sensor and the absolute pose estimated by the motion trajectory map, construct a corresponding target optimization function.

[0063] S8. Incorporate the preset penalty factor into the target optimization function, and through iterative transformation estimation, eliminate the cumulative error generated as the number of scan frames increases during inter-frame matching, and perform fusion mesh construction.

[0064] Specifically, in step S8, when incorporating the preset penalty factor into the objective optimization function, the expression formula of the objective optimization function E2 is as follows:

[0065] E2=∑ i,j ρ(e 2 (p i ,p j ;∑ i,j ,T i,j (6)

[0066] Here, the estimated absolute pose is taken as a node, p i Representing nodes i and p j Represents node j; T i,j Represents the relative pose between node i and node j, ∑ i,j This indicates that all constraint pairs are summed; e 2 (p i ,p j ;∑ i,j ,T i,j )=e(p i ,p j ;T i,j ) T ∑ i,j -1 e(p i ,p j ;T i,j ), e(p i ,p j ;T i,j ) = T i,j -p i -1 p j ρ is the penalty factor for integration.

[0067] In one embodiment, u = d 2 d represents the surface diameter of the reconstructed object. Considering that a suitable penalty function can effectively perform verification and filtering without increasing computational cost, the current embodiment uses the Geman-mclure function from M-estimation, i.e.

[0068] Since the above formula (6) is difficult to optimize directly, we currently assume relation l, and assume the objective optimization function E2 is:

[0069] E2=∑ i，j l(e 2 (p i p j ;∑ i，j T i，j )+∑ i，j ψ(1); (7)

[0070] Among them, it is known Minimize the formula E2 by taking the partial derivative with respect to l. In actual calculations, l is considered as the confidence level. Constraints with smaller residuals have higher error weights and are considered more reliable; conversely, constraints with larger residuals are less reliable. This serves as a verification and elimination mechanism, resulting in robust optimization. Furthermore, the selection of the parameter μ is crucial; μ = d 2 , representing the surface diameter of the reconstructed object, controls the range of significant influence of the residuals on the objective. A larger μ makes the objective function smoother and allows more corresponding terms to participate in the optimization. As μ decreases, the objective function becomes sharper, more outlier matches are eliminated, and the data participating in the optimization becomes more accurate.

[0071] To solve this nonlinear squared error function problem, the transformation matrix is ​​also transformed according to formula (5) in the current embodiment. Considering that only a small number of nodes in the pose graph have direct edge connections, i.e., the sparsity of the pose graph, and for numerical stability, the current embodiment uses a sparse BA algorithm to solve it. Sparse BA is usually optimized using the LM method. LM adds a positive definite diagonal matrix to the Gauss-Newton method, i.e., through (J r T J r +λI)ξ=-J r T r Let's solve for ξ.

[0072] It should be noted that the effect after rapid optimization is as follows: Figure 4 As shown in (c), further fusion and mesh construction are performed, and the effect is as follows. Figure 4 As shown in (d).

[0073] In one embodiment, a 3D scanning system applied to the online matching optimization method is also provided, the system comprising:

[0074] The acquisition module is used to acquire a one-to-one corresponding depth texture image pair, wherein the depth texture image pair includes a depth image acquired by a depth sensor and a texture image acquired by a camera device.

[0075] The inter-frame motion estimation module is used to perform feature matching between the depth texture image pair corresponding to the current frame and the depth texture image pair corresponding to the sample frame using a progressive refinement strategy to estimate the initial pose of the depth sensor, and to optimize the estimated initial pose by combining geometric constraints and texture constraints to obtain a refined inter-frame motion estimate.

[0076] A multi-mode optimization module is used to segment the data obtained through inter-frame motion estimation to obtain multiple data segments, and optimize the pose of each data segment; wherein each data segment includes multiple image frames; for each data segment, a key frame is selected from the multiple image frames included in the data segment, and combined with the key frame and loop closure information, joint optimization between segments is performed, and the pose of the key frame in the corresponding data segment is fixed, and the pose of other image frames in the data segment is optimized to obtain a globally consistent and smooth motion trajectory map;

[0077] The cumulative error elimination module is used to construct a corresponding target optimization function by combining the relative pose measured by the depth sensor and the absolute pose estimated by the motion trajectory map; it is also used to incorporate a preset penalty factor into the target optimization function, and eliminate the cumulative error generated as the number of scan frames increases during inter-frame matching through iterative transformation estimation, and perform fusion mesh construction.

[0078] In one embodiment, the depth sensor includes a projection module and a depth information acquisition module. The projection module is used to project white light or a structured light beam of a specific wavelength onto the surface of the object being measured. The depth information acquisition module is used to acquire depth information of the surface of the object being measured when the projection module projects the structured light beam. The camera device includes a texture information acquisition module, which is used to acquire texture information of the surface of the object being measured when the camera device projects a texture illumination beam onto the surface of the object being measured.

[0079] Here, the structured beam and the textured illumination beam are projected alternately. When the structured beam is projected once, the depth information acquisition module acquires the depth information of the surface of the object being measured. Then, the projection of the textured illumination beam is started, and the camera device acquires the texture information of the surface of the object being measured projected by the textured illumination beam once.

[0080] The above is a single cycle of the measurement process of texture and depth information of the surface of the object being measured. When the above measurement process is repeated at a certain repetition frequency, the relative position and relative angle between the camera device, the depth sensor and the object being measured will change continuously, thus completing the continuous measurement of the structure of the object being measured.

[0081] In one embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the above-described online matching optimization methods.

[0082] In one embodiment, a 3D scanning device for an online matching optimization method is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0083] This invention discloses an online matching optimization method and 3D scanning system that combines geometry and texture. On one hand, it integrates both geometric and optical constraints, fully utilizing texture information. It proposes to calculate and solve for texture images, obtaining feature values ​​that are insensitive to illumination and have strong anti-interference capabilities to replace unprocessed pixel intensities. This makes the system more adaptable to optical changes and the registration results more robust. On the other hand, it employs a stepwise refinement strategy to decompose and simplify the complex problem. First, it preliminarily estimates the pose using features, then refines the pose to gradually obtain an accurate pose estimate. Furthermore, a penalty factor is added to the subsequently established optimization objective function. Without increasing additional computational costs, it can effectively verify and filter different constraint pairs, ensuring the accuracy and stability of the optimization. Finally, it employs a piecewise multi-mode optimization strategy, enabling modeling of the problem at different levels of abstraction, achieving fast and accurate optimization.

[0084] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A three-dimensional image matching optimization method, characterized in that, The method includes: Obtain a one-to-one corresponding depth-texture image pair, wherein the depth-texture image pair includes a depth image acquired by a depth sensor and a texture image acquired by a camera device; A progressive refinement strategy is adopted to perform feature matching between the depth texture image corresponding to the current frame and the depth texture image corresponding to the sample frame in order to estimate the initial pose of the depth sensor. The preliminary pose estimated in step S2 is optimized by combining geometric constraints and texture constraints to obtain a refined inter-frame motion estimate. The data obtained through inter-frame motion estimation is segmented to obtain multiple data segments, and the pose in each data segment is optimized; wherein each data segment includes multiple image frames; For each data segment, a key frame is selected from the multiple image frames included in the data segment. Combining the key frames and loop closure information, joint optimization is performed between segments. For each data segment, the poses of the key frames in the corresponding data segment are fixed, and the poses of other image frames in the data segment are optimized to obtain a globally consistent motion trajectory diagram with smooth transition. By combining the relative pose measured by the depth sensor and the absolute pose estimated by the motion trajectory map, a corresponding target optimization function is constructed. The preset penalty factor is incorporated into the target optimization function, and the cumulative error generated during inter-frame matching as the number of scan frames increases is eliminated through iterative transformation estimation, and then fused mesh is constructed. Acquire three-dimensional image frames and perform pose optimization on the three-dimensional image frames to obtain a motion trajectory map; The absolute pose is estimated based on the motion trajectory diagram. Relative pose obtained by depth sensor measurement; Construct a target optimization function based on the relative pose and the absolute pose; A preset penalty factor is incorporated into the target optimization function to perform inter-frame matching optimization on the image and then perform fusion mesh construction. Wherein, incorporating the preset penalty factor into the objective optimization function is as follows: E2=∑ i,j p(e 2 (p i ,p j ?∑ i,j ,T i,j )); Here, the estimated absolute pose is taken as a node, p i Representing nodes i and p j Represents node j; T i,j Represents the relative pose between node i and node j, ∑ i,j This indicates that all constraint pairs are summed; e 2 (p i ,p j ;∑ i,j ,T i,j )=e(p i ,p j ;T i,j ) T ∑ i,j -1 e(p i ,p j ;T i,j ), e(p i ,p j ;T i,j ) = T i,j -p i -1 p j ρ is the penalty factor for integration; , To reconstruct the surface diameter of the object, considering that an appropriate penalty function can perform good verification and filtering without increasing additional computational costs, the Geman-mclure function in M ​​estimation is selected.

2. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the steps of the method of claim 1.

3. A three-dimensional scanning system applied to an online matching optimization method, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method of claim 1.

Images