A method for optimizing the reconstruction of three-dimensional data from two-dimensional time series

Abstract

The application discloses a method for optimizing two-dimensional time sequence to reconstruct three-dimensional data, which comprises the following steps: step 1, synchronously collecting two-dimensional images of a target and corresponding spatial poses about time sequences to obtain original data; step 2, converting the collected spatial pose matrix sequences into six degrees of freedom about time sequences, respectively performing piecewise cubic spline fitting on the six degrees of freedom about time sequences and resampling, and converting the sampled six degrees of freedom sequences into spatial pose sequences in the form of matrix again; step 3, combining the two-dimensional image sequences, determining the best corresponding time points in the fitted spatial pose sequences according to the correlation between the images by using an iterative optimization method; and step 4, mapping the image sequences to a three-dimensional space according to corresponding spatial pose information to complete three-dimensional reconstruction. The application optimizes the spatial pose sequences of the original data about time to optimize the three-dimensional reconstruction result.

Description

Technical Field

[0001] This invention relates to a method for reconstructing three-dimensional data, and more particularly to a method for optimizing the reconstruction of three-dimensional data from two-dimensional time series. Background Technology

[0002] In computer vision, 3D reconstruction is the process of reconstructing the 3D information of a target object from single-view or multi-view images. Because single-view images are incomplete, 3D reconstruction relies on empirical information. Multi-view images, however, contain sufficient information to reconstruct a 3D model of the target using multiple 2D images.

[0003] Currently commonly used multi-view Figure 3 The 3D reconstruction process is as follows: image acquisition, camera calibration, feature extraction, stereo matching, and 3D reconstruction. Image acquisition involves capturing a set of images of the target object. Camera calibration involves obtaining the camera's intrinsic and extrinsic parameters by capturing an effective imaging model, facilitating the subsequent calculation of the 3D coordinates of image pixels in space. Feature extraction and stereo matching establish correspondences between images based on certain features in the 2D images. Finally, 3D reconstruction recovers the target's 3D information based on the relationships between images and the camera's intrinsic and extrinsic parameter matrices. In this process, feature extraction and stereo matching are susceptible to image quality and noise, which can significantly reduce positioning accuracy.

[0004] Another probe positioning method indirectly obtains the spatial pose information of the target by positioning the camera or probe used for imaging. Common positioning methods include mechanical positioning, electromagnetic positioning, and optical positioning. Among them, optical positioning is a convenient and low-cost method. It involves fixing an optical feature, such as a calibration plate, to the scanning camera or probe, and then using a fixed camera to photograph this feature to calculate the pose of the scanning camera or probe in space. This type of method is susceptible to problems such as shaking during scanning and system latency, and the 3D reconstruction results may contain offsets and jitter. Summary of the Invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is to provide a method for optimizing the reconstruction of three-dimensional data from two-dimensional time series, in order to address the shortcomings of the existing technology.

[0006] To address the aforementioned technical problems, this invention discloses a method for optimizing the reconstruction of three-dimensional data from two-dimensional time series data, comprising the following steps:

[0007] Step 1: Synchronously acquire the two-dimensional image of the target and the corresponding spatial pose sequence over time to obtain the raw data; the raw data includes the two-dimensional image sequence and the spatial pose matrix sequence;

[0008] Step 2: Convert the acquired spatial pose matrix sequence into a sequence of six degrees of freedom over time. Perform piecewise cubic spline fitting on the six degrees of freedom sequence and resample. Convert the sampled six degrees of freedom sequence back into a new spatial pose sequence in matrix form.

[0009] Step 3: Combining the two-dimensional image sequence, based on the correlation between images, an iterative optimization method is used to determine the optimal corresponding time node for each two-dimensional image in the fitted spatial pose sequence.

[0010] Step 4: Map the two-dimensional image sequence into three-dimensional space according to the corresponding spatial pose information to complete the three-dimensional reconstruction.

[0011] Furthermore, the synchronous acquisition method described in step 1 specifically includes: synchronizing the image acquisition device and the pose acquisition device through a synchronization signal.

[0012] Furthermore, in the original data described in step 1:

[0013] Spatial pose matrix sequence, that is, spatial pose information is regarded as rigid body transformation, which is represented by rotation and translation matrices;

[0014] A two-dimensional image sequence, denoted as P, is given, and the time node corresponding to the i-th image P(i) is denoted as t. i =T old (i), i = 0, 1, 2, ..., the sequence of spatial pose matrices is denoted as M. old (t i Let T be the value of the unoptimized value. old (i) = i.

[0015] Furthermore, the acquisition of the two-dimensional image of the target in step 1 refers to the acquisition of the two-dimensional image of the target using a linear scanning method, which includes at least an ultrasonic scanning method.

[0016] Furthermore, in step 2, the acquired spatial pose matrix sequence is transformed into a sequence of six degrees of freedom over time. Specifically, the rotation matrix part of the spatial pose matrix sequence is decomposed into three rotational components α, β, and γ according to the Rodriguez formula, and the translation matrix part is decomposed into three translational components x, y, and z along the coordinate axes.

[0017] Furthermore, the new spatial pose sequence described in step 2 is denoted as:

[0018] M new (t),t=0,Δt,2Δt,···,1,1+Δt,1+2Δt,···

[0019] Where Δt is the sampling interval.

[0020] Furthermore, the iterative optimization method described in step 3 specifically includes:

[0021] Step 3-1, Initialization: Initialize the time nodes corresponding to the two-dimensional image to the original time nodes, that is, let P(i) correspond to the new time node T to be optimized. new (i)=T old (i) = i;

[0022] The first and last images, along with their corresponding time points and spatial poses, are fixed and remain unchanged throughout the iteration process.

[0023] Step 3-2: Fix the time nodes corresponding to all even-numbered 2D images, then the new spatial pose corresponding to image P(i_even) is M. new (t i_even The two-dimensional images with even numbers are mapped to three-dimensional space.

[0024] Step 3-3: For a two-dimensional image P(i_odd) with an odd number i_odd, traverse the spatial pose between the corresponding time nodes of the two-dimensional images with even numbers before and after it according to the set range.

[0025] Map the image P(i_odd) to 3D space according to its spatial pose, project it onto the plane containing the preceding and following images, resample it, calculate the correlation coefficient between the image and the preceding and following images, and sum them. Select the time node t with the largest sum of correlation coefficients. i_odd As the new corresponding node of the image P(i_odd), that is:

[0026] T new (i odd )=t i_odd

[0027] Step 3-4: Perform the operation described in step 3-3 on all two-dimensional images with odd serial numbers;

[0028] Step 3-5: Referring to the method described in Step 3-2, fix all two-dimensional images with odd serial numbers and their corresponding time nodes;

[0029] Steps 3-6: Referring to the methods described in steps 3-2 to 3-5, perform operations on all two-dimensional images with even-numbered sequences to update the corresponding time nodes;

[0030] Steps 3-7: Determine if the condition for stopping iteration is met. If so, complete the iteration and set the current time point T. new (i) is the optimal corresponding time point; otherwise, return to step 3-2 and continue iterating.

[0031] Further, the set range described in step 3-3 is defined as the following traversal range:

[0032] The first range is to set the minimum time interval mΔt between two images, where m is a positive integer;

[0033] The second range is to set the maximum allowable deviation value l, that is, the traversal range satisfies the following condition:

[0034] T old (i odd )-[T new (i odd -1)+mΔt]<l or [T new (i odd +1)-mΔt]-T old (i odd )<l

[0035] Traverse M under the condition of the second range new (t),t=T new (i odd -1)+mΔt,T new (i odd -1)+(m+1)Δt,···,T new (i odd +1)-mΔt.

[0036] Further, the conditions for stopping iteration described in step 3-7 include:

[0037] The first condition is that the stability rate s is defined as the number of identical corresponding time nodes of the two-dimensional images between alternate iterations divided by the total number of corresponding time nodes, and the iteration is exited when s is greater than a set threshold;

[0038] The second condition is that the number of iterations n is defined, and the iteration is exited when n is greater than a set threshold.

[0039] Further, when executing the iterative optimization method described in step 3, parallel computing is adopted to reduce time consumption.

[0040] Further, completing the 3D reconstruction described in step 4 refers to mapping the 2D images to the 3D space according to the new spatial pose sequence and its corresponding time nodes obtained after the iteration in step 3 is completed, and displaying the final 3D reconstruction result by means of surface rendering, volume rendering or slice rendering.

[0041] Beneficial effects:

[0042] This invention designs a method for optimizing the spatial pose sequence corresponding to two-dimensional images. It adopts a piecewise cubic spline fitting method to initially reduce the impact of jitter during scanning. In addition, it uses an iterative method based on the correlation between images to further optimize the smoothness of the trajectory of the image sequence in space, making up for or reducing the shortcomings of offset and jitter in the three-dimensional reconstruction results. Attached Figure Description

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0044] Figure 1 This is a schematic diagram of the optimized process of the present invention.

[0045] Figure 2 This is a schematic diagram of a free-scanning three-dimensional ultrasound imaging system.

[0046] Figure 3 This is a diagram showing the coordinate positions.

[0047] Figure 4 This is a schematic diagram of the iterative optimization process. Detailed Implementation

[0048] An adaptive method for optimizing the reconstruction of three-dimensional data from two-dimensional time series data is presented, taking free-scanning three-dimensional ultrasound reconstruction as an example. However, the technical solution of this invention is applicable to any linear scanning imaging system, not limited to ultrasound imaging systems, such as... Figure 1 As shown, it includes the following steps:

[0049] Step 1: Synchronously acquire the two-dimensional image of the target and the corresponding spatial pose sequence over time to obtain the raw data.

[0050] Taking free-scanning 3D ultrasound reconstruction as an example, the process of obtaining raw data requires the construction of a free-scanning 3D ultrasound imaging system. The system connection method is as follows: Figure 2 As shown, the system consists of a control host, an ultrasonic transducer, an industrial camera, and an ultrasonic probe with a calibration plate fixed on it. The calibration plate features an asymmetric circular calibration pattern and four ArUco codes (ArUco, Augmented Reality University of Cordoba, is a miniature augmented reality library designed and developed by the Artificial Vision Applications (AVA) research group at the University of Cordoba. ArUco is a synthetic square marker used to estimate camera pose and is currently integrated into OpenCV; different ArUco codes can be generated by setting the number of squares in the marker).

[0051] The method for synchronously acquiring the two-dimensional image and spatial pose sequence of the target over time includes:

[0052] With the camera position fixed, a synchronization trigger signal is sent from the control host. Simultaneously, the camera captures an image of the calibration plate while acquiring a two-dimensional ultrasound image. The rotation and translation matrix representing the spatial pose information is calculated and denoted as R. w2c With t w2c Based on the pre-calibrated positional relationship between the ultrasound image and the calibration plate, R i2w and t i2w The spatial pose R of the ultrasound image in the camera coordinate system was calculated. i2c With t i2c Positional relationships such as Figure 3 As shown, the calculation relationship is:

[0053]

[0054] The sequence of ultrasound images is denoted as P, and the time node corresponding to the i-th ultrasound image P(i) is denoted as t. i =T old (i), i=0,1,2,···, by R i2c and t i2c The resulting spatial pose sequence is denoted as M. old (t i Let T be the value of the unoptimized value. old (i) = i.

[0055] Step 2: Convert the acquired spatial pose matrix sequence into a sequence of six degrees of freedom with respect to time, perform piecewise cubic spline fitting on each sequence and resample, and convert the sampled six-degree-of-freedom sequence back into a spatial pose sequence in matrix form.

[0056] The method for converting the spatial pose matrix sequence into a six-degree-of-freedom time sequence includes:

[0057] The spatial pose matrix sequence M acquired in step 1 old Decompose it into a sequence of six degrees of freedom with respect to time, that is, the rotation matrix R of the sequence elements. i2c According to Rodriguez's formula, it can be decomposed into three rotational components α, β, and γ, and the translation matrix t. i2c It can be decomposed into three translation components along the coordinate axes: x, y, and z.

[0058] The method for performing piecewise cubic spline fitting and resampling on the six-degree-of-freedom time sequence and converting it into a matrix-form spatial pose sequence includes:

[0059] Piecewise cubic spline fitting (Reference: POLLOCK DS G. Chapter 11 - Smoothing with Cubic Splines [M] / / POLLOCK DS G. Handbook of Time Series Analysis, Signal Processing, and Dynamics. London; Academic Press. 1999: 293-322.). Constructing the loss function L:

[0060]

[0061] In the formula, f(x) is the piecewise cubic spline function used for fitting. The loss function consists of two parts: an error term and a smoothing term. p∈[0,1] is the weight of the error term, and the weight (1-p) of the smoothing term is mutually constrained. When pursuing consistency with the original data, smoothness is sacrificed, and vice versa. i Let p be the error weight at each point, which is generally set to 1. Note that when the error term weight p = 0, the original fit can be simplified to a least-squares linear fit; when the error term weight p = 1, the original fit can be simplified to a piecewise cubic spline interpolation. Therefore, the smoothness of the fitted function or its consistency with the original data can be controlled by adjusting the error term weight p.

[0062] Piecewise cubic spline fitting is performed on the six-DOF time sequence. The six fitted curves are sampled at intervals Δt∈(0,1) while retaining the original time nodes. The new six-DOF sequence is then transformed back into a rotation and translation matrix sequence using the Rodriguez formula, denoted as M. new (t),t=0,Δt,2Δt,···1,1+Δt,1+2Δt,···.

[0063] Step 3: Combining the two-dimensional image sequence, and based on the correlation between images, an iterative optimization method is used to determine the optimal corresponding time point in the fitted spatial pose sequence. The specific process is as follows... Figure 4 As shown.

[0064] The iterative optimization method includes:

[0065] Iteration process 1: Before the iteration begins, initialize the time nodes corresponding to the two-dimensional image to the original time nodes, that is, let P(i) correspond to the new time node T to be optimized. new (i)=T old (i) = i; the first and last images and their corresponding time nodes and spatial poses are fixed and not changed throughout the iteration.

[0066] Iteration process 2: first, fix the time nodes corresponding to all images with even serial numbers, so that the new spatial pose corresponding to the even-numbered image P(i_even) is M new (t i_even ), and map it to the three-dimensional space.

[0067] For a certain odd-numbered image P(i_odd), traverse the spatial poses between the time nodes corresponding to the even images before and after it. In order to avoid the coincidence of the time nodes of the images or a large deviation from the original data, it is necessary to limit the traversal range: range 1, mΔt is the minimum time interval between two images; range 2, the maximum allowable deviation value l is defined, and the traversal range should satisfy T old (i odd )-[T new (i odd -1)+mΔt]<l or [T new (i odd +1)-mΔt]-T old (i odd )<l;

[0068] Under the condition of range 2, traverse M new (t), t=T new (i odd -1)+mΔt, T new (i odd -1)+(m+1)Δt, ···, T new (i odd +1)-mΔt, map the odd-numbered image to the three-dimensional space according to the spatial pose, project it onto the planes where the two preceding and following images are located for resampling, calculate and sum the correlation coefficients between it and the preceding and following images respectively, and select the time node t with the maximum sum of correlation coefficients i_odd as the new corresponding node of the odd-numbered image, that is, T new (i odd )=t i_odd ;

[0069] After performing the above operations on all odd-numbered images, fix the odd-numbered images and their time nodes. Similarly, perform the above operations of traversing spatial poses, projecting, resampling, and calculating correlation coefficients on all even-numbered images, update the corresponding time nodes, and complete one round of iteration;

[0070] Iteration process 3: regarding the iteration termination conditions, condition 1: stop the iteration when the time nodes corresponding to the two-dimensional images no longer change or the change degree is small, define the stability rate s as the number of identical time nodes corresponding to the images between alternate iterations divided by the total number of corresponding time nodes, and set to exit the iteration when s is greater than a certain threshold; condition 2: define the number of iterations n, and set to exit the iteration when n is greater than a certain threshold.

[0071] The optimal time node T for obtaining the spatial pose sequence corresponding to the ultrasound image after fitting is obtained. new .

[0072] In the above process, the time consumption of image sequences that need to be updated, as well as operations such as traversing spatial pose, projection, resampling, and calculating correlation coefficients, can be reduced by parallel computing.

[0073] Step 4: Map the image sequence into three-dimensional space according to the corresponding spatial pose information to complete the three-dimensional reconstruction. Specifically, conventional methods such as surface rendering, volume rendering, or slice rendering in existing technologies can be used to display the final three-dimensional reconstruction result.

[0074] The image sequence refers to the two-dimensional ultrasound images acquired in step 1; the spatial pose information corresponding to the image sequence is the M obtained in the above steps. new At time node T corresponding to ultrasound image P new A matrix of time points.

[0075] The method of mapping an image sequence to a three-dimensional space based on corresponding spatial pose information includes:

[0076] Establish a three-dimensional coordinate system with the top-left pixel of the image as the origin, the x-axis as the horizontal axis, the y-axis as the vertical axis, the image plane as the XOY plane, and the z-axis perpendicular to the image plane. Then the coordinates of each point on the image are [u, v, 0]. T Let the coordinates when transformed to the corresponding camera coordinate system be [x c ,y c ,z c ] T According to M new The corresponding matrix and The positional relationships are:

[0077]

[0078] By transforming all the acquired ultrasound image pixels into the camera coordinate system and mapping the planar points into three-dimensional space, three-dimensional ultrasound reconstruction is achieved.

[0079] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding the method for optimizing two-dimensional time series reconstruction of three-dimensional data, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0080] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0081] This invention provides a method and approach for optimizing the reconstruction of three-dimensional data from two-dimensional time series data. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for optimizing the reconstruction of three-dimensional data from two-dimensional time series, characterized in that, Includes the following steps: Step 1: Synchronously acquire the two-dimensional image of the target and the corresponding spatial pose sequence over time to obtain the raw data; the raw data includes the two-dimensional image sequence and the spatial pose matrix sequence; Step 2: Convert the acquired spatial pose matrix sequence into a sequence of six degrees of freedom over time. Perform piecewise cubic spline fitting on the six degrees of freedom sequence and resample. Convert the sampled six degrees of freedom sequence back into a new spatial pose sequence in matrix form. Step 3: Combining the two-dimensional image sequence, and based on the correlation between images, an iterative optimization method is used to determine the optimal corresponding time node for each two-dimensional image in the fitted spatial pose sequence. Specifically, the iterative optimization method includes: Step 3-1, Initialization: Initialize the time nodes corresponding to the two-dimensional image to the original time nodes, that is, let Corresponding to the new time nodes to be optimized ; The first and last images, along with their corresponding time points and spatial poses, are fixed and remain unchanged throughout the iteration process. Step 3-2: Fix the time nodes corresponding to all even-numbered two-dimensional images, then the images... The corresponding new spatial pose is The two-dimensional images with even numbers are mapped to three-dimensional space. Step 3-3, for the serial number Two-dimensional images with an odd number of elements Based on a set range, iterate through the spatial poses of the two-dimensional images corresponding to the time nodes before and after the even-numbered indices of the images; The image Based on the spatial pose mapping to 3D space, the image is projected onto the plane containing the preceding and following images and resampled. The correlation coefficient between the image and the preceding and following images is calculated and summed. The time node with the largest sum of correlation coefficients is selected. As the new corresponding node in this image, that is: ； Step 3-4: Perform the operation in step 3-3 on all two-dimensional images with odd numbers; Step 3-5: Referring to the method described in Step 3-2, fix all two-dimensional images with odd serial numbers and their corresponding time nodes; Steps 3-6: Referring to the methods described in steps 3-2 to 3-5, perform operations on all two-dimensional images with even-numbered sequences to update the corresponding time nodes; Steps 3-7: Determine if the condition for stopping iteration is met. If so, complete the iteration and record the current time point. If the best corresponding time point is selected, otherwise return to step 3-2 and continue iterating; Step 4: Map the two-dimensional image sequence into three-dimensional space according to the corresponding spatial pose information to complete the three-dimensional reconstruction.

2. The method for optimizing two-dimensional time series to reconstruct three-dimensional data according to claim 1, characterized in that, The synchronous acquisition method described in step 1 specifically includes: synchronizing the image acquisition device and the pose acquisition device through a synchronization signal; In the aforementioned raw data: Spatial pose matrix sequence, that is, spatial pose information is regarded as rigid body transformation, which is represented by rotation and translation matrices; A two-dimensional image sequence, denoted as The i-th image The corresponding time node is denoted as The spatial pose matrix sequence is denoted as Assuming no optimization .

3. The method for optimizing two-dimensional time series to reconstruct three-dimensional data according to claim 1, characterized in that, The acquisition of two-dimensional images of the target mentioned in step 1 refers to the acquisition of two-dimensional images of the target using a linear scanning method, which includes at least an ultrasonic scanning method.

4. The method for optimizing two-dimensional time series to reconstruct three-dimensional data according to claim 3, characterized in that, Step 2 involves converting the acquired spatial pose matrix sequence into a six-degree-of-freedom time sequence, specifically by decomposing the rotation matrix portion of the spatial pose matrix sequence into three rotation components according to the Rodrigues formula. The translation matrix is ​​partially decomposed into three translation components along the coordinate axes. .

5. The method for optimizing two-dimensional time series to reconstruct three-dimensional data according to claim 4, characterized in that, The new spatial pose sequence mentioned in step 2 is denoted as: ； in, The sampling interval is denoted as .

6. The method for optimizing two-dimensional time series to reconstruct three-dimensional data according to claim 5, characterized in that, The range set in step 3-3 limits the traversal range as follows: Range 1: Set the minimum time interval between two images. ,in Belongs to positive integers; Range 2: Set the maximum allowable deviation value That is, the traversal range satisfies the following condition: or ; Traversal under the condition of range two 。 7. The method for optimizing two-dimensional time series to reconstruct three-dimensional data according to claim 6, characterized in that, The conditions for stopping iterations described in steps 3-7 include: Condition 1: Define the stability rate The number of identical time points in the 2D image between iterations is divided by the total number of corresponding time points. Exit the iteration when the value exceeds the set threshold; Condition 2: Define the number of iterations. ,when Exit the iteration when the value exceeds the set threshold.

8. The method for optimizing two-dimensional time series to reconstruct three-dimensional data according to claim 7, characterized in that, When performing the iterative optimization method described in step 3, parallel computing is used to reduce the time consumption.

9. The method for optimizing two-dimensional time series to reconstruct three-dimensional data according to claim 1, characterized in that, The completion of 3D reconstruction in step 4 involves mapping the 2D image into 3D space based on the new spatial pose sequence obtained after the iteration in step 3 and its corresponding time nodes, and displaying the final 3D reconstruction result using surface rendering, volume rendering, or slice rendering.

Images