Data processing method, three-dimensional reconstruction method, device and electronic equipment

Abstract

This disclosure provides a data processing method, a 3D reconstruction method, an apparatus, and an electronic device. The data processing method is applied to a monocular 3D camera, which includes a camera and a projector. The data processing method includes: when the monocular 3D camera captures a target object, acquiring a first phase-shift image of the target object on the camera and a second phase-shift image of the target object on the projector; constructing a target matrix for each point on the target object based on the first and second phase-shift images; using the Jacobian rotation algorithm, performing singular value decomposition (SVD) on the target matrix of each point in parallel to obtain the SVD result of the target matrix; and determining the depth information of the points on the target object based on the SVD result. This disclosure enables efficient SVD of large-scale target matrices, thereby quickly determining the depth information of each point.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data processing method, a three-dimensional reconstruction method, an apparatus, and an electronic device. Background Technology

[0002] A monocular 3D camera mainly consists of a camera and a projector. To capture pixel information of a target object from different angles, the projector can be tilted and the camera can be placed vertically to achieve 3D reconstruction of the target object.

[0003] In particular, when a monocular 3D camera performs 3D reconstruction of a target object, it is necessary to determine the depth information of each point on the target object. However, since there are a large number of points on the target object, such as millions or tens of millions of points, it is very time-consuming to determine the depth information of these points through ordinary mathematical calculations, thus failing to meet the requirement of real-time 3D reconstruction of the target object. Summary of the Invention

[0004] This disclosure provides a data processing method, a 3D reconstruction method, an apparatus, and an electronic device to address the problem of time-consuming depth calculation of points on a target object during current 3D reconstruction.

[0005] The first aspect of this disclosure provides a data processing method applied to a monocular 3D camera, the monocular 3D camera including a camera and a projector; the data processing method includes: when the monocular 3D camera captures a target object, acquiring a first phase-shifted image of the target object on the camera and a second phase-shifted image of the target object on the projector; constructing a target matrix for each point on the target object based on the first phase-shifted image and the second phase-shifted image; using the Jacobian rotation algorithm, performing singular value decomposition on the target matrix of each point in parallel to obtain the singular value decomposition result of the target matrix; and determining the depth information of the points on the target object based on the singular value decomposition result.

[0006] A second aspect of this disclosure provides a three-dimensional reconstruction method, comprising: performing three-dimensional reconstruction of a target object based on depth data obtained by the data processing method of the first aspect.

[0007] A third aspect of this disclosure provides a data processing apparatus for executing the data processing method of the first aspect, applied to a monocular 3D camera, the monocular 3D camera including: a camera and a projector; the data processing apparatus including:

[0008] The acquisition module is used to acquire the first phase-shifted image of the target object on the camera and the second phase-shifted image of the target object on the projector when the monocular 3D camera captures the target object.

[0009] The construction module is used to construct the target matrix for each point on the target object based on the first phase shift map and the second phase shift map;

[0010] The decomposition module is used to perform singular value decomposition on the target matrix of each point in parallel using the Jacobi rotation algorithm, and obtain the singular value decomposition result of the target matrix.

[0011] The determination module is used to determine the depth information of points on the target object based on the singular value decomposition results.

[0012] A fourth aspect of this disclosure provides a three-dimensional reconstruction apparatus for performing a three-dimensional reconstruction method of the second aspect, comprising: a three-dimensional reconstruction module for performing three-dimensional reconstruction of a target object based on depth data obtained by the data processing method of the first aspect.

[0013] A fifth aspect of this disclosure provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method of the first aspect.

[0014] This disclosure applies to 3D reconstruction using a monocular 3D camera. The provided data processing method is applied to a monocular 3D camera, which includes a camera and a projector. The data processing method includes: when the monocular 3D camera captures a target object, acquiring a first phase-shift image of the target object on the camera and a second phase-shift image of the target object on the projector; constructing a target matrix for each point on the target object based on the first and second phase-shift images; using the Jacobian rotation algorithm to perform singular value decomposition (SVD) on the target matrix of each point in parallel to obtain the SVD result of the target matrix; and determining the depth information of the points on the target object based on the SVD result. This disclosure, by constructing a target matrix for each point on the target object and using the Jacobian rotation algorithm to perform SVD on the target matrix of each point in parallel to obtain the SVD result of the target matrix, can efficiently perform SVD on large-scale target matrices, thereby quickly determining the depth information of each point. Attached Figure Description

[0015] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings:

[0016] Figure 1 An application scenario diagram of a data processing method provided for an exemplary embodiment of this disclosure;

[0017] Figure 2 A flowchart illustrating the steps of a data processing method provided as an exemplary embodiment of this disclosure;

[0018] Figure 3 A flowchart illustrating the steps of another data processing method provided as an exemplary embodiment of this disclosure;

[0019] Figure 4 A structural block diagram of a data processing apparatus provided for an exemplary embodiment of this disclosure;

[0020] Figure 5 A schematic diagram of the structure of an electronic device provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0022] In related technologies, the acceleration is based on the singular value decomposition of ultra-large matrices. However, for image-based 3D reconstruction, a 4*4 matrix is ​​constructed. The surface of the target object has millions or tens of millions of points, and each point corresponds to a 4*4 matrix. However, using related technologies to perform singular value decomposition on millions or tens of millions of 4*4 matrices is slow and cannot meet real-time requirements, thus affecting the speed of 3D reconstruction by a monocular 3D camera.

[0023] Based on the above problems, the data processing method provided in this disclosure is applied to 3D reconstruction using a monocular 3D camera. The monocular 3D camera includes a camera and a projector. The data processing method includes: when the monocular 3D camera captures a target object, acquiring a first phase-shift image of the target object on the camera and a second phase-shift image of the target object on the projector; constructing a target matrix for each point on the target object based on the first and second phase-shift images; performing singular value decomposition (SVD) on the target matrix of each point in parallel using the Jacobian rotation algorithm to obtain the SVD result of the target matrix; and determining the depth information of the points on the target object based on the SVD result. This disclosure, by constructing a target matrix for each point on the target object and performing SVD on the target matrix of each point in parallel using the Jacobian rotation algorithm to obtain the SVD result of the target matrix, can efficiently perform SVD on large-scale target matrices, thereby quickly determining the depth information of each point.

[0024] In this embodiment, the data processing method can be an overall data processing method implemented using a cloud computing system. Furthermore, the server executing the data processing method can be a cloud server, so as to leverage the advantages of cloud resources to run various algorithms; relative to the cloud, the data processing method can also be applied to conventional servers or server arrays and other server-side devices, without limitation.

[0025] Furthermore, one application scenario of this disclosure embodiment is as follows: Figure 1 , Figure 1 The diagram includes an object W in space, where point P is a point on the surface of object W. I1 is the first phase-shift image of object W projected onto a monocular 3D camera, P1 is the imaging position of point P corresponding to the first phase-shift image, I2 is the second phase-shift image of object W projected onto the monocular 3D camera, P2 is the imaging position of point P corresponding to the second phase-shift image, O1 is the optical center of the camera, and O2 is the optical center of the projector. The monocular 3D camera is used to photograph object W, obtaining the first phase-shift image I1 and the second phase-shift image I2. Based on the first phase-shift image I1 and the second phase-shift image I2, the depth information of each point on the surface of object W can be calculated, thus enabling 3D reconstruction of object W.

[0026] also, Figure 1 This is merely one exemplary application scenario, and the embodiments disclosed herein can be applied to any scenario involving singular values ​​of a matrix. The embodiments disclosed herein do not limit the specific application scenario.

[0027] Figure 2 A flowchart illustrating the steps of a data processing method provided for an exemplary embodiment of this disclosure. Figure 2 The data processing method shown is applied to a monocular 3D camera, which includes a camera and a projector; specifically, it includes the following steps:

[0028] S201, when a monocular 3D camera is used to photograph a target object, the first phase-shifted image of the target object on the camera and the second phase-shifted image of the target object on the projector are acquired.

[0029] In this embodiment, the monocular 3D camera mainly consists of a camera and a projector. The camera primarily acquires a first phase-shifted image modulated by the surface of the target object, while the projector primarily projects a second phase-shifted image. The monocular 3D camera is structured with the projector tilted and the camera vertically positioned.

[0030] Among them, reference Figure 1 The first image is shifted as follows Figure 1 In I1, the second image shifted as follows Figure 2 I2 in the middle.

[0031] S202, construct the target matrix for each point on the target object based on the first phase shift diagram and the second phase shift diagram.

[0032] Among them, reference Figure 1 For point P of the target object, the world coordinates of point P are P0. wThe target matrix is ​​constructed as follows: a point P in space needs to have its depth information in actual space obtained using the principle of triangulation, based on its imaging position P1 (a pixel in the first phase-shift image I1) in the camera and its imaging position P2 (a pixel in the second phase-shift image I2) in the projector. Specifically, the imaging position P1 has pixel coordinates in the first phase-shift image I1. Imaging position P2 has pixel coordinates in the second phase shift image I2 Pixel coordinates and pixel coordinates The reprojection relationship has formulas (1) and (2):

[0033]

[0034]

[0035] In formulas (1) and (2), K is the intrinsic parameter matrix, and P W Let P1 be the world coordinates of point P. Let P1 be the coordinates of point P in camera coordinate system 1. W .

[0036] in, Let t = (t1 t2 t3) be the rotation matrix from camera coordinate system 2 to camera coordinate system 1, and let t = (t1 t2 t3) be the translation matrix from camera coordinate system 2 to camera coordinate system 1. (Refer to...) Figure 1 Camera coordinate system 1 is a coordinate system with O1 as the origin, and camera coordinate system 2 is a coordinate system with O2 as the origin.

[0037] Let matrix matrix Extract them into vector form as follows Then formula (1) can be rewritten as formula (3), and formula (2) can be rewritten as formula (4), as follows:

[0038]

[0039]

[0040] Furthermore, by performing a linear transformation on formula (3) to eliminate the depth information s1, we obtain formula (5). By performing a linear transformation on formula (4) to eliminate the depth information s2, we obtain formula (6). Here, the depth information s1 is the depth information of point P in camera coordinate system 1, and the depth information s2 is the depth information of point P in camera coordinate system 2, as detailed below:

[0041]

[0042]

[0043] Then, by subtracting P1 from formulas (5) and (6) respectively, we obtain formulas (7) and (8), as follows:

[0044]

[0045]

[0046] Combining formulas (7) and (8), we obtain formula (9), as follows:

[0047]

[0048] Then the target matrix A is

[0049] The target matrix is ​​a 4x4 matrix. Therefore, each point on each target object corresponds to a 4x4 target matrix, resulting in millions or tens of millions of target matrices.

[0050] S203 employs the Jacobi rotation algorithm to perform singular value decomposition on the target matrix at each point in parallel, obtaining the singular value decomposition result of the target matrix.

[0051] First, the Jacobi rotation algorithm can achieve fast and accurate singular value decomposition of the target matrix. Second, parallelism means that each target matrix corresponds to one thread for singular value decomposition.

[0052] Furthermore, performing singular value decomposition on the target matrix of each point in parallel can also be achieved by using n threads to perform singular value decomposition on m target matrices. In this case, the n threads first perform one-to-one singular value decomposition on n of the m target matrices. After one thread completes the singular value decomposition of one target matrix, it then obtains another target matrix for singular value decomposition, thereby improving the efficiency of singular value decomposition of the target matrices.

[0053] Where, the target matrix A = UΣV T Σ, where U is the left singular matrix, Σ is the singular value matrix, and V is the right singular matrix. In this disclosure, the singular value decomposition result only includes the right singular matrix V.

[0054] S204. Based on the singular value decomposition results, determine the depth data of points on the target object.

[0055] Specifically, singular value decomposition is performed on the target matrix to obtain the right singular matrix V. The last column of the right singular matrix V is... Furthermore, Z represents the depth information of point P. Specifically, Z is the depth information of point P in the world coordinate system.

[0056] Furthermore, in this embodiment of the disclosure, the singular value decomposition result obtained in S203 includes: a right singular matrix, referring to... Figure 3 The singular value decomposition method for each target matrix includes the following steps:

[0057] S301, Determine the symmetric positive semi-definite matrix based on the target matrix and its transpose.

[0058] The process of determining a symmetric positive semi-definite matrix based on the target matrix and its transpose includes: determining some elements of the symmetric positive semi-definite matrix based on the target matrix and its transpose, wherein the elements are either upper triangular elements or lower triangular elements of the symmetric positive semi-definite matrix; and determining the symmetric positive semi-definite matrix based on the elements and the symmetry of the symmetric positive semi-definite matrix.

[0059] In this embodiment of the disclosure, the symmetric positive semi-definite matrix is ​​a symmetric matrix, and the upper triangular elements and the lower triangular elements are symmetric. Therefore, when determining the symmetric positive semi-definite matrix, the upper triangular elements or the lower triangular elements of the symmetric positive semi-definite matrix are determined first, and then the elements of other parts are determined by using the property of symmetry. This can improve the speed of determining the symmetric positive semi-definite matrix, and thus improve the overall speed of singular value decomposition.

[0060] Specifically, the target matrix A is a 4x4 matrix, and the singular value decomposition of the target matrix A is A=U∑V T The left singular matrix U is a 4x4 matrix. The singular value matrix Σ is a 4x4 matrix, with all elements except those on the diagonal being 0. Each element on the main diagonal is called a singular value. The right singular matrix V is a 4x4 matrix. Both the left and right singular matrices U and V are unitary and orthogonal, satisfying U... T U = I, V T V = I, where I is the identity matrix.

[0061] In this embodiment of the disclosure, the symmetric positive semi-definite matrix D (1) The calculation method is as follows, referring to formula (12):

[0062] D (1) =A T A=VΣ 2 V T Formula (12)

[0063] The purpose of formula (12) is to transform the symmetric positive semi-definite matrix D (1) All eigenvectors span a 4x4 right singular matrix V.

[0064] S302. Based on the symmetric positive semidefinite matrix, the right singular matrix is ​​determined using the bilateral Jacobian rotation algorithm.

[0065] In this embodiment of the disclosure, the bilateral Jacobi rotation algorithm is a variant of the optimized Jacobi eigenvalue algorithm.

[0066] Specifically, based on the symmetric positive semi-definite matrix, a two-sided Jacobian rotation algorithm is used to determine the right singular matrix, including: constructing a first similarity transformation function, which is an iterative function. The first similarity transformation function includes: determining the first intermediate matrix of the next round based on the first Jacobian matrix of the current round, the transpose of the first Jacobian matrix, and the first intermediate matrix of the current round, wherein, in the first iteration, the first intermediate matrix is ​​a symmetric positive semi-definite matrix; substituting the symmetric positive semi-definite matrix into the first similarity transformation function to determine the first Jacobian matrix and the first intermediate matrix in each iteration process, wherein, in the iteration process, the elements of the first Jacobian matrix are represented by the sine and cosine values ​​of the first angle; and determining the right singular matrix based on the multiple first Jacobian matrices determined in the iteration process.

[0067] The first similarity transformation function is D. (k+1) =[J (k) ] T D (k) J (k) Among them, J (k) =J(p,q,θ), representing the first Jacobian matrix. D (k) Let D represent the first intermediate matrix of the current round. (k+1) This represents the first intermediate matrix in the next round.

[0068] The first Jacobian matrix is ​​determined using formula (13), as follows:

[0069]

[0070] In formula (13), the element in the p-th row and p-th column of the first Jacobian matrix is ​​cos(θ), the element in the p-th row and q-th column is sin(θ), the element in the q-th row and p-th column is -sin(θ), and the element in the q-th row and q-th column is cos(θ). Other elements on the diagonal are 1, and all other elements outside the diagonal are 0. In formula (13), col p represents the p-th column of the first Jacobian matrix, col q represents the q-th column of the first Jacobian matrix, row p represents the p-th row of the first Jacobian matrix, and row q represents the q-th row of the first Jacobian matrix.

[0071] In this embodiment of the disclosure, J (k) = J(p,q,θ), where 1≤p<q≤4. Then, in each iteration, J... (k) =J1 (k) J2 (k) J3 (k) J4(k) J5 (k) J6 (k) Among them, J1 (k) At that time, (p, q) = (1, 2); J2 (k) At that time, (p, q) = (1, 3); J3 (k) At that time, (p, q) = (1, 4); J4 (k) At that time, (p, q) = (2, 3); J5 (k) At that time, (p, q) = (2, 4); J6 (k) At that time, (p, q) = (3, 4). The objective of each iteration using the first similarity transformation function is to eliminate the first intermediate matrix D of the current round. (k) The first intermediate matrix D for the next round is obtained by taking the elements in the p-th row and q-th column and the elements in the q-th row and q-th column. (k+1) The element in row p, column q and the elements in row q and column q are both 0. Each iteration makes D... (k+1) It is closer to a diagonal form.

[0072] Among them, the symmetric positive semi-definite matrix D (1) Substitute into the first similarity transformation function D (k+1) =[J (k) ] T D (k) J (k) Determine the first Jacobian matrix J in each iteration process. (k) and the first intermediate matrix D (k+1) .

[0073] Furthermore, during the iteration process, the element in the p-th row and p-th column of the first Jacobian matrix is ​​the cosine value of the first angle θ, cos(θ), the element in the p-th row and q-th column is the sine value of the first angle θ, sin(θ), the element in the q-th row and p-th column is the negative value of the sine value of the first angle θ, -sin(θ), and the element in the q-th row and q-th column is the cosine value of the first angle θ, cos(θ).

[0074] Furthermore, based on the multiple first Jacobian matrices determined during the iteration process, the right singular matrix is ​​determined as follows:

[0075] In the convergence theory of the bilateral Jacobian rotation algorithm iteration, it is necessary to choose the absolute value of the first angle θ, |θ| < π / 4, to ensure convergence. Here, the first intermediate matrix D of the current round... (k) The element in the p-th row and p-th column is set to D. (k) pp =a pp The first intermediate matrix D of the current round (k) The element in the p-th row and q-th column is set to D. (k) pq =apq The first intermediate matrix D of the current round (k) The element in the q-th row and p-th column is set to D. (k) qp =a qp The first intermediate matrix D of the current round (k) The element in the qth row and qth column is set to D. (k) qq =a qq The first intermediate matrix D in the next round. (k+1) The element in the p-th row and p-th column is set to D. (k+1) pp =b pp The first intermediate matrix D in the next round (k+1) The element in the p-th row and q-th column is set to D. (k+1) pq =b pq The first intermediate matrix D in the next round (k+1) The element in the q-th row and p-th column is set to D. (k+1) qp =b qp The first intermediate matrix D in the next round (k+1) The element in the qth row and qth column is set to D. (k+1) qq =b qq Let cos(θ) be abbreviated as c, and sin(θ) as s. Then these elements satisfy formula (14), as follows:

[0076]

[0077] in, It is a diagonal matrix, therefore, we need b pq =b qp =a pq (c 2 -s 2 )+(a pp -a qq cs = 0. Where, if a pp =a qq , can make θ = π / 4. If a pp ≠a qq Then we can obtain This allows us to obtain the first angle. Since the arctangent function returns values ​​in the range (-π / 2, π / 2), therefore... The convergence requirement of the bilateral Jacobian rotation algorithm iteration is guaranteed: |θ| < π / 4.

[0078] Furthermore, assuming the first angle θ is very small, under this assumption, we can approximately obtain tan(2θ)≈2tanθ, where let satisfy That is, when θ is very small, It can be obtained Then let We can obtain: This leads to formula (15), as follows:

[0079]

[0080] In formula (15), the first intermediate matrix D is given. (k) The expression for decreasing off-diagonal elements during iteration. D represents the first intermediate matrix. (k) The changing trend of off-diagonal elements during the iteration process, i.e., the function of the optimal first angle θ, is an increasing function within the interval (0, π / 4). When the value of θ is very small, the function value... Approaching 0, this effectively leads to the elimination of off-diagonal elements. However, as the value of θ increases, the function value... It increases. For the function value when θ = π / 4... A value of 1 means that the off-diagonal elements remain unchanged; they are neither decreased nor eliminated.

[0081] Furthermore, by considering another given angle This solves the problem of the approximation method failing when θ≈π / 4 due to D. (k) The problem of poor performance due to off-diagonal elements not decreasing. It can be seen that this approximation It may lie outside the interval (-π / 4, π / 4). However, to guarantee the convergence of the bilateral Jacobian rotation algorithm, |θ| < π / 4. Therefore, we consider truncation. The probability of approximating the value when the calculated value is outside this interval. For a given angle, the conjugate off-diagonal element b pq as follows:

[0082]

[0083] This leads to formula (16).

[0084]

[0085] In formula (16), θ is a decreasing function in the interval [0, π / 4]. Both approximations of formulas (15) and (16) are even functions with θ as the independent variable and periodic functions with period T of π / 2, so it is sufficient to study them in the interval [0, π / 4]. For the two functions (15) and (16) that characterize the degree of reduction of off-diagonal elements, the best one can be selected based on the magnitude of the reduction they produce. The greater the reduction, the better. When θ is in (0, π / 4), the intersection of the two functions, formulas (15) and (16), is θ0 = arctan(2) / 2 ≈ 0.55357. At the intersection point The magnitude reduction ratio is equal to 1 / |tan(2θ0)|=0.5. Therefore, by choosing the best of the two approximations, this disclosure can guarantee that the size of the off-diagonal elements is reduced by at least 50%. So when θ<θ0, formula (15) is chosen, and when θ>=θ0, formula (16) is chosen.

[0086] Finally, by observing that a fixed angle is used only when the amplitude decreases by no more than 0.5, without relying on obvious angle criteria for θ and θ0, we can determine the optimal angle. Otherwise, use formula (15). The judgment form is as follows:

[0087]

[0088] In summary, the specific process for determining (c, s) is as follows: given a pp a qq and a pq ,calculate Then determine Is it less than the break (a) qq -a pp ) 2 If so, then s = ω*a pq c = ω*(a qq -a pp If not, then confirm. The J matrix for each iteration can be determined using c and s.

[0089] Furthermore, in this embodiment of the disclosure, the target matrix is ​​a 4*4 matrix, D (k+1) =[J1 (k) J2 (k) J3 (k) J4 (k) J5 (k) J6 (k) ] T D (k) [J1 (k) J2 (k)J3 (k) J4 (k) J5 (k) J6 (k) ].

[0090] Then determine v (k) =J1 (k) J2 (k) J3 (k) J4 (k) J5 (k) J6 (k) Repeat the previous step (e.g., 4 to 5 times) until D is reached. (k+1) When the matrix is ​​diagonal, v is used in each iteration. (k) Multiplying them together yields a right singular matrix V, where V = v (1) v (2) …v (n) Where k takes the values ​​1, 2, ..., n, and n is an integer greater than 1, representing the result of D after n iterations. (n+1) It is a diagonal matrix.

[0091] In this embodiment, when performing singular value decomposition on the target matrix of each point, the process includes: storing the data generated during the decomposition of the target matrix of each point using a structure array based on a preset computing platform.

[0092] Specifically, using a structure array (SOA) instead of an array structure (AOS) to store the data generated during the decomposition of the target matrix for each point can achieve an optimization strategy for adjacent threads accessing adjacent data in CUDA (a computing platform).

[0093] In one optional embodiment, singular value decomposition is performed on the target matrix of each point, including: when performing floating-point addition, subtraction, or square root reciprocal operations, the internal addition and subtraction functions and square root reciprocal of a preset computing platform are selected to perform singular value decomposition on the target matrix of each point.

[0094] Specifically, the extensive use of operations such as fadd_rn (an addition operation) and fsub_rn (a subtraction operation) means that when mapping to the compiler, manually selecting certain statements that only contain addition and subtraction but not multiplication will not be merged into the floating-point MAD instruction. (The FMAD instruction is the instruction unit of the arithmetic unit in the GPU stream processor, which automatically merges a single addition and a single multiplication of a floating-point number into a*b+c.) In addition, the frsqrt_rn() function is used to implement fast square root reciprocal.

[0095] Avoiding multiplication and division operations in the above singular value decomposition process can improve the efficiency of singular value decomposition.

[0096] In one optional embodiment, singular value decomposition is performed on the target matrix of each point, including: using a union data type and combining bitwise operations such as OR, AND, and NOT to perform singular value decomposition on the target matrix of each point. The union data type includes floating-point type and integer type, wherein the floating-point type is used for data calculation and data recording; the integer type is used for data judgment, and the floating-point type and the integer type share a storage space.

[0097] Specifically, by employing union data types and combining bitwise operations such as OR, AND, and NOT, the instruction flow in CUDA is optimized, branches that affect the parallel performance of CUDA are handled, and the impact of branches on performance is reduced.

[0098] In one optional embodiment, the Jacobi rotation algorithm is used to perform singular value decomposition on the target matrix of each point in parallel. Before obtaining the singular value decomposition result of the target matrix, the method further includes: pre-storing the sine and cosine values ​​of a preset angle for use during the singular value decomposition process.

[0099] This involves pre-storing the sine and cosine values ​​of preset angles for use during singular value decomposition (SVD), reducing redundant calculations during the process. Examples include cos(0), cos(π / 4), cos(π / 2), sin(0), sin(π / 4), and sin(π / 2). Furthermore, the algorithm can be optimized to replace some non-pre-set sine and cosine values ​​with other expressions.

[0100] Furthermore, the singular value decomposition process described above does not require time-consuming computational units such as cosine, sin, and sqrt (square root). By pre-calculating and storing the values ​​and then eliminating these calculations after merging the calculations, the efficiency of singular value decomposition can be improved.

[0101] In this disclosure, the singular value decomposition (SVD) time for millions of 4x4 target matrices on an NVIDIA Jetson TX2 (a computing platform) is limited to tens of milliseconds, exceeding the speed of related technologies. Real-world testing on an NVIDIA Jetson TX2 shows that the computation time for 1.3 million 4x4 symmetric matrices is 15ms, with an accuracy of 10⁻⁵ to 10⁻⁴. Therefore, this disclosure enables fast and efficient SVD for large-scale small matrices.

[0102] Furthermore, this disclosure also provides a three-dimensional reconstruction method, specifically, performing three-dimensional reconstruction of a target object based on the depth data obtained by the above data processing method.

[0103] Specifically, the above data processing method can determine the depth data of each point of the target object, and based on multiple depth data, a 3D reconstruction of the target object can be performed. This disclosure enables efficient and accurate 3D reconstruction of the target object by performing efficient and accurate singular value decomposition on the target matrix.

[0104] In this embodiment of the disclosure, reference is made to Figure 4 In addition to providing a data processing method, a data processing apparatus 40 is also provided for executing the aforementioned data processing method. This apparatus is applied to a monocular 3D camera, which includes a camera and a projector. The data processing apparatus 40 includes:

[0105] The acquisition module 41 is used to acquire the first phase-shifted image of the target object on the camera and the second phase-shifted image of the target object on the projector when the monocular 3D camera captures the target object.

[0106] Module 42 is used to construct a target matrix for each point on the target object based on the first phase shift map and the second phase shift map;

[0107] The decomposition module 43 is used to perform singular value decomposition on the target matrix of each point in parallel using the Jacobi rotation algorithm to obtain the singular value decomposition result of the target matrix.

[0108] The determination module 44 is used to determine the depth information of points on the target object based on the singular value decomposition results.

[0109] In one optional embodiment, the singular value decomposition result includes a right singular matrix. For each target matrix, the decomposition module 43 is specifically used to: determine a symmetric positive semi-definite matrix based on the target matrix and its transpose; and determine the right singular matrix based on the symmetric positive semi-definite matrix using a two-sided Jacobian rotation algorithm.

[0110] In one optional embodiment, when the decomposition module 43 determines the right singular matrix using a two-sided Jacobian rotation algorithm based on a symmetric positive semi-definite matrix, it is specifically used to: construct a first similarity transformation function, which is an iterative function, and includes: determining the first intermediate matrix for the next round based on the first Jacobian matrix of the current round, the transpose of the first Jacobian matrix, and the first intermediate matrix of the current round, wherein, in the first round of iteration, the first intermediate matrix is ​​a symmetric positive semi-definite matrix; substituting the symmetric positive semi-definite matrix into the first similarity transformation function to determine the first Jacobian matrix and the first intermediate matrix in each round of iteration, wherein, in the iteration process, the elements of the first Jacobian matrix are represented by the sine and cosine values ​​of the first angle; and determining the right singular matrix based on the multiple first Jacobian matrices determined in the iteration process.

[0111] In one optional embodiment, when the decomposition module 43 determines the symmetric positive semi-definite matrix based on the target matrix and the transpose of the target matrix, it is specifically used to: determine some elements of the symmetric positive semi-definite matrix based on the target matrix and the transpose of the target matrix, wherein the some elements are the upper triangular elements or lower triangular elements of the symmetric positive semi-definite matrix; and determine the symmetric positive semi-definite matrix based on the some elements and the symmetry of the symmetric positive semi-definite matrix.

[0112] In one optional embodiment, the decomposition module 43 is specifically used to: store the data generated during the process of decomposing the target matrix of each point using a structure array based on a preset computing platform.

[0113] In one optional embodiment, the decomposition module 43 is specifically used to: when performing floating-point addition, subtraction, or square root reciprocal operations, select the internal addition and subtraction functions and square root reciprocal of the preset computing platform to perform singular value decomposition on the target matrix of each point.

[0114] In one optional embodiment, the decomposition module 43 is specifically used to: use a union data type and combine bitwise calculations such as OR, AND, and NOT to perform singular value decomposition on the target matrix of each point. The union data type includes floating-point type and integer type, wherein the floating-point type is used for data calculation and data recording; the integer type is used for data judgment, and the floating-point type and the integer type share a storage space.

[0115] In one optional embodiment, a storage module (not shown) is further included, specifically for: pre-storing the sine and cosine values ​​of a preset angle before performing singular value decomposition on the target matrix of each point in parallel using the Jacobi rotation algorithm to obtain the singular value decomposition result of the target matrix, so as to be called during the singular value decomposition process.

[0116] The data processing apparatus provided in this embodiment constructs a target matrix for each point on a target object; employs the Jacobian rotation algorithm to perform singular value decomposition on the target matrix of each point in parallel, thereby obtaining the singular value decomposition result of the target matrix. This enables efficient singular value decomposition of large-scale target matrices, thereby quickly determining the depth information of each point on the target object.

[0117] In this embodiment of the disclosure, a three-dimensional reconstruction apparatus (not shown) is also provided for performing the above-described three-dimensional reconstruction method, including: a three-dimensional reconstruction module for performing three-dimensional reconstruction of a target object based on depth data obtained by the above-described data processing method.

[0118] The three-dimensional reconstruction device disclosed herein can quickly and efficiently perform three-dimensional reconstruction of target objects.

[0119] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0120] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an example embodiment of this disclosure. For example... Figure 5 As shown, the electronic device 50 includes a processor 51 and a memory 52 communicatively connected to the processor 51, the memory 52 storing computer-executed instructions.

[0121] The processor executes computer execution instructions stored in the memory to implement the data processing method or three-dimensional reconstruction method provided in any of the above method embodiments. The specific functions and technical effects that can be achieved will not be elaborated here.

[0122] This disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the data processing method or three-dimensional reconstruction method provided in any of the above method embodiments.

[0123] This disclosure also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the data processing method or three-dimensional reconstruction method provided in any of the above method embodiments.

[0124] In the embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0127] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0129] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0130] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to a monocular 3D camera, the monocular 3D camera comprising: a camera and a projector; the data processing method comprising: When the monocular 3D camera captures a target object, a first phase-shifted image of the target object on the camera and a second phase-shifted image of the target object on the projector are acquired. Based on the first phase shift map and the second phase shift map, construct the target matrix for each point on the target object; The Jacobi rotation algorithm is used to perform singular value decomposition on the target matrix of each point in parallel to obtain the singular value decomposition result of the target matrix; Based on the singular value decomposition results, the depth information of points on the target object is determined.

2. The data processing method according to claim 1, characterized in that, The singular value decomposition result includes: a right singular matrix. For each target matrix, the singular value decomposition includes: Based on the target matrix and its transpose, determine the symmetric positive semi-definite matrix; Based on the symmetric positive semi-definite matrix, the right singular matrix is ​​determined using the bilateral Jacobian rotation algorithm.

3. The data processing method according to claim 2, characterized in that, The step of determining the right singular matrix using the bilateral Jacobian rotation algorithm based on the symmetric positive semi-definite matrix includes: Construct a first similarity transformation function, which is an iterative function. The first similarity transformation function includes: determining the first intermediate matrix of the next round based on the first Jacobian matrix of the current round, the transpose of the first Jacobian matrix, and the first intermediate matrix of the current round, wherein, in the first round of iteration, the first intermediate matrix is ​​the symmetric positive semi-definite matrix. Substitute the symmetric positive semi-definite matrix into the first similarity transformation function to determine the first Jacobian matrix and the first intermediate matrix in each iteration process, wherein the elements of the first Jacobian matrix are represented by the sine and cosine values ​​of the first angle during the iteration process. The right singular matrix is ​​determined based on the multiple first Jacobian matrices determined during the iteration process.

4. The data processing method according to claim 2, characterized in that, The step of determining the symmetric positive semi-definite matrix based on the target matrix and its transpose includes: Based on the target matrix and its transpose, determine some elements of a symmetric positive semi-definite matrix, wherein the elements are either the upper triangular elements or the lower triangular elements of the symmetric positive semi-definite matrix. The symmetric positive semi-definite matrix is ​​determined based on the aforementioned partial elements and the symmetry of the symmetric positive semi-definite matrix.

5. The data processing method according to any one of claims 1 to 4, characterized in that, When performing singular value decomposition on the target matrix at each point, the following is included: Based on a pre-defined computing platform, the data generated during the decomposition of the target matrix for each point is stored using a structure array.

6. The data processing method according to any one of claims 1 to 4, characterized in that, The singular value decomposition of the target matrix at each point includes: When performing floating-point addition, subtraction, or square root reciprocal operations, the built-in addition and subtraction functions and square root reciprocal functions of the preset computing platform are selected to perform singular value decomposition on the target matrix of each point.

7. The data processing method according to any one of claims 1 to 4, characterized in that, The singular value decomposition of the target matrix at each point includes: A union data type is used, and singular value decomposition is performed on the target matrix of each point by combining OR, AND, and NOT bitwise computation. The union data type includes floating-point type and integer type, wherein the floating-point type is used for data calculation and data recording, and the integer type is used for data judgment. The floating-point type and the integer type share the same storage space.

8. The data processing method according to any one of claims 1 to 4, characterized in that, Before using the Jacobi rotation algorithm to perform singular value decomposition on the target matrix at each point in parallel, and obtaining the singular value decomposition result of the target matrix, the method further includes: The sine and cosine values ​​of preset angles are pre-stored for use during the singular value decomposition process.

9. A three-dimensional reconstruction method, characterized in that, include: The depth data obtained by the data processing method according to any one of claims 1 to 8 is used to perform three-dimensional reconstruction of the target object.

10. A data processing apparatus for performing the data processing method according to any one of claims 1 to 8, characterized in that, Applied to a monocular 3D camera, the monocular 3D camera includes: a camera and a projector; the data processing device includes: The acquisition module is used to acquire a first phase-shifted image of the target object on the camera and a second phase-shifted image of the target object on the projector when the monocular 3D camera captures the target object; A construction module is used to construct a target matrix for each point on the target object based on the first phase shift map and the second phase shift map; The decomposition module is used to perform singular value decomposition on the target matrix of each point in parallel using the Jacobi rotation algorithm to obtain the singular value decomposition result of the target matrix. The determination module is used to determine the depth information of points on the target object based on the singular value decomposition results.

11. A three-dimensional reconstruction apparatus for performing the three-dimensional reconstruction method of claim 9, characterized in that, include: A 3D reconstruction module is used to perform 3D reconstruction of a target object using the depth data obtained by the data processing method according to any one of claims 1 to 8.

12. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method as described in any one of claims 1 to 8 or the three-dimensional reconstruction method as described in claim 9.

Images