A synthetic dataset generation method for multi-focal surface three-dimensional reconstruction

Abstract

This invention belongs to the field of 3D reconstruction technology, specifically relating to a method for generating synthetic datasets for multi-focal plane 3D reconstruction. The method includes the following steps: First, grayscale images of different scenes are acquired using a camera, and then some depth-ill-indeterminate scenes are filtered out to obtain a scene image set; second, elementary functions, piecewise functions, and evolutionary algorithm test functions are used to simulate the depth and topography of the scene, and a 3D topography map is obtained by random transformation using a topography kernel function; finally, the scene image set and the 3D topography map are simulated using a point spread function to generate a multi-focal plane image dataset. This invention's method can achieve the simulation of massive multi-focal plane image datasets at low cost and high efficiency, helping to narrow the gap between synthetic datasets and real datasets.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, specifically relating to a method for generating synthetic datasets for multi-focal plane 3D reconstruction. Background Technology

[0002] Multifocal plane 3D topography reconstruction, as an important branch of computer vision, is mainly used for 3D modeling and quantitative analysis of object surfaces in fields such as healthcare, biopharmaceuticals, and precision manufacturing. With the increasing demand for applications in various scenarios such as precise measurement, intelligent modeling, and 3D interactive experiences, existing multifocal plane 3D reconstruction methods cannot meet the intelligent decision-making needs of diverse application scenarios.

[0003] Currently, methods for reconstructing the 3D topography of object surfaces using multifocal planes can be broadly categorized into two types: traditional mathematical modeling and neural network fitting. Traditional mathematical modeling first assesses the focused measurement volume of the scene by constructing a focusing measurement operator; then, it refines the sequential dimensions of the focused measurement volume to obtain the relative position information of the focal points, which forms the initial depth map; finally, it repairs the initial depth map using information such as the edges of the scene to obtain the final depth map. However, because the parameters of the focusing evaluation operator in mathematical modeling are fixed, and edge information cannot be effectively integrated with the initial depth information, it cannot adaptively predict the 3D topography structure of the scene. With the open-source availability of large-scale multifocal plane image datasets and the rapid development of computer hardware, neural networks can extract abstract focusing features and scene semantic features from multifocal plane image datasets to form intelligent decision models, effectively overcoming the shortcomings of traditional mathematical modeling methods in single-scene prediction. The decision of the neural network regarding the focused region in a multifocal plane image sequence depends on the dataset acquisition method and the annotation of the 3D topography, both of which directly affect the robustness and accuracy of the neural network in 3D reconstruction across multiple scenes. Currently, multifocal plane image datasets mainly collect mesoscopic indoor scenes and macroscopic architectural scenes. Mesoscopic and macroscopic scene datasets rely on image texture details and color distribution to assist in 3D reconstruction, but these scenes cannot be applied to the field of fine texture scenes. 3D annotations mostly use light field multi-view image data to simulate plane focusing or use Blender computational imaging to construct defocus, both of which only generate a small number of depth images and cannot achieve low-cost and high-efficiency multifocal plane image data.

[0004] The above research indicates that existing 3D reconstruction methods have the following two shortcomings: traditional mathematical modeling relies on the initial parameter construction and cannot adapt to the 3D shape reconstruction of multiple scenes; existing multifocal image datasets, due to the single image data acquisition method and the high cost of 3D annotation, cannot guide neural networks to fit the special shapes of many scenes.

[0005] In summary, we believe that improving the dataset acquisition method and refining the annotation of 3D structures are among the ways to solve the problems of neural network-based 3D reconstruction methods. Therefore, the method of this invention designs a synthetic dataset simulation method for multi-focal plane 3D reconstruction, which can achieve scene uniformity under multiple perspectives and low-cost annotation of 3D structures, thereby guiding neural networks to achieve cross-scene applications. Summary of the Invention

[0006] To overcome the shortcomings of existing methods, the purpose of this invention is to provide a method for generating synthetic datasets for multi-focal plane 3D reconstruction, comprising the following steps:

[0007] Step 1: The camera acquires grayscale images of different scenes to obtain a scene image set. n represents the number of scene images, and its value ranges from 1 to n to N, where N is the total number of images.

[0008] Step 2, process the scene image set obtained in Step 1. The image information entropy is calculated according to equation (1), and the image gradient sum is calculated according to equation (2). Then, the texture evaluation result of the scene image set is obtained through equation (3).

[0009]

[0010]

[0011] E(I n )=0.4×EN(I n ) / 8+0.6×GR(I n ) / (3×10 6 ), 1≤n≤N(3)

[0012] Where EN() represents the image information entropy function, I n (p) represents the statistical value of each pixel in the image, p represents the gray value in the image, GR() represents the image gradient accumulation and summation function, sum() represents the single-point summation function, * represents the convolution operator, and E() represents the image texture evaluation function.

[0013] Step 3, process the scene image set obtained in Step 1. According to the filtering rules of Equation (4), a set of scene images with rich texture and moderate grayscale range is obtained.

[0014]

[0015] Among them I n′ This represents the number of scene images after filtering, and its value ranges from 1 to n′ to N′, where N′ is the total number of images after filtering.

[0016] Step 4: Simulate the randomization process of scene morphology using elementary functions from equations (5) to (9), piecewise functions from equation (10), and evolutionary algorithm test functions from equations (11) to (15), and construct the initial three-dimensional morphology set of the scene accordingly. Its value range is 1≤n≤N, where N is the total number of images;

[0017] f(x)=k(5)

[0018] f(x)=kx+b(6)

[0019] f(x) = sinx + b (7)

[0020] f(x) = a x +b(8)

[0021] f(x) = log a x(9)

[0022]

[0023] f(x,y)=20k+[x 2 -100cos(2πx)]+[y 2 -10cos(2πy)](11)

[0024] f(x,y)=k1x 2 +k1y 2 (12)

[0025]

[0026]

[0027] f(x,y)=k(yx 2 ) 2 +(1-x) 2 (15)

[0028] Where a, b, and c are parameters, x and y are the range values ​​of the scene graph's width and height, and k and k n ,1≤n≤N is a random parameter value, and its value range is 0<k,k n ≤1, 1≤n≤N;

[0029] Step 5, from the initial 3D topography set in Step 4 Two groups of morphological S were randomly selected from the middle p With S q Then, arbitrarily select a morphology kernel function from equations (16) to (20) to obtain a highly randomized three-dimensional morphology atlas. As depth annotations for a scene, their values ​​range from 1 to n′ to N′, where N′ is the total number of depth annotations.

[0030]

[0031] S n " = RS p S q (17)

[0032] S n "=(S p S q +C) R (18)

[0033] S n " = tanh(RS) p S q +C)(19)

[0034]

[0035] Where A, R, C represent random parameters, and their values ​​range from -1 to A, R, C ≤ 1; tanh() represents the hyperbolic tangent function; p and q represent the serial numbers of the three-dimensional elementary topography diagrams, and their values ​​range from 1 to p, q ≤ N.

[0036] Step 6: Process the scene image set obtained in Step 3. Combined with the three-dimensional topography atlas obtained in step 5 A multifocal image dataset is generated using the point spread function from equation (21) to equation (23).

[0037] I d =I n′ *h(21)

[0038]

[0039]

[0040] Where * denotes the convolution operator, exp() is the exponential function, h is the point spread function h, x, y are the pixel coordinates, and σ h For variance, u f The focus position is set for the camera, m is the camera constant, and B is the ratio of the camera focal length f to the lens diameter.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] (1) This invention collects texture maps in a unified manner based on the unique texture features and colors in real-world scenarios, and proposes screening criteria to remove some depth-indeterminate scenes, which can effectively guide neural networks to learn multi-focal images and efficiently infer the three-dimensional structure of the scene.

[0043] (2) This invention proposes to use shape function to simulate the three-dimensional shape of the scene and use point spread function to generate standard multifocal image sequence at low cost and high efficiency, which helps to narrow the gap between synthetic multifocal image data and real multifocal image data. Attached Figure Description

[0044] Figure 1 This is a flowchart of a synthetic dataset generation method for multi-focal plane 3D reconstruction according to the present invention;

[0045] Figure 2 This is a schematic diagram of a synthetic dataset generation method for multi-focal plane 3D reconstruction according to the present invention;

[0046] Figure 3 This is a set of scene images from step 1 in Embodiment 1 of the present invention;

[0047] Figure 4 This is a set of scene images with rich texture and appropriate grayscale range in step 3 of embodiment 1 of the present invention;

[0048] Figure 5 This is a portion of the initial three-dimensional topography set in step 4 of embodiment 1 of the present invention;

[0049] Figure 6 This is a set of three-dimensional topography after kernel function transformation in step 5 of embodiment 1 of the present invention;

[0050] Figure 7 This is a set of multi-focal plane image datasets in step 6 of embodiment 1 of the present invention. Detailed Implementation

[0051] like Figure 1 , Figure 2 As shown, a method for generating synthetic datasets for multi-focal plane 3D reconstruction includes the following steps:

[0052] Step 1: The camera acquires grayscale images of different scenes to obtain a scene image set. n represents the number of scene images, and its value ranges from 1 to n to N, where N is the total number of images.

[0053] Step 2, process the scene image set obtained in Step 1. The image information entropy is calculated according to equation (1), and the image gradient sum is calculated according to equation (2). Then, the texture evaluation result of the scene image set is obtained through equation (3).

[0054]

[0055]

[0056] E(I n )=0.4×EN(I n ) / 8+0.6×GR(I n ) / (3×10 6 ), 1≤n≤N(3)

[0057] Where EN() represents the image information entropy function, I n (p) represents the statistical value of each pixel in the image, p represents the gray value in the image, GR() represents the image gradient accumulation and summation function, sum() represents the single-point summation function, * represents the convolution operator, and E() represents the image texture evaluation function.

[0058] Step 3, process the scene image set obtained in Step 1. According to the filtering rules of Equation (4), a set of scene images with rich texture and moderate grayscale range is obtained.

[0059]

[0060] Among them I n′ This represents the number of scene images after filtering, and its value ranges from 1 to n′ to N′, where N′ is the total number of images after filtering.

[0061] Step 4: Simulate the randomization process of scene morphology using elementary functions from equations (5) to (9), piecewise functions from equation (10), and evolutionary algorithm test functions from equations (11) to (15), and construct the initial three-dimensional morphology set of the scene accordingly. Its value range is 1≤n≤N, where N is the total number of images;

[0062] f(x)=k(5)

[0063] f(x)=kx+b(6)

[0064] f(x) = sinx + b (7)

[0065] f(x) = a x +b(8)

[0066] f(x) = log a x(9)

[0067]

[0068] f(x,y)=20k+[x 2 -100cos(2πx)]+[y 2-10cos(2πy)](11)

[0069] f(x,y)=k1x 2 +k1y 2 (12)

[0070]

[0071]

[0072] f(x,y)=k(yx 2 ) 2 +(1-x) 2 (15)

[0073] Where a, b, and c are parameters, x and y are the range values ​​of the scene graph's width and height, and k and k n ,1≤n≤N is a random parameter value, and its value range is 0<k,k n ≤1, 1≤n≤N;

[0074] Step 5, from the initial 3D topography set in Step 4 Two groups of morphological S were randomly selected from the middle p With S q Then, arbitrarily select a morphology kernel function from equations (16) to (20) to obtain a highly randomized three-dimensional morphology atlas. As depth annotations for a scene, their values ​​range from 1 to n′ to N′, where N′ is the total number of depth annotations.

[0075]

[0076] S′ n′ =RS p S q (17)

[0077] S′ n′ =(S p S q +C) R (18)

[0078] S′ n′ =tanh(RS) p S q +C) (19)

[0079]

[0080] Where A, R, C represent random parameters, and their values ​​range from -1 to A, R, C ≤ 1; tanh() represents the hyperbolic tangent function; p and q represent the serial numbers of the three-dimensional elementary topography diagrams, and their values ​​range from 1 to p, q ≤ N.

[0081] Step 6: Process the scene image set obtained in Step 3. Combined with the three-dimensional topography atlas obtained in step 5 A multifocal image dataset is generated using the point spread function from equation (21) to equation (23).

[0082] I d =I n′ *h (21)

[0083]

[0084]

[0085] Where * denotes the convolution operator, exp() is the exponential function, h is the point spread function h, x, y are the pixel coordinates, and σ h For variance, u f The focus position is set for the camera, m is the camera constant, and B is the ratio of the camera focal length f to the lens diameter.

[0086] Example 1

[0087] A method for generating synthetic datasets for multi-focal plane 3D reconstruction, such as Figure 1 and Figure 2 As shown, it includes the following steps:

[0088] Step 1: The camera acquires grayscale images of different scenes to obtain a scene image set. n represents the number of scene images, and its value ranges from 1 to n to 8. The total number of images is set to 8, such as... Figure 3 As shown;

[0089] Step 2, process the scene image set obtained in Step 1. The image information entropy is calculated according to equation (1), and the image gradient sum is calculated according to equation (2). Then, the texture evaluation result of the scene image set is obtained through equation (3).

[0090]

[0091]

[0092] E(I n )=0.4×EN(I n ) / 8+0.6×GR(I n ) / (3×10 6 ), 1≤n≤N(3)

[0093] Where EN() represents the image information entropy function, I n(p) represents the statistical value of each pixel in the image, p represents the gray value in the image, GR() represents the image gradient accumulation and summation function, sum() represents the single-point summation function, * represents the convolution operator, and E() represents the image texture evaluation function.

[0094] Step 3, process the scene image set obtained in Step 1. According to the filtering rules of Equation (4), a set of scene images with rich texture and moderate grayscale range is obtained. like Figure 4 As shown,

[0095]

[0096] Among them I n′ This represents the number of scene images after filtering, and its value ranges from 1 to n′ to 6. The total number of images after filtering is 6.

[0097] Step 4: Simulate the randomization process of scene morphology using elementary functions from equations (5) to (9), piecewise functions from equation (10), and evolutionary algorithm test functions from equations (11) to (15), and construct the initial three-dimensional morphology set of the scene accordingly. Its value range is 1≤n≤8, and the total number of images is set to 8, such as Figure 5 As shown;

[0098] f(x)=k(5)

[0099] f(x)=kx+b(6)

[0100] f(x) = sinx + b (7)

[0101] f(x) = a x +b(8)

[0102] f(x) = log a x(9)

[0103]

[0104] f(x,y)=20k+[x 2 -100cos(2πx)]+[y 2 -10cos(2πy)] (11)

[0105] f(x,y)=k1x 2 +k1y 2 (12)

[0106]

[0107]

[0108] f(x,y)=k(yx 2 ) 2 +(1-x) 2 (15)

[0109] Where a, b, and c are parameters, all set to 1 in this embodiment; x and y are the range values ​​of the scene graph width and height, set to 1 ≤ x, y ≤ 512 in this embodiment; k and k n , 1≤n≤8 are random parameter values ​​and are all set to 2 in this embodiment;

[0110] Step 5, from the initial 3D topography set in Step 4 Two groups of morphological S were randomly selected from the middle p With S q Then, arbitrarily select a morphology kernel function from equations (16) to (20) to obtain a highly randomized three-dimensional morphology atlas. As depth annotations for the scene, their value range is 1≤n′≤6, and the total number of depth annotations is set to 6.

[0111]

[0112] S′ n′ =RS p S q (17)

[0113] S′ n′ =(S p S q +C) R (18)

[0114] S′ n′ =tanh(RS) p S q +C) (19)

[0115]

[0116] Where A, R, and C represent random parameters, and in this embodiment, they are all set to 1. Their value range is -1≤A,R,C≤1. tanh() represents the hyperbolic tangent function. p and q represent the serial numbers of the three-dimensional elementary topography diagrams, and their value range is 1≤p,q≤8.

[0117] Step 6: Process the scene image set obtained in Step 3. Combined with the three-dimensional topography atlas obtained in step 5 A multifocal image dataset is generated using the point spread function from equation (21) to equation (23).

[0118] I d =I n′ *h (21)

[0119]

[0120]

[0121] Where * denotes the convolution operator, exp() is the exponential function, h is the camera's response per unit point source, x, y are pixel coordinates with a range of 1 ≤ x, y ≤ 512, and σ h For variance, and in this example set to 0.1, u f The focus position for the camera is set to 5 in this embodiment, m is the camera constant, which is set to 20 in this embodiment, and B is the ratio of the camera focal length f to the lens diameter, which is 1.2 in this embodiment.

Claims

1. A method for generating synthetic datasets for multi-focal plane 3D reconstruction, characterized by the following steps: Step 1, the camera collects gray-scale images of different scenes to obtain a scene image set n represents the number of scene images, and has a value range of 1≤n≤N, wherein N is the total number of images. Step 2, calculating the image information entropy according to formula (1) and the image gradient accumulation according to formula (2), and then obtaining the texture evaluation result of the scene image set through formula (3) According to formula (1) to calculate the image information entropy and formula (2) to calculate the image gradient accumulation, and then through formula (3) to obtain the texture evaluation result of the scene image set, E(I n )=0.4×EN(I n ) / 8+0.6×GR(I n ) / (3×10 6 ),1≤n≤N(3) Where EN() represents the image information entropy function, I n (p) represents the statistical value of each pixel in the image, p represents the gray value in the image, GR() represents the image gradient accumulation and summation function, sum() represents the single-point summation function, * represents the convolution operator, and E() represents the image texture evaluation function. Step 3, process the scene image set obtained in Step 1. According to the filtering rules of Equation (4), a set of scene images with rich texture and moderate grayscale range is obtained. where I n′ represents the number of screened scene images, and its value range is 1≤n′≤N′, N′ is the total number of screened images; Step 4: Simulate the randomization process of scene morphology using elementary functions from equations (5) to (9), piecewise functions from equation (10), and evolutionary algorithm test functions from equations (11) to (15), and construct the initial three-dimensional morphology set of the scene accordingly. Its value range is 1≤n≤N, where N is the total number of images; f(x)=k(5) f(x)=kx+b(6) f(x) = sinx + b (7) f(x) = a x + b(8) f(x) = log a x(9) f(x,y) = k1x 2 +k1y 2 (12) f(x,y)=k(y-x 2 ) 2 +(1-x) 2 (15) Where a, b, and c are parameters, x and y are the range values ​​of the scene graph's width and height, and k and k n ,1≤n≤N is a random parameter value, and its value range is 0<k,k n ≤1, 1≤n≤N; Step 5, from the initial 3D topography set in Step 4 Two groups of morphological S were randomly selected from the middle p With S q Then, arbitrarily select a morphology kernel function from equations (16) to (20) to obtain a highly randomized three-dimensional morphology atlas. As depth annotations for a scene, their values ​​range from 1 to n′ to N′, where N′ is the total number of depth annotations. S' n′ = RS p S q (17) S' n′ = (S p S q +C) R (18) S' n′ = tanh(RS p S q +C) (19) Where A, R, C represent random parameters, and their values ​​range from -1 to A, R, C ≤ 1; tanh() represents the hyperbolic tangent function; p and q represent the serial numbers of the three-dimensional elementary topography diagrams, and their values ​​range from 1 to p, q ≤ N. Step 6. Generating a set of scene images from the set of three-dimensional topography maps obtained in step 5 Combining the set of three-dimensional topography maps obtained in step 5 Generating a set of multi-focal plane image data from the point spread functions of equations (21) to (23) I d =I n′ *h (21) Where * denotes the convolution operator, exp() is the exponential function, h is the point spread function h, x, y are the pixel coordinates, and σ h For variance, u f The focus position is set for the camera, m is the camera constant, and B is the ratio of the camera focal length f to the lens diameter.

Images