A neural network-based gaussian splash monocular reconstruction method

Abstract

The application discloses a Gaussian splash monocular reconstruction method based on a neural network, which comprises the following steps: firstly, constructing a Gaussian splash monocular reconstruction model comprising an image depth recovery model and an SfM motion recovery model, training the image depth recovery model, then shooting a panoramic picture, training the panoramic picture by using a spherical convolution, recovering a distorted part, then performing sparse point cloud reconstruction by using the SfM motion recovery model, reconstructing an image surface by using a Gaussian reconstruction kernel, forming a basic geometric configuration of the image, finally performing Gaussian low-pass filtering and noise reduction processing on the picture by using an image processing technology, forming a complete surface structure of an object, and completing three-dimensional reconstruction of an image scene. The method is combined with a spherical convolution kernel by using three-dimensional Gaussian splash technology to construct an object geometric model, operability is obviously improved, the calculation speed is greatly improved, and the method has important application value in the fields of intelligent medical treatment and engineering flaw detection.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a Gaussian splash monocular reconstruction method based on a neural network. BACKGROUND

[0002] With the rapid development of machine learning, high-precision three-dimensional surface reconstruction technology is increasingly widely used in various fields, such as reverse engineering, digital modeling, medical diagnosis, virtual reality, etc. This technology can provide more accurate and more intuitive graphical expression through the remodeling and modeling of the geometry of an object. In the era of artificial intelligence, the level of three-dimensional reconstruction technology is crucial to the development of computer technology.

[0003] The document“R. Hartley and A. Zisserman,‘Multiple View Geometry in Computer Vision,’2nd ed. Cambridge: Cambridge University Press, 2004.” proposes a three-dimensional reconstruction method, which geometrically understands and analyzes the projection process from three dimensions to two dimensions, and designs a solution to the inverse problem from two dimensions to three dimensions. However, existing three-dimensional reconstruction usually requires a large number of images with known camera parameters, and multiple steps such as camera parameter estimation, dense point cloud reconstruction and surface reconstruction, which require a huge amount of calculation. In contrast, deep learning-based algorithms have a significant advantage in simplifying the calculation. In neural networks, processing data extracted from pictures becomes a process of encoding and decoding, and with the increase of feature point selection, the three-dimensional model of the object can be reconstructed from a single or multiple two-dimensional images without complex camera calibration. At present, there are three main forms of three-dimensional reconstruction: point cloud-based three-dimensional reconstruction, voxel-based three-dimensional reconstruction and mesh-based three-dimensional reconstruction. Among them, the advent of point cloud reconstruction method Nerf makes neural radiation field the mainstream, but its use of neural network for dense reconstruction still requires high GPU computing power.

[0004] Although the work of recovering layouts from existing images through geometry and deep learning has made rapid progress, there are still few works that use panoramic images to address these challenges. Panoramic cameras have the potential to improve task performance: their 360° field of view can capture the entire field of view around their optical center, and a large amount of information can be obtained at one time, thereby better restoring the layout. SUMMARY

[0005] To solve the above technical problems, the application provides a Gaussian splash monocular reconstruction method based on a neural network, which combines three-dimensional Gaussian with monocular reconstruction technology from the perspective of engineering application, and uses panoramic picture shooting and spherical convolution kernel picture depth recovery technology to reduce the number of pictures required by the previous three-dimensional reconstruction technology, greatly accelerating the speed of three-dimensional reconstruction, and has important application value.

[0006] The technical scheme adopted by the application is as follows: a Gaussian splash monocular reconstruction method based on a neural network, and the specific steps are as follows:

[0007] S1, constructing a Gaussian splash monocular reconstruction model including an image depth recovery model and an SfM motion recovery model;

[0008] The image depth recovery model is used for scene depth recovery of a monocular panoramic image, and includes a pre-training network and a prediction network. The specific structure of the pre-training network and the prediction network is a coding-decoding structure based on a spherical convolution, and the encoder fixedly uses VGG16. The SfM motion recovery model is used for sparse point cloud reconstruction of the recovered image.

[0009] In the coding-decoding structure based on the spherical convolution, the encoder includes a spherical convolution layer, a pooling layer, and a loss function including cross-entropy and regularization; and the encoding model has 5 main layers, each main layer includes a spherical convolution layer and a pooling layer; and the encoding part of the spherical convolution layer and the pooling layer includes 18 sub-layers. The decoder includes 5 layers of stacked deconvolution layers and a Relu activation function. The decoder is connected with the encoder in a skip connection mode, and uses up-sampling on the output.

[0010] S2, training the image depth recovery model based on step S1;

[0011] The points corresponding to the imaging space in the image are view-encoded to form a supervised learning sequence as the input of the pre-training network and the prediction network. The image depth recovery model embeds an adaptive matrix estimation algorithm Adam to optimize the learning rate, simultaneously uses an error back propagation algorithm to update the network weight, and uses the two algorithms to sequentially train the pre-training network and the prediction network. According to the loss function, the gradient change of each parameter is estimated, the learning rate and the learning step of the parameter are dynamically adjusted, the distance of each movement in the opposite direction of the gradient is determined, the weight of the neural network model is constantly updated, the loss function and the precision curve of the model output are observed, when the parameters in the model converge and the model precision reaches or approaches the preset precision, the training process is terminated, and the training is completed.

[0012] In the Adam, m t and v t are also calculated, and the calculation expression is as follows:

[0013]

[0014] m t = β1m t-1 + (1-β1)g t (2)

[0015] v t = β2v t-1 + (1-β2)g t 2 (3)

[0016] where θ denotes any parameter in the model, f t (θ) denotes the stochastic function value of the noisy objective function f(θ) at the subsequent time step t. m t denotes the estimate of the first moment of the gradient, m t-1 denotes the estimate of the first moment of the gradient before update, v t denotes the estimate of the second moment of the gradient, v t-1 denotes the estimate of the second moment of the gradient before update, β1 denotes the exponential decay rate of the first moment estimate, and β2 denotes the exponential decay rate of the second moment estimate. g t denotes the gradient of the function f t (θ) with respect to θ. In Adam, the biased first moment estimate and the second moment estimate are corrected by calculating the bias, and the calculation expression is as follows:

[0017]

[0018] where, denotes the first moment estimate of the time step t, denotes the second moment estimate of the current time step t, and the parameter update formula of Adam is generated as follows:

[0019]

[0020] where θ t+1 denotes the parameter value of the time step t+1, θ t denotes the parameter value of the time step t, η denotes the learning rate, ∈ denotes the bias value, and ∈ = 10 -8 .

[0021] S3, collect image information, input the image depth recovery model trained in step S2 to process the geometric information of the image, and complete the scene depth recovery of the monocular panoramic image;

[0022] S4, input the image with recovered scene depth obtained in step S3 into an SfM motion recovery model, perform sparse point cloud reconstruction, and reconstruct the image surface using a Gaussian reconstruction kernel to form a basic geometric configuration of the image;

[0023] S5, perform Gaussian low-pass filter noise reduction processing on the image reconstructed in step S4 using image processing technology to form a complete surface structure of the object and complete three-dimensional reconstruction of the image scene.

[0024] Further, the step S3 is specifically as follows:

[0025] S31, use a camera to shoot a monocular panoramic image of the object to obtain image information;

[0026] Use a camera and set a turntable for the camera so that it can shoot each direction of the object at a single viewing angle, use the camera to shoot a monocular panoramic image of the object to obtain image information.

[0027] Wherein, the obtained picture includes random high-frequency noise.

[0028] S32, based on the image information obtained in step S31, input the image into the image depth recovery model trained in step S2 to perform image recovery processing on the geometric information of the image, so that the image output contains correct depth information;

[0029] The processing process of the figure is regarded as a rotation change of three-dimensional coordinates, the image to be processed is a three-dimensional rotation group SO(3), that is, a set composed of the initial and final states in the rotation process, and the matrix form expression of the rotation operation is as follows:

[0030]

[0031] Wherein, θ represents the rotation angle corresponding to the rotation operation in three-dimensional space, R x (θ), R y (θ), R z (θ) respectively represent the rotation operation in each dimension in the three-dimensional coordinate system, R x (θ), R y (θ), R z (θ) corresponding to the matrix are the mathematical representations corresponding to each operation.

[0032] The inner product function expression of spherical convolution on the sphere S 2 is as follows:

[0033]

[0034] Wherein, S 2 represents the sphere, represents the pixel function at each position on the sphere, fk (x) represents a convolution kernel function, k represents the kth channel, K represents the total number of image channels, and x represents the independent variable.

[0035] In the trained image depth recovery model, the decoding part has 5 large layers, and the convolution kernel of the convolution part in the first layer is convolved on the sphere S 2 The convolution kernel of the convolution part in the second layer is convolved in the three-dimensional rotation group SO(3). When operating in the three-dimensional rotation group, for any element in the three-dimensional rotation matrix, there is an associated expression as follows:

[0036]

[0037] where x j represents the jth component of the original three-dimensional vector x in the standard basis, and j is the column index; x′ i represents the ith component of the three-dimensional vector x′ after rotation in the standard basis, and i is the row index. represents the element of the rotation matrix R, i.e. the mapping from the jth axis of the original coordinate system to the ith axis of the rotated coordinate system.

[0038] Based on the Euler angle decomposition, the form of motion occurring in SO(3) is represented as the product of the function of twice rotation around a single coordinate axis and the function of motion around another coordinate axis. First, consider the coordinate axes as the regular inertial reference axes, and analyze the motion on the two coordinate axes. Then, consider the coordinate axes as the carrier system with the motion center as the carrier, and describe the form of motion as roll-pitch, then the expression of the motion process is as follows:

[0039] ρ=R z (α)R y (β)R z (γ) (12)

[0040] where R z (α)R y (β)R z (γ) respectively represent the rotation operation in each dimension in the three-dimensional coordinate system, and α, β, γ respectively represent the angle between the rotation operation and the x, y, z coordinate axes; the inner product of R with the function on the rotation group can be regarded as the Hadamard product of the matrix, which satisfies the equivariance when convolved. The fast Fourier transform algorithm is selected for optimization.

[0041] The trained image depth recovery model uses the Wigner d function as the basis to convert the spherical convolution process into a form that can be optimized by the fast Fourier transform algorithm, as follows:

[0042] First, set the expression of the Wigner D function as follows:

[0043]

[0044] Wherein, J, M, M' represent three influence function characterization of the main indicators; J is a non-negative integer, and satisfies -J < M, M' < J.

[0045] Then the second term is Wigner d function, its expansion expression is as follows:

[0046]

[0047] Wherein, u represents the absolute value of M and M' difference, v represents the absolute value of M and M' sum, s represents the difference of J and (u+v) / 2, Indicates the Jacobi polynomial. Then the first term of Wigner d function expansion expression is as follows:

[0048]

[0049] Wherein, if J satisfies greater than M and M', the value of Wigner d function is all 0. The pre-computed sample array of Wigner d function is used as the standard two-dimensional shift fast Fourier transform array.

[0050] Let N represent three-dimensional manifold, U represent base function, then the expression of GFT transformation is as follows:

[0051]

[0052] Wherein, X represents a point on three-dimensional manifold of spherical convolution, Indicates the lth frequency component of function f after graph Fourier transform, that is, the spectral coefficient; f(X) represents a function defined on graph N, that is, the signal value of a point on three-dimensional manifold; Indicates the base function U l The value corresponding to the complex conjugate function at node X; the base function U l Is the orthogonal base function in graph Fourier transform, corresponding to the eigenvector of graph Laplacian operator. Using S 2 Manifold, U l (X) is a vector value Y l (X), using SO(3) manifold, U l (X) takes matrix value D l According to the inverse derivation of Wigner d function, the expression of the point on SO(3) is as follows:

[0053]

[0054] According to formula (16), using GFFT algorithm to formula (17), the expression is as follows:

[0055]

[0056] where D ′ denotes the complex conjugate of the Wigner d-function.

[0057] Further, the step S4 is specifically as follows:

[0058] After the SfM motion recovery model generates a sparse point cloud, a three-dimensional Gaussian is selected as a primitive, and a covariance matrix is used to define set information, and the value of the matrix is determined by the average value of the region space corresponding to the Gaussian kernel.

[0059] A three-dimensional column vector v = (x0, x1, x2) T is used to represent a point in the ray space. Given a central projection and a projection plane: coordinates x0 and x1 specify a point on the projection plane, and the third coordinate x2 specifies the Euclidean distance from the projection center to the point on the viewing ray. The expression of the light intensity on the light path is as follows:

[0060]

[0061] where T denotes the transpose operation, μ is an integral variable, and represents a certain position along the ray from the ray exit point to point ξ in the integral expression. represents the position of the point on the ray when the camera is taken as the origin; L represents the distance from the ray exit point to ; ξ represents the ray irradiation direction and is represented in the form of a 3D Gaussian unit vector; represents the emission coefficient, i.e., the color of the point ; g(x) represents a scatter function that defines the light obscuration rate; represents the emission coefficient at along the ray direction ξ, and the exponential term is an attenuation coefficient; is a light source term, which describes the light intensity scattered at along the ray direction ξ. The scatter function is a weighted sum of Gaussian reconstruction kernels , and the expression is as follows:

[0062]

[0063] where g kG represents the k G th weighted coefficient, which represents the contribution degree of the k G th Gaussian reconstruction kernel to the scatter function . The Gaussian reconstruction kernel is the k Ga reconstruction kernel describing the light scattering or attenuation properties at position Substituting equation (20) into equation (19), we get

[0064]

[0065] Then set the emission coefficient as a constant C0, and use the first two terms of the Taylor expansion as an approximation of the exponential, we get the simplified expression of equation (21) as follows:

[0066]

[0067] where, represents the integral reconstruction kernel, i.e., the integral of the Gaussian reconstruction kernel , and

[0068] Finally, the SfM motion recovery model completes the reconstruction of the image surface. For the output reconstructed image, the originally curved straight lines should become close to straight lines; if the image contains rectangular objects, the four edges of the reconstructed rectangular image should be close to straight line segments, and the four corners should be close to 90° right angles; part of the ellipse in the original image should be close to a circle after reconstruction; the angle of part of the angle in the original image should be reduced to a certain extent after reconstruction.

[0069] Further, in the step S5, after completing the three-dimensional reconstruction of the image scene, the reconstruction performance evaluation index of the Gaussian splash monocular reconstruction model is as follows:

[0070] The evaluation index includes: peak signal-to-noise ratio PSNR, structural similarity index SSIM, and learning perceptual image block similarity LPIPS.

[0071] (1) Peak signal-to-noise ratio PSNR;

[0072]

[0073] where, MSE represents the mean square error; m, n represent the width and height of the image respectively, I(i,j) represents the pixel value of the image at position (i,j) before processing, K(i,j) represents the pixel value of the image at position (i,j) after processing, and MAX I represents the maximum possible pixel value in the image.

[0074] (2) Structural similarity SSIM;

[0075]

[0076] where, u x , u yrespectively represent the mean value of the pixels of the images X, Y, the images X and Y represent the images before and after reconstruction via the Gaussian splash monocular reconstruction model respectively; c1, c2 are constants, and satisfy c1=(K1*L) 2 , c2=(K2*L) 2 , K1 and K2 represent two constants to avoid the denominator being zero when calculating SSIM, L represents the dynamic range of the pixel value, that is, the maximum value of the pixel value in the image; σ xy represents the covariance between the pixel values of the images X and Y, and measures the correlation of the corresponding pixel values in the two images; represents the variance of the pixel values of the image X; represents the variance of the pixel values of the image Y.

[0077] (3) learning the perceptual image block similarity LPIPS;

[0078]

[0079] wherein d(p, p0) represents the distance measurement between the two image blocks p and p0; H l , W l represent the length and width of the output of the lth layer; w l weight is used for weighted scaling of the output of each channel; and respectively represent the activation output of the lth layer network to the input image blocks p and p0 at the position (h, w), and h and w represent the spatial coordinates on the image.

[0080] Advantages of the present application: the method of the present application firstly constructs a Gaussian splash monocular reconstruction model including an image depth recovery model and an SfM motion recovery model, trains the image depth recovery model, then shoots a panoramic picture, trains it with a spherical convolution, recovers the distorted part, then performs sparse point cloud reconstruction through the SfM motion recovery model, and reconstructs the image surface with a Gaussian reconstruction kernel to form the basic geometric configuration of the image, finally adopts image processing technology to perform Gaussian low-pass filtering and noise reduction processing on the picture to form the complete surface structure of the object, and completes the three-dimensional reconstruction of the image scene. The method of the present application uses three-dimensional Gaussian splash technology combined with a spherical convolution kernel to construct an object geometric model, which has significantly improved operability, greatly improves the calculation speed, and has important application value in the fields of intelligent medical treatment, engineering defect detection, etc. The method of the present application provides technical support for clinical diagnosis, special engineering projects, etc. In clinical diagnosis, the Gaussian splash monocular reconstruction model can spread the varicose vein image of the patient's leg to a two-dimensional plane for reconstruction, which is convenient for doctors to diagnose. BRIEF DESCRIPTION OF DRAWINGS

[0081] Figure 1This is a flowchart of a Gaussian splash monocular reconstruction method based on neural networks according to the present invention.

[0082] Figure 2 This is a diagram of the overall structure of the encoder-decoder structure based on spherical convolution in an embodiment of the present invention.

[0083] Figure 3 This is a three-dimensional representation of SO(3) in geometric space in an embodiment of the present invention.

[0084] Figure 4 This is a schematic diagram of the reconstruction result of the method of the present invention in an embodiment of the present invention. Detailed Implementation

[0085] The method of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0086] like Figure 1 The flowchart of a Gaussian splash monocular reconstruction method based on neural networks of the present invention is shown below. The specific steps are as follows:

[0087] S1. Construct a Gaussian splash monocular reconstruction model that includes an image depth restoration model and an SfM motion restoration model;

[0088] The image depth restoration model is used for scene depth restoration of monocular panoramic images and includes a pre-trained network and a prediction network. Both the pre-trained and prediction networks have an encoder-decoder structure based on spherical convolution, with the encoder fixed at VGG16. The SfM motion restoration model is used for sparse point cloud reconstruction of the restored image.

[0089] The overall structure of the encoder-decoder structure based on spherical convolution is as follows: Figure 2 As shown, the encoder includes: spherical convolutional layers, pooling layers, and a loss function containing cross-entropy and regularization; the encoding model has a total of 5 main layers, each including one spherical convolutional layer and one pooling layer; the encoding part's spherical convolutional layers and pooling layers contain a total of 18 sub-layers, i.e. Figure 2 The red and green highlighted sections are shown in the image. The decoder consists of 5 stacked deconvolutional layers and a ReLU activation function. The decoder is skipped from the encoder, and upsampling is used on the output to avoid insufficient resolution.

[0090] S2. Based on step S1, train the image depth restoration model;

[0091] Points in the image corresponding to the imaging space are view-encoded to form a supervised learning sequence, which serves as the input to the pre-trained and prediction networks. The image depth restoration model embeds the Adam adaptive moment estimation algorithm to optimize the learning rate, while simultaneously using the backpropagation algorithm to update the network weights. These two algorithms are used to train the pre-trained and prediction networks sequentially. The gradient change of each parameter is estimated based on the loss function, and the learning rate and learning step size are dynamically adjusted. The distance moved in the opposite direction of the gradient is determined each time, continuously updating the weights of the neural network model. The loss function and accuracy curve output by the model are observed. The training process terminates when the parameters in the model converge (i.e., the parameters remain essentially unchanged throughout multiple training rounds) and the model accuracy reaches or approaches the preset accuracy, thus completing the training.

[0092] To improve the generalization ability of the neural network, a strategy of random data augmentation is adopted, and random dropout is designed to erase the image, avoiding its over-reliance on any single feature. After training to restore the normal viewpoint, Gaussian splashing can be used to reconstruct the image to the correct depth.

[0093] Adam calculates the average m of an exponentially decaying historical gradient. t It also calculates the average v of an exponentially decaying historical squared gradient. t Its calculation expression is as follows:

[0094]

[0095] m t = β1m t-1 + (1-β1)g t (2)

[0096] v t = β2v t-1 + (1-β2)g t 2 (3)

[0097] Where θ represents any parameter in the model, f t (θ) represents the random function value of the noise objective function f(θ) at subsequent time step t. t Let m represent the estimate of the first moment of the gradient. t-1 v represents the estimate of the first moment of the gradient before the update. t Let v represent the estimate of the second moment (nondeterministic error) of the gradient. t-1 β1 represents the estimate of the second moment (nondeterministic error) of the gradient before the update, β2 represents the exponential decay rate of the first moment estimate, and β2 represents the exponential decay rate of the second moment estimate. t The function f represents the expression. t(θ) the gradient with respect to θ. Adam corrects the bias by computing the first and second moment estimates of the bias correction in Adam, the computation expression is as follows:

[0098]

[0099] wherein, denotes the first moment estimate of the time step t, denotes the second moment estimate of the current time step t, the parameter update formula of Adam is generated as follows:

[0100]

[0101] wherein, θ t+1 denotes the parameter value of the time step t+1, θ t denotes the parameter value of the time step t, η denotes the learning rate, controls the step length of the parameter update, ∈ denotes the bias value, and ∈=10 -8 .

[0102] S3, collecting image information, inputting the image depth recovery model trained in step S2 to process the geometric information of the image, completing the scene depth recovery of the monocular panoramic image;

[0103] S4, inputting the image after the scene depth recovery obtained in step S3 into the SfM motion recovery model, performing sparse point cloud reconstruction, and reconstructing the image surface using a Gaussian reconstruction kernel (Gaussian splashing) to form the basic geometric configuration of the image;

[0104] S5, using image processing technology to perform Gaussian low-pass filtering and noise reduction processing on the image after reconstruction in step S4, forming the complete surface structure of the object, and completing the three-dimensional reconstruction of the image scene.

[0105] In the embodiment, the step S3 is specifically as follows:

[0106] S31, using a camera to shoot a monocular panoramic image of an object to obtain image information;

[0107] In the embodiment, a 360° fisheye camera capable of converting poses is used, and a turntable is designed for the camera, so that the camera can shoot each direction of the object at a single viewing angle. The monocular panoramic image of the object is shot using the camera to obtain image information.

[0108] Wherein, the obtained picture includes random high-frequency noise.

[0109] S32, based on the image information obtained in step S31, inputting the image depth recovery model trained in step S2 to perform image recovery processing on the geometric information of the image, so as to output an image containing correct depth information;

[0110] The equidistant cylindrical projection will cause the figure to produce serious distortion, and then affect the depth estimation, the embodiment proposes a spherical convolution kernel to carry out the depth estimation of panoramic image, the processing process of the figure is regarded as the rotation change of three-dimensional coordinates, the image to be processed is SO(3), which is a three-dimensional rotation group, that is, the set composed of the initial and ending states in the rotation process, and the matrix form expression of the rotation operation is as follows:

[0111]

[0112]

[0113] Wherein, θ represents the rotation angle corresponding to the rotation operation in three-dimensional space, R x (θ), R y (θ), R z (θ) respectively represent the rotation operation in each dimension in the three-dimensional coordinate system, R x (θ), R y (θ), R z (θ) corresponding matrix respectively is the mathematical expression corresponding to each operation.

[0114] When extracting features, the output in the plane is the inner product of the input feature map and the filter, which is also applicable to the sphere and the three-dimensional rotation group. In the calculation process, the embodiment defines R as the rotation group on SO(3), and because it has the invariance of the measure, it can be efficiently substituted in the integral operation. The inner product function expression of the spherical convolution on the sphere S 2 is as follows:

[0115]

[0116] Wherein, S 2 represents the sphere, represents the pixel function of each position on the sphere, f k (x) represents the convolution kernel function, k represents the kth channel, K represents the total number of image channels, and x represents the independent variable.

[0117] In the trained image depth recovery model, the decoding part has 5 large layers, wherein the convolution kernel of the convolution part in the first layer is convolved on the sphere S 2 , and the convolution kernel of the convolution part starts from the second layer and is convolved in the three-dimensional rotation group SO(3). When operating in the three-dimensional rotation group, because SO(3) is composed of linear transformations that keep vectors unchanged in three-dimensional Euclidean space, and the vectors in the three-dimensional coordinate system can be represented by the standard orthogonal basis, therefore, for any element in the three-dimensional rotation matrix, there is an associated expression as follows:

[0118]

[0119] where x j denotes the jth component of the original three-dimensional vector x in the standard basis, j is the column index; x ′i denotes the ith component of the rotated three-dimensional vector x' in the standard basis, i is the row index. denotes the element of the rotation matrix R, i.e., the mapping from the jth axis of the original coordinate system to the ith axis of the rotated coordinate system. The three-dimensional representation of SO(3) in the geometric space is shown in Figure 3 ω h denotes the angular range covered by the spherical convolution kernel in the three-dimensional space, a ω denotes the angular size corresponding to the projection of the angular range in the xOz plane, a h denotes the angular size corresponding to the projection of the angular range in the xOy plane; r ω h denotes the pixel range (i.e., resolution range) covered by the spherical convolution kernel in the three-dimensional space, r ω denotes the number of pixels contained in the projection of the convolution kernel in the width direction of the original image, r h denotes the number of pixels contained in the projection of the convolution kernel in the height direction of the original image.

[0120] In the observation process, based on the Euler angle decomposition, the form of motion occurring in SO(3) is represented as the product of the function of twice rotation around a single coordinate axis and the function of motion around another coordinate axis, without the comprehensive involvement of three coordinate axes. First, the coordinate axes are regarded as the regular inertial reference axes, and the motion on two coordinate axes is analyzed. Then, the coordinate axes are regarded as the carrier system with the motion center at the origin as the carrier, and the form of describing the motion is roll-pitch, then the expression of the motion process is given as follows:

[0121] p = R z ( a ) R y ( b ) R z ( g ) (12)

[0122] where R z ( a ) R y ( b ) R z ​​​(g) respectively represent the rotation operation in each dimension of the three-dimensional coordinate system, and a, b, g respectively represent the included angle of the rotation operation with the three coordinate axes x, y, z; the inner product of the function on the rotation group and R can be regarded as the Hadamard product of the matrix, and when the convolution is performed, the isomorphism is satisfied. However, due to the high time complexity of directly calculating the picture depth, a fast Fourier transform algorithm is selected for optimization. In order to make the spherical signal meet the form that can be processed by the fast Fourier transform, the Wigner D function is involved in the calculation in the embodiment, and the Wigner D equation is derived from the spherical harmonic function SH. The spherical harmonic function itself is also a special orthogonal basis form of the Laplace operator.

[0123] The trained image depth recovery model uses the Wigner d function (non-traditional Wigner D function) as the basis to convert the spherical convolution process into a form that can be processed by the fast Fourier transform algorithm, as follows:

[0124] First, the expression of the Wigner D function is set as follows:

[0125]

[0126] Wherein, J, M, M' represent three main indexes represented by the function; J is a non-negative integer, and satisfies -J < M, M' < J.

[0127] The second term is the Wigner d function, and the expansion expression is as follows:

[0128]

[0129] Wherein, u represents the absolute value of the difference between M and M', v represents the absolute value of the sum of M and M', and s represents the difference between J and (u+v) / 2, The Jacobi polynomial is represented by J. The expansion expression of the first term of the Wigner d function is as follows:

[0130]

[0131] Wherein, if J satisfies M and M', the value of the Wigner d function is 0. The pre-calculated sample array of the Wigner d function is used as a standard two-dimensional translation fast Fourier transform array. Since the space signal in SO(3) is a discrete three-dimensional array, the processing process undergoes two-dimensional translation and scaling, and therefore the Wigner d function plays a crucial role in the operation.

[0132] In the three-dimensional rotation group, the set of Wigner D functions can constitute a complete set of orthogonal group functions. Integrating the functions on SO(3) can be converted into the product of a normalization factor and three Kronecker functions. This means that it has extremely sensitive 0, 1 judgment ability and is suitable for use as a basis.

[0133] In this embodiment, N represents a three-dimensional manifold, and U represents a basis function. The expression of the GFT transform (graph Fourier transform) is as follows:

[0134]

[0135] where X represents a certain point on the three-dimensional manifold on which the spherical convolution is performed, represents the lth frequency component of the function f after the graph Fourier transform, that is, the spectral coefficient; f(X) represents a function defined on the graph N, that is, the signal value of a certain point on the three-dimensional manifold; represents the basis function U l represents the value corresponding to the complex conjugate function of the node X; the basis function U l is an orthogonal basis function in the graph Fourier transform, corresponding to the eigenvector of the graph Laplacian operator. S 2 is used when the manifold is SO(3) l (X) is a vector-valued Y l (X), U l (X) takes a matrix value D l According to the inverse derivation of the Wigner d function, the expression of the point on SO(3) is as follows:

[0136]

[0137] According to formula (16), the GFFT algorithm is used for formula (17), and the expression is as follows:

[0138]

[0139] where D' represents the complex conjugate function of the Wigner d function.

[0140] In this embodiment, the step S4 is specifically as follows:

[0141] This embodiment references the basic principles of neural radiation fields, but utilizes an elliptical Gaussian heterogeneous kernel as the foundational scene representation method. After generating a sparse point cloud using SfM, instead of using Colmap for dense point cloud reconstruction, a 3D Gaussian kernel is selected as the primitive. Since sparse point cloud images have significant noise, determining the normals is difficult, making the conventional method of calculating derivatives using numerical gradients and backpropagating challenging. Therefore, this embodiment uses a covariance matrix to define ensemble information. The value of this matrix is ​​determined by the average value of the region space corresponding to the Gaussian kernel.

[0142] Use a three-dimensional column vector v = (x0, x1, x2) T Let x0 and x1 represent a point in the ray space. Given a central projection and a projection plane: coordinates x0 and x1 specify a point on the projection plane, and the third coordinate x2 specifies the Euclidean distance from the projection center to the point on the viewing ray. Then the light intensity along the ray path... The expression is as follows:

[0143]

[0144] Where T represents the transpose operation, and μ is the integration variable, which in the integral expression represents a position along the ray from the ray exit point (i.e., the projection center) to point ξ. This represents the position of a point on the ray when the camera is taken as the origin; L represents the distance from the ray's exit point to... The distance at the point; ξ represents the direction of the ray, and is expressed using a 3D Gaussian unit vector. Represents the emission coefficient, i.e., point emission coefficient. The color at that location; g(x) represents the astigmatism function that defines the light occlusion rate; Indicates in The emission coefficient along the ray direction ξ, with the exponent term being the attenuation coefficient; It is a light source item, and its description is in The intensity of light scattered along the ray direction ξ at a given point. Astigmatism function. It is a Gaussian reconstruction kernel The weighted sum is expressed as follows:

[0145]

[0146] Among them, g k Indicates the kth G The weighting coefficient represents the k-th weight. G Gaussian reconstruction kernel Astigmatism function The degree of contribution. Gaussian reconstruction kernel It is the kth G A reconstruction kernel function, describing the location... The light scattering or attenuation properties of the particles are related to the position and shape of the individual particles. The spacing of the particles can be irregular and have different shapes. Therefore, equation (19) is not limited to regular data sets. Substituting equation (20) into equation (19) gives:

[0147]

[0148] The emission coefficient is then set to be a constant Co, and the first two terms of the Taylor expansion are used as an approximation of the exponential, giving the simplified expression of equation (21) as follows:

[0149]

[0150] where represents the integral reconstruction kernel, i.e., the Gaussian reconstruction kernel , and

[0151] Gaussian spattering shares the same image generation model as the neural radiance field, but the Gaussian reconstruction kernel has significant advantages in mathematical principles. The point defined by the Gaussian reconstruction kernel is an elliptical geometry that is rotated and stretched in any direction, which is mainly determined by two characteristics, i.e., the covariance matrix and the spherical harmonic function. According to its covariance matrix, each point can represent a limited area close to its mean value in space. The spherical harmonic function is a special function defined on the surface of a sphere, which can calculate such a function for any point on the sphere and obtain a value. This allows us to accurately define the relevant colors of the view. The advantage of the Gaussian reconstruction kernel is that it is closed during the affine transformation and resonance process, and the integral of it can convert a three-dimensional Gaussian to two dimensions. The reconstruction kernel r(x) only performs two-dimensional convolution calculation. This makes the stitching algorithm have higher rendering efficiency.

[0152] Finally, the SfM motion recovery model completes the reconstruction of the image surface. For the output reconstructed image, the originally curved straight line should become close to a straight line; if the image contains a rectangular object, the four edges of the reconstructed rectangular image should be close to straight line segments, and the four corners should be close to 90° right angles; part of the ellipse in the original image should be close to a circle after reconstruction; the angle of part of the angle in the original image should be reduced to a certain extent after reconstruction.

[0153] In this embodiment, after the three-dimensional reconstruction of the image scene is completed in step S5, the reconstruction performance evaluation index of the Gaussian spattering monocular reconstruction model is as follows:

[0154] ​To comprehensively evaluate the performance of the method of the present application, the evaluation indexes used in this embodiment include: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) and learned perceptual image patch similarity (LPIPS). These indexes jointly measure the quality of the reconstructed image, the similarity to the original image, and the difference in visual perception.

[0155] (1) Peak signal-to-noise ratio (PSNR);

[0156]

[0157]

[0158] wherein, MSE represents mean square error; m, n respectively represent the width and height of the image, I(i,j) represents the pixel value of the image before processing at the (i,j) position, K(i,j) represents the pixel value of the image after processing at the (i,j) position, MAX I represents the maximum possible pixel value in the image.

[0159] (2) Structural similarity (SSIM);

[0160]

[0161] wherein, u x , u y respectively represent the mean of the pixels of the images X and Y, the images X and Y respectively represent the images before and after reconstruction via the Gaussian splash monocular reconstruction model; c1, c2 are constants, in order to avoid the case that the denominator is 0, c1=(K1×L) 2 , c2=(K2×L) 2 , in this embodiment, K1=0.01, K2=0.03, L=255, K1 and K2 represent two constants for avoiding the denominator being zero when calculating SSIM, and L represents the dynamic range of the pixel value, i.e. the maximum possible value of the pixel value in the image; σ xy represents the covariance between the pixel values of the images X and Y, which measures the correlation of the corresponding pixel values in the two images; represents the variance of the pixel values of the image X; represents the variance of the pixel values of the image Y.

[0162] (3) Learned perceptual image patch similarity (LPIPS);

[0163]

[0164] wherein, d(p,p0) represents the distance measure between two image blocks p and p0, which is used to measure the perceptual similarity between them; H l , W l represent the length and width of the output of the l-th layer; wl weights are used to weight and scale the output of each channel; and respectively represent the activation output of the l-th layer network for the input image patches p and p0 at position (h, w), h and w represent the spatial coordinates on the image.

[0165] As Figure 4 shown, the embodiment further carries out experimental verification, selects a video as a material, uses time-lapse photography to shoot a plurality of panoramic pictures, and puts the pictures into the Gaussian splashing monocular reconstruction model for training, and the embodiment sets a breakpoint in the training process, so that the depth recovery process is visualized.

[0166] wherein, Figure 4 (a) is a partial panoramic image, Figure 4 (b) is a breakpoint image recovery state, which shows the recovery effect at the breakpoint position in the image recovery process; Figure 4 (c) is the reconstruction result of the method of the application, and it can be seen that the method of the application has obvious improvement in detail recovery and visual quality. The model training effect in the embodiment can be known by the evaluation index.

[0167] wherein, SSIM: 0.77, which indicates that the reconstructed image has high similarity with the original image in structural details; PSNR: 26.21, which shows a low noise level and a high signal fidelity; LPIPS: 0.268, which is a low value, indicating that the reconstructed image is closer to the expectation of human vision in perceptual quality.

[0168] In summary, the method of the application is composed of an image depth recovery model and an SfM motion recovery model, the image depth recovery model is composed of a pre-training network and a prediction network of an encoding-decoding structure, and the SfM motion recovery model uses a Gaussian reconstruction kernel to reconstruct a geometric body. The model overturns the previous mode of training with one hundred pictures, and only uses one picture for reconstruction, which has significantly improved the operability. Moreover, the method of the application uses the Gaussian splashing method to replace the traditional dense point cloud reconstruction, strips the neural network with relatively high computing power requirement, greatly improves the calculation speed, and has practical engineering application significance. In medical diagnosis work, the model shows high sensitivity and adaptability.

[0169] Those skilled in the art will appreciate that the embodiments described herein are presented for the purpose of helping the reader to understand the principles of the application, and should be understood as not limiting the scope of protection of the application to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations according to the technical inspiration disclosed in the application, which do not deviate from the essence of the application, and these modifications and combinations are still within the scope of protection of the application.

Claims

1. A Gaussian splash monocular reconstruction method based on neural networks, the specific steps of which are as follows: S1. Construct a Gaussian splash monocular reconstruction model that includes an image depth restoration model and an SfM motion restoration model; The image depth restoration model is used for scene depth restoration of monocular panoramic images, including: The pre-trained network and the prediction network are both based on a spherical convolution encoder-decoder structure, with the encoder being a fixed VGG16. The SfM motion recovery model is used to reconstruct sparse point clouds from the recovered images. In the spherical convolution-based encoder-decoder structure, the encoder includes: a spherical convolutional layer, a pooling layer, and a loss function containing cross-entropy and regularization; the encoding model has a total of 5 main layers, each including a spherical convolutional layer and a pooling layer; the encoding part contains 18 sub-layers in total, including the spherical convolutional layers and pooling layers; the decoder includes: 5 stacked deconvolutional layers and a ReLU activation function; and the decoder is skip-connected to the encoder, using upsampling on the output. S2. Based on step S1, train the image depth restoration model; Points in the image corresponding to the imaging space are view-encoded to form a supervised learning sequence, which serves as the input to the pre-trained network and the prediction network. The image depth restoration model embeds the Adam adaptive moment estimation algorithm to optimize the learning rate, while the backpropagation algorithm is used to update the network weights. The two algorithms are used to train the pre-trained network and the prediction network in sequence. The gradient change of each parameter is estimated according to the loss function, and the learning rate and learning step size of the parameters are dynamically adjusted. The distance to move in the opposite direction of the gradient is determined each time, and the weights of the neural network model are continuously updated. The loss function and accuracy curve of the model output are observed. When the parameters in the model converge and the model accuracy reaches or approaches the preset accuracy, the training process is terminated, and the training is completed. Adam calculates the average m of an exponentially decaying historical gradient. t It also calculates the average v of an exponentially decaying historical squared gradient. t Its calculation expression is as follows: m t =β1m t-1 +(1-β1)g t (2) v t =β2v t-1 +(1-β2)g t 2 (3) Where θ represents any parameter in the model, f t (θ) represents the random function value of the noise objective function f(θ) at subsequent time step t; m t Let m represent the estimate of the first moment of the gradient. t-1 v represents the estimate of the first moment of the gradient before the update. t Let v represent the estimate of the second moment of the gradient. t-1 β1 represents the estimated second moment of the gradient before the update, β2 represents the exponential decay rate of the first moment estimate, and β2 represents the exponential decay rate of the second moment estimate; g t The function f represents the expression. t (θ) Gradient with respect to θ; In Adam, bias is offset by calculating the first-order moment estimate and the second-order moment estimate for bias correction, as shown in the following expression: in, This represents the first moment estimate at time step t. Let represent the second moment estimate at the current time step t. The parameter update formula for generating Adam is as follows: Where, θ t+1 The parameter value θ represents the time step t+1. t Let represent the parameter value at time step t, η represent the learning rate, and ∈ represent the bias value, where ∈ = 10. -8 ; S3. Acquire image information and input the image depth restoration model trained in step S2 to process the geometric information of the image and complete the scene depth restoration of the monocular panoramic image. S4. Input the scene depth restored image obtained in step S3 into the SfM motion restoration model to perform sparse point cloud reconstruction, and use the Gaussian reconstruction kernel to reconstruct the image surface to form the basic geometric configuration of the image. S5. Apply image processing techniques to the image reconstructed in step S4 to perform Gaussian low-pass filtering for noise reduction, forming a complete surface structure of the object and completing the three-dimensional reconstruction of the image scene.

2. The Gaussian splash monocular reconstruction method based on neural networks according to claim 1, characterized in that, Step S3 is as follows: S31. Use a camera to capture a monocular panoramic image of the object and obtain image information; Use a camera and design a turntable to capture images of an object from all directions with a single viewpoint. Use this camera to capture a monocular panoramic image of the object and obtain image information. The resulting images include random high-frequency noise; S32. Based on the image information obtained in step S31, input it into the image depth restoration model trained in step S2 to perform image restoration processing on the geometric information of the image, so that the output image contains the correct depth information. If we consider the image processing process as a rotational transformation of three-dimensional coordinates, and the image being processed is a three-dimensional rotation group SO(3), that is, the set consisting of the initial and final states during the rotation process, then the matrix form expression of the rotation operation is as follows: Where θ represents the rotation angle corresponding to the rotation operation in three-dimensional space, and R x (θ), R y (θ), R z (θ) represent rotation operations in each dimension of the three-dimensional coordinate system, R x (θ), R y (θ), R z The matrices corresponding to (θ) are the mathematical representations of each operation; Spherical convolution on spherical surface S 2 The expression for the inner product function on the above is as follows: Among them, S 2 To represent a sphere, f is a pixel function representing each position on the sphere. k (x) represents the convolution kernel function, k represents the k-th channel, K represents the total number of image channels, and x represents the independent variable; In the trained image depth restoration model, the decoding part consists of 5 layers, where the convolution kernel in the first layer is located on a sphere S. 2 The convolution kernels of the convolutional part starting from the second layer are convolved in the three-dimensional rotation group SO(3); when operating in the three-dimensional rotation group, for any element in the three-dimensional rotation matrix, the associated expression is as follows: Where, x j This represents the j-th component of the original three-dimensional vector x under the standard basis, where j is the column index; x′ i Let represent the i-th component of the rotated 3D vector x′ under the standard basis, where i is the row index; The elements of the rotation matrix R represent the mapping from the j-th axis of the original coordinate system to the i-th axis of the rotated coordinate system. Based on Euler angle decomposition, the motion in SO(3) is represented as the product of a function of two rotations around a single coordinate axis and a function of motion around another coordinate axis. First, the coordinate axes are regarded as conventional inertial reference axes, and the motion on the two coordinate axes is analyzed. Then, the coordinate axes are regarded as a system of motion with the origin as the center of motion, and the motion is described as roll-pitch. The expression of the motion process is given as follows: p=R z (a)R y (b)R z (c) (12) Among them, R z (α)R y (β)R z (γ) represent rotation operations in each dimension of the three-dimensional coordinate system, and α, β, and γ represent the angles between the rotation operation and the x, y, and z coordinate axes, respectively. The inner product of R and the function on the rotation group can be regarded as the Hadamard product of matrices, which satisfies equivariance when convolved. The Fast Fourier Transform algorithm is selected for optimization. The trained image depth restoration model uses the Wigner d function as a basis to transform the spherical convolution process into a form that can be subjected to the Fast Fourier Transform algorithm, as follows: First, define the expression for the Wigner D function as follows: Where J, M, and M' represent the three main indices characterized by the influence function; J is a non-negative integer and satisfies -J <M,M'<J; The second term is the Wigner d function, whose expanded expression is as follows: Where u represents the absolute value of the difference between M and M', v represents the absolute value of the sum of M and M', and s represents the difference between J and (u+v) / 2. Let represent a Jacobian polynomial; then the expansion expression of the first term of the Wigner d function is as follows: Wherein, if J satisfies greater than M and M', the value of the Wigner d function is 0; the pre-computed sample array of the Wigner d function is used as a standard two-dimensional translational fast Fourier transform array; Let N represent a 3D manifold and U represent basis functions, then the expression for the GFT transformation is as follows: Where X represents a point on a 3D manifold where the sphere is convolved. The spectral coefficient represents the l-th frequency component of the function f after the graph Fourier transform; f(X) represents a function defined on graph N, i.e., the signal value at a point on the three-dimensional manifold. Representing the basis function U l The value of the complex conjugate function at node X; basis function U l These are the orthogonal basis functions in the graph Fourier transform, corresponding to the eigenvectors of the graph Laplace operator; using S... 2 manifold time U l (X) represents the vector value Y. l (X), U when using SO(3) manifold l (X) Take matrix value D l Based on the reverse derivation of the Wigner d function, the expression for the point on SO(3) is obtained as follows: Based on equation (16), applying the GFFT algorithm to equation (17) yields the following expression: Among them, D ′ This represents the complex conjugate function of the Wigner d function.

3. The Gaussian splash monocular reconstruction method based on neural networks according to claim 1, characterized in that, Step S4 is as follows: After generating sparse point clouds using the SfM motion recovery model, a three-dimensional Gaussian is selected as the primitive, and the covariance matrix is ​​used to define the set information. The value of this matrix is ​​determined by the average value of the region space corresponding to the Gaussian kernel. Use a three-dimensional column vector v = (x0, x1, x2) T Let x0 and x1 represent a point in the ray space; given a central projection and a projection plane: coordinates x0 and x1 specify a point on the projection plane, and the third coordinate x2 specifies the Euclidean distance from the projection center to the point on the viewing ray; then the light intensity along the ray path... The expression is as follows: Where T represents the transpose operation, and μ is the integration variable, which in the integral expression represents a position along the ray from the ray exit point to point ξ. This represents the position of a point on the ray when the camera is taken as the origin; L represents the distance from the ray's exit point to... The distance at the point; ξ represents the direction of the ray, and is expressed using a 3D Gaussian unit vector. Represents the emission coefficient, i.e., point emission coefficient. The color at that location; g(x) represents the astigmatism function that defines the light occlusion rate; Indicates in The emission coefficient along the ray direction ξ, with the exponent term being the attenuation coefficient; It is a light source item, and its description is in The intensity of light scattered along the ray direction ξ; astigmatism function It is a Gaussian reconstruction kernel The weighted sum is expressed as follows: in, Indicates the kth G The weighting coefficient represents the k-th weight. G Gaussian reconstruction kernel Astigmatism function The degree of contribution; Gaussian reconstruction kernel It is the kth G A reconstruction kernel function, describing the location The light scattering or attenuation characteristics at a certain point reflect the position and shape of a single particle; substituting equation (20) into equation (19), we get: Then set the emission coefficient. Let C0 be a constant, and use the first two terms of the Taylor expansion as approximations of the exponent, to obtain the simplified expression of equation (21) as follows: in, This refers to the integral reconstruction kernel, also known as the Gaussian reconstruction kernel. The integral, and Finally, the SfM motion recovery model completes the reconstruction of the image surface; for the output reconstructed image, the originally curved straight lines should become close to straight lines; if the image contains rectangular objects, the four sides of the reconstructed rectangular image should be close to straight line segments, and the four vertex angles should be close to 90° right angles; some ellipses in the original image should be close to circles after reconstruction; some included angles in the original image should be reduced to a certain extent after reconstruction.

4. The Gaussian splash monocular reconstruction method based on neural networks according to claim 1, characterized in that, In step S5, after completing the 3D reconstruction of the image scene, the specific performance evaluation indicators of the Gaussian splash monocular reconstruction model are as follows: The evaluation metrics include: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learning-Aware Image Patch Similarity (LPIPS). (1) Peak signal-to-noise ratio (PSNR); Where MSE represents the mean squared error; m and n represent the width and height of the image, respectively; I(i,j) represents the pixel value at position (i,j) in the image before processing; K(i,j) represents the pixel value at position (i,j) in the image after processing; and MAX... I This represents the maximum possible pixel value in the image; (2) Structural similarity (SSIM); Among them, u x u y Let c1 and c2 represent the mean pixel values ​​of images X and Y, respectively, where images X and Y represent the images before and after reconstruction using the Gaussian splash monocular reconstruction model; c1 and c2 are constants, and satisfy c1 = (K1 × L). 2 c2 = (K2 × L) 2 K1 and K2 represent two constants to avoid zero denominators when calculating SSIM, and L represents the dynamic range of pixel values, i.e., the maximum possible pixel value in the image; σ xy Covariance represents the covariance between pixel values ​​in image X and image Y, measuring the correlation between corresponding pixel values ​​in the two images; The variance of the pixel values ​​in image X; The variance of the pixel values ​​in image Y; (3) Learning Perceptual Image Patch Similarity LPIPS; Where d(p,p0) represents the distance metric between two image patches p and p0; H l W l This represents the length and width of the output of layer l; w l Weights are used to scale the output of each channel. and These represent the activation outputs of the l-th layer network for input image blocks p and p0 at positions (h, w), respectively, where h and w represent the spatial coordinates on the image.

Images