Gaussian splashing monocular reconstruction method based on neural network

Abstract

The invention discloses a Gaussian splash monocular reconstruction method based on a neural network, and the method comprises the steps: constructing a Gaussian splash monocular reconstruction model which comprises an image depth recovery model and an SfM motion recovery model, training the image depth recovery model, shooting a panoramic picture, training the panoramic picture through spherical convolution, recovering a distortion part, and carrying out the reconstruction of the Gaussian splash monocular reconstruction model; then sparse point cloud reconstruction is carried out through an SfM motion recovery model, a Gaussian reconstruction core is used for reconstructing the surface of the image to form a basic geometric configuration of the image, and finally, an image processing technology is used for carrying out Gaussian low-pass filtering noise reduction processing on the image to form a complete surface structure of an object, and three-dimensional reconstruction of an image scene is completed. According to the method, the three-dimensional Gaussian splashing technology is combined with the spherical convolution kernel to construct the geometric model of the object, the operability is remarkably improved, the calculation speed is greatly increased, and the method has important application value in the fields of intelligent medical treatment, engineering flaw detection and the like.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a Gaussian splashing monocular reconstruction method based on a neural network. Background Art

[0002] With the rapid development of machine learning, high-precision 3D surface reconstruction technology is increasingly being used in various fields, such as reverse engineering, digital modeling, medical diagnosis, virtual reality, etc. This technology can provide more accurate and intuitive graphic expressions by reshaping and modeling the geometric shapes of objects. In an era when artificial intelligence is prevalent, the technical level of 3D reconstruction is crucial to the development of computer technology.

[0003] The document "R. Hartley and A. Zisserman, 'Multiple View Geometry in Computer Vision, '2nd ed. Cambridge: Cambridge University Press, 2004." proposed a 3D reconstruction method, which geometrically understands and analyzes the projection process from 3D to 2D, and designs a solution to the inverse problem from 2D to 3D. However, the existing 3D reconstruction usually requires a large number of images with known camera parameters, and performs multiple steps such as camera parameter estimation, dense point cloud reconstruction, and surface reconstruction, which requires a huge amount of computing power. In contrast, algorithms based on deep learning have the advantage of significantly reducing computing power. In neural networks, processing data extracted from images becomes a process of encoding and decoding. With the increase in feature point selection, the 3D model of an object can be reconstructed from a single or multiple 2D images without complex camera calibration. At present, there are three mainstream 3D reconstruction forms: point cloud-based 3D reconstruction, voxel-based 3D reconstruction, and grid-based 3D reconstruction. Among them, the advent of point cloud reconstruction method Nerf has made neural radiation field mainstream, but because it uses neural networks for dense reconstruction, it still requires high GPU computing power.

[0004] While work on recovering layouts from existing images via geometry and deep learning has progressed rapidly, there is still little work on leveraging panoramic images to address these challenges. Panoramic cameras have the potential to improve the performance of the task: their 360° field of view captures the entire field around their optical center, allowing for a large amount of information to be acquired at once, leading to better layout recovery. Summary of the invention

[0005] To solve the above technical problems, the present invention provides a Gaussian splattering monocular reconstruction method based on a neural network. From the perspective of engineering application, it combines three-dimensional Gaussian with monocular reconstruction technology, and uses panoramic image shooting and spherical convolution kernel image depth recovery technology to reduce the number of images required for previous three-dimensional reconstruction technology, greatly speeding up the speed of three-dimensional reconstruction, and has important application value.

[0006] The technical solution adopted by the present invention is: a Gaussian splash monocular reconstruction method based on neural network, and the specific steps are as follows:

[0007] S1. Build a Gaussian splash monocular reconstruction model including image depth recovery model and SfM motion recovery model;

[0008] The image depth recovery model is used to recover the scene depth of a monocular panoramic image, including: a pre-trained network and a prediction network. The specific structures of the pre-trained network and the prediction network are both encoding-decoding structures based on spherical convolution, and the encoder is fixedly VGG16. The SfM motion recovery model is used to reconstruct sparse point clouds of the recovered image.

[0009] Among them, in the encoding-decoding structure based on spherical convolution, the encoder includes: spherical convolution layer, pooling layer, loss function including cross entropy and regularization; and the encoding model has 5 main layers, each of which includes a spherical convolution layer and a pooling layer; the encoding part spherical convolution layer and pooling layer contain 18 sublayers in total. The decoder includes: 5 stacked deconvolution layers and Relu activation function. The decoder is jump-connected with the encoder, and upsampling is used for the output.

[0010] S2, based on step S1, training an image depth recovery model;

[0011] The points in the image corresponding to the imaging space are view-encoded to form a supervised learning sequence as the input of the pre-trained network and the prediction network. The image depth recovery model is embedded with the adaptive moment estimation algorithm Adam to optimize the learning rate, and the error back propagation algorithm is used to update the network weights. The two algorithms are used to train the pre-trained network and the prediction network in sequence, and the gradient change of each parameter is estimated according to the loss function, and the learning rate and learning step of the parameter are dynamically adjusted to determine the distance moved in the opposite direction of the gradient each time. The weight of the neural network model is continuously updated, and the loss function and accuracy curve of the model output are observed. When the parameters in the model converge and the model accuracy reaches or approaches the preset accuracy, the training process is terminated and the training is completed.

[0012] Among them, Adam calculates an exponentially decaying average m of historical gradients t , and also calculates an exponentially decaying average of the historical squared gradient v t , and its calculation expression is as follows:

[0013]

[0014] m t = β1m t-1 + (1-β1)g t (2)

[0015] v t = β2v t-1 + (1-β2)g t 2 (3)

[0016] Among them, θ represents any parameter in the model, f t (θ) represents the random function value of the noise objective function f(θ) at the subsequent time step t. t Represents an estimate of the first-order moment of the gradient, m t-1 represents the estimate of the first-order moment of the gradient before the update, v t represents the estimate of the second-order moment of the gradient, v t-1 g represents the estimate of the second-order moment of the gradient before the update, β1 represents the exponential decay rate of the first-order moment estimate, and β2 represents the exponential decay rate of the second-order moment estimate. t Represents the function f t (θ) is the gradient of θ. In Adam, the deviation is offset by calculating the first-order moment estimate and the second-order moment estimate of the deviation correction. The calculation expression is as follows:

[0017]

[0018] in, represents the first-order moment estimate at time step t, Represents the second-order moment estimate of the current time step t, and the parameter update formula for generating Adam is as follows:

[0019]

[0020] Among them, θ t+1 represents the parameter value at time step t+1, θ t represents the parameter value at time step t, η represents the learning rate, ∈ represents the bias value, and ∈=10 -8 .

[0021] S3, collecting image information, inputting the image depth recovery model trained in step S2 to process the geometric information of the image, and completing the scene depth recovery of the monocular panoramic image;

[0022] S4, inputting the scene depth restored image obtained in step S3 into the SfM motion recovery model, reconstructing the sparse point cloud, and reconstructing the image surface with a Gaussian reconstruction kernel to form the basic geometric configuration of the image;

[0023] S5. Performing Gaussian low-pass filtering and noise reduction processing on the image reconstructed in step S4 using image processing technology to form a complete surface structure of the object and complete the three-dimensional reconstruction of the image scene.

[0024] Furthermore, the step S3 is specifically as follows:

[0025] S31, using a camera to capture a monocular panoramic image of the object to obtain image information;

[0026] A camera is used and a turntable is set for the camera so that it can capture objects in all directions with a single perspective. The camera is used to capture a monocular panoramic image of the object to obtain image information.

[0027] Among other things, the resulting image includes random high-frequency noise.

[0028] S32, based on the image information obtained in step S31, inputting it into the image depth recovery model trained in step S2 to perform image recovery processing on the geometric information of the image, so that it outputs an image containing correct depth information;

[0029] The graphics processing process is regarded as the rotation change of three-dimensional coordinates. The processed image is the three-dimensional rotation group SO(3), that is, the set of starting and ending states in the rotation process. The matrix form of the rotation operation is as follows:

[0030]

[0031] Among them, θ represents the rotation angle corresponding to the rotation operation in three-dimensional space, R x (θ), R y (θ), R z (θ) represents the rotation operation in each dimension in the three-dimensional coordinate system, R x (θ), R y (θ), R z The matrices corresponding to (θ) are the mathematical representations of each operation.

[0032] Spherical convolution on the sphere S 2 The inner product function expression on is as follows:

[0033]

[0034] Among them, S 2 represents a sphere, Represents the pixel function at each position on the sphere, fk (x) represents the convolution kernel function, k represents the kth channel, K represents the total number of image channels, and x represents the independent variable.

[0035] In the trained image depth recovery model, the decoding part has 5 layers, among which the convolution kernel of the convolution part in the first layer is on the spherical surface S 2 In the up-convolution, the convolution kernel of the convolution part starting from the second layer is convolved in the three-dimensional rotation group SO(3). When operating in the three-dimensional rotation group, for any element in the three-dimensional rotation matrix, the associated expression is as follows:

[0036]

[0037] Among them, x j represents the jth component of the original three-dimensional vector x in the standard basis, where j is the column index; x′ i represents the i-th component of the rotated three-dimensional vector x′ in the standard basis, where i is the row index. Represents the elements of the rotation matrix R, that is, the mapping from the j-th axis of the original coordinate system to the i-th axis of the rotated coordinate system.

[0038] Based on the Euler angle decomposition, SO(3) expresses the motion as the product of two rotations around a single coordinate axis and a function of motion around another coordinate axis. First, the coordinate axis is regarded as a conventional inertial reference axis, and the motion on the two coordinate axes is analyzed. Then, the coordinate axis is regarded as a carrier system with the origin as the carrier of the motion center, and the form of describing the motion is roll-pitch. The expression of the motion process is given as follows:

[0039] ρ=R z (α)R y (β)R z (γ) (12)

[0040] Among them, R z (α)R y (β)R z (γ) represents the rotation operation in each dimension in the three-dimensional coordinate system, α, β, γ represent the angles between the rotation operation and the three coordinate axes x, y, and z respectively; the inner product of R and the function on the rotation group can be regarded as the Hadamard product of the matrix, which satisfies the equivariance when performing convolution. The fast Fourier transform algorithm is selected for optimization.

[0041] The trained image depth recovery model uses the Wigner d function as a basis to transform the spherical convolution process into a form that can be used for the fast Fourier transform algorithm, as follows:

[0042] First, the expression of the Wigner D function is set as follows:

[0043]

[0044] Among them, J, M, and M' represent three main indicators that affect the characterization of the function; J is a non-negative integer and satisfies -J <M,M'<J。

[0045] Then the second term is the Wigner d function, and its expanded expression is as follows:

[0046]

[0047] Among them, u represents the absolute value of the difference between M and M', v represents the absolute value of the sum of M and M', and s represents the difference between J and (u+v) / 2. represents the Jacobi polynomial. Then the first term of the Wigner d function is expanded as follows:

[0048]

[0049] If J satisfies the condition that it is greater than M and M', the value of Wigner's d function is 0. The pre-calculated sample array of Wigner's d function is used as a standard two-dimensional translation fast Fourier transform array.

[0050] Assuming N represents the three-dimensional manifold and U represents the basis function, the expression for GFT transformation is as follows:

[0051]

[0052] Where X represents a point on the three-dimensional manifold of the spherical convolution. represents the lth frequency component of function f after Fourier transform of the graph, i.e., the spectral coefficient; f(X) represents a function defined on the graph N, i.e., the signal value of a point on the three-dimensional manifold; Represents the basis function U l The value corresponding to the complex conjugate function at the node X; the basis function U l is the orthogonal basis function in the graph Fourier transform, corresponding to the eigenvector of the graph Laplacian operator. 2 Manifold Time U l (X) is the vector value Y l (X), when using SO(3) manifold U l (X) Take the matrix value D l , according to the reverse derivation of Wigner d function, the expression of the point on SO(3) is as follows:

[0053]

[0054] According to formula (16), using GFFT algorithm for formula (17), the expression can be obtained as follows:

[0055]

[0056] Among them, D ′ Represents the complex conjugate function of the Wigner d function.

[0057] Furthermore, the step S4 is specifically as follows:

[0058] After the SfM motion recovery model generates a sparse point cloud, a three-dimensional Gaussian is selected as the primitive, and the covariance matrix is ​​used to define the set information. The value of the matrix is ​​determined by the average value of the regional space corresponding to the Gaussian kernel.

[0059] Use a three-dimensional column vector v = (x0, x1, x2) T To represent a point in ray space. Given a central projection and a projection plane: coordinates x0 and x1 specify a point on the projection plane, and a third coordinate x2 specifies the Euclidean distance from the projection center to the point on the viewing ray. Then the light intensity on the light path is The expression is as follows:

[0060]

[0061] Where T represents the transpose operation, μ is the integral variable, and in the integral expression represents a position along the ray from the ray exit point to the point ξ. It indicates the position of the point on the ray when the camera is taken as the origin; L indicates the distance from the ray exit point to the ray exit point. ξ represents the irradiation direction of the ray and is represented in the form of a 3D Gaussian unit vector; Represents the emission coefficient, that is, point The color at the point; g(x) represents the scattering function that defines the light shielding rate; Indicated in The emission coefficient along the ray direction ξ at , and the exponential term is the attenuation coefficient; is the light source term, which is described in The intensity of light scattered along the ray direction ξ at . is the Gaussian reconstruction kernel The weighted sum of is expressed as follows:

[0062]

[0063] Among them, g kG Indicates the kth G weighting coefficient, which represents the kth G Gaussian reconstruction kernel Astigmatism function The contribution of Gaussian reconstruction kernel is the kth GA reconstruction kernel function, described at position The light scattering or attenuation characteristics at the location reflect the position and shape of a single particle. Substituting equation (20) into equation (19), we get:

[0064]

[0065] Then set the emissivity As a constant C0, and using the first two terms of Taylor expansion as approximate values ​​of the exponent, the simplified expression of equation (21) is obtained as follows:

[0066]

[0067] in, Represents the integral reconstruction kernel, that is, the Gaussian reconstruction kernel The integral of

[0068] Finally, the SfM motion recovery model completes the reconstruction of the image surface. For the output reconstructed image, the originally curved straight line should become close to a straight line; if the image contains a rectangular object, the four sides of the reconstructed rectangular image should be close to straight line segments, and the four vertex angles should be close to 90° right angles; some ellipses in the original image should be close to circles after reconstruction; and some angles in the original image should be reduced to a certain extent after reconstruction.

[0069] Furthermore, in step S5, after completing the three-dimensional reconstruction of the image scene, the reconstruction performance evaluation index of the Gaussian splattering monocular reconstruction model is specifically as follows:

[0070] The evaluation indicators include: peak signal-to-noise ratio PSNR, structural similarity index SSIM and learning-perceptual image block similarity LPIPS.

[0071] (1) Peak signal-to-noise ratio (PSNR);

[0072]

[0073] Where MSE stands for mean square error; m and n represent the width and height of the image respectively; I(i,j) represents the pixel value at position (i,j) of the image before processing; K(i,j) represents the pixel value at position (i,j) of the image after processing; MAX I Represents the maximum possible pixel value in an image.

[0074] (2) Structural similarity (SSIM);

[0075]

[0076] Among them, u x 、u yRespectively represent the mean values ​​of the pixels of images X and Y, where images X and Y represent the images before and after reconstruction by the Gaussian splattering monocular reconstruction model; c1 and c2 are constants, and satisfy c1 = (K1 × L) 2 , c2=(K2×L) 2 , K1 and K2 represent two constants to avoid the denominator being zero when calculating SSIM, L represents the dynamic range of the pixel value, that is, the maximum possible value of the pixel value in the image; σ xy Represents the covariance between the pixel values ​​of image X and image Y, measuring the correlation between the corresponding pixel values ​​in the two images; Represents the variance of the pixel values ​​of image X; Represents the variance of the pixel values ​​of image Y.

[0077] (3) Learning to perceive image patch similarity LPIPS;

[0078]

[0079] Where d(p,p0) represents the distance measure between two image patches p and p0; H l , W l represents the length and width of the output of layer l; w l The weight is used to perform weighted scaling on the output of each channel; and They represent the activation outputs of the l-th layer network for the input image blocks p and p0 at the position (h, w), respectively, where h and w represent the spatial coordinates on the image.

[0080] Beneficial effects of the present invention: The method of the present invention first constructs a Gaussian splash monocular reconstruction model including an image depth recovery model and an SfM motion recovery model, and trains the image depth recovery model, then takes a panoramic picture, and trains it with spherical convolution to restore the distorted part, then reconstructs the sparse point cloud through the SfM motion recovery model, and reconstructs the image surface with a Gaussian reconstruction kernel to form the basic geometric configuration of the image, and finally uses image processing technology to perform Gaussian low-pass filtering and noise reduction on the image to form a complete surface structure of the object and complete the three-dimensional reconstruction of the image scene. The method of the present invention uses a three-dimensional Gaussian splash technology combined with a spherical convolution kernel to construct an object geometric model, which has significantly improved operability and greatly improved the speed of calculation. It has important application value in the fields of smart medical care, engineering flaw detection, etc. The method of the present invention provides technical support for scenes such as clinical diagnosis and special engineering projects. In clinical diagnosis, the Gaussian splash monocular reconstruction model can expand the varicose vein image of the patient's leg to a two-dimensional plane for reconstruction, which is convenient for doctors to diagnose. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1The present invention is a flowchart of a Gaussian splashing monocular reconstruction method based on a neural network.

[0082] Figure 2 It is an overall structural diagram of the encoding-decoding structure based on spherical convolution in an embodiment of the present invention.

[0083] Figure 3 3D representation of SO(3) in geometric space in an embodiment of the present invention.

[0084] Figure 4 It is a schematic diagram of the reconstruction result of the method of the present invention in an embodiment of the present invention. DETAILED DESCRIPTION

[0085] The method of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0086] like Figure 1 As shown in FIG. 1 , a flow chart of a Gaussian splashing monocular reconstruction method based on a neural network of the present invention is shown in FIG. 1 , and the specific steps are as follows:

[0087] S1. Build a Gaussian splash monocular reconstruction model including image depth recovery model and SfM motion recovery model;

[0088] The image depth recovery model is used to recover the scene depth of a monocular panoramic image, including: a pre-trained network and a prediction network. The specific structures of the pre-trained network and the prediction network are both encoding-decoding structures based on spherical convolution, and the encoder is fixedly VGG16. The SfM motion recovery model is used to reconstruct sparse point clouds of the recovered image.

[0089] The overall structure of the encoding-decoding structure based on spherical convolution is as follows: Figure 2 As shown in the figure, the encoder includes: a spherical convolution layer, a pooling layer, and a loss function including cross entropy and regularization; and the encoding model has 5 main layers, each of which includes a spherical convolution layer and a pooling layer; the encoding part of the spherical convolution layer and the pooling layer contains a total of 18 sublayers, namely Figure 2 As shown in the red and green parts. The decoder includes: 5 stacked deconvolution layers, Relu activation function. The decoder is skipped and connected to the encoder, and upsampling is used for the output to avoid the problem of insufficient resolution.

[0090] S2, based on step S1, training an image depth recovery model;

[0091] The points in the image corresponding to the imaging space are view-encoded to form a supervised learning sequence as the input of the pre-trained network and the prediction network. The image depth recovery model is embedded with the adaptive moment estimation algorithm Adam to optimize the learning rate, and the error back propagation algorithm is used to update the network weights, and the two algorithms are used to train the pre-trained network and the prediction network in sequence, and the gradient change of each parameter is estimated according to the loss function, and the learning rate and learning step of the parameter are dynamically adjusted to determine the distance moved in the opposite direction of the gradient each time, and the weight of the neural network model is continuously updated. The loss function and accuracy curve of the model output are observed. When the parameters in the model converge (that is, the parameters remain basically unchanged in multiple rounds of training), and the model accuracy reaches or approaches the preset accuracy, the training process is terminated and the training is completed.

[0092] In order to improve the generalization ability of the neural network, a random data enhancement strategy is set up and random dropout is designed to erase the image to prevent it from over-relying on a certain feature element. After training to restore the normal perspective, Gaussian splashing can be used to reconstruct the image restored to the correct depth.

[0093] Among them, Adam calculates an exponentially decaying average m of historical gradients t , and also calculates an exponentially decaying average of the historical squared gradient v t , and its calculation expression is as follows:

[0094]

[0095] m t = β1m t-1 + (1-β1)g t (2)

[0096] v t = β2v t-1 + (1-β2)g t 2 (3)

[0097] Among them, θ represents any parameter in the model, f t (θ) represents the random function value of the noise objective function f(θ) at the subsequent time step t. t Represents an estimate of the first-order moment of the gradient, m t-1 represents the estimate of the first-order moment of the gradient before the update, v t Represents an estimate of the second-order moment (non-deterministic error) of the gradient, v t-1 g represents the estimate of the second-order moment (non-deterministic error) of the gradient before update, β1 represents the exponential decay rate of the first-order moment estimate, and β2 represents the exponential decay rate of the second-order moment estimate. t Represents the function f t(θ) is the gradient of θ. In Adam, the deviation is offset by calculating the first-order moment estimate and the second-order moment estimate of the deviation correction. The calculation expression is as follows:

[0098]

[0099] in, represents the first-order moment estimate at time step t, Represents the second-order moment estimate of the current time step t, and the parameter update formula for generating Adam is as follows:

[0100]

[0101] Among them, θ t+1 represents the parameter value at time step t+1, θ t represents the parameter value at time step t, η represents the learning rate, which controls the step size of parameter update, ∈ represents the bias value, and ∈=10 -8 .

[0102] S3, collecting image information, inputting the image depth recovery model trained in step S2 to process the geometric information of the image, and completing the scene depth recovery of the monocular panoramic image;

[0103] S4, inputting the scene depth restored image obtained in step S3 into the SfM motion recovery model, reconstructing the sparse point cloud, and reconstructing the image surface with a Gaussian reconstruction kernel (Gaussian splattering) to form the basic geometric configuration of the image;

[0104] S5. Performing Gaussian low-pass filtering and noise reduction processing on the image reconstructed in step S4 using image processing technology to form a complete surface structure of the object and complete the three-dimensional reconstruction of the image scene.

[0105] In this embodiment, step S3 is specifically as follows:

[0106] S31, using a camera to capture a monocular panoramic image of the object to obtain image information;

[0107] This embodiment uses a 360° fisheye camera that can change its position and posture, and sets a turntable design for the camera so that it can capture objects in all directions with a single perspective. The camera is used to capture a monocular panoramic image of the object to obtain image information.

[0108] Among other things, the resulting image includes random high-frequency noise.

[0109] S32, based on the image information obtained in step S31, inputting it into the image depth recovery model trained in step S2 to perform image recovery processing on the geometric information of the image, so that it outputs an image containing correct depth information;

[0110] Equirectangular projection will cause serious distortion of the image, which will affect the depth estimation. This embodiment proposes a spherical convolution kernel to perform depth estimation of the panoramic image. The image processing process is regarded as the rotation change of the three-dimensional coordinates. The processed image is a three-dimensional rotation group SO(3), that is, a set of the starting and ending states in the rotation process. The matrix form of the rotation operation is as follows:

[0111]

[0112]

[0113] Among them, θ represents the rotation angle corresponding to the rotation operation in three-dimensional space, R x (θ), R y (θ), R z (θ) represents the rotation operation in each dimension in the three-dimensional coordinate system, R x (θ), R y (θ), R z The matrices corresponding to (θ) are the mathematical representations of each operation.

[0114] When extracting features, the output in the plane is the inner product of the input feature map and the filter, which is also applicable to the spherical surface and the three-dimensional rotation group. In the calculation process, this embodiment defines R as the rotation group on SO(3). Since it has metric invariance, it can be efficiently replaced in the integral operation. Spherical convolution on the sphere S 2 The inner product function expression on is as follows:

[0115]

[0116] Among them, S 2 represents a sphere, Represents the pixel function at each position on the sphere, f k (x) represents the convolution kernel function, k represents the kth channel, K represents the total number of image channels, and x represents the independent variable.

[0117] In the trained image depth recovery model, the decoding part has 5 layers, among which the convolution kernel of the convolution part in the first layer is on the spherical surface S 2 Convolution, starting from the second layer, the convolution kernel of the convolution part is convolved in the three-dimensional rotation group SO(3). When operating in the three-dimensional rotation group, since SO(3) is composed of linear transformations that keep vectors unchanged in three-dimensional Euclidean space, and vectors in three-dimensional coordinate systems can be represented by standard orthogonal bases, for any element in the three-dimensional rotation matrix, there is an associated expression as follows:

[0118]

[0119] Among them, x j represents the jth component of the original three-dimensional vector x in the standard basis, where j is the column index; x ′i represents the i-th component of the rotated three-dimensional vector x′ in the standard basis, where i is the row index. represents the element of the rotation matrix R, that is, the mapping from the j-th axis of the original coordinate system to the i-th axis of the rotated coordinate system. The three-dimensional representation of SO(3) in geometric space is as follows Figure 3 As shown in the figure, α ω ×α h Represents the angle range covered by the spherical convolution kernel in three-dimensional space, α ω Indicates the angle corresponding to the projection of the angle range on the xOz plane, α h Indicates the angle corresponding to the projection of the angle range on the xOy plane; r ω × h represents the pixel range (i.e., resolution range) covered by the spherical convolution kernel in three-dimensional space, r ω Represents the number of pixels contained in the projection of the convolution kernel in the width direction of the original image, r h Indicates the number of pixels contained in the projection of the convolution kernel in the height direction of the original image.

[0120] During the observation process, based on the Euler angle decomposition, SO(3) expresses the motion as the product of two rotations around a single coordinate axis and a function of motion around another coordinate axis, without the need for full involvement of the three coordinate axes. First, the coordinate axes are regarded as conventional inertial reference axes, and the motion on the two coordinate axes is analyzed. Then, the coordinate axes are regarded as a carrier system with the origin as the carrier of the motion center, and the form of describing the motion is roll-pitch, then the expression of the motion process is given as follows:

[0121] ρ=R z (α)R y (β)R z (γ) (12)

[0122] Among them, R z (α)R y (β)R z(γ) respectively represents the rotation operation in each dimension in the three-dimensional coordinate system, α, β, γ respectively represent the angles between the rotation operation and the three coordinate axes x, y, z; the inner product of R and the function on the rotation group can be regarded as the Hadamard product of the matrix, which satisfies the equivariance when performing convolution. However, due to the high time complexity of directly calculating the depth of the image, the fast Fourier transform algorithm is selected for optimization. In order to make the spherical signal meet the form that can be processed by the fast Fourier transform, the Wigner D function of this embodiment is involved in the calculation. The Wigner D equation is extended from the spherical harmonic function SH, and the spherical harmonic function itself is also a special orthogonal basis form of the Laplace operator.

[0123] The trained image depth recovery model uses the Wigner d function (non-traditional Wigner D function) as a basis to transform the spherical convolution process into a form that can be used for the fast Fourier transform algorithm, as follows:

[0124] First, the expression of the Wigner D function is set as follows:

[0125]

[0126] Among them, J, M, and M' represent three main indicators that affect the characterization of the function; J is a non-negative integer and satisfies -J <M,M'<J。

[0127] Then the second term is the Wigner d function, and its expanded expression is as follows:

[0128]

[0129] Among them, u represents the absolute value of the difference between M and M', v represents the absolute value of the sum of M and M', and s represents the difference between J and (u+v) / 2. represents the Jacobi polynomial. Then the first term of the Wigner d function is expanded as follows:

[0130]

[0131] Among them, if J satisfies greater than M and M', the value of Wigner d function is 0. The pre-calculated sample array of Wigner d function is used as a standard two-dimensional translation fast Fourier transform array. Since the spatial signal in SO(3) is a discrete three-dimensional array, its processing process undergoes two-dimensional translation and scaling, so Wigner d function plays a vital role in the calculation.

[0132] In the three-dimensional rotation group, the set of Wigner D functions can form a complete set of orthogonal group functions. By integrating the function on SO(3), it can be transformed into the product of a normalization factor and three Kronecker functions. This means that it has extremely sensitive 0,1 determination ability and is suitable as a basis.

[0133] In this embodiment, N is assumed to represent a three-dimensional manifold, and U represents a basis function. Then, the expression for performing GFT transformation (graph Fourier transform) is as follows:

[0134]

[0135] Where X represents a point on the three-dimensional manifold of the spherical convolution. represents the lth frequency component of function f after Fourier transform of the graph, i.e., the spectral coefficient; f(X) represents a function defined on the graph N, i.e., the signal value of a point on the three-dimensional manifold; Represents the basis function U l The value corresponding to the complex conjugate function at the node X; the basis function U l is the orthogonal basis function in the graph Fourier transform, corresponding to the eigenvector of the graph Laplacian operator. 2 Manifold Time U l (X) is the vector value Y l (X), when using SO(3) manifold U l (X) Take the matrix value D l , according to the reverse derivation of Wigner d function, the expression of the point on SO(3) is as follows:

[0136]

[0137] According to formula (16), using GFFT algorithm for formula (17), the expression can be obtained as follows:

[0138]

[0139] Wherein, D′ represents the complex conjugate function of the Wigner d function.

[0140] In this embodiment, step S4 is specifically as follows:

[0141] This embodiment refers to the basic principle of neural radiation field, but uses an elliptical Gaussian heterogeneous kernel as the basic scene representation method. After SfM generates a sparse point cloud, Colmap is not used to reconstruct the dense point cloud, but a three-dimensional Gaussian is selected as the primitive. Since sparse point cloud images have large noise, it is difficult to determine the normal, and it is difficult to implement the conventional method of using numerical gradients to calculate derivatives and back propagation. Therefore, this embodiment uses a covariance matrix to define the collective information. The value of the matrix is ​​determined by the average value of the regional space corresponding to the Gaussian kernel.

[0142] Use a three-dimensional column vector v = (x0, x1, x2) T To represent a point in ray space. Given a central projection and a projection plane: coordinates x0 and x1 specify a point on the projection plane, and a third coordinate x2 specifies the Euclidean distance from the projection center to the point on the viewing ray. Then the light intensity on the light path is The expression is as follows:

[0143]

[0144] Among them, T represents the transpose operation, μ is the integral variable, and in the integral expression represents a certain position along the ray from the ray exit point (i.e., the projection center) to the point ξ. It indicates the position of the point on the ray when the camera is taken as the origin; L indicates the distance from the ray exit point to the ray exit point. ξ represents the irradiation direction of the ray and is represented in the form of a 3D Gaussian unit vector; Represents the emission coefficient, that is, point The color at the point; g(x) represents the scattering function that defines the light shielding rate; Indicated in The emission coefficient along the ray direction ξ at , and the exponential term is the attenuation coefficient; is the light source term, which is described in The intensity of light scattered along the ray direction ξ at . is the Gaussian reconstruction kernel The weighted sum of is expressed as follows:

[0145]

[0146] Among them, g k Indicates the kth G weighting coefficient, which represents the kth G Gaussian reconstruction kernel Astigmatism function The contribution of Gaussian reconstruction kernel is the kth G A reconstruction kernel function, described at position The light scattering or attenuation characteristics at the location reflect the position and shape of a single particle. The spacing between particles can be irregular and have different shapes. Therefore, equation (19) is not limited to regular data sets. Substituting equation (20) into equation (19), we get:

[0147]

[0148] Then set the emissivity As a constant C0, and using the first two terms of Taylor expansion as approximate values ​​of the exponent, we can obtain a simplified expression of formula (21), which is as follows:

[0149]

[0150] in, Represents the integral reconstruction kernel, that is, the Gaussian reconstruction kernel The integral of

[0151] Gaussian splashing shares the same image generation model as neural radiation field, but the Gaussian reconstruction kernel has significant advantages in mathematical principles. The points defined by the Gaussian reconstruction kernel are elliptical geometries that can be rotated and stretched in any direction, which are mainly determined by two properties, one is the covariance matrix, and the other is the spherical harmonic function. According to its covariance matrix, each point can represent a finite area in space close to its mean. The spherical harmonic function is a special function defined on the surface of a sphere. Such a function can be calculated for any point on the sphere and a value is obtained. This allows us to accurately define the relevant color of the view. The advantage of the Gaussian reconstruction kernel is that it is closed during affine transformation and resonance, and by integrating it, the three-dimensional Gaussian can be transformed into two dimensions. The reconstruction kernel r(x) only performs the calculation of two-dimensional convolution. This allows the stitching algorithm to have higher rendering efficiency.

[0152] Finally, the SfM motion recovery model completes the reconstruction of the image surface. For the output reconstructed image, the originally curved straight line should become close to a straight line; if the image contains a rectangular object, the four sides of the reconstructed rectangular image should be close to straight line segments, and the four vertex angles should be close to 90° right angles; some ellipses in the original image should be close to circles after reconstruction; and some angles in the original image should be reduced to a certain extent after reconstruction.

[0153] In this embodiment, in step S5, after completing the three-dimensional reconstruction of the image scene, the reconstruction performance evaluation index of the Gaussian splattering monocular reconstruction model is specifically as follows:

[0154] In order to comprehensively evaluate the performance of the method of the present invention, the evaluation indicators used in this embodiment include: peak signal-to-noise ratio PSNR, structural similarity index SSIM and learning perceptual image block similarity LPIPS. These indicators jointly measure the quality of the reconstructed image, the similarity with the original image, and the difference in visual perception.

[0155] (1) Peak signal-to-noise ratio (PSNR);

[0156]

[0157]

[0158] Where MSE stands for mean square error; m and n represent the width and height of the image respectively; I(i,j) represents the pixel value at position (i,j) of the image before processing; K(i,j) represents the pixel value at position (i,j) of the image after processing; MAX I Represents the maximum possible pixel value in an image.

[0159] (2) Structural similarity (SSIM);

[0160]

[0161] Among them, u x 、u y Respectively represent the mean values ​​of the pixels of images X and Y, where image X and image Y represent the images before and after reconstruction by the Gaussian splattering monocular reconstruction model; c1 and c2 are constants. In order to avoid the denominator being 0, c1=(K1×L) 2 , c2=(K2×L) 2 In this embodiment, K1=0.01, K2=0.03, L=255, K1 and K2 represent two constants to avoid the denominator being zero when calculating SSIM, and L represents the dynamic range of the pixel value, that is, the possible maximum value of the pixel value in the image; σ xy Represents the covariance between the pixel values ​​of image X and image Y, measuring the correlation between the corresponding pixel values ​​in the two images; Represents the variance of the pixel values ​​of image X; Represents the variance of the pixel values ​​of image Y.

[0162] (3) Learning to perceive image patch similarity LPIPS;

[0163]

[0164] Where d(p,p0) represents the distance metric between two image patches p and p0, which is used to measure the perceptual similarity between them; H l , W l represents the length and width of the output of layer l; wl The weight is used to perform weighted scaling on the output of each channel; and They represent the activation outputs of the l-th layer network for the input image blocks p and p0 at the position (h, w), respectively, where h and w represent the spatial coordinates on the image.

[0165] like Figure 4 As shown, this embodiment further conducts experimental verification, selects a video as the material, uses time-lapse photography to take several panoramic pictures, and puts the pictures into the Gaussian splash monocular reconstruction model for training. In addition, this embodiment sets breakpoints during the training process to visualize the depth recovery process.

[0166] in, Figure 4 (a) is a partial panoramic image. Figure 4 (b) is the breakpoint image recovery state, showing the recovery effect at the breakpoint position during the image recovery process; Figure 4 (c) is the reconstruction result of the method of the present invention, which shows that the method of the present invention has obvious improvements in detail restoration and visual quality. The evaluation index can be used to understand the model training effect of this embodiment.

[0167] Among them, SSIM: 0.77, which indicates that the reconstructed image has a high similarity with the original image in structural details; PSNR: 26.21, showing a lower noise level and higher signal fidelity; LPIPS: 0.268, this value is low, indicating that the reconstructed image is closer to human visual expectations in perceptual quality.

[0168] In summary, the method of the present invention consists of two parts: an image depth recovery model and an SfM motion recovery model. The image depth recovery model consists of a pre-trained network and a prediction network of a coding and decoding structure, while the SfM motion recovery model uses a Gaussian reconstruction kernel to reconstruct the geometric body. The model overturns the previous model of using one hundred pictures for training and only uses one picture for reconstruction, which has significantly improved operability. In addition, the method of the present invention uses the Gaussian splashing method to replace the traditional dense point cloud reconstruction, which strips off the neural network with relatively high computing power requirements, greatly improves the speed of calculation, and has practical engineering application significance. In medical diagnosis work, the model has shown high sensitivity and adaptability.

[0169] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A Gaussian splash monocular reconstruction method based on neural network, the specific steps are as follows: S1. Build a Gaussian splash monocular reconstruction model including image depth recovery model and SfM motion recovery model; The image depth recovery model is used to perform scene depth recovery on a monocular panoramic image, including: Pre-trained network and prediction network; the specific structures of the pre-trained network and the prediction network are both encoding-decoding structures based on spherical convolution, and the encoder is fixedly VGG16; the SfM motion recovery model is used to reconstruct sparse point clouds of the recovered images; Among them, in the encoding-decoding structure based on spherical convolution, the encoder includes: a spherical convolution layer, a pooling layer, and a loss function including cross entropy and regularization; and the encoding model has 5 main layers, each of which includes a spherical convolution layer and a pooling layer; the encoding part spherical convolution layer and pooling layer contain a total of 18 sublayers; the decoder includes: 5 stacked deconvolution layers, Relu activation function; and the decoder is jump-connected with the encoder, and upsampling is used for the output; S2, based on step S1, training an image depth recovery model; The points in the image corresponding to the imaging space are view-encoded to form a supervised learning sequence as the input of the pre-trained network and the prediction network; the adaptive moment estimation algorithm Adam is embedded in the image depth recovery model to optimize the learning rate, and the error back propagation algorithm is used to update the network weights, and the two algorithms are used to train the pre-trained network and the prediction network in sequence, and the gradient change of each parameter is estimated according to the loss function, and the learning rate and learning step of the parameter are dynamically adjusted to determine the distance moved in the opposite direction of the gradient each time, and the weight of the neural network model is continuously updated, and the loss function and accuracy curve of the model output are observed. When the parameters in the model converge and the model accuracy reaches or approaches the preset accuracy, the training process is terminated and the training is completed; Among them, Adam calculates an exponentially decaying average m of historical gradients t , and also calculates an exponentially decaying average of the historical squared gradient v t , and its calculation expression is as follows: m t =β1m t-1 +(1-β1)g t (2) v t =β2v t-1 +(1-β2)g t 2 (3) Among them, θ represents any parameter in the model, f t (θ) represents the random function value of the noise objective function f(θ) at the subsequent time step t; m t Represents an estimate of the first-order moment of the gradient, m t-1 represents the estimate of the first-order moment of the gradient before the update, v t represents the estimate of the second-order moment of the gradient, v t-1 represents the estimate of the second-order moment of the gradient before the update, β1 represents the exponential decay rate of the first-order moment estimate, and β2 represents the exponential decay rate of the second-order moment estimate; g t Represents the function f t (θ) is the gradient of θ; Adam offsets the deviation by calculating the first-order moment estimate and the second-order moment estimate of the deviation correction. The calculation expression is as follows: in, represents the first-order moment estimate at time step t, Represents the second-order moment estimate of the current time step t, and the parameter update formula for generating Adam is as follows: Among them, θ t+1 represents the parameter value at time step t+1, θ t represents the parameter value at time step t, η represents the learning rate, ∈ represents the bias value, and ∈=10 -8 ; S3, collecting image information, inputting the image depth recovery model trained in step S2 to process the geometric information of the image, and completing the scene depth recovery of the monocular panoramic image; S4, inputting the scene depth restored image obtained in step S3 into the SfM motion recovery model, reconstructing the sparse point cloud, and reconstructing the image surface with a Gaussian reconstruction kernel to form the basic geometric configuration of the image; S5. Performing Gaussian low-pass filtering and noise reduction processing on the image reconstructed in step S4 using image processing technology to form a complete surface structure of the object and complete the three-dimensional reconstruction of the image scene.

2. The Gaussian splattering monocular reconstruction method based on neural network according to claim 1, characterized in that: The step S3 is specifically as follows: S31, using a camera to capture a monocular panoramic image of the object to obtain image information; A camera is used, and a turntable is set for the camera so that it can capture all directions of the object with a single perspective. The camera is used to capture a monocular panoramic image of the object to obtain image information. Among them, the resulting image includes random high-frequency noise; S32, based on the image information obtained in step S31, inputting it into the image depth recovery model trained in step S2 to perform image recovery processing on the geometric information of the image, so that it outputs an image containing correct depth information; The graphics processing process is regarded as the rotation change of three-dimensional coordinates. The processed image is the three-dimensional rotation group SO(3), that is, the set of starting and ending states in the rotation process. The matrix form of the rotation operation is as follows: Among them, θ represents the rotation angle corresponding to the rotation operation in three-dimensional space, R x (θ), R y (θ), R z (θ) represents the rotation operation in each dimension in the three-dimensional coordinate system, R x (θ), R y (θ), R z The matrices corresponding to (θ) are the mathematical representations of each operation; Spherical convolution on the sphere S 2 The inner product function expression on is as follows: Among them, S 2 represents a sphere, Represents the pixel function at each position on the sphere, f k (x) represents the convolution kernel function, k represents the kth channel, K represents the total number of image channels, and x represents the independent variable; In the trained image depth recovery model, the decoding part has 5 layers, among which the convolution kernel of the convolution part in the first layer is on the spherical surface S 2 For the up-convolution, the convolution kernel of the convolution part starting from the second layer is convolved in the three-dimensional rotation group SO(3); when operating in the three-dimensional rotation group, for any element in the three-dimensional rotation matrix, the associated expression is as follows: Among them, x j represents the jth component of the original three-dimensional vector x in the standard basis, where j is the column index; x′ i represents the i-th component of the rotated three-dimensional vector x′ in the standard basis, where i is the row index; represents the elements of the rotation matrix R, that is, the mapping from the j-th axis of the original coordinate system to the i-th axis of the rotated coordinate system; Based on the Euler angle decomposition, SO(3) expresses the motion as the product of two rotations around a single coordinate axis and a function of motion around another coordinate axis. First, the coordinate axis is regarded as a conventional inertial reference axis, and the motion on the two coordinate axes is analyzed. Then, the coordinate axis is regarded as a carrier system with the origin as the carrier of the motion center, and the form of describing the motion is roll-pitch. The expression of the motion process is given as follows: p=R z (a)R y (b)R z (c) (12) Among them, R z (α)R y (β)R z (γ) represents the rotation operation in each dimension in the three-dimensional coordinate system, α, β, γ represent the angles between the rotation operation and the three coordinate axes x, y, and z respectively; the inner product of R and the function on the rotation group can be regarded as the Hadamard product of the matrix, which satisfies the equivariance when performing convolution; the fast Fourier transform algorithm is selected for optimization; The trained image depth recovery model uses the Wigner d function as a basis to transform the spherical convolution process into a form that can be used for the fast Fourier transform algorithm, as follows: First, the expression of the Wigner D function is set as follows: Among them, J, M, and M' represent three main indicators that affect the characterization of the function; J is a non-negative integer and satisfies -J <M,M'<J; Then the second term is the Wigner d function, and its expanded expression is as follows: Among them, u represents the absolute value of the difference between M and M', v represents the absolute value of the sum of M and M', and s represents the difference between J and (u+v) / 2. represents the Jacobi polynomial; then the first term of the Wigner d function is expanded as follows: Wherein, if J satisfies that it is greater than M and M', the value of Wigner d function is 0; the pre-calculated sample array of Wigner d function is used as a standard two-dimensional translation fast Fourier transform array; Assuming N represents the three-dimensional manifold and U represents the basis function, the expression for GFT transformation is as follows: Where X represents a point on the three-dimensional manifold of the spherical convolution. represents the lth frequency component of function f after Fourier transform of the graph, i.e., the spectral coefficient; f(X) represents a function defined on the graph N, i.e., the signal value of a point on the three-dimensional manifold; Represents the basis function U l The value corresponding to the complex conjugate function at the node X; the basis function U l is the orthogonal basis function in the graph Fourier transform, corresponding to the eigenvector of the graph Laplacian operator; using S 2 Manifold Time U l (X) is the vector value Y l (X), when using SO(3) manifold U l (X) Take the matrix value D l , according to the reverse derivation of Wigner d function, the expression of the point on SO(3) is as follows: According to formula (16), using GFFT algorithm for formula (17), the expression can be obtained as follows: Among them, D ′ Represents the complex conjugate function of the Wigner d function.

3. The Gaussian splashing monocular reconstruction method based on neural network according to claim 1 is characterized in that: The step S4 is specifically as follows: After the SfM motion recovery model generates a sparse point cloud, a three-dimensional Gaussian is selected as the primitive, and the covariance matrix is ​​used to define the set information. The value of the matrix is ​​determined by the average value of the regional space corresponding to the Gaussian kernel. Use a three-dimensional column vector v = (x0, x1, x2) T to represent a point in ray space; given a central projection and a projection plane: coordinates x0 and x1 specify a point on the projection plane, and the third coordinate x2 specifies the Euclidean distance from the projection center to the point on the viewing ray; then the light intensity on the optical path is The expression is as follows: Where T represents the transposition operation, μ is the integral variable, and in the integral expression represents a position along the ray from the ray exit point to the point ξ; It indicates the position of the point on the ray when the camera is taken as the origin; L indicates the distance from the ray exit point to the ray exit point. ξ represents the irradiation direction of the ray and is represented in the form of a 3D Gaussian unit vector; Represents the emission coefficient, that is, point The color at the point; g(x) represents the scattering function that defines the light shielding rate; Indicated in The emission coefficient along the ray direction ξ at , and the exponential term is the attenuation coefficient; is the light source term, which is described in The intensity of light scattered along the ray direction ξ at the position; astigmatism function is the Gaussian reconstruction kernel The weighted sum of is expressed as follows: in, Indicates the kth G weighting coefficient, which represents the kth G Gaussian reconstruction kernel Astigmatism function The contribution of Gaussian reconstruction kernel is the kth G A reconstruction kernel function, described at position The light scattering or attenuation characteristics at the location reflect the position and shape of a single particle; Substituting equation (20) into equation (19), we get: Then set the emissivity As a constant C0, and using the first two terms of Taylor expansion as approximate values ​​of the exponent, the simplified expression of equation (21) is obtained as follows: in, Represents the integral reconstruction kernel, that is, the Gaussian reconstruction kernel The integral of Finally, the SfM motion recovery model completes the reconstruction of the image surface; for the output reconstructed image, the originally curved straight line should become close to a straight line; if the image contains a rectangular object, the four sides of the reconstructed rectangular image should be close to straight line segments, and the four vertex angles should be close to 90° right angles; some ellipses in the original image should be close to circles after reconstruction; the angles of some angles in the original image should be reduced to a certain extent after reconstruction.

4. The Gaussian splattering monocular reconstruction method based on neural network according to claim 1, characterized in that: In step S5, after completing the three-dimensional reconstruction of the image scene, the reconstruction performance evaluation index of the Gaussian splattering monocular reconstruction model is as follows: The evaluation indicators include: peak signal-to-noise ratio PSNR, structural similarity index SSIM and learning-perceptual image block similarity LPIPS; (1) Peak signal-to-noise ratio (PSNR); Where MSE stands for mean square error; m and n represent the width and height of the image respectively; I(i,j) represents the pixel value at position (i,j) of the image before processing; K(i,j) represents the pixel value at position (i,j) of the image after processing; MAX I Represents the maximum possible pixel value in an image; (2) Structural similarity (SSIM); Among them, u x 、u y Respectively represent the mean values ​​of the pixels of images X and Y, where images X and Y represent the images before and after reconstruction by the Gaussian splattering monocular reconstruction model; c1 and c2 are constants, and satisfy c1 = (K1 × L) 2 , c2=(K2×L) 2 , K1 and K2 represent two constants to avoid the denominator being zero when calculating SSIM, L represents the dynamic range of the pixel value, that is, the maximum possible value of the pixel value in the image; σ xy Represents the covariance between the pixel values ​​of image X and image Y, measuring the correlation between the corresponding pixel values ​​in the two images; Represents the variance of the pixel values ​​of image X; Represents the variance of the pixel values ​​of image Y; (3) Learning to perceive image patch similarity LPIPS; Where d(p,p0) represents the distance measure between two image patches p and p0; H l , W l represents the length and width of the output of layer l; w l The weight is used to perform weighted scaling on the output of each channel; and They represent the activation outputs of the l-th layer network for the input image blocks p and p0 at the position (h, w), respectively, where h and w represent the spatial coordinates on the image.

Images