Multi-view synthesis method based on tensor Gaussian mixture representation, electronic equipment and medium

Abstract

The invention discloses a multi-view synthesis method based on tensor Gaussian mixture representation, electronic equipment and a medium. The multi-view synthesis method comprises the following steps: initializing a three-dimensional Gaussian body set; setting a global appearance dictionary, and extracting a feature vector corresponding to each three-dimensional Gaussian body from the local tensor grid; respectively inputting each feature vector and a camera pose corresponding to the target picture into a first multi-layer perceptron and a second multi-layer perceptron to obtain opacity and color corresponding to each three-dimensional Gaussian body; adding a mask to each three-dimensional Gaussian body with known opacity and color; rasterization is carried out on the three-dimensional Gaussian body set after the mask is added, and a synthesized multi-view is obtained; completing one iteration; and calculating mean square error loss based on the target picture and the synthesized multi-view, performing back propagation optimization, optimizing the scale, direction and position of the three-dimensional Gaussian body, optimizing parameters of the local tensor grid, the first multi-layer perceptron, the second multi-layer perceptron and the global appearance dictionary, and repeating iteration until the synthesized multi-view is consistent with the target picture.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a multi-view synthesis method based on tensor Gaussian mixture representation, electronic equipment, and medium. Background Art

[0002] In the field of novel view synthesis, significant progress has been made in recent years. Its goal is to generate realistic new views of 3D scenes or objects by interpolating from a sparse set of images. This technology has been widely used in the real world, including virtual reality, augmented reality, digital twins, and autonomous driving. Among them, Neural Radiance Field (NeRF) has received widespread attention as a pioneering technology that uses an implicit multi-layer perceptron (MLP) to accumulate RGB values ​​along sampling rays to achieve realistic image rendering. However, in most practices, NeRF has inherent problems such as high overhead and slow synthesis speed. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a multi-view synthesis method, electronic device, and medium based on tensor Gaussian mixture representation.

[0004] In a first aspect, an embodiment of the present invention provides a multi-view synthesis method based on tensor Gaussian mixture representation, the method comprising:

[0005] Initialize a set of three-dimensional Gaussian volumes;

[0006] Set up a global appearance dictionary and extract the feature vector corresponding to each three-dimensional Gaussian volume from the local tensor grid;

[0007] Input each feature vector and the camera pose corresponding to the target image into the first multi-layer perceptron and the second multi-layer perceptron respectively to obtain the opacity and color corresponding to each three-dimensional Gaussian body;

[0008] Add a mask to each 3D Gaussian volume of known opacity and color;

[0009] Rasterize the masked 3D Gaussian volume set to obtain a synthetic multi-view; complete 1 iteration;

[0010] The mean square error loss is calculated based on the target image and the synthesized multi-views, and back-propagation optimization is performed to optimize the scale, direction, and position of the three-dimensional Gaussian body, the parameters of the local tensor grid, the first multi-layer perceptron, the second multi-layer perceptron, and the global appearance dictionary. The iteration is repeated until the synthesized multi-views are consistent with the target image.

[0011] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned multi-view synthesis method based on tensor Gaussian mixture representation.

[0012] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-view synthesis method based on tensor Gaussian mixture representation.

[0013] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the above-mentioned multi-view synthesis method based on tensor Gaussian mixture representation when executed by a processor.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] The invention discloses a multi-view synthesis method based on tensor Gaussian mixture representation, by initializing a set of three-dimensional Gaussian bodies; setting a global appearance dictionary, extracting a feature vector corresponding to each three-dimensional Gaussian body from a local tensor grid; inputting each feature vector and the camera posture corresponding to a target image into the first and second multi-layer perceptrons respectively, obtaining the opacity and color corresponding to each three-dimensional Gaussian body; adding a mask to each three-dimensional Gaussian body with known opacity and color; rasterizing the masked three-dimensional Gaussian body set to obtain a synthesized multi-view; completing one iteration; calculating the mean square error loss based on the target image and the synthesized multi-view, back propagation optimization, optimizing the scale, direction, and position of the three-dimensional Gaussian body, optimizing the local tensor grid, the first and second multi-layer perceptrons, and the parameters of the global appearance dictionary, and repeating the iteration until the synthesized multi-view is consistent with the target image. The invention reduces the computational overhead and improves the synthesis speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0017] Figure 1 A schematic diagram of a multi-view synthesis method based on tensor Gaussian mixture representation provided by an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0021] like Figure 1 As shown, an embodiment of the present invention provides a multi-view synthesis method based on tensor Gaussian mixture representation, the method comprising the following steps:

[0022] Step S1, initialize the three-dimensional Gaussian volume set

[0023] In the formula, represents normal distribution, χ i represents the spatial position, ∑ i represents the covariance matrix, σ i Indicates opacity, c i Represents the color value, and n represents the number of three-dimensional Gaussians.

[0024] Step S2, extracting a feature vector corresponding to the three-dimensional Gaussian volume from the local tensor grid;

[0025] Specifically, the step S2 includes the following sub-steps:

[0026] The local tensor grid is factorized by CP decomposition, as follows:

[0027]

[0028] In the formula, φ f represents the feature vector associated with the position, represents the outer product, R f Represents the number of tensor components, Z f，r Represents the decomposed component tensor value, b r represents the feature dimension, Represents the vector values ​​in the x, V, and z directions respectively.

[0029] Using matrix B c As a global appearance dictionary, this dictionary links the color correlations between different local tensors. For any position χ, after trilinear interpolation within each tensor, we can obtain its feature representation containing opacity and color:

[0030]

[0031] In the formula, Respectively represent the vector values ​​obtained after interpolation at any position in the x, v, and z directions. , Represents the component tensor value after decomposition and interpolation at any position, Represents the matrix obtained by concatenating all tensor components.

[0032] It should be noted that this example uses a single grid Z f To represent the features containing opacity and color to establish joint optimization. This allows us to control the optimization direction of the Gaussian function by the combined effects of opacity and color in the rendered image.

[0033] Features are extracted from the k nearest local tensors, and the feature vectors are sent to two different tiny MLPs for decoding to obtain transparency and color, taking into account the viewing direction d during color decoding. Compared with optimizing the attributes separately using independent grids, this approach significantly reduces the number of redundant Gaussian components and provides enhanced optimization direction control.

[0034] Step S3, input the feature vector and the camera pose corresponding to the target image into the first multi-layer perceptron and the second multi-layer perceptron respectively to obtain opacity and color; the set of three-dimensional Gaussian volumes with known opacity and color is recorded as

[0035] In the formula, ψ 1 (.) represents the first multi-layer perceptron, ψ 2 (.) represents the second multi-layer perceptron.

[0036] Step S4, adding a mask to the three-dimensional Gaussian volume with known opacity and color;

[0037] Specifically, the step S4 comprises the following steps:

[0038] Set a mask parameter Thus, a binary mask M∈{0, 1} is generated n , the expression is as follows:

[0039]

[0040] Where Γ(·) represents the stop gradient operator, represents the indicator function, sigmoid(·) represents the activation function, δ represents the mask threshold, n represents the number of three-dimensional Gaussian bodies, and i represents the i-th three-dimensional Gaussian body.

[0041] Add masks to the three dimensions of position, size, and opacity of the three-dimensional Gaussian body. The expressions are as follows:

[0042]

[0043]

[0044]

[0045] In the formula, Indicates the scale after masking, s n represents the scale before masking, Indicates the opacity after adding the mask, o n Indicates the opacity before adding the mask. Indicates the spatial position before adding the mask, h n Indicates the spatial position after masking.

[0046] Step S5, rasterizing the masked three-dimensional Gaussian volume set to obtain a synthesized multi-view; completing one iteration;

[0047] Step S6, based on the target image and the synthesized multi-view, the mean square error loss is calculated, the back propagation optimization is performed, the scale, direction, and position of the three-dimensional Gaussian body are optimized, the local tensor grid, the first multi-layer perceptron, the second multi-layer perceptron, and the global appearance dictionary B are optimized c The parameters are adjusted and the iteration is repeated until the synthesized multi-view is consistent with the target image.

[0048] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-view synthesis method based on tensor Gaussian mixture representation as described above. Figure 2 As shown, a hardware structure diagram of any device with data processing capability for the multi-view synthesis method based on tensor Gaussian mixture representation provided by an embodiment of the present invention is shown, except Figure 2 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0049] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by the processor, the multi-view synthesis method based on tensor Gaussian mixture representation as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0050] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A multi-view synthesis method based on tensor Gaussian mixture representation, characterized in that: The method comprises: Initialize a set of three-dimensional Gaussian volumes; Set up a global appearance dictionary and extract the feature vector corresponding to each three-dimensional Gaussian volume from the local tensor grid; Input each feature vector and the camera pose corresponding to the target image into the first multi-layer perceptron and the second multi-layer perceptron respectively to obtain the opacity and color corresponding to each three-dimensional Gaussian body; Add a mask to each 3D Gaussian volume of known opacity and color; Rasterize the masked 3D Gaussian volume set to obtain a synthetic multi-view; complete 1 iteration; The mean square error loss is calculated based on the target image and the synthesized multi-views, and back-propagation optimization is performed to optimize the scale, direction, and position of the three-dimensional Gaussian body, the parameters of the local tensor grid, the first multi-layer perceptron, the second multi-layer perceptron, and the global appearance dictionary. The iteration is repeated until the synthesized multi-views are consistent with the target image.

2. According to claim 1, a multi-view synthesis method based on tensor Gaussian mixture representation is characterized in that: Will In the formula, represents normal distribution, χ i represents the spatial position, ∑ i represents the covariance matrix, σ i Indicates opacity, c i Represents the color value, and n represents the number of three-dimensional Gaussians.

3. The multi-view synthesis method based on tensor Gaussian mixture representation according to claim 1, characterized in that: Extracting the eigenvectors corresponding to the three-dimensional Gaussian volume from the local tensor grid includes: The local tensor grid is factorized by CP decomposition, as follows: In the formula, Φ f represents the feature vector associated with the position, represents the outer product, R f Represents the number of tensor components, Z f,r Represents the decomposed component tensor value, b r represents the feature dimension, Represent the vector values ​​in the x, y, and z directions respectively; Using matrix B c As a global appearance dictionary, this dictionary links the color correlations between different local tensors; for any position χ, after trilinear interpolation within each tensor, we can obtain its feature representation containing opacity and color: In the formula, Respectively represent the vector values ​​obtained after interpolation at any position in the x, y, and z directions, Represents the component tensor value after decomposition and interpolation at any position, Represents the matrix obtained by concatenating all tensor components.

4. The multi-view synthesis method based on tensor Gaussian mixture representation according to claim 1, characterized in that: The set of three-dimensional Gaussian volumes with known opacity and color is denoted as Where ψ1(.) represents the first multilayer perceptron, and ψ2(.) represents the second multilayer perceptron.

5. The multi-view synthesis method based on tensor Gaussian mixture representation according to claim 1, characterized in that: Adding a mask to a 3D Gaussian volume of known opacity and color involves: Set a mask parameter Thus generating a binary mask M∈{0,1} n , the expression is as follows: Where Γ(·) represents the stop gradient operator, represents the indicator function, sigmoid(·) represents the activation function, δ represents the mask threshold, n represents the number of three-dimensional Gaussian bodies, and i represents the i-th three-dimensional Gaussian body; Add masks to the three dimensions of position, size, and opacity of the three-dimensional Gaussian body. The expressions are as follows: In the formula, Indicates the scale after masking, s n represents the scale before masking, Indicates the opacity after adding the mask, o n Indicates the opacity before adding the mask. Indicates the spatial position before adding the mask, h n Indicates the spatial position after masking.

6. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multi-view synthesis method based on tensor Gaussian mixture representation as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-view synthesis method based on tensor Gaussian mixture representation as described in any one of claims 1 to 5 is implemented.

8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the multi-view synthesis method based on tensor Gaussian mixture representation described in any one of claims 1 to 5 is implemented.

Images