Method for multi-view synthesis based on tensor gaussian mixture representation, electronic device, medium

Abstract

The application discloses a multi-view synthesis method based on tensor Gaussian mixture representation, electronic equipment and medium, comprising: initializing a three-dimensional Gaussian body set; setting a global appearance dictionary, extracting a feature vector corresponding to each three-dimensional Gaussian body from a local tensor grid; inputting each feature vector and a camera pose corresponding to a target picture into a first and a second multi-layer perception machine respectively to obtain opacity and color corresponding to each three-dimensional Gaussian body; adding a mask to each three-dimensional Gaussian body with known opacity and color; rasterizing the three-dimensional Gaussian body set after adding the mask to obtain a synthesized multi-view; completing one iteration; calculating mean square error loss based on the target picture and the synthesized multi-view, backward propagation optimization, optimizing the scale, direction and position of the three-dimensional Gaussian body, optimizing the parameters of the local tensor grid, the first and second multi-layer perception machines and the global appearance dictionary, and repeating the iteration until the synthesized multi-view is consistent with the target picture.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a multi-view synthesis method based on tensor Gaussian mixture representation, an electronic device and a medium. BACKGROUND

[0002] In the field of novel view synthesis, significant progress has been made in recent years. The goal is to generate realistic new views of three-dimensional scenes or objects by interpolating from a sparse set of images. This technology has been widely applied in the real world, including virtual reality, augmented reality, digital twin and autonomous driving, etc. Among them, Neural Radiance Field (NeRF) has attracted widespread attention as a pioneering technology, which uses implicit multi-layer perceptron (MLP) to accumulate RGB values along the sampling ray to achieve realistic image rendering. However, in most practices, NeRF has inherent problems such as large overhead and slow synthesis speed. SUMMARY

[0003] In view of the deficiencies of the prior art, the application provides a multi-view synthesis method based on tensor Gaussian mixture representation, an electronic device and a medium.

[0004] In a first aspect, an embodiment of the application provides a multi-view synthesis method based on tensor Gaussian mixture representation, which comprises:

[0005] Initializing a set of three-dimensional Gaussian bodies;

[0006] Setting a global appearance dictionary, extracting a feature vector corresponding to each three-dimensional Gaussian body from a local tensor grid;

[0007] Inputting each feature vector and the corresponding camera pose of the target picture into a first multi-layer perceptron and a second multi-layer perceptron respectively to obtain the opacity and color corresponding to each three-dimensional Gaussian body;

[0008] Adding a mask to each three-dimensional Gaussian body with known opacity and color;

[0009] Rasterizing the set of three-dimensional Gaussian bodies after adding the mask to obtain a synthesized multi-view; completing one iteration;

[0010] Calculating the mean square error loss based on the target picture and the synthesized multi-view, optimizing the scale, direction and position of the three-dimensional Gaussian body, optimizing the parameters of the local tensor grid, the first multi-layer perceptron, the second multi-layer perceptron and the global appearance dictionary, and repeating the iteration until the synthesized multi-view is consistent with the target picture.

[0011] In a second aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned multi-view synthesis method based on tensor Gaussian mixture representation.

[0012] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-view synthesis method based on tensor Gaussian mixture representation.

[0013] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which implements the above-mentioned multi-view synthesis method based on tensor Gaussian mixture representation when executed by a processor.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] The present invention discloses a multi-view synthesis method based on tensor Gaussian mixture representation. The method includes initializing a set of three-dimensional Gaussian bodies; setting a global appearance dictionary, extracting a feature vector corresponding to each three-dimensional Gaussian body from a local tensor grid; inputting each feature vector and the camera pose corresponding to a target image into a first and a second multi-layer perceptron, respectively, to obtain the opacity and color corresponding to each three-dimensional Gaussian body; adding a mask to each three-dimensional Gaussian body with known opacity and color; rasterizing the masked set of three-dimensional Gaussian bodies to obtain a synthesized multi-view; completing one iteration; calculating a mean square error loss based on the target image and the synthesized multi-view; performing backpropagation optimization to optimize the scale, orientation, and position of the three-dimensional Gaussian bodies; optimizing the parameters of the local tensor grid, the first and second multi-layer perceptrons, and the global appearance dictionary; and repeating the iteration until the synthesized multi-views are consistent with the target image. The present invention reduces computational overhead and improves synthesis speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic diagram of a multi-view synthesis method based on tensor Gaussian mixture representation provided by an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.

[0021] like Figure 1 As shown, an embodiment of the present invention provides a multi-view synthesis method based on tensor Gaussian mixture representation, the method comprising the following steps:

[0022] Step S1, initialize the three-dimensional Gaussian volume set

[0023] Where, represents normal distribution, χ i represents the spatial position, ∑ i represents the covariance matrix, σ i Indicates opacity, c i Represents the color value, and n represents the number of three-dimensional Gaussian volumes.

[0024] Step S2, extracting the feature vector corresponding to the three-dimensional Gaussian volume from the local tensor grid;

[0025] Specifically, step S2 includes the following sub-steps:

[0026] The local tensor grid is factorized by CP decomposition, which is expressed as follows:

[0027]

[0028] Where, φ f represents the feature vector associated with the position, represents the outer product, R f Indicates the number of tensor components, Z f，r Represents the decomposed component tensor value, b r represents the feature dimension, Represents the vector values ​​in the x, V, and z directions respectively.

[0029] Using matrix B c As a global appearance dictionary, this dictionary links the color correlations between different local tensors. For any position χ, after trilinear interpolation within each tensor, we can obtain its feature representation containing opacity and color:

[0030]

[0031] Where, Represents the vector value obtained after interpolation at any position in the x, v, and z directions respectively. Represents the component tensor value after decomposition and interpolation at any position, Represents the matrix resulting from the concatenation of all tensor components.

[0032] It should be noted that this example uses a single grid Z f To represent the features containing opacity and color to establish joint optimization. This allows us to control the optimization direction of the Gaussian function by the combined effects of opacity and color in the rendered image.

[0033] Features are extracted from the k nearest local tensors, and the feature vectors are sent to two different tiny MLPs for decoding to obtain transparency and color, taking into account the viewing direction d during the color decoding process. Compared with optimizing the attributes separately using independent grids, this approach significantly reduces the number of redundant Gaussian components and provides enhanced optimization direction control.

[0034] Step S3: Input the feature vector and the camera pose corresponding to the target image into the first multi-layer perceptron and the second multi-layer perceptron respectively to obtain the opacity and color; the set of three-dimensional Gaussian volumes with known opacity and color is recorded as

[0035] Where ψ1(.) represents the first multilayer perceptron, and ψ2(.) represents the second multilayer perceptron.

[0036] Step S4, adding a mask to the three-dimensional Gaussian volume with known opacity and color;

[0037] Specifically, step S4 includes the following steps:

[0038] Set a mask parameter Thus, a binary mask M∈{0, 1} is generated n , the expression is as follows:

[0039]

[0040] Where Γ(·) represents the stop gradient operator, represents the indicator function, sigmoid(·) represents the activation function, δ represents the mask threshold, n represents the number of three-dimensional Gaussian bodies, and i represents the i-th three-dimensional Gaussian body.

[0041] Add a mask to the position, size, and opacity of the three-dimensional Gaussian body. The expression is as follows:

[0042]

[0043]

[0044]

[0045] Where, Indicates the scale after masking, s n Indicates the scale before masking, Indicates the opacity after adding the mask, o n Indicates the opacity before adding the mask. Indicates the spatial position before adding the mask, h n Indicates the spatial position after masking.

[0046] Step S5, rasterizing the masked 3D Gaussian volume set to obtain a synthesized multi-view; completing one iteration;

[0047] Step S6, based on the target image and the synthesized multi-view, calculate the mean square error loss, perform back propagation optimization, optimize the scale, direction, and position of the three-dimensional Gaussian, optimize the local tensor grid, the first multi-layer perceptron, the second multi-layer perceptron, and the global appearance dictionary B c The parameters are set and the iteration is repeated until the synthesized multi-view is consistent with the target image.

[0048] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-view synthesis method based on tensor Gaussian mixture representation as described above. Figure 2 As shown in FIG, a hardware structure diagram of any device with data processing capability for the multi-view synthesis method based on tensor Gaussian mixture representation provided by an embodiment of the present invention is shown, except Figure 2 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0049] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the multi-view synthesis method based on tensor Gaussian mixture representation as described above. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities as described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0050] The above embodiments are intended only to illustrate the design concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. The scope of protection of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design concepts disclosed in the present invention are within the scope of protection of the present invention.

Claims

1. A multi-view synthesis method based on tensor Gaussian mixture representation, characterized in that: The method comprises: Initialize a set of three-dimensional Gaussian volumes; Set up a global appearance dictionary and extract the feature vector corresponding to each 3D Gaussian volume from the local tensor grid; Input each feature vector and the camera pose corresponding to the target image into the first multi-layer perceptron and the second multi-layer perceptron respectively to obtain the opacity and color corresponding to each three-dimensional Gaussian body; Add a mask to each 3D Gaussian volume with known opacity and color; Rasterize the masked 3D Gaussian volume set to obtain a composite multi-view; complete 1 iteration; Based on the target image and the synthesized multi-view, the mean square error loss is calculated and back-propagation optimization is performed to optimize the scale, orientation, and position of the 3D Gaussian. The parameters of the local tensor grid, the first multi-layer perceptron, the second multi-layer perceptron, and the global appearance dictionary are optimized. The iteration is repeated until the synthesized multi-view is consistent with the target image. The process of extracting the eigenvector corresponding to the three-dimensional Gaussian volume from the local tensor grid includes: The local tensor grid is factorized by CP decomposition, which is expressed as follows: ； Where, represents the feature vector associated with the position, represents the outer product, represents the number of tensor components, Represents the decomposed component tensor values, represents the feature dimension, Represents the vector values ​​in the x, y, and z directions respectively; Using the Matrix As a global appearance dictionary, this dictionary links the color correlations between different local tensors; for any position , after trilinear interpolation within each tensor, we obtain its feature representation including opacity and color: ； Where, 、 、 Respectively represent the vector values ​​obtained after interpolation at any position in the x, y, and z directions, Represents the component tensor value after decomposition and interpolation at any position, Represents the matrix resulting from the concatenation of all tensor components.

2. The multi-view synthesis method based on tensor Gaussian mixture representation according to claim 1, characterized in that: The initialized three-dimensional Gaussian volume set is denoted as ; Where, represents a normal distribution, Indicates spatial location, represents the covariance matrix, Indicates opacity, Represents the color value, Represents the number of three-dimensional Gaussian volumes.

3. The multi-view synthesis method based on tensor Gaussian mixture representation according to claim 1, characterized in that: The set of three-dimensional Gaussian volumes with known opacity and color is denoted as: ； Where, represents the first multi-layer perceptron, Represents the second multi-layer perceptron, and d represents the direction.

4. The multi-view synthesis method based on tensor Gaussian mixture representation according to claim 1, characterized in that: Adding a mask to a 3D Gaussian volume of known opacity and color involves: Set a mask parameter , thus generating a binary mask , the expression is as follows: ； Where, represents the stop gradient operator, represents the indicator function, represents the activation function, represents the mask threshold, Represents the number of three-dimensional Gaussian bodies, i represents the i-th three-dimensional Gaussian body; Add a mask to the position, size, and opacity of the three-dimensional Gaussian body. The expression is as follows: ； ； ； Where, represents the scale after masking, Indicates the scale before masking, Indicates the opacity after adding the mask. Indicates the opacity before adding the mask. Indicates the spatial position before masking. Indicates the spatial position after masking.

5. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multi-view synthesis method based on tensor Gaussian mixture representation according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-view synthesis method based on tensor Gaussian mixture representation is implemented as described in any one of claims 1 to 4.

7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the multi-view synthesis method based on tensor Gaussian mixture representation according to any one of claims 1 to 4 is implemented.

Images