A method for constructing a 3D scene representation model based on scene-adaptive local voxel multi-plane images

By introducing a voxel-level adaptive local multi-plane image representation method in three-dimensional scenes and combining it with an MLP network with a self-attention mechanism, the problem of poor local texture rendering in existing methods is solved, and higher quality three-dimensional scene rendering effects are achieved.

CN119229013BActive Publication Date: 2025-10-03HUAZHONG UNIV OF SCI & TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411331210.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-24
Publication Date
2025-10-03
Estimated Expiration
2044-09-24

AI Technical Summary

Technical Problem

Existing three-dimensional scene representation methods do not perform well in local texture rendering, especially in scenes with complex textures and fine structures. In addition, global learning methods cannot accurately express local information.

Method used

A method based on scene-adaptive local voxel multi-plane images is adopted to divide the three-dimensional scene into a voxel grid, construct a global multi-plane image for each voxel, and introduce a self-attention mechanism in the MLP network of each voxel. Through fine modeling of independently encoded voxel MLPs, the scene information of adjacent voxel blocks is coordinated to enhance consistency.

Benefits of technology

It significantly improves the rendering effect of three-dimensional scenes, can effectively capture the detailed features of local areas, improve the geometry and content rendering continuity of the scene, and is suitable for scenes with complex textures and fine structures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119229013B_ABST
    Figure CN119229013B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of 3D reconstruction and specifically relates to a method for constructing a 3D scene representation model based on scene-adaptive local voxel multi-plane images. The method comprises the following steps: dividing the 3D scene to be represented into a voxel grid and constructing a global multi-plane image (MPI) for each voxel; determining the intersection points of different light rays with each voxel MPI under each known viewing angle, encoding the 3D position coordinates of each intersection point corresponding to each light ray as a multi-dimensional position vector, and encoding the direction as a multi-dimensional direction vector. The set of multi-dimensional position vectors and direction vectors corresponding to each light ray, as well as the color label of the pixel corresponding to the light ray in the captured scene image, constitute a training sample; constructing a 3D scene representation network, including a multi-layer MLP network for each voxel position, a multi-layer MLP network for each voxel direction, and a color calculation unit; and training the network using the training samples to obtain a 3D scene representation model. The present invention can effectively capture detailed features of local areas.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional reconstruction, and more specifically, relates to a method for constructing a three-dimensional scene representation model based on scene-adaptive local voxel multi-plane images. Background Art

[0002] Existing 3D scene representation methods based on global multi-plane images decompose the 3D scene into multiple parallel forward-facing semi-transparent planes (MPIs) along the reference camera's viewing cone. These planes are constructed under the reference camera at a fixed depth perpendicular to the reference camera. Points on the planes are observed by the reference camera and the target camera, respectively. This perspective consistency allows the MPIs to represent the entire scene from a certain perspective with fewer sampled rays. A typical example is NeX, which uses a global MPI with a fixed depth distribution to represent 3D scenes and uses a large MLP network for global learning to encode the color and opacity information of the MPIs. However, this method is not rich enough for rendering local textures in 3D scenes, and the synthesized 3D scene images from a certain perspective are not very effective.

[0003] Recent research has improved the local texture representation of a scene by using local planes. For example, Neurmips uses point clouds to construct local planes. Specifically, it uses the SFM 3D reconstruction method based on colmap software to obtain a 3D point cloud (the 3D point cloud coordinates in the scene). The 3D point cloud is clustered into multiple local planes, and a large global MLP network is used to learn the texture and geometric features of the local planes. Finally, a small MLP network is used for distillation learning and fine-tuning to achieve the goal of local scene learning. However, this method still fails to achieve good rendering results. Summary of the Invention

[0004] In response to the defects of the existing technology and the need for improvement, the present invention provides a method for constructing a three-dimensional scene representation model based on scene-adaptive local voxel multi-plane images, which aims to effectively capture the detailed features of local areas and improve the rendering effect of three-dimensional scenes.

[0005] To achieve the above object, according to one aspect of the present invention, a method for constructing a three-dimensional scene representation model based on scene-adaptive local voxel multi-plane images is provided, comprising:

[0006] The three-dimensional scene to be represented is divided into a voxel grid, and a global multi-plane image (MPI) of each voxel is constructed. The intersection points of different light rays with the MPI of each voxel under each known viewing angle are determined, and the three-dimensional position coordinates of each intersection point corresponding to each light ray are encoded as multi-dimensional position vectors, and the direction is encoded as a multi-dimensional direction vector. The set of multi-dimensional position vectors and multi-dimensional direction vectors corresponding to the light ray and the color label of the pixel corresponding to the light ray in the acquired scene image constitute a training sample.

[0007] Construct a position multi-layer MLP network, a direction multi-layer MLP network and a color calculation unit for each voxel; the position multi-layer MLP network includes n layers of self-attention units connected to the output ends of the first n fully connected layers, which are respectively used to divide the feature vector output by the previous fully connected layer into multiple non-overlapping windows, and perform self-attention learning on each window. The learning results of each window are spliced ​​and input into the next fully connected layer; the input of the position multi-layer MLP network is the multi-dimensional position vector of all intersections of each light on the corresponding voxel of the network, and the output is the color coefficient vector and opacity corresponding to all intersections; the input of the direction multi-layer MLP network is the multi-dimensional direction vector of all intersections of each light on the corresponding voxel of the network, and the output is the basis function corresponding to all intersections; the color calculation unit is used to calculate the color corresponding to all intersections based on the basis function and color coefficient vector of all intersections of each received light on all voxels, combined with the basic color vector to be optimized, so as to synthesize the opacity corresponding to all intersections to obtain the pixel rendering color corresponding to the light;

[0008] The training samples are used to optimize the position multi-layer MLP network, orientation multi-layer MLP network and basic color vector of each voxel until the preset termination condition is reached to obtain a three-dimensional scene representation model.

[0009] Furthermore, a sine function with a frequency length of m is used to perform m-dimensional vector encoding to obtain the multi-dimensional position vector and the multi-dimensional direction vector.

[0010] Furthermore, the encoding method of the multidimensional position vector is as follows: using a sine function with a frequency length of 20 to encode the plane position coordinates of each intersection point corresponding to each ray into a 40-dimensional vector; using a sine function with a frequency length of 8 to encode the depth coordinates of each intersection point corresponding to the ray into a 16-dimensional vector; splicing the 40-dimensional vector and the 16-dimensional vector to obtain the multidimensional position vector of the corresponding intersection point;

[0011] The encoding method of the multi-dimensional direction vector is: using a sine function with a frequency length of 4 to encode the direction of each intersection point corresponding to each light ray into an 8-dimensional vector, to obtain the multi-dimensional direction vector of the intersection point.

[0012] Furthermore, the value of n is 2.

[0013] Furthermore, the calculation of the color and the opacity is performed using a preset linear relationship between a basis function set, a color coefficient vector set, and a basic color vector.

[0014] The present invention also provides a method for synthesizing a new perspective of a three-dimensional scene, which uses the three-dimensional scene representation model constructed by the three-dimensional scene representation model construction method as described above to perform color rendering of pixels corresponding to multiple light rays under a new perspective, thereby synthesizing a new perspective image.

[0015] The present invention also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the above-mentioned three-dimensional scene representation model construction method and / or the above-mentioned three-dimensional scene new perspective synthesis method.

[0016] The present invention also provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the three-dimensional scene representation model construction method and / or the three-dimensional scene new perspective synthesis method as described above.

[0017] The present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the above-mentioned three-dimensional scene representation model construction method and / or the above-mentioned three-dimensional scene new perspective synthesis method.

[0018] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0019] (1) The present invention proposes a voxel-based multi-plane three-dimensional scene representation method, which constructs a local MPI at the voxel level (that is, constructs an MPI for each voxel), retaining the simplicity and perspective consistency of the MPI structure, and at the same time finely models each local MPI through the independently encoded MLP (implicit neural radiation field) of each voxel, which can effectively capture the detailed features of each local area. Among them, in order to solve the block effect existing in the independent modeling of each voxel, a voxel plane attention mechanism is introduced into the MLP network of each voxel, which enhances the content and geometric consistency of the scene by coordinating the scene information independently encoded in adjacent voxel blocks, realizes scene adaptation, improves scene expression performance, ensures modeling accuracy, and can significantly improve the continuity of scene geometry and content rendering. The three-dimensional scene representation model obtained by this method is more refined for the representation of the three-dimensional scene, and can achieve better subsequent new perspective rendering effects in scenes with complex geometry and color textures. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1A flowchart of a method for constructing a three-dimensional scene representation model based on scene-adaptive local voxel multi-plane images provided by an embodiment of the present invention;

[0021] Figure 2 A voxel multi-plane construction diagram provided by an embodiment of the present invention;

[0022] Figure 3 A voxel fully connected network graph with a voxel attention block provided by an embodiment of the present invention;

[0023] Figure 4 End-to-end learning of color and opacity maps provided by embodiments of the present invention; DETAILED DESCRIPTION

[0024] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0025] Example 1

[0026] A method for constructing a 3D scene representation model based on scene-adaptive local voxel multi-plane images, such as Figure 1 Shown, including:

[0027] The three-dimensional scene to be represented is divided into a voxel grid, and a global multi-plane image (MPI) of each voxel is constructed. The intersection points of different light rays with the MPI of each voxel under each known viewing angle are determined, and the three-dimensional position coordinates of each intersection point corresponding to each light ray are encoded as multi-dimensional position vectors, and the direction is encoded as a multi-dimensional direction vector. The set of multi-dimensional position vectors and multi-dimensional direction vectors corresponding to the light ray and the color label of the pixel corresponding to the light ray in the acquired scene image constitute a training sample.

[0028] Construct a 3D scene representation network, including a multi-layer MLP network for the position of each voxel, a multi-layer MLP network for the direction, and a color calculation unit; wherein, the multi-layer MLP network for the position contains n layers of self-attention units connected to the output ends of the first n layers of fully connected layers, which are used to divide the feature vector output by the previous fully connected layer into a series of non-overlapping windows, and perform self-attention learning on each window, and splice the self-attention learning results of each window and input them into the next fully connected layer; the input of the multi-layer MLP network for the position is the multi-dimensional position vector of all intersections of each light on the corresponding voxel, and the output is the total intersection of each light on the corresponding voxel. The color coefficient vector set and opacity set of the intersection points are as follows: the input of the directional multi-layer MLP network is the multidimensional direction vector of all the intersection points of each ray on the corresponding voxel, and the output is the basis function set of all the intersection points of each ray on the corresponding voxel; the color calculation unit is used to calculate the color set and opacity set of all the intersection points of each ray on the corresponding voxel based on the basis function set and color coefficient vector set of all the intersection points of each ray on all voxels, combined with the basic color vector to be optimized, so as to perform opacity synthesis on all the intersection points of each ray with all voxels to obtain the pixel rendering color corresponding to the ray;

[0029] The training samples are used to optimize the position multi-layer MLP network, orientation multi-layer MLP network and basic color vector of each voxel until the preset termination condition is reached to obtain a three-dimensional scene representation model.

[0030] Considering that the existing multi-plane learning method using fixed depth and global learning cannot capture the local geometric features and texture information of the scene in detail, especially in scenes with complex textures and fine structures, the global multi-plane learning method will cause texture aliasing or missing when rendering the scene. The existing three-dimensional scene expression method based on local planes constructs local planes through point clouds, which can effectively capture geometric shapes and textures in simple scenes. However, this method ignores the correlation between cameras, resulting in the plane structure (position and normal vector) being too flexible. When using a large global MLP network to learn the texture and geometric features of the local plane, it is difficult to accurately learn the local plane (color, opacity) used for local texture rendering, resulting in holes or repetitions in the texture of the new perspective image rendered based on the local plane. In addition, it relies on the global MLP network learning method to fit the local plane without independent modeling, making it still a challenge to accurately express local information in complex scenes.

[0031] This embodiment proposes a voxel-based multi-plane 3D scene representation method. In scenes with complex geometry and color textures, this method uses a reference camera to construct a local MPI at the voxel level (that is, to construct an MPI for each voxel), retaining the simplicity and perspective consistency of the MPI structure. At the same time, each local MPI is finely modeled through independently encoded voxel MLPs (implicit neural radiance fields). To address the blocking effect of independent modeling of each voxel, a voxel plane attention mechanism is introduced into the MLP network of each voxel. By coordinating the scene information independently encoded in adjacent voxel blocks, the content and geometric consistency of the scene are enhanced.

[0032] The above method first constructs a local voxel multi-plane image MPI. Specifically, Figure 2 As shown, first the coordinates of the multi-plane image Define an axis-aligned bounding box (AABB), denoted as bd min :(x min ,y min , d min ), bd max :(x max ,y max , d max ), where bd min , bd max The entire scene is then subdivided into blocks with a resolution of M = M x ·M y ·M d Assume that each voxel V i,j,k Located at a specific position determined by its index in the 3D grid, where i, j, and k correspond to the voxel's index (number) in the direction. Inside each voxel, there is a global multiplanar image corresponding to its position, which is a local multiplanar image of the entire 3D scene.

[0033] For each specific voxel V i,j,k , where the multi-plane coordinate set can be defined as P i,j,k , where one of the plane coordinates x: (x i ,y j , d k ), and assigned to the corresponding three-dimensional voxels through voxel identification, as follows:

[0034] V(x)=[(x-bd min ) / ((bd max -bd min ) / M)]

[0035] Here V(x)∈V i,j,k, each voxel corresponds to two voxel MLP (multi-layer perceptron) networks containing two sets of independent parameters, fitting the color and opacity of multiple planes within the voxel.

[0036] Secondly, it involves voxel fully connected networks and attention mechanisms. Unlike the large global MLP network outlined in NeX, voxel MLP uses a small fully connected network that is learned independently within each voxel, such as Figure 3 Specifically, consider a point pixel x in a plane within a voxel: (x i ,y j , d k ), preferably, according to the formula: γ L (u)=[sin(u), cos(u),..., sin(2 L-1 u,2 L-1 u), cos(2 L-1 x, 2 L-1 u)], where u represents the three-dimensional position coordinates or direction information of a point in a plane, and L represents the encoding dimension. Preferably, a sine function with a frequency length of 20 is used to convert a position (x i ,y j ) is encoded into 40 dimensions. Similarly, a sine function with a frequency length of 8 is used to encode the depth d k Encoded into 16 dimensions, the vector of the 40-dimensional and 16-dimensional splicing of a point is used as the multidimensional position vector E. The plane direction is converted into Converted to 8 dimensions, as the multidimensional direction vector dir of a point.

[0037] Each multidimensional position vector E is input into the position multilayer MLP network F of the voxel where the multidimensional position vector E corresponds to, and is converted into a D-dimensional point embedding T through the first fully connected layer in the position multilayer MLP network. i,j,k (Eigenvector of each local MPI), Add the bias vector T i,j,k =W i,j,k E i,j,k +b i,j,k .

[0038] In order to promote the collaborative learning of local multi-planar representations across different voxel blocks, a voxel attention block is designed. The above D-dimensional embedding is input into the voxel attention block, where the attention network adopts a lightweight self-attention module: MobileAttention V2 to effectively balance model complexity and performance. This embedding T i,j,k is divided into a series of non-overlapping small windows: T i,j,k →{(Ti,j,k )1,(T i,j,k )2,...,(T i,j,k ) W}, each window contains L points. Self-attention is then applied independently within each window. The self-attention outputs of these windows are combined using a multi-head self-attention mechanism, expressed as:

[0039] As a preferred implementation, there are 1 to 2 voxel attention layers, and after stacking learning of voxel fully connected layers, the last layer of the fully connected network outputs color coefficients and opacity.

[0040] In order to better learn the color variation of local voxel global multi-plane image with view angle, the plane p in the voxel global multi-plane MPI is transformed into l Color Parameterized as viewing direction dir=(dir x ,dir y ) basis function. The pixel color is determined by the basic color vector b(x i ,y j , d k ), and the basis functions related to the light direction The linear combination is expressed as: p l represents the lth local plane in voxel (i, j, k), and N represents the order (dimension).

[0041] Each voxel involves two multi-layer MLP networks: the position multi-layer MLP network F and the direction multi-layer MLP network G. Among them, the position multi-layer MLP network F concatenates the multi-dimensional position vector corresponding to a point and its corresponding embedding vector and maps it to opacity and color coefficients Expressed as:

[0042] At the same time, another multi-layer MLP network, the directional multi-layer MLP network G, is specifically used to understand the basis function based on the observation direction dir, which is expressed as:

[0043]

[0044] Here, Representing voxels The pth l The transparency value of the global multi-plane image, b(x i ,y j , d k ), and Respectively represent the pth voxel (i, j, k)l The base color vector set, color coefficient set and basis function set of a global multi-plane image. F, G are two independently trained multi-layer MLP networks for each voxel (i, j, k). It is worth noting that the scale of the F network (i.e., the position multi-layer MLP network F) is much smaller than that of the existing MPI-based global learning method NeX. It can be used as a preferred method and can have only 64 hidden neurons, balancing complexity and accuracy. In this case, its network parameters are only 1 / 36 of the F network in nex, while the scale of the G network is very close to nex, maintaining the same accuracy as neX. x The same number of layers as the G network.

[0045] like Figure 4 As shown, the color calculation unit is used to calculate the color set and opacity set corresponding to all intersections based on the basis function set and color coefficient vector set of all intersections of each received light ray and each voxel MPI, combined with the basic color vector to be optimized, to perform opacity synthesis on all the intersections and obtain the pixel rendering color corresponding to the light ray.

[0046] Specifically, to optimize the base color, MLP, and attention block weights, Voxel-MPI is rendered into the target view and the loss value for backpropagation is calculated. Specifically, this embodiment uses a variant of the Axis Aligned Bounding Box (AABB) algorithm to find the exact intersection of the line of sight with the voxel's local multiplane. Then, using bilinear interpolation and inference MLP, the color of the target pixel is calculated by alpha synthesis in the following formula:

[0047]

[0048] Among them, N L is the number of planes that the ray intersects with Voxel-MPI, and Respectively represent the pth voxel in voxel (i, j, k) l The color image and transparency in the sub-plane are calculated. The loss value between the rendered result and the real result is calculated. The weights and base colors of the two multi-layer Voxel MLP are optimized by minimizing the following loss function: in, and I t They are rendered images and real images, represents the gradient operator, TV(b) represents the total variational error of the base color image, and ω and γ are balancing weights.

[0049] Compared with existing methods, the results of this embodiment show that the voxel multi-planar method surpasses the existing multi-planar method and the Nerf method in multiple data sets, and compared with the benchmark Nex, the voxel multi-planar method produces better results while requiring less memory and training time.

[0050] Example 2

[0051] A method for synthesizing a new perspective of a three-dimensional scene uses the three-dimensional scene representation model constructed by the three-dimensional scene representation model construction method as described above to perform color rendering of pixels corresponding to multiple light rays under a new perspective, thereby synthesizing a new perspective image.

[0052] The relevant technical solutions are the same as those in Example 1 and will not be described again here.

[0053] Example 3

[0054] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, it implements the three-dimensional scene representation model construction method described in the first embodiment and / or the three-dimensional scene new perspective synthesis method described in the second embodiment.

[0055] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory may be used to store computer programs and / or modules, and the processor may perform various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory.

[0056] The relevant technical solutions are the same as those in the above embodiment and will not be described again here.

[0057] Example 4

[0058] A computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device containing the storage medium is controlled to execute the three-dimensional scene representation model construction method described in the first embodiment and / or the three-dimensional scene new perspective synthesis method described in the second embodiment.

[0059] Specifically, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0060] The relevant technical solutions are the same as those in the above embodiment and will not be described again here.

[0061] Example 5

[0062] A computer program product includes a computer program or instructions, which, when executed by a processor, implements the three-dimensional scene representation model construction method described in the first embodiment and / or the three-dimensional scene new perspective synthesis method described in the second embodiment.

[0063] The relevant technical solutions are the same as those in the above embodiment and will not be described again here.

[0064] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a three-dimensional scene representation model based on scene-adaptive local voxel multi-plane images, characterized in that: include: The three-dimensional scene to be represented is divided into a voxel grid, and a global multi-plane image MPI of each voxel is constructed; Determine the intersection points of different rays with each voxel MPI under each known viewing angle, encode the three-dimensional position coordinates of each intersection point corresponding to each ray into a multidimensional position vector, and encode the direction into a multidimensional direction vector. The set of multidimensional position vectors and multidimensional direction vectors corresponding to the ray, as well as the color label of the pixel corresponding to the ray in the acquired scene image, constitute a training sample; Construct a position multi-layer MLP network, a direction multi-layer MLP network and a color calculation unit for each voxel; the position multi-layer MLP network includes n layers of self-attention units connected to the output ends of the first n fully connected layers, which are respectively used to divide the feature vector output by the previous fully connected layer into multiple non-overlapping windows, and perform self-attention learning on each window. The learning results of each window are spliced ​​and input into the next fully connected layer; the input of the position multi-layer MLP network is the multi-dimensional position vector of all intersections of each light on the corresponding voxel of the network, and the output is the color coefficient vector and opacity corresponding to all intersections; the input of the direction multi-layer MLP network is the multi-dimensional direction vector of all intersections of each light on the corresponding voxel of the network, and the output is the basis function corresponding to all intersections; the color calculation unit is used to calculate the color corresponding to all intersections based on the basis function and color coefficient vector of all intersections of each received light on all voxels, combined with the basic color vector to be optimized, so as to synthesize the opacity corresponding to all intersections to obtain the pixel rendering color corresponding to the light; The training samples are used to optimize the position multi-layer MLP network, orientation multi-layer MLP network and basic color vector of each voxel until the preset termination condition is reached to obtain a three-dimensional scene representation model.

2. The method for constructing a three-dimensional scene representation model according to claim 1, wherein: A sine function with a frequency length of m is used to perform m-dimensional vector encoding to obtain the multi-dimensional position vector and the multi-dimensional direction vector.

3. The method for constructing a three-dimensional scene representation model according to claim 2, wherein: The multidimensional position vector is encoded as follows: using a sine function with a frequency length of 20 to encode the plane position coordinates of each intersection point corresponding to each ray into a 40-dimensional vector; using a sine function with a frequency length of 8 to encode the depth coordinates of each intersection point corresponding to the ray into a 16-dimensional vector; and concatenating the 40-dimensional vector and the 16-dimensional vector to obtain the multidimensional position vector of the corresponding intersection point; The encoding method of the multi-dimensional direction vector is: using a sine function with a frequency length of 4 to encode the direction of each intersection point corresponding to each light ray into an 8-dimensional vector, to obtain the multi-dimensional direction vector of the intersection point.

4. The method for constructing a three-dimensional scene representation model according to claim 1, wherein: The value of n is 2.

5. The method for constructing a three-dimensional scene representation model according to claim 1, wherein: The calculation of the color and the opacity is performed using a preset linear relationship between a basis function set, a color coefficient vector set, and a basic color vector.

6. A method for synthesizing a new perspective of a three-dimensional scene, characterized in that: The three-dimensional scene representation model constructed by the three-dimensional scene representation model construction method according to any one of claims 1 to 5 is used to perform color rendering of pixels corresponding to multiple light rays under a new perspective, thereby synthesizing a new perspective image.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method for constructing a three-dimensional scene representation model according to any one of claims 1 to 5 or the method for synthesizing a new perspective of a three-dimensional scene according to claim 6 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed by a processor, the device where the storage medium is located is controlled to execute the three-dimensional scene representation model construction method as described in any one of claims 1 to 5 or the three-dimensional scene new perspective synthesis method as described in claim 6.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method for constructing a three-dimensional scene representation model according to any one of claims 1 to 5 or the method for synthesizing a new perspective of a three-dimensional scene according to claim 6 is implemented.

Citation Information

Patent Citations

  • Pyramid attention-based scene recognition method and device, scene recognition model training method and device

    CN113822232A

  • Virtual viewpoint synthesis method based on multi-plane image type parameter coding

    CN115578474A