Free viewpoint video synthesis method, device, electronic device and storage medium

Abstract

The present invention relates to a free viewpoint video synthesis method, device, electronic device and storage medium. The free viewpoint video synthesis method of the present invention comprises: obtaining a video to be synthesized, the video comprising voxels with depth and color; for each voxel, aggregating voxel embeddings of 8 vertices to obtain a representation of a query point inside the voxel; using latent code embedding to represent the representation of the query point inside the voxel learned at different time frames to obtain a deformable latent code network; inputting the deformable latent code network into a sparse convolutional network to obtain a dense latent code; constructing a neural radiation field based on a multilayer perceptron according to the dense latent code; inputting the three-dimensional position and viewpoint direction of each voxel point into the multilayer perceptron network model of the neural radiation field to obtain the color and density of each voxel; performing neural rendering on the neural radiation field to obtain a video under a free viewpoint. The method of the present invention implicitly represents the scene content with voxels to achieve the synthesis of free viewpoint views.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a free viewpoint video synthesis method, device, electronic equipment and storage medium. Background Art

[0002] Images and videos are one of the important means for humans to obtain information. With the development of digital technology with images and videos as the core, the popularization of three-dimensional (3D) visual video, monitoring and other service applications, three-dimensional multimedia services can attract more attention from users than traditional two-dimensional ones, allowing users to have an immersive experience. In 3D video technology, people use binocular or multi-eye cameras to shoot the same scene from different angles, and then use computer vision technology to form a stereoscopic video. However, whether it is traditional two-dimensional video or three-dimensional video, viewers can only rely on the placement of the camera to obtain video information from a certain angle. In order to change this passive way of receiving information, people have proposed multi-view video technology, which can select the desired viewing angle and position on the device terminal. In order to realize this technology, it is necessary to arrange camera acquisition equipment at the corresponding viewpoint position around the captured scene. With the development of digital video technology, it has developed from video with two viewpoints to multi-view video, and then developed to free viewpoint video (FVV) technology based on this. Among them, free viewpoint video technology has the strongest immersive experience. Users can watch videos at any angle without being restricted by the number of viewpoints.

[0003] FVV technology is widely used, such as live broadcast of sports events (fencing and dance, etc.) and 3D immersive classrooms, etc., and its application prospects are broad, but its popularization still faces many difficulties. In order to obtain videos from any viewpoint, the traditional method is to place multiple cameras at viewpoints around the observed object, and the data generated also increases exponentially. The existing hardware equipment obviously cannot meet the requirements of watching videos from any viewpoint. In order to reduce the pressure of data storage and network transmission, relying on virtual viewpoint synthesis technology, that is, synthesizing videos of the same scene under unknown viewpoints through the collected video data, the number of camera acquisition devices can be reduced. How to improve the quality of scene videos under free viewpoints and improve rendering efficiency is still a challenging task.

[0004] The process of synthesizing new viewpoints is also the process of modeling the scene. The core of Neural Radiance Field (NeRF) is a neural implicit representation method. Figure 1 As shown in Figure 1, a fully connected neural network is used to fit a continuous voxel representation. This fully connected network inputs the desired position (x, y, z), and the viewing angle (θ, ), output the color (r, g, b) of the position and the transparency (σ) of the point, that is, given multiple pictures of the same scene at different positions and the corresponding camera positions, a whole multi-layer perceptron (Multilayer Perceptron) network is used to render pictures of the scene at any angle.

[0005] Because NeRF needs to optimize the representation of each scene independently, over-evaluation along each view ray results in high computational cost and a lot of computing time. In addition, NeRF is not suitable for implicitly representing dynamic scenes. If dynamic video is directly input, objects in the scene will be occluded due to time inconsistency when synthesizing new views, resulting in a large number of blurred new views. Summary of the invention

[0006] Based on this, the purpose of the present invention is to provide a free viewpoint video synthesis method, device, electronic device and storage medium, construct a deformable network latent code, record scene texture and motion information at different time frames, share scene representation, and propagate scene content across time, so that the audience can obtain scene information without being restricted by the placement of the camera, thereby realizing the synthesis of free viewpoint views.

[0007] In a first aspect, the present invention provides a free viewpoint video synthesis method, comprising the following steps:

[0008] Obtain a video to be synthesized, wherein the video consists of RGB-D video frames, and the RGB-D video frames include voxels with depth and color;

[0009] For each voxel, aggregate the voxel embeddings of the eight vertices of the voxel to obtain a representation of the query point inside the voxel;

[0010] For the representation of the query point inside the voxel, latent code embedding is used to represent the representation of the query point inside the voxel learned at different time frames, so as to obtain a deformable latent code network composed of latent code embeddings on different time frames;

[0011] Inputting the deformable latent code network into a sparse convolutional network to obtain a dense latent code formed by diffusing the sparse latent code in the deformable latent code network into a nearby 3D space;

[0012] According to the dense latent code, a neural radiation field based on a multi-layer perceptron is constructed;

[0013] Inputting the three-dimensional position and viewpoint direction of each voxel point into the multi-layer perceptron network model of the neural radiation field to obtain the color and density of each voxel;

[0014] The neural radiation field is subjected to neural rendering to obtain a video under a free viewpoint.

[0015] Further, performing neural rendering on the neural radiation field includes:

[0016] For each voxel point, the color c and density σ of the point are mapped through the implicit field according to the voxel data, and the color of the two-dimensional pixel point corresponding to the voxel is rendered through volume rendering technology;

[0017] Using volume rendering technology, dense sampling is performed in non-empty areas along the light rays to render the corresponding scene appearance and geometric information at different three-dimensional positions, and obtain videos under free viewpoints.

[0018] Further, for each voxel, the voxel embeddings of the eight vertices of the voxel are aggregated to obtain a representation of the query point inside the voxel, including:

[0019] For the 8 vertices of each voxel, a trilinear interpolation method is used to form a feature representation embedding;

[0020] The feature representation embeddings of the eight vertices are aggregated to obtain the representation of the query point inside the voxel.

[0021] Further, the sparse convolutional network includes three layers connected in sequence;

[0022] Each layer consists of a sparse convolutional layer, a batch normalization layer, and a renormalization layer connected sequentially.

[0023] Furthermore, the neural radiation field based on the multi-layer perceptron includes a plurality of multi-layer perceptrons;

[0024] Each multi-layer perceptron processes a portion of the scene corresponding to the multi-layer perceptron individually.

[0025] Furthermore, the three-dimensional position and viewpoint direction of each voxel point are input into the multi-layer perceptron network model of the neural radiation field to obtain the color of each voxel, including:

[0026] The color at voxel point x in frame t is calculated using the following formula:

[0027] c t (x) = M c (ψ(x,z,S t ),γ d (d),γ x (x),L t )

[0028] Among them, c t (x) is the color of voxel point x in frame t, M c represents a two-layer MLP network, ψ(x,Z,S t ) is the latent code of the voxel point x, S trepresents the parameters of the deformable latent code network, Z represents the dense latent code after sparse transformation; γ d and γ x are the position encoding functions for viewing direction and spatial position, L t represents latent embedding.

[0029] Furthermore, the three-dimensional position and viewpoint direction of each voxel point are input into the multi-layer perceptron network model of the neural radiation field to obtain the density of each voxel, including:

[0030] The density at voxel point x in frame t is calculated using the following formula:

[0031] σ t (x) = M σ (ψ(x,Z,S t ))

[0032] Among them, σ t (x) is the density at voxel point x in frame t, M σ represents a four-layer MLP network, ψ(x,Z,S t ) is the latent code of the voxel point x, S t represents the parameters of the deformable latent code network, and Z represents the dense latent code after sparse transformation.

[0033] In a second aspect, the present invention further provides a free viewpoint video synthesis device, comprising:

[0034] A video acquisition module, used to acquire a video to be synthesized, wherein the video is composed of RGB-D video frames, and the RGB-D video frames include voxels with depth and color;

[0035] A query point representation calculation module is used for aggregating the voxel embeddings of the eight vertices of each voxel to obtain a representation of the query point inside the voxel;

[0036] A latent code network acquisition module is used to represent the representation of the query point inside the voxel using latent code embedding to represent the representation of the query point inside the voxel learned at different time frames, so as to obtain a deformable latent code network composed of latent code embeddings on different time frames;

[0037] A dense latent code acquisition module is used to input the deformable latent code network into a sparse convolutional network to obtain a dense latent code formed by diffusing the sparse latent code in the deformable latent code network to a nearby 3D space;

[0038] A neural radiation field construction module, used to construct a neural radiation field based on a multi-layer perceptron according to the dense latent code;

[0039] A voxel information calculation module, used for inputting the three-dimensional position and viewpoint direction of each voxel point into the multi-layer perceptron network model of the neural radiation field to obtain the color and density of each voxel;

[0040] A neural rendering module is used to perform neural rendering on the neural radiation field to obtain a video under a free viewpoint.

[0041] In a third aspect, the present invention further provides an electronic device, characterized in that it includes:

[0042] at least one memory and at least one processor;

[0043] The memory is used to store one or more programs;

[0044] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a free viewpoint video synthesis method as described in any one of the first aspects of the present invention.

[0045] In a fourth aspect, the present invention further provides a computer-readable storage medium, characterized in that:

[0046] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a free viewpoint video synthesis method as described in any one of the first aspects of the present invention are implemented.

[0047] The free viewpoint video synthesis method, device, electronic device and storage medium provided by the present invention can make the transition color at the intersection of point light sources no longer just a monotonous color, but a fusion of the mixed colors of multiple light sources at the intersection, mixing the colors of multiple lights at the intersection of the light sources, thereby improving the final coloring effect of the scene.

[0048] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a network architecture based on a multi-layer perceptron;

[0050] Figure 2 A schematic diagram of a flow chart of a free viewpoint video synthesis method provided by the present invention;

[0051] Figure 3 This is a framework diagram of the MLP-based implicit neural radiation field scene model used in the present invention;

[0052] Figure 4 This is a schematic diagram of the structure of the sparse convolutional network used in the present invention;

[0053] Figure 5A schematic diagram of the intersection of light and a scene in one embodiment;

[0054] Figure 6 A schematic structural diagram of a free viewpoint video synthesis device provided by the present invention. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0056] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0057] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0058] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0059] In addition, in the description of this application, unless otherwise specified, "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0060] In a specific embodiment, the present invention provides a free viewpoint video synthesis method, which uses dynamic scene videos under multi-viewing angles obtained by three handheld commercial-grade RGB-D sensors, Microsoft Azure Kinect, as input, and adopts the following method to efficiently synthesize scene videos under free viewpoints.

[0061] Through NeRF, we can see that learning implicit neural representations of static 3D scenes can achieve higher view synthesis quality when the input views are dense. If dynamic videos are directly input into the static NeRF model to learn scene representation, the learned new views will become blurry.

[0062] Before implementing the free viewpoint video synthesis method provided by the present invention, it is necessary to calibrate the three Kinect devices and configure the sensor parameters to obtain consistent results between the devices. The Volumetric Capture multi-sensor capture system invented by the team of Nikolaos Zioulis et al. is used to record the captured RGB-D frames and generate voxel or point cloud data with depth and color at three different viewing angles.

[0063] Based on the above hardware devices, in view of the problems in the background technology, the present application embodiment provides a free viewpoint video synthesis method, such as Figure 2 As shown, the method comprises the following steps:

[0064] S01: Obtain a video to be synthesized, wherein the video is composed of RGB-D video frames, and the RGB-D video frames include voxels with depth and color.

[0065] Voxel is the smallest unit of data segmentation in three-dimensional space, similar to pixel being the smallest unit of two-dimensional space. Voxel uses a cube block of fixed size as the smallest unit.

[0066] S02: For each voxel, aggregate the voxel embeddings of the eight vertices of the voxel to obtain a representation of a query point inside the voxel.

[0067] In a preferred embodiment, the RGB-D sequence of the scene captured by the Kinect device is converted into a voxel data set containing scene information through a multi-sensor capture system, each voxel has eight vertices, and the feature representation embedding is formed for these eight vertices using a trilinear interpolation method, and the representation of the query point inside the voxel is obtained by aggregating the feature representation embedding of these eight vertices.

[0068] Trilinear interpolation is a method of linear interpolation on a tensor product grid of three-dimensional discrete sampled data. This tensor product grid may have any non-overlapping grid points in each dimension, but it is not a triangulated finite element analysis grid. This method linearly approximates the value of a point (x, y, z) by using the data points on the grid to approximate the value of a point on a local rectangular prism.

[0069] S03: For the representation of the query point inside the voxel, latent code embedding is used to represent the representation of the query point inside the voxel learned at different time frames, and a deformable latent code network composed of latent code embeddings on different time frames is obtained.

[0070] Preferably, the latent code is another simpler representation of the multi-dimensional pixel points of the scene, and the latent code is embedded in L t To represent the representation of query points inside voxels learned at different time frames, the latent embeddings at different time frames are combined into a deformable latent network (z1, z2.....z t ).

[0071] S04: Inputting the deformable latent code network into a sparse convolutional network to obtain a dense latent code formed by diffusing the sparse latent code in the deformable latent code network to a nearby 3D space.

[0072] Since the latent code is relatively sparse in three-dimensional space, directly inserting the latent code will cause the vectors of most three-dimensional points to be zero. Therefore, the sparse convolutional network architecture is used to process the sparse latent code, and the sparse latent code in the deformable latent code network is diffused to the nearby 3D space to form a denser latent code. After processing, the dense latent codes at each time frame t represent different neural radiation fields. At the same time, the dense latent code at any video frame can also be mapped to other video frames, so that the implicit field can integrate and share the observations at different video frames.

[0073] Preferably, Figure 4 As shown, the sparse convolutional network used in the present invention has three layers in total, each layer consists of three parts: sparse convolution, batch normalization, and renormalization, which can efficiently process the input latent code.

[0074] S05: Constructing a neural radiation field based on a multi-layer perceptron according to the dense latent code.

[0075] Preferably, the neural radiation field based on the multi-layer perceptron includes multiple multi-layer perceptrons; each multi-layer perceptron independently processes a part of the scene corresponding to the multi-layer perceptron.

[0076] When light passes through the radiation field, NeRF needs to query the MLP millions of times, which seriously slows down the rendering process. Therefore, in the setting of the present invention, instead of using a large MLP to represent the entire scene, it is decomposed into several smaller, faster-evaluated MLPs, and each individual MLP only needs to process part of the scene. In this way, combined with further optimization, the rendering speed is faster than the original NeRF model without incurring higher storage costs.

[0077] S06: Input the three-dimensional position and viewpoint direction of each voxel point into the multi-layer perceptron network model of the neural radiation field to obtain the color and density of each voxel.

[0078] Preferably, Figure 3 As shown, the spatial position, viewpoint direction and latent code contained in each voxel data in the 3D space are embedded into the input implicit field.

[0079] For any voxel data in the implicit field, at each time frame t, the volume density at the voxel point x is predicted as a function of the latent code ψ(x, Z, S t ).

[0080] Preferably, the density at voxel point x in frame t is calculated using the following formula:

[0081] σ t (x) = M σ (ψ(x,Z,S t ))

[0082] Among them, σ t (x) is the density at voxel point x in frame t, M σ represents a four-layer MLP network, ψ(x, Z, S t ) is the latent code of the voxel point x, S t represents the parameters of the deformable latent code network, and Z represents the dense latent code after sparse transformation.

[0083] The latent code ψ(x, Z, S t ) and viewing direction d are used as inputs for color regression. To model the position-dependent incident light, the color model also takes the spatial position x as input. Observing the time-varying factors that affect the scene appearance, such as secondary lighting and self-occlusion, we assign a latent embedding L to each video frame t. t to encode factors that change over time. That is, for frame t, the color at x is predicted as the latent code View direction d, spatial location x, and latent embedding L t function.

[0084] Preferably, the color at voxel point x in frame t is calculated using the following formula:

[0085] c t (x) = M c (ψ(x,z,S t ), γ d (d), γ x (x), L t )

[0086] Among them, c t (x) is the color of voxel point x in frame t, M c represents a two-layer MLP network, ψ(x, Z, S t ) is the latent code of the voxel point x, S t represents the parameters of the deformable latent code network, Z represents the dense latent code after sparse transformation; γ d and γ x are the position encoding functions for viewing direction and spatial position, L t represents latent embedding.

[0087] S07: Perform neural rendering on the neural radiation field to obtain a video under a free viewpoint.

[0088] In a preferred embodiment, for each voxel point, the color c and density σ of the point are mapped through the implicit field according to the voxel data, and the color of the two-dimensional pixel point corresponding to the voxel is rendered by volume rendering technology. Using volume rendering technology, dense sampling is performed in non-empty areas along the light, and the appearance and geometric information of the corresponding scene at different three-dimensional positions are rendered to obtain the video under the free viewpoint.

[0089] Compared to NeRF's rendering of the neural implicit representation of the entire space, the rendering of the present invention is more efficient because it avoids rendering voxels in empty areas and terminates ray marching early.

[0090] Preferably, rendering is performed in two steps:

[0091] ① Ray-voxel intersection

[0092] The ray r intersects the scene as Figure 5 As shown. The voxel data is mapped to the color c and density σ of the point through the implicit field, and the color of the two-dimensional pixel corresponding to the voxel is rendered through volume rendering technology. Usually, the numerical quadrature equation is applied to the light that intersects the voxel point to calculate the pixel color. Among them, due to the cubic structure characteristics of the voxel, a bounding box is set outside the voxel point, and the axis-aligned bounding box intersection test (AABBtest) is applied to each ray. It checks whether the ray intersects the voxel by comparing the distance from the origin of the ray to each of the six boundary planes of the voxel. For the intersecting ray, the color of the pixel corresponding to the voxel point The calculation formula is:

[0093]

[0094] in represents the cumulative transparency along the ray r, and t1 and t2 represent video frames at different times.

[0095] The rendered pixel color is usually different from the real scene color. Here we introduce a color loss L, which is the cumulative rendered color. The difference between the actual scene color C is calculated as follows:

[0096]

[0097] Among them, L is the color loss, r is the rendering light, is the accumulated rendered color, and C is the real scene color.

[0098] ②Traveling light within a voxel

[0099] Volume rendering requires dense sampling in non-empty areas along rays to achieve high-quality rendering. Dense evaluation at uniformly sampled points throughout space is inefficient because empty areas are often tested unnecessarily. The present invention uses sparse voxel-based rejection sampling to create a set of query points that enables denser sampling at the same evaluation cost.

[0100] For solid surfaces, the proposed volume rendering will scatter the surface color along the rays, which means that many unnecessary accumulation steps are required on the surface to make the cumulative transparency T(t) reach 0. Therefore, in a preferred embodiment, a specific threshold is set, and when the cumulative transparency drops below the threshold, stopping the evaluation can significantly speed up the rendering process without causing a significant decrease in quality.

[0101] The present invention provides a free viewpoint video synthesis method, which improves the final coloring effect of the scene by modifying the code for calculating illumination in the shader and mixing the colors of multiple lights at the intersection of light sources.

[0102] The present application also provides a free viewpoint video synthesis device, such as Figure 6 As shown, the free viewpoint video synthesis device 400 includes:

[0103] The video acquisition module 401 is used to acquire a video to be synthesized, wherein the video is composed of RGB-D video frames, and the RGB-D video frames include voxels with depth and color;

[0104] A query point representation calculation module 402 is used for aggregating the voxel embeddings of the eight vertices of each voxel to obtain a representation of the query point inside the voxel;

[0105] A latent code network acquisition module 403 is used to use latent code embedding to represent the representation of the query point inside the voxel learned at different time frames, so as to obtain a deformable latent code network composed of latent code embeddings on different time frames;

[0106] A dense latent code acquisition module 404 is used to input the deformable latent code network into a sparse convolutional network to obtain a dense latent code formed by diffusing the sparse latent code in the deformable latent code network to a nearby 3D space;

[0107] A neural radiation field construction module 405 is used to construct a neural radiation field based on a multi-layer perceptron according to the dense latent code;

[0108] The voxel information calculation module 406 is used to input the three-dimensional position and viewpoint direction of each voxel point into the multi-layer perceptron network model of the neural radiation field to obtain the color and density of each voxel;

[0109] The neural rendering module 407 is used to perform neural rendering on the neural radiation field to obtain a video under a free viewpoint.

[0110] Preferably, the neural rendering module includes:

[0111] A color rendering unit is used to map the color c and density σ of each voxel point through an implicit field according to the voxel data, and render the color of the two-dimensional pixel point corresponding to the voxel through a volume rendering technology;

[0112] The scene rendering unit is used to use volume rendering technology to perform dense sampling in non-empty areas along the light, render the corresponding scene appearance and geometric information at different three-dimensional positions, and obtain the video under the free viewpoint.

[0113] Preferably, the representation calculation module of the query point includes:

[0114] A feature representation embedding calculation unit, used to form a feature representation embedding using a trilinear interpolation method for the eight vertices of each voxel;

[0115] The aggregation unit is used to aggregate the feature representation embeddings of the eight vertices to obtain the representation of the query point inside the voxel.

[0116] Preferably, the sparse convolutional network comprises three layers connected in sequence;

[0117] Each layer consists of a sparse convolutional layer, a batch normalization layer, and a renormalization layer connected sequentially.

[0118] Preferably, the neural radiation field based on the multi-layer perceptron includes a plurality of multi-layer perceptrons;

[0119] Each multi-layer perceptron processes a portion of the scene corresponding to the multi-layer perceptron individually.

[0120] Preferably, the voxel information calculation module includes a color calculation unit, which is used to calculate the color at the voxel point x in the frame t using the following formula:

[0121] c t (x) = M c (ψ(x,z,S t ), γ d (d), γ x (x), L t )

[0122] Among them, c t (x) is the color of voxel point x in frame t, M c represents a two-layer MLP network, ψ(x, Z, S t ) is the latent code of the voxel point x, S t represents the parameters of the deformable latent code network, Z represents the dense latent code after sparse transformation; γ d and γ x are the position encoding functions for viewing direction and spatial position, L t represents latent embedding.

[0123] Preferably, the voxel information calculation module includes a density calculation unit for calculating the density at the voxel point x in the frame t using the following formula:

[0124] σ t (x) = M σ (ψ(x,Z,S t ))

[0125] Among them, σ t (x) is the density at voxel point x in frame t, M σ represents a four-layer MLP network, ψ(x, Z, S t ) is the latent code of the voxel point x, S t represents the parameters of the deformable latent code network, and Z represents the dense latent code after sparse transformation.

[0126] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0127] The present application also provides an electronic device, including:

[0128] at least one memory and at least one processor;

[0129] The memory is used to store one or more programs;

[0130] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the free viewpoint video synthesis method as described above.

[0131] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0132] The present application also provides a computer-readable storage medium.

[0133] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the free viewpoint video synthesis method as described above are implemented.

[0134] Computer usable storage media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0135] Traditional free viewpoint synthesis technology relies on a high-quality, high-cost, dense array of video acquisition devices, which has a limited scope of application and is expensive. The present invention provides a free viewpoint video synthesis method, device, electronic device and storage medium, and proposes an RGB-D video acquisition device, which is affordable and easy for ordinary scientific researchers to obtain, and is compact, easy to carry, and highly applicable to various scenarios.

[0136] Compared with the viewpoint synthesis technology based on NeRF static scenes, the present invention provides a free viewpoint video synthesis method, device, electronic device and storage medium, which take dynamic scene video sequences as input, are not limited to a specific viewing angle range, and realize the synthesis of free viewpoint views. In addition, unnecessary light sampling is skipped on the traditional rendering method to improve rendering efficiency.

[0137] The above-mentioned embodiments only express several implementation methods of the present invention, and the description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A free viewpoint video synthesis method, characterized in that: The following steps are involved: Obtain a video to be synthesized, wherein the video consists of RGB-D video frames, and the RGB-D video frames include voxels with depth and color; For each voxel, aggregate the voxel embeddings of the eight vertices of the voxel to obtain a representation of the query point inside the voxel; For the representation of the query point inside the voxel, latent code embedding is used to represent the representation of the query point inside the voxel learned at different time frames, so as to obtain a deformable latent code network composed of latent code embeddings on different time frames; Inputting the deformable latent code network into a sparse convolutional network to obtain a dense latent code formed by diffusing the sparse latent code in the deformable latent code network into a nearby 3D space; According to the dense latent code, a neural radiation field based on a multi-layer perceptron is constructed; Inputting the three-dimensional position and viewpoint direction of each voxel point into the multi-layer perceptron network model of the neural radiation field to obtain the color and density of each voxel; The neural radiation field is subjected to neural rendering to obtain a video under a free viewpoint.

2. A free viewpoint video synthesis method according to claim 1, characterized in that: Performing neural rendering on the neural radiation field includes: For each voxel point, the color c and density σ of the point are mapped through the implicit field according to the voxel data, and the color of the two-dimensional pixel point corresponding to the voxel is rendered through volume rendering technology; Using volume rendering technology, dense sampling is performed in non-empty areas along the light rays to render the corresponding scene appearance and geometric information at different three-dimensional positions, and obtain videos under free viewpoints.

3. The free viewpoint video synthesis method according to claim 1, characterized in that: For each voxel, aggregate the voxel embeddings of the eight vertices of the voxel to obtain a representation of the query point inside the voxel, including: For the 8 vertices of each voxel, a trilinear interpolation method is used to form a feature representation embedding; The feature representation embeddings of the eight vertices are aggregated to obtain the representation of the query point inside the voxel.

4. The free viewpoint video synthesis method according to claim 1, characterized in that: The sparse convolutional network includes three layers connected in sequence; Each layer consists of a sparse convolutional layer, a batch normalization layer, and a renormalization layer connected sequentially.

5. The free viewpoint video synthesis method according to claim 1, characterized in that: The neural radiation field based on the multi-layer perceptron includes a plurality of multi-layer perceptrons; Each multi-layer perceptron processes a portion of the scene corresponding to the multi-layer perceptron individually.

6. The free viewpoint video synthesis method according to claim 1, characterized in that: The three-dimensional position and viewpoint direction of each voxel point are input into the multi-layer perceptron network model of the neural radiation field to obtain the color of each voxel, including: The color at voxel point x in frame t is calculated using the following formula: c t (x)=M c (ψ(x,z,S t ),c d (d),c x (x),L t ) Among them, c t (x) is the color of voxel point x in frame t, M c represents a two-layer MLP network, ψ(x, Z, S t ) is the latent code of the voxel point x, S t represents the parameters of the deformable latent code network, Z represents the dense latent code after sparse transformation; γ d and γ x are the position encoding functions for viewing direction and spatial position, L t Represents latent embedding.

7. The free viewpoint video synthesis method according to claim 1, characterized in that: The three-dimensional position and viewpoint direction of each voxel point are input into the multi-layer perceptron network model of the neural radiation field to obtain the density of each voxel, including: The density at voxel point x in frame t is calculated using the following formula: σ t (x)=M σ (ψ(x,Z,S t )) Among them, σ t (x) is the density at voxel point x in frame t, M σ represents a four-layer MLP network, ψ(x, Z, S t ) is the latent code of the voxel point x, S t represents the parameters of the deformable latent code network, and Z represents the dense latent code after sparse transformation.

8. A free viewpoint video synthesis device, characterized in that: include: A video acquisition module, used to acquire a video to be synthesized, wherein the video is composed of RGB-D video frames, and the RGB-D video frames include voxels with depth and color; A query point representation calculation module is used for aggregating the voxel embeddings of the eight vertices of each voxel to obtain a representation of the query point inside the voxel; A latent code network acquisition module is used to represent the representation of the query point inside the voxel using latent code embedding to represent the representation of the query point inside the voxel learned at different time frames, so as to obtain a deformable latent code network composed of latent code embeddings on different time frames; A dense latent code acquisition module is used to input the deformable latent code network into a sparse convolutional network to obtain a dense latent code formed by diffusing the sparse latent code in the deformable latent code network to a nearby 3D space; A neural radiation field construction module, used to construct a neural radiation field based on a multi-layer perceptron according to the dense latent code; A voxel information calculation module, used for inputting the three-dimensional position and viewpoint direction of each voxel point into the multi-layer perceptron network model of the neural radiation field to obtain the color and density of each voxel; A neural rendering module is used to perform neural rendering on the neural radiation field to obtain a video under a free viewpoint.

9. An electronic device, characterized in that: include: at least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a free viewpoint video synthesis method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a free viewpoint video synthesis method as described in any one of claims 1 to 7 are implemented.

Images