A novel view synthesis method and system based on real-time rendering and generalizable neural radiation field

Abstract

This invention provides a novel view synthesis method and system based on a real-time rendering generalizable neural radiation field. The method includes: constructing a generalizable neural radiation field algorithm network model based on the spatial feature similarity of image patches; wherein the generalizable neural radiation field algorithm network model introduces global information from the input image through the similarity of spatial features of image patches; training the generalizable neural radiation field algorithm network model with data collected by a data acquisition device; wherein the data includes images, depth, and motion data of real-world scenes; and performing novel view synthesis of free-viewpoint video using the trained generalizable neural radiation field algorithm network model. Applying this invention can adapt to different scenarios and tasks, accelerate the 3D scene reconstruction process, improve rendering quality, provide more refined visual effects, and reduce unnecessary computational resource consumption.

Description

Technical Field

[0001] This invention relates to the field of novel view synthesis technology, and more specifically, to a novel view synthesis method and system based on real-time rendering of a generalizable neural radiation field. Background Technology

[0002] With the continuous development of digital multimedia technology, people have increasingly higher demands for video quality and content, and are paying more and more attention to the interactivity and visual sensory experience of videos. The explosive growth of the online entertainment industry has also prompted the industry itself to examine the value of breakthroughs in interactive experience. Based on this, people are constantly exploring higher-quality visual experiences: in terms of image quality, visual media has evolved from standard definition and high definition to 4K ultra-high definition and 8K ultra-high definition; in terms of interactivity, it has evolved from 2D to 3D and 4D, with free-viewpoint video being one of the directions for the development of multimedia interactivity.

[0003] Free-viewpoint video combines dynamic and static perspectives. From the perspectives of visual viewing angle and interactive spatial dimension, the free movement of the panoramic viewpoint gives viewers greater autonomy and a better immersive experience. It allows viewers to flexibly choose their viewing angle and position without being limited by the content creator's control. It has wide applications in many fields such as public safety, healthcare, autonomous driving, cultural entertainment, and e-commerce, making it an important research area in virtual reality. Among these applications, how to efficiently model and render high-quality free-viewpoint video has become a research focus for many scholars.

[0004] There are two main methods for creating free-viewpoint videos: image-based new view synthesis and model-based 3D reconstruction. Among these, image-based new view synthesis has become a research hotspot due to its photorealistic quality and enhanced immersion. New view synthesis, a task combining computer vision and computer graphics, refers to generating a target image with arbitrary target camera poses from a given source image and its camera pose. However, new view synthesis is an underdetermined problem; without considering prior knowledge or constraints, multiple solutions exist. A good solution requires a complete 3D understanding of all visible objects in the unseen view and must handle complex situations such as occlusion and textureless surfaces. Therefore, traditional free-viewpoint video generation techniques typically require significant computational resources and time, as well as complex processing steps, to produce high-quality free-viewpoint videos. These techniques suffer from slow processing speeds, high costs, and difficulties in achieving real-time interactivity.

[0005] With the development of deep learning in image understanding tasks, deep learning-based methods can effectively construct information about three-dimensional scenes from two-dimensional images. Many new view synthesis methods based on deep learning and combined with traditional methods have emerged, such as NeRF (Neural Radiance Fields).

[0006] NeRF is the first algorithm to achieve photorealistic new view synthesis using implicit representations. It implicitly learns a static 3D scene by fitting a continuous function to an MLP (Multilayer Perceptron). The function is input with the coordinates of 3D points and the viewing direction to obtain the corresponding color and volume density. Differentiable neural bodies are then used for rendering to obtain pixel colors. NeRF uses only the input image as supervision information to fit accurate implicit functions to high-resolution geometry, enabling photorealistic new view synthesis for complex scenes. The overall algorithm flow is as follows: Figure 1 As shown, a large number of views from different perspectives are input to construct the neural radiation field of the 3D model, and finally the view from the specified perspective is rendered.

[0007] NeRF uses volumetric rendering to incorporate the rendering process into the neural network, allowing the network to be trained directly from the errors in the rendered images. Figure 2 The differentiable rendering process of NeRF is shown, such as Figure 2 As shown, NeRF fits a five-dimensional vector function, which describes the geometric and color information of a 3D model and is implemented using a multilayer perceptron. The input to this five-dimensional function consists of the three-dimensional coordinate vector x = (x, y, z) of a point in space and the two-dimensional view direction vector d = (θ, φ). The function output is the volume density б of that point in space and the color c = (r, g, b) of that point along the d direction. In the real world, the color of an object is related to lighting conditions; different colors will be observed when viewing the same position of the same object from different viewpoints. Therefore, in the specific calculation, б is only related to the coordinate vector x, while c is determined by both x and d. For each pixel in the image, a corresponding ray is modeled. This ray originates from the camera's optical center o and is denoted as r = o + td. After obtaining б and c for all spatial points along this ray, the corresponding pixels are rendered using volume rendering.

[0008] However, the function representation learned by NeRF is only applicable to a single scene, and has problems such as poor generalization, only applicable to static scenes, requiring a large number of multi-view images as input, and slow training and inference speed, making it difficult to generate free viewpoint videos in real time for any scene.

[0009] Therefore, how to build a network that can generalize to any unseen static scene and achieve real-time rendering speed for new views has become one of the research directions in the field of new view synthesis technology for free viewpoint video. Summary of the Invention

[0010] In view of the problems existing in the current traditional free viewpoint video generation technology, such as slow processing speed, high cost and difficulty in achieving real-time interaction, the purpose of this invention is to provide a new view synthesis method and system based on real-time rendering of generalizable neural radiation fields, so as to build a network that supports generalization to any unseen static scene and achieve real-time rendering speed of new views.

[0011] On one hand, the present invention provides a novel view synthesis method based on real-time rendering of a generalizable neural radiation field, comprising:

[0012] S100: Construct a generalizable neural radiation field algorithm network model based on the spatial feature similarity of image patches; wherein, the generalizable neural radiation field algorithm network model introduces global information of the input image through the similarity of the spatial features of image patches;

[0013] S200: The generalizable neural radiation field algorithm network model is trained using data collected by a data acquisition device; wherein, the data collected by the data acquisition device includes image data and motion data of real-world scenes;

[0014] S300: Synthesize new views of free-viewpoint videos using a trained generalizable neural radiation field algorithm network model.

[0015] One optional approach is that the generalizable neural radiation field algorithm network model includes:

[0016] The 2D feature extraction module is used to extract multi-scale two-dimensional features from the input image;

[0017] The 3D feature extraction module is used to construct a cost volume through the multi-scale two-dimensional features, and to extract the depth probability volume and three-dimensional feature volume of the cost volume through UNet-3D.

[0018] The sampling guidance module is used to filter sampling points in the space based on the depth probability volume, so as to retain sampling points that meet the preset depth probability requirements;

[0019] The neural radiation field module is used to calculate the low-resolution target image features of the target image by using a preset MLP network and preserving the multi-scale two-dimensional and three-dimensional feature volumes of the sampling points.

[0020] The upsampling module is used to generate high-resolution target image features of the input image by upsampling based on the low-resolution target image features, and to render a high-resolution image based on the high-resolution target image features.

[0021] One optional approach is that the method for the 2D feature extraction module to extract multi-scale two-dimensional features includes: extracting features from an input image of size 3×H×W. In the middle, downsampling yielded a size of 32× 2D low-resolution image features F i,1 Then, bilinear interpolation is used to interpolate the 2D low-resolution feature F. i,1 Upsample the feature map to obtain a size of 2D features F i,2 Where H represents the height of the input image, W represents the width of the input image, N represents the number of input images, and i represents the i-th image among the N images.

[0022] Alternatively, the 2D feature extraction module may further include a 3D mapping unit, used to map 3D points in space to 2D features F of the input image. i,2 Above, the pixel-aligned feature {f} is obtained. i |i=1,...,N}; and, the pixel-aligned features on different input views are aggregated and then pooled to obtain the image features f of the three-dimensional points. img =ψ(f1,...,f N ).

[0023] One optional approach is that the 3D feature extraction module constructs a cost volume using the multi-scale two-dimensional features, including:

[0024] Using the 2D feature F i,2 The cost volume is constructed based on the camera frustum of the target view through differentiable homography changes; wherein, N input images are projected onto several sampling planes {L j |j=1,...,D}, where D is the number of sampling planes, and j represents the j-th sampling plane among the D sampling planes, by using the 2D feature F i,2 The cost volume is constructed by mapping it onto D sampling planes; where,

[0025] Given the camera intrinsic parameter matrix K of input view Ii i Rotation matrix R i Translation matrix T i The camera intrinsic parameter matrix K of the target view It t Rotation matrix R t Translation matrix T t The homography change is defined as follows:

[0026]

[0027] Among them, a T The matrix representing the transpose of the principal axis 'a' of the target view camera, I is the identity matrix, and H is the transpose of the target view camera. i (z) The feature value obtained by projecting the pixels (u,v) in the target view onto the target view on the depth axis and performing homography is defined as:

[0028]

[0029] Based on the mapping features after homography transformation, multi-view features of each voxel are calculated. The variance yields the cost volume, where [u,v,1] T Let [u,v,1] be the transpose of the matrix.

[0030] One optional approach is that, after extracting the three-dimensional feature volume of the cost body, the 3D feature extraction module further includes: performing trilinear interpolation on the three-dimensional feature volume to obtain voxel alignment features f with spatial geometric information. voxel .

[0031] One optional approach is that the neural radiation field module calculates the low-resolution target image features in the multi-scale two-dimensional features and the three-dimensional feature volume through a preset MLP network, including:

[0032] The 2D and 3D features of the three-dimensional points are input into a preset MLP network to obtain the point features and volume density of the three-dimensional points, defined as:

[0033] f p ,σ=φ(f img ,f voxel )

[0034] Where φ is the MLP network, f p Let f be the point feature of the three-dimensional point, σ be the volume density of the three-dimensional point, and f be the point feature of the three-dimensional point. p The image features f of the three-dimensional points img And the mixing weight w for predicting the color of the input view image based on the relative viewing direction of 3D points in the target view under the input view. i Then, by the mixed weight w i The color features observed when viewing a 3D point from a certain direction in the target view are defined as:

[0035]

[0036] Among them, f i The color features are the two-dimensional image features of the input view. The final cumulative color feature of each ray is obtained by 2D aggregation of the volume density σ.

[0037] One option is to generate high-resolution target image features of the input image based on the low-resolution target image features by upsampling, including: generating high-resolution target image features of the input image by performing sub-pixel convolution image feature upsampling operation on the low-resolution target image features.

[0038] Alternatively, the sampling guidance module may further include:

[0039] The first-level filtering unit is used to obtain the probability and standard deviation of a pixel (u,v) in a target image by linear interpolation of the depth probability volume, thereby obtaining the depth range existing on the surface of the depth probability volume, so as to skip blank areas in the space and narrow the sampling range according to the depth range.

[0040] The secondary screening unit is used to guide precise sampling within a narrowed sampling range using the cumulative density function of the depth probability volume, so as to screen out the retained sampling points.

[0041] This invention also provides a novel view synthesis system based on real-time rendering of generalizable neural radiation fields, which performs novel view synthesis of free-viewpoint videos based on the aforementioned novel view synthesis method based on real-time rendering of generalizable neural radiation fields, including:

[0042] A network model building unit is used to construct a generalizable neural radiation field algorithm network model based on the similarity of spatial features of image patches; wherein, the generalizable neural radiation field algorithm network model introduces global information of the input image through the similarity of spatial features of image patches;

[0043] A network model training unit is used to train the generalizable neural radiation field algorithm network model using data collected by a data acquisition device; wherein, the data collected by the data acquisition device includes image data and motion data of real-world scenes;

[0044] The new view synthesis unit is used to synthesize new views of free-viewpoint videos using a trained generalizable neural radiation field algorithm network model.

[0045] The present invention also provides an electronic device, the electronic device comprising:

[0046] At least one processor; and,

[0047] A memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the steps in the novel view synthesis method based on real-time rendering generalizable neural radiation fields as described above.

[0049] As can be seen from the above technical solution, the novel view synthesis method and system based on real-time rendering generalizable neural radiation field provided by this invention improves the generation method of free viewpoint video from three aspects: data acquisition, network model, and engine rendering by adopting a real-time rendering generalizable neural radiation field based on image blocks. It uses a novel view synthesis technology to create free viewpoint video, splits dynamic scenes into frames, realizes a generalizable and real-time rendering neural radiation field algorithm, constructs a network that supports generalization to any unseen static scene, and achieves real-time rendering speed of the new view. Thus, it generates free viewpoint video in a more efficient and real-time manner, eliminates the complex calculation and time delay problems in traditional methods, and provides a better user experience for virtual reality, game development, live streaming and other fields. Attached Figure Description

[0050] Other objects and results of the invention will become more apparent and readily understood by referring to the following description taken in conjunction with the accompanying drawings, and with a more complete understanding of the invention. In the drawings:

[0051] Figure 1 This is a schematic diagram of the algorithm flow for neural radiation fields;

[0052] Figure 2 This is a schematic diagram of the differentiable rendering process for a neural radiation field.

[0053] Figure 3 This is a flowchart illustrating a novel view synthesis method based on a real-time rendered generalizable neural radiation field according to an embodiment of the present invention.

[0054] Figure 4 This is a schematic diagram of the framework structure of a generalizable neural radiation field algorithm network model based on image patch spatial feature similarity according to an embodiment of the present invention.

[0055] Figure 5 This is a schematic diagram of the data training process of a generalizable neural radiation field algorithm network model based on image patch spatial feature similarity according to an embodiment of the present invention.

[0056] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0057] In the following description, numerous specific details are set forth for illustrative purposes and to provide a thorough understanding of one or more embodiments. However, it will be apparent that these embodiments may also be implemented without these specific details. In other instances, well-known structures and devices are shown in block diagram form for ease of description of one or more embodiments.

[0058] In contrast to existing methods, this invention provides a novel view synthesis method and system based on real-time rendering of generalizable neural radiation fields. To better illustrate the technical solution of this invention, some of the technical terms involved in this invention will be briefly explained below.

[0059] Neural Radiance Fields (NeRF) is a computer vision technique used to generate high-quality 3D reconstructed models. It utilizes deep learning to extract the geometric shape and texture information of objects from images viewed from multiple perspectives, and then uses this information to generate a continuous 3D radiance field, thus presenting highly realistic 3D models at any angle and distance. NeRF technology has broad application prospects in computer graphics, virtual reality, augmented reality, and other fields.

[0060] Multi-Layer Perceptron (MLP): A multi-layer perceptron (MLP) is a basic neural network architecture used for deep learning tasks. An MLP consists of multiple layers, each composed of neurons (or nodes). These neurons are typically connected in a feedforward manner, where each neuron receives the output of the neuron in the previous layer and passes it to the next layer.

[0061] Volume density: In NeRF, volume density refers to the density value of each point in three-dimensional space. This density value is used to characterize whether there is an object or surface in the scene at a given point. It can be seen as a measure of the probability that an actual object exists at a certain point.

[0062] The Depth Probability Volume (DPV) is a distribution of the depth probability of pixels based on the cost volume. The cost volume is a tensor of size B×C×D×H×W used to store the matching degree of pixels in each of the multiple views of an image. The DPV also has dimensions of B×C×D×H×W and stores the depth probability information of each pixel.

[0063] The present invention provides a novel view synthesis method and system based on real-time rendering of generalizable neural radiation fields. This method improves the synthesis of novel views from the following three perspectives by employing real-time rendering of generalizable neural radiation fields based on image patches:

[0064] 1. Data acquisition equipment: This equipment uses cameras and other devices to efficiently capture images of real-world scenes, including image and motion data.

[0065] 2. Network Model: Designing and training highly optimized neural network models that can transform collected data into three-dimensional neural radiation fields without requiring a large amount of computing resources;

[0066] 3. Rendering Engine: Develop a real-time rendering engine consisting of an upsampling module and a differentiable surface rendering method. Learn the implicit representation of the scene through deep learning, and then render the scene through the real-time rendering engine. This will enable the generation of neural radiation fields to present videos with high-quality details and free viewpoint effects in real time.

[0067] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] It should be noted that the following description of exemplary embodiments is merely illustrative and is not intended to limit the invention or its application or use in any way. Technologies and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such technologies and equipment should be considered part of the specification.

[0069] To illustrate the novel view synthesis method and system based on real-time rendering generalizable neural radiation fields provided by this invention... Figure 3 , Figure 4 The framework structure of the novel view synthesis method based on real-time rendering generalizable neural radiation field and the generalizable neural radiation field algorithm network model based on image patch spatial feature similarity in embodiments of the present invention is illustrated by way of example.

[0070] like Figure 3 and Figure 4 As shown in the figure, the novel view synthesis method based on real-time rendering generalizable neural radiation field provided by the present invention mainly includes the following steps:

[0071] S100: Construct a generalizable neural radiation field algorithm network model based on the spatial feature similarity of image patches; wherein, the generalizable neural radiation field algorithm network model introduces global information of the input image through the similarity of the spatial features of image patches.

[0072] Global information of an image refers to the overall attributes of the image, such as color features, texture features, and shape features. Global information description is not applicable to situations with aliasing or occlusion. Local information, on the other hand, is feature information extracted from local regions of the image. The correlation between features is relatively small, and in the case of occlusion, the disappearance of some features will not affect the detection and matching of other features.

[0073] S200: The generalizable neural radiation field algorithm network model is trained using data collected by a data acquisition device; wherein, the data collected by the data acquisition device includes image data and motion data of real-world scenes.

[0074] Specifically, as an example, motion data of real-world scenes can be obtained through COLMAP. By inputting the captured image data of the real-world scene into COLMAP, the corresponding camera intrinsic parameters, extrinsic parameters, depth, and motion data can be obtained.

[0075] S300: Synthesize new views of free-viewpoint videos using a trained generalizable neural radiation field algorithm network model.

[0076] In order to build a network that can generalize to any unseen static scene and achieve real-time rendering speed for new views, this invention first needs to build a generalizable neural radiation field algorithm network model based on the similarity of image patch spatial features.

[0077] NeRF synthesizes new views of a scene by encoding volume density and color, but it essentially overfits the geometric and color information of the scene, relying on the network's memory to present the scene, thus lacking the ability to generalize to new scenes. To address the generalization problem, many methods introduce image features and use convolution to extract features from the input image, thereby generating richer and more general scene representations.

[0078] Currently, multi-view reconstruction is one approach to address the generalization problem. Multiple View Stereo (MVS) is a generalization of stereo vision, enabling the observation and acquisition of 3D scene images from multiple perspectives, and performing matching and depth estimation. Most deep learning-based MVS methods rely on planar scanning algorithms to generate cost volumes. The core of these algorithms is verifying depth hypotheses. After projecting pixels onto different planes in space, if a projection point is captured by different cameras with similar light intensity, then the depth value of that point is valid. This method divides the depth interval into discrete values, selecting the most valid depth from all hypotheses to estimate the final depth. This approach allows for both estimation of depth information in a scene and generalization. Given a 3D point in space, it is projected onto various input views. By determining the consistency of local features across views, the volume density is determined, learning the mapping relationship between spatial points and input views, thus enabling generalization to any new scene.

[0079] During the camera imaging process, the acquired image data is discretized. Due to the limitations of the photosensitive element itself, each pixel on the imaging surface only represents the nearby color. Macroscopically, pixels are adjacent, but microscopically, there is still a certain distance between two pixels. Pixels that exist between two actual physical pixels are called subpixels.

[0080] Subpixel algorithms are a method used to improve image resolution. However, subpixel convolutional layers do not contain non-linear operations; they directly utilize data from the feature maps of low-resolution images to generate high-resolution images. Assuming a magnification factor of r, if traditional interpolation methods are used to achieve the size enlargement operation, the convolutional layer's computation occurs in the high-resolution space, increasing r by a factor of r. 2 The computational cost is several times higher. If a deconvolutional network is used, each input pixel is multiplied and added with the elements of the deconvolution kernel, and the result is obtained by superposition, making the calculation more complex. Since the deconvolution kernel has a span of r during the sliding process, some pixels in the result are accumulated more times than other positions, which can easily produce checkerboard noise and affect image edges and details. Therefore, the sub-pixel convolutional layer used in this invention can ensure both speed and better image super-resolution generation effect.

[0081] There are two approaches to applying neural radiation fields to dynamic scenes: one is dynamic NeRF, which requires additional temporal information in addition to viewpoint and image-related information; the other is cross-scene generalized NeRF, which involves splitting the dynamic scene into frames and synthesizing each frame as a separate scene. This invention uses the second method.

[0082] This invention presents a neural radiation field algorithm based on the spatial feature similarity of image patches. It processes images in a patch-by-patch manner, incorporating global information through spatial feature similarity. An image patch is analogous to a patch in deep learning; when the image resolution is too large and resources are limited (e.g., video memory, computing power), the image can be divided into smaller patches. This invention uses an image patch approach, distinct from the ray-by-ray processing of the original NeRF. Assuming an image patch size of h×w, processing it in this way simultaneously involves h×w rays, and also introduces the correlation between rays.

[0083] Figure 5 This is a schematic diagram of the data training process for a generalizable neural radiation field algorithm network model based on image patch spatial feature similarity according to an embodiment of the present invention.

[0084] like Figure 4 and Figure 5As shown in the embodiments of the present invention, the generalizable neural radiation field algorithm network model 400 based on the similarity of image patch spatial features mainly includes five parts: a 2D feature extraction module 410, a 3D feature extraction module 420, a neural radiation field module 430, an upsampling module 440, and a sampling point guidance module 450.

[0085] The system includes the following modules: a 2D feature extraction module, which extracts multi-scale two-dimensional features from several input images using a 2D convolutional neural network (2D CNN); a 3D feature extraction module, which constructs a cost volume from the multi-scale two-dimensional features using a 3D convolutional neural network (3D CNN), and extracts the depth probability volume and three-dimensional feature volume of the cost volume using UNet-3D to provide geometric perception information for the construction of the neural radiation field; a sampling guidance module, which filters sampling points in space based on the depth probability volume to retain sampling points that meet preset depth probability requirements; a neural radiation field module, which feeds the interpolated two-dimensional and three-dimensional features into a preset MLP to calculate the low-resolution target image features of the retained sampling points; and an upsampling module, which upsamples the low-resolution target image features obtained by the aforementioned neural radiation field module to generate high-resolution target image features of the input image, and renders a high-resolution image based on the high-resolution target image features.

[0086] Specifically, as an example, methods for extracting multi-scale two-dimensional features using a 2D feature extraction module include:

[0087] For an input image of size 3×H×W Inputting it into the 2D feature extraction module yields a result of size [size missing]. 2D low-resolution features F i,1 Then, bilinear interpolation is used to upsample the feature map, resulting in a size of... 2D features F i,2 Where H represents the height of the input image, W represents the width of the input image, N represents the number of input images, and i represents the i-th image out of N images. Multi-scale features are used to construct the cost volume and neural radiation field, where F i,1 Used to construct the cost body, F i,2 Used to reconstruct the neural radiation field.

[0088] The purpose of the aforementioned four-fold downsampling is to reduce the image size, thereby reducing video memory and computational load. The purpose of using bilinear interpolation for two-fold upsampling is to obtain feature information at different scales. Although the low-resolution feature map obtained by the feature extraction module is downsampled four times in each dimension compared to the original image, the neighborhood information of the retained pixels is encoded and stored in the 32-channel feature channel, containing rich semantic information.

[0089] To build a network that can generalize to any unseen static scene, the 2D feature extraction module also includes a 3D mapping unit, which maps any 3D point in space to the 2D features F of the input image. i,2 Above, or mapped to the 2D low-resolution feature F obtained by the above downsampling. i,1 Above, the pixel-aligned feature {f} is obtained. i |i=1,...,N}, aggregate these pixel features from different input views and then perform pooling to obtain the image features f of the 3D point. img =ψ(f1,...,f N The image features f of the three-dimensional point. img The neural radiation field module will be used to calculate the features of low-resolution target images.

[0090] As an example, the specific method for the 3D feature extraction module to construct the cost volume using the multi-scale two-dimensional features includes: utilizing the 2D features F obtained from the upsampling described above. i,2 The cost volume is constructed based on the camera frustum of the target view through differentiable homography changes. N input images are projected onto several sampling planes {L j |j=1,...,D}, where D is the number of sampling planes, and j represents the j-th sampling plane among the D sampling planes, by using the 2D feature F i,2 The cost volume is constructed by mapping it onto D sampling planes;

[0091] Wherein, the camera intrinsic parameter matrix K is given by the input view Ii. i Rotation matrix R i Translation matrix T i The camera intrinsic parameter matrix K of the target view It t Rotation matrix R t Translation matrix T t Homography can be defined as:

[0092]

[0093] Where a T The matrix representing the transpose of the principal axis 'a' of the target view camera, I is the identity matrix, and H is the transpose of the target view camera. i(z) The feature value obtained by projecting the pixels (u,v) in the target view onto the target view on the depth axis and performing homography is defined as:

[0094]

[0095] Based on the mapping features after homography transformation, multi-view features of each voxel are calculated. The variance yields the cost volume, where [u,v,1] T Let [u, v, 1] be the transpose of the matrix. Then, the depth probability volume and 3D feature volume of the cost volume are extracted using UNet-3D.

[0096] Extracting the depth probability volume and 3D feature volume using UNet-3D can be broadly divided into two parts: the first is the backbone feature extraction part, which involves stacking 3D convolutional layers and activation functions to obtain several effective features; the second is the enhancement feature extraction part, which upsamples the effective features obtained in the previous step and performs feature fusion. Specifically, as an example, UNet-3D includes encoder downsampling, decoder upsampling, and skip connections. Skip connections pass layers of the same resolution from the encoding path to the decoding path, providing high-resolution features for the decoding layer. Finally, the depth probability volume and 3D feature volume are obtained through two 3D convolutional layers.

[0097] In addition, after extracting the three-dimensional feature volume of the cost body, the 3D feature extraction module also performs trilinear interpolation on the three-dimensional feature volume to obtain voxel alignment features f with spatial geometric information. voxel The voxel alignment feature f voxel The neural radiation field module will be used to compute features from low-resolution target images. Using voxel alignment features as input to the MLP network provides the network with spatial geometric information.

[0098] After obtaining the 2D and 3D features of a 3D point, they are input into an MLP network. The neural radiation field module then obtains the point features and volume density of the 3D point, defined as follows:

[0099] f p ,σ=φ(f img ,f voxel )

[0100] Where φ is the MLP network, f p Let f be the point feature of the three-dimensional point, σ be the volume density of the three-dimensional point, and f be the point feature of the three-dimensional point. p The pixel features f of the three-dimensional point img And the mixing weight w for predicting the color of the input view image based on the relative viewing direction of 3D points in the target view under the input view. i Then by wi The color features observed when viewing a 3D point from a certain direction in the target view are defined as:

[0101]

[0102] Among them, f i Given the 2D image features of the input view, the color features and volume density predicted by the network are aggregated in 2D to obtain the final cumulative color features of each ray. Specifically, as an example, the color features and volume density of the sampling points on a ray are weighted and summed to obtain the color features of the corresponding pixel.

[0103] Among them, the mixed weight w i The formula for obtaining it is Where, Δd i This represents the difference between the viewing angle of the target view and the viewing angle of the input view.

[0104] This invention improves the final image quality through subpixel convolution image feature upsampling. A low-resolution target feature map is input into this upsampling module, which processes it to obtain a high-resolution RGB image as the high-resolution target image feature. This module is located at the end of the model, allowing the feature map to retain more texture regions in the low-resolution space while maintaining speed. Furthermore, compared to interpolating a low-resolution RGB image into a high-resolution RGB image, upsampling the feature map can better learn spatial correlations, resulting in a higher-quality image. Subpixel convolution is essentially an upsampling method for pixel rearrangement. Its principle is explained with an example: [H,W,C] is a low-resolution image; convolution yields an image of size [H,W,C*r]. 2 The feature map is obtained by taking r as the upsampling factor and then performing a shuffle transformation on the feature map to obtain a super-resolution image of size [H*r, W*r, C], thereby achieving image upsampling.

[0105] Subpixel convolution does not perform convolution on integer pixels, resulting in lower network complexity. Using subpixel convolution for upsampling can significantly improve the overall training and inference speed of the network. The process of obtaining a high-resolution RGB image from the upsampling module is shown in the figure.

[0106] The sampling guidance module primarily utilizes the Depth Probability Volume (DPV) obtained from the 3D feature extraction module. The DPV is associated with 3D points in the scene and stores depth estimates as well as probability information about surface presence. Using the surface presence probability information in the DPV for sampling guidance reduces the number of sampling points, thus lowering computational load and saving time. It also ensures more accurate sampling locations. This step is similar to the coarse sampling step in NeRF, but unlike NeRF, it doesn't require multiple MLP layers to obtain the volume density for precise sampling. Since the Depth Probability Volume contains the depth probability information of the sampling points, after estimating the surfaces in the 3D scene based on this depth probability information, the cumulative density function is calculated. The calculation result determines which sampling points will be retained for further refinement and which will be discarded. Sampling points that meet preset depth probability requirements can be retained; for example, several sampling points with high depth probability values ​​will be retained, processed, and input into the network for further calculation. Sampling points with low depth probability values ​​can be discarded, thus reducing unnecessary sampling. By using the retained sampling points with high depth probability values, more detail and accurate rendering results can be obtained. This sampling guidance strategy can focus on areas with uncertainty without consuming a lot of computing resources, thus enabling more efficient scene reconstruction.

[0107] The sampling guidance module 450 extracts the depth probability volume of the cost volume using UNet-3D, and obtains the probability density function of the sampling points from the depth probability volume to retain a number of sampling points with larger depth probability values. Specifically, as an example, the sampling guidance module 450 further includes:

[0108] The first-level filtering unit is used to obtain the probability and standard deviation of a pixel (u,v) in a target image by linear interpolation of the depth probability volume, thereby obtaining the depth range existing on the surface of the depth probability volume, so as to skip blank areas in the space and narrow the sampling range according to the depth range.

[0109] The secondary screening unit is used to guide precise sampling within a narrowed sampling range using the cumulative density function of the depth probability volume, so as to screen out the retained sampling points.

[0110] The original NeRF sampled 128 points on the light source for each pixel; however, most of these sampling points were located in empty areas, resulting in a waste of computational resources. This invention utilizes a depth probabilistic volume to estimate the probability of a surface existing in 3D space. Surface estimation reduces the number of sampling points, improving computational speed. Furthermore, because the surface is estimated, the placement of sampling points is relatively accurate, thus improving the final scene reconstruction image.

[0111] As can be seen from the above embodiments, the novel view synthesis method based on real-time rendering generalizable neural radiation fields provided by this invention is a NeRF-type method capable of efficient computation, used for 3D scene reconstruction and free-viewpoint video generation. It utilizes a depth probability volume to estimate the probability of surface existence in 3D space and uses the estimation results of the depth probability volume to intelligently guide the sampling process, more accurately finding suitable sampling locations and reducing unnecessary waste of computational resources. Furthermore, this invention uses the depth probability volume to estimate surface probabilities for sampling guidance, replacing the conventional coarse sampling step and directly performing fine-grained sampling, thereby improving computational efficiency. In addition, this invention introduces a sampling guidance strategy for sampling points to further improve efficiency. Specifically, spatial correlation upsampling is introduced in the upsampling module to upsampling low-resolution feature maps, better learning spatial correlations, thereby obtaining higher-quality images and simultaneously improving efficiency.

[0112] On the other hand, the present invention also provides a novel view synthesis system based on real-time rendering generalizable neural radiation fields, which performs novel view synthesis of free-viewpoint videos based on the novel view synthesis method based on real-time rendering generalizable neural radiation fields as described above, including:

[0113] A network model building unit is used to construct a generalizable neural radiation field algorithm network model based on the similarity of spatial features of image patches; wherein, the generalizable neural radiation field algorithm network model introduces global information of the input image through the similarity of spatial features of image patches;

[0114] A network model training unit is used to train the generalizable neural radiation field algorithm network model using data collected by a data acquisition device; wherein, the data collected by the data acquisition device includes image data and motion data of real-world scenes;

[0115] The new view synthesis unit is used to synthesize new views of free-viewpoint videos using a trained generalizable neural radiation field algorithm network model.

[0116] The generalizable neural radiation field algorithm network model based on image patch spatial feature similarity, constructed by the aforementioned network model building unit, mainly protects the following five parts:

[0117] The 2D feature extraction module is used to extract multi-scale two-dimensional features from the input image;

[0118] The 3D feature extraction module is used to construct a cost volume through the multi-scale two-dimensional features, and to extract the depth probability volume and three-dimensional feature volume of the cost volume through UNet-3D.

[0119] The sampling guidance module is used to filter sampling points in the space based on the depth probability volume, so as to retain sampling points that meet the preset depth probability requirements;

[0120] The neural radiation field module is used to calculate the multi-scale two-dimensional features and the low-resolution target image features that retain sampling points in the three-dimensional feature volume through a preset MLP network.

[0121] The upsampling module is used to generate high-resolution target image features of the input image by upsampling based on the low-resolution target image features, and to render a high-resolution image based on the high-resolution target image features.

[0122] The aforementioned novel view synthesis system based on real-time rendering generalizable neural radiation fields is an implementation system corresponding to the aforementioned novel view synthesis method based on real-time rendering generalizable neural radiation fields. Its specific execution steps can be referred to the specific embodiments of the aforementioned novel view synthesis method based on real-time rendering generalizable neural radiation fields, which will not be described in detail here.

[0123] As can be seen from the above embodiments, the novel view synthesis method and system based on real-time rendering generalizable neural radiation fields provided by the present invention have the following advantages compared with existing novel view synthesis schemes:

[0124] 1. High computational efficiency: One of the core objectives of this invention is to improve the computational efficiency of the NeRF-based novel view synthesis scheme. Through intelligent sampling guidance strategies, unnecessary computational overhead can be reduced, thereby making the computation more efficient.

[0125] 2. More accurate 3D scene reconstruction: Using depth probabilistic volume surface probability estimation helps to find the location of the surface more accurately; upsampling in the feature dimension can learn spatial correlation, obtain higher quality images, and improve the accuracy of 3D scene reconstruction.

[0126] 3. Reduced resource consumption: By avoiding uniform coarse sampling of the entire scene, the waste of computing resources is reduced, which helps to improve performance in computationally intensive tasks.

[0127] 4. Wide range of applications: This invention has potential applications in multiple fields such as virtual reality, game development, augmented reality, and free-viewpoint video generation, and can improve the quality of products and services in these fields.

[0128] By applying the novel view synthesis method and system based on real-time rendering generalizable neural radiation fields provided by this invention, the following technical effects can be achieved:

[0129] 1. Efficient 3D Reconstruction: This invention can accelerate the 3D scene reconstruction process, making it more suitable for real-time or interactive applications, such as virtual reality experiences, games, and live streaming.

[0130] 2. Improved rendering quality: Through a more intelligent sampling guidance strategy, this invention is expected to improve rendering quality and provide more refined visual effects.

[0131] 3. Resource saving: Reducing unnecessary consumption of computing resources helps to lower hardware requirements and improve performance.

[0132] 4. Scalability: This invention can be adapted to different scenarios and tasks, and therefore has good scalability and adaptability in various applications.

[0133] As another aspect of the present invention, such as Figure 6 As shown, the present invention also provides an electronic device, the electronic device comprising:

[0134] At least one processor; and,

[0135] A memory that is communicatively connected to at least one processor; wherein,

[0136] The memory stores a computer program that can be executed by at least one processor to enable the at least one processor to perform the steps in the aforementioned novel view synthesis method and system method based on real-time rendering generalizable neural radiation fields.

[0137] It will be understood by those skilled in the art that Figure 6 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0138] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0139] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0140] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0141] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0142] The memory 11 in the electronic device 1 stores a novel view synthesis program 12 based on real-time rendering of a generalizable neural radiation field. This program is a combination of multiple instructions that, when run in the processor 10, can achieve the following:

[0143] S100: Construct a generalizable neural radiation field algorithm network model based on the spatial feature similarity of image patches; wherein, the generalizable neural radiation field algorithm network model introduces global information of the input image through the similarity of the spatial features of image patches;

[0144] S200: The generalizable neural radiation field algorithm network model is trained using data collected by a data acquisition device; wherein, the data collected by the data acquisition device includes image data and motion data of real-world scenes;

[0145] S300: Synthesize new views of free-viewpoint videos using a trained generalizable neural radiation field algorithm network model.

[0146] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figure 3 and Figure 4 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0147] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0148] The novel view synthesis method and system based on real-time rendering generalizable neural radiation fields proposed according to the present invention have been described above by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that various modifications can be made to the novel view synthesis method and system based on real-time rendering generalizable neural radiation fields proposed in the present invention without departing from the scope of the present invention. Therefore, the scope of protection of the present invention should be determined by the content of the appended claims.

Claims

1. A novel view synthesis method based on real-time rendering of generalizable neural radiation fields, characterized in that, include: S100: Construct a generalizable neural radiation field algorithm network model based on the spatial feature similarity of image patches; wherein, the generalizable neural radiation field algorithm network model introduces global information of the input image through the similarity of the spatial features of image patches; S200: The generalizable neural radiation field algorithm network model is trained using data collected by a data acquisition device; wherein, the data collected by the data acquisition device includes image data and motion data of real-world scenes; S300: New view synthesis of free-viewpoint video using a trained generalizable neural radiation field algorithm network model; The generalizable neural radiation field algorithm network model includes: The 2D feature extraction module is used to extract multi-scale two-dimensional features from the input image; The 3D feature extraction module is used to construct a cost volume through the multi-scale two-dimensional features, and to extract the depth probability volume and three-dimensional feature volume of the cost volume through UNet-3D. The sampling guidance module is used to filter sampling points in the space based on the depth probability volume, so as to retain sampling points that meet the preset depth probability requirements; The neural radiation field module is used to calculate the low-resolution target image features of the target image by using a preset MLP network and preserving the multi-scale two-dimensional and three-dimensional feature volumes of the sampling points. The upsampling module is used to generate high-resolution target image features of the input image by upsampling based on the low-resolution target image features, and to render a high-resolution image based on the high-resolution target image features; The sampling guidance module further includes: The first-level filtering unit is used to filter pixels in the target image. By linear interpolating the depth probability volume, the probability and standard deviation of the pixel point on a certain depth plane are obtained, thereby obtaining the depth range existing on the surface of the depth probability volume, so as to skip blank areas in space and narrow the sampling range according to the depth range. The secondary screening unit is used to guide precise sampling within a narrowed sampling range using the cumulative density function of the depth probability volume, so as to screen out the retained sampling points.

2. The novel view synthesis method based on real-time rendering generalizable neural radiation field as described in claim 1, characterized in that, The method for extracting multi-scale two-dimensional features by the 2D feature extraction module includes: From size Input image In the middle, downsampling extracts a size of 2D low-resolution image features Then, bilinear interpolation is used to interpolate the 2D low-resolution features. Upsample the feature map to obtain a size of 2D features ;in, H Indicates the height of the input image. W Indicates the width of the input image. N Indicates the number of input images. i express N The first image i open.

3. The novel view synthesis method based on real-time rendering generalizable neural radiation field as described in claim 2, characterized in that, The 2D feature extraction module further includes a 3D mapping unit, used to map 3D points in space to 2D features of the input image. Above, pixel alignment features are obtained. ; Furthermore, the pixel-aligned features from different input views are aggregated and then pooled to obtain the image features of the three-dimensional points. .

4. The novel view synthesis method based on real-time rendering generalizable neural radiation field as described in claim 3, characterized in that, The 3D feature extraction module constructs a cost volume using the multi-scale two-dimensional features, including: Utilizing the 2D features The cost volume is constructed based on the camera frustum of the target view through differentiable homography changes; wherein N input images are projected onto several sampling planes. ,in, D The number of sampling planes, j express D The th sampling plane j One, by the 2D features Mapped to D The cost volume is constructed on each sampling plane; among them, Given the camera intrinsic parameter matrix of input view Ii K i Rotation matrix R i Translation matrix T i and the camera intrinsic matrix of the target view It K t Rotation matrix R t Translation matrix T t The homography change is defined as follows: in, Represents the target view camera axis The transpose of the matrix, It is the identity matrix. Pixels in the target view The eigenvalues ​​obtained by projecting onto the target view on the depth axis and undergoing homography are defined as follows: Based on the mapping features after homography transformation, multi-view features of each voxel are calculated. The variance is used to obtain the cost body, where, express[ u,v, The transpose of 1].

5. The novel view synthesis method based on real-time rendering generalizable neural radiation field as described in claim 3, characterized in that, After extracting the three-dimensional feature volume of the cost volume, the 3D feature extraction module further includes: performing trilinear interpolation on the three-dimensional feature volume to obtain voxel alignment features with spatial geometric information. .

6. The novel view synthesis method based on real-time rendering generalizable neural radiation field as described in claim 5, characterized in that, The neural radiation field module calculates the low-resolution target image features in the multi-scale two-dimensional features and the three-dimensional feature volume through a preset MLP network, including: The 2D and 3D features of the three-dimensional points are input into a preset MLP network to obtain the point features and volume density of the three-dimensional points, defined as: in, For MLP networks, The point features of the three-dimensional points, The volume density of the three-dimensional points is determined by the point features of the three-dimensional points. Image features of the three-dimensional points And the mixing weights for predicting the color of the input view image based on the relative viewing directions of 3D points in the target view under the input view. Then by mixed weights The color features observed when viewing a 3D point from a certain direction in the target view are defined as: in, f i The color features are the two-dimensional image features of the input view. and volume density 2D aggregation is performed to obtain the final cumulative color characteristics of each ray.

7. The novel view synthesis method based on real-time rendering generalizable neural radiation field as described in claim 6, characterized in that, Generating high-resolution target image features of the input image based on upsampling of the low-resolution target image features includes: generating high-resolution target image features of the input image by performing sub-pixel convolution image feature upsampling operation on the low-resolution target image features.

8. A novel view synthesis system based on real-time rendering generalizable neural radiation fields, comprising performing novel view synthesis of free-viewpoint videos based on the novel view synthesis method based on real-time rendering generalizable neural radiation fields as described in any one of claims 1-7, including: A network model building unit is used to construct a generalizable neural radiation field algorithm network model based on the similarity of spatial features of image patches; wherein, the generalizable neural radiation field algorithm network model introduces global information of the input image through the similarity of spatial features of image patches; A network model training unit is used to train the generalizable neural radiation field algorithm network model using data collected by a data acquisition device; wherein, the data collected by the data acquisition device includes image data and motion data of real-world scenes; The new view synthesis unit is used to synthesize new views of free-viewpoint videos using a trained generalizable neural radiation field algorithm network model.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, which enables the at least one processor to perform the steps in the novel view synthesis method based on real-time rendering generalizable neural radiation fields as described in any one of claims 1-7.

Images