Rendering a two-dimensional image of a dynamic three-dimensional scene

Abstract

Implementations are provided for rendering 2D images of a dynamic 3D scene from different viewpoints using a Neural Radiance Field (NeRF). In this regard, an AI image processing system is configured to process coordinates of points in a dynamic 3D scene over a plurality of time steps indicated by a time point of interest with an RNN to produce motion information for the points at the time point of interest, process the motion information with a fully connected neural network to produce displacements of the points from the coordinates in the dynamic 3D scene, and process the displaced points from a viewpoint of interest with a NeRF to render the points on a 2D image of the dynamic 3D scene at the time point of interest. The displaced points are generated based on an estimate of the displacements of the points by utilizing motion cues.

Description

Technical Field

[0001] This disclosure generally relates to the processing of visual data, and more specifically, to rendering two-dimensional images of dynamic three-dimensional scenes from a new perspective using neural networks. Background Technology

[0002] Novel perspective rendering refers to the process of generating a two-dimensional (2D) image or view of a three-dimensional (3D) scene from a different angle or perspective that is not part of the original input data. In the context of computer graphics and computer vision, this task involves creating a new 2D image of a 3D scene from viewpoints that were not explicitly captured during the initial data acquisition.

[0003] A common application of new-view rendering is in virtual reality (VR) and augmented reality (AR) systems. In these systems, users can move their heads or change their positions, requiring the generation of new views of the virtual environment to maintain a realistic and immersive experience. New-view rendering can also be relevant in the context of 3D reconstruction, where the goal is to generate new images of a scene from an arbitrary viewpoint based on a finite set of input images.

[0004] Various techniques can be used for novel perspective rendering, including traditional computer graphics and vision-based methods, as well as state-of-the-art approaches based on deep learning and neural networks. While computer graphics / vision methods have been widely used for novel perspective rendering, they have certain drawbacks. These drawbacks include, but are not limited to, limited realism, sparsity in the reconstructed 3D scene structure, manual modeling and parameter tuning, limited generalization, limited handling of complex lighting, lack of scene understanding, and computational complexity.

[0005] For example, deep learning models, such as Neural Radiation Field Networks (NeRF), can be trained to understand the 3D structure of a scene and generate realistic images of the scene from new viewpoints. These models learn to capture the spatial relationships and properties of 3D scenes, allowing them to generalize to unseen perspectives.

[0006] Deep learning models are typically used for perspective rendering of static scenes (i.e., when objects do not move or change their configuration over time). Furthermore, conventional deep learning models for photorealistic rendering require manual and computational work to create the scene and render a realistic image.

[0007] Therefore, to fully realize the potential of new perspective rendering methods and achieve even broader impact, it is crucial to extend the capabilities of perspective rendering to dynamic scenes (i.e., scenes where objects may change position over time). The most promising methods for creating rendered content for dynamic scenes typically utilize deformation fields. They use Deformation Field Neural Networks (DFNs) for modeling. DFNs are designed to handle spatial deformations in scenes due to their dynamic nature. The main idea behind DFNs is to model and predict the deformations of 3D points that may occur between different views or frames in a dynamic 3D scene.

[0008] However, these methods have a major drawback: they use a DFN to directly deform the 3D volume corresponding to each frame of a dynamic 3D scene into a common canonical frame. From a learning perspective, this presents a challenge because the DFN must learn a wide range of motion, including frames that are temporally distant from the canonical frame, nearby frames, and everything in between. In other words, the DFN must be trained on a training dataset representing the various deformations and dynamic scenes that may be encountered during viewpoint rendering. Furthermore, designing a network capable of learning such complex motion patterns is challenging. Summary of the Invention

[0009] New perspective rendering plays a central role in AR / VR applications. Perspective rendering can be used for things like creating virtual flights of tourist sites and creating advertising displays for cars. Other applications of new perspective rendering include film and games, allowing for use cases such as rendering action sequences from more visually appealing viewpoints and creating pleasing visualizations of character movement.

[0010] Some embodiments of this disclosure are based on the understanding that new perspective rendering tasks are relatively easier to complete if the 3D scene being rendered is static, i.e., the objects in the 3D scene do not move or change their configuration over time. However, this limits the scope of perspective rendering techniques. Therefore, to realize the potential of such perspective rendering techniques, it is crucial to extend the capabilities of these techniques to dynamic scenes, such as scenes where the position and configuration of objects change over time.

[0011] The ability to render dynamic scenes from new perspectives expands the applicability of perspective rendering technology to fields such as film and games, thereby allowing use cases such as rendering action sequences from more visually appealing viewpoints and creating pleasing visualizations of character movement.

[0012] In some cases, NeRF-based methods are used for novel perspective rendering in dynamic 3D scenes. NeRF-based methods overcome the sparsity challenge in the recovered geometric representation faced by traditional computer vision-based methods by modeling a function that maps the volume representation of a scene to its color and opacity values, such as those rendered from a given viewpoint, via a deep neural network. As an example, a multilayer perceptron (MLP) is used to map the volume representation of a scene to its color and opacity values. The ability of NeRF-based methods to render scenes with detail while maintaining quality has led to the widespread adoption of these techniques for novel perspective rendering tasks.

[0013] However, these NeRF-based methods are only well-suited for static scenarios, i.e., scenarios where the pose and configuration of objects do not change over time.

[0014] The purpose of this disclosure is to enable efficient rendering of 3D scenes from a new perspective when the scene is dynamic (i.e., a scene whose content or objects change over time).

[0015] Some implementations are based on the understanding that NeRF-based methods for rendering new perspectives in 3D dynamic scenes can model the rendering task in a common frame (called a canonical frame) and use a deformation field to project 3D volume information from all frames into this common space.

[0016] It can be noted that the Dynamic NeRF method builds upon the original NeRF method by extending it to handle dynamic scenes or objects that undergo time-varying changes. While the original NeRF was designed for static scenes, Dynamic NeRF allows for the representation and rendering of scenes with objects that move or change their appearance over time. The basic idea in Dynamic NeRF is to model not only spatial changes in the scene but also temporal changes. This is achieved by introducing an additional temporal dimension into the NeRF representation, making it a function of both space and time. The Dynamic NeRF (DNF) model represents the radiation or color of points in space and time, capturing the appearance of a scene or object at different points in time. Typically, such a representation is instantiated using a cascaded DFN and a regular NeRF, where the DFN is responsible for estimating the displacement of 3D points, and the NeRF renders that displacement point from the provided viewing direction.

[0017] To train a dynamic NeRF model, a dataset containing spatiotemporal information, such as a sequence of images captured over time, is typically required. Each training sample in the dataset consists of an image, camera pose, and a corresponding timestamp. The dynamic NeRF model parameters are then optimized to minimize the difference between the rendered image from the dynamic NeRF model and the actual captured image at different time points. During inference, the dynamic NeRF model can be used to generate new perspectives of dynamic scenes or objects at any time point. By evaluating the dynamic NeRF model function at different spatiotemporal coordinates, it can render frames of temporal changes and movement within the captured scene. As an example, dynamic NeRF has implications in various applications such as virtual reality, augmented reality, robotics, and autonomous driving. It allows for realistic rendering and compositing of dynamic scenes, enabling more immersive experiences and accurate simulations of real-world dynamics.

[0018] Therefore, some implementations are based on the observation that using only one DFN to render the entire scene is inefficient for dynamic scenes in which the configuration, position, etc. of objects in the scene may change in many ways.

[0019] Some implementations are based on the understanding that traditional methods for rendering new perspectives of a scene rely on recovering the 3D geometry of the scene given multiple input views. However, these methods depend on matching keypoints between the input views and upscaling them into 3D space, which can lead to sparsity in the recovered geometric representation, as the 3D geometry is recovered only for the matched keypoints rather than for the entire scene. Therefore, these methods perform poorly in dynamic scenes.

[0020] Some implementations are based on the understanding that modeling scene dynamics of dynamic 3D scenes via deformation fields has a major drawback. Specifically, NeRF-based methods using deformation fields employ a Deformation Neural Network (DFN) to directly deform the 3D volume of each frame into a common canonical frame. From a learning perspective, this presents a challenge because the DFN must learn a wide range of motion, including for frames that are temporally far from the canonical frame, nearby frames, and other frames between distant and nearby frames.

[0021] Therefore, embodiments of this disclosure aim to avoid problems related to learning a wide range of motion for tasks involving the efficient rendering of 3D dynamic scenes from new perspectives. Embodiments of this disclosure also allow for interpretable representations of the temporal dynamics of the scene.

[0022] As an example, the challenge of estimating the displacement of 3D points in a dynamic scene leads to a solution in which a DFN can be instantiated using a cascade of a recurrent neural network (RNN) and a fully connected neural network (FCNN) trained to estimate the displacement of points from a specific time point to a frame at a common timestamp. An example of such a fully connected neural network is a multilayer perceptron (MLP), a feedforward artificial neural network that generates a set of outputs from a set of inputs. An MLP is characterized by several layers of nodes connected as a directed graph between the input and output layers. However, the MLP itself can receive the time point of interest and the coordinates of the point, and generate the displacement of that point from the common time point (typically corresponding to the time point of the first frame) to the time point of interest. While FCNNs (such as MLPs) are advantageous for training on dynamic scenes, MLPs themselves do not consider the concept of time as part of their architecture.

[0023] Some embodiments of this disclosure are based on the understanding that the architectural flaws of MLPs that do not model the concept of time can lead to inaccurate estimations of the displacement of points over time. Therefore, some embodiments aim to address this flaw.

[0024] Some implementations are based on the understanding that while the temporal embedding of FCNN can directly estimate the displacement of points over the entire time span, it is necessary to recursively estimate the displacement of each point to better represent the dynamic temporal evolution of a dynamic scene. Doing so in this way mimics the concept of time in the displacement estimation network.

[0025] Therefore, some embodiments of this disclosure are based on the understanding that adding a recurrent neural network (RNN) to a pipeline with an architecture that implements a short-term memory with a recursive structure that mimics temporal evolution can overcome the shortcomings of FCNN. Examples of such recurrent neural networks include gated recurrent units (GRUs) that act as gating mechanisms within the recurrent neural network. The GRU can receive the coordinates of a point of interest and a time point of interest to generate a representation of the motion of the coordinates over a fixed time step from the start of time (the first frame) to the time point of interest. The MLP is then modified to receive this motion representation to generate the displacement of the coordinates from it.

[0026] In this way, the temporal concept is moved from FCNN to RNN to more accurately capture motion cues. Therefore, motion can be predicted more accurately over an expanded prediction range. Furthermore, due to the internal memory of RNN, the displacement of points can be estimated recursively, making RNN and FCNN function as recursive units.

[0027] Some embodiments of this disclosure are based on the understanding that the task of deforming a 3D volume can be considered as the deformation of a set of points sampled in 3D space. Furthermore, the deformation of a 3D point over time is equivalent to estimating a vector that connects the 3D point at its current position in the current time frame to its position in the canonical frame time frame.

[0028] Some embodiments of this disclosure are based on the understanding that the vector of a 3D point can be decomposed into a piecewise linear manner, wherein each decomposed segment represents the displacement of the 3D point between two consecutive time ranges.

[0029] Furthermore, some embodiments of this disclosure are based on the understanding that a model based on gated recursive unit-NeRF (G-NeRF) can be used to model the movement of 3D points across different time steps in a dynamic 3D scene.

[0030] Therefore, the deformable network architecture of the G-NeRF-based model includes gated recurrent units (GRUs) and an MLP. As an example, the GRU tracks the history of displacement of 3D points in past frames, enabling prediction of future displacements of 3D points for motion in a 3D scene. To this end, the GRU tracks past displacements of 3D points, while the MLP predicts the displacement of 3D points within each fixed-length time interval or within the remaining time interval.

[0031] In one aspect, an artificial intelligence (AI) image processing system is provided. The AI ​​image processing system employs a neural radiation field (NeRF) to render a two-dimensional (2D) image of a dynamic three-dimensional (3D) scene from different viewpoints and at different time points based on an implicit representation of the 3D scene. The AI ​​image processing system includes at least one processor and a memory storing instructions thereon. These instructions cause the at least one processor of the AI ​​image processing system to process the coordinates of points in the dynamic 3D scene at multiple time steps indicated by a time point of interest using a recurrent neural network, to generate motion information of the points at the time points of interest. Furthermore, the stored instructions cause the at least one processor to process the motion information using a fully connected neural network to generate displacements of the points from the coordinates in the dynamic 3D scene, and to process the displacement points from the viewpoint of interest using a NeRF trained on a static 3D scene, to render the points on the 2D image of the dynamic 3D scene at the time points of interest. As an example, displacement points are generated based on the displacement of the points.

[0032] According to an additional implementation, the recurrent neural network and the fully connected neural network execute recursively at multiple time points until the time point of interest is reached, such that the displacement of the point determined during the current iteration is the input for subsequent iterations.

[0033] According to an additional implementation, the recurrent neural network includes gated recurrent units (GRUs), and the execution of each GRU corresponds to the movement of a point over a fixed time step.

[0034] According to an additional implementation, the maximum number of iterations of the GRU defines the maximum length of the prediction range. Furthermore, the recurrent neural network and the fully connected neural network execute recursively at different time points until the time point of interest is reached, such that the displacement of the point determined during the current iteration is the input for subsequent iterations.

[0035] According to the additional implementation, each recursion operates over a time range whose length is no greater than the maximum length of the predicted range.

[0036] According to an additional implementation, the displacement of a point determined during the current iteration is input into subsequent iterations and combined with the hidden state at the subsequent iteration using a nonlinear mapping.

[0037] According to an additional implementation, the output at the current iteration is a function of the GRU's current input and hidden state.

[0038] According to an additional implementation, the output of the GRU at the current iteration is a nonlinear function of the weighted sum of the current input and the hidden state at the current iteration.

[0039] According to an additional implementation, NeRF is trained to implicitly represent the 3D geometry and appearance of a 3D scene, and learns a continuous 3D function with learnable parameters based on the viewpoint of interest and the coordinates of the displacement point.

[0040] According to an additional implementation, backpropagation time (BPTT) is used to train NeRF.

[0041] According to an additional implementation, the recurrent neural network (RNN) is a bidirectional RNN.

[0042] According to an additional implementation, rendering points on a 2D image of a dynamic 3D scene generates a new view of the dynamic 3D scene.

[0043] According to an additional implementation, the point represents the position of an object or a portion thereof in a dynamic 3D scene.

[0044] On the other hand, an artificial intelligence (AI)-based image processing method is provided. This AI-based image processing method utilizes an AI image processing system employing Neural Radiation Field (NeRF) to render a 2D image of a dynamic 3D scene from different viewpoints and time points based on an implicit representation of the 3D scene. The AI-based image processing method includes processing the coordinates of points in the dynamic 3D scene at multiple time steps indicated by a time point of interest using a recurrent neural network to generate motion information of the points at that time point. The AI-based image processing method also includes processing the motion information using a fully connected neural network to generate displacements of points from the coordinates in the dynamic 3D scene, and processing the displacement points from the viewpoint of interest using a NeRF trained on a static 3D scene to render the points on the 2D image of the dynamic 3D scene at the time point of interest. As an example, displacement points are generated based on the displacement of the points.

[0045] In another aspect, a non-transitory computer-readable storage medium contains a program executable by a processor for performing a method. The method includes processing the coordinates of points in a dynamic 3D scene at multiple time steps indicated by a time point of interest using a recurrent neural network to generate motion information of the points at those time points. As an example, a neural radiation field (NeRF) is used to render a two-dimensional (2D) image of the dynamic 3D scene from different viewpoints and at different time points based on an implicit representation of the 3D scene. The method also includes processing the motion information with a fully connected neural network to generate displacements of points from the coordinates in the dynamic 3D scene, and processing the displacement points from the viewpoint of interest using a NeRF trained for a static 3D scene to render the points on a 2D image of the dynamic 3D scene at the time point of interest. As an example, displacement points are generated based on the displacement of the points.

[0046] The present disclosure is further described in the following detailed description section with reference to the numerous accompanying drawings, which are non-limiting examples of exemplary embodiments of the present disclosure, and in several views of the drawings, the same reference numerals denote similar parts. The drawings shown are not necessarily drawn to scale, but generally focus on illustrating the principles of the embodiments of the present disclosure. Attached Figure Description

[0047] [ Figure 1 ]

[0048] Figure 1 An exemplary environment is shown, according to one or more embodiments, in which an artificial intelligence-based image processing system is implemented for viewpoint rendering in a dynamic scene.

[0049] [ Figure 2 ]

[0050] Figure 2 The following are illustrated according to one or more embodiments. Figure 1 A block diagram of the system.

[0051] [ Figure 3A ]

[0052] Figure 3A Example architectures of recurrent neural networks (RNNs) according to one or more implementations are shown.

[0053] [ Figure 3B ]

[0054] Figure 3B An example architecture of a gated recurrent unit (GRU) according to one or more implementations is shown.

[0055] [ Figure 3C ]

[0056] Figure 3C An example architecture of NeRF according to one or more implementations is shown.

[0057] [ Figure 4 ]

[0058] Figure 4 A block diagram of a NeRF-based method for rendering 2D images of dynamic scenes is shown.

[0059] [ Figure 5A ]

[0060] Figure 5A An example block diagram of the architecture of a displacement network of a system according to one or more implementations is shown.

[0061] [ Figure 5B ]

[0062] Figure 5B Example architectures of rendering networks according to one or more implementations are shown.

[0063] [ Figure 6 ]

[0064] Figure 6 An example flowchart of a method for training a system according to one or more embodiments is shown.

[0065] [ Figure 7 ]

[0066] Figure 7 An example flowchart of a method for generating images of a dynamic scene according to one or more embodiments is shown.

[0067] [ Figure 8 ]

[0068] Figure 8 A block diagram of a system according to some implementations is shown. Detailed Implementation

[0069] While the accompanying drawings illustrate the embodiments currently disclosed, other embodiments are contemplated, as pointed out in the discussion. This disclosure presents illustrative embodiments by way of representation and not limitation. Those skilled in the art can devise many other modifications and embodiments falling within the scope and spirit of the principles of the embodiments currently disclosed.

[0070] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, apparatus and methods are shown only in block diagram form to avoid obscuring this disclosure.

[0071] The terms “for example,” “like,” and “such as,” as well as the verbs “comprising,” “having,” “including,” and other verb forms thereof, as used in this specification and claims, when used in conjunction with a list of one or more components or other items, are to be interpreted as open-ended, meaning that the list is not considered to exclude other additional components or items. The term “based on” means at least partially based on. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0072] Overview of Implementations of the Artificial Intelligence (AI) Image Processing System Based on the Present Disclosure

[0073] The disclosed AI image processing system includes a Neural Radiation Field (NeRF) model for 3D scene representation and new viewpoint rendering of dynamic scenes. The NeRF model is capable of performing viewpoint rendering tasks, allowing the synthesis of high-quality images from new viewpoints. The key principle of the NeRF model is to model a volume scene function that captures both the geometry and appearance of a 3D scene. However, NeRF is typically only successful for viewpoint rendering in static scenes and exhibits poor performance in dynamic scene modeling. For example, traditional NeRF models cannot accurately model the deformation of points in the scene over time.

[0074] To this end, embodiments of this disclosure provide an AI image processing system for performing novel perspective rendering in dynamic 3D scenes. As an example, the AI ​​image processing system relies on dividing the displacement estimation task into time ranges or time steps of fixed time units. Furthermore, for each time range of a time unit, the displacement of point 'x' is estimated for that time range. As an example, the displacement at the current iteration is then used to estimate another displacement for subsequent time ranges or subsequent iterations. This process is repeated until the time of interest is reached.

[0075] To achieve the aforementioned steps for viewpoint rendering in dynamic 3D scenes, a NeRF model is used. The NeRF model comprises a deformable network and a rendering network. For example, the deformable network includes gated recurrent units (GRUs) or gated recurrent neural networks. The proposed NeRF architecture enables efficient modeling of objects in dynamic 3D scenes for rendering images from new viewpoints. To this end, the proposed GRU-based NeRF model allows for interpretable representations of the temporal dynamics of 3D scenes, while addressing the challenges associated with large learning or training datasets for capturing wide ranges of motion that are difficult to capture.

[0076] Figure 1 An exemplary environment 100 according to one or more embodiments is shown, in which an artificial intelligence (AI) image processing system 102 is implemented for viewpoint rendering in a dynamic scene 104. Additional, fewer, or other elements may be added to the environment 100.

[0077] The AI ​​image processing system 102 (hereinafter referred to as system 102) is a deep learning neural network. As an example, system 102 is communicatively connected to the dynamic scene 104 via a sensor (not shown). As an example, the sensor is an optical sensor, such as an image sensor, camera, video camera, etc. The sensor is configured to record or capture images, image sequences, or videos of the dynamic scene 104.

[0078] As an example, dynamic scene 104 refers to a setting or environment that involves movement, change, and activity over time. For example, dynamic scene 104 means that there are moving objects, interactions, and transformations. Dynamic scenes 104 are characterized by their temporal variability and may include elements such as people, vehicles, or any objects (depicted as objects 106) that undergo movement or change.

[0079] According to this example, dynamic scene 104 is represented as a football match. In this respect, the sports field can be considered dynamic scene 104. For this purpose, moving players can correspond to dynamic objects 106 in dynamic scene 104. For example, players on the field are in constant motion due to running, dribbling, and participating in various activities. Their positions change rapidly, and they interact with each other as they move toward the ball, defend, or attempt to score. Subsequently, these players can form dynamic objects in the dynamic scene.

[0080] Although this example represents a player as dynamic object 106, this should not be construed as limiting. In dynamic scene 104, other dynamically moving objects may exist that can be rendered with viewpoints. Examples of such objects may include, but are not limited to, balls, crowds, cameras, etc. Furthermore, the representation of a football match as dynamic scene 104 is merely exemplary and should not be construed as limiting. Other examples of dynamic scenes may include, but are not limited to, bodies of water (such as waterfalls, beaches, oceans, etc.), vehicle traffic on roads, motorcycles, crowds in areas, and cityscapes. Additionally, it can be noted that dynamic scenes may also include static objects, such as trees, buildings, etc. Therefore, in some cases, embodiments of this disclosure may also be implemented in static scenes, or used to render images of static objects from new viewpoints.

[0081] Returning to this example, dynamic scene 104 involves the constant or variable motion of object 106 over time. Therefore, understanding and accurately representing such dynamic scenes poses a challenge to computer vision systems, especially when tasks such as viewpoint rendering are involved, as dynamic scenes evolve over time.

[0082] As mentioned above, typical NeRF-based models are well-suited for static scenes; however, they cannot accurately model objects in dynamic scenes where the scene content changes over time.

[0083] For example, NeRF-based models can model rendering tasks for new perspective rendering in common or canonical frames, using deformable fields to project 3D volume information from all frames to the canonical frame. However, this technique suffers from drawbacks during implementation in dynamic scenes. From a learning perspective, this presents a challenge because the 3D volume of each frame is directly deformed into the common canonical frame. Models capable of capturing the complexity of this motion are difficult to develop, as deformable field neural networks need to learn a wide range of motions for different possible dynamic objects across multiple frames of the scene. Consequently, effective perspective rendering of the scene from new viewpoints is hindered. As an example, deformable field neural networks estimate incorrect displacements or shifts for points over long time periods, leading to inaccurate perspective rendering.

[0084] Novel perspective synthesis using radiation field models or neural radiation field (NeRF) models has proven highly successful in the task of novel perspective synthesis for static scenes. These models learn from 3D points... and its viewing direction ( ) to its color ( ) and opacity ( A mapping of 3D points. For example, opacity provides a measure of how much light passes through a 3D point. This is achieved using a multilayer perceptron (MLP) network. Learning this mapping, the Multilayer Perceptron (MLP) network Defined as: (1) In some cases, the parameters of an MLP network are trained to minimize the photometric reconstruction loss of a scene within a given viewpoint set. However, the initial design of a NeRF model with an MLP network presents several challenges, including slow training and inference speeds and an inability to handle scenes with objects at different depths.

[0085] Some implementations are based on the understanding that the Kilo (Thousand-Part NeRF)-NeRF model can be used to address the speed-dependent issues in the traditional architecture of NeRF models. At this point, the scene can be segmented into several parts, and each part can be rendered using a separate, but much smaller, NeRF. In some cases, the MLP network can be completely removed from the NeRF model to directly use spherical harmonic functions on a 3D voxel mesh to optimize opacity and view-dependent color. In some other examples, the voxel mesh can be decomposed into a set of 2D planes for projecting 3D points onto these 2D planes. To implement any of the above techniques in a traditional NeRF model, mappings are learned on the mesh on each of these 2D planes, which produces the color and opacity values ​​of the 3D points given a viewing orientation by synthesizing the outputs from each of the 2D planes. However, the scenes processed by these methods are inherently static, meaning that objects in the scene do not change their position and / or configuration over time. This presents a barrier to extending these methods to scenes with motion.

[0086] In some cases, Deformable Neural Radiation Field (D-NeRF) can be used to render dynamic scenes from a new perspective. In this regard, dynamic scenes can include objects whose position, configuration, latent shape, and other properties change over time. There are two main categories of work in D-NeRF models. In the first approach, a deformable field neural network is used to explicitly model the displacement of 3D points over time, and the displaced points are rendered using a standard NeRF model. As an example, in this approach, the deformable field neural network... It can be represented as: (2) Furthermore, the standard NeRF model learns a mapping between the displacement point and a given viewing direction. This mapping can be learned as: (3) Alternatively, in the second approach, the temporal encoding is fed as input into a standard NeRF model. While in the first approach the displacement of each point of an object in the scene is explicitly modeled, in the second approach the displacement of each point is implicitly learned. Subsequently, the mapping in the second approach can be represented or learned as: (4) Where f(·) represents some encoding of the timestamp of the current time frame t. As an example, position encoding is used to encode time information.

[0087] To address this, some existing work has attempted to alleviate the challenges of modeling dynamic scenes by decoupling the task of rendering new perspectives of dynamic scenes into two subtasks. In the first subtask, the positional displacement of 3D points in the scene is estimated over time. Then, a standard NeRF-based method is used to render the displacement points from the new perspective.

[0088] Furthermore, some approaches seeking to improve the rendering of dynamic scenes have sought to derive improved representations of dynamic scenes using scene motion cues and / or disentangled semantics. A common element of all these aforementioned methods for dynamic scene rendering is that they utilize a two-stage rendering pipeline. In the first stage, the displacement of a point's position in 3D coordinates is estimated, while in the second stage, the appearance of that point from the desired viewpoint is predicted. Therefore, the conventional dynamic NeRF architecture is decoupled into pixel displacement (or deformable network) and static NeRF. In this paper, all motion predictions are encapsulated by a deformable network that triggers a static NeRF for the displacement pixels.

[0089] While these methods offer promising initial directions for modeling dynamic scenes, the task of capturing all kinds of displacements within any time frame (short or long) makes the learning task challenging, error-prone, and inefficient. Regardless of whether the mapping between 3D points and opacity and color is explicitly or implicitly modeled via deformable networks, learning the displacements of all moving points in a scene that may belong to many different types of objects is difficult.

[0090] In some cases, a dynamic scene can be decomposed by modeling each object in the scene using a separate dynamic NeRF. However, like other deformable network-based dynamic NeRF techniques, this approach attempts to directly learn the displacement of 3D points at a given time step. Therefore, this approach may also fail to accurately model the deformation of points.

[0091] Therefore, estimating the displacement of 3D points in a scene for use in dynamic NeRF can be challenging for several reasons. For example, dynamic NeRF requires a dataset with spatiotemporal information, which includes sequences of images captured over time. However, obtaining such a dataset with ground-value 3D point displacements can be difficult and time-consuming. Acquiring accurate ground-value data for dynamic scenes typically involves sophisticated motion capture systems or specialized equipment. Furthermore, estimating the motion and displacement of 3D points from image sequences is an ill-posed problem. Consequently, designing a model capable of successfully performing this task is difficult. For example, multiple 3D motions can interpret the same 2D motion observed in an image, leading to ambiguity in motion estimation. This ambiguity makes it challenging to accurately infer the displacement of 3D points in a scene during training and inference. In addition, dynamic scenes often involve occlusion, rapid motion, deformation, and other complex dynamics. These factors introduce additional challenges when estimating accurate point displacements. For example, occlusion can obstruct the visibility of certain points in a scene, making it difficult to accurately track their motion, while rapid motion and deformation can cause significant changes in the appearance of objects, further complicating the estimation of point displacements. Furthermore, dynamic scenes can contain non-rigid and deformable objects, such as people or animals. In this regard, estimating the displacement of such objects requires modeling their complex deformations by considering their shape and appearance changing over time. Therefore, accurately capturing and modeling deformations is a challenging task, especially for highly flexible and jointed objects.

[0092] Therefore, the purpose of this disclosure is to provide a technique for accurately modeling the deformation of 3D points in a dynamic scene over time.

[0093] According to this embodiment, system 102 is configured to render a two-dimensional (2D) image 110 of dynamic scene 104 or objects 106 in dynamic scene from a new viewpoint or a different viewpoint using NeRF 108 (also known as NeRF model 108 or NeRF network 108).

[0094] As an example, NeRF 108 combines recurrent neural networks and fully connected neural networks. For this purpose, system 102 is configured to perform displacement estimation tasks within fixed time units of C frames. Furthermore, system 102 is configured to estimate the displacement of a 3D point separately for each of these time units. Based on the final displacement point, the viewpoint of interest, and the time of interest, an image or view of the object at the new viewpoint is generated. This is combined with, for example... Figure 2 Further details of system 102 are described.

[0095] Figure 2A block diagram 200 of an AI image processing system 102 according to an example implementation is shown. The AI ​​image processing system 102 employs NeRF 108 to render a 2D image 214 of a dynamic 3D scene. As an example, the dynamic 3D scene may correspond to a video game scene, a real-world scene, or an augmented reality scene. The dynamic 3D scene (such as dynamic scene 104) may include moving objects that can change their position and / or configuration over time.

[0096] To this end, system 102 is configured to render 2D images 214 of the 3D scene from different perspectives and at different points in time based on the implicit representation of the 3D scene.

[0097] As an example, implicit representation refers to a mathematical description or model of an object or scene for which the surface or boundaries of the object are not explicitly defined. In implicit representation, the object is represented by an implicit function that can determine whether a given point is inside or outside the object. As an example, implicit representation is defined as a function in three-dimensional space. ,in These are the coordinates of a point in the scene or on an object. The sign of the function's value at a particular point indicates whether that point is inside or outside the object. For example, if If the point is outside the object, then the point is outside the object; if The point is inside the object; if The point is on the surface of the object.

[0098] As an example, system 102 includes a recurrent neural network (RNN) 204, a fully connected neural network (FCNN) 206, and a NeRF 108.

[0099] As an example, RNN 204 is configured to process sequential data by maintaining a hidden state that captures information from previous time steps. RNN 204 can include loops that allow information to be carried across different time steps, making them suitable for tasks involving sequences and capturing dependencies across time steps. Subsequently, RNN 204 is configured to determine the displacement of points within different time steps or time ranges. As an example, RNN 204 is implemented as a Gated Recurrent Unit (GRU), Long Short-Term Memory (LSTM), Temporal Convolutional Network (TCN), etc.

[0100] Furthermore, FCNN 206 is a feedforward neural network where each neuron in one layer connects to every neuron in the next layer, without any skipping or connections within the same layer. FCNN 206 is configured to implement a mapping between a viewpoint or perspective of interest for an object or scene and parameters for the scene, such as color and opacity. Examples of FCNN 206 can include, but are not limited to, multilayer perceptrons (MLPs).

[0101] Going further, NeRF 108 is configured to represent and render 3D scenes, particularly rendering images 214 of dynamic scenes from new perspectives and times of interest. As an example, NeRF 108 is configured to model volume scene functions to capture the geometry and appearance of the 3D scene. NeRF 108 can implicitly represent the 3D scene for predicting radiance values ​​for any 3D point in the scene. To this end, NeRF 108 can generate realistic images of dynamic 3D scenes from new viewpoints.

[0102] In operation, system 102 can receive dynamic 3D scene 202. Static 3D scene can actually be associated with dynamic 3D scene 202 (hereinafter referred to as 3D scene 202), wherein static 3D scene can be an image of dynamic 3D scene captured at a certain point in time.

[0103] The 3D scene 202 is fed into the RNN 204 for processing. As an example, the RNN 204 is configured to process the coordinates of points in the 3D scene 202 at multiple time steps (e.g., ...). The number of time steps used for processing is indicated by the time point of interest. As an example, the time point of interest can be the time point for which an image of the dynamic scene is to be generated. Subsequently, the prediction range {1, ..., t} between time point t and the current time point associated with the capture of the 3D scene 202 is divided into fixed-length... Different time steps or time ranges, where the final time range or remaining time range can span the remaining time amount (i.e., the remaining time range can have a time length). The motion information 208 for points in the 3D scene 202 is estimated as displacement over various time ranges or time steps. Similarly, motion information 208 is determined for each point in the 3D scene 202.

[0104] According to embodiments of this disclosure, the RNN 204 may include or be implemented as a gated recursive unit (GRU). For this purpose, the execution of each iteration of the GRU may correspond to the movement of the point within a time step. As an example, each iteration of the GRU may generate a displacement for the point within a corresponding time step. Subsequently, the final iteration of the GRU may indicate the displacement of the point for the time point of interest t.

[0105] Continuing further, motion information 208 is fed into FCNN 206. As an example, FCNN 206 is configured to process motion information 208 to generate displacements of points from coordinates in the 3D scene 202. For example, based on the output of RNN 204 or motion information 208, FCNN 206 is configured to predict the position or coordinate displacement of a point in the scene after displacement, from the current time point to the time point of interest. In this way, FCNN 206 determines a new position, such as point coordinates, for a point at the time point of interest. Subsequently, based on the new position or new point coordinates, the point can be displaced. Once displaced, the point (now referred to as displacement point 210) is fed into NeRF 108. As an example, displacement point 210 is generated based on the displacement of this point across different time ranges or time steps.

[0106] As an example, RNN 204 and FCNN 206 are recursively executed for different time points until the time point of interest is reached. In other words, for each different time range, RNN 204 is executed to generate motion information 208 for the corresponding time range, and FCNN 206 is executed on the generated motion information to generate displacements of points for the corresponding time range. Thus, the displacements of points are calculated in each different time range or time step. Subsequently, the displacement points determined during the current iteration or current time range are the inputs for subsequent iterations or subsequent time ranges. To this end, displacement points 210 are generated for the time point of interest based on the displacements generated in each iteration corresponding to different time ranges spanning the prediction range.

[0107] Subsequently, NeRF 108 was configured to operate from the viewpoint of interest. 212 processes displacement point 210. As an example, NeRF 108 is trained on a 3D scene 202 to render displacement point 210 on a 2D image 214 of the dynamic 3D scene at time t of interest. Specifically, NeRF 108 is specifically trained on the 3D scene 202 at time t so that NeRF 108 can accurately model points of the 3D scene 202 within the image 214 generated from the new viewpoint 212 of interest. Subsequently, using the 3D scene 202, other points can be reliably displacementd and rendered from the desired viewpoint and the desired time point of interest without needing to train the model once or all at once on a large dataset corresponding to the motion of various objects in the dynamic scene.

[0108] As an example, rendering a point or displacement point 210 on a 2D image 214 of a dynamic 3D scene generates a new view of the 3D scene. For example, since the displacement point 210 is processed based on the viewpoint of interest and the time point of interest, the point is updated to correspond to the new view in the modified scene. Therefore, rendering the displacement point 210 processed based on the viewpoint of interest generates a new view of the 3D scene.

[0109] Overview of Recurrent Neural Networks

[0110] Figure 3A This describes an example architecture 300 for a recurrent neural network (RNN). An RNN is a type of feedforward neural network. RNNs are capable of predicting, processing, and modeling sequential and temporal data. Architecture 300 corresponds to RNN 204 in the implementation described herein.

[0111] Typically, an RNN includes an input layer 302, multiple hidden layers (described as hidden layers 304A and 304B, collectively referred to as hidden layer 304), and an output layer 306. In particular, an RNN may include a chain of repeating neural network modules called units, where each unit takes an input from the current time step and an output from a previous time step. This allows the RNN to capture dependencies and patterns in sequential data.

[0112] As an example, the hidden layers 304 of an RNN are configured to remember specific information about the sequence. These hidden layers 304 allow past outputs to be used as inputs and store information about the sequence, including the time series. As an example, information in a recurrent neural network can be circulated through loops in the intermediate hidden layers 304.

[0113] As an example, input layer 302 receives and preprocesses the input to the RNN before passing it to hidden layer 304. Furthermore, multiple hidden layers 304 can form intermediate layers of the RNN. For example, each hidden layer 304A or 304B can have its own corresponding activation function, weights, and biases. For example, the different activation functions, weights, and biases of hidden layers 304 can be normalized to ensure that each hidden layer has the same characteristics. As an example, the activation function used in the hidden layer 304 of the RNN can be, for example, the sigmoid function, the hyperbolic tangent (tanh) function, or the rectified linear unit (ReLU) function.

[0114] It should be noted that the number of hidden layers 304 being two is merely exemplary. In some cases, a single hidden layer may exist, on which there are loops a predetermined number of times. In some other cases, each of the multiple hidden layers may have a corresponding loop a predetermined number of times.

[0115] For example, information can be passed through loops within an RNN. To do this, before making a decision or prediction, the RNN evaluates the current input and what it has learned from past inputs (i.e., the outputs of past inputs). For instance, an RNN can retrieve past outputs due to internal memory to produce an output for the current iteration.

[0116] As an example, input layer 302 can take an input vector as input. The input vector can represent the input data at the current time step. Furthermore, output layer 306 produces an output vector that serves as the hidden state for the prediction of the time step of interest. The circular connections between hidden layers 304 allow information or output from previous layers or time steps preceding the time step of interest to influence the prediction for that time step.

[0117] RNNs can be trained using, for example, backpropagation in time (BPTT), gradient clipping, loss functions, or combinations thereof. As an example, during RNN training via BPTT, the backpropagation algorithm is applied to the RNN, where time-series data is used as its input for training. For instance, typically in an RNN, a single input is sent to the input layer 302 at a time, resulting in a single output. However, during backpropagation, both the current input and previous inputs are used. Subsequently, a time step can include multiple time-series data points entering the RNN simultaneously. Afterward, once the RNN has predicted the output of the time-series data, the output from the output layer of the RNN is used to compute and collect the error. The RNN is then rolled back, and its weights are recalculated and adjusted to account for faults or errors.

[0118] Examples of different types of RNNs include, but are not limited to, vanilla RNNs, Long Short-Term Memory (LSTM) RNNs, and Gated Recurrent Units (GRUs). Typical vanishing gradient RNNs suffer from the gradient vanishing problem, limiting their ability to capture long-range dependencies. Furthermore, while LSTMs address the vanishing gradient problem by introducing memory units and gating mechanisms to selectively remember or forget information, their modeling can be complex.

[0119] Some implementations are based on the understanding that RNNs are widely used for modeling time series data. RNNs can operate in an autoregressive manner. For example, at each time step... RNN accepts input and using nonlinear mapping and trainable parameters Its hidden state relative to the previously entered information Combining to produce output The process of generating output, for example, a rendered image for dynamic scene 104, continues until the point of interest is reached, i.e., for each time step. .

[0120] According to embodiments of this disclosure, an RNN is trained using a method known as Backpropagation in Time (BPTT) by backpropagating gradients from the point of interest to the beginning of each time step. For example, a basic RNN processes the input sequence or input vector only in the forward or backward direction. Therefore, the RNN used in this disclosure can be a bidirectional RNN, a variant of the basic RNN, which can process the input sequence or input vector in both directions.

[0121] According to the example implementation, a GRU is used to implement an RNN 204 for system 102. A GRU is a simplified version of an LSTM, providing a balance between efficiency and performance with fewer parameters. Combined with, for example... Figure 3B Describe the details of GRU.

[0122] Overview of gated recursive units

[0123] refer to Figure 3B This diagram illustrates an example architecture 310 of a gated recurrent unit 312 according to some embodiments of the present disclosure. The GRU 312 is a recurrent neural network (RNN) and can provide a simpler alternative to LSTM networks. The GRU 312 is capable of processing sequential data, such as text, speech, and time-series data.

[0124] As an example, the GRU 312 can use a gating mechanism to selectively update its hidden layers, such as hidden layers 304A or 304B, at each time step. As an example, in the GRU 312, the output at a specific time step... The input at the corresponding time step And the hidden state of GRU 312 for time steps prior to a specific time step. The function.

[0125] Furthermore, the GRU 312 can have two gating mechanisms: a reset gate 316 and an update gate 318. These gating mechanisms control the flow of information into and out of the GRU 312. As an example, the outputs of the reset gate 316 and the update gate are the current inputs. and hidden state A nonlinear function of weighted summation.

[0126] For example, reset gate 316 determines how much of the previously hidden state should be forgotten. In addition, updating gate 318 determines how many new inputs should be used. This is used to update hidden layer 304 or the hidden state. As an example, the output of these gates is a non-linear function of the weighted sum of the current input and the hidden state. Furthermore, the sequence of operations in the GRU 312 is given by the following equation: (5) (6) (7) (8) in , W and W are learnable weight matrices, while , And learnable bias. Furthermore... This represents element-wise product, and This represents the Sigmoid activation function.

[0127] Therefore, according to this embodiment, the GRU 312 uses a gating mechanism to selectively update the hidden state at each time step, thereby allowing them to effectively model sequential data. In operation, the GRU 312 can use the current input as a vector. and the previously hidden state As input. GRU 312 is then configured to compute values ​​for different gates. For example, for each gate, parameterized current input and previous hidden state vectors are computed by performing element-wise multiplication 314. As an example, the Hadamard product is used to perform element-wise multiplication 314. Furthermore, in the relevant input vector... With the corresponding weights for each gate ,and Element-wise multiplication is performed between them. Furthermore, the corresponding activation function for each gate is applied to the parameterized vector. Element-wise multiplication 314.

[0128] To calculate the current hidden state, based on the current input... Regarding the previous time step The hidden state and learnable parameters of reset gate 316 To calculate the output 320 of reset gate 316, The output 320 will be compared with the previous time step. The hidden states are multiplied by 322 to obtain a parameterized vector representation of the output from the reset gate 316. Then, in the input vector... An element-wise multiplication 314 is performed between the parameterized vector from the reset gate 316 and the parameterized vector. Based on the element-wise multiplication 314, the hidden state vector at the current time step is generated. Adaptive modifications. Afterwards, based on the current input... Hidden states for previous time steps And update the learnable parameters of gate 318 To calculate the output from update gate 318 324. Finally, the output 324 of update gate 318 is added to the previous hidden state vector to determine the current hidden state vector. ,326.

[0129] Overview of Neural Radiation Fields (NeRF)

[0130] Figure 3C An example architecture 330 of a neural radiation field (NeRF) 332 according to some implementations is shown. As an example, the NeRF 332 is a neural network configured to parameterize a signal. This signal may correspond to a single 3D scene or object. To this end, the neural field can embed the scene into the weights of the NeRF 332 to overfit the NeRF to a specific scene.

[0131] Typically, voxel meshes or polygon meshes are used to store 3D scenes. While storing voxel meshes is expensive, polygon meshes may only represent hard surfaces that might not be suitable for certain situations, such as medical scene examination or medical imaging. To overcome the aforementioned challenges associated with storing 3D scenes, NeRF 332 is used. Specifically, NeRF 332 is configured to efficiently and compactly generate and store 3D representations of objects or scenes. The 3D representation is differentiable and continuous, and can have arbitrary dimensions and resolution. Furthermore, NeRF 332 is domain-independent, i.e., it does not depend on the input for each task.

[0132] According to the example implementation, the task of rendering a new perspective requires creating a new view of the scene that has not yet been provided to the system. NeRF can rely on the principle of implicitly representing the 3D geometry and appearance of the scene through a deep neural network 334. Subsequently, NeRF 332 is mathematically represented as follows.

[0133] Given the coordinates of a point in a 3D scene 3 and its viewing direction ( The NeRF 332 is configured to learn continuous 3D functions that can be represented using a deep neural network 334 or an MLP. Deep neural networks 334 can include learnable parameters. Alternatively, learnable parameters can be used. This indicates that it represents the coordinates of the point. Mapped to its color and opacity Then, a deep neural network 334 of NeRF 332 is trained using a backpropagation-based method (such as ADAM).

[0134] Based on the example, 3D functions It can be represented as: (9) It can be noted that NeRF 332 is widely used to solve viewpoint synthesis tasks. As an example, viewpoint synthesis is a task of generating 3D objects or scenes given a set of images from different angles (or perspectives).

[0135] In operation, NeRF 332 is configured to sample the coordinates of the scene and feed the sampled coordinates into a deep neural network 334 to generate a field quantity. In this regard, NeRF 332 is configured to accept a single, continuous 3D coordinate of a point 338 in scene 336 or an object 340 in scene 336, along with two viewpoints as input. The 5D coordinates may include the spatial location of point 338. Viewing direction at point 338 Subsequently, the 3D coordinates of point 338 were sampled. and viewing direction It is fed into a deep neural network 334. As an example, point 338 represents the position of object 340 in the 3D scene 336.

[0136] Furthermore, the deep neural network 334 is configured to output the color intensity corresponding to point 338. and opacity or volume density These output color intensities and opacity or volume density This is called a field quantity. For example, output color intensity. This could include points 338. In addition, opacity Indicates the spatial location passing through point 338 How much radiation (or brightness) is accumulated by the light. For example, opacity. It can provide a measure of the potential impact of point 338 on the rendering of the entire scene 336, and can also indicate the predicted color value. .

[0137] To this end, NeRF 332 is configured to predict field quantities from the desired reconstruction domain (e.g., the desired viewpoint of the problem (e.g., the viewpoint rendering problem or viewpoint rendering task)). The NeRF 332 is then configured to map the reconstruction back to the sensor domain, for example, to generate a 2D RGB image of the scene from a new or desired perspective. In this way, the NeRF 332 can output different representations of the same point 338 when viewed from different angles. As a result, the NeRF 332 can capture various lighting effects, such as reflections and transparency, making it well-suited for rendering the same scene 336 from different perspectives.

[0138] During the training of NeRF 332, NeRF 332 can compute the reconstruction loss and optimize the deep neural network 334. In this regard, typically, the target opacity and color for point 338 are unknown. Therefore, differentiable methods can be used to convert the generated field quantity... The images are then mapped back to 2D images. These images are then compared to ground truth images (e.g., scene images) to represent the rendering loss that the deep neural network 334 was retrained for.

[0139] As an example, volume rendering is used to generate field quantities. A mapping 342 is used to revert to the 2D image. For example, mapping 342 includes a first mapping corresponding to a point on ray 1 and a second mapping corresponding to a point on ray 2. In this way, a mapping or volume rendering is generated based on the field quantity generated for each point in the different rays passing through scene 336 and / or object 340 in the scene. Based on the volume rendering mapping 342, a standard reconstruction loss 344 is calculated for each ray. As an example, the reconstruction loss 344 is calculated using an autoencoder with an input image of scene 336. Based on the reconstruction loss 344, a deep neural network 334 can be trained, retrained, or optimized to generate and render realistic images of scene 336 and / or object 340 from a new perspective.

[0140] An overview of the process of rendering dynamic scenes from a new perspective

[0141] NeRF can also be extended to render dynamic scenes from new perspectives. (See reference) Figure 4 A block diagram 400 illustrates a method for rendering a 2D image 214 of a dynamic scene. As an example, for a given point in a 3D scene 336 (e.g., by x∈R...),... 3 The indicated point 338), its expected viewing direction And the point of interest for rendering point 338. Used to deform point 338 to a new location or coordinates. Based on the new location or coordinates and new perspective in a dynamic scene. and location This involves generating 2D images or frames, such as a 2D image 214 of a dynamic scene viewed from a new perspective. To this end, rendering a dynamic scene involves generating images or frames of the 3D scene as it evolves over time and from different or new angles.

[0142] As an example, dynamic scene rendering is typically performed using a deformable network 402 coupled to a NeRF 404. In this respect, it is represented as... The deformable network 402 is configured to use the coordinates of point 338 based on the time point of interest. Mapped to public normative space , making This can be expressed mathematically as:

[0143] Furthermore, the coordinates of the repositioned point Train NeRF 404 to view from a given viewing direction. Render the repositioned points. After training, NeRF 404 is configured to generate color and opacity values ​​for the repositioned points.

[0144] Among other methods used for rendering dynamic scenes, embedding networks (not shown) can display the current time step number. Encoding to latent space In the middle. Then, the encoding of the time step is fed into NeRF 404 as follows:

[0145] The networks in both methods can be trained via backpropagation. However, since the typical deformable network 402 directly converts points to canonical form, it may have to learn a wide range of motion, including for frames that are temporally distant. This may be infeasible in some cases and affects the output of these models. Furthermore, even with temporal encoding, the task of capturing all kinds of displacements within any time range (short or long) makes the learning task difficult and challenging.

[0146] An overview of an AI image processing system for rendering dynamic scenes from a novel perspective, based on this disclosure.

[0147] According to embodiments of this disclosure, an AI image processing system 102 is provided for performing a process of rendering a 2D image 214 of a dynamic scene based on a 3D scene 202 or a scene image. The system 102 is configured to render the 2D image 214 from new perspectives and angles never captured by the 3D scene 202. As an example, the 2D image 214 may include objects captured from the new angles and perspectives, such as dynamically moving objects in a dynamic scene.

[0148] The embodiments of this disclosure are based on the understanding that the task of deforming a 3D volume can be considered as the deformation of a set of sampled points in 3D space. Furthermore, the deformation of a 3D point (such as point 338) over time can be equivalent to estimating a vector connecting the 3D point 338 at its current position in the current time step to its position in the canonical frame time step. Subsequently, the vector can be decomposed into different segments or fragments using a piecewise linear function, such that each segment represents the displacement of point 338 between two consecutive time steps.

[0149] Figure 5A and Figure 5BA block diagram of the architecture of an AI image processing system 102 according to some embodiments of the present disclosure is shown together. As an example, the AI ​​image processing system 102 includes an RNN 204, an FCNN 206, and a NeRF 108. For this purpose, the RNN 204 and FCNN 206 can implement deformation or translation networks for translating points 338 of a 3D scene 336 or 202. Furthermore, the NeRF 108 can implement a rendering network for rendering translated points 210 in a 2D image 214 of a dynamic scene.

[0150] refer to Figure 5A Example block diagram 500 is shown corresponding to the architecture of the displacement network of system 102 according to some implementations. As described above, the displacement network is implemented using RNN 204 and FCNN 206. For example, RNN 204 is implemented using GRU, and FCNN 206 is implemented using MLP.

[0151] According to embodiments of this disclosure, RNN 204 includes multiple GRUs, depicted as GRU 502A, 502B, ..., 502N (collectively referred to as GRU 502). Similarly, FCNN 206 includes multiple MLPs, depicted as MLP 504A, 504B, ..., 504N (collectively referred to as MLP 504).

[0152] To implement the displacement network of system 102, GRU 502 is implemented to track the displacement history of point 338 in past frames or previous time ranges, such that the future displacement prediction of point 338 conforms to the motion determined so far. Furthermore, the ability of GRU to capture temporal dependencies is utilized by using the same GRU units at each step or on GRUs 502A, 502B, ..., 502N. This also contributes to parameter efficiency, as only the weights of this shared GRU need to be learned during training. Similarly, for parameter efficiency reasons, a single MLP is used to implement MLPs 504A, 504B, ..., 504N.

[0153] The displacement network of system 102 uses MLP to predict the displacement of 3D point 338 at different time steps.

[0154] like Figure 4 As described, a key feature of deformable network 402-based methods for rendering dynamic scenes from a new perspective is that they directly estimate the displacement of 3D points over the entire time span, up to the time step of interest. This presents a challenge because it requires learning a wide range of motion that can be performed by objects in a dynamic scene over long time spans, especially for rendering images or frames over long time ranges.

[0155] In particular, deformable networks Learning diverse motion patterns corresponding to different objects in a scene across different time spans is challenging due to the wide variation in possible outputs for a given scene. For example, consider a point representing the position of a car in the real world. Given a time unit, such as... Depending on the car's movement, the point in the 3D world might move very fast or very slow. On the other hand, if the point represents a location on a tree, it might remain in the same 3D coordinates for any given time period. This wide range of possibilities makes training... It has become difficult.

[0156] Therefore, to overcome the problems associated with training deformation networks, a novel displacement network architecture is provided. The proposed displacement network works by dividing the predicted range of deformation within a point into shorter fixed-length time windows and a remaining time range occupying the remaining time. For example, the fixed-length time window for each time range can be indicated as... And the remaining time range can be represented as Or t modulo Δt in seconds. For example, each time range can form a time step. Subsequently, RNN 204 and FCNN 206 can determine the displacement of the point for each time range or time step.

[0157] In operation, the displacement network predicts the range of data, indicated by the time step t of interest. The time is divided into fixed-length time ranges that indicate the number of time steps. Subsequently, each of the time steps, except for the remaining or last time step, can span [a certain period]. The time range is seconds, and the remaining time step can span the remaining time amount, i.e. .

[0158] As an example, RNN 204 could include a GRU 502 expanded over a fixed number of time steps, enabling prediction based on the range of predictions. The maximum length determines the number of time steps. As an example, the number of times MLP 504 is invoked is determined based on the number of time steps or segments into which the prediction range up to the point of interest is divided.

[0159] For each time step or time range up to the point of interest, recursively execute RNN 204 and FCNN 206. To do this, each recursion... The above operation. It can be noted that if the length of the time range... Greater than the prediction range The maximum length is then determined by recursion, which operates only once in time steps modulo Δt (t modulo Δt). It can be noted that the points of interest are those at which the image is rendered from a new perspective (such as image 214 of a dynamic scene).

[0160] As an example, during the first iteration, GRU 502A and MLP 504A are executed within the first time step to generate the displacement of point 338 after the first time step. Subsequently, after the first iteration, the displacement of the point is determined, indicating how much the point has displaced in the first iteration. The displacement point determined in the first iteration is then used as input for the second iteration. In the second iteration, GRU 502B and MLP 504B are executed within the second time step to generate another displacement of the point after the second time step. This process is repeated until the time point of interest is reached. For this purpose, the displacement determined during the current iteration is provided as input for subsequent iterations until the time point of interest is reached. In this way, the displacement of the point over multiple time steps is generated. In practice, to make the model parameters efficient, the same GRU and MLP models can be used across different time steps.

[0161] For the current iteration, GRU is configured based on the coordinates of the point and the current time range. The motion information 208 of the point is determined. Furthermore, for the current iteration, the MLP is configured to determine or generate the displacement of the point within the current time range based on the motion information 208. As an example, based on the motion information 208, the displacement of the point from its current coordinates to another coordinate is generated. As an example, when the point corresponds to a dynamic object in the scene, the point's current coordinates may differ from the other coordinate. Alternatively, when the point corresponds to a static or stationary object in the scene, the point's current coordinates may not change from the other coordinate or may only change slightly.

[0162] As an example, the displacement of the x-coordinate of the point in the current iteration is generated by the MLP based on the motion information generated by the GRU. In this respect, the system combining GRU and MLP is represented as having a learnable set of parameters. of Subsequently, in the displacement network, the forward propagation through the displacement network for the first time step can be expressed as: (10) in Indicates the coordinates of a point in 3D scene 202, and This refers to the time range of the first time step. Similarly, for subsequent time steps, such as from the second time step up to the penultimate time step (i.e., the time step before the remaining time steps) or the second-to-last time step, the forward propagation through the displacement network can be represented as: (11) Furthermore, for the final time step or remaining time step, the forward propagation through the displacement network can be expressed as: (12) Where k= and

[0163] As an example, in each iteration, the GRU 502 tracks the past displacement of the point. Specifically, the GRU for the current iteration or time step can receive displacements from previous time ranges or iterations. The GRU can generate motion information for the current iteration based on the previous displacements or displacement points displaced based on displacements generated in previous iterations. Subsequently, the MLP for the current iteration can receive the motion information generated by the GRU for the current iteration. Based on the received motion information, the MLP can generate the displacement of the point.

[0164] In this way, GRU and MLP can operate recursively on each iteration to generate displacements for the corresponding iteration. For example, the displacement of a point determined during the current iteration is input to another GRU in subsequent iterations. For example, the displacement or output of a point at the current iteration is a function of the input to the current iteration and the hidden state of GRU 502.

[0165] As an example, in addition to displacement, subsequent iterations of the GRU are also fed its current hidden state. As another example, a nonlinear mapping is used to combine the displacement from the current iteration fed to the GRU for subsequent iterations with the hidden state of the GRU for those subsequent iterations. For example, the nonlinear mapping allows the GRU to capture complex temporal dependencies between the displacements of points captured in previous iterations.

[0166] Once displacements have been generated for different time ranges up to the point of interest, a displacement point, such as displacement point 210, is determined for that point. Displacement point 210 can indicate that the initial point is within the prediction range so as to correspond to the final displacement at the point of interest. As can be understood, GRU 502 and MLP 504 can operate in different iterations for the corresponding time ranges to determine the displacement.

[0167] Displacement point 210 is generated based on the displacement produced by the displacement network at each iteration. As an example, the net displacement or displacement coordinates for a point are determined by utilizing the fact that displacement is a vector and can be determined by piecewise displacements over each iteration for each time range. Based on the net displacement, displacement point 210 is generated. As an example, displacement point 210 is displaced to a new location or coordinates that may differ from the coordinates of the point determined in the previous iteration or the coordinates of the point in the 3D scene.

[0168] Subsequently, once the displacement point 210 is determined, the resulting displacement point 210 is fed into the rendering network. It can be noted that, for example, the displacement network described herein is used to generate the displacement position of each point in the 3D scene 202. The displacement points of the 3D scene 202 are then fed into the rendering network to allow the rendering network to generate new perspectives, images, or frames of the dynamic scene. As an example, the rendering network is implemented using NeRF 108.

[0169] Combining, for example Figure 3C and Figure 5B Describe the details of the rendering network or NeRF.

[0170] refer to Figure 5B An example architecture 510 of a rendering network according to some implementations is shown. As an example, a NeRF 108 is used to implement the rendering network. For example, displacement points 210 are fed into the NeRF to utilize learnable parameters. Forward propagation.

[0171] As an example, in order to render displacement point 210, NeRF 108 is configured to receive a viewpoint of interest 212. For example, the viewpoint of interest 212 could be a desired viewpoint or desired angle for rendering a dynamic scene. As an example, the viewpoint of interest 212 is received from the user of system 102 and / or from upstream processes associated with games, AR / VR displays, video playback, etc.

[0172] Furthermore, based on the received view of interest 212, NeRF 108 is trained to implicitly represent the 3D geometry and appearance of the 3D scene. Additionally, NeRF 108 is trained based on the set of visible parameters of the view of interest 212 and the displacement point 210 to learn learnable parameters. Continuous 3D functions Examples of visible parameters for displacement point 210 may include, but are not limited to, 3D coordinates, surface normals, material properties, lighting, depth information, visibility or occlusion information, color or radiation, opacity or transparency, and / or motion information.

[0173] As an example, NeRF 108 is trained to implicitly represent the 3D geometry of a 3D scene 202 by modeling the volume rendering of the 3D scene 202 as a continuous function. For example, NeRF 108 is trained on the 3D scene 202 (such as an image of the 3D scene 202) to learn the parameters of the volume representation of the 3D scene 202. In this respect, NeRF 108 can learn, for example... 3D spatial coordinates, viewing direction and such as color and opacity value This allows NeRF 108 to map between objects without using meshes or voxels to represent 3D geometry in the traditional sense. This enables NeRF 108 to handle complex and detailed scenes with fine-grained structures.

[0174] As an example, NeRF 108 is configured to define continuum functions. It will use 3D spatial coordinates As input and output radiation values, i.e., color and opacity value As an example, NeRF 108 can use ray parameterization to represent points, such as point 338 or displacement point 210 along a ray passing through scene 336. As an example, NeRF 108 may include a multilayer perceptron (MLP) 508. For example, for displacement point 210 along a ray, the MLP 508 of NeRF 108 is configured to evaluate 3D functions. For example, radiance and opacity values ​​are obtained in a view-dependent manner based on the view of interest 212.

[0175] As an example, MLP 508 is used to model 3D functions of NeRF 108 that map 3D spatial coordinates to radiance and opacity values. For instance, MLP 508 can implement implicit representations of 3D scenes. As can be noted, NeRF 108 aims to estimate continuum functions representing radiance and opacity values ​​for a scene. This is represented as... The 3D function is implemented as MLP 508 in NeRF 108. The input to MLP 508 is the 3D spatial coordinates of points in the scene, specifically the coordinates of displacement point 210. The MLP 508 takes the input and processes it through multiple layers to capture complex nonlinear relationships. For example, an MLP 508 can include several hidden layers, each with a learnable set of weights and biases. The number of layers and the size of each layer are hyperparameters that can be tuned based on the complexity of the scene and the desired model capacity. Furthermore, the MLP 508 can introduce nonlinearity after each layer of the MLP using nonlinear activation functions such as ReLU, hyperbolic tangent (tanh), or sigmoid, enabling it to capture complex patterns in the input data.

[0176] Subsequently, the output layer of the MLP 508 can generate radiation or color values ​​and opacity or transparency values ​​for a given 3D displacement point 210. This output represents the color and opacity of the displacement point 210 in the scene considered from the viewpoint 212 of interest. In some cases, a separate MLP is used to model the color and transparency values.

[0177] As an example, supervised learning is used to train the parameters of MLP 508 on a dataset containing 3D spatial coordinates and corresponding ground truth radiance values. The training process involves minimizing a loss function that measures the difference between the predicted radiance values ​​and the ground truth radiance values. Furthermore, during training and inference, NeRF uses ray stepping to sample displacement points along the rays traversing the scene. At each sampling point, MLP 504 is evaluated to obtain radiance and opacity values, and these values ​​are integrated along the rays to generate the final 2D image 214.

[0178] The determined radiance and opacity values ​​for displacement point 210 can take into account how displacement point 210 might appear from the viewpoint 212 of interest. Furthermore, based on the determined radiance and opacity values, displacement point 210 is rendered as, for example, a portion of a 2D image 214. In this way, a new viewpoint for a dynamic scene, or a 2D image forming a new viewpoint, is generated. The execution system 102 performs the new viewpoint rendering task in an efficient and effective manner to generate a photorealistic 2D image for a dynamic 3D scene.

[0179] Overview of the training system

[0180] Figure 6 An example flowchart 600 is shown, illustrating a method for training system 102 according to some embodiments. (In conjunction with...) Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 5A and Figure 5B explain Figure 6 Components.

[0181] To this end, system 102 includes a translation network implemented using GRU 502 and MLP 504, and a rendering network implemented using NeRF 108 and MLP 508. For example, the learnable parameters of GRU 502 can be derived from... Instructions, and the learnable parameters of the MLP 504 can be derived from... The GRU 502 takes the point to be rendered at the time of interest as input, considers its state information to derive the temporal context, and continues to generate its output, namely motion information 208. Furthermore, the MLP 504 processes the output of the GRU 502 to predict the position of the point (i.e., displacement point 210) after being displaced over a prediction range encompassing different time points up to the time of interest. The displacement point is then fed into either the NeRF 108 or the MLP 508. The learnable parameters of the MLP 504 can be derived from... Instructions. Therefore, the total set of learnable parameters for system 102. Defined by the following formula:

[0182] As an example, backpropagation in time (BPTT) can be used to train the translation network and the rendering network together. The model or network (i.e., RNN 204 or GRU 502, FCNN 206 or MLP 504, and NeRF 108 or MLP 508) is trained to optimize its respective learnable model parameters (i.e., ... This system accurately represents 3D scenes and generates realistic images from different viewpoints. Training the system may include the following steps.

[0183] At position 602, generate the training dataset. As an example, the training dataset is generated based on a dataset containing pairs of 3D scenes captured from various viewpoints and their corresponding images. As another example, the training dataset may be partitioned into at least two sets corresponding to training and testing or validation phases based on viewpoints. The training set will typically contain more than one viewpoint, while the test set will contain viewpoints not included in the training set.

[0184] At position 604, the training dataset is fed into the system for training. In this regard, the training dataset is processed by the translational network and the rendering network, either in combination or independently. For example, BPTT is used to process the training dataset for training all models. Based on this processing, the learnable parameters of each model are defined and / or updated.

[0185] As an example, the training dataset is preprocessed to associate each point in the scene in the training dataset with its corresponding features (such as color, transparency, or other relevant parameters).

[0186] At position 606, a loss function is determined for gradient computation and for validating the learnable weights for each model. As an example, the loss function could measure the difference between a prediction made by the rendered model and the ground truth image. For instance, each model might make predictions against the validation set based on a viewpoint provided in the training dataset. Similarly, such predictions could be made against a viewpoint provided in the training set itself to compute gradients. Examples of loss functions could include, but are not limited to, mean squared error (MSE), perceptual loss, or adversarial loss.

[0187] The loss function is computed only for the image ultimately rendered by System 102 and compared with the ground truth image captured from the viewpoint that performed the prediction. To this end, based on the loss function, the learnable weights or model parameters of each model (i.e., RNN 204, FCNN 206, and NeRF108) are updated to optimize the operation and improve the accuracy of System 102.

[0188] As an example, an optimization function such as stochastic gradient descent (SGD) or adaptive moment estimation (Adam) can be used to minimize a determined loss function. For instance, the optimization function can be manipulated by adjusting the learning rate and other hyperparameters to achieve efficient convergence.

[0189] Overview of the steps for generating 2D images of dynamic scenes from a new perspective

[0190] Figure 7 An example flowchart 700 is shown, illustrating a method for generating images for dynamic scenes according to some embodiments. (Combined with...) Figure 1 , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 5A , Figure 5B and Figure 6 To explain the components Figure 7 Components.

[0191] At position 702, the coordinates of points in a dynamic 3D scene are processed. As an example, the coordinates of a point... The data is fed into the RNN 204 for processing. For example, the RNN 204 can process data from points of interest. The coordinates of points at multiple time steps are specified to generate motion information 208 for points at time points of interest. As an example, this continues until the time point of interest... The time length is divided or segmented into time ranges, where each time range can correspond to a time step. Subsequently, in each iteration (such as the current iteration), the RNN 204 or GRU can process the coordinates of the points to generate motion information for the corresponding time range (i.e., a time step).

[0192] At 704, motion information 208 is processed to generate point displacements from coordinates in the dynamic 3D scene 202. As an example, motion information 208 is processed by FCNN 206 or an MLP. For example, based on motion information determined by GRU for the current iteration, an MLP can generate displacements of points in the scene. As an example, displacements are generated based on coordinates in the 3D scene 202. For example, after the first iteration, the displacement determined by the MLP can indicate the displacement of the point from the input 3D scene 202. For example, in subsequent iterations, the displacement determined by the MLP can indicate the displacement of the point relative to previously determined displacements.

[0193] It can be noted that for subsequent iterations after the current iteration, the displacement of the point determined by FCNN 206 or MLP in the current iteration is fed as input to the RNN 204 or GRU for the subsequent iteration to determine motion information for the subsequent iteration. For example, the output of GRU 502, i.e., the output of GRU or motion information 208 at the current iteration, is a nonlinear function of the weighted sum of the current input and the hidden state at the current iteration. In other words, at the current iteration j, the GRU is configured to be based on the input fed to the GRU in the current iteration (e.g., the coordinate x of the point from the static 3D scene 202 or the coordinates based on the displacement determined by MLP in the previous iteration). And the hidden state of GRU at the current iteration. The output is generated using a non-linear function.

[0194] At point 706, the displacement point is processed from the viewpoint of interest 212. Specifically, once the displacement is determined for each time step or time range spanning the entire length of the prediction range, the net displacement of the point is determined. This is based on the coordinates and net displacement of the point in the 3D scene 202, for example, by the MLP in the last iteration. N Displacement point 210 is generated.

[0195] Subsequently, displacement point 210 is fed into NeRF 108. NeRF 108 can also receive viewpoints of interest 212 and further process displacement point 210 based on viewpoints of interest 212 to render that point on a 2D image 214 of the dynamic 3D scene for the time point of interest. In particular, NeRF 108 is trained for the 3D scene 202.

[0196] Overview of the system's example diagram

[0197] Figure 8 A block diagram 800 of a system 102 according to some embodiments is shown. System 102 may include multiple interfaces for connecting system 102 to other systems and devices. In this regard, a network interface controller (NIC) 802 is configured to connect system 102 to a network 806 via a bus 804, which connects system 103 to a sensing device (not shown). For example, system 102 includes a transmitter interface 808 configured to command transmitter 810 to transmit pulse waves. Using a receiver interface 812 connected to receiver 814, system 102 can receive images or image sequences of 3D scenes 202 or 336, such as dynamic scenes. In some embodiments, system 102 receives information about static 3D scenes 202 via network 806.

[0198] System 102 includes an output interface 818 configured to render a 2D image 214 for a dynamic scene or objects within a dynamic scene. For example, the output interface 818 can display the 2D image 214 on a display device, store the image 214 in a storage medium, and / or transmit the image 214 via a network 806. For example, system 102 can be linked to a display interface via a bus 804, which is adapted to connect system 102 to a display device such as a computer monitor, camera, television, projector, or mobile device. System 102 can also be connected to an application interface adapted to connect the system to devices for performing various tasks.

[0199] In some implementations, system 102 includes an input interface for receiving a 3D scene 202. Examples of input interfaces include a NIC 802, a receiver interface 812, and a human-machine interface (HMI) 820. The HMI 820 within system 102 connects system 102 to a keyboard 822 and a pointing device 824, etc., wherein the pointing device 824 may include a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touchscreen, etc.

[0200] System 102 includes a processor 826 configured to execute stored instructions stored in storage device 828, and a memory 830 storing instructions executable by processor 826. Processor 826 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Memory 830 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Processor 826 may be connected to one or more input and output devices via bus 804.

[0201] The instructions can use RNN 204, FCNN 206, and NeRF 108 to implement methods for rendering new perspectives of scenes (such as dynamic and / or static scenes). The instructions can include trained neural networks, namely RNN 204, FCNN 206, and NeRF 108, used to determine the displacement of points and the new perspective of points for rendering points in image 214.

[0202] The embodiments of the present invention described above can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can execute on any suitable processor or set of processors (whether provided on a single computer or distributed among multiple computers). Such a processor can be implemented as an integrated circuit, with one or more processors within an integrated circuit assembly. However, the processor can be implemented using circuitry of any suitable format.

[0203] Furthermore, embodiments of the present invention can be implemented as methods, examples of which have been provided. Actions performed as part of a method can be ordered in any suitable manner. Therefore, embodiments in which actions are performed in a different order than those shown can be constructed, which may include performing some actions simultaneously, even if they are shown as sequential actions in the illustrative embodiments.

[0204] The above description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the above description of the exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing one or more exemplary embodiments. Various changes to the function and arrangement of the elements are contemplated without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims.

[0205] Specific details are set forth in the foregoing description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary detail. In other cases, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Furthermore, the same reference numerals and names in the various figures indicate the same elements.

[0206] Furthermore, various implementations can be described as processes depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Moreover, the order of operations can be rearranged. A process may terminate upon completion of its operations, but may have additional steps not discussed or included in the diagrams. Furthermore, not all operations in any particular described process may occur in all implementations. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the function may correspond to the function returning to the calling function or the main function.

[0207] Furthermore, implementations of the disclosed subject matter can be carried out, at least partially, manually or automatically. They can be performed, or at least assisted by, using machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, for manual or automatic implementation. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks can be stored in a machine-readable medium. The processor can then perform the necessary tasks.

[0208] The various methods or processes outlined herein can be encoded as software that can be executed on one or more processors employing any of a variety of operating systems or platforms. Furthermore, such software can be written using a variety of suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine. Typically, in various implementations, the functionality of program modules can be combined or distributed as needed.

[0209] The embodiments of this disclosure can be implemented as a method, examples of which have been provided. The actions performed as part of the method can be ordered in any suitable manner. Therefore, embodiments in which actions are performed in a different order than those shown can be constructed, which may include performing some actions simultaneously, even if they are shown as sequential actions in the illustrative embodiments. Furthermore, the use of ordinal terms such as “first” or “second” in the claims to modify the claim elements themselves does not imply any priority, order of precedence, or sequence of one claim element relative to another claim element, or the temporal order of the actions of the method, but is merely used as labels to distinguish one claim element with a specific name from another element with the same name (other than using ordinal terms to distinguish claim elements).

[0210] Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adjustments and modifications can be made within the spirit and scope of this disclosure. Therefore, aspects of the appended claims cover all such variations and modifications that fall within the true spirit and scope of this disclosure.

Claims

1. An artificial intelligence (AI) image processing system, wherein the AI ​​image processing system employs Neural Radiation Field (NeRF) to render two-dimensional 2D images of the 3D scene from different perspectives and at different time points based on an implicit representation of a dynamic 3D scene, the AI ​​image processing system comprising: At least one processor and a memory, wherein instructions are stored in the memory, the instructions causing the at least one processor of the AI ​​image processing system to: A recurrent neural network is used to process the coordinates of points in a dynamic 3D scene at multiple time steps indicated by a time point of interest, in order to generate motion information of the points at the time points of interest. The motion information is processed using a fully connected neural network to generate the displacement of the point from the coordinates in the dynamic 3D scene; and The NeRF trained on a static 3D scene is used to process displacement points from a viewpoint of interest to render the points on the 2D image of the dynamic 3D scene at the time of interest, wherein the displacement points are generated based on the displacement of the points.

2. The AI image processing system of claim 1, wherein, The recurrent neural network and the fully connected neural network are recursively executed for different time points until the time point of interest is reached, such that the displacement of the point determined during the current iteration is the input for subsequent iterations. 3.The AI image processing system of claim 1, wherein, The recurrent neural network includes gated recurrent units (GRUs), and the execution of each GRU corresponds to the movement of the point at a time step.

4. The AI ​​image processing system according to claim 3, wherein, The recurrent neural network includes a fixed number of iterations of the gated recurrent unit (GRU) that defines the maximum length of the prediction range, and The recurrent neural network and the fully connected neural network are recursively executed for different time points until the time point of interest is reached, such that the displacement of the point determined during the current iteration is the input for subsequent iterations.

5. The AI image processing system of claim 4, wherein, Each recursion operates over a time range of the maximum length, which is no greater than the predicted range.

6. The AI image processing system of claim 4, wherein, The displacement of the point determined during the current iteration is input into the subsequent iteration and combined with the hidden state at the subsequent iteration using a nonlinear mapping.

7. The AI image processing system of claim 4, wherein, The output at the current iteration is a function of the current input and hidden state of the GRU. 8.The AI image processing system of claim 4, wherein, The output of the GRU at the current iteration is a nonlinear function of the weighted sum of the current input and the hidden state at the current iteration. 9.The AI image processing system of claim 1, wherein, The NeRF was trained as follows: Implicitly representing the 3D geometry and appearance of the 3D scene; and Based on the viewpoint of interest and the coordinates of the displacement point, a continuous 3D function with learnable parameters is learned. 10.The AI image processing system of claim 1, wherein, The NeRF was trained using backpropagation time-backpropagation (BPTT). 11.The AI image processing system of claim 1, wherein, The recurrent neural network RNN ​​is a bidirectional RNN. 12.The AI image processing system of claim 1, wherein, Rendering the points on the 2D image of the dynamic 3D scene generates a new view of the 3D scene.

13. The AI ​​image processing system according to claim 1, wherein, The point represents the position of the object in the 3D scene.

14. A method for image processing based on artificial intelligence (AI), the method being implemented by an AI image processing system, the AI ​​image processing system employing Neural Radiation Field (NeRF) to render two-dimensional 2D images of the 3D scene from different viewpoints and at different time points based on an implicit representation of a dynamic 3D scene, the method comprising the following steps: A recurrent neural network is used to process the coordinates of points in a dynamic 3D scene at multiple time steps indicated by a time point of interest, in order to generate motion information of the points at the time points of interest. The motion information is processed using a fully connected neural network to generate the displacement of the point from the coordinates in the dynamic 3D scene; as well as The NeRF trained on a static 3D scene is used to process displacement points from a viewpoint of interest to render the points on the 2D image of the dynamic 3D scene at the time of interest, wherein the displacement points are generated based on the displacement of the points.

15. The method according to claim 14, wherein, The recurrent neural network and the fully connected neural network are recursively executed for different time points until the time point of interest is reached, such that the displacement of the point determined during the current iteration is the input for subsequent iterations.

16. The method of claim 14, wherein, The recurrent neural network includes gated recurrent units (GRUs), and the execution of each GRU corresponds to the movement of the point at a time step.

17. The method according to claim 16, wherein, The recurrent neural network includes a fixed number of iterations of the gated recurrent unit (GRU) that defines the maximum length of the prediction range, and The recurrent neural network and the fully connected neural network are recursively executed for different time points until the time point of interest is reached, such that the displacement of the point determined during the current iteration is the input for subsequent iterations.

18. The method according to claim 17, wherein, Each recursion operates over a time range of the maximum length, which is no greater than the predicted range.

19. The method of claim 14, wherein, Rendering the points on the 2D image of the dynamic 3D scene generates a new view of the 3D scene.

20. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium contains a program executable by a processor to perform a method, the method comprising the steps of: A recurrent neural network is used to process the coordinates of points in a dynamic 3D scene at multiple time steps indicated by a time point of interest to generate motion information of the points at the time point of interest. Neural radiation field (NeRF) is used to render two-dimensional 2D images of the 3D scene from different perspectives and at different time points based on an implicit representation of the dynamic 3D scene. The motion information is processed using a fully connected neural network to generate the displacement of the point from the coordinates in the dynamic 3D scene; and The NeRF trained on a static 3D scene is used to process displacement points from a viewpoint of interest to render the points on the 2D image of the dynamic 3D scene at the time of interest, wherein the displacement points are generated based on the displacement of the points.

Images