Dynamic (4D) scene reconstruction using multiple neural radiance fields

Abstract

Systems, methods, and software are disclosed that generally improve computer vision techniques, particularly 4D scene reconstruction. An artificial intelligence (AI) image processing system employs multiple dynamic neural radiance fields (NeRFs) to render two-dimensional (2D) images of a four-dimensional (4D) scene from different viewpoints and different time instances. The AI image processing system collects observation parameters for rendering the 2D images of the 4D scene, e.g., from a desired observation direction and at a desired time instance. The system then executes multiple dynamic NeRFs to obtain color and volume density values of voxels associated with pixels in the 2D images based on the observation parameters. The system then renders the 2D images based on a combination of the color and volume density values obtained from the multiple dynamic NeRFs.

Description

TECHNICAL FIELD

[0001] Aspects of the present disclosure relate to the field of computer vision, and in particular, to rendering views of dynamic scenes using neural networks. BACKGROUND

[0002] Neural Radiance Fields (or NeRF) are neural networks trained on a sparse set of two-dimensional (2D) images of a three-dimensional (3D) scene to provide new views of the 3D scene. NeRF represents the 3D scene as a continuous function that maps 3D coordinates to color and density values. The core idea of these methods is to implicitly represent the 3D geometry and appearance of a scene through a deep neural network that learns a mapping of points in 3D space and a mapping of viewing directions to color and density values.

[0003] Training a NeRF typically involves projecting a ray from an image pixel into the scene and inputting the 3D coordinates of the voxels along the ray to the neural network. The network outputs a respective color and density value for each voxel. The color and density values of the voxels along the ray are used to compute a predicted color and density for the image pixel. A loss function evaluates the predicted values against known values for the image pixel and updates the network parameters accordingly.

[0004] After training, the NeRF can be integrated into a rendering pipeline to predict the color and density values of voxels along a ray projected from a pixel in a desired view of the scene. The predicted color and density values are processed to determine the color and density values of the pixel. The same steps are performed for all pixels in the desired view to produce a synthesized image. Ideally, the NeRF would be trained on every point in the scene. However, if every possible pixel and every possible voxel for that pixel were sampled, this high-resolution sampling would result in too many ground truths needed for training.

[0005] Recently, some research has extended these NeRF-based methods to 4D scenes in which objects in a 3D scene can change in configuration, position, etc. However, existing methods fail to consider the effectiveness of the space-time sampling steps involved in rendering such scenes, which compromises the quality of the rendering and reduces efficiency. SUMMARY

[0006] Disclosed herein are systems, methods, and software that generally improve computer vision techniques, particularly 4D scene reconstruction. In various implementations, an artificial intelligence (AI) image processing system employs multiple dynamic neural radiance fields (NeRFs) to render two-dimensional (2D) images of a four-dimensional (4D) scene from different viewpoints at different times. The NeRFs implicitly represent the 4D scene, and the 4D itself is a 3D scene that changes over time.

[0007] An AI image processing system collects observation parameters for rendering a two-dimensional (2D) image of a 4D scene, e.g., from a desired observation direction and at a desired time instant. The system then performs multiple dynamic NeRFs to obtain color and volume density values of voxels associated with pixels in the 2D image based on the observation parameters. The system then renders the 2D image based on a combination of the color and volume density values obtained from the multiple dynamic NeRFs.

[0008] In some implementations, the dynamic NeRFs can be trained on the 4D scene at different spatial-temporal sampling resolutions from each other. Additionally, or alternatively, training the NeRFs at different spatial-temporal resolutions can be achieved by leveraging semantic information about the 4D scene. The semantic information can be used to segment the 4D scene into regions, e.g., based on the relative motion intensity of the regions, which can then be assigned to specific NeRFs. For example, one NeRF can be trained on one (or more) regions with less motion relative to another region or regions at one spatial-temporal sampling resolution. A second NeRF can then be trained on other region(s) of the 4D scene at a different (e.g., higher) spatial-temporal sampling resolution. Additionally, or alternatively, one or more of the NeRFs can also be trained on semantic embeddings of the 4D scene, thereby providing almost cost-free object tracking in the 4D scene.

[0009] This Summary is intended to introduce in simplified form a series of concepts that are further described below in the technical disclosure. It is not the intention to define or frame the claimed subject matter by this Summary. This Summary is not intended to be used to limit the scope of the claimed subject matter.

[0010] Many aspects of the disclosure can be better understood with reference to the following drawings. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. Furthermore, in the drawings, like reference numerals designate corresponding parts throughout the several views. While several embodiments are described in connection with these drawings, the disclosure is not limited to the embodiments disclosed herein. On the contrary, it is intended to cover all alternatives, modifications, and equivalents. BRIEF DESCRIPTION OF DRAWINGS

[0011] [ Figure 1 ]

[0012] Figure 1 An architecture of a computer vision system in an implementation is shown.

[0013] [ Figure 2 ]

[0014] Figure 2 A training process in an implementation is shown.

[0015] [ Figure 3 ]

[0016] Figure 3 A rendering process in an implementation is shown.

[0017] [ Figure 4A ]

[0018] Figure 4A A training pipeline in an implementation is shown.

[0019] [ Figure 4B ]

[0020] Figure 4B A run scheme in an implementation regarding training is shown.

[0021] [ Figure 5 ]

[0022] Figure 5 A rendering pipeline in an implementation.

[0023] [ Figure 6A ]

[0024] Figure 6A A run scheme in an implementation regarding training and rendering modes is shown.

[0025] [ Figure 6B ]

[0026] Figure 6B A run scheme in an implementation regarding training and rendering modes is shown.

[0027] [ Figure 6C ]

[0028] Figure 6C A run scheme in an implementation regarding training and rendering modes is shown.

[0029] [ Figure 6D ]

[0030] Figure 6D A run scheme in an implementation regarding training and rendering modes is shown.

[0031] [ Figure 6E ]

[0032] Figure 6E A run scheme in an implementation regarding training and rendering modes is shown.

[0033] [ Figure 6F ]

[0034] Figure 6F A run scheme in an implementation regarding training and rendering modes is shown.

[0035] [ Figure 6G ]

[0036] Figure 6G Operation schemes in implementations regarding training and rendering modes are shown.

[0037] [ Figure 6H ]

[0038] Figure 6H Operation schemes in implementations regarding training and rendering modes are shown.

[0039] [ Figure 7 ]

[0040] Figure 7 Operation framework in implementations is shown.

[0041] [ Figure 8 ]

[0042] Figure 8 Computing systems suitable for implementing various operating environments, architectures, processes, schemes, sequences, and frameworks discussed below in connection with other figures are shown. DETAILED DESCRIPTION

[0043] Reconstructing 3D scenes has a wide range of applications, including virtual reality / augmented reality (VR / AR), 3D animation, game production, and movie creation, allowing users to observe the scene from any desired perspective. While reconstructing static 3D scenes is crucial and has made significant progress, reconstructing dynamic 3D scenes (or 4D as referred to herein) is even more important because the world around us is usually characterized by a constant flow of state, with many objects in motion.

[0044] To this end, an improved artificial intelligence (AI) image processing system is disclosed herein that enhances dynamic novel view synthesis through smarter space-time sampling, which in some implementations allows for improved tracking of objects of interest in the novel view. A brand new gear concept allows for hierarchical modeling of dynamic regions of a 3D scene according to their degree of motion. This differentiation allows the space-time sampling resolution of individual regions to be adjusted according to their motion scale, resulting in more realistic, dynamic novel view synthesis. In some implementations, the new method is able to track the viewpoint for objects of interest with very low additional computational cost.

[0045] In particular, the image processing system employs multiple dynamic NeRFs to render 2D images of a 4D scene from different viewpoints and different time instances. The NeRFs collectively perceive the distinction between static and dynamic regions in the scene (and / or the difference between regions of the scene that are less dynamic and regions that are more dynamic). As a result, the NeRFs can represent regions that are relatively more dynamic more faithfully than other regions, which inherently requires more resources per unit due to their temporal evolution nature. The regions can be further stratified according to their motion scales, and the samples are allocated accordingly.

[0046] In some implementations, the disclosed AI image processing system follows a framework that models a 4D scene hierarchically with semantic embeddings of image segmentation models. The system is optimized for 4D semantic embeddings, and based on that, the system determines appropriate region-specific spatio-temporal sampling resolutions in the NeRFs. Regions with larger motion scales are allocated higher spatio-temporal sampling resolutions and are applied differentially to the NeRFs. The NeRFs can be trained on regions that are all, some, or none different. In general, each NeRF is trained based on a combination of region and spatio-temporal sampling resolution that is different from the combinations of regions and resolutions of other NeRFs.

[0047] With respect to this motion-aware sampling strategy, various technical effects can be achieved and embodied, including improved quality of synthesized images. In addition, or as an alternative, viewpoint object tracking can also be supported and implemented through the semantic embedding representation. Empirical studies validate the effectiveness of the disclosed techniques, achieving state-of-the-art rendering and tracking performance on multiple challenging datasets.

[0048] Turning now to the drawings, Figure 1 An architecture of a computational vision system in an implementation is shown represented by system architecture 100. System architecture 100 includes training pipeline 101, rendering pipeline 105, and NeRFs 110 (including NeRF 111, NeRF 113, and NeRF 115). Training pipeline 101 represents a computational hardware, software, and / or firmware system that takes 4D video data as input and trains an artificial neural network (e.g., NeRFs 110) on the data. Rendering pipeline 105 represents a computational hardware, software, and / or firmware that takes observation parameters as input, interfaces with a neural network (e.g., NeRFs 110) to obtain voxel color and density values, and produces 2D images as output.

[0049] It will be appreciated that while shown together, the actual implementation of the system architecture 100 can implement the training pipeline 101 and the rendering pipeline 105 separately from one another. For example, the training pipeline 101 can be implemented on a computing device configured for training artificial neural networks, while the rendering pipeline 105 can be implemented in a different computing device. In some cases, however, the training pipeline 101 can be implemented on the same computing device as the rendering pipeline 105. For example, the rendering pipeline 105 can be deployed to a runtime environment, such as a personal computer, a mobile phone, an embedded computing system, a server computer, etc., in a collocated and / or cooperative computing manner with the training pipeline 101.

[0050] Figure 2 A training process 200 using the training pipeline 101 in the context of training multiple NeRFs is briefly illustrated. The training process 200 can be implemented in program instructions in the context of the software and / or firmware elements of the training pipeline 101. When executed by one or more processing devices of one or more computing systems (e.g., the computing device 801 in Figure 8 Figure 2

[0051] In operation, the computing device initializes a region to be sampled and a sampling level or “gear” at which to sample the region (step 201). The sampling gear represents a stage, level, or other indicator for distinguishing one spatial-temporal sampling resolution from another. The region to be sampled is composed of 4D space-time coordinates, which can be conceptualized in relation to which pixels in a 2D video frame provide ground truth for training. At the outset, the 4D region can relate to all pixels in all frames, while in training, the region to be sampled focuses more on points in the 4D scene that relate to pixels with greater motion intensity (see step 207 below). The four dimensions of the 4D scene include the x, y, and z dimensions, as well as time, t.

[0052] The computing device continues to sample points in the current region of the 4D scene at a spatial-temporal sampling resolution corresponding to the current gear (step 203). The spatial-temporal sampling resolution includes two components: a spatial component, which defines the granularity of the sampled points in the 4D scene in their x-y-z locations; and a temporal component, which characterizes the granularity of the sampled points in the video frames or t dimension.

[0053] ​​Taking a single pixel as an example, the pixel is represented in a plurality of consecutive video frames captured by the camera from a particular viewing direction. The 4D points in the scene that correspond to the pixel are the points that fall along the ray cast from the pixel at the plurality of times. The temporal component of the space-time sampling resolution relates to the temporal component of each 4D point associated with the pixel, while the spatial component relates to the x-y-z components of each 4D point associated with the pixel.

[0054] By keeping the temporal component constant and limiting to a single time instant, the spatial component can be better understood. At full spatial resolution at a single time point, all 3D points on the ray cast from the given pixel at the single time instant will be sampled; at half resolution, half of the 3D points will be sampled; at quarter resolution, a quarter of the points will be sampled; and so on for the rest of the temporal sampling resolutions.

[0055] By keeping the spatial component constant, the temporal component can be better understood. As mentioned above, a spatial component of 1 / 2 will result in half of the 3D points on the ray cast from the pixel at a given time instant being sampled. However, assuming full temporal resolution, the same 3D points are sampled for each possible time instant (i.e. for each frame of the captured video). Halving the temporal resolution will result in the set of 3D points being sampled on only half the frames; halving the temporal resolution again (to a quarter) will result in the set of 3D points being sampled on only a quarter of the frames; and so on for the rest of the temporal sampling resolutions.

[0056] Combining the two concepts further illustrates the technique. A 1 / 2 spatial sampling resolution and a 1 / 2 temporal sampling resolution for a given pixel will result in half of the 3D points on the ray cast from the pixel being sampled in half of the video frames (assuming a uniform spacing in both spatial and temporal resolutions, although such an assumption is not required). A 1 / 4 spatial sampling resolution and a 1 / 4 temporal sampling resolution for a given pixel will result in a quarter of the 3D points on the ray cast from the pixel being sampled in a quarter of the video frames (again assuming the same uniform spacing, but not necessarily).

[0057] The sampled points are fed forward through one of the corresponding NeRFs (step 205). For each point, the input to the NeRF is a vector representation of the 4D coordinates of the point, along with the viewing direction. Each feature vector is an m-dimensional feature vector of each point sampled in the 4D scene at the current space-time sampling resolution. The m-dimensionality of the feature vector allows for each combination of x, y, z, and t coordinates to be uniquely encoded or embedded into the values that make up the feature vector. For each sampled 4D point, the input to the NeRF includes the feature vector for the point and the viewing direction d.

[0058] The output produced by the NeRF includes predicted color values (c) and predicted volume density levels (r) for each input combination of feature vectors and viewing directions. The output of the NeRF also includes semantic embeddings (s) for each input combination. The predicted pixel values are computed based on the predicted color and volume density levels for the points (or voxels) associated with a given pixel. For example, for a pixel in a frame at time t, the color of the pixel is computed based on the color and volume density values of the 3D points output by the NeRF along the rays that pass through the pixel and project into the scene at time t. The computing device also computes the semantic features for a given pixel in a similar manner to how it computes the pixel values.

[0059] The loss function compares the predicted output to the ground truth and updates the parameters of the NeRF accordingly. The predicted pixel values are compared to the ground truth pixel values in the 2D frames of the captured video data, while the predicted semantic features are compared to the ground truth 2D semantic feature maps corresponding to the respective 2D frames. The training of the first NeRF eventually stops or is otherwise paused, at which point the computing device examines one or more images generated based on the output of the NeRF to identify sub-regions of the sampled region of the 4D scene that have poor quality (step 207).

[0060] The quality of the NeRF is examined by using the NeRF to generate 2D images at multiple time steps from multiple viewpoints (or viewing directions) and comparing them to the ground truth 2D images of the video frames. In addition, corresponding feature maps are produced for the 2D images and compared to the ground truth 2D feature maps. The differences between the predicted red-green-blue (RGB) values in the predicted 2D images and the ground truth images, as well as the differences between the predicted semantic features and the ground truth semantic features, contribute to the quality assessment.

[0061] The computing device computes a 2D rendering loss map based on the RGB differences and the semantic differences, which is then analyzed to find the center coordinates of the blocks of images that have poor quality. Together with the ground truth 2D images, the center coordinates are fed into a segmentation decoder to generate 2D upshift masks. The upshift masks generally correspond to sub-regions in the current region that have poor quality (i.e., cannot be satisfactorily rendered using the current spatial-temporal sampling resolution). It will be appreciated that multiple upshift masks are produced, each at a different viewpoint and time.

[0062] The computing device then proceeds to determine whether the process has reached the last gear (step 209). If the last gear is reached, the process ends. Otherwise, the computing device proceeds to select the next region or regions for sampling (step 211). The upshift masks generated above are used to define the next region or regions. That is, since the upshift masks are 2D, they are used to filter the pixels in the respective frames that are used as ground truth for the next round of training. Overall, the 2D upshift masks across multiple time instants result in the formation of 4D motion regions during sampling.

[0063] Take the example of an object in a 3D scene captured from multiple directions by multiple cameras. Each camera records multiple frames over multiple time instants, the pixels of each frame forming a 2D image. Assume for the sake of example that a first round of training of a first NeRF at a first spatio-temporal sampling resolution results in poor quality of RGB and semantic features corresponding to the object. The poor quality will result in a downshift mask being created from the viewpoints provided by at least some of the cameras and many of the frames. The downshift mask is then applied to the 2D images at the next gear to determine on which pixels to train.

[0064] As mentioned above, sampling 4D pixels involves projecting rays from the training pixels and sampling 3D points at different time stamps along the rays that pass through these pixels. Applying a downshift mask to a given training image eliminates some of the pixels in the training set, leaving only the pixels related to the object in this example. Consider a subsequent frame to further illustrate this point, the object in each subsequent frame will occupy a different set of pixels relative to the previous frame. However, assume for the sake of example that the low quality rendering is performed, a downshift mask is also created for the viewpoints and time instants corresponding to these frames. Overall, the role of the downshift mask is to define a focal region that can be sampled at a higher spatio-temporal resolution than the previous region.

[0065] After identifying the next region or regions to sample, the computing device increments the gear (step 213) and returns to step 203. At step 203, the computing device samples the 4D points associated with the new region at a next spatio-temporal sampling resolution associated with the next NeRF. The next spatio-temporal sampling resolution can be higher than the previous resolution in terms of both spatial and temporal components, although these increases need not be proportional to each other. Furthermore, or instead, the boundaries of the next region to sample can be smaller than the previous region. For example, while the boundaries of the initial region can have encompassed the entire 4D scene, subsequent regions can be limited to a small fraction of the 4D scene. The training process 200 continues until no gears remain, at which point the set of NeRFs can be deployed into a runtime environment, the rendering process 300 leveraging their motion-aware properties to construct new views of the 4D scene.

[0066] Figure 3 A rendering 300 employed in an implementation using the rendering pipeline 105 is briefly illustrated. The rendering process 300 can be implemented in program instructions in the context of software and / or firmware elements of the rendering pipeline 105. When executed by one or more processing devices (e.g., computing device 801 in FIG. 1) of one or more computing systems, the program instructions direct the one or more computing systems to operate as follows, for clarity referred to in brackets Figure 8 Figure 3 ​the steps in FIG. 1 and refer to the computing device in the singular.

[0067] In operation, the computing device collects observation parameters including a desired viewpoint (or direction, defined in terms of azimuth and inclination angles) and a desired time instant (step 301) to render a new view of the 4D scene. Optionally, the observation parameters can also include an object of interest. In this case, the user can indicate the object of interest by selecting the object in the user interface, for example by clicking on the object in an initial 2D image of the scene.

[0068] The new view consists of a grid of pixels that together form a 2D image. To render the 2D image, the computing device queries the plurality of NeRFs to obtain the color and volume density of the voxels associated with the pixels. The color and density values of the respective sets of voxels associated with the respective pixels are used to compute the color (or radiance) and density values of the voxels. The computing device iterates through the 2D image pixel by pixel (although actual implementations can employ parallel processing). Thus, the computing device identifies a next pixel (step 303) and, for that pixel, identifies a next voxel in the set of voxels associated with the pixel (step 305).

[0069] For the identified voxel, the computing device selects one of the NeRFs from the set of NeRFs to query for the color and volume density values (c, r) and, optionally, the semantic feature value (s) of the voxel (step 307). The selection of the NeRF is based on the 4D coordinates of the voxel, which correspond to a particular region in the 4D scene for which the selected NeRF was trained at the NeRF-specific spatio-temporal resolution. The selection of the NeRF can be implemented by, for example, looking up the NeRF based on the 4D coordinates of the voxel using a function mapping of regions to NeRFs. The function mapping can be implemented by a computational function, a lookup table, etc.

[0070] After identifying the appropriate NeRF, the computing device executes the selected NeRF to obtain c, r, and (optionally) s for the current voxel (step 309). After obtaining these values for the voxel, the computing device determines whether there are any voxels remaining in the set of voxels associated with the pixel (step 311). If so, the computing device repeats steps 305 through 309 until there are no voxels for the current pixel. Once there are no voxels remaining, the computing device determines whether there are any more pixels in the target image that need to be computed (step 313). If so, the computing device repeats steps 303 through 311 until there are no more pixels remaining, at which point the computing device can render the 2D image based on the color and density values obtained for the respective sets of voxels associated with the respective pixels in the image.

[0071] In the alternative where the observation parameter indicates an object of interest, the rendering process 300 includes steps 321-329, which describe a sub-process that allows for object tracking through the 4D scene. At step 321, the computing device identifies the 2D location of the object of interest in the initial 2D image of the 4D scene. For example, the user can click on a point in the initial image, thereby defining the x-y coordinates of the selected point. Next, the computing device determines the corresponding 3D location of the object in the scene based on the 2D coordinates of the object in the image (step 323). From there, the computing device maps the 3D scene location to an x-y location in the new image being created (step 325).

[0072] At this point, the computing device is able to obtain an object mask for the object of interest using the semantic feature values (s) output by the NeRF for each voxel (step 327). Specifically, the computing device generates a 2D feature map based on the feature values and submits the feature map to a semantic decoder along with the x-y coordinates in the new image. The semantic decoder generates an object mask based on the feature map and the center point represented by the x-y coordinates. The object mask can then be applied to the new 2D image (step 329), which can occur before the 2D image is rendered in step 315 or after the 2D is rendered in step 315.

[0073] Figure 4A A training pipeline 400 in implementation is shown, Figure 4B A run scheme 450 is shown with respect to the training pipeline 400. The training pipeline 400 represents computing hardware, software, and / or firmware components that are able to train multiple NeRFs on 4D scene data 410 at dynamic spatio-temporal sampling resolutions to generate new views of the scene. The training pipeline 400 includes a controller 401, a vector engine 403, a semantic engine 405, a rendering function 407, a loss function 409, and NeRFs 420. The NeRFs 420 include a NeRF 421, a NeRF 423, and a NeRF 425.

[0074] Overall, the controller 401 directs two processes that occur in an alternating fashion: 1) a radiance field optimization process (training the NeRFs); and 2) a gear assignment update. These two processes are interrelated in that the gear assignment determines which NeRFs are trained and on which feature volumes. The information flow associated with the radiance field optimization is shown by solid lines connecting the elements of the training pipeline 400, while the steps specific to the gear assignment update are shown by dashed lines.

[0075] With respect to the radiation field optimization process, the controller 401 provides the vector engine 403 with the upshift mask (or none initially) and the gear assignments. The vector engine 403 generates a characteristic volume for each gear. For example, the vector engine 403 generates characteristic volume 411 for the first gear assignment, characteristic volume 413 for the second gear assignment, and characteristic volume 415 for the third gear assignment.

[0076] The NeRFs 420 take as input the feature vectors supplied by the characteristic volumes 411, 413, and 415, as well as the direction input. In addition, each NeRF is trained on a respective one of the characteristic volumes: NeRF 421 is trained on characteristic volume 411; NeRF 423 is trained on characteristic volume 413; and NeRF 425 is trained on characteristic volume 415. Thus, since each characteristic volume represents a different spatial-temporal sampling resolution, each of the NeRFs 420 is trained at a different spatial-temporal sampling resolution than each of the other NeRFs 420.

[0077] The NeRFs 420 output, for each input combination of a feature vector (f) and a direction (d), a color (c), a volume density (r), and a semantic feature (s). The rendering function 407 computes pixel values based on the color and volume density values of the voxels associated with the pixels. The rendering function 407 also computes the semantic features of the pixels based on the semantic feature values output by the NeRFs 420 for the associated voxels.

[0078] The loss function 409 evaluates the predicted pixel values against the ground truth pixel values provided by the 4D scene data 410. The loss function 409 also evaluates the predicted semantic values against the ground truth semantic values provided by the semantic engine 405. The loss function 409 updates the parameters of the NeRFs 420 based on the comparison results.

[0079] With respect to the gear assignment update process, the controller 401 determines the gear assignments based on the ground truth 2D images in the 4D scene data 410, the 2D feature maps produced by the semantic engine 405, and the test output images produced by the rendering function 407.

[0080] More specifically, while the radiation field optimization process is paused, the controller 401 provides the NeRFs 420 with test inputs in the form of test feature vectors and test direction inputs for multiple viewpoints corresponding to multiple times, for which the ground truth is known from the 4D scene data 410. The NeRFs 420 produce test outputs, which are processed by the rendering function 407 to generate test output images and feature maps at different viewpoints and different times.

[0081] The controller 401 generates a render loss map for each of the 2D test images and test feature maps based on a comparison of the test images and test feature maps to ground truth 2D images and ground truth feature maps. Those test images and test features that are of insufficient quality are used to generate a shift-up mask for the next gear (assuming the next gear is available). The controller provides the new shift-up mask to the vector engine 403 along with the updated gear assignment and restarts the radiance field optimization process.

[0082] The restarted radiance field optimization process trains the next NeRF on the next feature volume according to the new shift-up mask and gear assignment. For example, if the first round of radiance field optimization trains a NeRF 421 on feature volume 411 (and thus, at a first spatial-temporal sampling resolution), the next round of radiance field optimization trains a NeRF 423 on feature volume 413 (and thus, at a second spatial-temporal sampling resolution). In addition, the new shift-up mask is used to concentrate the sampling in areas of the 4D scene that are of lower quality as determined earlier by the controller 401, and the vector engine 403 generates feature vectors for the feature volume 413 using the new shift-up mask.

[0083] The run scheme 450 illustrates a sequence of operations to better illustrate the above-described radiance field optimization process and the gear assignment update process. The upper half of the run scheme 450 relates to the radiance field optimization, while the lower half relates to the gear assignment update.

[0084] In operation, the controller 401 identifies the current gear n, where n is a gear number from 1 to k. Next, the controller 401 determines which n-region(s) to sample. In the initial phase, sampling can be performed on the entire 4D scene, and in subsequent phases, the regions can be based on the shift-up masks computed during the gear assignment update process. The controller 401 passes the shift-up mask, values derived from the shift-up mask, or other such information to the vector engine 403 to enable the vector engine 403 to generate a feature volume at a particular spatial-temporal resolution that is tailored to the current gear. Accordingly, the controller 401 also identifies the current gear to the vector engine 403. The vector engine 403 selects 4D points based on the shift-up mask and the spatial-temporal sampling resolution and generates feature vectors corresponding to the selected 4D points. The feature vectors and directions are input to one of the selected NeRFs 420 corresponding to the current gear.

[0085] The selected NeRF processes each input to output predicted color, volume density, and semantic values for each 4D point and direction combination. The rendering function 407 computes predicted color, volume density, and semantic values for each pixel based on the values obtained for the voxels of the pixel. The loss function 409 compares the predicted results with ground truth pixel values supplied in the 4D scene data 410 and semantic feature values provided by the semantic engine 405, respectively. Finally, the loss function 409 updates the parameters of the selected NeRF based on the comparison results.

[0086] The gear assignment update process is next performed, in which the controller 401 submits input values to the currently designated NeRF under training to test the NeRF’s ability to accurately reconstruct 2D images from known viewpoints and timestamps. The test input values include, for example, various 4D point and direction inputs associated with pixels in the test images being constructed. The controller 401 uses the output of the NeRF to construct not only predicted 2D images, but also predicted 2D feature maps. The controller 401 also obtains ground truth images and feature maps from the 4D scene data 410 and the semantic engine 405, respectively.

[0087] The controller 401 generates a rendering loss map based on the differences between the predicted images and feature maps and the ground truth images and feature maps. Next, the controller 401 submits the loss map to the semantic engine 405, which is able to generate a gear up mask corresponding to low quality areas in the loss map. The controller 401 utilizes the gear up mask during the next round of radiance field optimization, which continues with the next gear assignment, and so on until no more gears are available and all NeRFs have been trained.

[0088] The trained NeRFs can be deployed in an environment of a rendering pipeline, Figure 5 An example of which is provided. The rendering pipeline 500 represents a computational hardware, software, and / or firmware assembly capable of constructing new 2D images for desired viewpoints at desired times. The rendering pipeline 500 includes the controller 401, the NeRFs 520, the routing function 503, the mask function 505 (optional), and the pixel function 507. The NeRFs 520 include the NeRF 521, the NeRF 523, and the NeRF 525, which represent neural radiance fields trained at mutually different spatial-temporal sampling resolutions, respectively, as described previously.

[0089] In operation, the controller 501 receives as input observation parameters. The observation parameters include a viewpoint (direction), a desired time, and a desired 2D location of an object of interest in a current viewing frame. The controller 501 queries the NeRFs 520 for color, volume density, and semantic feature value predictions of voxels associated with pixels in a 2D image being constructed. (It will be appreciated that the object tracking aspects of the rendering pipeline 500 are non-limiting, optional features of the pipeline.)

[0090] More specifically, for each pixel, the controller 501 identifies a set of voxels in 4D space-time corresponding to the pixel. The controller 501 then computes a feature vector for each voxel based on the 4D coordinates and time stamp of the voxel. For each voxel, the controller 501 selects one of the associated NeRFs 520 and passes the computed feature vector and direction component as input to the selected NeRF. The selected NeRF outputs a color, volume density, and semantic feature value for the voxel.

[0091] The association between NeRFs and voxels can be predefined or predetermined by mapping 4D space-time coordinates to NeRFs. That is, which region or regions each NeRF is trained on can be persisted or otherwise defined in a mapping function, mapping table, or other suitable function or data structure. By maintaining and persisting the association into the rendering runtime environment, the most suitable and capable NeRF can be queried on a voxel-by-voxel basis, improving the accuracy of 2D image rendering.

[0092] Overall, the controller pushes feature vectors and direction inputs for each voxel associated with a pixel and each pixel in the 2D image being constructed. The predicted color, volume density, and feature values output by the NeRFs 520 are distributed by the routing function 503 to the pixel function 507 or the mask function 505.

[0093] The pixel function 507 accepts color and volume density values from the routing function 503 and uses the routing function 503 to compute color and density values for the pixel. The mask function 505 accepts semantic feature values from the routing function 503 and uses the semantic feature values to construct a 2D feature map. In addition, the mask function 505 accepts a 2D location of the object of interest from the controller 501, which is used to Figure 1 The object mask allows the mask function 505 to compute an object mask for the object of interest. The object mask can be applied by the pixel function 507 to eliminate most, if not all, extraneous pixels in the final image that are not relevant to the object of interest.

[0094] Figures 6A to 6H Various aspects of the enhanced AI image processing techniques disclosed herein are further illustrated. Figure 6A An extension to the rendering pipeline 500 is shown in FIG. 6. The extension includes a 3D object tracker 600 that is configured to track the 3D location of the object of interest in the 3D scene being rendered. The 3D object tracker 600 is configured to receive as input a 3D location of the object of interest in the 3D scene being rendered. The object tracker 600 is configured to track the 3D location of the object of interest in the 3D space-time being rendered. The object tracker 600 is configured to output as output a 3D location of the object of interest in the 3D scene being rendered. Figure 6Hthe first stage (stage 600A) of the operational scheme. Stage 600A of the operational scheme involves a scene 601 captured in video over time from multiple viewing directions by multiple cameras. Scene 601 includes multiple object locations in 3D space represented by object 603 (a tree), object 607 (a turtle), and object 609 (a rabbit).

[0095] Camera 610 records scene 601 from a first direction in video frames 611, camera 620 records scene 601 from a second direction in video frames 621, and camera 630 records scene 601 from a third direction in video frames 631. For example purposes, it is assumed that the duration of recording of the scene by each camera is approximately the same and synchronized with one another. Thus, it is assumed that each set of video frames includes the same number of frames synchronized with one another relative to their timestamps. The recorded videos provide 4D scene data that is provided to a training pipeline 640 that trains a set of NeRFs as described herein to construct new images of scene 601 from desired viewpoints at desired times.

[0096] Figure 6B Stage 600B in FIG. 6B illustrates motion assumed to occur in scene 601 for example purposes. For example, between time tl and time t2, object 607 moves a distance dl from left to right. However, over the same duration, object 609 moves a greater distance (d2). To provide further contrast, object 603 has little to no motion.

[0097] Figure 6C Stage 6C in FIG. 6C illustrates various motion regions resulting from the motion (or lack thereof) of objects 607, 609, and 603 in scene 601. The motion of object 603 (because it is low) is characterized with the general background, thus falling into motion region 623. In contrast, the motion of object 607 creates motion region 627 in the scene, while the motion of object 609 creates motion region 629. Motion regions 627 and 629 are intended to visualize the four-dimensional nature of the video, which consists of three spatial dimensions (x, y, and z) and a temporal dimension t. In a sense, the motion regions can be conceptualized as a “tunnel” through the 3D scene.

[0098] The various motion regions can be sampled at different spatial-temporal resolutions and provided as input to different NeRFs. For example, a first NeRF within training pipeline 640 can be trained on motion region 623 at a first sampling resolution, while a second NeRF can be trained on motion region 627 at a sampling resolution that is greater than the first sampling resolution, and a third NeRF can be trained on motion region 629 at another sampling resolution that is greater than both the first sampling resolution and the second sampling resolution. (It will be appreciated that while the example of three NeRFs is provided, there can be any number of NeRFs, each trained on a different motion region at a different sampling resolution.) Figures 6A to 6HA 1 : 1 correspondence between the motion regions and the NeRF / space-time sampling resolution is shown, but a given NeRF can be trained on multiple motion regions at the specified space-time sampling resolution for that NeRF. Figure 6D A correspondence between the motion regions and the NeRF is shown in more detail.

[0099] In Figure 6D , stage 600D of the run scheme illustrates a correspondence between the motion regions and the NeRFs, and the temporal component of the specified space-time sampling resolution. Specifically, stage 600D shows three NeRFs represented by NeRF 653, NeRF 657, and NeRF 659. NeRF 653 is trained on motion region 623 at a first temporal sampling resolution A, while NeRF 657 is trained on motion region 627 at a second temporal sampling resolution B, and NeRF 659 is trained on motion region 629 at a third temporal sampling resolution.

[0100] For the purposes of illustration, assume that temporal sampling resolution A is less than temporal sampling resolution B, which is less than temporal sampling resolution C. This is because motion region 623 includes the least amount of motion, while motion region 627 includes more motion than motion region 623, but less than motion region 629.

[0101] The difference in temporal sampling resolution is represented by the change in the sampled pixels from the individual cameras. For example, for NeRF 653 and temporal sampling resolution A, pixel 612 in video frame 611 is only sampled twice. Translating this concept into 4D space, any sample point corresponding to pixel 612 would only be drawn from the time points corresponding to the two frames shown here with respect to video frame 611. Similarly, a pixel in video frame 621 would only be sampled twice, as would another pixel in video frame 631. In 4D space, any sample point corresponding to these pixels would only be drawn from the time points corresponding to the two frames shown here with respect to video frames 621 and 631.

[0102] In contrast, temporal sampling resolution B results in pixels (or their related points in 4D) being sampled more frequently. For example, for NeRF 657 and temporal sampling resolution B, a pixel in video frame 611 would be sampled three times. Translating this concept into 4D space, any sample point corresponding to that pixel would only be drawn from the time points corresponding to the three frames shown here with respect to video frame 611. Similarly, a pixel in video frame 621 would only be sampled three times, as would another pixel in video frame 631. In 4D space, any sample point corresponding to these pixels would only be drawn from the time points corresponding to the three frames shown here with respect to video frames 621 and 631.

[0103] Further by comparison, the temporal sampling resolution C will result in the pixel (or its associated point in 4D) being sampled more frequently than in the first two examples. For example, for NeRF 659 and temporal sampling resolution C, the pixel in video frame 611 will be sampled six times. Translating this concept into 4D space, any sampling point corresponding to this pixel will be drawn from the time points corresponding to the six frames illustrated here with respect to video frame 611. Similarly, the pixel in video frame 621 will be sampled only three times, as will another pixel in video frame 631. In 4D space, any sampling point corresponding to these pixels will be drawn from only the time points corresponding to the six frames illustrated here with respect to video frames 621 and 631.

[0104] Figure 6D Stage 600E in FIG. 6 illustrates the correspondence between the motion regions and the NeRFs, as well as the spatial component of the specified space-time sampling resolution. Stage 600E also illustrates NeRF 653, NeRF 657, and NeRF 659 and their respective motion regions. NeRF 653 is trained on motion region 623 with a first spatial sampling resolution A, while NeRF 657 is trained on motion region 627 with a second spatial sampling resolution B, and NeRF 659 is trained on motion region 629 with a third spatial sampling resolution.

[0105] For the purposes of illustration, assume that temporal sampling resolution A is less than spatial sampling resolution B, which is less than spatial sampling resolution C. This is again because motion region 623 includes the least amount of motion, while motion region 627 includes more motion than motion region 623, but less than motion region 629.

[0106] The differences in spatial sampling resolution are represented by the change in the sampled voxels for a given pixel in a single frame, while keeping the temporal component constant at a single time instant. For example, for NeRF 653 and spatial sampling resolution A, the same pixel (pixel 612) in a single frame of video frame 611 is sampled only twice. Translating this concept into 4D space, for a single time instant, only two points on ray 613 corresponding to pixel 612 that is projected from pixel 612 will be sampled. However, translating this concept into 4D space, for a single time instant, only two points on ray 613 corresponding to pixel 612 that is projected from pixel 612 will be sampled. Figure 6E Combining the spatial sampling resolution in FIG. 6 with the temporal sampling resolution illustrated in FIG. 7 will result in four (4) total sampling points: the two points along the ray will be sampled at the two time instants specified by the temporal sampling resolution, respectively. Figure 6D

[0107] ​Similarly, a pixel at a single instance of video frame 621 will be spatially sampled only twice, as will another pixel in video frame 631. Translating this concept into 4D space, only two points along the ray projected from the pixel in video frame 621 will be sampled, and only two points along the ray projected from the pixel in video frame 631 will be sampled. Again, the spatial sampling resolution in Figure 6E in combination with the temporal sampling resolution shown in Figure 6D will yield four (4) total sampling points for each of video frames 621 and 623: four points along each ray will be sampled at the two time instants specified by the temporal sampling resolution in Figure 6D

[0108] In contrast, spatial sampling resolution B will result in more points being sampled than spatial sampling resolution A. Keeping the temporal component constant at a single time instant, the difference in spatial sampling resolution is represented by a change in the sampling voxels for a given pixel in a single frame. For example, for NeRF 657 and spatial sampling resolution B, a pixel at a single time instant in video frame 611 is sampled four times. Translating this concept into 4D space, four points on the ray projected from the pixel will be sampled for a single time instant. Again, the spatial sampling resolution in Figure 6E in combination with the temporal sampling resolution shown in Figure 6D will yield twelve (12) total sampling points: four points along each ray will be sampled at the three time instants specified by the temporal sampling resolution in

[0109] Similarly, a pixel at a single instance of video frame 621 will be spatially sampled four times, as will another pixel in video frame 631. Translating this concept into 4D space, four points along the ray projected from the pixel in video frame 621 will be sampled, and four points along the ray projected from the pixel in video frame 631 will be sampled. Again, the spatial sampling resolution in Figure 6E in combination with the temporal sampling resolution shown in Figure 6D will yield twelve (12) total sampling points for each of video frames 621 and 631: four points along each ray will be sampled at the three time instants specified by the temporal sampling resolution in Figure 6D

[0110] Further in contrast, spatial sampling resolution C will result in more points being sampled than spatial sampling resolution A or B. Keeping the temporal component constant at a single time instant, the difference in spatial sampling resolution is represented by a change in the sampling voxels for a given pixel in a single frame. For example, for NeRF 659 and spatial sampling resolution C, a pixel at a single time instant in video frame 611 is sampled eight times. Translating this concept into 4D space, eight points on the ray projected from the pixel will be sampled for a single time instant. Again, the spatial sampling resolution in​​ Figure 6E the spatial sampling resolution in Figure 6D combined with the temporal sampling resolution shown in

[0111] Similarly, a single instance of a pixel in video frame 621 will be spatially sampled eight times, as will another pixel in video frame 631. Translating this concept into 4D space, eight points along the ray cast from the pixel in video frame 621 will be sampled, and eight points along the ray cast from the pixel in video frame 631 will be sampled. Again, Figure 6E the spatial sampling resolution in Figure 6D combined with the temporal sampling resolution shown in Figure 6D

[0112] Figure 6F Stage 600F in a running scenario is shown. Here, the user wishes to view the scene 601 from a desired viewpoint at a desired time. The user can input the viewing parameters through a user interface, which are fed to the rendering pipeline 690. The desired view 681 represents a perspective view captured by a virtual camera 680 oriented in a virtual direction suitable for rendering a 2D image according to the user’s selection. Note that the position and orientation of the virtual camera 680 is new relative to the placement of the cameras 610, 620, and 630 in Figure 6A The rendering pipeline 690 accepts the viewing parameters and produces a rendered image according to the operational flow discussed above with respect to the rendering pipelines 105 and 500.

[0113] Figure 6G Stage 600G in a running scenario is shown. Stage 600H involves a user interface 691 through which the user can view an initial 2D image of the scene 601 as well as input viewing parameters. The user interface 691 includes a time bar 692, which the user can use to adjust a slider 693 to specify a desired time. The user interface 691 also includes a viewing direction bar 694, which the user can use to adjust a slider 694 to specify a desired viewing direction. The rendering pipeline 690 accepts the viewing parameters as input and produces a 2D image 696 from the perspective of the desired viewing direction and the desired time. Note that the 2D image 696 looks at least slightly different (or new) relative to any image or frame captured by the cameras 610, 620, and 630.

[0114] Finally, Figure 6H An optional stage in a running scenario is shown. Stage 600H shows a 2D image 696 produced by the rendering pipeline 690 from the perspective of a desired viewing direction and a desired time.​ Figure 6G The same user interface 691 as before, with the addition of an object selection 697. The object selection 697 is used to identify an object of interest to the rendering pipeline 690. As a result, the rendering pipeline 690 is able to filter out most or all other pixels from the 2D image 696 that are not related to the object of interest. The resulting 2D image 699 includes only those pixels associated with the object of interest. Again, note that the 2D image 699 looks at least slightly different (or new) relative to any image or frame captured by the cameras 610, 620, and 630, in addition to its object tracking features.

[0115] Framework: Gear-NeRF

[0116] The following sections describe in more detail a framework that implements the concepts described above. The framework can be implemented in program instructions in the context of software and / or firmware elements. When executed by one or more processing devices of one or more computing systems, the program instructions direct the one or more computing systems to operate as described below with respect to the framework.

[0117] Neural radiation field (NeRF): Standard NeRF employs a multi-layer perceptron (MLP) with sinusoidal position encoding to map 3D space coordinates and viewing directions to volumetric densities and emitted . Rendering individual image pixels involves casting a ray along a direction through the pixel The predicted color of the corresponding pixel is computed as follows:

[0118] where and . Standard NeRF is trained by minimizing the mean squared error between input images and predicted images obtained by rendering the scene from the viewpoints that captured the input images, with the training loss given by:

[0119] where is the set of all rays cast from the input image.

[0120] Planar decomposition of 4D volumes: A recent emerging trend in handling dynamics using radiance field representations is to directly adapt them to frames in addition to and . This can be done by learning from to . These methods attempt to learn the 4D feature vector of each projection to a set of 2D planes. The embeddings of these projections on these planes can then be integrated to obtain an embedding for the 4D point. This can be mathematically expressed as follows:

[0121] where, and are functions that embed coordinate tuples into features of dimension m (evaluated by bilinear interpolation on a regularly spaced 2D grid of features), denotes element-wise multiplication, and denotes a linear transformation that maps the product to a feature vector. Subsequently, a small MLP can map the feature vector to the volume density and the view-dependent emitted color given an observation direction .

[0122] The proposed method

[0123] Given a set of input videos of a dynamic scene , with calibrated camera poses, the disclosed framework (referred to herein as Gear-NeRF) represents the scene using a series of 4D feature volumes and 4D semantic embeddings.

[0124] Similar to the multiple gears in a motor vehicle used to optimize engine performance, Gear-NeRF hierarchically N gear levels this semantically embedded scene representation based on the scale of motion. Each of these levels is referred to as a gear. With an enhanced training scheme, regions with greater motion are assigned higher gear levels, receiving denser spatio-temporal sampling to improve dynamic novel view synthesis. The proposed 4D semantic embeddings also enable a new functionality, viewpoint tracking of target objects at almost no cost, given a simple user cue such as a click.

[0125] Figure 7 The entire pipeline 700 of Gear-NeRF is shown. In Figure 7In particular, the Gear-NeRF 703 takes the multi-view video 701 as input. After optimizing the serial 4D feature volume, the Gear-NeRF 703 maps the space-time coordinates to 4D semantic embeddings, in addition to volume density and view-dependent radiance color. The Gear-NeRF 703 allows for volume rendering of desired 2D RGB output images 707 as well as 2D feature maps 709. Regions 711 with larger motion are automatically assigned higher gear levels and thus receive higher resolution space-time sampling. In addition, the Gear-NeRF is able to perform free-viewpoint tracking of target objects 713 with such simple cues as user clicks 705.

[0126] Series of 4D feature bodies

[0127] The representation method of this framework is not to represent the dynamic scene using a uniform 4D volume, but to include a series of feature volumes, each of which corresponds to a gear level . Specifically, for any space-time coordinate , the feature vector corresponding to is calculated as follows:

[0128] The vector-valued function and the linear transformation are shared by all gears, while each gear has its own space-time embedding . Therefore, each gear describes a region of a certain motion scale, while pure spatial features can be shared between all gears.

[0129] This framework also obtains the gear level at any space-time coordinate from the planar decomposed 4D feature volume. Specifically, the gear level at is calculated as follows:

[0130] where is a vector consisting of 1s, and are embedding functions. However, this defines a continuous feature volume. In order to map it to gear level integers, we apply the following projection operation:

[0131] Based on this gear level volume, we define the 4D mask for regions at gear level as:

[0132] The final feature vector at is calculated as follows:

[0133] Subsequently, the tiny MLP, maps these feature vectors and the observed direction to a volume density and radiance / color values . This allows the framework to obtain photometric renderings of the scene.

[0134] 4D semantic embedding

[0135] Gear-NeRF leverages the powerful object priors of the Segmented Arbitrary Model (SAM) model to obtain semantic understanding of the scene to improve photometric rendering and free-viewpoint object tracking. To this end, the framework leverages a SAM encoder to obtain 2D feature maps from the frames of the individual videos. The framework then optimizes the 4D SAM embedding field by supervising it using these 2D feature maps. In particular, the above MLP is configured to output, in addition to density and color, a 4D semantic embedding for a given spatio-temporal coordinate. To render a 2D semantic feature map in a given view, the framework computes the semantic features of the pixels in the feature map by tracing rays through the framework and performing volume rendering, similar to Equation 1:

[0136] The SAM embedding is supervised by minimizing the mean squared error between the prediction and the ground truth features of the SAM encoder, as follows:

[0137] Training program using gear allocation

[0138] In the case of gear initialization , the (semantically embedded) radiance field optimization and gear assignment update are done in an alternating fashion.

[0139] Gear assignment update: When updating the gear assignment after a period of radiance field optimization, the framework finds the regions that are rendered worst from the rendering loss map and increases their gear for denser spatio-temporal sampling. The following steps outline the process of updating the gear assignment of a region: The framework samples multiple viewpoints and time steps and renders 2D images / SAM features for them. For each rendered RGB-SAM map, the framework computes a rendering loss map. Each pixel of the rendering loss map is computed using .

[0140] The framework then stitches the individual render loss maps to find the image patches with the top k largest / smallest average loss. The center coordinates of these image patches are used as positive / negative cues for the next step.

[0141] The framework feeds the ground truth RGB images along with the positive and negative cues into the SAM decoder to estimate upshift masks. These masks tend to cover regions with motion and cannot be satisfactorily rendered with the current sampling resolution. Note that there are multiple upshift masks at different viewpoints and time steps.

[0142] For each pixel of the upshift mask, the framework traces the ray and samples along the points of the ray and updates the gear assignment by pushing the value towards the incremented value.

[0143] By updating the gear assignment, the framework improves the spatio-temporal sampling resolution of the gear shift regions and then resumes the task of optimizing the radiance field.

[0144] The framework alternates between two processes: radiance field optimization (L rounds are performed each time) and gear assignment update until the average variance of the individual render loss maps is below a predetermined threshold. After this, the framework performs additional L' rounds of optimization on the radiance field without further gear assignment update.

[0145] Motion-sensing spatial-temporal sampling

[0146] This subsection explains how the framework allows differential treatment of regions at different gear levels based on the motion-aware spatio-temporal sampling strategy specified by the gear. Temporal sampling means choosing a temporal resolution for the planar decomposed 4D feature volume and, through spatial sampling, choosing a strategy for sampling points along each ray for volume rendering.

[0147] Motion sensing time sampling To handle the increase in object motion intensity, as reflected by their growing gear level, the framework increases the temporal resolution of the voxel grid. Specifically, the The temporal resolution along the time axis is increasingly higher, enabling the 4D feature volume to better model dynamics along the time axis. This ensures that fast-moving objects can be modeled more faithfully without appearing blurry. The temporal resolution of each gear feature volume is determined by a linear interpolation between (for ) to the total number of frames (for ).

[0148] Motion-sensing spatial sampling ​While denser point sampling can improve reconstruction accuracy, increasing the number of sampling points across the entire scene can lead to high computational costs. Therefore, this framework employs a 3D point segmentation strategy. This strategy is implemented by sampling a relatively small number of n samples along each ray, assuming the framework is at its lowest setting. If a sampling point belongs to... For regions with higher gears, the frame samples points more densely within those regions. For each sample point within that region, the frame breaks it down into... The points are equidistantly distributed within the corresponding ray segment (at that gear level).

[0149] Cost-free viewpoint tracking with user prompts

[0150] The proposed 4D SAM embedding facilitates high-quality cueable mask segmentation and enables another useful feature: near-cost viewpoint object tracking, allowing users to extract target objects based on the 4D embedding with a single click. The following describes how the framework obtains the corresponding object mask at a new viewpoint and time step, given a click provided by the user at any arbitrary viewpoint and time step.

[0151] Masking for the new viewpoint: The first step in this task is to find the 3D correspondence of the user's click. To achieve this, the framework traces a ray through selected pixels and determines the depth at which the ray intersects the first object surface it encounters by utilizing volume density. This yields the 3D coordinates of the intersection point. Subsequently, using the camera pose of the new viewpoint, the 3D coordinates of this intersection point can be easily mapped to 2D coordinates within any new viewpoint image. In addition to the rendered SAM feature map of the new view, the framework also feeds these coordinates into the SAM decoder to generate the object mask for the new view.

[0152] Mask for new time step For this task, the framework propagates the object mask to its adjacent time steps. Specifically, for the object mask of a specific frame t, the framework computes the bounding box of the mask and uses this bounding box as the reference for adjacent frames. or The SAM hint. By combining this hint with... SAM features rendered at the location Figure 1 By inputting the SAM decoder, we can obtain the target The object mask.

[0153] Combining the two processes described above, we can obtain the object mask from any viewpoint and time step, starting with a single click.

[0154] Figure 8A computing device 801 is shown, which is representative of any system or collection of systems in which the various processes, programs, services, frameworks, and schemes disclosed herein can be implemented. Examples of the computing device 801 include, without limitation, desktop and laptop computers, tablet computers, mobile computers, server computers, network servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server, container, and any variation or combination thereof.

[0155] The computing device 801 can be implemented as a single device, system, or apparatus, or can be implemented in a distributed manner as multiple devices, systems, or apparatuses. The computing device 801 includes, without limitation, a processing system 802, a storage system 803, software 805, a communication interface system 807, and a user interface system 809. The processing system 802 is operatively coupled with the storage system 803, the communication interface system 807, and the user interface system 809. The computing device 801 can optionally include additional devices, features, or functionality not discussed for the sake of brevity.

[0156] The processing system 802 loads and executes software 805 from the storage system 803. The software 805 includes and implements a computer vision process 806, which is representative of the training process 200 and the rendering process 300. When executed by the processing system 802, the software 805 instructs the processing system 802 to operate as described herein at least with respect to the various processes, operational schemes, and sequences discussed in the implementations described above.

[0157] Still referring to Figure 8 The processing system 802 can include a microprocessor and other circuitry that fetches and executes software 805 from the storage system 803. The processing system 802 can be implemented within a single processing apparatus, but can also be distributed across multiple processing apparatuses or subsystems that cooperate in executing program instructions. Examples of the processing system 802 include general purpose central processing units, graphics processing units, digital signal processors, specialized processors, and logic apparatuses, as well as any other type of processing apparatus, combinations, or variations thereof.

[0158] The storage system 803 can include any computer readable storage media readable by the processing system 802 and capable of storing software 805. The storage system 803 can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read only memory, magnetic disks, optical disks, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case is the computer readable storage media a propagated signal.

[0159] In addition to computer-readable storage media, in some implementations storage system 803 can also include computer-readable communication media over which at least some of software 805 can be communicated internally or externally. Storage system 803 can be implemented as a single storage device but can also be implemented across multiple storage devices or sub-systems confi gurated to operate collectively.

[0160] Software 805, including computer vision processes 806, can be implemented in program instructions and, among other functions, can direct processing system 802 to operate in accordance with various operational scenarios, sequences, frameworks, and processes as shown and / or discussed herein when executed by processing system 802. For example, software 805 can include program instructions for implementing the sampling, training, and / or rendering processes described herein, as well as for executing multiple NeRFs as described herein.

[0161] In particular, the program instructions can include various components or modules that cooperate or otherwise interact to perform the various processes and operational scenarios described herein. The various components or modules can be embodied in compiled or interpreted instructions, or in some other variant or combination of instructions. The various components or modules can be executed in a synchronous or asynchronous manner, serially or in parallel, within a single threaded environment or a multi-threaded environment, or according to any other suitable execution paradigm, variant, or combination thereof. Software 805 can include additional processes, programs, or components, such as operating system software, virtualization software, or other application software. Software 805 can also include firmware or some other form of machine-readable processing instructions executable by processing system 802.

[0162] In general, software 805, when loaded into processing system 802 and executed, can transform a suitable device, system, or apparatus (of which computing device 801 is representative) from a general-purpose computing system into a special- purpose computing system customized to facilitate the execution of computer vision processes in an optimized manner. Indeed, coding software 805 on storage system 803 can physically alter the internal structure of storage system 803. The particular transformation of the physical structure can depend on various factors in different implementations of this description. Examples of such factors can include, but are not limited to, the technology used to implement the storage media of storage system 803 and whether the computer-storage media is characterized as main storage or secondary storage, as well as other factors.

[0163] For example, if the computer-readable storage media are implemented with semiconductor-based memory, software 805 can transform the physical state of the semiconductor memory when the software is encoded therein, e.g., by transforming the state of transistors, capacitors or other discrete circuit elements constituting the semiconductor memory. Similarly, if the computer-readable storage media are implemented with magnetic or optical memory, software 805 can transform the physical state of magnetic or optical media, e.g., by transforming the state of magnetic domains or polarization states of light pulses.

[0164] Communication interface system 807 can include communication connections and devices that allow for communication with other computing systems (not shown) over a communication network (not shown). Examples of connections and devices that together allow for inter-system communication can include network interface cards, antennas, power amplifiers, RF circuitry, transceivers, and other communication circuitry. The connections and devices can communicate through communication media to exchange communications with other computing systems or networks of systems, such as metal, glass, air, or any other suitable communication media. The aforementioned media, connections, and devices are well known and need not be discussed at length here.

[0165] Communication between computing device 801 and other computing systems (not shown) can occur through one or more communication networks and according to various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, internets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software-defined networks, data center buses and backplanes, or any other type of network, combination of networks, or variation thereof. The aforementioned communication networks and protocols are well known and need not be discussed at length here.

[0166] Those skilled in the art will appreciate that aspects of the present application can be embodied as a system, method, or computer program product. Accordingly, aspects of the present application can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "circuit," "module" or "system." Furthermore, aspects of the present application can take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0167] Indeed, the descriptions and drawings depict specific embodiments in order to teach those skilled in the art how to make and use the best mode. Some well-known aspects are simplified or omitted in order not to obscure the concept of the application. Those skilled in the art will appreciate variations from these embodiments that fall within the scope of the disclosure. Those skilled in the art will also appreciate that the features described above can be combined in various ways to form many different embodiments of the application. As a result, the application is not limited to the specific embodiments described above, but only by the claims and their equivalents.

Claims

1. An artificial intelligence (AI) image processing system, wherein the AI ​​image processing system renders two-dimensional (2D) images of a four-dimensional (4D) scene from different viewpoints and at different times using multiple dynamic neural radiation fields (NeRF), the AI ​​image processing system comprising: At least one processor and a memory thereon storing instructions that cause the at least one processor of the AI ​​image processing system to perform the following operations: Collect observation parameters for rendering a two-dimensional (2D) image of the 4D scene, wherein the observation parameters include a desired observation direction and a desired time, and wherein the 4D scene includes a three-dimensional (3D) scene that changes over time. The plurality of dynamic NeRFs are executed to obtain voxel color and volume density values ​​associated with pixels in the 2D image based on the observation parameters, wherein the plurality of dynamic NeRFs include implicit neural representations of the 4D scene; and The 2D image is rendered based on a combination of the color and the volume density values ​​obtained from the plurality of dynamic NeRFs.

2. The AI ​​image processing system according to claim 1, wherein, The observation parameters also include objects of interest from the 4D scene, and the instructions further cause the at least one processor to: Execute one or more of the dynamic NeRFs to obtain a feature map of the 2D image; At least in part based on the feature map, an object mask corresponding to the object of interest is obtained; and The object mask is applied to the 2D image.

3. The AI ​​image processing system according to claim 2, further comprising an input interface configured to receive selections of the observation parameters, wherein, The selection includes choosing the object of interest from different 2D images corresponding to times different from the desired time.

4. The AI ​​image processing system according to claim 3, wherein, In order to obtain the object mask at least in part based on the feature map, the instruction causes the at least one processor to: Determine the position of the object of interest in the different 2D images; The position of the object of interest in the 3D scene is determined at least based on the position of the object of interest in the different 2D images; Based on the position of the object of interest in the 3D scene, the position of the object of interest in the 2D image is determined; and The object mask is generated based on the location of the object of interest in the 2D image.

5. The AI ​​image processing system according to claim 1, wherein, In order to execute the plurality of dynamic NeRFs to obtain the color and volume density values ​​of the voxels associated with the pixels in the 2D image, the instructions cause the at least one processor to perform the following operations for each pixel in the 2D image: Identify the voxel group associated with the pixel, wherein each voxel in the voxel group is associated with a different four-dimensional (4D) spatial-temporal coordinate; and For each voxel in the voxel group: Based on the 4D space-time coordinates associated with the voxel, one of the NeRFs is selected for execution; and Execute one of the NeRFs to obtain the color value and volume density value of the voxel.

6. The AI ​​image processing system according to claim 1, wherein, In order to execute the plurality of dynamic NeRFs to obtain the color and volume density values ​​of the voxels associated with the pixels in the 2D image, the instructions cause the at least one processor to execute only one of the plurality of dynamic NeRFs for each voxel associated with each pixel in the 2D image to obtain the color and volume density values ​​of the voxels.

7. The AI ​​image processing system according to claim 1, wherein, In order to execute the plurality of dynamic NeRFs to obtain the color and volume density values ​​of the voxels associated with the pixels in the 2D image, the instructions cause the at least one processor to execute all of the plurality of dynamic NeRFs for each voxel associated with each pixel in the 2D image to obtain the color and volume density values ​​of the voxels.

8. The AI ​​image processing system according to claim 1, wherein, Each of the plurality of dynamic NeRFs has a spatial-temporal resolution that is different from that of each other dynamic NeRF in the plurality of dynamic NeRFs.

9. The AI ​​image processing system according to claim 8, wherein, Each of the plurality of dynamic NeRFs includes an implicit neural representation of a portion of the 4D scene, which is different from the portions of the 4D scene implicitly represented by each of the other dynamic NeRFs.

10. The AI ​​image processing system according to claim 1, wherein, The plurality of dynamic NeRFs include a first dynamic NeRF corresponding to a first moving object in the 4D scene and a second dynamic NeRF corresponding to a second moving object in the 4D scene, wherein the motion of the first moving object is greater than the motion of the second moving object, and wherein the spatial-temporal resolution of the first dynamic NeRF is greater than the spatial-temporal resolution of the second dynamic NeRF.

11. A method for rendering images of a four-dimensional (4D) scene from different viewpoints at different times using multiple dynamic neural radiation fields (NeRF), the method comprising the steps of: Collect observation parameters for rendering a two-dimensional (2D) image of the 4D scene, wherein the observation parameters include a desired observation direction and a desired time, and wherein the 4D scene includes a three-dimensional (3D) scene that changes over time. The plurality of dynamic NeRFs are executed to obtain voxel color and volume density values ​​associated with pixels in the 2D image based on the observation parameters, wherein the plurality of dynamic NeRFs include implicit neural representations of the 4D scene; and The 2D image is rendered based on a combination of the color and the volume density values ​​obtained from the plurality of dynamic NeRFs.

12. The method according to claim 11, wherein, The observation parameters also include objects of interest from the 4D scene, and the method further includes the following steps: Execute one or more of the dynamic NeRFs to obtain a feature map of the 2D image; At least in part based on the feature map, an object mask corresponding to the object of interest is obtained; and The object mask is applied to the 2D image.

13. The method according to claim 12, further comprising: The selection of the observation parameters is received via an input interface, wherein the selection includes selecting the object of interest from different 2D images corresponding to times different from the desired time.

14. The method according to claim 13, wherein, The step of obtaining the object mask based at least in part on the feature map includes: Determine the position of the object of interest in the different 2D images; The position of the object of interest in the 3D scene is determined at least based on the position of the object of interest in the different 2D images; Based on the position of the object of interest in the 3D scene, the position of the object of interest in the 2D image is determined; and The object mask is generated based on the location of the object of interest in the 2D image.

15. The method according to claim 11, wherein, The step of performing the plurality of dynamic NeRF operations to obtain the color and volume density values ​​of voxels associated with pixels in the 2D image includes: for each pixel in the 2D image, Identify the voxel group associated with the pixel, wherein each voxel in the voxel group is associated with a different four-dimensional (4D) spatial-temporal coordinate; and For each voxel in the voxel group: Based on the 4D space-time coordinates associated with the voxel, one of the NeRFs is selected for execution; and Execute one of the NeRFs to obtain the color value and volume density value of the voxel.

16. The method according to claim 11, wherein, The step of performing the plurality of dynamic NeRFs to obtain the color and volume density values ​​of voxels associated with pixels in the 2D image includes: performing only one of the plurality of dynamic NeRFs for each voxel associated with each pixel in the 2D image to obtain the color and volume density values ​​of the voxel.

17. The method according to claim 11, wherein, The step of performing the plurality of dynamic NeRFs to obtain the color and volume density values ​​of voxels associated with pixels in the 2D image includes: performing all of the plurality of dynamic NeRFs for each voxel associated with each pixel in the 2D image to obtain the color and volume density values ​​of the voxel.

18. The method according to claim 11, wherein, Each of the plurality of dynamic NeRFs has a spatial-temporal resolution that is different from that of each other dynamic NeRF, and each of the plurality of dynamic NeRFs includes an implicit neural representation of a portion of the 4D scene that is different from the portion of the 4D scene implicitly represented by each other dynamic NeRF.

19. The method according to claim 11, further comprising: The plurality of NeRFs are trained at different spatial-temporal sampling resolutions and on different parts of the 4D scene, wherein the motion intensity of each part of the 4D scene targeted by each of the plurality of NeRFs is different from the motion intensity of each other part of the 4D scene targeted by each of the other NeRFs.

20. A computing device, the computing device comprising: One or more computer-readable storage media storing program instructions that, when read and executed by one or more processors, instruct the computing device to at least: Multiple dynamic NeRFs are executed to obtain voxel color and volume density values ​​associated with pixels in a 2D image based on observation parameters including the desired observation direction and desired time, wherein the multiple dynamic NeRFs include implicit neural representations of a 4D scene. The 2D image is rendered based on a combination of the color and the volume density values ​​obtained from the plurality of dynamic NeRFs; Execute one or more of the dynamic NeRFs to obtain a feature map of the 2D image; Based at least in part on the feature map, an object mask corresponding to the object of interest in the 4D scene is generated; and The object mask is applied to the 2D image.

Images