Inverting neural radiance fields for pose estimation

Abstract

Provided are systems and methods of inverting a trained NeRF model, where the NeRF model stores the structure of a scene or object to estimate 6D poses from images taken at new viewpoints. 6D pose estimation has a wide range of applications, including visual localization and object pose estimation for robotic manipulation.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 114,399, filed November 16, 2020. Each of the above applications is incorporated herein by reference in its entirety. Technical Field

[0003] This disclosure generally relates to systems and methods for attitude estimation. More specifically, this disclosure relates to inverting neural radiation fields used for attitude estimation. Background Technology

[0004] Neural Radiance Field (NeRF) demonstrates its advantages in novel view image synthesis by capturing the 3D geometry and appearance of scenes or objects using neural network models. Differentiable neural models are designed to be trained using a set of 2D images with known camera poses, and employ gradient descent to optimize occupancy and color appearance in 3D space. Summary of the Invention

[0005] Various aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description or by practice of the embodiments.

[0006] One example aspect of this disclosure relates to a computational system for determining camera pose. The computational system includes one or more processors and one or more non-transitory computer-readable media, which jointly store: a machine learning neural radiation field model previously trained to model a scene; and instructions that, when executed by the one or more processors, cause the computational system to perform operations. The operations include: obtaining a subject image captured by a camera. The operations further include: for each of one or more pose update iterations: obtaining a current estimated pose of the camera; processing data describing the current estimated pose with the machine learning neural radiation field to generate one or more synthetic pixels of a synthetic image of the scene from the current estimated pose; evaluating a loss function that compares the one or more synthetic pixels with one or more observed pixels included in the subject image captured by the camera; and updating the current estimated pose of the camera, at least in part based on the gradient of the loss function.

[0007] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0008] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0009] A detailed discussion of embodiments relating to those skilled in the art is set forth in the description with reference to the accompanying drawings, wherein:

[0010] Figure 1 A block diagram depicting an example process for training a neural radiation field model for machine learning according to an example embodiment of the present disclosure.

[0011] Figure 2 A block diagram depicts an example process for estimating camera pose using a neural radiation field model based on an example embodiment of the present disclosure.

[0012] Figure 3A A block diagram depicting an example computing system according to an example embodiment of the present disclosure.

[0013] Figure 3B A block diagram depicting an example computing device according to an example embodiment of the present disclosure.

[0014] Figure 3C A block diagram depicting an example computing device according to an example embodiment of the present disclosure.

[0015] Repeated reference numerals across multiple figures are intended to identify the same features in different embodiments. Detailed Implementation

[0016] Overview

[0017] Generally, this disclosure relates to systems and methods for estimating pose (e.g., represented in six dimensions (6D)) from images taken from a new perspective by reversing a trained neural radiation field (NeRF) model of the structure of a stored scene or object. Pose estimation has wide applications, including visual localization for robot manipulation / navigation and object pose estimation.

[0018] Recent advances in novel view synthesis have sparked interest in addressing challenges through analytical synthesis methods. However, obtaining high-quality 3D models of scenes typically requires specialized sensors, which are usually only feasible in laboratory settings. NeRF offers a method for capturing 3D structures using a set of images and provides an opportunity to apply synthetic analysis methods to real-world scenes.

[0019] Furthermore, for object pose estimation, specific systems typically employ RGB-D cameras for accurate prediction. However, this introduces numerous problems, including resolution, frame rate, field of view, and depth range. This disclosure demonstrates that NeRF can perform accurate pose estimation using only RGB training and testing images.

[0020] Specifically, this disclosure provides a framework (an example implementation of which may be referred to as "iNeRF") for performing pose estimation by "inverting" a neural radiation field (NeRF) trained on a "reverse" basis. NeRF has proven highly effective for viewpoint synthesis tasks (synthesizing realistic new perspectives of real-world scenes or objects). This disclosure demonstrates that NeRF can be used to perform the same quality improvement on the task of 3D pose estimation (fitting a 3D model to an image of that model). As an example, given an initial pose estimate, the example system can use gradient descent to minimize the residuals between pixels rendered from an already trained NeRF and pixels in the observed image.

[0021] The example experiments included in U.S. Provisional Patent Application No. 63 / 114,399 thoroughly investigated: 1) how ray sampling is performed during pose refinement in iNeRF to collect informative gradients, and 2) how different loss functions affect iNeRF on synthetic datasets proposed by NeRF. iNeRF can perform visual localization in complex real-world scenes (e.g., as shown relative to the LLFF dataset). Furthermore, the proposed pose estimation pipeline, which includes feature-based initialization and iNeRF refinement, outperforms all other RGB-based methods on LINEMOD without providing true pose labels.

[0022] The systems and methods disclosed herein offer numerous technical effects and advantages. As an example, this disclosure enables pose estimation to be performed more efficiently than existing techniques. Specifically, many existing techniques require rendering the entire image for each candidate pose to determine which pose is best. Rendering and then analyzing the entire image is computationally expensive. In contrast, the example systems and methods described herein can selectively render only a small number of composite pixels instead of the entire image. This results in significant computational savings, thereby conserving computational resources such as processors, memory, and / or network bandwidth.

[0023] As another example of the technical effects and advantages, the techniques described herein can be used to determine the pose of autonomous robots or to localize autonomous robots. By providing improved pose estimates for the robot, the robot's performance can be improved (e.g., autonomous vehicles can drive more efficiently and safely).

[0024] Exemplary embodiments of this disclosure will now be discussed in further detail with reference to the accompanying drawings.

[0025] Example NeRF model

[0026] Figure 1 A block diagram depicting an example process for training a neural radiation field model for machine learning according to an example embodiment of the present disclosure.

[0027] refer to Figure 1 The training dataset 12 may include existing training images depicting the scene. In some implementations, the training images may be unrestricted and may exhibit various inconsistencies with each other. As shown in 14, the computational system may perform a model optimization or training process on the training dataset 12 to generate a machine learning neural radiation field model 16. After training, the model 16 may be provided with the location of the desired synthetic image. In response, the model 16 may generate a synthetic image depicting the scene from that location.

[0028] Example neural radiance field models and example techniques for training neural radiance field models are described in Mildenhall et al.’s “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis” published on arXiv:2003.08934 [cs.CV] and Martin-Brualla et al.’s “NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections” published on arXiv:2008.02268 [cs.CV]. In some cases, the models and techniques described in these articles can be used as models16 and for training14.

[0029] Specifically, given a set of N RGB images and known camera pose NeRF can learn to synthesize novel perspectives associated with unseen camera poses. NeRF does this by representing the scene as a “radiation field”: a volumetric density modeling the scene shape, and a view-dependent color modeling the appearance of occupied regions of the scene, both within a bounded 3D volume. The density σ and RGB color c at each point are parameterized by the weights Θ of a multilayer perceptron (MLP) F, which assigns the point’s 3D position x = (x, y, z) and its unit norm viewing direction d = (d...). x d y d z As input, (σ, c) ← F Θ (x, d).

[0030] To render a pixel, NeRF can emit a camera ray from the center of the camera's projection through that pixel on the image plane. Along the ray, a set of points can be sampled as input to an MLP, which outputs a set of densities and colors. These values ​​can then be used to approximate the image formation behind volumetric rendering using digital orthogonality, thus producing an estimate of the pixel's color. NeRF can use a certain set of samples from the ray. Training to minimize photometric loss Where C(r) is the observed RGB value of the pixel corresponding to ray r in a certain image, and The predictions are generated from neural volumetric rendering. To improve rendering efficiency, two MLPs can be trained: a "coarse" one and a "fine" one, where the coarse model is used to bias the samples used for the fine model.

[0031] Example of NeRF model inversion

[0032] Example Idea

[0033] This disclosure provides a framework (an example implementation thereof may be referred to as "iNeRF") for performing pose estimation by "inverting" a NeRF trained by Θ. It is assumed that the NeRF of the scene or object parameterized by Θ has been recovered, and the camera intrinsics are known, but the camera pose T of image observation I has not yet been determined. Unlike optimizing the NeRF of Θ using a given set of camera poses and image observations, given the weights Θ and image I as input, the example implementation can solve the problem of inverting the recovery of the camera pose T:

[0034]

[0035] To address this optimization problem, the example implementation utilizes the capability from NeRF to take an estimated camera pose T∈SE(3) within a coordinate frame from the NeRF model and render the corresponding image observation. The example implementation can then use the same photometric loss function as used in NeRF. (See description above), but instead of backpropagation to update the weights Θ of the MLP, the computation system can instead update the pose T to minimize This will be discussed in more detail below. Figure 2 The overall process is shown.

[0036] While the concept of inverting NeRF to perform pose estimation can be briefly explained, it's not immediately apparent how practically this problem can be solved in a useful way. Loss function The SE(3) is non-convex in the 6DoF space, and full-image NeRF rendering is computationally expensive, especially if used in loops within the optimization process. The example formulas and experiments described in this paper aim to address these challenges. The following sections will discuss (i) gradient-based SE(3) optimization processes, (ii) ray sampling strategies, and (iii) how to improve NeRF using predicted poses from iNeRF.

[0037] Gradient-based SE(3) optimization

[0038] Let Θ be used as a parameter of the trained and fixed NeRF. As the estimated camera pose for the current optimization step i, I is the observed image, and This serves as the loss used to train the fine-grained model in NeRF. An example implementation can employ gradient-based optimization to solve the problem defined in Equation 1. To ensure the pose is estimated during gradient-based optimization Continuing in the SE(3) manifold, the example implementation can be parameterized using exponential coordinates. Given an initial pose estimate from the camera bounding box to the model bounding box. Example implementation methods can Represented as:

[0039]

[0040] in,

[0041]

[0042] in, Let θ represent the helical axis, θ represent the amplitude, and [w] represent the skew-symmetric 3×3 matrix of w. Through this parameterization, our goal is to solve for the optimal relative transformation starting from the initial estimated pose T0:

[0043]

[0044] The example implementation can use an MLP to iteratively differentiate the loss function to obtain the gradient for updating the estimated relative transformation. The example implementation can use the Adam optimizer with an exponentially decaying learning rate.

[0045] For each observed image, the example implementation can... Initialize to near 0, where each element is distributed from a zero-mean normal distribution. Randomly selected from the pool. In practice, using... Instead The parameterization causes the rotation center to be located at the center of the initial estimate, rather than at the center of the camera frame. This reduces the coupling between rotation and translation during optimization.

[0046] Example techniques for ray sampling

[0047] In a typical differentiable rendering and comparison pipeline, one would like to utilize the gradient contributed by all output pixels in the rendered image. However, due to NeRF, the value of each output pixel is determined by the gradient along each ray during the ray's travel. The values ​​of the n sample points are weighted to calculate the result; therefore, given a batch The amount of sampled rays in the underlying NeRF MLP, then Each forward / backward pass will be queried. Therefore, calculating and backpropagating the loss of all pixels in the image (i.e., b = HW, where H and W represent the height and width of the high-resolution image) requires significantly more memory than on any commercial GPU.

[0048] While some exemplary implementations of this disclosure may perform multiple forward and backward passes to accumulate these gradients, this becomes very slow for each step of an already iterative optimization process. Below, methods for selecting the loss function to evaluate at each optimization step are provided. Sampling ray set The strategy. Example experiments indicate that when sampling only b=2048 rays at each gradient step, the example implementation is able to recover the accurate pose, which corresponds to a single forward / backward pass suitable for GPU memory, and provides a faster gradient step of 150× on a 640×480 image.

[0049] Random sampling

[0050] The intuitive strategy is to sample M pixel positions on the image plane. And it calculates the corresponding ray. In fact, NeRF itself uses this strategy when optimizing Θ (assuming no image batching is used). However, when the batch size of ray b is small, the performance of this random sampling strategy can be inefficient. For certain types of scenes, most randomly sampled pixels correspond to flat, textureless regions in the image, which provide little information about the pose (consistent with the well-known aperture problem).

[0051] Point of interest sampling

[0052] Another sampling method is interest point sampling. In interest point sampling, the computational system first uses an interest point detector to locate a set of candidate pixel positions in the observed image. Then, the computational system can sample M points from the detected interest points, and if not enough interest points are detected, it reverts to random sampling. While this strategy allows for faster optimization convergence due to the introduction of less randomness, it can be prone to local minima because it only considers interest points in the observed image, rather than interest points from both the observed and rendered images. However, obtaining interest points in the rendered image requires… The forward MLP pass-through is expensive, so its use in optimization can be costly.

[0053] Region of interest sampling

[0054] To prevent local minima caused by sampling only from points of interest, some example implementations of this disclosure may perform "region of interest" sampling. This is a strategy that relaxes interest point sampling and samples from local regions centered on or otherwise adjacent to the interest point (e.g., a dilated mask). As an example, after the interest point detector locates the interest point, the computational system may apply a 5×5 morphological dilation for I iterations to expand the sampling area. In practice, region-based sampling can speed up optimization when the batch size of rays is small. It should be noted that if I is set to an excessively large number, region of interest sampling reverts to random sampling.

[0055] Example visualization of data flow

[0056] Figure 2 A block diagram depicts an example process for estimating camera pose using a neural radiation field model 16 employing machine learning, according to an example embodiment of the present disclosure.

[0057] More specifically, after or during training 14, model 16 can be used to estimate the pose of the camera capturing the subject image 18. This process can be performed iteratively to determine the camera pose. As an example, the current estimated pose is represented as exponential coordinates with six degrees of freedom.

[0058] like Figure 2 As shown, in each of one or more update iterations, the computational system can obtain the current estimated pose 20 of the camera capturing the subject image 18. The computational system can process the data describing the current estimated pose 20 with a neural radiation field 16 of machine learning to generate one or more synthetic pixels 22 of a synthetic image of the scene from the current estimated pose 20.

[0059] As an example, using a neural radiation field 16 from machine learning to process data describing the currently estimated pose 20 to generate one or more synthetic pixels 22 may include: determining a subsampled image plane that identifies one or more sampled pixels from a plurality of possible pixels; determining the ray and orientation of each of the one or more sampled pixels; and processing data describing the ray and orientation to generate one of the synthetic pixels for each sampled pixel.

[0060] As an example, using a neural radiation field 16 from machine learning to process data describing the currently estimated pose 20 to generate one or more synthetic pixels 22 may include: identifying one or more points of interest; identifying regions of interest surrounding each of the one or more points of interest; sampling points from the regions of interest; and generating synthetic pixels from the sampled points.

[0061] After generating the composite pixel 22, the computing system can evaluate a loss function 24 that compares one or more composite pixels 22 with one or more observed pixels included in the subject image 18 captured by the camera.

[0062] As an example, a loss function can compare the corresponding intensity of one or more synthetic pixels with the corresponding intensity of one or more observed pixels included in the subject image captured by the camera.

[0063] As another example, the loss function may additionally or alternatively compare the corresponding chromaticity of one or more synthetic pixels with the corresponding chromaticity of one or more observed pixels included in the subject image captured by the camera.

[0064] As yet another example, the loss function can be the L1 loss function, the L2 loss function, and / or the Huber loss function.

[0065] The computational system can update the camera's current estimated pose 20 based at least in part on the gradient of the loss function 24. For example, the loss function 24 can be backpropagated through model 16 to update the estimated pose 20.

[0066] After one or more iterations are completed, the current estimated pose 20 can be output as the final estimated pose of the main image 18.

[0067] In some implementations, the camera is associated with an autonomous robot (e.g., an autonomous vehicle) that navigates the scene based on the camera's estimated pose.

[0068] In some implementations, the subject image 18 depicts the same scene as the training dataset 12. In other implementations, the subject image 18 depicts a scene that is different from but similar to the training dataset 12.

[0069] Use iNeRF to self-supervise NeRF

[0070] In addition to using iNeRF to perform pose estimation given a trained NeRF, this disclosure also provides techniques for using the estimated pose to feed back into the trained NeRF representation. Specifically, some example implementations may first (1) be given a camera with a known pose. A set of training RGB images is used to train NeRF, generating NeRF parameters Θ. train Then, the example implementation can (2) use iNeRF to obtain additional observation images of unknown poses. And solve for the estimated attitude. Given these estimated poses, the example implementation can (3) use self-supervised pose labels to... Add to the training set. This process allows NeRF to be trained in a semi-supervised environment.

[0071] Example devices and systems

[0072] Figure 3A A block diagram of an example computing system 100 according to an example embodiment of the present disclosure is described. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a network 180.

[0073] User computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0074] User computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118, wherein instructions 118 are executed by processor 112 to cause user computing device 102 to perform operations.

[0075] In some implementations, the user computing device 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be, or may otherwise include, various machine learning models such as neural networks (e.g., deep neural networks) or other types of machine learning models (including nonlinear and / or linear models). Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Example machine learning model 120 has been referenced. Figure 1 and Figure 2 Let's discuss this.

[0076] In some implementations, one or more machine learning models 120 may be received from server computing system 130 over network 180, stored in memory 114 of user computing device, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing device 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., performing parallel perspective synthesis between multiple instances of the same or different scenes).

[0077] Alternatively or additionally, one or more machine learning models 140 may be included in, or otherwise stored in and implemented by, a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine learning model 140 may be implemented by the server computing system 140 as part of a web service (e.g., a view synthesis service). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0078] User computing device 102 may also include one or more user input components 122 for receiving user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to a user input object (e.g., a finger or stylus). Touch-sensitive components can be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide input.

[0079] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138, wherein instructions 138 are executed by processor 132 to cause server computing system 130 to perform operations.

[0080] In some implementations, the server computing system 130 includes or is otherwise implemented therein one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, these devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0081] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, model 140 may be, or can otherwise include, various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Example model 140 has been referenced. Figure 1 and 2 Let's discuss this.

[0082] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 via interaction with training computing system 150, which is communicatively coupled to network 180. Training computing system 150 may be separate from server computing system 130, or may be part of server computing system 130.

[0083] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158, wherein the instructions 158 are executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0084] Training computing system 150 may include model trainer 160, which trains machine learning models 120 and / or 140 stored at user computing device 102 and / or server computing system 130 using various training or learning techniques, such as, for example, backpropagation of errors. For example, a loss function can be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update parameters over several training iterations.

[0085] In some implementations, backpropagation of the execution error may include backpropagation with time-executed truncation. The model trainer 160 may perform several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0086] Specifically, model trainer 160 can train machine learning models 120 and / or 140 based on a set of training data 162. Training data 162 may include, for example, unconstrained image data, such as "in the wild" photos.

[0087] In some implementations, training examples may be provided by the user computing device 102 if the user has consented. Therefore, in this implementation, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0088] Model trainer 160 includes computer logic for providing desired functionality. Model trainer 160 can be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some embodiments, model trainer 160 includes a program file stored on a storage device, which is loaded into memory and executed by one or more processors. In other embodiments, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium, such as a RAM hard disk or an optical or magnetic medium.

[0089] Network 180 can be any type of communication network, such as a local area network (e.g., intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. Generally, communication on Network 180 can be conducted using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL) via any type of wired and / or wireless connection.

[0090] Figure 3A An example computing system that can be used to implement this disclosure is shown. Other computing systems may also be used. For example, in some embodiments, user computing device 102 may include model trainer 160 and training dataset 162. In this embodiment, model 120 may be trained and used locally at user computing device 102. In some such embodiments, user computing device 102 may implement model trainer 160 to personalize model 120 based on user-specific data.

[0091] Figure 3B A block diagram depicts an example computing device 10 that performs according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0092] The computing device 10 includes several applications (e.g., applications 1 to N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0093] like Figure 3B As shown, each application can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0094] Figure 3C A block diagram depicts an example computing device 50 that performs according to an example embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0095] The computing device 50 includes several applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0096] The central intelligence layer comprises several machine learning models. For example, such as... Figure 3C As shown, a corresponding machine learning model (e.g., a model) can be provided for each application and managed by a central intelligence layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some embodiments, the central intelligence layer is included in or otherwise implemented by the operating system of computing device 50.

[0097] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for computing device 50. For example... Figure 3C As shown, the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0098] Additional disclosure

[0099] This paper discusses technologies related to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and divisions of tasks and functions among components. For example, the processes discussed in this paper can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0100] While the subject matter has been described in detail with reference to various specific example embodiments, each example is provided by way of explanation and not as a limitation of this disclosure. Those skilled in the art will readily understand the foregoing and will be able to make changes, variations, and equivalent substitutions to these embodiments. Therefore, the disclosure of this subject matter does not preclude modifications, variations, and / or additions to the subject matter that are readily apparent to those skilled in the art. For example, features shown or described as a part of one embodiment may be used with another embodiment to produce further embodiments. Therefore, this disclosure is intended to cover such modifications, variations, and equivalent substitutions.

Claims

1. A computational system for determining camera pose, comprising: One or more processors; and One or more non-transitory computer-readable media, which share the following storage: A neural radiation field model for machine learning, which was previously trained to model the scene; and Instructions, when executed by one or more processors, cause the computing system to perform operations, said operations including: Obtain the subject image captured by the camera; and For each of one or more pose update iterations: Obtain the current estimated pose of the camera; The neural radiation field of machine learning is used to process the data describing the current estimated pose in order to generate one or more synthetic pixels of a synthetic image of the scene from the current estimated pose. Evaluate the loss function that compares one or more synthetic pixels with one or more observed pixels included in the subject image captured by the camera; and The camera's current estimated pose is updated based at least in part on the gradient of the loss function.

2. The computing system according to claim 1, wherein, The current estimated pose is represented as exponential coordinates with six degrees of freedom.

3. The computing system according to claim 1, wherein, The loss function compares the corresponding intensity of one or more synthesized pixels with the corresponding intensity of one or more observed pixels included in the subject image captured by the camera.

4. The computing system according to claim 1, wherein, The loss function compares the corresponding chromaticity of one or more synthesized pixels with the corresponding chromaticity of one or more observed pixels included in the subject image captured by the camera.

5. The computing system according to claim 1, wherein, The loss function includes the Huber loss function.

6. The computing system according to claim 1, wherein, The loss function includes either the L1 or L2 loss function.

7. The computing system according to claim 1, wherein, The camera is associated with an autonomous robot that navigates in a scene based on the camera's estimated pose.

8. The computing system according to claim 1, wherein, Using neural radiation fields from machine learning to process data describing the current estimated pose to generate one or more synthetic pixels includes: Determine a subsampled image plane that identifies one or more sampled pixels from a plurality of possible pixels; Determine the ray and direction for each of one or more sampled pixels; and Process the data describing the ray and direction to generate one of the composite pixels for each sampled pixel.

9. The computing system according to any one of claims 1 to 8, wherein, The main image depicts the scene.

10. The computing system according to any one of claims 1 to 8, wherein, The main images depict different but similar scenes.

11. A computer-implemented method for determining camera pose, comprising: A subject image captured by a camera is obtained by a computing system comprising one or more computing devices; The neural radiation field model, previously trained by the computing system to model the scene, is accessed by the computing system. as well as For each of one or more pose update iterations: The current estimated pose of the camera is obtained by the computing system; The computing system uses a neural radiation field from machine learning to process the data describing the current estimated pose in order to generate one or more synthetic pixels of a synthetic image of the scene from the current estimated pose. The loss function that compares one or more synthetic pixels with one or more observed pixels included in the subject image captured by the camera is evaluated by the computational system. as well as The current estimated pose of the camera is updated by the computational system based at least in part on the gradient of the loss function.

12. The computer-implemented method according to claim 11, wherein, The current estimated pose is represented as exponential coordinates with six degrees of freedom.

13. The computer-implemented method according to claim 11, wherein, The loss function compares the corresponding intensity of one or more synthesized pixels with the corresponding intensity of one or more observed pixels included in the subject image captured by the camera.

14. The computer-implemented method according to claim 11, wherein, The loss function compares the corresponding chromaticity of one or more synthesized pixels with the corresponding chromaticity of one or more observed pixels included in the subject image captured by the camera.

15. The computer-implemented method according to claim 11, wherein, The loss function includes the Huber loss function.

16. The computer-implemented method according to claim 11, wherein, The loss function includes either the L1 or L2 loss function.

17. The computer-implemented method according to claim 11, wherein, The camera is associated with an autonomous robot that navigates in a scene based on the camera's estimated pose.

18. The computer-implemented method according to claim 11, wherein, Using neural radiation fields from machine learning to process data describing the current estimated pose to generate one or more synthetic pixels includes: Determine a subsampled image plane that identifies one or more sampled pixels from a plurality of possible pixels; Determine the ray and direction for each of one or more sampled pixels; and Process the data describing the ray and direction to generate one of the composite pixels for each sampled pixel.

19. The computer-implemented method according to any one of claims 11 to 18, wherein, The main image depicts the scene.

20. One or more non-transitory computer-readable media, collectively storing instructions that, when executed, cause a computing system to perform operations, said operations including: The computer system obtains the subject image captured by the camera; The neural radiation field model, previously trained by the computing system to model the scene, is accessed by the computing system. as well as For each of one or more pose update iterations: The current estimated pose of the camera is obtained by the computing system; The computing system uses a neural radiation field from machine learning to process the data describing the current estimated pose in order to generate one or more synthetic pixels of a synthetic image of the scene from the current estimated pose. The loss function that compares one or more synthetic pixels with one or more observed pixels included in the subject image captured by the camera is evaluated by the computational system. as well as The current estimated pose of the camera is updated by the computational system based at least in part on the gradient of the loss function.

Images