Method and system for training quantized neural radiance fields

Abstract

A computer-implemented method comprising: encoding a radiance field of an object onto a machine learning model; performing a training process on the machine learning model based on a set of training images of the object to obtain a trained machine learning model, wherein the training process comprises a first training process using a plurality of first test sample points followed by a second training process using a plurality of second test sample points located within a threshold distance from a surface region of the object; obtaining a target view parameter indicative of a view direction of the object; obtaining a plurality of rays associated with a target image of the object; obtaining rendering sample points on the plurality of rays associated with the target image; and rendering a color associated with a pixel of the target image by inputting a rendering sample point to the trained machine learning model.

Description

Technical Field

[0001] This invention generally relates to the technical fields of computer vision and machine learning, and more specifically, to methods and systems for training quantized neural radiation fields. Background Technology

[0002] Neural radiation fields, or NeRF, are frameworks that allow the reconstruction and rendering of objects by utilizing their radiation fields. An object's radiation field can generally be considered as a representation or visualization of the object in a three-dimensional (3D) rendering space, through which various renderings of the object, such as images or videos, can be generated. Novel views or video animations of objects can be rendered based on radiation fields by using machine learning models such as multilayer perceptrons (MLPs).

[0003] The various rendering frameworks used to generate objects in NeRF can be computationally intensive, which greatly limits the application of NeRF technology in scenarios requiring real-time responses. Reducing the number of sampling points can speed up rendering. However, the quality of the rendering results deteriorates as the number of sampling points decreases. Summary of the Invention

[0004] In view of the limitations of the prior art described above, the present invention provides a method and system for training quantized neural radiation fields that overcomes these limitations.

[0005] One aspect of the present invention relates to a computer-implemented method. The method may include: encoding the radiation field of an object into a machine learning model using a computer device; and training the machine learning model using the computer device and based on a set of training images of the object to obtain a trained machine learning model. The training process may include a first training process using a plurality of first test sample points, followed by a second training process using a plurality of second test sample points. Each of the first and second sample points may lie on a plurality of training rays derived from the training images.

[0006] The method may further include: obtaining target view parameters indicating the viewpoint and view direction of the object; obtaining a plurality of sample rays associated with a target image of the object based on the view parameters, the target image being associated with the viewpoint and the view direction; obtaining rendered sample points on the plurality of sample rays associated with the target image by the computer device; and rendering colors associated with the pixels of the target image by inputting the rendered sample points into the trained machine learning model.

[0007] In some embodiments, the radiation field may include a three-dimensional (3D) rendering space surrounding the object.

[0008] In some embodiments, the machine learning model may be a fully connected neural network comprising one or more nodes, each node having associated weights.

[0009] In some embodiments, the machine learning model can be configured to accept the position vector and orientation vector of a point in the 3D rendering space as input, and output the density and radiance of the point. The position vector may indicate the position of the point relative to the viewpoint, and the orientation vector may indicate the relative orientation of the point relative to the viewpoint.

[0010] In some embodiments, during the second training process, the values ​​of the weights, elements of the position vector, elements of the orientation vector, the density, and the radiance are quantized.

[0011] In some embodiments, obtaining rendering sample points on the plurality of sample rays may include obtaining the rendering sample points, wherein the rendering sample points are located within the threshold distance from the surface region of the object.

[0012] In some embodiments, the surface region of the object may be obtained using the traveling cube technique based on the set of training images, and the surface region may be in the form of a triangular mesh.

[0013] In some embodiments, the threshold distance may be 2 cm.

[0014] The present invention also proposes an apparatus. The apparatus may include a processor and a memory. The memory may store instructions executable by the processor. When executed by the processor, the instructions may cause the processor to perform any of the methods described in the method embodiments.

[0015] The present invention also proposes a non-transitory storage medium. The storage medium can store instructions executable by a processor. When executed by the processor, the instructions can cause the processor to perform any of the methods in the method embodiments.

[0016] In the computer implementation method disclosed in this invention, the surface region of an object is determined based on multiple test images under a NeRF model. Then, to render a target image of the object from a given viewpoint based on the NeRF model, sample points for rendering can be collected on or near the surface region of the object. Since sample points far from the surface region of the object contribute little to the rendering of the object, this sampling scheme can achieve the same rendering quality with fewer sample points than conventional sampling methods. This specification also discloses a training scheme for the NeRF model, which includes a first training process and a subsequent second training process. During the second training process, sample points for training can be collected on or near the surface region of the object. Compared to conventional methods, this training scheme improves training efficiency. Additionally, a quantization framework is disclosed, which can be applied to one or more of the training and rendering processes. The quantization process optimizes the data in the training and / or rendering processes to allow it to be processed more efficiently by certain hardware such as artificial intelligence (AI) chips.

[0017] The foregoing general description and the following detailed description are exemplary and illustrative only, and do not limit the inventive concept. Attached Figure Description

[0018] The accompanying drawings, which are included in and form a part of this specification, illustrate embodiments consistent with the invention and, together with the specification, explain the disclosed principles. It will be apparent that these drawings present only some embodiments of the invention, and that those skilled in the art can derive other embodiments from them without any inventive effort.

[0019] Figure 1 A schematic diagram of a neural radiation field system according to various embodiments of this specification is shown.

[0020] Figure 2 A schematic diagram of an exemplary volume rendering module according to various embodiments of this specification is shown.

[0021] Figure 3 A schematic diagram of the rendering process based on some technologies is shown.

[0022] Figure 4 A schematic diagram of the rendering process according to various embodiments of this specification is shown.

[0023] Figure 5 A schematic diagram of a training strategy according to various embodiments of this specification is shown.

[0024] Figure 6 Block diagrams of computer systems according to various embodiments of this specification are shown.

[0025] Figure 7A block diagram of a computer system in which any of the embodiments described in this invention may be implemented is shown. Detailed Implementation

[0026] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments may be implemented in many forms and should not be construed as limited to those set forth herein. Rather, these embodiments are presented to provide a full and thorough understanding of the inventive concept and to fully convey the concept of the embodiments to others skilled in the art. Furthermore, the described features, structures, and characteristics may be combined in any suitable manner in one or more embodiments. In view of the inventive concept herein, those skilled in the art will recognize that various embodiments may be practiced without one or more specific details or using other methods, components, materials, etc. In some cases, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the embodiments.

[0027] The accompanying drawings are provided for illustration and explanation of the inventive concept and are not necessarily drawn to scale. In the drawings, the same reference numerals denote the same or similar elements, and therefore repeated detailed descriptions are omitted.

[0028] Neural Radiation Field (NeRF) is a framework that allows for the reconstruction and rendering of complex scenes or objects by optimizing the underlying radiation field using multiple input images. Within the NeRF framework, static scenes or objects can be represented by a continuous five-dimensional (5D) function.

[0029] To render a scene or object, the 5D functions representing the scene or object can be implemented using a fully connected neural network (such as a multilayer perceptron (MLP)). The neural network can output a position vector indicating the location of a point (e.g., 3D Euclidean coordinates (x, y, z)) and a direction vector indicating the direction of that point relative to the viewpoint (e.g., spherical coordinates). The function takes a point as input and outputs the density (ρ) at that point and the radiance at that point along that direction (e.g., the RGB value at that point). The radiance at all points in space can form a radiation field, based on which an image from any given viewpoint can be generated through volumetric rendering.

[0030] To improve the accuracy of images generated by a neural network, the network's parameters can be optimized by calibrating the generated images with training images (i.e., ground truth images). Once trained, the neural network can generate images of scenes or objects at any given viewpoint.

[0031] Figure 1 A schematic diagram of a neural radiation field (NeRF) system 100 according to various embodiments of this specification is shown. Figure 1As shown, the Neural Radiation Field System 100 may include an Object View Composition Module 110, which can be configured to generate real-time rendering of objects based on the NeRF framework.

[0032] In some embodiments, the object view compositing module 110 may be implemented in part or in whole as software, hardware, or any combination thereof. In some embodiments, the object view compositing module 110 may be implemented in part or in whole as software running on one or more computing devices or systems, such as cloud computing systems. For example, the object view compositing module 110 may be implemented in part or in whole on a cloud computing system to generate images of objects within the NeRF framework from various selected perspectives or viewpoints.

[0033] In some embodiments, the object view composition module 110 may include a training data preparation module 112, a radiation field encoding module 114, and a volume rendering module 116. Each of these modules is discussed below.

[0034] In some embodiments, such as Figure 1 As shown, the NeRF system 100 may also include one or more data memories 120. The object view synthesis module 110 may be configured to communicate with and / or operate with one or more data memories 120. The one or more data memories 120 may store various types of data associated with the object view synthesis module 110.

[0035] For example, at least one data storage device 120 may store training data used to train a machine learning model to encode the radiation field of an object onto the machine learning model. The training data may include, for example, images depicting the object from various viewpoints. For example, at least one data storage device 120 may store multiple images depicting a dog to train a machine learning model to encode the dog's radiation field onto the machine learning model. In some embodiments, at least one data storage device 120 may store data related to the radiation field, such as radiance and volume density accessible to the object view synthesis module 110.

[0036] In some embodiments, at least one data memory 120 may store various data relating to a triangular mesh and depth map accessible to the object view compositing module 110. In some embodiments, at least one data memory 120 may store machine-readable instructions (e.g., code). When executed, the instructions may cause one or more computing systems to perform training of a machine learning model or to render an image based on the radiation field. Other embodiments of the neural radiation field system 100 are contemplated.

[0037] In some embodiments, the training data preparation module 112 can be configured to generate training data for training a machine learning model to encode the radiation field of an object onto the machine learning model. Typically, the radiation field of an object can be a representation or visualization of the object in a three-dimensional rendering space through which various views of the object can be rendered (e.g., synthesized, constructed, etc.). The training data used to encode the radiation field of the object can include a set of images depicting the object at various viewpoints. For example, a first image in a set of images may depict the object in a frontal view, a second image in the set of images may depict the object in a side view, a third image in the set of images may depict the object in a top view, and so on.

[0038] To reduce the complexity and time required for training machine learning models, the training data preparation module 112 can convert a set of images into a continuous five-dimensional (5D) representation. In the continuous 5D representation, each pixel in each image of the set of images can be represented by a position vector and a direction vector. The position vector can be, for example, Euclidean coordinates (x, y, z), and the direction vector can be, for example, spherical coordinates. Thus, each pixel in each image of the set of images can be represented by parameters x, y, z, θ, and φ, or by five dimensions, and the set of images can be represented by a continuous string of parameters x, y, z, θ, and φ.

[0039] Representing training data in this way significantly reduces the dimensionality of training machine learning models to encode the radiation field of objects, thereby reducing the time required to train the machine learning models. In some embodiments, the position and orientation vectors of pixels in an image can be determined based on the pose associated with the image. The image pose is an estimate of the position and orientation (or direction) of the objects depicted in the image from the center of the camera that captured the image.

[0040] In one implementation, the pose of an image can be estimated based on Structure of Motion (SfM) technology. In another implementation, the pose of an image can be estimated based on Simultaneous Localization and Mapping (SLAM) technology. Other methods for estimating the pose of an image are conceivable, and this specification is not limiting in this regard.

[0041] In some embodiments, the radiation field encoding module 114 can be configured to encode the radiation field of an object onto a machine learning model based on training data provided by the training data preparation module 112. Once the radiation field of the object is encoded onto the machine learning model, the machine learning model can be queried to output the radiance and volume density associated with points in the radiation field from various advantageous points. Typically, an advantageous point can be a point in the radiation field (i.e., 3D rendering space) from which imaginary rays can be injected into the radiation field in directions passing through that point. Thus, an advantageous point can have a position vector indicating its location in the radiation field and a direction vector indicating its direction.

[0042] As an illustrative example, in some embodiments, a machine learning model can be queried based on advantageous points including position vectors connected to direction vectors. Imaginary rays can be generated to travel through the radiation field at the points indicated by the position vectors and in the directions indicated by the direction vectors. In this example, the machine learning model can output radiance and volume density, as well as the points in the radiation field through which the imaginary rays have traveled.

[0043] In some embodiments, a fully connected neural network, such as a multilayer perceptron (MLP), can be used to implement the machine learning model. For example, in one embodiment, a machine learning model can be implemented using a neural network comprising nine fully connected perceptron layers. This neural network can be trained to encode the radiation field of an object. In this embodiment, the neural network can take the position vector corresponding to a point as input and output the volume density and feature vector of the point at the eighth layer of the neural network. The feature vector can then be concatenated with the orientation vector corresponding to that point and passed to the last layer of the neural network to output the radiance of that point. Other machine learning models are conceivable, and this specification is not limiting in this regard.

[0044] In some embodiments, the machine learning model configured by the radiation field encoding module 114 can be represented as follows:

[0045] f(m,s)=[ρ,r] (1)

[0046] Where m is the position vector of a point in the radiation field, s is the direction vector of a point in the radiation field, ρ is the volume density along the direction of the direction vector in the radiation field, and r is the radiance along the direction of the direction vector in the radiation field. In this respect, the machine learning model can be represented as a function f, which takes the position vector and direction vector as input and outputs the radiance and volume density along the direction of the direction vector in the radiation field.

[0047] A machine learning model can include multiple parameters. In an embodiment where the machine learning model is a fully connected neural network, these parameters can be weights of the neural network, each weight associated with a node in the network. During training of the machine learning model, the parameters associated with the model can be optimized, for example, through backpropagation, such that the output from the function f converges to a reference radiation field (e.g., the ground truth radiation field of an image). Once the output from the function f has sufficiently converged to the reference radiation field (e.g., the difference between the output and the reference field is less than a threshold), the training of the machine learning model is considered complete, and the parameters of the model become fixed. The trained machine learning model can output radiance and volume density (e.g., ρ, r) in any direction corresponding to any point (e.g., m, s) in the radiation field.

[0048] In some embodiments, to prevent excessive smoothing of the radiation field, position encoding (PE) techniques can be applied during the training of the machine learning model. PE transforms the position vector m and direction vector s from a low-dimensional space (e.g., 5D space) to a higher-dimensional space to increase the fidelity of the radiation field. In some embodiments, PE can be implemented based on a sine expression as shown below:

[0049] γ(x)=[sin(2 0 x),cos(2 0 x),sin(2 1 x),cos(2 1 x),...,sin(2 L x),cos(2 L x)] (2)

[0050] Here, L is a hyperparameter. In one implementation, L can be set to 9 for γ(m) and 4 for γ(s). In this implementation, PE technology allows machine learning models to take 76-dimensional vectors instead of 5-dimensional vectors (e.g., x, y, z, θ, φ) as input to output radiance and volume density. PE can prevent the signal from being over-smoothed by MLPs, a typical drawback of MLPs.

[0051] In some embodiments, the volume rendering module 116 can be configured to generate an object rendering (e.g., an image, video, etc.) based on the object's radiation field. The volume rendering module 116 can generate a depth map of the object based on the volume density associated with the radiation field.

[0052] Figure 2 A schematic diagram of an exemplary volume rendering module according to various embodiments of this specification is shown. In some embodiments, Figure 1 The volume rendering module 116 can be implemented as Figure 2 Volume rendering module 200.

[0053] like Figure 2 As shown, in some embodiments, the volume rendering module 200 may include a depth map generation module 202, a viewpoint generation module 204, and an image rendering module 206. Each of these modules will be discussed below.

[0054] In some embodiments, the depth map generation module 202 can be configured to generate a depth map of an object based on its radiation field. Initially, the depth map generation module 202 can evaluate the volume density of points associated with the object's radiation field. Volume density can represent the opacity of points. The depth map generation module 202 can discretize the radiation field into multiple voxels. A voxel can be a unit of graphical information in three-dimensional space, such as a pixel in a two-dimensional image. The depth map generation module 202 can obtain the volume density associated with each voxel by querying a machine learning model that encodes the radiation field to output the volume density of voxels.

[0055] Based on the volume density of multiple voxels, the depth map generation module 202 can generate the surface of an object (e.g., an isosurface). In one example, the radiation field can be discretized, and the density of each discretized voxel can be evaluated. The isosurface in the form of a triangular mesh can then be extracted using a traveling cube technique.

[0056] Generally, voxels corresponding to the surface of an object have approximately equal volume density, voxels corresponding to the region near the surface of the object can have high volume density, and voxels corresponding to the region far from the surface of the object can have low volume density.

[0057] Based on these principles, a triangular mesh of an object can be generated. Accordingly, a depth map of the object can be generated based on the triangular mesh using conventional techniques.

[0058] In some embodiments, the view generation module 204 can be configured to generate a camera transformation matrix and a projection matrix, which are used to render an image of an object through its radiation field. The camera transformation matrix can represent the transformation that places the camera in the correct position and orientation in world space. The projection matrix can represent the transformation that transforms pixel information (e.g., vertices) from camera space to device space (the space on which the resulting image is displayed).

[0059] Camera transformation and projection matrices can be generated based on points (e.g., viewpoints) associated with the radiation field of the viewed object. For example, the radiation field can depict artifacts. In this example, a point in the radiation field can be selected such that an artifact is located and oriented within the radiation field from that point's viewpoint (e.g., a "framing" artifact). In this respect, the transformation and projection matrices can be transformations that, when combined, transform the vertices of the object from the radiation field to a two-dimensional image space. An image of the object can then be rendered using the image space.

[0060] In some embodiments, the image rendering module 206 can be configured to render an image of an object for a given transformation matrix and a given projection matrix.

[0061] Typically, image rendering module 204 can render images using volumetric rendering techniques. In such techniques, images are rendered (constructed) by synthesizing pixels in an image space indicated by camera transformation and projection matrices. Image rendering module 206 can render the color of each pixel based on an absorption and emission particle model, where the color of the pixel is determined based on the light injected into the radiation field of the object.

[0062] In some embodiments, given a camera transformation matrix and a projection matrix, volumetric rendering can be used to render each pixel, and the final image can be constructed by compositing all pixels. Ray rendering can be performed using line integration, assuming an absorbing and emitting particle model. In one example, the extent of the integration path can be defined by the intersections of the ray with the near and far planes. The near and far planes can be determined by the actual size of the scene. Rendering can be performed using the following equation:

[0063]

[0064] in, This represents a line segment that begins at the far plane and ends at the near plane. C(l) represents the evaluated color of ray l.

[0065] In some embodiments, the image rendering module 204 may render the color of a pixel based on the following numerical orthogonal expression:

[0066]

[0067] in, It is the color of the pixel, ρ i The volume density r of the i-th voxel is queried through a machine learning model that encodes the radiation field. i It is the radiance of the i-th voxel queried through a machine learning model that encodes the radiation field, and δ i It is the Euclidean distance between the i-th and (i+1)-th voxels, and N is the total number of voxels along ray l.

[0068] One limitation of the rendering process described above is its heavy computational burden. In one example, in volumetric rendering techniques, N can be 196. Thus, as shown in the expression, rendering the color of the pixels in the image requires 196 summation iterations (i.e., N = 196), taking into account the radiosity and volume density of 196 voxels. Therefore, these volumetric rendering techniques may be computationally impractical for real-time image rendering.

[0069] Several improvements have been made to increase the efficiency of the rendering process. For example, instead of using a single neural network to represent the scene, a hierarchical volumetric sampling scheme using two neural networks (a "coarse" network and a "fine" network) can be used to improve rendering efficiency.

[0070] Figure 3 A schematic diagram illustrating the rendering process based on several technologies is shown. For example... Figure 3 As shown, the rendering process 300 employs a layered volumetric sampling scheme, which uses two sampling schemes and two neural networks to improve rendering efficiency.

[0071] like Figure 3 As shown, the rendering process 300 may include steps 302 to 308. In step 302, multiple rays may be constructed based on a given viewpoint. In step 304, a first set of sample points may be obtained. The first set of sample points may be any point on a ray located between the near plane and the far plane.

[0072] After obtaining the first set of sample points, the position vector and orientation vector associated with each sample point in the first set of sample points can be provided to the first neural network (i.e., the first MLP) to determine the volume density ρ and radiance r at each sample point in the first set of sample points.

[0073] In step 306, a second set of sample points can be obtained based on the output from the first MLP. In one example, using equation (4), the estimated color at each sample point in the first set of sample points can be obtained based on the density and radiance from the first MLP. Then, the estimated color can be based on each sample point in the first set of sample points. The estimated color distribution in space is determined. A second set of sampling points can then be obtained based on this estimated color distribution. In one example, a second set of sampling points can be obtained from portions of a volume that are likely more relevant to the final rendered result (e.g., the volume containing visible content), based on the color distribution. The position and orientation vectors associated with the second set of sample points can then be input into a second neural network (i.e., a second MLP).

[0074] In step 308, the final volume rendering result can be obtained based on the output from the second neural network.

[0075] This specification provides an improved workflow for rendering. Figure 4 A schematic diagram of an improved rendering process according to various embodiments of this specification is shown.

[0076] like Figure 4 As shown, the improved rendering process 400 may include the following steps 402 to 410.

[0077] In step 402, multiple pixel rays can be constructed based on a given viewpoint and a given viewing direction. The viewpoint can be a point in the target image of the observed object, and the viewing direction can be the direction in which the target image is observed. The viewpoint and viewing direction can be determined based on target view parameters. In one example, the view parameters may include a 3D position in space representing the viewpoint and a 2D vector representing the viewing direction. Other types of view parameters are conceivable, and this specification is not limiting in this regard.

[0078] In step 404, a set of sampling points can be obtained. This set of sampling points can lie between the near plane and the far plane along each pixel ray. The near plane and the far plane can be two planes defined in the space involved in the rendering of the target image. In one example, the near plane and the far plane can be determined in advance based on specific requirements. Other methods for determining the near plane and the far plane are conceivable, and this specification is not limited to them.

[0079] Each sampling point may include information containing a position vector m and a direction vector s at that sampling point. In one example, the position vector m may be the 3D Euclidean coordinates (x, y, z) of the sampling point, and the direction vector s may be spherical coordinates indicating the direction. The tuple. In one example, the position vector m can represent the position of the sampling point relative to the viewer's position determined in step 402, and the direction vector s can represent the relative direction between the sampling point and the viewer's position.

[0080] In step 404, a set of sampling points can be obtained using a modified ray sampling scheme along the pixel rays. In the modified ray sampling scheme, this set of sampling points can be points on the pixel rays and close to the surface region of the object. Various methods can be used to determine the surface region of the object. In one example, the surface region can be determined based on a NeRF model using an existing image of the object. For example, based on an existing image of the object, the volume density of voxels in the space surrounding the object can be determined, and the surface region of the object can include voxels with approximately equal volume densities (e.g., volume densities within a preset range). In one example, a traveling cubes technique can be used to extract the surface region of the object based on volume density, and the surface region of the object can be represented by a triangular mesh. In another example, the surface region of the object can be provided by a third-party source (e.g., by a user).

[0081] If the sampling point is on the surface area or within a threshold distance from the surface area, the sampling point is considered close to the surface area of ​​the object. The threshold distance can be determined according to specific needs. In one example, the threshold distance could be 2 cm. Other threshold distances are conceivable, and this specification does not limit this.

[0082] In step 406, the PE technique can be applied to the sampling points to increase their dimensionality. In one example, the sine expression of equation (2) can be used for the PE technique. In equation (2), L can be set to 9 for γ(m) and 4 for γ(s). Other implementations of the PE technique are conceivable, and this specification is not limiting in this regard.

[0083] In step 408, after processing the sample points using PE technology, the processed data γ(m) and γ(s) can be input into a neural network model such as an MLP to determine the radiance and density of each voxel in the 3D space surrounding the object. The size of the 3D space and the size of each voxel can be determined according to specific needs and are not limited thereto.

[0084] In step 410, based on the density and radiance of the voxels, the evaluation color C(l) for each ray l can be obtained according to equation (4). Then, the target image can be obtained using existing volumetric rendering techniques based on C(l) for each ray l.

[0085] Figure 4 The rendering process shown employs a modified sampling scheme, where only points located on or near the surface region of an object are sampled. This is because points far from the surface region of the object contribute very little (if anything) to the rendering result. The modified sampling scheme can significantly reduce the number of sampling points required, and thus reduce the computational complexity associated with rendering, without compromising the rendering result.

[0086] Based on the above inventive concept, this specification also proposes a training process for training neural networks in the NeRF framework. Figure 5 A schematic diagram of a training strategy according to various embodiments of this specification is shown.

[0087] like Figure 5 As shown, the training process may include a first training process, followed by a second training process. The first training process may employ a first sampling method to collect multiple first test sample points for training. The second training process may employ a modified sampling method, different from the first sampling method, to collect multiple second test sample points for training.

[0088] In one example, the multiple first test sample points can be sample points obtained from each ray between the near plane and the far plane. Various methods can be used to determine the location of the first sample point along each ray. In one example, the first test sample points can be voxels uniformly distributed along each ray between the near plane and the far plane. The distance between adjacent first sample points on each ray can be determined according to specific needs and is not limited in this specification.

[0089] Multiple second sample points can be sample points located on or near the surface area of ​​the object. In one example, the second sample points can be distributed within a threshold distance from the surface area of ​​the object. The threshold distance can be determined according to specific needs. In one example, the threshold distance can be 2 cm. Other threshold distances are conceivable, and this specification is not limited to them.

[0090] The goal of the training process can be to minimize the difference between the image output by the neural network and the ground truth image. In one example, the goal of the training process is to minimize the loss function. It can be represented as:

[0091]

[0092] in, It is a set of pixel rays used in training, along with C(r) and These are the ground truth color and predicted color of the image of pixel ray r, respectively, obtained by the neural network.

[0093] This specification also provides a quantization process to accelerate the rendering process. The quantization process may involve quantizing the values ​​of the weights, 3D position, density, individual elements of the orientation vector, and radiosity of the neural network. That is, the quantization process can restrict the values ​​of these parameters to multiple discrete values. In one example, the quantization process can make these parameters integers, and can be implemented by assigning each of the individual elements of the weights, 3D position, density, orientation vector, and radiosity to the nearest integer to its actual value. Other implementations of the quantization process are conceivable, and this specification is not limiting in this regard.

[0094] The quantization process can be applied to one or more of the first training, second training, and rendering processes. In one example, the quantization process may be applied only to the second training process. The quantization process optimizes the data in the training and / or rendering processes to allow it to be processed more efficiently by certain hardware such as artificial intelligence (AI) chips.

[0095] In the computer implementation method disclosed in this invention, the surface region of an object is determined based on multiple test images under the NeRF model. Then, in order to render a target image of the object based on the NeRF model at a given viewpoint, sample points for rendering can be collected on or near the surface region of the object. Since sample points far from the surface region of the object contribute little to the rendering of the object, this sampling scheme can achieve the same rendering quality with fewer sample points than conventional sampling methods. This specification also discloses a training scheme for the NeRF model, which includes a first training process followed by a second training process. During the second training process, sample points for training can be collected on or near the surface region of the object. This training scheme improves training efficiency compared to conventional methods. Additionally, a quantization framework is disclosed, which can be applied to one or more of the training and rendering processes. The quantization process optimizes the data in the training and / or rendering processes to allow it to be processed more efficiently by certain hardware such as artificial intelligence (AI) chips.

[0096] Figure 6 Block diagrams of computer systems according to various embodiments of this specification are shown.

[0097] like Figure 6 As shown, the computer system 600 may include a training module 602, an acquisition module 604, and a rendering module 606.

[0098] The training module 602 can be configured to encode the radiation field of an object into a machine learning model; and to train the machine learning model based on a set of training images of the object to obtain a trained machine learning model, wherein the training process includes a first training process using a plurality of first test sample points, followed by a second training process using a plurality of second test sample points, each of the first and second test sample points being located on a plurality of rays derived from the training images, the second test sample points being located within a threshold distance from a surface region of the object.

[0099] The acquisition module 604 can be configured to acquire target view parameters indicating the view direction of an object; acquire multiple rays associated with a target image of the object based on the view parameters, the target image being associated with the view direction; and acquire rendered sample points on the multiple rays associated with the target image.

[0100] The rendering module 606 can be configured to render colors associated with pixels of the target image by inputting rendered sample points into a trained machine learning model.

[0101] Based on the inventive concept presented in the method embodiments, this specification also provides an apparatus. The apparatus may include a processor and a memory. The memory may store instructions executable by the processor. When executed by the processor, the instructions may cause the processor to perform any of the methods described in the above method embodiments.

[0102] Based on the inventive concept presented in the method embodiments, this specification also provides a non-transitory storage medium. The storage medium can store instructions executable by a processor. When executed by a processor, the instructions can cause the processor to perform any of the methods described in the above method embodiments.

[0103] Figure 7 This is a block diagram illustrating a computer system 700 on which any embodiment of the invention described herein may be implemented. The computer system 700 may include a bus 702 or other communication mechanism for transmitting information, and one or more hardware processors 704 coupled to the bus 702 for processing information. The hardware processors 704 may be, for example, one or more general-purpose microprocessors.

[0104] Computer system 700 may also include main memory 706 coupled to bus 702, such as random access memory (RAM), cache, and / or other dynamic storage devices, for storing information and instructions to be executed by processor 704. Main memory 706 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 704. When stored in storage media accessible to processor 704, such instructions make computer system 700 a dedicated machine customized to perform the operations specified in the instructions. Main memory 706 may include non-volatile media and / or volatile media. Non-volatile media may include, for example, optical discs or magnetic disks. Volatile media may include dynamic memory. Common forms of media may include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM and EPROM, FLASH-EPROM, NVRAM, any other memory chips or cassette tapes and their networking versions.

[0105] Computer system 700 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which, in combination with the computer system, enable computer system 700 or program computer system 700 as a special-purpose machine. According to one embodiment, the techniques of the present invention are executed by computer system 700 in response to processor 704 executing one or more sequences of one or more instructions contained in main memory 706. Such instructions may be read into main memory 706 from another storage medium (such as storage device 708). Execution of the instruction sequence contained in main memory 706 causes processor 704 to perform the method steps described herein. For example, Figure 1 and Figure 2 The method steps shown and described in conjunction with these figures can be implemented by computer program instructions stored in main memory 706. When these instructions are executed by processor 704, they can perform actions such as... Figure 3 , Figure 4 and Figure 5 The steps are shown. In alternative embodiments, hardwired circuitry may be used instead of software instructions or in combination with software instructions.

[0106] Computer system 700 may also include a communication interface 710 coupled to bus 702. Communication interface 710 can provide bidirectional data communication coupled to one or more network links connected to one or more networks. As another example, communication interface 710 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN) component for communication with a WAN) network. A wireless link may also be implemented.

[0107] The execution of certain operations can be distributed across processors, residing not only within a single machine but also deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may be located in a single geographic location (e.g., within a home environment, office environment, or server cluster). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0108] Some embodiments are described in this invention as including logic or components. Components may constitute software components (e.g., code embodied in a machine-readable medium) or hardware components (e.g., tangible units capable of performing certain operations that can be configured or arranged in some physical manner).

[0109] While examples and features of the disclosed principles have been described in this invention, modifications, adaptations, and other implementations are possible without departing from the spirit and scope of the disclosed embodiments. Furthermore, the terms “comprising,” “having,” “containing,” and “including,” and other similar forms, are intended to be equivalent in meaning and are open-ended, as one or more items following any of these terms do not imply an exhaustive list of such items or a limitation to only the listed items. It must also be noted that, unless the context clearly specifies otherwise, the singular forms “a,” “an,” and “described” as used in this invention and the appended claims include plural references.

Claims

1. A computer-implemented method, characterized in that, include: The radiation field of an object is encoded into a machine learning model by computer equipment; A training process is performed on the machine learning model using the computer device and based on a set of training images of the object to obtain a trained machine learning model. The training process includes a first training process using a plurality of first test sample points, followed by a second training process using a plurality of second test sample points. The first training process employs a first sampling method to collect the plurality of first test sample points for training, and the second training process employs a modified sampling method different from the first sampling method to collect the plurality of second test sample points for training. Each of the first and second test sample points lies on a plurality of training rays derived from the training images. The first test sample points are sample points obtained from each ray between the near and far planes, and the second test sample points are located within a threshold distance from the surface region of the object. During the second training process, the weights of the machine learning model, the elements of the position vector and orientation vector of the second test sample points, and the density and radiance values ​​output by the machine learning model are quantized. Obtain target view parameters that indicate the viewpoint and view direction of the object; Based on the view parameters, multiple sample rays associated with a target image of the object are obtained, the target image being associated with the viewpoint and the view direction; The computer device obtains the rendered sample points on the plurality of sample rays; and The colors associated with pixels in the target image are rendered by inputting the rendered sample points into the trained machine learning model.

2. The method according to claim 1, characterized in that, The radiation field includes a three-dimensional (3D) rendering space surrounding the object.

3. The method according to claim 2, characterized in that, The machine learning model is a fully connected neural network consisting of one or more nodes, each with associated weights.

4. The method according to claim 3, characterized in that, The machine learning model is configured to accept the position vector and orientation vector of a point in the 3D rendering space as input, and output the density and radiance of the point, wherein the position vector indicates the position of the point relative to the viewpoint, and the orientation vector indicates the relative orientation of the point relative to the viewpoint.

5. The method according to claim 4, characterized in that, During the second training process, the values ​​of the weights, the elements of the position vector, the elements of the direction vector, the density, and the radiance are quantized.

6. The method according to claim 5, characterized in that, Obtaining rendering sample points on the plurality of sample rays includes: obtaining the rendering sample points, wherein the rendering sample points are located within the threshold distance from the surface region of the object.

7. The method according to claim 5, characterized in that, The surface region of the object was obtained using the traveling cube technique based on the set of training images, and the surface region is in the form of a triangular mesh.

8. The method according to claim 5, characterized in that, The threshold distance is 2cm.

9. A device, characterized in that, include: processor; A memory storing instructions executable by the processor, wherein, when executed by the processor, the instructions cause the processor to perform operations, the operations including: Encode the radiation field of an object into a machine learning model; The machine learning model is trained based on a set of training images of the object to obtain a trained machine learning model. The training process includes a first training process using multiple first test sample points, followed by a second training process using multiple second test sample points. The first training process uses a first sampling method to collect the multiple first test sample points for training, and the second training process uses a modified sampling method different from the first sampling method to collect the multiple second test sample points for training. Each of the first and second test sample points lies on multiple training rays derived from the training images. The first test sample points are obtained from each ray between the near and far planes, and the second test sample points are located within a threshold distance from the surface region of the object. During the second training process, the weights of the machine learning model, the elements of the position vector and orientation vector of the second test sample points, and the density and radiance values ​​output by the machine learning model are quantized. Obtain target view parameters that indicate the viewpoint and view direction of the object; Based on the view parameters, multiple sample rays associated with a target image of the object are obtained, the target image being associated with the viewpoint and the view direction; Obtain the rendered sample points on the multiple sample rays; and The colors associated with pixels in the target image are rendered by inputting the rendered sample points into the trained machine learning model.

10. The device according to claim 9, characterized in that, The radiation field includes a three-dimensional (3D) rendering space surrounding the object.

11. The device according to claim 10, characterized in that, The machine learning model is a fully connected neural network consisting of one or more nodes, each with associated weights.

12. The device according to claim 11, characterized in that, The machine learning model is configured to accept the position vector and orientation vector of a point in the 3D rendering space as input, and output the density and radiance of the point, wherein the position vector indicates the position of the point relative to the viewpoint, and the orientation vector indicates the relative orientation of the point relative to the viewpoint.

13. The device according to claim 12, characterized in that, During the second training process, the weights, elements of the position vector, elements of the direction vector, the density, and the radiance are quantized.

14. The device according to claim 13, characterized in that, Obtaining rendering sample points on the plurality of sample rays includes: obtaining the rendering sample points, wherein the sample points are located within the threshold distance from the surface region of the object.

15. The device according to claim 13, characterized in that, The surface region of the object was obtained using the traveling cube technique based on the set of training images, and the surface region is in the form of a triangular mesh.

16. The device according to claim 13, characterized in that, The threshold distance is 2cm.

17. A non-transitory storage medium for storing instructions executable by a processor, wherein, When executed by the processor, the instructions cause the processor to perform operations, the operations including: Encode the radiation field of an object into a machine learning model; The machine learning model is trained based on a set of training images of the object to obtain a trained machine learning model. The training process includes a first training process using multiple first test sample points, followed by a second training process using multiple second test sample points. The first training process uses a first sampling method to collect the multiple first test sample points for training, and the second training process uses a modified sampling method different from the first sampling method to collect the multiple second test sample points for training. Each of the first and second test sample points lies on multiple training rays derived from the training images. The first test sample points are obtained from each ray between the near and far planes, and the second test sample points are located within a threshold distance from the surface region of the object. During the second training process, the weights of the machine learning model, the elements of the position vector and orientation vector of the second test sample points, and the density and radiance values ​​output by the machine learning model are quantized. Obtain target view parameters that indicate the viewpoint and view direction of the object; Based on the view parameters, multiple sample rays associated with a target image of the object are obtained, the target image being associated with the viewpoint and the view direction; Obtain rendered sample points on the plurality of sample rays associated with the target image; and The colors associated with pixels in the target image are rendered by inputting the rendered sample points into the trained machine learning model.

18. The storage medium according to claim 17, wherein, The radiation field includes a three-dimensional (3D) rendering space surrounding the object.

19. The storage medium according to claim 18, wherein, The machine learning model is a fully connected neural network consisting of one or more nodes, each with associated weights.

20. The storage medium according to claim 19, wherein, The machine learning model is configured to accept the position vector and direction vector of a point in the 3D rendering space as input, and output the density and radiance of the point, wherein the position vector indicates the position of the point relative to the viewpoint, and the direction vector indicates the relative direction of the point relative to the viewpoint.

Images