Neural body radiation field rendering acceleration method and device based on adaptive region division

Abstract

The present invention discloses a neural body radiation field rendering acceleration method and device based on adaptive region division. The present invention adaptively divides the target rendering image into a combination of several region blocks, then traverses each region block, and renders the color information corresponding to the block, and finally quickly generates the target rendering image based on the rendering result of each block. While ensuring high-quality and high-fidelity rendering effects, this rendering method can also adaptively allocate computing resources in a two-dimensional area. The adaptive block division of this rendering method can be completed synchronously when the target scene is created, and will not significantly increase the additional computing consumption. The computing efficiency is generally equivalent to or even better than the existing neural body radiation field rendering method. At the same time, thanks to the improvement in computing efficiency, feature maps with higher resolution can be rendered, thereby effectively alleviating the imaging artifact problem caused by excessive upsampling in the existing method.

Description

Technical Field

[0001] The present invention relates to the fields of image rendering, graphics rendering and neural rendering technology, and in particular to a neural body radiation field rendering acceleration method and device based on adaptive region division. Background Art

[0002] In recent years, neural volume rendering techniques based on neural volume radiance fields have been widely applied to 3D vision and graphics tasks such as novel perspective synthesis, scene modeling, and avatar creation. For novel perspective synthesis, for example, conventional techniques are primarily divided into image-based rendering and image-based modeling. The former, such as light field rendering, may require dense perspective input and massive data storage to ensure high-quality perspective rendering, while the latter typically requires pre-recovery or reconstruction of high-quality, high-precision 3D geometry and texture information to aid rendering, which is often quite difficult. Unlike these approaches, neural volume rendering methods based on neural volume radiance fields implicitly represent and model the radiance field of the target scene using neural networks (such as multilayer perceptrons). These inputs require only sparse perspective color images, or even single-view images, to achieve photorealistic rendering quality. They also offer advantages such as efficient model storage. However, this method suffers from low rendering efficiency due to the multiple network computations required, significantly hindering its development and widespread adoption in related applications and tasks.

[0003] There are currently three main approaches to accelerating neural volumetric radiance fields. One approach involves densely partitioning the target scene or object's area into voxels and then pre-storing the corresponding neural volumetric radiance field calculation results at each voxel to accelerate rendering. While this approach does effectively accelerate the rendering process of neural volumetric radiance fields, it significantly increases the storage cost of each model, hindering subsequent model encoding and transmission. Another approach aims to achieve acceleration by reducing the number of sampling points per ray to save computational overhead. This can be achieved by adding depth information to adaptively adjust the sampling interval for each ray and using models to predict or store the sampling start point for each ray. This approach generally requires additional computation or training to extract the corresponding auxiliary information, which increases the cost of model creation. The final approach combines neural volumetric rendering with 2D neural rendering. Specifically, instead of directly rendering a color image at the final rendering resolution, this approach typically first renders a low-resolution, high-channel feature map, then uses a 2D neural network to predict the final rendering result. Related methods can generally ensure a certain rendering quality and multi-view consistency, and the rendering efficiency can be greatly improved. However, if the resolution difference between the feature map rendered by this method and the target color map is too large, it may introduce imaging artifacts and multi-view inconsistency problems caused by excessive upsampling. Summary of the Invention

[0004] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide a neural body radiation field rendering acceleration method and device based on adaptive region division. The present invention adaptively divides the target rendering image into a combination of several region blocks, then traverses each region block, and volume renders the color information corresponding to the block, and finally quickly generates the target rendering image based on the rendering result of each block. While ensuring high-quality and high-fidelity rendering effects, this rendering method can also adaptively allocate computing resources in a two-dimensional area. The adaptive block division of this rendering method can be completed synchronously when the target scene is created, and will not significantly increase the additional computing consumption. The computing efficiency is generally equivalent to or even better than the existing volume rendering method. At the same time, thanks to the improvement in computing efficiency, feature maps with higher resolution can be rendered, thereby effectively alleviating the imaging artifact problem caused by excessive upsampling in the existing method.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] According to a first aspect of this specification, a neural network rendering acceleration method for adaptively dividing a rendering area is provided, comprising:

[0007] The target rendered image is adaptively divided into several block combinations, the block combinations are parameterized, and the block division results, i.e., the block representation parameters, are predicted based on the camera parameters. It is required that all valid areas obtained by the adaptive division can cover the entire target rendered image.

[0008] Traverse each region block and render the color information corresponding to the region block based on the neural volume radiation field. The volume rendering process is required to be differentiable for the representation parameters of the region block.

[0009] Based on the rendering results of each area block, the rendering results of the target rendering image are quickly generated.

[0010] Furthermore, the adaptive segmentation results are adjusted through self-supervised constraints. Specifically, the image information of the relevant scenes is collected in advance, the rendering results of the corresponding image information are generated as the predicted images, a loss function is established, the neural body radiation field corresponding to the relevant scenes is trained using gradient backpropagation, and the representation parameters of the regional blocks are updated, thereby achieving the adjustment of the adaptive segmentation results.

[0011] Furthermore, the loss function includes but is not limited to: constraining the predicted image to be consistent with the collected image, constraining all valid areas obtained by adaptive division to cover the entire target rendered image, constraining the overlap of different area blocks obtained by division to be as small as possible, etc.

[0012] Furthermore, during the adaptive partitioning process, a repulsive constraint is required between the regional blocks. Specifically, the following repulsive model is established for each regional block:

[0013]

[0014] In the above formula, d i (i,·) represents the repulsive force function corresponding to the i-th block, which is determined by the representation parameters of the block, d i (i,j) returns the repulsive influence of the i-th region block on the j-th region block.

[0015] Furthermore, the region block division result is predicted according to the camera parameters, where the camera parameters refer to the camera intrinsic parameters and extrinsic parameters, and all other parameters that can be converted into camera parameters should also be treated equally.

[0016] Furthermore, volume rendering is performed on each divided area block to obtain the color information of the area. The specific volume rendering process is as follows:

[0017] Back-project the current area block to obtain the three-dimensional space area for rendering the two-dimensional area block, and further divide the three-dimensional space area. The radiation field information of each divided three-dimensional area block is predicted using the following implicit function;

[0018] F θ:(c,x,d)→(f,σ)

[0019] In the above formula, F is the neural body radiation field, which is represented by a neural network structure, θ is a learnable network parameter, c is an optional conditional feature vector, x is the position information of the 3D region, d is the optional target rendering camera's viewing angle information of the 2D region, f is the feature information of the predicted 3D spatial region, and σ is the predicted corresponding density information;

[0020] Then, the feature information of each two-dimensional area block is generated using the following calculation method. The process can be described as follows:

[0021]

[0022]

[0023] In the above formula, f p represents the feature information of the two-dimensional area block obtained by volume integration, t is the integral differential element of the three-dimensional space area obtained by back projection, w(t) is the opacity of the integral differential element t, and r(·) represents the position information of the divided three-dimensional area block.

[0024] Furthermore, the back-projection method of the current region block to the three-dimensional space region includes but is not limited to ray projection, cylindrical projection, and cone projection.

[0025] Furthermore, the area block division includes but is not limited to circular area division, brush-shaped strip area division, and other indefinite-shaped area division.

[0026] For circular area division, the representation parameters of the area block are the center and radius.

[0027] For the brush-shaped strip area division, the representation parameters of the area block adopt the vector parameters of the brush, and the vector parameters include control points, radius, transparent gradient parameters, etc.

[0028] Furthermore, based on the rendering results of each area block, a predicted image of the target rendered image is generated. This process generates a feature vector describing the pixel color based on the distance from the pixel to each area block, and then uses a neural network to predict the final color value of the pixel based on the feature vector.

[0029] Furthermore, the neural body radiation field is a fully connected and convolutional neural network model, and the relevant network layers include: input layer, convolution layer, pooling layer, fully connected layer and loss layer.

[0030] According to the second aspect of this specification, a neural body radiation field rendering acceleration device based on adaptively divided areas is provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the neural body radiation field rendering acceleration method based on adaptively divided areas as described in the first aspect.

[0031] The beneficial effects of the present invention are:

[0032] 1) This method can effectively improve the rendering efficiency of neural volumetric rendering while maintaining the ability of neural volumetric rendering to implicitly model geometric information, resulting in excellent multi-view consistency in the rendering results. Furthermore, this method can achieve rendering quality comparable to or even better than that before acceleration.

[0033] 2) Thanks to the improvement in rendering efficiency, this method can effectively reduce the construction cost of the target scene, that is, reduce the relevant model training time.

[0034] 3) The entire adaptive region division process can be completed automatically without human intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a neural body radiation field rendering acceleration method based on adaptive region division is provided as an exemplary embodiment;

[0036] Figure 2 A conceptual demonstration of adaptive region partitioning provided for an exemplary embodiment;

[0037] Figure 3 A conceptual illustration of back-projection of a divided area into a rendering area provided by an exemplary embodiment;

[0038] Figure 4 This is a structural diagram of a neural body radiation field rendering acceleration device based on adaptive region division provided by an exemplary embodiment. DETAILED DESCRIPTION

[0039] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0040] It should be clear that the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0041] The terms used in the embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. The singular forms "a", "an", "the" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0042] In recent years, neural volume rendering technology based on neural volume radiation field has been widely used in 3D vision and graphics tasks such as new perspective synthesis, scene modeling, and virtual image creation. Although this method can achieve photo-level rendering quality, excellent multi-perspective consistency, and perspective-dependent rendering effects, the inefficient training process and slow rendering speed have greatly hindered the further promotion and development of this method. Therefore, this paper proposes a neural volume radiation field rendering acceleration method based on adaptive region division, such as Figure 1 As shown, the method of the present invention comprises:

[0043] The target rendered image is adaptively divided into several block combinations, the block combinations are parameterized, and the block division results, i.e., the block representation parameters, are predicted based on the camera parameters. It is required that all valid areas obtained by the adaptive division can cover the entire target rendered image.

[0044] Traverse each region block and render the color information corresponding to the region block based on the neural volume radiation field. The volume rendering process is required to be differentiable for the representation parameters of the region block.

[0045] Based on the rendering results of each area block, the rendering results of the target rendering image are quickly generated.

[0046] Specifically, the area block division includes but is not limited to circular area division, brush-shaped strip area division, and other indefinite-shaped area divisions.

[0047] For circular area division, the representation parameters of the area block may be the center and radius of the circle.

[0048] For the brush-shaped strip area division, the representation parameters of the area block may adopt the vector parameters of the brush, and the vector parameters include control points, radius, transparent gradient parameters, etc.

[0049] In one embodiment, the adaptive partitioning result is adjusted through self-supervision constraints, specifically:

[0050] Collect image information of relevant scenes in advance. Scene images include but are not limited to multi-view images taken at the same time, images taken at different times, etc.; generate rendering results corresponding to the image information as predicted images, and establish a loss function. The loss function includes but is not limited to: constraining the predicted image to be consistent with the collected image, constraining all valid areas obtained by adaptive division to cover the entire target rendered image, constraining the overlap of different area blocks obtained by division to be as small as possible, etc.; use gradient backpropagation to train the neural body radiation field corresponding to the relevant scene, update the representation parameters of the area blocks, and thus realize the adjustment of the adaptive division results.

[0051] In one embodiment, during the adaptive partitioning process, a repulsive constraint is required between the region blocks. Specifically, the following repulsive model is established for each region block:

[0052]

[0053] In the above formula, d i (i,·) represents the repulsive force function corresponding to the i-th block, which is determined by the representation parameters of the block, d i (i, j) returns the repulsive influence of the i-th region block on the j-th region block. Assuming the region block is circular, the representation parameter of the region block can be set to the center position μ i and radius γ i , then the repulsion function can be defined as:

[0054]

[0055] In one embodiment, volume rendering is performed on each divided area block to obtain color information of the area. The volume rendering process is specifically as follows:

[0056] Back-project the current area block to obtain a three-dimensional space area for rendering the two-dimensional area block. The back-projection method of the current area block to the three-dimensional space area includes but is not limited to ray projection, cylindrical projection and cone projection, etc. Figure 3 The three-dimensional space is further divided, and the radiation field information of each divided three-dimensional area block is predicted using the following implicit function;

[0057] F θ :(c,x,d)→(f,σ)

[0058] In the above formula, F is the neural body radiation field, which is represented by a neural network structure, θ is a learnable network parameter, c is an optional conditional feature vector, x is the position information of the 3D region, d is the optional target rendering camera's viewing angle information of the 2D region, f is the feature information of the predicted 3D spatial region, and σ is the predicted corresponding density information;

[0059] Then, the feature information of each two-dimensional area block is generated using the following calculation method. The process can be described as follows:

[0060]

[0061] w(t)=exp(-∫0 t σ(r(s))ds)·σ(r(t))

[0062] In the above formula, f p represents the feature information of the two-dimensional region obtained by volume integration, which includes but is not limited to color information, high-dimensional feature vectors, and high-dimensional feature vector maps. t is the integral differential element of the three-dimensional space region obtained by back projection, w(t) is the opacity of the integral differential element t, and r(·) represents the position information of the divided three-dimensional region block, such as Figure 2 shown.

[0063] Based on the rendering results of each area block, a predicted image of the target rendered image is generated. This process generates a feature vector describing the color of the pixel based on the distance from the pixel to each area block, and then uses a neural network to predict the final color value of the pixel based on the feature vector.

[0064] In one embodiment, a neural volume radiation field rendering acceleration method based on adaptively dividing a rendering area includes the following steps:

[0065] Step 1: Collect multi-view images of relevant scenes and corresponding camera intrinsic and extrinsic parameter information.

[0066] Step 2: Use a multi-layer perceptron to represent the neural body radiation field of the target scene and initialize the relevant network parameters.

[0067] Step 3: For each image in the image set, perform relevant neural rendering based on its camera information. The process is as follows:

[0068] Step 3.1: Use a circular area division method, use the center and radius as the representation parameters of the area block, and use a neural network to predict the area block division result, that is, the representation parameters of the area block, based on the camera information.

[0069] Step 3.2: Based on neural volume rendering, predict the feature vector of each region block. This process must ensure that the parameters representing the region block are differentiable. Specifically, it can be described as:

[0070]

[0071] in represents the feature vector generated by the i-th region block, μ i ,γ i Represent the center and radius respectively, f p(·) represents the volume rendering function, which requires μ i ,γ i is differentiable.

[0072] Step 3.3: For each pixel of the target rendered image, weight the feature vectors corresponding to each region block according to the distance from the pixel to the center of each region block, and use the weighted result as the feature vector of the pixel.

[0073] Step 3.4: Predict the rendering color of the corresponding pixel based on the weighted feature vector obtained above.

[0074] Step 3.5: Traverse all pixels in sequence to obtain the neural volume rendering results corresponding to the camera parameters.

[0075] Step 4: Calculate the loss function based on the collected images and predicted images, and train the network parameters of the multilayer perceptron.

[0076] Step 5: Randomly traverse the image and execute the above process until the training is completed.

[0077] Step 6: For the trained neural body radiation field of the relevant scene, given the target camera parameter information, predict the relevant rendering results.

[0078] Compared with the previous neural network rendering method, the embodiment of the present invention has the following main advantages:

[0079] 1) It can significantly reduce the calls of volume rendering integrals, thereby effectively improving rendering efficiency.

[0080] 2) Computing resources can be adaptively allocated, thereby reducing unnecessary calculations to a certain extent.

[0081] 3) The entire process can be completed in a self-supervised manner while ensuring a certain rendering quality and multi-view consistency.

[0082] Corresponding to the aforementioned embodiment of the neural body radiation field rendering acceleration method based on adaptive region division, the present invention also provides an embodiment of a neural body radiation field rendering acceleration device based on adaptive region division.

[0083] See also Figure 4 An embodiment of the present invention provides a neural body radiation field rendering acceleration device based on adaptive region division, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement the neural body radiation field rendering acceleration method based on adaptive region division in the above embodiment.

[0084] The embodiment of the neural body radiation field rendering acceleration device based on adaptively divided regions of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 4 As shown, this is a hardware structure diagram of any device with data processing capability where the neural body radiation field rendering acceleration device based on adaptive division of regions is located. Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0085] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0086] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0087] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the neural body radiation field rendering acceleration method based on adaptive region division in the above embodiment is implemented.

[0088] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0089] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0090] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0091] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0092] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0093] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A neural body radiation field rendering acceleration method based on adaptive region division, characterized in that: The method includes: The target rendered image is adaptively divided into several block combinations, the block combinations are parameterized, and the block division results, i.e., the block representation parameters, are predicted based on the camera parameters. It is required that all valid areas obtained by the adaptive division can cover the entire target rendered image. Traversing each region block, rendering the color information corresponding to the region block based on the neural volume radiation field, requiring the volume rendering process to be differentiable for the representation parameters of the region block; Based on the rendering results of each area block, the rendering results of the target rendering image are quickly generated; The adaptive partitioning results are adjusted through self-supervision constraints, specifically: Pre-collect image information of relevant scenes, generate rendering results of the corresponding image information as predicted images, establish a loss function, and constrain the predicted image to be consistent with the collected image, constrain all valid areas obtained by adaptive partitioning to cover the entire target rendered image, and constrain the overlap of different area blocks obtained by partitioning to be as small as possible. Use gradient backpropagation to train the neural body radiation field corresponding to the relevant scenes and update the representation parameters of the area blocks. During the adaptive partitioning process, the following repulsion model is established for each region block: where d i (i,·) represents the repulsion function corresponding to the i-th region block, which is determined by the representation parameters of the region block, d i (i,j) returns the repulsive influence of the i-th region block on the j-th region block; Volume rendering is performed on each divided area block to obtain the color information of the area. The specific volume rendering process is as follows: Back-project the current area block to obtain a three-dimensional spatial area for rendering the two-dimensional area block, and further divide the three-dimensional spatial area. The radiation field information of each divided three-dimensional area block is predicted using the following implicit function; F θ :(c,x,d)→(f,σ) Where F is the neural body radiation field, θ is a learnable network parameter, c is an optional conditional feature vector, x is the position information of the 3D area block, d is the optional target rendering camera's viewing angle information of the 2D area block, f is the feature information of the predicted 3D spatial area, and σ is the predicted corresponding density information; Generate feature information for each two-dimensional area block using the following formula: where f p represents the feature information of the two-dimensional area block obtained by volume integration, t is the integral differential element of the three-dimensional space area obtained by back projection, w(t) is the opacity of the integral differential element t, and r(·) represents the position information of the divided three-dimensional area block.

2. The neural body radiation field rendering acceleration method based on adaptive region division according to claim 1, characterized in that: The region block division result is predicted based on camera parameters, where the camera parameters include camera intrinsic parameters, extrinsic parameters, and all other parameters that can be converted into camera parameters.

3. The neural body radiation field rendering acceleration method based on adaptive region division according to claim 1, characterized in that: Ways of back-projecting the current region block into the three-dimensional space region include ray projection, cylindrical projection, and conical projection.

4. The neural body radiation field rendering acceleration method based on adaptive region division according to claim 1, characterized in that: The area block division includes circular area division, brush-shaped strip area division, and other indefinite shape area division; For circular area division, the representation parameters of the area block are the center and radius; For the brush-shaped strip area division, the representation parameters of the area blocks adopt brush vector parameters.

5. The neural body radiation field rendering acceleration method based on adaptive region division according to claim 1, characterized in that: Based on the rendering results of each area block, a predicted image of the target rendered image is generated. This process generates a feature vector describing the pixel color based on the distance from the pixel to each area block, and then uses a neural network to predict the final color value of the pixel based on the feature vector.

6. The neural body radiation field rendering acceleration method based on adaptive region division according to claim 1, characterized in that: The neural body radiation field is a fully connected and convolutional neural network model, and the relevant network layers include: input layer, convolution layer, pooling layer, fully connected layer and loss layer.

7. A neural body radiation field rendering acceleration device based on adaptive region division, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it is used to implement the neural body radiation field rendering acceleration method based on adaptive region division as described in any one of claims 1 to 6.

Images