A scene-adaptive foveated neural radiance field rendering method and system

Abstract

The present invention relates to a scene-adaptive foveated neural radiation field rendering method and system, the method comprising: S1: constructing a scene visual perception sensitivity body based on the viewpoint position, direction of all images in the training set of the current scene, and the saliency value and depth value of the image pixels; sampling the ellipsoid radius according to the scene visual perception sensitivity body, and constructing a multi-resolution ellipsoid neural radiation field representation structure; S2: constructing a foveated neural radiation field rendering network based on rectangular mapping, which is used to generate a foveated rendered image under the current viewpoint; S3: constructing a loss function for optimizing the parameters of the foveated neural radiation field rendering network based on rectangular mapping, while keeping the surrounding image structure information of the foveated rendered image intact, and relaxing the constraints on the quality of the surrounding images. The method provided by the present invention can be based on a small number of scene shooting images, and synthesize high-quality foveated rendering results of the scene at any viewing angle for a near-eye display device at a high frame rate.

Description

Technical Field

[0001] The present invention relates to the field of computer graphics, and in particular to a scene-adaptive foveated neural radiation field rendering method and system. Background Art

[0002] With the development of computer graphics and deep learning technology, researchers have proposed a series of neural rendering technologies, combining traditional rendering technology with learning-based three-dimensional scene representation technology, and achieved results far beyond traditional rendering technology. The most representative of these is the neural radiation field technology. In 2021, Mildenhall et al. proposed the neural radiation field NeRF to generate rendered images of complex scenes from a new perspective. NeRF uses a fully connected deep network to represent the scene. During training, it uses a set of multi-perspective object images as data to optimize the neural network to achieve reconstruction of the three-dimensional scene. When rendering new perspective images, NeRF takes the three-dimensional position and two-dimensional direction of the perspective as input, outputs the color and volume density of each voxel position in the scene, and then renders the scene image from the new perspective through the volume rendering method. Experimental results show that compared with the traditional neural rendering method, NeRF improves the rendering quality by 18%-27%. In response to the problem that NeRF is prone to blurring and distortion when processing training / test images of different resolutions, Barron et al. proposed Mip-NeRF in 2021. Mip-NeRF extends NeRF, represents scenes with continuous values, and replaces ray tracing in the voxel rendering step in NeRF with cone tracing. Experimental results show that compared with NeRF, Mip-NeRF reduces the error rate of the dataset by 17%. In 2021, Park et al. proposed Nerfies to address the problem that NeRF can only handle static scenes but not dynamic scenes. Nerfies maps the input frame coordinates to deformed canonical coordinates by introducing an additional multi-layer perceptron (MLP), and adds elastic regularization, background regularization, and coarse-to-fine deformation regularization through adaptive mask position encoding, thereby achieving dynamic scene reconstruction and new perspective rendering. In response to the problem of long rendering time of NeRF, Yu et al. proposed an accelerator architecture specifically customized for NeRF rendering in 2022-ICARUS. ICARUS uses a full optical core (PLCore) to execute the complete NeRF pipeline, while converting fully connected operations into approximate reconfigurable multiple constant multiplications (MCMs) to improve computational efficiency.

[0003] In order to further improve the synthesis performance of the neural radiation field-based rendering method to achieve real-time frame rate, Deng et al. proposed a foveation-based neural radiation field representation and image synthesis method (FovNeRF) in 2022, which uses foveation rendering to accelerate the neural radiation field rendering technology. Specifically, this method constructs a user-centered, inside-out concentric sphere three-dimensional neural radiation field representation structure, and then synthesizes the foveation rendering result of the current viewpoint in real time based on the neural radiation field representation structure combined with the visual acuity of the human eye. However, the concentric sphere three-dimensional neural radiation field representation structure of FovNeRF is difficult to represent the radiance of complex scenes well, and this method requires two image generation networks to synthesize the foveation rendering results, which is difficult to meet the frame rate requirements of near-eye display devices. Therefore, how to improve the synthesis performance of the foveation image without reducing the synthesis quality of the foveation image, so as to meet the frame rate requirements of the near-eye display device, has become an urgent problem to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a scene-adaptive foveated neural radiation field rendering method and system.

[0005] The technical solution of the present invention is: a scene-adaptive foveated neural radiation field rendering method, comprising:

[0006] Step S1: estimating the saliency values ​​and depth values ​​of all image pixels in the training set of the current scene; constructing a scene visual perception sensitivity body based on the viewpoint position, direction, and saliency values ​​and depth values ​​of the image pixels of all images; sampling the ellipsoid radius according to the scene visual perception sensitivity body, and constructing a multi-resolution ellipsoid neural radiation field representation structure according to the sampled ellipsoid radius;

[0007] Step S2: constructing a foveated neural radiation field rendering network based on rectangular mapping, including: a rectangular mapping encoding network module, an inference network module and an inverse rectangular mapping decoding network module; given the current viewpoint position and direction, obtaining a scene image corresponding to the current viewpoint position and direction in a training set, inputting the current viewpoint position and direction, the width and height of the output image and the multi-resolution ellipsoid neural radiation field representation structure into the foveated neural radiation field rendering network based on rectangular mapping, and generating a foveated rendered image under the current viewpoint;

[0008] Step S3: constructing a loss function for optimizing the parameters of the foveated neural radiation field rendering network based on rectangular mapping, so as to relax the constraints on the image quality of the surrounding area while keeping the structural information of the foveated rendered image in the surrounding area intact, thereby improving the quality of the foveated rendered image.

[0009] Compared with the prior art, the present invention has the following advantages:

[0010] 1. The present invention discloses a scene-adaptive foveation neural radiation field rendering method, which implements a multi-resolution ellipsoid neural radiation field representation structure, adaptively adjusts the ellipsoid radius according to the complex scene content to increase the sphere density in the scene's high visual perception sensitive area, and improves the radiation field representation capability of complex scenes. Under the condition of the same number of sphere layers, compared with the concentric sphere structure of the existing FovNeRF method, the present invention improves the radiation field representation capability of complex scenes.

[0011] 2. The present invention designs and implements a foveated neural radiation field rendering network based on rectangular mapping, integrates rectangular mapping into the foveated neural radiation field rendering network, and uses a single end-to-end network structure to efficiently synthesize foveated rendered images. Under the condition of similar synthesis quality, compared with the FovNeRF method, the present invention can synthesize high-quality foveated rendered images for complex scenes at a high frame rate (≥60FPS) in a near-eye display device, and the synthesis performance is improved by 1.3-1.8 times.

[0012] 3. The present invention constructs a loss function to optimize the parameters of the foveation neural radiation field rendering network based on rectangular mapping, and achieves the premise that the parameter quantity satisfies the network to synthesize the foveation image at a high frame rate while maintaining the high-quality synthesis result of the foveation area. The training data required by the present invention is consistent with the traditional neural radiation field rendering method, and there is no need to reconstruct the foveation image for the target scene for training. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of a scene-adaptive foveated neural radiance field rendering method in an embodiment of the present invention;

[0014] Figure 2 It is an overall flow chart of the scene-adaptive foveated neural radiance field rendering method of an embodiment of the present invention;

[0015] Figure 3 Schematic diagram of the comparison of the gaze point images synthesized by the present invention and the FovNeRF method with respect to the benchmark;

[0016] Figure 4 It is a structural block diagram of a scene-adaptive foveated neural radiance field rendering system in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention provides a scene-adaptive foveated neural radiation field rendering method, which can capture images based on a small number of scenes and synthesize high-quality foveated rendering results of the scene at any viewing angle for a near-eye display device at a high frame rate.

[0018] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0019] In order to better understand the embodiments of the present invention, the principle of the present invention is firstly described as follows:

[0020] (1) By reducing the image synthesis quality of the peripheral area, the number of parameters of the neural radiation field rendering network is reduced, thereby improving the image synthesis performance to meet the frame rate requirements of the near-eye display device;

[0021] (2) The image content of the training set is used to estimate the scene's high visual perception sensitivity areas, and the radiation field representation ability of the neural radiation field representation structure in high visual perception sensitivity areas is improved to improve the image synthesis quality under new viewpoints;

[0022] (3) By integrating rectangular mapping into the foveated neural radiance field rendering network, the goal of efficiently synthesizing foveated rendered images using a single end-to-end network structure is achieved, thereby improving the synthesis performance of foveated images.

[0023] Embodiment 1

[0024] like Figure 1 As shown, a scene-adaptive foveated neural radiance field rendering method provided by an embodiment of the present invention includes the following steps:

[0025] Step S1: estimate the saliency value and depth value of all image pixels in the training set of the current scene; construct a scene visual perception sensitivity body based on the viewpoint position, direction, and saliency value and depth value of image pixels of all images; sample the ellipsoid radius according to the scene visual perception sensitivity body, and construct a multi-resolution ellipsoid neural radiation field representation structure according to the sampled ellipsoid radius;

[0026] Step S2: constructing a foveated neural radiation field rendering network based on rectangular mapping, including: a rectangular mapping encoding network module, an inference network module, and an inverse rectangular mapping decoding network module; given the current viewpoint position and direction, obtaining the scene image corresponding to the current viewpoint position and direction in the training set, inputting the current viewpoint position and direction, the width and height of the output image, and the multi-resolution ellipsoid neural radiation field representation structure into the foveated neural radiation field rendering network based on rectangular mapping, and generating a foveated rendered image under the current viewpoint;

[0027] Step S3: Construct a loss function for optimizing the parameters of the foveated neural radiation field rendering network based on rectangular mapping, so that the constraints on the image quality of the surrounding area are relaxed while keeping the structural information of the foveated rendered image in the surrounding area intact, thereby improving the quality of the foveated rendered image.

[0028] In real scenes, areas of high visual perception sensitivity are usually concentrated in a few limited range areas of the scene rather than throughout the entire scene. In the neural radiation field rendering method, improving the radiation field representation capability within these limited ranges can improve the synthesis quality of the foveated rendered image. FovNeRF uses a concentric sphere structure to represent the scene radiance, and the structure has consistent radiance representation capabilities in all areas of the scene. Due to real-time requirements, the concentric sphere structure of FovNeRF can only use a limited number of sphere layers to represent the scene radiance in virtual reality applications, and lacks the ability to accurately represent the radiance of areas with high visual perception sensitivity. Therefore, it is necessary to improve the radiance representation capability of areas with high visual perception sensitivity under the condition of a limited number of sphere layers. The present invention adopts a radiance representation algorithm based on a multi-resolution ellipsoid structure to solve the above problems.

[0029] First, the visual perception sensitivity of the entire scene is estimated, and the scene visual perception sensitivity body is constructed. Then, based on the scene visual perception sensitivity body, all ellipsoid radii in the multi-resolution ellipsoid structure are sampled using the Gaussian kernel density estimation method. Since the scene visual perception sensitivity body has a higher visual perception sensitivity in high visual perception sensitivity areas, the sampled ellipsoid radii will also be more concentrated in high visual perception sensitivity areas, thereby increasing the spherical layer density in these areas. A higher spherical layer density has a more accurate radiosity representation capability.

[0030] The following is a pseudo code of the radiometric representation algorithm based on the multi-resolution ellipsoid structure of the present invention, wherein:

[0031] Algorithm input: training set viewpoint data set rays, training set image set imgs, number of sphere layers N, width and height of training set images (W, H), camera field of view FOV, the closest distance and the farthest distance of the radiance field (d min ,d max );

[0032] Algorithm output: multi-resolution ellipsoidal radiometric representation structure Ω;

[0033] / / Initialize scene perception sensitivity body Vol;

[0034] 1 Vol←initVolume(N,d min ,d max )

[0035] / / Calculate the visual perception sensitivity value of each voxel in Vol

[0036] 2 for[img,ray]∈[imgs,rays]do

[0037] 3 salImg←saliency(img)

[0038] 4 depthImg←depth(img)

[0039] 5 for px∈img do

[0040] 6 px.o,px.dir←pxPosDir(px,W,H,FOV,ray)

[0041] 7 px.depth←normDepth(depthImg[px],d min ,d max )

[0042] 8 pos←rayCast(px.o,px.dir,px.depth)

[0043] 9

[0044] 10

[0045] 11 end for

[0046] 12 end for

[0047] / / Estimate the probability density function of the visual perception sensitivity of the entire scene based on the Gaussian kernel density estimation method and sample N points to generate a set of sampling points

[0048] 13 points←GaussianKernelSampling(Vol,N)

[0049] / / Sort the x, y, and z values ​​of all points in points to get the radius of all ellipsoids on the x, y, and z axes in the multi-resolution ellipsoid structure sortedRadius x,y,z

[0050] 14 sortedRadius x,y,z ←sort(points)

[0051] / / According to sortedRadius x,y,z The multi-resolution ellipsoid structure Ω is constructed by the radius on the x, y, and z axes

[0052] 15Ω←construct(sortedRadius x,y,z )

[0053] 16 returnΩ

[0054] In one embodiment, the above step S1: estimating the saliency values ​​and depth values ​​of all image pixels in the training set of the current scene; constructing a scene visual perception sensitivity body based on the viewpoint position, direction, and saliency values ​​and depth values ​​of image pixels of all images; sampling the ellipsoid radius according to the scene visual perception sensitivity body, and constructing a multi-resolution ellipsoid neural radiation field representation structure according to the sampled ellipsoid radius, specifically includes:

[0055] Step S11: using the minimum obstacle salient object detection method to obtain the salient values ​​of the image pixels in the training set; defining the image pixels with salient values ​​higher than the threshold in the area corresponding to the scene as a high visual perception sensitive area as the subsequent output gaze point rendering image;

[0056] Given a training set of viewpoint data sets rays, each viewpoint data ray in rays includes the viewpoint position ray.o and the line of sight direction ray.dir, a training set of images imgs, the number of sphere layers N used to define the complexity of the multi-resolution ellipsoid structure, the width and height (W, H) of the training set images, the closest distance and the farthest distance (d min ,d max );

[0057] First, initialize the scene visual perception sensitivity volume Vol, (as shown in the first line of the above pseudo code). According to the closest distance and the farthest distance (d min ,d max ) constructs a voxel structure Ω containing N×N×N voxels to represent the visual perception sensitivity of the entire scene, that is, the distance (d min ,d min ,d min )The nearest voxel is Distance (d min ,d min ,d min )The position of the farthest voxel is The value of each voxel represents the visual perception sensitivity of that area in the scene;

[0058] Then, for each image img in the training set, the minimum obstacle salient object detection method is used to obtain the salient image salImg corresponding to img (as shown in the third line of the above pseudo code);

[0059] Step S12: Use the monocular image depth estimation method to obtain the relative depth values ​​of the image pixels in the training set. The range of the relative depth values ​​is [0,1]. The relative depth values ​​of all image pixels are calculated based on the nearest distance d between them and the predefined neural radiation field representation structure. min and the maximum distance d maxMapping is performed to obtain the depth values ​​of all image pixels in the training set;

[0060] Use the monocular image depth estimation method to obtain the depth image depthImg corresponding to img (as shown in line 4 of the above pseudo code);

[0061] Step S13: According to the viewpoint position and direction of all images in the training set, as well as the saliency values ​​and depth values ​​corresponding to all image pixels, a ray casting method is used to cast light into the current scene based on the viewpoint position and direction of all images in the training set. The light starts from the viewpoint position and passes through the step length of the depth value to obtain all collision points of the light in the current scene. The scene visual perception sensitivity volume Vol is calculated according to the position of the collision point and the saliency value of the pixel corresponding to the collision point;

[0062] For each pixel px in img, first calculate the ray position px.o and direction px.dir of px in world space. Since the ray position of each pixel is consistent with the viewpoint position, let px.o = ray.o, and use the pixel ray direction calculation method to get px.dir. The calculation formula 1 is as follows:

[0063] px.dir=(Vec(Px,Py,-1)-ray.o)·cameraToWorldMatrix (1)

[0064] Among them, cameraToWorldMatrix is ​​the coordinate transfer matrix from camera space to world space, (Px, Py) is the coordinate of px in camera space, and the calculation formula is shown in Formula 2:

[0065]

[0066] Among them, (px.x, px.y) is the coordinate of the pixel in the screen space, and FOV is the camera field of view;

[0067] Then based on the scene range [d min ,d max ] The depth value of px on the depth image depthImg is regularized to obtain the depth estimation value PxDepth of Px in the scene (as shown in line 6 of the above pseudo code), and the calculation formula is shown in Formula 3:

[0068]

[0069] Where depthImg[Px] is the depth value of Px on the depth image depthImg. Based on the ray position px.o and direction px.dir of px in world space and the depth estimation value pxDepth, the position pos of px in the scene is obtained using a ray casting method based on depth estimation (as shown in line 8 of the above pseudo code).

[0070] Since pos is a position point in a three-dimensional scene, the present invention uses the round function to calculate the voxel index corresponding to pos in the scene visual perception sensitivity volume. (as shown in line 9 of the above pseudo code), the calculation formula is shown in Formula 4:

[0071]

[0072] in, is the rounding operation. is a voxel index ranging from (0,0,0) to (N,N,N). Accumulate the saliency value of px in the saliency image salImg to In the result, we get Vol at index The visual perception sensitivity value at (shown in line 10 of the above pseudo code).

[0073] After traversing each pixel px in img, the scene visual perception sensitivity volume Vol is obtained.

[0074] Step S14: Use the Gaussian kernel density estimation method to sample the scene visual perception sensitivity volume Vol on the three axes xyz respectively, and construct a multi-resolution ellipsoid neural radiation field representation structure Ω with the sampling result as the radius.

[0075] After constructing the scene visual perception sensitivity volume, the probability density function of the visual perception sensitivity of the entire scene is estimated using the Gaussian kernel density estimation method, and N values ​​are sampled according to the visual perception sensitivity probability density function to obtain the set of all sampling points (as shown in line 13 of the above pseudo code). The values ​​of points on the x, y, and z axes are counted, and the value sets of x, y, and z are sorted respectively to obtain the radius of all ellipsoids on the x, y, and z axes in the multi-resolution ellipsoid structure sortedRadius x,y,z (shown in line 14 of the pseudo code above), and according to sortedRadius x,y,z Construct a multi-resolution ellipsoid structure Ω and return it (as shown in lines 15-16 of the above pseudo code).

[0076] In one embodiment, the above step S2: constructing a foveated neural radiation field rendering network based on rectangular mapping, including: a rectangular mapping encoding network module, an inference network module and an inverse rectangular mapping decoding network module; given the current viewpoint position and direction, obtaining a scene image corresponding to the current viewpoint position and direction in a training set, inputting the current viewpoint position and direction, the width and height of the output image, and the multi-resolution ellipsoid neural radiation field representation structure into the foveated neural radiation field rendering network based on rectangular mapping, generating a foveated rendered image under the current viewpoint, specifically including:

[0077] Step S21: constructing a fixation point neural radiation field rendering network based on rectangular mapping, including: a rectangular mapping encoding network module, an inference network module and an inverse rectangular mapping decoding network module; the rectangular mapping encoding network module, the inference network module and the inverse rectangular mapping decoding network module are all constructed based on a fully connected network;

[0078] The following is the pseudo code of the foveated rendering image synthesis algorithm based on rectangular mapping of the present invention, wherein:

[0079] Input: output image width and height (W, H), camera field of view FOV, rectangular transformation compression coefficient σ, gaze point position (x g ,y g ), multi-resolution ellipsoidal radiosity representation structure Ω, viewpoint data ray;

[0080] Output: the gaze point rendering result COL under the current viewpoint ray;

[0081] / / Construct the screen space pixel orientation set P according to the width and height of the output image, the camera field of view and the viewpoint data sceen

[0082] 1P sceen ←pxPosDir(W,H,FOV,ray)

[0083] / / P sceen The pixel position and direction data in the matrix are passed to the matrix mapping space pixel bit set P rec

[0084] 2P rec ←recMapping(P sceen ,ray,σ,x g ,y g )

[0085] / / Based on P rec The ray positions and directions corresponding to all pixels in the multi-resolution ellipsoid radiometric representation structure Ω are intersected to obtain the radiometric sampling point p. The sampling points p of all pixels on Ω construct a set P

[0086] 3 P←ellipsoidSampling(Ω,P rec )

[0087] / / Encode the sampling point set P into high-frequency features

[0088] 4

[0089] / / Use voxel rendering method to obtain the radiance estimation value set RAD of all pixels in rectangular mapping spacerec

[0090] 5

[0091] / / Write RAD rec The radiance estimation values ​​of all pixels in the inverse transformation to the screen space to obtain the radiance estimation value set RAD in the screen space

[0092] 6 RAD←invRecMapping(RAD rec ,σ,W,H,x g ,y g )

[0093] / / Send RAD into the fully connected network for decoding and output the synthesis result of the gaze point rendering image

[0094] 7 COL←decodeNN(RAD)

[0095] return COL

[0096] The above algorithm first uses the rectangular mapping encoding network module to perform pixel-by-pixel encoding (lines 1-4); then the pixel encoding features are passed to the inference network module to obtain the radiance estimation result of the pixel in the rectangular mapping space under the current viewpoint (line 5); then the inverse rectangular mapping decoding network module is used to perform feature decoding on the radiance estimation result in the rectangular mapping space, and the gaze point rendering result of the pixel in the screen space under the current viewpoint is obtained and output (lines 6-8).

[0097] Step S22: Given the current viewpoint position and direction, obtain the scene image corresponding to the current viewpoint position and direction in the training set, input the current viewpoint position, direction and the width and height of the output image and the multi-resolution ellipsoid neural radiation field representation structure into the rectangular mapping encoding network module, and output the pixel-by-pixel features of the scene image, specifically including:

[0098] Step S221: construct a screen space pixel orientation set P according to the width and height [W, H] of the output image and the viewpoint data ray sceen , where the viewpoint data ray includes: viewpoint position ray.o and direction ray.dir;

[0099] In the rectangular mapping encoding network module, the screen space pixel orientation set P is first constructed based on the width and height [W, H] of the output image and the viewpoint data ray sceen (Line 1). sceenis a two-dimensional list that stores the ray position and ray direction data of each pixel in the screen space. That is, for any pixel px, its coordinates in the screen space are (x, y). Since the ray position of each pixel is consistent with the viewpoint position, the ray position px.pos of the pixel px = ray.o, and the ray direction px.dir of the pixel px is calculated using the above formula (1), and [px.pos, px.dir] is assigned to P sceen [x,y].

[0100] Step S222: Move the screen space pixels to the set P sceen The pixel position and direction data in the matrix are passed to the matrix mapping space pixel bit set P rec ;

[0101] In order to reduce the number of parameters of the foveated rendering image synthesis network, the pixels in the screen space are compressed into a low-resolution rectangular mapping space based on the rectangular transformation, and the screen space pixels are moved to the set P according to the mapping result. sceen The pixel position and direction data in the matrix are passed to the matrix mapping space pixel bit set P rec (Line 2). Specifically, for any pixel px, its coordinates in screen space are (x, y), and the rectangular transformation uses formula (5) to transform it to matrix mapping space coordinates (u, v), and passes the pixel bit direction data at this position into the rectangular mapping space pixel bit direction set P rec In px.dirP rec [u,v]=P rec [u,v]∪P sceen [x,y].

[0102]

[0103] Among them, the x range is [0, W], the y range is [0, H], and the u range is In the range of v In the case of the rectangular mapping space, the number of pixels is the number of pixels in the output image. f is the shading rate reduction control parameter, which is set to 0.35

[138] ; function N X (·) and N Y (·)As shown in formula (6).

[0104]

[0105] Since the rectangular mapping transformation compresses the pixels in the screen space into the low-resolution rectangular mapping space, a single pixel in the rectangular mapping space may correspond to multiple pixels in the screen space, that is, there are rectangular mapping space coordinates (u, v), P rec[u,v] contains multiple position data and direction data, which are averaged so that the rectangular mapping space pixel position set P rec Each coordinate position in contains only one position data and direction data.

[0106] Step S223: Based on P rec The positions and directions corresponding to all pixels in the multi-resolution ellipsoid radiometric representation structure Ω are intersected to obtain the radiometric sampling point p, and the sampling points p of all pixels on Ω construct a set P;

[0107] Using formula (7) based on the matrix mapping spatial pixel position set P rec The ray positions px.o and directions px.dir corresponding to all pixels px in the radiance representation structure Ω are intersected with the multi-resolution ellipsoid radiance representation structure Ω to obtain the radiance sampling point p, and the set P is constructed based on the sampling points p of all pixels on Ω (the third line).

[0108]

[0109] Where Δ=B 2 -4·A·C, the calculation formulas of A, B, and C are shown in formula (8).

[0110]

[0111] Where radius is the radius of each ellipsoid in Ω on the xyz axis. Then, the high-frequency conversion method proposed by Mildenhall et al. is used to convert the sampling point set P into high-frequency features, as shown in formula (9), and the converted high-frequency features are sent to the fully connected network encoding to obtain the encoded features: (Line 4).

[0112] Step S224: Encode the sampling point set P into pixel-by-pixel features

[0113] γ(p)=(sin(2 0 ·π·p),cos(2 0 ·π·p),...,sin(2 4 ·π·p),cos(2 4 ·π·p))(9)

[0114] Where p is a sampling point in P.

[0115] Step S23: Input the pixel-by-pixel features into the inference network module, sample and calculate the pixel-by-pixel radiance from the multi-resolution ellipsoid neural radiation field representation structure based on the pixel-by-pixel features, and obtain the view synthesis result in the rectangular mapping space, specifically including:

[0116] The pixel-by-pixel features Input the inference network module and use the voxel rendering method to obtain the radiance estimation value set RAD of all pixels in the rectangular mapping space rec As a result of view synthesis;

[0117] The radiometric reasoning and rendering algorithm of the present invention is consistent with the radiometric reasoning and rendering algorithm in FovNeRF, and the encoding feature They are respectively passed into the inference network module, and then the voxel rendering method is used to obtain the radiance estimation value set RAD of all pixels in the rectangular mapping space. rec (Line 5).

[0118] Step S24: inputting the view synthesis result into the inverse rectangular mapping decoding network module for decoding to generate a foveated rendering image under the current viewpoint, specifically including:

[0119] The view synthesis result RAD rec Input the inverse rectangular mapping decoding network module for decoding, and convert RAD rec The inverse transformation is performed to the set of radiance estimation values ​​RAD in the screen space, and RAD is decoded by a fully connected network to generate a foveated rendering image COL under the current viewpoint.

[0120] In the inverse rectangular mapping decoding network module, RAD is converted into rec The radiance estimation values ​​of all pixels in the image are inversely transformed to the screen space (line 6), and the converted radiance features are sent to the fully connected network for decoding and outputting the synthetic result of the foveated rendering image (lines 7-8).

[0121]

[0122] Where (u,v) is the RAD rec The coordinates of any pixel in the function N U (·) and N V (·) As shown in formula (11):

[0123]

[0124] In the traditional foveation-based neural radiation field rendering method FovNeRF, synthesizing foveation-rendered images requires constructing multiple view synthesis networks to synthesize the foveation area image and the surrounding area image in the foveation-rendered image respectively, and then mixing the foveation area image and the surrounding area image to obtain the final foveation-rendered image. This requires the neural radiation field rendering method to train multiple view synthesis networks to generate foveation-rendered images, which reduces the foveation image synthesis performance of the neural radiation field rendering method. In addition, the generation results of multiple networks in FovNeRF need to be accurately aligned based on the current viewpoint, otherwise the foveation area and the surrounding area images will be broken, affecting the quality of the final mixed foveation-rendered image, which increases the training cost of the neural radiation field rendering method based on the foveation. Therefore, the present invention designs a foveation neural radiation field rendering network based on rectangular mapping, and uses a single end-to-end network structure to synthesize foveation-rendered images. Compared with FovNeRF, the present invention can synthesize foveation-rendered images while reducing the number of parameters in the network structure, and improve the synthesis performance of foveation images of the neural radiation field rendering method in virtual reality applications.

[0125] In one embodiment, the above step S3: constructing a loss function for optimizing the parameters of the foveated neural radiance field rendering network based on rectangular mapping, so as to relax the constraints on the image quality of the surrounding area while keeping the structural information of the foveated rendered image in the surrounding area intact, and improve the quality of the foveated rendered image, specifically includes:

[0126] Construct the loss function loss:

[0127]

[0128] Among them, COL is the generated gaze point rendering image under the current viewpoint, GT is the real image under the current viewpoint, σ is the compression coefficient based on rectangular transformation, is the image after COL and GT are mean compressed based on σ; px∈Fovea represents all pixels in the fixation area, MSE is the mean square error function, and SSIM is the structural similarity function; The calculation formula (13) for the pixel with coordinates (x, y) is as follows:

[0129]

[0130] Where W and H are the width and height of COL and GT respectively; They are and width and height.

[0131] Since in foveated rendering, the image rendering quality of the surrounding area can be reduced to a certain extent without reducing the perceived rendering quality, the scene-adaptive foveated neural radiation field rendering method of the present invention proposes a loss function of the foveated neural radiation field rendering network based on rectangular mapping to optimize it. While ensuring that the synthesized foveated rendered image has high quality in the foveated area and the overall structural information is not lost, the restrictions on the image synthesis quality of the surrounding area are adaptively reduced. The required training data is consistent with the traditional neural radiation field rendering method, and there is no need to reconstruct the foveated image for training for the target scene.

[0132] Figure 2 The overall flow chart of the scene-adaptive foveated neural radiance field rendering method of the present invention is shown.

[0133] Figure 3 This is a comparison diagram of the gaze point images synthesized by the present invention and the FovNeRF method relative to the benchmark. It can be seen that the gaze point rendering image generated by the method of the present invention is of higher quality and closer to the benchmark image than that by the FovNeRF method.

[0134] The hardware configuration used in the embodiment of the present invention is an Intel i7-10700KF processor, an Nvidia Geforce3080Ti graphics card, and 64G memory. It runs on a Windows 10 64-bit system. All programs involved in the efficiency test are 64-bit programs compiled using O2-level optimization.

[0135] The present invention discloses a scene-adaptive foveated neural radiation field rendering method, which realizes a multi-resolution ellipsoid neural radiation field representation structure, adaptively adjusts the ellipsoid radius according to the complex scene content to increase the sphere density in the scene's high visual perception sensitive area, and improves the radiation field representation capability of complex scenes. Under the condition of the same number of sphere layers, compared with the concentric sphere structure of the existing FovNeRF method, the present invention improves the radiation field representation capability of complex scenes.

[0136] The present invention designs and implements a foveated neural radiance field rendering network based on rectangular mapping, integrates rectangular mapping into the foveated neural radiance field rendering network, and uses a single end-to-end network structure to efficiently synthesize foveated rendered images. Under the condition of similar synthesis quality, compared with the FovNeRF method, the present invention can synthesize high-quality foveated rendered images for complex scenes at a high frame rate (≥60FPS) in a near-eye display device, and the synthesis performance is improved by 1.3-1.8 times.

[0137] The present invention aims at a foveation neural radiation field rendering network based on rectangular mapping, constructs a loss function to optimize its parameters, and achieves a high-quality synthesis result of the foveation area under the premise that the parameter quantity satisfies the network to synthesize the foveation image at a high frame rate. The training data required by the present invention is consistent with the traditional neural radiation field rendering method, and there is no need to reconstruct the foveation image for the target scene for training.

[0138] Embodiment 2

[0139] like Figure 4 As shown, an embodiment of the present invention provides a scene-adaptive foveated neural radiance field rendering system, including the following modules:

[0140] Constructing a multi-resolution ellipsoid neural radiation field representation structure module 41, which is used to estimate the saliency value and depth value of all image pixels in the training set of the current scene; constructing a scene visual perception sensitivity body based on the viewpoint position, direction and saliency value and depth value of the image pixels of all images; sampling the ellipsoid radius according to the scene visual perception sensitivity body, and constructing a multi-resolution ellipsoid neural radiation field representation structure according to the sampled ellipsoid radius;

[0141] A foveated rendering image generation module 42 is used to construct a foveated neural radiation field rendering network based on rectangular mapping, including: a rectangular mapping encoding network module, an inference network module, and an inverse rectangular mapping decoding network module; given the current viewpoint position and direction, a scene image corresponding to the current viewpoint position and direction in a training set is obtained, and the current viewpoint position and direction, the width and height of the output image, and the multi-resolution ellipsoid neural radiation field representation structure are input into the foveated neural radiation field rendering network based on rectangular mapping to generate a foveated rendering image under the current viewpoint;

[0142] Constructing a loss function module 43, which is used to construct a loss function for optimizing the parameters of the foveated neural radiation field rendering network based on rectangular mapping, so that while keeping the structural information of the foveated rendered image in the surrounding area intact, the constraints on the image quality of the surrounding area are relaxed, thereby improving the quality of the foveated rendered image.

[0143] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included within the scope of the present invention.

Claims

1. A scene-adaptive foveated neural radiance field rendering method, characterized in that: include: Step S1: Estimate the saliency value and depth value of all image pixels in the training set of the current scene; Constructing a scene visual perception sensitivity body based on viewpoint positions, directions, and saliency values ​​and depth values ​​of pixels of all images; sampling an ellipsoid radius according to the scene visual perception sensitivity body, and constructing a multi-resolution ellipsoid neural radiation field representation structure according to the sampled ellipsoid radius; Step S2: constructing a foveated neural radiation field rendering network based on rectangular mapping, including: a rectangular mapping encoding network module, an inference network module and an inverse rectangular mapping decoding network module; given the current viewpoint position and direction, obtaining a scene image corresponding to the current viewpoint position and direction in a training set, inputting the current viewpoint position and direction, the width and height of the output image and the multi-resolution ellipsoid neural radiation field representation structure into the foveated neural radiation field rendering network based on rectangular mapping, and generating a foveated rendered image under the current viewpoint; Step S3: constructing a loss function for optimizing the parameters of the foveated neural radiation field rendering network based on rectangular mapping, so as to relax the constraints on the image quality of the surrounding area while keeping the structural information of the foveated rendered image in the surrounding area intact, thereby improving the quality of the foveated rendered image.

2. The scene-adaptive foveated neural radiance field rendering method according to claim 1, characterized in that: The step S1: estimating the saliency values ​​and depth values ​​of all image pixels in the training set of the current scene; constructing a scene visual perception sensitivity body based on the viewpoint positions, directions, and saliency values ​​and depth values ​​of the image pixels of all images; The ellipsoid radius is sampled according to the scene visual perception sensitivity body, and a multi-resolution ellipsoid neural radiation field representation structure is constructed according to the sampled ellipsoid radius, specifically including: Step S11: using a minimum obstacle salient object detection method to obtain salient values ​​of image pixels in the training set; defining image pixels with salient values ​​higher than a threshold in an area corresponding to the scene as a high visual perception sensitivity area; Step S12: Using a monocular image depth estimation method to obtain the relative depth values ​​of the image pixels in the training set, the range of the relative depth values ​​is [0,1], and the relative depth values ​​of all image pixels are the closest distance d to the predefined neural radiation field representation structure. min and the maximum distance d max Mapping is performed to obtain depth values ​​of all image pixels in the training set; Step S13: according to the viewpoint position and direction of all images in the training set, and the saliency values ​​and depth values ​​corresponding to the pixels of all images, a ray casting method is used to cast a ray into the current scene based on the viewpoint position and direction of all images in the training set. The ray starts from the viewpoint position and passes through the step length of the depth value to obtain all collision points of the ray in the current scene. The scene visual perception sensitivity volume Vol is calculated according to the position of the collision point and the saliency value of the pixel corresponding to the collision point. Step S14: Use the Gaussian kernel density estimation method to sample the scene visual perception sensitivity volume Vol on the three axes xyz respectively, and construct a multi-resolution ellipsoid neural radiation field representation structure Ω with the sampling result as the radius.

3. The scene-adaptive foveated neural radiance field rendering method according to claim 2, characterized in that: The step S2: constructing a foveated neural radiation field rendering network based on rectangular mapping, including: a rectangular mapping encoding network module, an inference network module and an inverse rectangular mapping decoding network module; given the current viewpoint position and direction, obtaining a scene image corresponding to the current viewpoint position and direction in a training set, inputting the current viewpoint position and direction, the width and height of the output image and the multi-resolution ellipsoid neural radiation field representation structure into the foveated neural radiation field rendering network based on rectangular mapping, and generating a foveated rendered image under the current viewpoint, specifically including: Step S21: constructing a fixation point neural radiation field rendering network based on rectangular mapping, including: a rectangular mapping encoding network module, an inference network module and an inverse rectangular mapping decoding network module; the rectangular mapping encoding network module, the inference network module and the inverse rectangular mapping decoding network module are all constructed based on a fully connected network; Step S22: given the current viewpoint position and direction, obtain the scene image corresponding to the current viewpoint position and direction in the training set, input the current viewpoint position, direction and the width and height of the output image and the multi-resolution ellipsoid neural radiation field representation structure into the rectangular mapping encoding network module, and output the pixel-by-pixel features of the scene image; Step S23: inputting the pixel-by-pixel features into the inference network module, sampling and calculating the pixel-by-pixel radiance from the multi-resolution ellipsoid neural radiance field representation structure based on the pixel-by-pixel features, and obtaining a view synthesis result in a rectangular mapping space; Step S24: input the view synthesis result into the inverse rectangular mapping decoding network module for decoding to generate a foveated rendering image under the current viewpoint.

4. The scene-adaptive foveated neural radiance field rendering method according to claim 3, characterized in that: The step S22: given the current viewpoint position and direction, obtaining the scene image corresponding to the current viewpoint position and direction in the training set, inputting the current viewpoint position, direction and the width and height of the output image and the multi-resolution ellipsoid neural radiation field representation structure into the rectangular mapping encoding network module, and outputting the pixel-by-pixel features of the scene image, specifically includes: Step S221: construct a screen space pixel orientation set P according to the width and height [W, H] of the output image and the viewpoint data ray sceen , where the viewpoint data ray includes: viewpoint position ray.o and direction ray.dir; Step S222: Move the screen space pixels to a set P sceen The pixel position and direction data in the matrix are passed to the matrix mapping space pixel bit set P rec ; Step S223: Based on P rec The positions and directions corresponding to all pixels in the multi-resolution ellipsoid radiometric representation structure Ω are intersected to obtain the radiometric sampling point p, and the sampling points p of all pixels on Ω construct a set P; Step S224: Encode the sampling point set P into pixel-by-pixel features 5. The scene-adaptive foveated neural radiance field rendering method according to claim 4, characterized in that: The step S23: inputting the pixel-by-pixel features into the inference network module, sampling and calculating the pixel-by-pixel radiance from the multi-resolution ellipsoid neural radiation field representation structure based on the pixel-by-pixel features, and obtaining the view synthesis result in the rectangular mapping space, specifically includes: The pixel-by-pixel features Input the inference network module and use the voxel rendering method to obtain the radiance estimation value set RAD of all pixels in the rectangular mapping space rec As the view synthesis result.

6. The scene-adaptive foveated neural radiance field rendering method according to claim 5, characterized in that: The step S24: inputting the view synthesis result into the inverse rectangular mapping decoding network module for decoding to generate a foveated rendering image under the current viewpoint specifically includes: The view synthesis result RAD rec Input the inverse rectangular mapping decoding network module for decoding, and convert RAD rec The inverse transformation is performed to the set of radiance estimation values ​​RAD in the screen space, and RAD is decoded by a fully connected network to generate a foveated rendering image COL under the current viewpoint.

7. The scene-adaptive foveated neural radiance field rendering method according to claim 6, characterized in that: The step S3: constructing a loss function for optimizing the parameters of the foveated neural radiance field rendering network based on rectangular mapping, so as to relax the constraints on the image quality of the surrounding area while keeping the structural information of the foveated rendered image in the surrounding area intact, and improve the quality of the foveated rendered image, specifically includes: Construct the loss function: Among them, COL is the generated gaze point rendering image under the current viewpoint, GT is the real image under the current viewpoint, σ is the compression coefficient based on rectangular transformation, is the image after COL and GT are mean compressed based on σ; px∈Fovea represents all pixels in the fixation area, MSE is the mean square error function, and SSIM is the structural similarity function; The calculation formula for the pixel with coordinates (x, y) is as follows: Where W and H are the width and height of COL and GT respectively; They are and width and height.

8. A scene-adaptive foveated neural radiance field rendering system, characterized in that: Includes the following modules: Constructing a multi-resolution ellipsoid neural radiation field representation structure module for estimating the saliency value and depth value of all image pixels in the training set of the current scene; constructing a scene visual perception sensitivity body based on the viewpoint position, direction and the saliency value and depth value of the image pixels of all images; sampling the ellipsoid radius according to the scene visual perception sensitivity body, and constructing a multi-resolution ellipsoid neural radiation field representation structure according to the sampled ellipsoid radius; A foveated rendering image generation module is used to construct a foveated neural radiation field rendering network based on rectangular mapping, including: a rectangular mapping encoding network module, an inference network module, and an inverse rectangular mapping decoding network module; given a current viewpoint position and direction, a scene image corresponding to the current viewpoint position and direction in a training set is obtained, and the current viewpoint position and direction, the width and height of the output image, and the multi-resolution ellipsoid neural radiation field representation structure are input into the foveated neural radiation field rendering network based on rectangular mapping to generate a foveated rendering image under the current viewpoint; A loss function module is constructed to construct a loss function for optimizing the parameters of the foveated neural radiation field rendering network based on rectangular mapping, so that the constraints on the image quality of the surrounding area are relaxed while keeping the structural information of the foveated rendering image in the surrounding area intact, thereby improving the quality of the foveated rendering image.

Images