Personalized human head neural radiation field basis representation and reconstruction method based on multi-level hash coding

Abstract

The present invention discloses a personalized human head neural radiation field basis representation and reconstruction method based on multi-level hash coding. The input RGB video is reconstructed based on a grid blendshape basis; the expression coefficient and camera parameters are obtained by solving the inverse rendering problem; the head portion of each frame is segmented to obtain a head semantic segmentation map; the neural radiation field basis is combined with the expression coefficient of each frame to obtain a multi-resolution hash table, and the sampling points are queried on the multi-resolution hash table to obtain voxel features. The voxel features are converted into sampling point RGB and volume density using a multi-layer perceptron, and a rendered image is obtained through volume rendering; during the sampling process, the density information in the space is recorded using a density grid to guide the sampling, and a density grid update strategy that considers the expression coefficient is adopted to fully consider the maximum range that can be achieved by each facial expression. By optimizing and training the basis and the multi-layer perceptron, a high-quality personalized neural radiation field human head parameterized model is finally obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a personalized human head neural radiation field substrate representation and reconstruction method based on multi-level hash coding. Background Art

[0002] 3D face / head representation is a key research problem in computer vision and computer graphics, with widespread applications in AR / VR, digital gaming, and the film industry. Representing dynamic head scenes and faithfully reconstructing head models from monocular RGB video is a crucial and challenging research topic. Parametric head models that represent the head using low-dimensional coefficients, such as the blendshape basis, have been studied and refined for a long time. The blendshape basis generates facial geometry by linearly combining head expression bases using expression coefficients. This approach offers two advantages: first, it is a semantic parameterization, where the magnitude of the expression coefficients characterizes the influence of a specific expression base on the final combination. Second, the blendshape basis constructs a reasonable low-dimensional deformation space, which reduces the difficulty for users to edit facial expressions.

[0003] Current explicit blendshape bases such as FaceWarehouse and FLAME are designed to model different individuals with different expressions, and therefore may ignore many personalized geometric and texture details. To construct a personalized blendshape base, traditional mesh-based methods usually use deformation transfer or construct a multilinear tensor 3D face model. However, these methods usually have the following disadvantages. First, it is difficult for mesh-based parameterized explicit models to represent non-facial parts such as hair and teeth. Second, if RGB supervision is to be used, approximate differentiable rendering techniques must be used to alleviate the non-differentiable problem. Third, due to limited representation capabilities, deformation transfer cannot realistically reconstruct expressions. Finally, facial expressions are related to many factors such as age and muscle movement, which are difficult to be expressed by a generalized explicit blendshape base model.

[0004] Recently, methods based on neural radiance fields (NeRFs) have made it possible to synthesize realistic head images. Some work has combined neural radiance fields with generative adversarial networks. However, this generative model couples expression and identity, making expression editing difficult. HeadNeRF proposes a strategy to decouple the different semantic attributes of the neural radiance field of a human head. However, due to the limited capacity of its generalization model, it struggles to accurately model individual facial details and dynamic patterns. AD-NeRF and NerFACE can generate highly personalized facial animations. Their one-person-one-training strategy enables the model to learn more personalized facial details. However, due to their use of Fourier encoding and tensor concatenation, they often require a long time to train a reasonable dynamic head field. Although several methods have been developed to accelerate the training and inference of static neural radiance fields, achieving fast training for dynamic scenes (such as complex head deformations) remains a challenge. Summary of the Invention

[0005] In response to the shortcomings of the existing technology, the present invention proposes a new type of personalized parametric model of the human head - the neural radiation field basis (English name NeRF blendshape) and provides an efficient reconstruction method for this model. The neural radiation field basis is a personalized human head basis defined on a multi-level voxel field. It has the property of fast convergence and can construct a realistic personalized human head basis in about 10-20 minutes. It has strong expressive power. It not only inherits good semantics from the mesh blendshape basis, but also can learn more personalized details, including nonlinear deformation (cheek wrinkles, forehead wrinkles), user-specific attributes (moles, beards), and non-facial head features. Compared with the traditional mesh blendshape basis, the neural radiation field basis proposed in the present invention can be constructed from a monocular RGB video. The technical solution of the present invention can solve the problems of cross-identity driving of the human head and synthesis of new perspectives in virtual reality technology.

[0006] The purpose of the present invention is achieved through the following technical solution: a personalized human head neural radiation field base representation and reconstruction method based on multi-level hash coding, comprising:

[0007] Reconstruct the input RGB video based on the mesh blendshape basis; obtain the expression coefficient and camera parameters by solving the inverse rendering problem; segment the head part of each frame to obtain the head semantic segmentation map;

[0008] The expression coefficients of each frame are combined with the neural radiation field basis to obtain a multi-resolution hash table corresponding to the expression coefficients. The sampling points of the head scene space are queried on the multi-resolution hash table to obtain voxel features. The voxel features are converted into sampling point RGB and volume density using a multi-layer perceptron, and the rendered image is obtained through volume rendering.

[0009] Furthermore, each basis of the neural radiation field basis is a radiation field expressed by a multi-level voxel field, and the multi-level voxel field is an L-layer feature storage structure, each layer is a voxel field, and each voxel block stores features of length F. A multi-resolution hash table is used to compress and store the multi-level voxel field. The multi-resolution hash table h is a feature table of size L×T×F, where T is the hash table size.

[0010] Furthermore, the head representation is written as:

[0011]

[0012] Where I is the final rendered image, θ represents the learnable weights of the multilayer perceptron, Represents sampling, sampling point feature access and volume rendering, C is the camera internal and external parameters, h0∈R L×T×F A multi-resolution hash table representing an average face, H = {h1, h2, ..., h K}, h i ∈R L×T×F A multi-resolution hash table representing different expression offsets, K is the number of bases, w = {w1, w2, ..., w K}∈R K is the expression coefficient; the combination of the multi-resolution hash table corresponding to the expression coefficient is:

[0013] Furthermore, during the sampling process, the density information in the density grid recording space is used to guide the sampling, and a density grid update strategy that considers the expression coefficient is adopted to fully consider the maximum range that each facial expression can reach.

[0014] Furthermore, for the density grid update strategy, the hash table corresponding to each basis

[0015]

[0016]

[0017] Where N is the total number of frames in the video, is the i-th dimension element of the expression coefficient of the j-th frame, h0 represents the multi-resolution hash table of the average face, h i A multi-resolution hash table representing different expression offsets; take all The final density grid is obtained by taking the element-by-element maximum value of the corresponding density grid.

[0018] Furthermore, the method of sampling light based on a density grid is as follows: sampling starts from the viewpoint first, and the density value of each sampling point is queried in the density grid. If it is higher than the threshold, sampling continues; if it is lower than the threshold, sampling is directly jumped to the intersection of the light and the next pixel of the grid.

[0019] Furthermore, for any point x in space, the method for calculating its stored features in the multi-resolution hash table is as follows: first calculate the diagonal point index of x on each layer, input it into the hash function to obtain the hash value, use the hash value to find the feature in the hash table, and then use linear interpolation to find the obtained feature, so as to obtain the voxel feature corresponding to x.

[0020] Furthermore, using the multi-layer perceptron g θ The voxel features of the sampling point x are converted into sampling point RGB and volume density, expressed as:

[0021] g θ (η(x;h),γ(d))→(σ,c)

[0022] Among them, c and σ represent the sampling point RGB and volume density respectively, η(x;h) represents the voxel feature obtained by querying the sampling point x on the hash table h, and γ(d) represents the position encoding of the view vector d.

[0023] Furthermore, the rendered image is obtained through volume rendering, and the calculation formula for rendering a single ray RGB is:

[0024]

[0025]

[0026] Among them, I(r) represents the color corresponding to the light r, t and s are the parameterized variables of the light, c(r(t)) is the RGB corresponding to the point r(t) on the light, σ(r(s)) and σ(r(t)) are the volume densities corresponding to the points r(s) and r(t) on the light, respectively.

[0027] Furthermore, during the training phase, the neural radiation field basis is obtained through the supervision of image RGB, head semantic segmentation map and perceptual loss function; at the beginning of training, the neural radiation field basis is randomly initialized and continuously updated during the training process.

[0028] It can be seen from the technical solution provided by the present invention that: 1) The architecture of the present invention has taken into account the relationship between global conditions and local features (including expression coefficients and position codes obtained by queries, etc.). This feature combination architecture can reduce the learning burden of the multi-layer perceptron. Such a structure is easier to learn and converge than the widely used tensor splicing structure, and has stronger expressive power. 2) The neural radiation field basis proposed in the present invention can capture multi-scale details at the same time, and the features of multi-resolution hash tables at different levels can be optimized together. 3) The density grid update strategy proposed by the present invention can ensure that the sampling skips the parts that cannot be covered by the dynamics of the face. The density grid update strategy of the present invention takes into account the dynamics of the expression to ensure that every voxel that may be occupied by the head area is taken into account. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 A neural radiation field substrate construction pipeline is provided for an embodiment of the present invention.

[0031] Figure 2 This is a diagram showing the rendering results of different methods provided by the embodiments of the present invention.

[0032] Figure 3 This is a curve showing the change of PSNR over training time on a certain identity test set provided by an embodiment of the present invention.

[0033] Figure 4 Rendering effect diagrams on the test set when different methods provided in the embodiments of the present invention are trained to different stages.

[0034] Figure 5 A schematic diagram of a cross-identity expression-driven application provided by an embodiment of the present invention.

[0035] Figure 6 A schematic diagram of a new perspective synthesis application of a human head provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0037] The present invention provides a personalized human head neural radiation field base representation and reconstruction method based on multi-level hash coding. Figure 1 Schematic diagram of the pipeline of the method of the present invention.

[0038] Each basis of the neural radiation field basis is a radiation field expressed by a multi-level voxel field. For performance reasons, the present invention uses a multi-resolution hash table to store the multi-level voxel field. Any human head scene can be expressed as a combination of expression coefficients and bases, and the number of bases is consistent with the dimension of the expression coefficients. The present invention uses a multi-layer perceptron (MLP) to convert the voxel field into a volume-rendered radiation field, and further determines a density grid based on the range of facial motion to guide efficient sampling.

[0039] The present invention uses a multi-level voxel field to represent the radiation field of a human head corresponding to a specific expression. The multi-level voxel field is an L-layer feature storage structure, with each layer being a voxel field, and each voxel block storing features of length F. Because multi-level voxel fields occupy a large amount of storage space, in actual implementation, a multi-resolution hash table is used to compress and store the multi-level voxel field. The multi-resolution hash table h is a feature table of size L × T × F, where T is the hash table size.

[0040] The head representation in the present invention can be written as:

[0041]

[0042] Where I is the final rendered image, θ represents the learnable weights of the multilayer perceptron, Represents sampling, sampling point feature access and volume rendering, C is the camera internal and external parameters, h0∈R L×T×F A multi-resolution hash table representing an average face, H = {h1, h2, ..., h K}, h i ∈R L×T×F A multi-resolution hash table representing different expression offsets, K is the number of bases, and w = {w1, w2, ..., w K}∈R K is the expression coefficient. The combination of the multi-resolution hash table corresponding to the expression coefficient is: For a sampling point x in the head scene space, query it on the hash table h to obtain the voxel feature, which is then converted into the sampling point RGB and volume density σ using a multi-layer perceptron. Finally, the rendered image is obtained through volume rendering.

[0043] In one embodiment, the neural radiation field substrate is constructed as follows:

[0044] Based on a grid blendshape base, we optimize facial parameters to fit and reconstruct each frame of a single-person, single-object RGB video, obtaining the expression coefficient and camera parameters for each frame. We also segment the head of each frame to obtain a semantic segmentation map.

[0045] The neural radiation field basis is combined with the expression coefficients of each frame to obtain a multi-resolution hash table corresponding to the expression coefficients. The hash table is then queried for each sampling point in the head scene space to obtain the sampling point's RGB and volume density, and finally rendered to obtain the corresponding pixel color. The neural radiation field basis is trained using the image RGB, the head semantic segmentation map, and the perceptual loss function for supervision. The neural radiation field basis is randomly initialized at the beginning of training and continuously updated during training.

[0046] In order to speed up sampling and rendering, the present invention uses a density grid to record the density information in the space to guide sampling. In order to fully consider the density distribution of all expressions, the hash table corresponding to each basis is first calculated.

[0047]

[0048]

[0049] Where N is the total number of frames in the video, is the i-th dimension element of the expression coefficient of the j-th frame; then take all The final density grid is obtained by taking the element-by-element maximum value of the corresponding density grid.

[0050] The method for sampling light based on a density grid is as follows: sampling starts from the viewpoint, and at each sampling point, the density value is queried in the density grid. If it is above the threshold, sampling continues; if it is below the threshold, sampling is directly jumped to the intersection of the light and the next pixel in the grid.

[0051] For a sampling point x in the head scene space, query it on the hash table to obtain voxel features. Specifically, for any point x in the space, the method for calculating its stored features on the multi-resolution hash table is: first calculate the diagonal point index of x on each layer, input it into the hash function to obtain the hash value, use the hash value to search for features on the hash table, and then use linear interpolation to find the obtained features, so as to obtain the voxel features corresponding to x.

[0052] In one embodiment, for a cube voxel field, the side length is N1, and the diagonal point index is the calculated corner point and .

[0053] In one embodiment, a multilayer perceptron g is used θ The voxel features of the sampling point x are converted into sampling point RGB and volume density, expressed as:

[0054] g θ (η(x;h),γ(d))→(σ,c)

[0055] Among them, c and σ represent the sampling point RGB and volume density respectively, η(x;h) represents the voxel feature obtained by querying the sampling point x on the hash table h, and γ(d) represents the position encoding of the view vector d.

[0056] The rendered image is obtained by volume rendering. The calculation formula for rendering a single ray RGB is:

[0057]

[0058]

[0059] Among them, I(r) represents the color corresponding to the light r, t and s are the parameterized variables of the light, c(r(t)) is the RGB corresponding to the point r(t) on the light, σ(r(s)) and σ(r(t)) are the volume densities corresponding to the points r(s) and r(t) on the light, respectively.

[0060] The calculation formula for rendering the head semantic segmentation map is:

[0061]

[0062] Among them, M(r) is the pixel value of the head semantic segmentation map corresponding to the light r.

[0063] In one embodiment, the main steps of the personalized human head neural radiation field basis representation and reconstruction method based on multi-level hash coding are as follows:

[0064] 1. Data Acquisition Phase. The reconstruction scheme of this invention only requires a monocular RGB video sequence to construct a personalized neural radiation field substrate for the human head. First, the camera is fixed so that the lens is aimed at the subject, and then the subject is asked to make 46 facial expressions that meet the FACS (Facial Action Coding System) standard, see https: / / www.paulekman.com / facial-action-coding-system / ). The subject is then asked to make a number of random expressions, rotating their head while doing so, so that the training data includes images of the human head captured from various angles. The total length of a single-person video is approximately 3-5 minutes.

[0065] 2. Data preprocessing. First, the input RGB video is reconstructed based on a mesh blendshape basis. By solving the inverse rendering problem, the expression coefficient and camera parameters for each frame are obtained. Simultaneously, the head is segmented in each frame to obtain a semantic segmentation map of the head.

[0066] 3. Combine the expression coefficients of the taken frame with the neural radiation field basis to obtain a multi-resolution hash table of the corresponding frame. For the sampling points in the head scene space, query the hash table to obtain the sampling point RGB and volume density, and finally render the color of the corresponding pixel.

[0067] 4. Training phase. Rendering and calculation of loss function. The loss function has the following items:

[0068] (1) Color loss function L color :

[0069]

[0070] Among them, IGT(r) is the true color value corresponding to light r, is the sampling light set;

[0071] (2) Semantic segmentation loss function L mask :

[0072]

[0073] Among them, M GT (r) is the true value of the pixel value of the semantic segmentation map of the head corresponding to the light r;

[0074] (3) Perceptual loss function: In order to improve the details of face rendering, the VGG network is used to implement the perceptual loss function. The actual calculation is to sample the light of a small window on the image and then apply the perceptual loss function.

[0075] During training, the density grid update formula of each neural radiation field basis is as follows:

[0076]

[0077]

[0078] Take All The final density grid is obtained by taking the maximum value of each element of the corresponding density grid. Experiments have shown that this update strategy can ensure that the final density grid can take into account the maximum area that can be reached by each facial expression.

[0079] The effects of the method of the present invention are demonstrated below.

[0080] 1. Comparison with similar methods

[0081] We compare with the most advanced face-driven or head modeling works, which are:

[0082] First Order Motion Model for Image Animation (NeurIPS 2019) hereinafter referred to as FOMM;

[0083] NerFACE: Dynamic Neural Radiance Fields for Monocular 4D Facial AvatarReconstruction (CVPR 2021 Oral Presentation), hereinafter referred to as NerFACE;

[0084] Neural Head Avatars from Monocular RGB Videos (CVPR 2022) is referred to as NHA;

[0085] We trained on monocular RGB videos of ten subjects with different hairstyles, genders, skin colors, and ages. The table below shows the difference between the generated results and the true values ​​under five different metrics on the test set:

[0086] MSE(1e-3)↓ L1(1e-2)↓ PSNR↑ SSIM(1e-1)↑ LPIPS(1e-2)↓ FOMM 1.75(0.81) 1.87(0.52) 28.32(2.45) 9.29(0.28) 5.30(1.73) NHA 0.69(0.54) 0.80(0.29) 32.85(3.01) 9.69(0.16) 3.37(1.88) NerFACE 0.75(0.45) 0.84(0.30) 32.24(2.70) 9.67(0.15) 3.50(1.64) Method of the present invention 0.48(0.32) 0.70(0.23) 34.15(2.58) 9.73(0.13) 2.67(1.32)

[0087] The values ​​outside the brackets are the mean values, and the values ​​inside the brackets are the standard deviations. It can be seen that the neural radiation field base constructed by the method of the present invention outperforms similar methods in terms of rendering quality and model expression ability. The rendering image results are shown as follows Figure 2 shown.

[0088] 2. Comparison of training efficiency

[0089] In the training task of human head NeRF, the method widely used so far is to concatenate the expression coefficients and position codes as the input of the multi-layer perceptron to achieve the purpose of dynamic expression driving, and the position code generally uses Fourier coding. Similar strategies can be seen in famous works such as ADNeRF, HeadNeRF, and NerFACE. Here we show that our multi-level voxel field position coding method is better than Fourier coding. At the same time, our expression combination structure such as neural radiation field basis can more effectively use expression coefficients to globally control local features compared to tensor splicing, thereby reducing the learning pressure of the multi-layer perceptron. NerFACE is a structure of Fourier coding plus tensor splicing. The present invention also implements a structure (concatenate case) of direct tensor splicing of multi-layer voxel features and expression coefficients as a benchmark method, and then compares the training efficiency of the present invention method (NeRF blendshape) with the other two.

[0090] Figure 3 The curve of PSNR changing with training time on a certain identity test set is shown. Figure 3 The results show that the method of directly concatenating multi-level voxel features with expression coefficients is superior to NerFACE, demonstrating that the use of multi-level voxel features can improve training efficiency and the expressiveness of the model. The method of the present invention is faster than the method of directly concatenating multi-level voxel features with expression coefficients, demonstrating that the neural radiation field substrate structure has advantages over tensor concatenation in training.

[0091] Figure 4 The rendering effects of the three methods on the test set when they are trained to different stages are shown. It can be seen that NerFACE cannot obtain a clear face within 20 minutes, while the baseline method and the method of the present invention can achieve convergence of the rigid area of ​​the face within 20 minutes. However, compared with the baseline method, the method of the present invention can learn the non-rigid deformation of the face in a shorter time. This not only illustrates the advantage of the multi-level voxel field in the scene convergence speed, but also verifies that the combined structure of the neural radiation field substrate can better reduce the learning pressure of the multi-layer perceptron compared to the tensor splicing structure.

[0092] 3. Application display

[0093] By extracting the expression coefficient of the source identity and then multiplying it by the neural radiation field basis of the corresponding identity, cross-identity expression-driven applications can be realized after volume rendering, such as Figure 5 shown.

[0094] Since the neural radiation field base itself is a three-dimensional face model, it also supports new perspective synthesis applications of the human head. Figure 6It is the result of adjusting the perspective rendering of a human head in a certain frame.

[0095] In summary, compared with other methods for constructing parametric head models, the present invention has the following advantages:

[0096] 1) The use of multi-level voxel fields enables our head model to simultaneously capture and learn facial details at different scales. Our blendshape base structure pre-modulates positional encodings, reducing the learning burden on the multi-layer perceptron. This results in faster head model construction and higher-quality generated images.

[0097] 2) The present invention extends the concept of blendshape basis from the traditional explicit mesh model to NeRF. The neural radiation field basis of the present invention not only has the same decoupled semantics as the mesh model, but also can better reflect the personalized facial muscles and texture changes of the human face, thereby better modeling dynamic human head scenes.

[0098] 3) Volume rendering allows for fully differentiable supervised training using images of human heads. The semantic segmentation loss function allows the model to learn geometric distributions more quickly. The perceptual loss function allows the modeling results to include more facial details.

[0099] 4) The use of density grids can enable sampling to skip empty areas, improving sampling efficiency. Our density grid update strategy, which takes into account the expression coefficient, can take into account the maximum range that a face can reach.

[0100] Through the description of the above embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented through software or by using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or mobile hard disk) and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in various embodiments of the present invention.

[0101] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A personalized human head neural radiation field base representation and reconstruction method based on multi-level hash coding, characterized by: include: Reconstruct the input RGB video based on the mesh blendshape basis; By solving the inverse rendering problem, the expression coefficient and camera parameters are obtained; Segment the head part of each frame to obtain the head semantic segmentation map; Combine the neural radiation field basis with the expression coefficient of each frame to obtain a multi-resolution hash table corresponding to the expression coefficient; Each basis of the neural radiation field basis is a radiation field expressed by a multi-level voxel field. The multi-level voxel field is an L-layer feature storage structure, where each layer is a voxel field and each voxel block stores features of length F. A multi-resolution hash table is used to compress and store the multi-level voxel field. The multi-resolution hash table h is a feature table of size L×T×F, where T is the hash table size. The head is represented as: Where I is the final rendered image, θ represents the learnable weights of the multilayer perceptron, Represents sampling, sampling point feature access and volume rendering, C is the camera internal and external parameters, h0∈R L×T×F A multi-resolution hash table representing an average face, H = {h1,h2,…,h K },h i ∈R L×T×F A multi-resolution hash table representing different expression offsets, K is the number of bases, w={w1,w2,…,w K }∈R K is the expression coefficient; The combination of the multi-resolution hash table corresponding to the expression coefficient is: The sampling point x in the head scene space is queried on the multi-resolution hash table h to obtain voxel features, and the voxel features are converted into sampling point RGB and volume density σ using a multi-layer perceptron, and the rendered image is obtained through volume rendering.

2. The method for representing and reconstructing a personalized human head neural radiation field base based on multi-level hash coding according to claim 1, characterized in that: During the sampling process, the density information in the density grid recording space is used to guide the sampling, and a density grid update strategy that considers the expression coefficient is adopted to fully consider the maximum range of facial expressions that can be achieved.

3. The method for representing and reconstructing a personalized human head neural radiation field base based on multi-level hash coding according to claim 2, characterized in that: For the density grid update strategy, the hash table corresponding to each basis Where N is the total number of frames in the video, is the i-th dimension element of the expression coefficient of the j-th frame, h0 represents the multi-resolution hash table of the average face, h i A multi-resolution hash table representing different expression offsets; take all The final density grid is obtained by taking the element-by-element maximum value of the corresponding density grid.

4. A personalized human head neural radiation field base representation and reconstruction method based on multi-level hash coding according to claim 2 or 3, characterized in that: The method for sampling light based on a density grid is as follows: sampling starts from the viewpoint, and at each sampling point, the density value is queried in the density grid. If it is above the threshold, sampling continues; if it is below the threshold, sampling is directly jumped to the intersection of the light and the next pixel in the grid.

5. A personalized human head neural radiation field base representation and reconstruction method based on multi-level hash coding according to claim 1 or 2, characterized in that: For any point x in space, the method for calculating its stored features in the multi-resolution hash table is as follows: first calculate the diagonal point index of x on each layer, input it into the hash function to obtain the hash value, use the hash value to find the feature in the hash table, and then use linear interpolation to find the obtained feature to obtain the voxel feature corresponding to x.

6. The method for representing and reconstructing a personalized human head neural radiation field base based on multi-level hash coding according to claim 1, characterized in that: Multilayer Perceptron g θ The voxel features of the sampling point x are converted into sampling point RGB and volume density, expressed as: Among them, c, σ represent the sampling point RGB and volume density respectively, η(x; h) represents the voxel feature obtained by querying the sampling point x on the hash table h, and γ(d) represents the position encoding of the view vector d.

7. The method for representing and reconstructing a personalized human head neural radiation field base based on multi-level hash coding according to claim 1, characterized in that: The rendered image is obtained by volume rendering. The calculation formula for rendering a single ray RGB is: Where I(r) represents the color corresponding to the light r, t and s are the parameterized variables of the light, c(r(t)) is the RGB corresponding to the point r(t) on the light, and σ(r(s)) and σ(r(t)) are the volume densities corresponding to the points r(s) and r(t) on the light, respectively.

8. The method for representing and reconstructing a personalized human head neural radiation field base based on multi-level hash coding according to claim 1, characterized in that: During the training phase, the neural radiation field basis is obtained through the supervision of image RGB, head semantic segmentation map and perceptual loss function; at the beginning of training, the neural radiation field basis is randomly initialized and continuously updated during the training process.

Images