Three-dimensional sight line redirection method based on neural radiation field

Abstract

The invention discloses a three-dimensional sight line redirection method based on a neural radiation field, and the method comprises the following steps: S1, collecting and processing a data set, and carrying out the face model processing of an image in the data set; s2, carrying out feature coding and carrying out constraint learning on a multi-layer perceptron (MLP); s3, extracting a three-plane feature T by using a three-plane feature extractor EF through the image and the three-dimensional face parameters obtained in the step S1, and performing volume rendering through the three-plane feature T to generate a low-resolution feature FT; and S4, performing model overall optimization by executing a gradient descent method to update the overall weight of the model, and performing iterative training for a plurality of rounds. According to the three-dimensional sight line redirection method based on the neural radiation field, end-to-end sight line redirection model training is achieved, explicit decoupling of the head posture and the gazing direction is achieved under the condition that a single-view-angle image is used, and the effect of controlling the gazing direction and the head posture at the same time is achieved.

Description

Technical Field

[0001] The present invention relates to the technical fields of image generation, sight line estimation and sight line redirection, and in particular to a three-dimensional sight line redirection method based on a neural radiation field. Background Art

[0002] Gaze estimation technology uses various technical means to determine a person's gaze direction, while gaze redirection refers to changing the gaze direction or head posture of the target image based on the existing gaze tag or the gaze direction of the existing image. The authenticity of face rendering has long been the focus of researchers' exploration, and gaze redirection technology can help present more realistic face rendering results, achieving the effect of "finishing touches", and gaze redirection technology can be widely used in various fields, such as virtual reality (VR), digital human (DigitalHuman), film and television special effects production (CG) and other fields, bringing a more realistic perception and user experience.

[0003] The sight line of a person is affected by two factors in the face, one is the rotation of the eyeball, and the other is the rotation of the head posture. For the sight line redirection of the face, it is possible to process the eye image or process the eyeball and head posture at the same time. For the redirection task of the eye image, since the operated image is often the aligned eye image, and the redirection process only involves the change of the pixels in the eye area, the whole process is often relatively simple, as described in patent CN114879843B. However, for the sight line redirection of the face image, it is necessary to comprehensively consider the movement of the eyeball and the head posture. However, in the single-view face image, the eyeball and the head posture are closely linked. Separating the two from a physical point of view may involve a complex modeling process. The present invention uses a deep learning network to learn the relationship between the eyeball and the head posture, and then uses the camera extrinsic parameters to explicitly change the head posture, so as to achieve the ideal redirection effect.

[0004] For the gaze redirection task, single-view images are easier to obtain and the cost of image acquisition is relatively low. However, when using single-view images for redirection, it is often difficult to handle the control of head posture, because the information provided by single-view images is often very limited, and the gaze direction and head posture are often coupled and nested in the high-dimensional feature space. The features provided by a single view are not enough to support the decoupling of the gaze features and head posture features, and it is difficult to achieve the control of the gaze direction and head posture separately. Multi-view images provide more facial feature information, and more importantly, multi-view images are accompanied by position information in three-dimensional space, which creates the possibility of decoupling the gaze features and head posture features of the display. However, the cost of multi-view image acquisition is more expensive, and it is usually impossible to define how many different perspectives are needed to achieve the purpose of redirection.

[0005] In order to solve the above problems, the present invention proposes a new three-dimensional gaze redirection method, which is based on a single-perspective facial image, utilizes the existing three-plane feature extractor to provide spatial position information, and combines neural radiation field technology to generate facial images with different gaze directions and head posture directions. Summary of the invention

[0006] The purpose of the present invention is to provide a method for three-dimensional gaze redirection based on neural radiation field, realize an end-to-end gaze redirection model training, and realize the display decoupling of head posture and gaze direction when using single-view images, so as to achieve the effect of simultaneously controlling gaze direction and head posture.

[0007] The present invention provides a method for three-dimensional sight line redirection based on neural radiation field, comprising the following steps: 1. A method for three-dimensional sight line redirection based on neural radiation field, characterized in that it comprises the following steps:

[0008] S1, collect and process the data set, and process the images in the data set into face models;

[0009] S2, the model passes through the feature encoder E GH Perform feature encoding and constraint learning with multi-layer perceptron MLP;

[0010] The initial stage of the model uses the line of sight feature and head posture feature encoder E GH For the input face image I s and I d Perform feature extraction to obtain the sight features F G and head posture feature F H ;

[0011] The model uses a multi-layer perceptron MLP to extract the line of sight features F G and head posture feature F H Perform regression, and constrain the generated gaze direction and head posture direction with the true label;

[0012] S3, using the image and three-dimensional face parameters obtained in step S1, using the three-plane feature extractor E F Extract the three-plane feature T and generate a low-resolution feature F by volume rendering of the three-plane feature T T , F T After the super-resolution module, a high-resolution sight redirection image I′s is obtained;

[0013] The model transforms image I s Input to the pre-trained 3D face reconstruction model E F In the above example, extract the three-plane feature T;

[0014] By sampling the three-plane feature space T, the model can generate the corresponding color c and volume density σ, and use the volume rendering formula to generate low-resolution features F T ;

[0015] In obtaining the low-resolution feature F T After that, the model transforms the low-resolution feature F T Combined with other feature channels, it is input into the super-resolution module to generate a high-resolution gaze redirection image I' s ;

[0016] The model introduces the line of sight consistency loss L cg , Line of sight feature consistency loss L cgf , identity feature consistency constraint L id , pre-trained image feature perception network L LPIPS ;

[0017] S4. Overall model optimization updates the overall model weights by executing the gradient descent method and iterating training for several rounds.

[0018] Preferably, in step S1, a data set of single-view images is collected, the data set of single-view images is cleaned, the single-view images are screened according to label information provided by the author of the data set, the line of sight is regularized on the single-view images, the background information in the single-view images is removed, and the single-view images are processed using the FLAME face model.

[0019] Preferably, in step S2, the model includes a line of sight feature and a head posture feature encoder E GH , pre-trained three-plane feature extractor E F , feature renderer R, a super-resolution module SR, two face images I s and I d , the camera extrinsic and intrinsic parameters corresponding to the image, and the information of the three-dimensional face.

[0020] Preferably, in step S2, the initial stage of the model is to use the line of sight feature and head posture feature encoder E GH For the input face image I s and I d Perform feature extraction to obtain the sight features F G and head posture feature F H ;

[0021] The encoding process can be expressed as:

[0022] F G , F H =E GH (I);

[0023] Among them, F G and F H Respectively represent the sight features and head posture features corresponding to the image;

[0024] Regress the gaze features and head posture features, predict the gaze direction and head posture direction, and constrain the obtained gaze direction and head posture direction with the real gaze label and head posture label respectively:

[0025] L gaze =||MLP G (F G )-g||;

[0026] L head =||MLP H (F H )-h||;

[0027] Among them, MLP G and MLP H represents the inference head of the sight feature and head posture feature, g and h represent the true sight direction and head posture direction respectively; and L head denote the constraints on using the true labels for the inferred gaze direction and head pose, respectively.

[0028] Preferably, in step S3, the model transforms image I s Input to the pre-trained three-plane feature extractor E F In the above example, extract the three-plane feature T;

[0029] In addition to the image I s In addition, E F It also accepts features from the line of sight and head pose encoder E GH Two sets of encoded sight features and head posture features:

[0030]

[0031] Represents image I s The three orthogonal two-dimensional feature planes obtained by encoding.

[0032] Respectively represent E GH For image I s and I d Encoded gaze features and head posture features.

[0033] Preferably, in step S3, the three-plane feature space can represent a point at any spatial position, and then a series of spatial sampling points can be generated. The three-plane feature space is sampled according to the position information of the sampling points. For any point x, the color c and volume density σ of the corresponding sampling point can be obtained:

[0034] (c,σ)=MLP(T xy (x)+T yz (x)+T zx (x));

[0035] Among them, c corresponds to the color of the sampling point; σ corresponds to the volume density of the sampling point; x is the three-dimensional position coordinate, T. represents the feature of the projection of x on the corresponding plane, and a fully connected layer is used to encode the feature after the sum of the three features. The dimension of σ is one-dimensional, and the dimension of c can be specified arbitrarily;

[0036] After obtaining the color c and volume density σ of all sampling points, the volume rendering formula in the neural radiance field can be used to render all points sampled according to the camera extrinsics:

[0037] F T =R(c,σ,θ);

[0038] F T represents the low-resolution feature obtained after rendering, θ is the external parameter of the camera, and the image I is used d The camera extrinsics of

[0039] For the low-resolution feature F T For example, the first three channels are interpreted as the RGB channels in the image, and other feature channels are sent to the super-resolution module to obtain a high-resolution image with a changed line of sight:

[0040] I′ s =SR(I′ ls , F T [3:]);

[0041] I′ ls =F T [:3];

[0042] I' ls and I' s Denote the high-resolution and low-resolution gaze redirection images respectively.

[0043] Preferably, in step S3, the model introduces a line of sight consistency loss L cg , Line of sight feature consistency loss L cgf , identity feature consistency constraint L id , pre-trained image feature perception network L LPIPS :

[0044] To generate image I' s and the original image I d Use the line-of-sight constraint:

[0045] L cg =1-cos(GazeNet(I′ s ),GazeNet(I′ d ));

[0046] Among them, GaxeNet() means to obtain the image line of sight direction; L cg Represents the aligned redirected image I' s With the source image I d The direction of sight between

[0047] The consistency loss of line of sight features is added to assist in the generation of line of sight consistent images:

[0048] L cgf =1-cos(GazeFea(I′ s ),GazeFea(I d ));

[0049] GaxeFea() means to obtain the sight line features of the image, L cgf Indicates that the image I' will be redirected at the feature level s With the source image I d Alignment of sight features between them;

[0050] Add an identity consistency constraint:

[0051] L id =1-cos(IDFea(I′ s ),IDFea(I s ));

[0052] Where IDFea() means to obtain the identity features of the image; L id Indicates that image I' will be redirected s With the source image I s Align the identity features between them and retain the key attributes of the face;

[0053] The learning-aware image patch similarity loss is used to constrain the generated images:

[0054] L LPIPS =LPIPS(I′ s , I s );

[0055] LPIPS is a pre-trained image feature perception network;

[0056] Preferably, the specific steps of the gradient descent method are as follows: First, calculate L tot ,

[0057] L tot =L gaze +h ead +λ 1 L cg +λ 2 c gf +λ 3 L id +λ 4 L LPIPS ;

[0058] Then find L tot The gradient with respect to the model parameters θ Then update the parameters in the direction of this gradient, and the update formula is:

[0059]

[0060] α represents the learning rate; L tot represents the sum of all losses; Represents the gradient of the model during the iteration.

[0061] Therefore, the present invention adopts the above-mentioned method of three-dimensional gaze redirection based on neural radiation field to realize an end-to-end gaze redirection model training. When using a single-view image, it realizes the display decoupling of head posture and gaze direction, thereby achieving the effect of simultaneously controlling gaze direction and head posture.

[0062] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is an overall flow chart of a method for three-dimensional sightline redirection based on neural radiation field of the present invention;

[0064] Figure 2 This is an overall structural diagram of a CNN model of a method for three-dimensional sightline redirection based on neural radiation field in the present invention. DETAILED DESCRIPTION

[0065] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0066] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0067] The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprises" and similar terms mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0068] Embodiment 1

[0069] like Figure 1-Figure 2 As shown, the present invention provides a method for three-dimensional sightline redirection based on neural radiation field, comprising the following steps:

[0070] S1, collect and process the data set, and process the images in the data set into face models;

[0071] Collect a data set of monoscopic images for gaze redirection, clean the data set of monoscopic images to ensure that there are clear face images in the monoscopic images, and filter the monoscopic images according to the label information provided by the author of the data set to ensure the accuracy of the gaze direction. Regularize the monoscopic images to eliminate the influence of head posture on the gaze. The size of the face in the regularized image is basically the same, ensuring the consistency of gaze estimation and gaze redirection. In the process of processing, the gaze direction and head posture in the camera coordinate system are obtained, the background information in the monoscopic images is removed to prevent the extra information in the background from interfering with the redirection, and the monoscopic images are processed using the FLAME face model; obtain the three-dimensional face parameters and face posture information.

[0072] S2, the model passes through the feature encoder E GH Perform feature encoding and constraint learning with multi-layer perceptron MLP;

[0073] The model includes the line of sight feature and head posture feature encoder E GH , pre-trained three-plane feature extractor E F , feature renderer R, a super-resolution module SR, two face images I s and I d , the camera extrinsic and intrinsic parameters corresponding to the face image, and the information of the three-dimensional face.

[0074] The initial stage of the model uses the line of sight feature and head posture feature encoder E GH For the input face image I sand I d Perform feature extraction to obtain the sight features F G and head posture feature F H ;

[0075] The encoding process can be expressed as:

[0076] F G , F H =E GH (I);

[0077] Among them, F G and F H Respectively represent the sight features and head posture features corresponding to the image;

[0078] The model uses a multi-layer perceptron MLP to extract the line of sight features F G and head posture feature F H Perform regression, and constrain the generated gaze direction and head posture direction with the true label;

[0079] Regress the gaze features and head posture features, predict the gaze direction and head posture direction, and constrain the obtained gaze direction and head posture direction with the real gaze label and head posture label respectively:

[0080] L gaze =||MLP G (F G )-g||;

[0081] L head =||MLP H (F H )-h||;

[0082] Among them, MLP G and MLP H L represents the inference head of the sight feature and head posture feature, g and h represent the true sight direction and head posture direction respectively. gaze and L head denote the constraints on using the true labels for the inferred gaze direction and head pose, respectively.

[0083] The purpose is to ensure that the model can learn the line of sight information and head posture information in the image, and in the high-dimensional feature space, the line of sight feature F G and head posture feature F H Highly coupled, there is no explicit decoupling of the two features, but the head posture feature encoder E GH Automatically learn the relationship between gaze and head pose.

[0084] S3, using the image and three-dimensional face parameters obtained in step S1, using the three-plane feature extractor E F Extract the three-plane feature T and generate a low-resolution feature F by volume rendering of the three-plane feature T T , F T After the super-resolution module, a high-resolution gaze redirection image I′ is obtained. s ;

[0085] The model transforms image I s Input to the pre-trained 3D face reconstruction model E F In the above example, extract the three-plane feature T;

[0086] E F It also receives the features from the line of sight and head pose encoder E GH Two sets of encoded sight features and head posture features:

[0087]

[0088] They represent three orthogonal two-dimensional feature planes obtained by encoding the image Is; Respectively represent E GH For image I s and I d The encoded sight features and head posture features. Through these three feature planes, the feature information of any point in the three-dimensional space can be described.

[0089] Since the pre-trained three-plane feature extractor E F The built-in ability to perceive three-dimensional facial information can generate a standard three-plane feature representation of the face, and at the same time use the gaze features and head posture features of the two images as additional information prompts for gaze redirection.

[0090] S3.2. By sampling the three-plane feature space T, the model can generate the corresponding color c and volume density σ, and use the volume rendering formula to generate low-resolution features F T ;

[0091] The three-plane feature space can represent a point at any spatial position, and then a series of spatial sampling points can be generated. The three-plane feature space is sampled according to the position information of the sampling points. For any point x, the color c and volume density σ of the corresponding sampling point can be obtained:

[0092] (c,σ)=MLP(T xy (x)+T yz (x)+T zx (x));

[0093] Among them, x is the three-dimensional position coordinate, T. represents the feature of the projection of x on the corresponding plane, and a fully connected layer is used to encode the feature after the sum of the three features. The dimension of σ is one-dimensional, and the dimension of c can be specified arbitrarily; the higher the dimension, the more feature information and the higher the video memory consumption, and the lower the dimension, the opposite is true.

[0094] S3.3, after obtaining the low-resolution feature F T After that, the model transforms the low-resolution feature F T Combined with other feature channels, it is input into the super-resolution module to generate a high-resolution gaze redirection image I' s ;

[0095] After obtaining the color c and volume density σ of all sampling points, all points can be rendered using the volume rendering formula in the neural radiance field:

[0096] F T =R(c,σ,θ);

[0097] Color c and volume density σ are the information of all sampling points; F T represents the low-resolution feature obtained after rendering, θ is the external parameter of the camera, and the image I is used d The camera extrinsics;

[0098] The camera extrinsics can be understood as taking pictures from different angles. If the position of the face is fixed, the position of the head posture can be controlled by controlling the position of the camera. Let the line of sight and head posture encoder learn the relationship between the line of sight and head posture autonomously. The encoder converts I s The gaze feature is converted to I d That is to say, by letting the coupled sight line and head posture features be processed by the deep neural network, and by controlling the external parameters of the camera to explicitly process the head posture, we can solve the problem of being able to control the gaze position and head posture well at the same time in the case of a small number of images.

[0099] For the low-resolution feature F T For example, the first three channels are interpreted as the RGB channels in the image, and are sent to the super-resolution module with other feature channels to obtain a high-resolution image with a changed line of sight:

[0100] I′ s =SR(I′ ls , F T [3:]);

[0101] I′ ls =F T [:3];

[0102] I' lsand I' s Denote the high-resolution and low-resolution gaze redirection images respectively.

[0103] After obtaining these two images, the network as a whole is constrained. In order to ensure the consistency of the line of sight before and after generation, the model introduces the line of sight consistency loss L cg , Line of sight feature consistency loss L cgf , identity feature consistency constraint L id , pre-trained image feature perception network L LPIPS :

[0104] To generate image I' s and the original image I d Use the line-of-sight constraint:

[0105] L cg =1-cos(GazeNet(I′ s ),GazeNet(I d ));

[0106] Among them, GazeNet() means to obtain the image line of sight direction; L cg Represents the aligned redirected image I' s With the source image I d The direction of sight between

[0107] To ensure the visual consistency of the final generated image, the consistency loss of sight features is added to assist in the generation of sight consistency images:

[0108] L cgf =1-cos(GazeFea(I′ s ),GazeFea(I d ));

[0109] GazeFea() means to obtain the sight features of the image, L cgf Indicates that the image I' will be redirected at the feature level s With the source image I d The line of sight features between them are aligned; L cgf Constraining the line of sight at the feature level can further ensure the consistency between the line of sight features of the generated image and the line of sight features of the target image.

[0110] To prevent the image I d The encoded features have an impact on the generated images because during the training process I d May come from different objects, adding a constraint on identity consistency:

[0111] L id =1-cos(IDFea(I′s ),IDFea(I s ));

[0112] Where IDFea() means to obtain the identity features of the image, L id Indicates that image I' will be redirected s With the source image I s The identity features between them are aligned to retain the key attributes of the face; a pre-trained face feature extractor is used to ensure that the original image I s and generate image I' s The consistency of identity can preserve the details of the identity information in the original image. It is worth mentioning that the present invention does not use pixel-level reconstruction loss to constrain the images before and after generation, because under the premise of generating a good redirection effect, the generated image I' s and the original image I s and I d There are pixel-level differences between them. Using pixel-level reconstruction loss is not conducive to preserving the characteristic details of the human eye area.

[0113] The learning-aware image patch similarity loss is used to constrain the generated images:

[0114] L LPIPS =LPIPS(I′ s , I s );

[0115] LPIPS is a pre-trained image feature perception network. By comparing the differences between two images in the feature space, it takes into account the perceptual characteristics of the human visual system to images, can better reflect the visual similarity of images, and is more tolerant to subtle pixel differences than pixel reconstruction loss.

[0116] S4. Overall model optimization updates the overall model weights by executing the gradient descent method, and iterates the training for several rounds to ensure that the model can generate high-fidelity images that convert the line of sight; as shown in the following formula:

[0117] L tot =L gaze +L head +λ 1 L cg +λ 2 L cgf +λ 3 L id +λ 4 L LPIPS ;

[0118] The specific steps of the gradient descent method are as follows: First, calculate L tot , then find L totThe gradient with respect to the model parameters θ Then update the parameters in the direction of this gradient, and the update formula is:

[0119]

[0120] Among them, α represents the learning rate; L tot represents the sum of all losses; Represents the gradient of the model during the iteration.

[0121] By integrating the above steps into an overall end-to-end deep neural network framework, end-to-end model training can be performed.

[0122] Therefore, the present invention adopts the above-mentioned method of three-dimensional gaze redirection based on neural radiation field to realize an end-to-end gaze redirection model training. When using a single-view image, it realizes the display decoupling of head posture and gaze direction, thereby achieving the effect of simultaneously controlling gaze direction and head posture.

[0123] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for three-dimensional sight line redirection based on neural radiation field, characterized in that: The following steps are involved: S1, collect and process the data set, and process the images in the data set into face models; S2, the model passes through the feature encoder E GH Perform feature encoding and constraint learning with multi-layer perceptron MLP; The initial stage of the model uses the line of sight feature and head posture feature encoder E GH For the input face image I s and I d Perform feature extraction to obtain the sight features F G and head posture feature F H ; The model uses a multi-layer perceptron MLP to extract the line of sight features F G and head posture feature F H Perform regression, and constrain the generated gaze direction and head posture direction with the true label; S3, using the image and three-dimensional face parameters obtained in step S1, using a three-plane feature extractor, extracting three-plane features T, and generating low-resolution features F by volume rendering the three-plane features T. T , F T After the super-resolution module, a high-resolution gaze redirection image I′ is obtained. s ; The model transforms image I s Input to the pre-trained 3D face reconstruction model E F In the above example, extract the three-plane feature T; By sampling the three-plane feature space T, the model can generate the corresponding color c and volume density σ, and use the volume rendering formula to generate low-resolution features F T ; In obtaining the low-resolution feature F T After that, the model transforms the low-resolution feature F T Combined with other feature channels and input into the super-resolution module to generate a high-resolution gaze redirection image I′ s ; The model introduces the line of sight consistency loss L cg , Line of sight feature consistency loss L cgf , identity feature consistency constraint L id , pre-trained image feature perception network L LPIPS ; S4. Overall model optimization updates the overall model weights by executing the gradient descent method and iterating training for several rounds.

2. The method for three-dimensional sightline redirection based on neural radiation field according to claim 1, characterized in that: In step S1, a data set of single-view images is collected, the data set of single-view images is cleaned, and the single-view images are screened according to the label information provided by the author of the data set, the line of sight of the single-view images is regularized, the background information in the single-view images is removed, and the single-view images are processed using the FLAME face model.

3. The method for three-dimensional sightline redirection based on neural radiation field according to claim 1, characterized in that: In step S2, the model includes the line of sight feature and head posture feature encoder E GH , pre-trained three-plane feature extractor E F , feature renderer R, a super-resolution module SR, two face images I s and I d , the camera extrinsic and intrinsic parameters corresponding to the image, and the information of the three-dimensional face.

4. The method for three-dimensional sightline redirection based on neural radiation field according to claim 1, characterized in that: In step S2, the initial stage of the model is through the line of sight feature and head posture feature encoder E GH For the input face image I s and I d Perform feature extraction to obtain the sight features F G and head posture feature F H ; The encoding process can be expressed as: F G ,F H =E GH (I); Among them, F G and F H Respectively represent the sight features and head posture features corresponding to the image; Regress the gaze features and head posture features, predict the gaze direction and head posture direction, and constrain the obtained gaze direction and head posture direction with the real gaze label and head posture label respectively: L gaze =||MLP G (F G )-g||; L head =||MLP H (F H )-h||; Among them, MLP G and MLP H They represent the inference heads of sight features and head posture features, g and h represent the true sight direction and head posture direction, respectively; L gaze and L head denote the constraints on using the true labels for the inferred gaze direction and head pose, respectively.

5. The method for three-dimensional sightline redirection based on neural radiation field according to claim 1, characterized in that: In step S3, the model transforms image I s Input to the pre-trained three-plane feature extractor E F In the above example, extract the three-plane feature T; In addition to the image I s In addition, E F It also accepts features from the line of sight and head pose encoder E GH Two sets of encoded sight features and head posture features: Represents image I s The three orthogonal two-dimensional feature planes obtained by encoding; Respectively represent E GH For image I s and I d Encoded gaze features and head posture features.

6. The method for three-dimensional sightline redirection based on neural radiation field according to claim 1, characterized in that: In step S3, the three-plane feature space can represent a point at any spatial position, and then a series of spatial sampling points can be generated. The three-plane feature space is sampled according to the position information of the sampling points. For any point x, the color c and volume density σ of the corresponding sampling point can be obtained: (c,σ)=MLP(T xy (x)+T yz (x)+T zx (x)); Among them, c corresponds to the color of the sampling point; σ corresponds to the volume density of the sampling point; x is the three-dimensional position coordinate, T· represents the feature of the projection of x on the corresponding plane, and a fully connected layer is used to encode the feature after the sum of the three features. The dimension of σ is one-dimensional, and the dimension of c can be specified arbitrarily; After obtaining the color c and volume density σ of all sampling points, the volume rendering formula in the neural radiance field can be used to render all points sampled according to the camera extrinsics: F T =R(c,σ,θ) ； F T represents the low-resolution feature obtained after rendering, θ is the external parameter of the camera, and the image I is used d The camera extrinsics; For the low-resolution feature F T For example, the first three channels are interpreted as the RGB channels in the image, and are sent to the super-resolution module with other feature channels to obtain a high-resolution image with a changed line of sight: I′ s =SR(I′ ls ,F T [3:]); I′ ls =F T [:3]; I′ ls and I′ s Denote the high-resolution and low-resolution gaze redirection images respectively.

7. The method for three-dimensional sightline redirection based on neural radiation field according to claim 1, characterized in that: In step S3, the model introduces the line of sight consistency loss L cg , Line of sight feature consistency loss L cgf , identity feature consistency constraint L id , pre-trained image feature perception network L LPIPS : To generate image I′ s and the original image I d Use the line-of-sight constraint: L cg =1-cos(GazeNet(I′ s ),GazeNet(I d )); Among them, GazeNet() means to obtain the image line of sight direction; L cg Represents the aligned redirected image I′ s With the source image I d The direction of sight between The consistency loss of line of sight features is added to assist in the generation of line of sight consistent images: L cgf =1-cos(GazeFea(I′ s ),GazeFea(I d )); GazeFea() means to obtain the sight line features of the image, L cgf Indicates that the image I′ will be redirected at the feature level s With the source image I d Alignment of sight features between them; Add an identity consistency constraint: L id =1-cos(IDFea(I′ s ),IDFea(I s )); Where IDFea() means to obtain the identity features of the image; L id Indicates that the image I′ will be redirected s With the source image I s Align the identity features between them and retain the key attributes of the face; The learning-aware image patch similarity loss is used to constrain the generated images: L LPIPS =LPIPS(I′ s ,I s ); LPIPS is a pre-trained image feature perception network.

8. The method for three-dimensional sightline redirection based on neural radiation field according to claim 1, characterized in that: The specific steps of the gradient descent method are as follows: First, calculate L tot , L tot =L gaze +L head +λ1L cg +λ2L cgf +λ3L id +λ4L LPIPS ; Then find L tot The gradient with respect to the model parameters θ Then update the parameters in the direction of this gradient, and the update formula is: α represents the learning rate; L tot represents the sum of all losses; Represents the gradient of the model during the iteration.

Images