A method of three-dimensional line-of-sight redirection based on neural radiance fields

Abstract

The application discloses a kind of methods for three-dimensional line of sight redirection based on neural radiation field, comprising the following steps: S1, gather and process data set, image in data set is handled face model;S2, feature coding and multilayer perception MLP are carried out constraint learning;S3, by the image obtained in step S1 and three-dimensional face parameters, utilize three-plane feature extractor E F Three-plane feature T is extracted and low-resolution feature F is generated by volume rendering through three-plane feature T T ;S4, model overall optimization is updated by executing gradient descent method weight of model whole, and iteratively trains several rounds.The application uses the above-mentioned method for three-dimensional line of sight redirection based on neural radiation field, realizes an end-to-end line of sight redirection model training, realizes explicit decoupling of head posture and gaze direction in the case of using single-view image, achieves the effect of simultaneously controlling gaze direction and head posture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image generation, gaze estimation and gaze redirection, and in particular to a method for three-dimensional gaze redirection based on neural radiance field. BACKGROUND

[0002] Gaze estimation technology determines the gaze direction of a person through various technical means, and gaze redirection refers to changing the gaze direction or head pose of a target image according to the existing gaze label or the gaze direction of the existing image. The realism of face rendering has long been the focus of researchers, and gaze redirection technology can help present more realistic face rendering results, achieving the effect of "drawing the dragon's eye", and gaze redirection technology can be widely applied to various fields, such as virtual reality (VR), digital human, film and television special effect production (CG), etc., bringing a more realistic visual experience and use experience.

[0003] The gaze of a person is affected by two factors in the face, one is the rotation of the eyeball, and the other is the rotation of the head pose. The gaze redirection of the face can be processed for the eye image, or the eyeball and the head pose can be processed at the same time. For the redirection task of the eye image, the image operated is usually the aligned eye image, and the redirection process only involves the change of the pixel in the eyeball region, so the whole process is usually relatively simple, as described in patent CN114879843B. However, for the gaze redirection of the face image, the motion of the eyeball and the head pose must be considered comprehensively, but in a single-view face image, the eyeball and the head pose are closely related, and separating them from a physical point of view may involve a complex modeling process. The present application uses a deep learning network to learn the relationship between the eyeball and the head pose, and then uses the camera extrinsic parameter to explicitly change the head pose to achieve the desired redirection effect.

[0004] For the gaze redirection task, single-view images are easier to obtain and the cost of image acquisition is relatively low, but when using single-view images for redirection, it is often difficult to control the head pose, because the information provided by the single-view image is very limited, and the gaze direction and the head pose are often coupled and nested in a high-dimensional feature space. The features provided by the single-view image are not enough to support the decoupling of the gaze features and the head pose features, so it is difficult to achieve separate control of the gaze direction and the head pose. Multi-view images provide more face feature information, and more importantly, multi-view images contain position information in three-dimensional space, which creates the possibility of decoupling the gaze features and the head pose features, but the cost of multi-view image acquisition is more expensive, and it is usually impossible to define how many different view images are needed to complete the redirection purpose.

[0005] To solve the above problems, the application provides a new three-dimensional view redirection method, which is based on a single-view face image, uses an existing three-plane feature extractor to provide spatial position information, combines a neural radiance field technology, and generates a face image with different view directions and head posture directions. SUMMARY

[0006] The application aims to provide a neural radiance field-based three-dimensional view redirection method, realize an end-to-end view redirection model training, realize display decoupling of head posture and gaze direction in the case of using a single-view image, and achieve the effect of simultaneously controlling the gaze direction and the head posture.

[0007] The application provides a neural radiance field-based three-dimensional view redirection method, which comprises the following steps: 1. A neural radiance field-based three-dimensional view redirection method, characterized in that it comprises the following steps:

[0008] S1, collect and process a data set, and perform face model processing on images in the data set;

[0009] S2, the model is processed by a feature encoder E GH , feature encoding and constraint learning are performed by a multilayer perceptron MLP;

[0010] The initial stage of the model is processed by a view feature and head posture feature encoder E GH , and an input face image I s is processed to obtain view features F d and head posture features F G , respectively;

[0011] The model uses a multilayer perceptron MLP to regress the extracted view features F H and head posture features F G , and the generated view direction and head posture direction are constrained with real labels;

[0012] S3, the image obtained in step S1 and three-dimensional face parameters are used to extract three-plane features T by a three-plane feature extractor E H , and low-resolution features F T are generated by volume rendering based on the three-plane features T T , and the F s is subjected to a super-resolution module to obtain a high-resolution view redirection image I′s;

[0013] The model inputs the image I F into a pre-trained three-dimensional face reconstruction model E T , and extracts three-plane features T;

[0014] By sampling the three-plane feature T space, the model can generate the corresponding color c and volume density σ, and generate a low-resolution feature F using the volume rendering formula T ;

[0015] After obtaining the low-resolution feature F T , the model combines the low-resolution feature F T with other feature channels and inputs them into the super-resolution module to generate a high-resolution view redirection image I' s ;

[0016] The model introduces a view consistency loss L cg , a view feature consistency loss L cgf , an identity feature consistency constraint L id , and a pre-trained image feature perception network L LPIPS ;

[0017] S4, the model overall optimization updates the weights of the whole model by performing gradient descent method, and iteratively trains for several rounds.

[0018] Preferably, in step S1, a single-view image dataset is collected, the single-view image dataset is cleaned, and the single-view image is filtered according to the label information provided by the author of the dataset, the single-view image is regularized, the background information in the single-view image is removed, and the single-view image is processed using the FLAME face model.

[0019] Preferably, in step S2, the model includes a view feature and head pose feature encoder E GH , a pre-trained three-plane feature extractor E F , a feature renderer R, a super-resolution module SR, two face images I s and I d , camera extrinsic and intrinsic parameters corresponding to the images, and three-dimensional face information.

[0020] Preferably, in step S2, the initial stage of the model extracts features from the input face images I GH and I s using the view feature and head pose feature encoder E d , respectively, to obtain the view feature F G and the head pose feature F H .

[0021] The encoding process can be represented as:

[0022] F G , F H = E GH (I);

[0023] Among them, F G and F H Respectively represent the sight features and head posture features corresponding to the image;

[0024] Regress the gaze features and head pose features to predict the gaze direction and head pose direction, and constrain the obtained gaze direction and head pose direction with the real gaze label and head pose label respectively:

[0025] L gaze =||MLP G (F G )-g||;

[0026] L head =||MLP H (F H )-h||;

[0027] Among them, MLP G and MLP H represents the inference head of the gaze feature and head posture feature, g and h represent the true gaze direction and head posture direction respectively; and L head denote the constraints on using the true labels for the inferred gaze direction and head pose, respectively.

[0028] Preferably, in step S3, the model transforms image I s Input to the pre-trained three-plane feature extractor E F In the process, three-plane features T are extracted;

[0029] In addition to the image I s In addition, E F Also accepts the encoder E from the line of sight feature and head pose feature GH Two sets of encoded gaze features and head posture features:

[0030]

[0031] Represents image I s The three orthogonal two-dimensional feature planes obtained by encoding.

[0032] Respectively represent E GH For image I s and I d Encoded gaze features and head posture features.

[0033] Preferably, in step S3, the three-plane feature space can represent a point at any spatial position, and then a series of spatial sampling points can be generated to sample the three-plane feature space according to the position information of the sampling points. For any point x, the color c and the volume density σ of the corresponding sampling point can be obtained:

[0034] (c,σ)=MLP(T xy (x)+T yz (x)+T zx (x));

[0035] where c is the color of the corresponding sampling point; σ is the volume density of the corresponding sampling point; x is a three-dimensional position coordinate, T. represents the feature of the projection of x on the corresponding plane, and a fully connected layer is used to encode the feature after the three features are summed, the dimension of σ is one-dimensional, and the dimension of c can be arbitrarily specified;

[0036] After obtaining the color c and the volume density σ of all the sampling points, all the points obtained by sampling according to the camera extrinsic parameters can be rendered using the volume rendering formula in the neural radiance field:

[0037] F T =R(c,σ,θ);

[0038] F T represents the low-resolution feature obtained after rendering, and θ is the camera extrinsic parameter, which adopts the camera extrinsic parameter of the image I d .

[0039] For the low-resolution feature F T , the first three channels thereof are interpreted as the RGB channels in the image, and other feature channels are sent into the super-resolution module to obtain a high-resolution view redirection image:

[0040] I′ s =SR(I′ ls , F T [3:]);

[0041] I′ ls =F T [:3];

[0042] I' ls and I' s represent the high-resolution and low-resolution view redirection images, respectively.

[0043] Preferably, in step S3, the model introduces a view consistency loss L cg , a view feature consistency loss L cgf , an identity feature consistency constraint L id , and a pre-trained image feature perception network L LPIPS :

[0044] To generate image I' s and original image I d Use the constraint of gaze consistency:

[0045] L cg =1-cos(GazeNet(I′ s ),GazeNet(I′ d ));

[0046] Where GaxeNet() represents the gaze direction of the image; L cg represents the alignment of the gaze direction between the redirected image I' s and the source image I d ;

[0047] Add the consistency loss of the gaze feature to assist the generation of the gaze consistency image:

[0048] L cgf =1-cos(GazeFea(I′ s ),GazeFea(I d ));

[0049] GaxeFea() represents the gaze feature of the image, L cgf represents the alignment of the gaze feature between the redirected image I' s and the source image I d ;

[0050] Add a constraint of identity feature consistency:

[0051] L id =1-cos(IDFea(I′ s ),IDFea(I s ));

[0052] Where IDFea() represents the identity feature of the image; L id represents the alignment of the identity feature between the redirected image I' s and the source image I s , retaining the key attributes of the face;

[0053] Learn the perceptual image block similarity loss to constrain the generated image:

[0054] L LPIPS =LPIPS(I′ s , I s );

[0055] LPIPS is a pre-trained image feature perception network;

[0056] The specific steps of the gradient descent method are as follows: first, L tot ,

[0057] L tot = L gaze + h ead + λ1L cg + λ2c gf + λ3L id + λ4L LPIPS ;

[0058] Then, L tot is obtained, which is the gradient of the model parameter θ Then, the parameter is updated in the direction of the gradient, and the update formula is as follows:

[0059]

[0060] αwherein represents the learning rate; L tot represents the sum of all losses; represents the gradient of the model in the iteration process.

[0061] Therefore, the method for three-dimensional line-of-sight redirection based on neural radiation field is adopted, an end-to-end line-of-sight redirection model training is realized, the display decoupling of the head posture and the gaze direction is realized in the case of using a single-view image, and the effect of simultaneously controlling the gaze direction and the head posture is achieved.

[0062] The technical solutions of the present application will be further described in detail below with reference to the drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0063] Fig. 1 It is a whole flow chart of the method for three-dimensional line-of-sight redirection based on neural radiation field of the present application;

[0064] Fig. 2 It is a whole structure chart of the CNN model of the method for three-dimensional line-of-sight redirection based on neural radiation field of the present application. DETAILED DESCRIPTION

[0065] The technical solutions of the present application will be further described in detail below with reference to the drawings and embodiments.

[0066] Unless otherwise defined, the technical terms or scientific terms used in the present application should be understood as the usual meanings understood by those skilled in the art in the field of the present application.

[0067] The words "first", "second" and similar terms used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects preceding the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0068] Example 1

[0069] like Figs. 1-2 As shown, the present invention provides a three-dimensional sightline redirection method based on neural radiation field, comprising the following steps:

[0070] S1. Collect and process the data set, and process the images in the data set into a face model;

[0071] A dataset of monoscopic images for gaze redirection is collected and cleaned to ensure clear facial images. The images are then filtered based on the label information provided by the dataset authors to ensure the accuracy of gaze direction. The images are then normalized to eliminate the influence of head pose on gaze. The faces in the normalized images are roughly the same size, ensuring consistency between gaze estimation and gaze redirection. During this processing, the gaze direction and head pose in the camera coordinate system are obtained. Background information is removed from the monoscopic images to prevent any interference with the redirection. The monoscopic images are then processed using the FLAME face model to obtain 3D facial parameters and facial pose information.

[0072] S2, the model passes the feature encoder E GH Perform feature encoding and constrained learning with multi-layer perceptron MLP;

[0073] The model includes the gaze feature and head pose feature encoder E GH , pre-trained three-plane feature extractor E F , feature renderer R, a super-resolution module SR, two face images I s and I d , camera extrinsics and intrinsics corresponding to the face image, and three-dimensional face information.

[0074] The initial stage of the model is achieved by using the line of sight feature and head posture feature encoder E GH For the input face image I sand I d Feature extraction is performed to obtain the gaze feature F G and the head pose feature F H ;

[0075] The encoding process can be represented as:

[0076] F G , F H = E GH (I);

[0077] wherein F G and F H represent the gaze feature and the head pose feature corresponding to the image respectively;

[0078] The model uses a multi-layer perception MLP to regress the extracted gaze feature F G and the head pose feature F H , and the generated gaze direction and head pose direction are constrained by the real labels;

[0079] The gaze feature and the head pose feature are regressed to predict the gaze direction and the head pose direction, and the obtained gaze direction and head pose direction are respectively constrained by the real gaze label and the head pose label:

[0080] L gaze = ||MLP G (F G )-g||;

[0081] L head = ||MLP H (F H )-h||;

[0082] wherein MLP G and MLP H represent the inference heads of the gaze feature and the head pose feature respectively, and g and h represent the real gaze direction and the head pose direction respectively. L gaze and L head represent the constraints of using the real labels for the inferred gaze direction and head pose respectively.

[0083] The purpose is to ensure that the model can learn the gaze information and the head pose information in the image, and in the high-dimensional feature space, the gaze feature F G and the head pose feature F H are highly coupled, without explicitly decoupling the two features respectively, but letting the head pose feature encoder E GH automatically learn the relationship between the gaze and the head pose.

[0084] S3, through the image and three-dimensional face parameters obtained in step S1, using a three-plane feature extractor E F extract three-plane features T and generate low-resolution features F by volume rendering through three-plane features T T , F T get high-resolution view redirection image I' through the super-resolution module s ;

[0085] The model inputs the image I s into a pre-trained three-dimensional face reconstruction model E F , extracts three-plane features T;

[0086] E F also accepts two sets of view features and head pose features encoded by view feature and head pose feature encoders E GH :

[0087]

[0088] respectively represent three orthogonal two-dimensional feature planes encoded from image Is; respectively represent E GH encoded view features and head pose features of image I s and I d . Through these three feature planes, the feature information of any point in three-dimensional space can be described.

[0089] Due to the three-dimensional face information perception ability of the pre-trained three-plane feature extractor E F itself, it can generate standardized face three-plane feature representation, and the view features and head pose features of the two images are used as additional information prompts for view redirection prompts.

[0090] S3.2, by sampling the three-plane feature T space, the model can generate corresponding color c and volume density σ, and generate low-resolution features F T using volume rendering formula;

[0091] The three-plane feature space can represent a point at any spatial position, and then a series of spatial sampling points can be generated. According to the position information of the sampling points, the three-plane feature space is sampled. For any point x, the color c and volume density σ of the corresponding sampling point can be obtained:

[0092] (c,σ)=MLP(T xy (x)+T yz (x)+T zx (x));

[0093] Where x is the three-dimensional position coordinate, T represents the projection of x on the corresponding plane, and a fully connected layer is used to encode the features after summing the three features, and the dimension of sigma is one-dimensional, and the dimension of c can be arbitrarily specified; the higher the dimension means more feature information and higher memory consumption, and the lower the dimension is the opposite.

[0094] S3.3, obtaining low-resolution features F T After that, the model will obtain low-resolution features F T Combined with other feature channels, input into the super-resolution module to generate high-resolution gaze redirection image I' s ;

[0095] After obtaining the color c and the density sigma of all sampling points, the body rendering formula in the neural radiation field can be used to render all points:

[0096] F T =R(c, sigma, theta);

[0097] Color c and density sigma are information of all sampling points; F T represents the low-resolution feature obtained after rendering, and theta is the camera extrinsic parameter, which uses the camera extrinsic parameter of image I d ;

[0098] The camera extrinsic parameter can be understood as taking pictures from different angles. If the position of the face is fixed, the position of the head pose can be controlled by controlling the position of the camera. Let the gaze and head pose encoder learn the relationship between the gaze and the head pose independently, and the encoder converts the gaze point feature of I s into the gaze feature of I d , that is, by giving the coupled gaze and head pose features to the deep neural network for processing, and by controlling the camera extrinsic parameter to explicitly process the head pose, the problem of controlling the gaze position and the head pose well at the same time under the condition of a small number of images is solved.

[0099] For low-resolution features F T , the first three channel numbers are explained as RGB channels in the image, and other feature channels are sent into the super-resolution module to obtain high-resolution images that change the gaze:

[0100] I′ s =SR(I′ ls , F T [3:]);

[0101] I′ ls =F T [:3];

[0102] I' lsand I' s represent the high-resolution and low-resolution gaze redirection images, respectively.

[0103] After obtaining the two images, the entire network is constrained, and in order to ensure the consistency of the generated and original gaze, the model introduces gaze consistency loss L cg , gaze feature consistency loss L cgf , identity feature consistency constraint L id , pre-trained image feature perception network L LPIPS :

[0104] The generated image I' s and the original image I d use the gaze consistency constraint:

[0105] L cg =1-cos(GazeNet(I′ s ),GazeNet(I d ));

[0106] Where GazeNet() represents the gaze direction of the image; L cg represents the alignment of the gaze direction between the redirected image I' s and the source image I d ;

[0107] To ensure the visual consistency of the final generated image, a gaze feature consistency loss is added to assist the generation of gaze consistency images:

[0108] L cgf =1-cos(GazeFea(I′ s ),GazeFea(I d ));

[0109] GazeFea() represents the gaze feature of the image, and L cgf represents the alignment of the gaze feature between the redirected image I' s and the source image I d ; L cgf restricts the gaze at the feature level, which can further ensure the consistency of the gaze feature of the generated image and the gaze feature of the target image.

[0110] In order to prevent the features encoded by the image I d from affecting the generated image, because I d may come from different objects during the training process, an identity feature consistency constraint is added:

[0111] L id =1-cos(IDFea(I′s ), IDFea(I s ));

[0112] where IDFea() represents the identity feature of the image, L id represents the alignment of the identity features between the redirected image I' s and the source image I s , preserving the key attributes of the face; a pre-trained face feature extractor is used, which can ensure the consistency of the original image I s and the generated image I' s identity, and can preserve the details of the identity information in the original image. It is worth mentioning that the present application does not use pixel-level reconstruction loss to constrain the images before and after generation, because under the premise of generating good redirection effect, there are pixel-level differences between the generated image I' s and the original image I s and I d , and the use of pixel-level reconstruction loss is not conducive to preserving the feature details of the eye region.

[0113] The learning perception image block similarity loss is used to constrain the generated image:

[0114] L LPIPS = LPIPS(I′ s , I s );

[0115] LPIPS is a pre-trained image feature perception network; by comparing the differences between two images in the feature space, the human visual system considers the perception characteristics of the image, which can better reflect the visual similarity of the image, and the tolerance for subtle pixel differences is more relaxed than the pixel reconstruction loss.

[0116] S4, the overall model optimization updates the overall weight of the model by performing gradient descent method, and iteratively trains for several rounds to ensure that the model can generate high-fidelity view conversion images; as shown in the following formula:

[0117] L tot = L gaze + L head + λ1L cg + λ2L cgf + λ3L id + λ4L LPIPS ;

[0118] The specific steps of the gradient descent method are as follows: first, L tot is calculated, then L tot is calculated, and the gradient of the model parameters θ is calculated Then the parameters are updated in the direction of the gradient, and the formula of the update is:

[0119]

[0120] Wherein, a represents a learning rate; L tot Indicates the sum of all losses. Indicates the gradient of the model in the iteration process.

[0121] The above steps are integrated into an overall end-to-end deep neural network framework, so that end-to-end model training can be performed.

[0122] Therefore, the application adopts the above-mentioned method for three-dimensional line-of-sight redirection based on a neural radiation field, realizes end-to-end line-of-sight redirection model training, realizes display decoupling of head posture and gaze direction in the case of using a single-view image, and achieves the effect of simultaneously controlling the gaze direction and the head posture.

[0123] The above embodiments are only used to illustrate the technical solutions of the present application but not to limit it, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can still be modified or replaced by equivalents, and these modifications or replacements cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A method of neural-radiance-field based three-dimensional line-of-sight redirection, the method comprising: The method comprises the following steps: S1, collecting and processing a data set, and performing face model processing on images in the data set; S2, the model passes through a feature encoder Feature encoding is performed and constrained learning is performed with a multi-layer perceptron (MLP). The initial stage of the model encodes the gaze feature and the head pose feature through a gaze feature and a head pose feature encoder on the input face image and extracts features, respectively, to obtain the gaze feature and the head pose feature ; The model uses a multi-layer perceptron, MLP, to regress on the extracted gaze features and head pose features The generated gaze and head pose directions are constrained against the true labels; S3, extracting three-plane features from the image and three-dimensional face parameters obtained in step S1 using a three-plane feature extractor extracting three-plane features and passing the three-plane features performing volume rendering to generate low-resolution features , obtaining a high-resolution line-of-sight redirection image through a super-resolution module ; The model will image input into a pre-trained three-dimensional face reconstruction model In the pre-trained three-dimensional face reconstruction model, three-plane features are extracted; By sampling the three-plane feature space, the model is able to generate corresponding color and volume density , and generate low-resolution features using volume rendering equation ; After obtaining the low-resolution feature , the model will combine the low-resolution feature with other feature channels and input them into the super-resolution module to generate a high-resolution line-of-sight redirection image ; The model introduces a gaze consistency loss , a gaze feature consistency loss , an identity feature consistency constraint , a pre-trained image feature perception network ; S4, model overall optimization updates the weight of the overall model by performing gradient descent method, and iteratively trains for several rounds.

2. The method of claim 1, wherein, In step S1, a data set of single-view images is collected, the data set of single-view images is cleaned, and the single-view images are screened according to label information provided by an author of the data set, the single-view images are subjected to gaze regularization, background information in the single-view images is removed, and the single-view images are processed using a FLAME face model.

3. The method of claim 1, wherein, In step S2, the model includes a line-of-sight feature and head pose feature encoder , a pre-trained , a feature renderer , a super-resolution module , two face images and , camera extrinsic and intrinsic parameters and three-dimensional face information corresponding to the image pair.

4. The method of claim 1, wherein, In step S2, the initial stage of the model encodes the gaze feature and the head pose feature by a gaze feature and head pose feature encoder For the input face image And Feature extraction is performed to obtain the gaze feature And the head pose feature respectively; The encoding process can be represented as: ； wherein, and respectively represent the line-of-sight feature and the head pose feature corresponding to the image. The gaze feature and the head pose feature are subjected to regression, the gaze direction and the head pose direction are predicted, and the obtained gaze direction and head pose direction are respectively constrained by real gaze labels and head pose labels: ； ； wherein, and respectively represent inferred head of line-of-sight and head pose features, and respectively represent true line-of-sight and head pose directions; and respectively represent constraints using true labels on inferred line-of-sight and head pose.

5. The method of claim 1, wherein, In step S3, the model extracts three-plane features from the image input to the pre-trained three-plane feature extractor ; In addition to the image Two sets of gaze and head pose features encoded from the gaze feature and head pose feature encoders are also accepted:​ ； respectively represent images three orthogonal two-dimensional feature planes are obtained by encoding 、 、 、 respectively represent encoded gaze features and head pose features for the image and ​ 6. The method of claim 1, wherein, In step S3, the three-plane feature space can represent a point at any spatial position, and then a series of spatial sampling points can be generated, and the three-plane feature space is sampled according to the position information of the sampling points. For any point , the color and the body density corresponding to the sampling point can be obtained. ； wherein, a color of the corresponding sampling point; a body density of the corresponding sampling point is a three-dimensional position coordinate, represents a feature of the projection on the corresponding plane, and a feature encoding of the sum of the three features is obtained using a fully connected layer, is one-dimensional, the dimension can be arbitrarily specified; After obtaining the color of all the sampling points and volume density After that, all the points sampled according to the camera extrinsic parameters can be rendered using the volume rendering formula in the neural radiance field: ； denotes the low resolution feature obtained after rendering, is the extrinsic parameter of the camera, denoted as image camera extrinsic parameter; For low resolution features The first three channels are interpreted as RGB channels in the image, while the other channels are fed into the super-resolution module to get the high resolution image with changed view. ； ； and denote a line of sight redirection image of high and low resolution, respectively.

7. The method of claim 1, wherein, In step S3, the model introduces a gaze consistency loss , a gaze feature consistency loss , an identity feature consistency constraint , a pre-trained image feature perception network : To generate an image and the original image Using a constraint of line of sight consistency: ； wherein, represents determining a line of sight direction of the image; represents aligning the redirected image with the line of sight direction of the source image between the source image A consistency loss of the gaze feature is added to assist generation of gaze consistency images: ； representing a line of sight feature of the image, representing aligning the line of sight feature between the redirected image with the source image at a feature level; An identity feature consistency constraint is added: ； wherein represents obtaining identity features of the image; represents aligning identity features between the redirected image and the source image while preserving facial key attributes; A learning perception image block similarity loss is used to constrain the generated images: ； is a pre-trained image feature perception network.

8. The method of claim 1, wherein, The specific steps of the gradient descent method are as follows: first, the following is calculated , ； Then the gradient of the model parameters with respect to the loss function is computed, and then the parameters are updated in the direction of this gradient, with the update formula ​ ； where represents the learning rate; represents the sum of all losses; represents the gradient of the model in the iteration process.

Images