Three-dimensional interaction two-hand rendering method and device based on knowledge distillation

Abstract

The invention provides a three-dimensional interaction two-hand rendering method and device based on knowledge distillation. According to the method, a single image is used as an input teacher network, and training is carried out; when the student network is trained, training images are input into the teacher network, sampling point coordinates under all light rays are obtained from the processing process of the teacher network to serve as input of the student network, rendering images output by the teacher network serve as ideal output, and student network training samples are formed; and training the student network by using the student network training sample. And during actual rendering, inputting a to-be-rendered image into the student network, predicting a pixel value corresponding to the direction by the student network through the direction vector of the target view angle direction, and outputting a rendered image. The knowledge distillation technology is applied to interactive two-hand rendering, the high-quality pseudo data generated by the pre-trained teacher network is used for training the student network based on the residual error convolution, and the efficient and real-time two-hand rendering effect is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a three-dimensional interactive two-handed rendering method and device based on knowledge distillation. Background Art

[0002] Hand reconstruction is an important research direction in the field of computer vision. As the most important interactive tool, the accuracy and naturalness of hand reconstruction directly affect the user's immersion and operating experience. Vision-based hand reconstruction methods refer to reconstructing the user's actual hand posture and shape in three-dimensional space by extracting effective information from two-dimensional images. Interactive hand reconstruction further requires simultaneous reconstruction of the posture and shape of both hands of the user to solve the problems of structural similarity and mutual occlusion of the two hands. After achieving accurate interactive hand reconstruction, it is necessary to convert the reconstructed three-dimensional model of both hands into a realistic two-dimensional image through hand rendering. By implementing different skin textures and detail processing, high-quality, photo-realistic rendering can provide natural visual feedback in virtual reality, augmented reality and other environments.

[0003] NeRF (Neural Radiance Field) rendering is a method for generating high-fidelity 3D models in the field of computer vision. It receives 2D images from multiple perspectives and their corresponding camera parameters as input, uses deep learning technology to extract geometric and texture information from the image, and converts this information into the color and density of each point in the 3D scene through MLP (Multilayer Perceptron). Finally, it uses volume rendering technology to synthesize the color and density of the sampled points along the target perspective direction to obtain a highly realistic 2D image.

[0004] Reference 1 (Huang, Xuan, et al. "3D Visibility-Aware Generalizable Neural Radiance Fields for Interacting Hands." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.) proposed a single-image-based NeRF rendering method for interactive hands. This method first estimates the parameterized mesh of the interacting hands, designs two encoder branches to extract geometric and texture features respectively, introduces a visibility-aware feature fusion module in the feature branch, and uses the visibility of the mesh vertices to enhance the texture and geometric features of the visible sampling points under the target perspective. This method supplements the information of the occluded part of the interacting hands through visibility feature fusion, reduces the NeRF rendering method's need for multi-perspective views, and realizes single-perspective input rendering.

[0005] Reference 2 (Mundra, Akshay, et al. "Livehand: Real-time and photorealistic neural hand rendering." Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023.) proposes a method for real-time rendering of single-hand images. This method first uses a parameterized coarse grid model to capture the basic geometric structure of the hand, and reduces invalid sampling by limiting the sampling points near the surface of the hand model during the NeRF sampling stage. In addition, this method also introduces a super-resolution module, which can perform super-resolution processing on the low-resolution image rendered by NeRF to generate the final full-resolution image. These two improvements jointly improve the reasoning efficiency of the network, enabling this method to achieve real-time rendering effects.

[0006] In virtual reality, augmented reality and other interactive applications, the real-time performance of hand reconstruction is crucial. However, traditional NeRF rendering methods usually perform better in static scenes, but face significant challenges in dynamic hand reconstruction. Existing high-fidelity two-hand reconstruction methods are generally unable to achieve lightweight real-time reconstruction, which limits their widespread use in practical applications. Reference 1 uses the most basic NeRF rendering method to solve the occlusion problem of visibility perception features on two-hand reconstruction and the NeRF rendering method's requirement for multi-view views, but it still cannot achieve real-time rendering effects. Reference 2 only implements NeRF real-time rendering of a single hand, and cannot achieve reconstruction of interactive two hands, and its usage scenarios are limited. Summary of the invention

[0007] In view of this, the present invention provides a 3D interactive two-handed rendering method based on knowledge distillation, which realizes a lighter interactive two-handed rendering scheme by reducing invalid sampling during the training process of the teacher network and using a student network based on a residual convolutional network to learn the input and output of the teacher network.

[0008] In order to solve the above technical problems, the present invention is implemented as follows.

[0009] A three-dimensional interactive two-handed rendering method based on knowledge distillation, characterized by comprising:

[0010] Step S1: teacher network construction: construct a teacher network including a parameterized network generation module and a rendering module; the parameterized network generation module provides the rendering module with a reduced rendering area processed by the MANO parameter model; and the teacher network is trained using the data set;

[0011] Step S2: Student network construction: The student network includes a preprocessing module, a residual network layer formed by stacking multiple residual blocks, multiple upsampling layers and activation function layers; the preprocessing module maps the coordinates of the sampling points in the image to a high-dimensional space to form a direction vector, and after processing by the convolution layer, upsampling layer and activation function layer, the RGB pixel value corresponding to each light ray under the target viewing angle is obtained, that is, the rendered image;

[0012] When training the student network, the input image is extracted from the data set, and the input image is processed by the trained teacher network to obtain the rendered image under the target perspective; the coordinates of the sampling points under each light are obtained from the teacher network as the input of the student network, and the rendered image output by the teacher network is used as the ideal output to form the student network training sample; the student network is trained using the student network training sample;

[0013] Step S3: During actual rendering, the image to be rendered is input into the student network. The student network obtains the direction vector of the known target perspective from the image to be rendered, uses the image to be rendered and the direction vector to predict the pixel value of the target perspective, and outputs the rendered image.

[0014] Preferably, the parameterized network generation module uses a high-resolution network HRNet to generate a two-dimensional heat map of both hands and a MANO parameter model; the two-dimensional heat map of both hands and the original input image are subjected to feature extraction through a ResNet-18 network, and the extracted features are input into a fully connected layer, thereby predicting the relative spatial displacement of the two hands, and providing it to the MANO parameter model to correct the relative spatial relationship of the two hands.

[0015] Preferably, when performing network training on the parameterized network generation module, a spatial relationship constraint loss is added to the loss function.

[0016] Preferably, the rendering module adopts a NeRF network. The student network adopts a NeLF network.

[0017] Preferably, the residual block in the student network includes a depth convolution layer, a first batch normalization layer, a GeLU activation layer, a point-by-point convolution layer, and a second batch normalization layer.

[0018] Preferably, the preprocessing module maps the input sampling point coordinates to a high-dimensional space by concatenating the input sampling point coordinates into a 256×256 vector, and then downsampling to generate a direction vector with a dimension of 64×64; after being processed by the residual network layer, it undergoes a two-level upsampling operation, the size is restored to 256×256, and finally the pixel value is predicted by the sigmoid function.

[0019] Preferably, after step S2 uses the student network training samples to train the student network, it further includes a fine-tuning step, specifically:

[0020] The image to be rendered in the dataset is directly input into the student network. The student network generates the direction vector of the image to be rendered, and outputs the rendered image after processing. The rendered image is compared with the ideal output corresponding to the image to be rendered, and the parameters in the student network are adjusted according to the comparison results.

[0021] The present invention also provides a 3D interactive two-handed rendering device based on knowledge distillation, which includes a teacher network, a student network and a training module;

[0022] The teacher network includes a parameterized network generation module and a rendering module; the parameterized network generation module provides the rendering module with a rendering area processed by the MANO parameter model; the rendering module adopts a NeRF network;

[0023] The student network includes a preprocessing module, a residual network layer formed by stacking multiple residual blocks, multiple upsampling layers and an activation function layer; the preprocessing module maps the sampling point coordinates of the target perspective of the input image to a high-dimensional space to form a direction vector, the direction vector and the input image are processed by the residual network layer, the upsampling layer and the activation function layer to obtain the corresponding RGB pixel value under the target perspective, and output the rendered image;

[0024] The training module is used to train the teacher network and the student network; when training the student network, the trained teacher network is used to process the input image to obtain the rendered image under the target perspective; the coordinates of all sampling points on the light are obtained from the teacher network as the input of the student network, and the rendered image output by the teacher network is used as the ideal output to form the student network training sample; the student network is trained using the student network training sample.

[0025] Beneficial effects:

[0026] (1) The present invention constructs a teacher network with a single image as input and a student network with a target view direction vector as input, respectively, uses the teacher network to generate pseudo data to train the student network, and only uses the student network in actual rendering. The present invention applies knowledge distillation technology to interactive two-handed rendering, and uses the high-quality pseudo data generated by the pre-trained NeRF teacher network to train the student network based on residual convolution, achieving efficient and real-time two-handed rendering effects.

[0027] (2) The student network of the present invention directly outputs the RGB pixel value corresponding to the light on the image through multiple residual blocks and upsampling operations, omitting the Alpha synthesis process required for NeRF rendering, and realizing real-time two-handed rendering.

[0028] (3) In a preferred embodiment, the parameterized mesh model in the teacher network incorporates a correction based on the spatial relationship between the two hands. Under the constraints of such a mesh model, the NeRF network generates the color and opacity of the sampling points required for volume rendering through MLP, reduces the invalid sampling area to accelerate the rendering process, thereby effectively reducing the use of computing resources while ensuring the rendering quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the 3D two-handed rendering solution based on knowledge distillation of the present invention;

[0030] Figure 2 is a schematic diagram of the teacher network;

[0031] Figure 3 A schematic diagram of the student network;

[0032] Figure 4 This is a block diagram of the composition of the 3D two-handed rendering device based on knowledge distillation of the present invention. DETAILED DESCRIPTION

[0033] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0034] The present invention provides a 3D two-handed rendering scheme based on knowledge distillation, which constructs a teacher network with a single RGB image as input and a student network with a target viewing direction vector as input. The teacher network is used to generate pseudo data to train the student network. Only the student network is used in actual rendering, ultimately achieving fast rendering of highly realistic 2D images in the target viewing direction.

[0035] The method framework proposed by the present invention is as follows Figure 1 As shown in the figure, the parameterized hand mesh generation module and the NeRF rendering module constitute the teacher network. The HandNeLF rendering module is the student network. The method is divided into a training process and an actual rendering process. The training process is divided into a teacher network construction process and a student network construction process.

[0036] Step S1: Teacher network construction.

[0037] In this embodiment, the teacher network includes a parameterized network generation module and a NeRF rendering module. In this embodiment, the rendering module in the teacher network adopts the NeRF network, and a similar network structure can also be used in practice. The parameterized network generation module provides the rendering module with a reduced rendering area after processing by the MANO parameter model, thereby reducing invalid sampling; the NeRF rendering module renders the input image according to the rendering area. By learning the input and output of the teacher network, the student network can obtain a more lightweight interactive two-handed rendering network.

[0038] Figure 2 The schematic diagram of the structure of the parameterized network generation module is shown. The parameterized network generation module includes a multi-scale image feature extraction part and a hand parameter regression part; preferably, the present invention may also include a spatial relationship enhancement part for extracting the spatial position relationship of the two hands and providing it to the MANO parameter model to correct the relative spatial relationship of the two hands.

[0039] Among them, the multi-scale image feature extraction part uses an encoder with HRNet (High Resolution Network) as the backbone architecture to extract multi-scale features (including texture features and geometric features) from the input image, thereby fully capturing the details and structural information of the image at all scales. The decoder generates two-dimensional heat maps (HeatMaps) of the two hands, describing the probability distribution of the position of each joint point on the image plane. The feature information output by the encoders at each scale and the output of the decoders at each scale are input into the hand parameter regression part based on the graph convolutional neural network for fusion processing to obtain the shape parameters and rotation parameters required by the MANO parameter model. At the same time, the last level decoder finally outputs a two-dimensional heat map, which is provided to the spatial relationship enhancement part.

[0040] The spatial relationship enhancement part includes the ResNet-18 network and the fully connected layer. The input of the ResNet-18 network is a two-dimensional heat map of both hands and the original input image. The wrist plays a key role in representing the spatial relationship of all hand joints. Since the key points in the heat map determine the positions of the left and right wrists, these positions are extracted together with the original image through the ResNet-18 network, and then bilinear interpolation is used to sample features at the precise wrist coordinates, so that the features of both wrists can be selectively extracted, thereby estimating the relative spatial position of the two hands. Subsequently, the extracted features are input into the fully connected layer to predict the relative spatial displacement of the two hands, which is provided to the MANO parameter model to correct the relative spatial relationship of the two hands, and finally a parameterized mesh model of the interacting two hands is obtained.

[0041] The parameterized mesh model is output to the NeRF network. Under the constraints of the mesh model, the NeRF network generates the color and opacity of the sampling points required for volume rendering through MLP, reduces the invalid sampling area to speed up the rendering process, and thus optimizes the traditional NeRF rendering method.

[0042] The teacher network needs to be trained. The samples provided by the InterHand2.6M dataset can be used. During training, the following loss function can be constructed Optimize the teacher network:

[0043]

[0044] Among them, the two-dimensional heat map constraint loss The mean square error loss function is used to measure the difference between the predicted heat map and the true heat map to ensure the accuracy of the generated two-dimensional heat map; reconstruction constraint loss The grid points and joint positions of the reconstructed hand model are constrained using a smooth L1 loss to ensure the accuracy of its shape and posture; MANO model parameter constraint loss The L2 loss function is used to constrain the shape and posture parameters of the generated MANO model; while the spatial relationship constraint loss By calculating the directed distance between the two hand models, the spatial relative displacement between the two hand models is constrained to prevent mutual penetration. b , h , r , m are the weights of each loss component.

[0045] Spatial relationship constraint loss The specific calculation process is as follows: For each grid point x on the left-hand model, first calculate its relative right-hand model grid vertex V right The shortest distance min and the center point of the right-hand model grid distance center Finally, the shortest directed distance SD relative to the right-hand model is obtained left ;

[0046]

[0047] The shortest directed distance SD of the right-hand grid point relative to the left-hand model right The processing is consistent.

[0048] The final spatial relationship constraint loss Defined as follows:

[0049]

[0050] Among them, ReLU() represents the ReLU function, and the threshold is set to 0.01 to constrain those mesh vertices whose relative directed distance is less than 0.01, thereby prompting the network to generate a three-dimensional hand shape that minimizes mutual penetration.

[0051] During training, the weights of these loss functions can be set to λ b =100,λ h =10,λ r =10 and λ m =0.1 to achieve the best balance and effect.

[0052] During training, the input image is first scaled to a resolution of 256×256. The Adam optimizer in the Pytorch framework can be used, and the batch size is set to 4. The initial learning rate is set to 0.001 and decays exponentially with each training round. During the training process, the mean square error loss function can be used to measure the difference between the predicted RGB pixel value and the true value.

[0053] After the teacher network is trained, the NeRF rendering module can generate the rendered image at the target perspective. The RGB pixel values ​​corresponding to each ray of the final target perspective will be used together with the coordinates of all sampling points on the ray as pseudo data for the training of the student network in the next step. Among them, the RGB pixel values ​​corresponding to each ray of the target perspective are the output of the teacher network and the ideal output of the student network; the coordinates of all sampling points on the ray can be obtained from the teacher network and are the input of the student network.

[0054] Step S2: Student network construction.

[0055] The student network is implemented using the NeLF network. The student network in which the NeLF network is applied to hand image rendering is called the HandNeLF rendering model. Figure 3 As shown, it specifically includes a preprocessing module, a residual network layer formed by stacking multiple residual blocks, multiple upsampling layers and an activation function layer. In this embodiment, there are 2 upsampling layers.

[0056] The preprocessing module maps the sampling point coordinates of the target view in the input image to a high-dimensional space to form the direction vector of the target view. During the training process, the sampling point coordinates are obtained from the teacher network, that is, the sampling points are randomly sampled along the ray using stratified sampling to obtain the sampling point coordinates on the ray, and the direction vector under the corresponding light is obtained by connecting them in series, thereby improving generalization. In actual rendering, the target view is known, and the sampling point coordinates are obtained from the image to be rendered according to the target view.

[0057] Specifically, the preprocessing module maps the scalar coordinates of all sampling points in the target view direction in the teacher network to a high-dimensional space through position encoding, thereby generating an input direction vector of dimension 64×64 as the input data of the student network. The mapping to the high-dimensional space is as follows: the input sampling point coordinates are concatenated to form a 256×256 vector, and then downsampled to generate a direction vector of dimension 64×64.

[0058] Next, the direction vector and the input image enter the residual network layer. The specific structure of each residual block can be selected as: deep convolution layer-batch normalization layer-GeLU activation-point-by-point convolution layer-batch normalization layer; the convolution kernel size of the deep convolution layer is 3 and the stride is 1; the convolution kernel size and stride of the point-by-point convolution layer are both 1. Through this structure, the batch normalization layer ensures the standardized processing of the data, while the GeLU activation function provides stronger nonlinear expression capabilities.

[0059] After the direction vector and the input image are processed by the residual network layer, they are subjected to upsampling operations by two upsampling layers, and the size is restored to, for example, 256×256. Finally, they are processed by the activation function layer to obtain the RGB pixel value corresponding to the light on the image, i.e., the rendered image. The activation function layer can use a sigmoid function.

[0060] When training the student network, the 10k sets of student network training samples generated by the teacher network are used to train the student network. The Adam optimizer is used during training, and the batch size is set to 4. The initial learning rate is set to 0.0005 and decays exponentially with each training round.

[0061] Preferably, after completing the preliminary training using the student network training samples, the student network can be further fine-tuned using the InterHand2.6M dataset. Specifically, the image to be rendered in the dataset is directly input into the student network, and the student network processes the target rendering perspective direction vector of the image to be rendered and outputs the rendered image, which is compared with the ideal rendering result of the target rendering perspective, and the parameters in the student network are adjusted according to the comparison result.

[0062] Step S3: actual rendering.

[0063] During actual rendering, the image to be rendered is input into the student network. The student network obtains the direction vector under the known target viewing direction from the image to be rendered, uses the image to be rendered and the direction vector to predict the RGB pixel value corresponding to the direction, and outputs the rendered image.

[0064] This is the end of the process.

[0065] In the above process, the parameterized mesh hand reconstruction part of step S1 can also use other types of reconstruction methods as long as the MANO parameter model can be obtained. In the data distillation step of step S2, the student network can use other lightweight network architectures. The student network needs to be efficient and concise so as to reduce the consumption of computing resources while maintaining performance and improve the efficiency of the overall process.

[0066] Based on the above method, the present invention also provides a 3D interactive two-handed rendering device based on knowledge distillation, such as Figure 4 As shown, it includes a teacher network, a student network and a training module.

[0067] The teacher network includes a parameterized network generation module and a rendering module; the parameterized network generation module provides the rendering module with the rendering area processed by the MANO parameter model; the rendering module adopts the NeRF network;

[0068] The student network includes a preprocessing module, a residual network layer composed of multiple residual blocks, multiple upsampling layers and activation function layers; the preprocessing module maps the sampling point coordinates of the target view of the input image to a high-dimensional space to form a direction vector. The direction vector and the input image are processed by the residual network layer, the upsampling layer and the activation function layer to obtain the RGB pixel value corresponding to the light on the image, that is, the rendered image;

[0069] The training module is used to train the teacher network and the student network; when training the student network, the trained teacher network is used to process the input image to obtain a rendered image at the target perspective; the coordinates of all sampling points under each light at the target perspective are obtained from the teacher network as the input of the student network, and the rendered image output by the teacher network is used as the ideal output to form a student network training sample; the student network is trained using the student network training sample.

[0070] Compared with document 1, the present invention realizes a more lightweight interactive two-handed rendering. The present invention effectively reduces the use of computing resources while ensuring the rendering quality through optimized parameterized grid reconstruction and NeRF rendering methods. The efficiency and performance of the overall system are further improved through data distillation and lightweight network design. By selecting an efficient student network architecture, this application not only maintains a high model accuracy, but also significantly reduces computing and storage overhead, thereby having a wider range of application potential and market value. This not only improves the operating efficiency of the system, but also makes the interactive experience in practical applications smoother and more real-time.

[0071] Compared with document 2, the present invention performs real-time rendering for both hands, while document 2 only performs real-time rendering for one hand. The present invention achieves synchronous rendering of both hands by expanding the application scope of parameterized grids, which is particularly important when dealing with complex interactive scenes. By simultaneously capturing and rendering the movements of both hands, the present invention can provide a richer and more realistic interactive experience.

[0072] The above specific embodiments only describe the design principle of the present invention. The shapes and names of the components in the description may be different and are not limited. Therefore, those skilled in the art in the field of the present invention may modify or replace the technical solutions recorded in the above embodiments; and these modifications and replacements do not deviate from the creative purpose and technical solutions of the present invention and should all fall within the protection scope of the present invention.

Claims

1. A 3D interactive two-handed rendering method based on knowledge distillation, characterized in that: include: Step S1: Teacher network construction: construct a teacher network including a parameterized network generation module and a rendering module; The parameterized network generation module provides the rendering module with the reduced rendering area processed by the MANO parameter model; the teacher network is trained using the data set; Step S2: Student network construction: The student network includes a preprocessing module, a residual network layer formed by stacking multiple residual blocks, multiple upsampling layers and activation function layers; the preprocessing module maps the sampling point coordinates in the input image to a high-dimensional space to form a direction vector, and the direction vector and the input image are processed by the residual network layer, the upsampling layer and the activation function layer to obtain the pixel values ​​corresponding to each light ray under the target viewing angle, that is, the rendered image; When training the student network, training images are extracted from the data set, and the trained teacher network is used to process the training images to obtain the rendered images under the target perspective; the coordinates of the sampling points under each light are obtained from the teacher network and combined with the original training images as the input of the student network, and the rendered images output by the teacher network are used as the ideal output to form the student network training samples; the student network is trained using the student network training samples; Step S3: During actual rendering, the image to be rendered is input into the student network. The student network obtains the direction vector of the known target perspective from the image to be rendered, uses the image to be rendered and the direction vector to predict the pixel value of the target perspective, and outputs the rendered image.

2. The method according to claim 1, characterized in that The parameterized network generation module uses a high-resolution network HRNet to generate a two-dimensional heat map of both hands and a MANO parameter model; the two-dimensional heat map of both hands and the original input image are extracted through a ResNet-18 network, and the extracted features are input into a fully connected layer to predict the relative spatial displacement of the two hands, which is provided to the MANO parameter model to correct the relative spatial relationship of the two hands.

3. The method according to claim 2, characterized in that When training the parameterized network generation module, the spatial relationship constraint loss is added to the loss function.

4. The method according to claim 1, characterized in that The rendering module adopts NeRF network.

5. The method according to claim 1, characterized in that The student network adopts the NeLF network.

6. The method according to claim 1, characterized in that The residual block in the student network includes a depth convolution layer, a first batch of normalization layers, a GeLU activation layer, a point-by-point convolution layer, and a second batch of normalization layers.

7. The method according to claim 1, characterized in that The preprocessing module maps the input sampling point coordinates to a high-dimensional space by concatenating the input sampling point coordinates into a 256×256 vector, and then downsampling to generate a direction vector with a dimension of 64×64; after being processed by the residual network layer, it undergoes a two-level upsampling operation and restores the size to 256×256, and finally predicts the RGB pixel value through the sigmoid function.

8. The method according to claim 1, characterized in that, After the student network is trained using the student network training samples, step S2 further includes a fine-tuning step, specifically: The image to be rendered in the dataset is directly input into the student network. The student network processes the target rendering perspective direction vector of the image to be rendered and then outputs the rendered image, which is compared with the ideal rendering result of the target rendering perspective. The parameters in the student network are adjusted according to the comparison results.

9. A 3D interactive two-handed rendering device based on knowledge distillation, characterized in that: The device includes a teacher network, a student network and a training module; The teacher network includes a parameterized network generation module and a rendering module; the parameterized network generation module provides the rendering module with a rendering area processed by the MANO parameter model; The rendering module uses NeRF network; The student network includes a preprocessing module, a residual network layer formed by stacking multiple residual blocks, multiple upsampling layers and an activation function layer; the preprocessing module maps the sampling point coordinates of the target perspective of the input image to a high-dimensional space to form a direction vector, the direction vector and the input image are processed by the residual network layer, the upsampling layer and the activation function layer to obtain the corresponding RGB pixel value under the target perspective, and output the rendered image; The training module is used to train the teacher network and the student network; When training the student network, the trained teacher network is used to process the training image to obtain the rendered image under the target perspective; the coordinates of all sampling points on each light ray are obtained from the teacher network, combined with the training image as the input of the student network, and the rendered image output by the teacher network is used as the ideal output to form the student network training sample; the student network is trained using the student network training sample.

Images