A three-dimensional interactive dual-hand rendering method and device based on knowledge distillation

Abstract

The present disclosure provides a three-dimensional interactive two-hand rendering method and device based on knowledge distillation. The method takes a single image as input of a teacher network and performs training; when training a student network, input a training image into the teacher network, obtain coordinates of each light ray down-sampling point from the teacher network processing as input of the student network, take a rendering image output by the teacher network as ideal output, and form a student network training sample; and train the student network by using the student network training sample. In actual rendering, input an image to be rendered into the student network, and the student network predicts a pixel value corresponding to a direction of a target view direction by using a direction vector of the direction, and outputs a rendering image. The present disclosure applies knowledge distillation technology to interactive two-hand rendering, uses high-quality pseudo data generated by a pre-trained teacher network to train a student network based on residual convolution, and realizes efficient and real-time two-hand rendering effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a three-dimensional interactive double-hand rendering method and device based on knowledge distillation. BACKGROUND

[0002] Hand reconstruction is an important research direction in the field of computer vision. As the most important interactive tool, the accuracy and naturalness of hand reconstruction directly affect the user's immersion and operation experience. Visual-based hand reconstruction methods refer to extracting effective information from two-dimensional images to reconstruct the actual hand pose and shape of the user in three-dimensional space. Interactive hand reconstruction further requires simultaneously reconstructing the pose and shape of both hands of the user, solving the problem of structural similarity and mutual occlusion of the two hands. After achieving accurate interactive hand reconstruction, the reconstructed three-dimensional model of the two hands needs to be converted into a realistic two-dimensional image through hand rendering. Through the implementation of different skin textures and detail processing, high-quality, photo-level rendering can provide natural visual feedback in virtual reality, augmented reality, and other environments.

[0003] NeRF (Neural Radiance Field) rendering is a method for generating high-fidelity three-dimensional models in the field of computer vision. By receiving multiple two-dimensional images and their corresponding camera parameters as input, deep learning techniques are used to extract geometric and texture information from the images. These information is converted into the color and density of each point in the three-dimensional scene through MLP (Multilayer Perceptron). Finally, using volume rendering technology, the color and density of the sampling points are synthesized along the target view direction, resulting in a high-fidelity two-dimensional image.

[0004] Document 1 (Huang, Xuan, et al. "3D Visibility-Aware Generalizable Neural Radiance Fields for Interacting Hands." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 3. 2024.) proposes an interactive double-hand NeRF rendering method based on a single image. This method first estimates the parameterized mesh of the interactive double hands, designs two encoder branches to extract geometric and texture features, and introduces a visibility-aware feature fusion module in the feature branch. The visibility of the mesh vertices enhances the texture and geometric features of the visible sampling points under the target view. This method supplements the information of the occluded parts of the interactive hands through visibility feature fusion, reduces the demand for multiple-view views of the NeRF rendering method, and realizes single-view input rendering.

[0005] Document 2 (Mundra, Akshay, et al. "Livehand: Real-time and photorealistic neural hand rendering." Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023.) proposes a method for real-time rendering of single-hand images. This method first uses a parameterized coarse mesh model to capture the basic geometric structure of the hand, and reduces invalid sampling by limiting the sampling points to the surface of the hand model during the NeRF sampling stage. In addition, this method also introduces a super-resolution module, which can perform super-resolution processing on the low-resolution images rendered by NeRF to generate the final full-resolution images. These two improvements together improve the inference efficiency of the network, enabling the method to achieve real-time rendering.

[0006] In virtual reality, augmented reality and other interactive applications, real-time performance of hand reconstruction is crucial. However, traditional NeRF rendering methods generally perform better in static scenes, but face significant challenges in dynamic hand reconstruction. Existing high-fidelity double-hand reconstruction methods generally cannot achieve lightweight real-time reconstruction, limiting their widespread use in practical applications. Document 1 uses the most basic NeRF rendering method to solve the problem of occlusion of visibility-aware features in double-hand reconstruction and the need for multi-view views in NeRF rendering methods, but still cannot achieve real-time rendering. Document 2 only achieves real-time rendering of single-hand NeRF, and cannot achieve interactive double-hand reconstruction, limiting the use of the scene. SUMMARY

[0007] Therefore, the present application provides a three-dimensional interactive double-hand rendering method based on knowledge distillation, which reduces invalid sampling during the training process of the teacher network and uses a student network based on residual convolutional network to learn the input and output of the teacher network, achieving a more lightweight interactive double-hand rendering scheme.

[0008] To solve the above technical problems, the present application is implemented as follows.

[0009] A three-dimensional interactive double-hand rendering method based on knowledge distillation, characterized in that it comprises:

[0010] Step S1: Teacher network construction: constructing a teacher network comprising a parameterized network generation module and a rendering module; the parameterized network generation module provides a reduced range of rendering areas processed by the MANO parameter model to the rendering module; training the teacher network using a data set;

[0011] Step S2: student network construction: the student network includes a preprocessing module, a residual network layer stacked by a plurality of residual blocks, a plurality of up-sampling layers and an activation function layer; the preprocessing module maps the sampling point coordinates in the image to a high-dimensional space to form a direction vector, and after processing by the convolution layer, the up-sampling layer and the activation function layer, the RGB pixel value corresponding to each light ray under the target view angle is obtained, that is, the rendering image;

[0012] During the training of the student network, the input image is extracted from the data set, the input image is processed by using the trained teacher network to obtain the rendering image under the target view angle; the sampling point coordinates under each light ray are obtained from the teacher network as the input of the student network, and the rendering image output by the teacher network is taken as the ideal output to form the student network training sample; the student network is trained by using the student network training sample;

[0013] Step S3: during actual rendering, the image to be rendered is input into the student network, the student network obtains the direction vector of the known target view from the image to be rendered, and the pixel value of the target view is predicted by using the image to be rendered and the direction vector, and the rendering image is output.

[0014] Preferably, the parameterized network generation module generates a high-resolution network HRNet to generate a two-dimensional heat map of the hands and a MANO parameter model; the two-dimensional heat map of the hands and the original input image are extracted by a ResNet-18 network, and the extracted features are input into a fully connected layer, so that the relative spatial displacement of the hands is predicted and provided to the MANO parameter model to correct the relative spatial relationship of the hands.

[0015] Preferably, when the parameterized network generation module is trained, a spatial relationship constraint loss is added to the loss function.

[0016] Preferably, the rendering module adopts a NeRF network. The student network adopts a NeLF network.

[0017] Preferably, the residual block in the student network includes a deep convolution layer, a first batch normalization layer, a GeLU activation layer, a point-wise convolution layer and a second batch normalization layer.

[0018] Preferably, the preprocessing module maps the input sampling point coordinates to a high-dimensional space as follows: the input sampling point coordinates are concatenated to form 256x256 vectors, and then down-sampling is performed to generate direction vectors with a dimension of 64x64; after processing by the residual network layer, two levels of up-sampling operations are accepted, the size is restored to 256x256, and finally the pixel value is predicted by a sigmoid function.

[0019] Preferably, after the student network is trained by using the student network training sample in step S2, a fine-tuning step is further included, specifically:

[0020] The student network is directly input with an image to be rendered in the dataset, the student network generates a direction vector of the image to be rendered, and after processing, a rendered image is output, compared with an ideal output corresponding to the image to be rendered, and parameters in the student network are adjusted according to the comparison result.

[0021] The application also provides a three-dimensional interactive double-hand rendering device based on knowledge distillation, which comprises a teacher network, a student network and a training module.

[0022] The teacher network comprises a parameterized network generation module and a rendering module; the parameterized network generation module provides a rendering area processed by a MANO parameter model to the rendering module; and the rendering module adopts a NeRF network.

[0023] The student network comprises a preprocessing module, a residual network layer stacked by multiple residual blocks, multiple up-sampling layers and an activation function layer; the preprocessing module maps the coordinates of sampling points in a target view of an input image to a high-dimensional space to form a direction vector, and the direction vector and the input image are processed by the residual network layer, the up-sampling layer and the activation function layer to obtain corresponding RGB pixel values in the target view, and a rendered image is output.

[0024] The training module is used for training the teacher network and the student network; when the student network is trained, the input image is processed by the trained teacher network to obtain a rendered image in a target view; all the coordinates of sampling points on a light ray are obtained from the teacher network as the input of the student network, and the rendered image output by the teacher network is taken as an ideal output to form a student network training sample; and the student network is trained by using the student network training sample.

[0025] Advantages:

[0026] (1) The teacher network taking a single image as input and the student network taking a target view direction vector as input are respectively constructed, the student network is trained by using the pseudo data generated by the teacher network, and only the student network is used in actual rendering.

[0027] (2) The student network directly outputs the corresponding RGB pixel values of the light ray on the picture by using multiple residual blocks and up-sampling operations, omits the Alpha synthesis process required by the NeRF rendering, and realizes real-time double-hand rendering.

[0028] (3) In a preferred embodiment, the parameterized mesh model in the teacher network adds a correction based on the spatial relationship of the two hands, and the NeRF network generates the color and opacity of the required sampling points through the MLP under the constraint of such mesh model, reduces the invalid sampling area to speed up the rendering process, thereby effectively reducing the use of computing resources while ensuring the rendering quality. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A schematic diagram of the three-dimensional two-hand rendering scheme based on knowledge distillation of the present application;

[0030] Figure 2 A schematic diagram of the teacher network;

[0031] Figure 3 A schematic diagram of the student network;

[0032] Figure 4 A composition block diagram of the three-dimensional two-hand rendering device based on knowledge distillation of the present application. DETAILED DESCRIPTION

[0033] The present application will be described in detail below with reference to the accompanying drawings and embodiments.

[0034] The present application provides a three-dimensional two-hand rendering scheme based on knowledge distillation, which respectively constructs a teacher network taking a single RGB image as input and a student network taking a target view direction vector as input, trains the student network using the teacher network to generate pseudo data, and only uses the student network in actual rendering, finally realizing fast rendering of a high-fidelity two-dimensional image of a target view direction.

[0035] The method framework proposed by the present application is shown in Figure 1 The parameterized hand mesh generation module and the NeRF rendering module constitute the teacher network. The HandNeLF rendering module is the student network. The method is divided into a training process and an actual rendering process. The training process is divided into a teacher network construction process and a student network construction process.

[0036] Step S1: teacher network construction.

[0037] In this embodiment, the teacher network includes a parameterized network generation module and a NeRF rendering module. In this embodiment, the rendering module in the teacher network adopts a NeRF network, and in practice, a similar network structure can also be adopted. The parameterized network generation module provides a reduced range of rendering areas processed by the MANO parameter model to the rendering module, thereby reducing invalid sampling; the NeRF rendering module renders the input image according to the rendering area. The student network can obtain a lighter interactive two-hand rendering network by learning the input and output of the teacher network.

[0038] Figure 2 A schematic diagram of the parametric network generation module is shown. The parametric network generation module includes a multi-scale image feature extraction part and a hand parameter regression part; preferably, the present invention may further include a spatial relationship enhancement part, used to extract the spatial positional relationship of the hands and provide it to the MANO parameter model to correct the relative spatial relationship of the hands.

[0039] The multi-scale image feature extraction section employs an encoder with an HRNet (High Resolution Network) backbone architecture to extract multi-scale features (including texture and geometric features) from the input image, thereby comprehensively capturing the details and structural information of the image at various scales. The decoder generates two-dimensional heatmaps for both hands, describing the probability distribution of the position of each joint on the image plane. The feature information output from each scale encoder and the output from each scale decoder are fed into the hand parameter regression section based on a graph convolutional neural network for fusion processing, obtaining the shape and rotation parameters required by the MANO parameter model. Simultaneously, the final decoder outputs a two-dimensional heatmap, which is provided to the spatial relationship enhancement section.

[0040] The spatial relationship enhancement component comprises a ResNet-18 network and fully connected layers. The ResNet-18 network takes as input two-dimensional heatmaps of both hands and the original input image. The wrists play a crucial role in representing the spatial relationships of all hand joints. Since keypoints in the heatmap determine the positions of the left and right wrists, these positions are used for feature extraction along with the original image via the ResNet-18 network. Then, bilinear interpolation is employed to sample features at precise wrist coordinates, enabling selective extraction of wrist features to estimate the relative spatial positions of the hands. Subsequently, the extracted features are input to the fully connected layers to predict the relative spatial displacement of the hands. This prediction is then provided to the MANO parametric model to correct the relative spatial relationships of the hands, ultimately resulting in a parametric mesh model of the interacting hands.

[0041] The parametric mesh model is output to the NeRF network. Under the constraints of the mesh model, the NeRF network generates the required color and opacity of the sampling points through MLP to reduce invalid sampling areas and accelerate the rendering process, thereby optimizing the traditional NeRF rendering method.

[0042] This teacher network needs to be trained. Samples provided by the InterHand2.6M dataset can be used. During training, the following loss function can be constructed. Optimize the teacher network:

[0043]

[0044] Among them, the two-dimensional heatmap constraint loss The mean square error loss function is used to measure the difference between the predicted heat map and the real heat map, to ensure the accuracy of the generated two-dimensional heat map; reconstruction constraint loss The smooth L1 loss is used to constrain the mesh points and joint positions of the hand model obtained by reconstruction, to ensure the accuracy of its shape and pose; MANO model parameter constraint loss The L2 loss function is used to constrain the shape and pose parameters of the generated MANO model; and the spatial relationship constraint loss The spatial relative displacement between the two hand models is constrained by calculating the directed distance between them, to prevent mutual penetration. b , λ h , λ r , λ m are the weights of each loss component.

[0045] Spatial relationship constraint loss The specific calculation process is as follows: for each mesh point x on the left hand model, first calculate its shortest distance distance right to the mesh vertex V min of the right hand model, and its distance distance center to the mesh center point of the right hand model, and finally obtain its shortest directed distance SD left to the right hand model.

[0046]

[0047] The shortest directed distance SD right of the right hand mesh point to the left hand model is processed in the same way.

[0048] The final spatial relationship constraint loss is defined as follows:

[0049]

[0050] where ReLU() represents the ReLU function, and threshold is set to 0.01 to constrain those mesh vertices with relative directed distance less than 0.01, so as to encourage the network to generate three-dimensional hand shapes with as little mutual penetration as possible.

[0051] During training, the weights of these loss functions can be set to λ b = 100, λ h = 10, λ r = 10 and λ m = 0.1, respectively, to achieve the best balance and effect.

[0052] During training, the input image is first scaled to a resolution of 256x256. The Adam optimizer in the Pytorch framework can be used, and the batch size is set to 4. The initial learning rate is set to 0.001 and is exponentially decayed with each training round. The mean square error loss function can be used to measure the difference between the predicted RGB pixel value and the true value during training.

[0053] After the teacher network is trained, the NeRF rendering module can generate a rendered image at the target view. The RGB pixel value corresponding to each ray of the final target view will be used as the training pseudo-data of the next student network together with the coordinates of all sampling points on the ray. Among them, the RGB pixel value corresponding to each ray of the target view is the output of the teacher network and the ideal output of the student network; the coordinates of all sampling points on the ray can be obtained from the teacher network and are the input of the student network.

[0054] Step S2: student network construction.

[0055] The student network is implemented using a NeLF network. The application of the NeLF network to the student network for hand image rendering is referred to as a HandNeLF rendering model, as shown in Figure 3 which specifically includes a preprocessing module, a residual network layer formed by stacking multiple residual blocks, multiple upsampling layers, and an activation function layer. In this embodiment, there are 2 upsampling layers.

[0056] The preprocessing module maps the sampling point coordinates of the target view in the input image to a high-dimensional space to form a direction vector of the target view. During training, the sampling point coordinates are obtained from the teacher network, i.e., the sampling points along the ray are randomly sampled using hierarchical sampling to obtain the sampling point coordinates on the ray, and the direction vector under the corresponding ray is obtained by concatenating them, thereby improving the generalization. In actual rendering, the target view is known, and the sampling point coordinates are obtained from the image to be rendered according to the target view.

[0057] Specifically, the preprocessing module maps the scalar coordinates of all sampling points in the target view direction in the teacher network to a high-dimensional space through position encoding, thereby generating an input direction vector with a dimension of 64x64 as the input data of the student network. The mapping to the high-dimensional space is specifically as follows: the input sampling point coordinates are concatenated to form 256x256 vectors, and then down-sampling is performed to generate a direction vector with a dimension of 64x64.

[0058] Then, the direction vector and the input image are jointly input into the residual network layer. The specific structure of each residual block can be selected as: a deep convolutional layer-a batch normalization layer-a GeLU activation-a pointwise convolutional layer-a batch normalization layer; wherein the convolution kernel size of the deep convolutional layer is 3 and the stride is 1; the convolution kernel size and the stride of the pointwise convolutional layer are both 1. Through this structure, the batch normalization layer ensures the standardization of the data, and the GeLU activation function provides stronger nonlinear expression capability.

[0059] After the direction vector and the input image are processed by the residual network layer, they are subjected to the upsampling operation of two-level upsampling layers, and the size is restored to, for example, 256x256, and finally subjected to the processing of an activation function layer to obtain the corresponding RGB pixel value of the light on the picture, that is, the rendered image. The activation function layer can adopt a sigmoid function.

[0060] When training the student network, the 10k groups of student network training samples generated by the teacher network are used to train the student network. During the training process, the Adam optimizer is adopted, and the batch size is set to 4. The initial learning rate is set to 0.0005 and is exponentially attenuated with each training round.

[0061] Preferably, after the preliminary training is completed by using the student network training samples, the student network can be further fine-tuned by using the InterHand2.6M dataset. Specifically, the student network is directly input with the to-be-rendered image in the dataset, and the student network processes the target rendering perspective direction vector of the to-be-rendered image and outputs a rendered image, which is compared with the ideal rendering result of the target rendering perspective, and the parameters in the student network are adjusted according to the comparison result.

[0062] Step S3: actual rendering.

[0063] During the actual rendering, the to-be-rendered image is input into the student network, the student network obtains the direction vector under the known target perspective direction from the to-be-rendered image, predicts the RGB pixel value corresponding to the direction by using the to-be-rendered image and the direction vector, and outputs a rendered image.

[0064] At this point, the process ends.

[0065] In the above process, the parameterized mesh hand reconstruction part of step S1 can also use other types of reconstruction methods as long as the MANO parameter model can be obtained. In the data distillation step of step S2 described above, the student network can use other lightweight network architectures. The student network needs to have the characteristics of high efficiency and simplicity so as to reduce the consumption of computing resources while maintaining the performance and improve the efficiency of the overall process.

[0066] Based on the above method, the application also provides a three-dimensional interactive double-hand rendering device based on knowledge distillation, which comprises a parameterized mesh hand reconstruction unit, a teacher network, a student network and a rendering unit.Figure 4 As shown, it includes a teacher network, a student network and a training module.

[0067] The teacher network includes a parameterized network generation module and a rendering module; the parameterized network generation module provides the rendering area processed by the MANO parameter model to the rendering module; the rendering module adopts a NeRF network;

[0068] The student network includes a preprocessing module, a residual network layer stacked by multiple residual blocks, multiple up-sampling layers and an activation function layer; the preprocessing module maps the sampling point coordinates in the target view of the input image to a high-dimensional space to form a direction vector, and the direction vector and the input image are processed by the residual network layer, the up-sampling layer and the activation function layer to obtain the corresponding RGB pixel value of the light on the picture, i.e. the rendering image;

[0069] The training module is used for training the teacher network and the student network; when training the student network, the trained teacher network is used to process the input image to obtain the rendering image in the target view; all sampling point coordinates under each light in the target view are obtained from the teacher network as the input of the student network, and the rendering image output by the teacher network is taken as the ideal output to form the student network training sample; the student network is trained by using the student network training sample.

[0070] Compared with document 1, the present application realizes more lightweight interactive double-hand rendering. Through the optimized parameterized grid reconstruction and NeRF rendering method, the present application effectively reduces the use of computing resources while ensuring the rendering quality. Through data distillation and the design of a lightweight network, the efficiency and performance of the overall system are further improved. By selecting an efficient student network architecture, the present application not only maintains high model accuracy, but also significantly reduces the computational and storage overheads, thereby having wider application potential and market value. This not only improves the running efficiency of the system, but also makes the interactive experience in practical applications more smooth and real-time.

[0071] Compared with document 2, the present application is aimed at real-time rendering of double hands, while document 2 is only aimed at single hand. By expanding the application range of the parameterized grid, the present application realizes the synchronous rendering of double hands, which is particularly important when dealing with complex interactive scenes. By simultaneously capturing and rendering the movements of both hands, the present application can provide a more rich and realistic interactive experience.

[0072] The above specific embodiments only describe the design principles of the present application, and the shapes and names of the components in the description can be different and are not limited. Therefore, those skilled in the art of the present application can modify or equivalently replace the technical solutions described in the foregoing embodiments; and these modifications and replacements do not deviate from the purpose and technical solutions of the present application, and should all belong to the protection scope of the present application.

Claims

1. A method for three-dimensional interactive dual-hand rendering based on knowledge distillation, characterized in that, The method comprises the following steps: Step S1: teacher network construction: constructing a teacher network comprising a parameterized network generation module and a rendering module; The parameterized network generation module provides a MANO parameter model processed reduced range rendering area to the rendering module; the teacher network is trained using a data set; The parameterized network generation module comprises a multi-scale image feature extraction part, a hand parameter regression part, a spatial relationship enhancement part and a MANO parameter model; the multi-scale image feature extraction part extracts multi-scale features from an input image using an encoder with a high-resolution network (HRNet) as the backbone architecture, and a decoder generates a two-dimensional hand heat map; the feature information output by each scale encoder and the output of each scale decoder are input into the hand parameter regression part based on a graph convolutional neural network for fusion processing to obtain shape parameters and rotation parameters required by the MANO parameter model; The spatial relationship enhancement part extracts the spatial position relationship of the two hands based on the two-dimensional hand heat map and the input image, and provides the MANO parameter model to correct the relative spatial relationship of the two hands, and finally obtains a parameterized mesh model of the interactive hands, which is output to the rendering module; Step S2: student network construction: the student network comprises a preprocessing module, a residual network layer stacked by multiple residual blocks, multiple upsampling layers and an activation function layer; the preprocessing module maps the sampling point coordinates in the input image to a high-dimensional space to form a direction vector, and the direction vector and the input image are processed by the residual network layer, the upsampling layer and the activation function layer to obtain the pixel values corresponding to each light ray under the target view angle, i.e. a rendering image; During the training of the student network, training images are extracted from a data set, and the training images are processed using the trained teacher network to obtain a rendering image under a target view angle; the sampling point coordinates under each light ray are obtained from the teacher network and combined with the original training image as the input of the student network, and the rendering image output by the teacher network is taken as the ideal output to form a student network training sample; the student network is trained using the student network training sample; Step S3: during actual rendering, input the image to be rendered into the student network, the student network obtains the direction vector of the known target view angle from the image to be rendered, and predicts the pixel value of the target view angle using the image to be rendered and the direction vector, and outputs a rendering image.

2. The method of claim 1, wherein, The spatial relationship enhancement part comprises a ResNet-18 network and a fully connected layer; the two-dimensional hand heat map and the input image are extracted together by the ResNet-18 network, the extracted features are input into the fully connected layer, and the relative spatial displacement of the two hands is predicted, which is provided to the MANO parameter model to correct the relative spatial relationship of the two hands.

3. The method of claim 2, wherein, When training the parameterized network generation module, a spatial relationship constraint loss is added to the loss function.

4. The method of claim 1, wherein, The rendering module adopts a NeRF network.

5. The method of claim 1, wherein, The student network adopts a NeLF network.

6. The method of claim 1, wherein, The residual block in the student network comprises a depth convolution layer, a first batch normalization layer, a GeLU activation layer, a point-wise convolution layer and a second batch normalization layer.

7. The method of claim 1, wherein, The pre-processing module maps the input sample point coordinates to a high-dimensional space: concatenating the input sample point coordinates to form 256x256 vectors, and then down-sampling to generate direction vectors with a dimension of 64x64; after processing by the residual network layer, two levels of up-sampling operations are performed to restore the size to 256x256, and finally the RGB pixel values are predicted by the sigmoid function.

8. The method of claim 1, wherein, After the student network is trained by the student network training sample in step S2, a fine-tuning step is further included, specifically: The student network directly inputs the to-be-rendered image in the data set, processes the target rendering perspective direction vector of the to-be-rendered image, and outputs a rendered image, which is compared with the ideal rendering result of the target rendering perspective, and the parameters in the student network are adjusted according to the comparison result.

9. An apparatus for three-dimensional interactive dual-hand rendering based on knowledge distillation, comprising: The device comprises a teacher network, a student network and a training module; The teacher network comprises a parameterized network generation module and a rendering module; the parameterized network generation module provides a rendering area processed by a MANO parameter model to the rendering module; The rendering module adopts a NeRF network; The parameterized network generation module comprises a multi-scale image feature extraction part, a hand parameter regression part, a spatial relationship enhancement part and a MANO parameter model; the multi-scale image feature extraction part extracts multi-scale features from the input image by using an encoder with a high-resolution network (HRNet) as the main architecture, and a decoder generates a two-dimensional heat map of both hands; the feature information output by each scale encoder and the output of each scale decoder are input into the hand parameter regression part based on the graph convolutional neural network for fusion processing to obtain shape parameters and rotation parameters required by the MANO parameter model; The spatial relationship enhancement part is used to extract the spatial position relationship of both hands based on the two-dimensional heat map of both hands and the input image, and provide the MANO parameter model for correcting the relative spatial relationship of both hands, so as to finally obtain a parameterized grid model of the interactive hands and output to the rendering module; The student network comprises a pre-processing module, a residual network layer stacked by a plurality of residual blocks, a plurality of up-sampling layers and an activation function layer; The pre-processing module maps the sample point coordinates of the input image in the target perspective to a high-dimensional space to form a direction vector, and the direction vector and the input image are processed by the residual network layer, the up-sampling layer and the activation function layer to obtain the corresponding RGB pixel value in the target perspective, and a rendered image is output; The training module is used for training the teacher network and the student network; When training the student network, the trained teacher network is used to process the training image to obtain a rendered image in the target perspective; all sample point coordinates on each light line are obtained from the teacher network, combined with the training image as the input of the student network, and the rendered image output by the teacher network is taken as the ideal output to form a student network training sample; and the student network is trained by the student network training sample.

Images