Ultra-low bandwidth video call transmission system and method based on neural radiation field

Abstract

The present invention provides an ultra-low-bandwidth video call transmission system and method based on neural radiation fields, comprising: a three-dimensional digital face reconstruction model that extracts facial features from facial images for ultra-low-bandwidth video call transmission; and can use facial features as input to explicitly control facial expressions, achieving three-dimensional reconstruction; a facial feature clustering analysis module that ensures real-time ultra-low-bandwidth video call transmission using facial features; a facial feature compression module that compresses facial features; and a video call demonstration module that displays the real-time ultra-low-bandwidth video call transmission process. The present invention uses a dynamic neural radiation field to implement a face reconstruction model, achieving high-fidelity, infinite-resolution face reconstruction. Based on the model's one-to-one correspondence between facial expression features and facial images, the original transmission of facial videos is converted to the transmission of facial features. Cluster analysis and feature compression are then used to ultimately achieve ultra-low-bandwidth, near-real-time face video call transmission.

Description

Technical Field

[0001] The present invention relates to a video call transmission system, and in particular to an ultra-low bandwidth video call transmission system and method based on neural radiation fields. Background Art

[0002] Reconstructing three-dimensional digital humans, especially digital faces, is a hot research topic in computer vision and computer graphics. Three-dimensional digital faces are essential for telepresence applications such as augmented reality (AR) and virtual reality (VR), as well as for video editing (e.g., dubbing in the film industry). These applications require a realistic reconstruction of the human face's appearance and the ability to adapt to changes in view angle and head pose (common in VR) and expressions (e.g., dubbing in the film industry). Previous facial reconstruction models used explicit geometric and material features (e.g., albedo and reflectance) to represent the face (or head). However, this approach is challenging because human skin exhibits subsurface scattering effects, the eyes are highly reflective, and the hair has complex geometric features and fine details, making this representation ineffective. While explicit reconstruction of high-quality skin surface geometry can be easily achieved with multi-camera, multi-view studios, hair is typically approximated by simply retrieving and refining the hairstyle, resulting in a highly unrealistic visual effect.

[0003] At the same time, the use and research of portrait video calls and video transmission are also attracting much attention. In recent years, people's lifestyles have increasingly shifted from offline to online, with online learning, online work, online entertainment, and other services emerging one after another. Among these, video calls and transmission play a very important role: With the shift of student education to online, in order to maintain teaching quality, online multi-person video conferencing is necessary to help students learn better, and the same is true for online office reports. In addition, due to the lockdown and isolation caused by the epidemic, only video calls can build a "face-to-face" communication bridge between us and our loved ones. Online entertainment such as live broadcasts and short videos also have a large demand for portrait video calls and transmission. Some live broadcasts with virtual images (where the host presents their face to the audience as a virtual image, and their facial expressions can be roughly captured) have higher requirements for video calls.

[0004] Furthermore, whether it's a video call or video transmission, a key consideration is the bandwidth required for video transmission. Furthermore, some specialized video transmission scenarios, such as those with extremely limited bandwidth, air-to-ground communications, underwater communications, polar expeditions, Earth-Moon communications, and space station communications, as well as large-scale online conferences with large numbers of participants, require sufficiently low bandwidth.

[0005] The primary method for reducing video bandwidth is video compression. Over the years, many different methods have been devised to compress video, spanning a long history. This includes the first inter-frame compression method, DCT image compression, the first video compression protocol H.120, the first commercially successful digital video coding standard H.261, and the H.266 / VCC standard released in 2020. These various methods exploit the temporal, spatial, coding, and visual redundancy inherent in video information, reducing the bandwidth required for transmission by removing these redundancies. These evolving video compression methods have enabled higher-definition video calls and transmissions, but they may struggle in the specific situations described above. Summary of the Invention

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide an ultra-low bandwidth video call transmission system and method based on neural radiation field.

[0007] According to one aspect of the present invention, there is provided an ultra-low bandwidth video call transmission system based on neural radiation field, comprising:

[0008] A three-dimensional digital face reconstruction module, wherein the three-dimensional digital face reconstruction model can extract facial expression features from an input face image; and can implicitly control facial expressions using the facial expression features as input to achieve three-dimensional reconstruction;

[0009] A video call demonstration module, which displays a real-time video call process and includes a server sending end and a client receiving end. The server sending end transmits the facial expression features collected by the three-dimensional digital face reconstruction module to the client receiving end;

[0010] A facial feature compression module, connected to the server sending end, compresses the transmitted facial expression features;

[0011] A facial feature clustering analysis module is connected to the client receiving end, performs cluster analysis on the facial expression features compressed by the facial feature compression module, and uses the three-dimensional digital face reconstruction module to perform sampling pre-rendering and result replacement to achieve real-time video calls.

[0012] Preferably, the three-dimensional digital face reconstruction module has the characteristic of matching the input facial image with the facial expression features one by one, so as to realize the transmission of facial expression features instead of facial images, thereby reducing bandwidth;

[0013] The facial feature compression module compresses the transmitted facial expression features again to reduce the transmission bandwidth;

[0014] The facial feature cluster analysis module realizes real-time performance in the process of rendering and re-presenting the received facial expression features at the client receiving end.

[0015] Preferably, the three-dimensional digital face reconstruction model includes:

[0016] a face tracking unit, wherein the face tracking unit estimates a face pose and a face expression feature in a given face image;

[0017] a background extraction unit, wherein the background extraction unit decouples the face and the background in a given face image to obtain a target face and the background respectively;

[0018] a dynamic neural radiation field unit, wherein the dynamic neural radiation field unit dynamically and implicitly represents a scene with the target face as the main body;

[0019] A face stereo rendering unit, which renders a scene with the target face represented by the dynamic neural radiation field unit as the main body to generate a face image;

[0020] The facial posture and facial expression features extracted by the facial tracking unit serve as part of the input of the dynamic neural radiation field unit;

[0021] The target face and background extracted by the background extraction unit are used by the face stereo rendering unit to calculate the loss in the training phase and to generate the face image in the testing phase.

[0022] Preferably, the dynamic neural radiation field unit uses the face posture P, facial expression feature δ and learnable latent code γ obtained by the face tracking unit as conditional parameters to restrict the face, and uses a multi-layer perceptron MLP to represent the dynamic neural radiation field unit D of the face. Θ ;

[0023] The input of the multi-layer perceptron MLP includes five types of parameters: a position p of a point in a three-dimensional space composed of (x, y, z), a two-dimensional viewing direction composed of (θ, φ), and a three-dimensional viewing direction composed of (θ, φ). The head pose P, facial expression features δ and learnable hidden code γ; its output is the voxel density σ corresponding to the point in three-dimensional space, and the radiation color RGB related to the viewing direction of the point, that is, DΘ (x, y, z, θ, φ; P, δ, γ) = (RGB, σ).

[0024] Preferably, the radiation color RGB and voxel density σ are the dynamic neural radiation field unit D Θ The network results, the acquisition process includes:

[0025] The facial expression feature δ and the facial posture P obtained by the facial tracking unit are processed by the embedding network and the attention network to obtain a processing result;

[0026] Combining the processing result with the hidden code to obtain the encoded feature;

[0027] Input the encoded features into 8 sequentially arranged fully connected layer networks with 256 channels;

[0028] The position p of a point in three-dimensional space is used as the input of the fifth fully connected layer network with 256 channels;

[0029] All fully connected layer networks use ReLu as the activation function, and the last fully connected layer network outputs the voxel density σ and a 256-dimensional feature vector; the 256-dimensional feature vector is combined with the viewing direction Combined, the input is fed into a 128-channel fully connected layer to output the radiance color RGB pixel value.

[0030] Preferably, the embedding network and the attention network perform window embedding and smoothing processing on the facial expression features, respectively; the learnable latent code γ compensation reconciles the errors in the facial expression features and the facial posture in the facial tracking unit.

[0031] Preferably, the face stereo rendering unit simulates the sampling process of camera rays passing through each pixel of each frame of the face image, accumulates the voxel density σ and radiation color RGB obtained by the dynamic neural radiation field unit, and obtains the final image pixel value G, including:

[0032] Given a camera ray The camera center is c and the viewing direction is The nearest boundary and the farthest boundary are z near and z far In the case of , the estimated pixel value C is expressed as:

[0033] Among them RGB Θ (·) and σ Θ (·) is the dynamic neural radiation field model D ΘThe corresponding point on the camera ray is calculated by head pose P, expression feature δ and learnable hidden code γ. T(t) represents the distance from z along the camera ray. near The cumulative transmittance from t to t is expressed as:

[0034] Preferably, the facial feature clustering analysis module includes:

[0035] A facial feature acquisition unit, which collects facial videos of multiple different subjects and obtains and integrates all facial expression features;

[0036] A cluster analysis unit, wherein the cluster analysis unit performs a k-means cluster analysis on the facial expression features obtained by the facial feature acquisition unit, calculates the distance between each type of facial expression feature point and its cluster center, and sums and compares the distances to ultimately determine the optimal number of clusters;

[0037] A sampling and pre-rendering unit, which samples each type of facial feature points after the cluster analysis in proportion, pre-renders the sampled points using a three-dimensional digital face reconstruction model, and stores the obtained rendering results;

[0038] The result replacement unit compares the transmitted target facial features with the facial features obtained by cluster sampling, selects the pre-rendering result of the feature closest to the target facial feature, and approximately replaces the rendering result of the currently transmitted facial feature.

[0039] Preferably, the facial feature compression module includes:

[0040] A residual coding compression unit, wherein the residual coding compression unit performs residual coding compression on facial expression features;

[0041] A DCT coding compression unit, wherein the DCT coding compression unit performs DCT coding compression on facial expression features;

[0042] In the video call demonstration module, the server sending end is constructed through the socket library in Python and is connected to the client receiving end, and is used to send the target facial expression features to the client receiving end; the client receiving end is constructed through the socket library in Python and is connected to the server sending end, and is used to receive the transmitted facial expression features and render the results for display.

[0043] According to a second aspect of the present invention, a method for transmitting ultra-low bandwidth video calls based on neural radiation fields is provided, which uses the above-mentioned ultra-low bandwidth video call transmission system based on neural radiation fields, including:

[0044] Extract facial expression features from images;

[0045] compressing the facial expression features;

[0046] Send the compressed facial expression features from the server sending end to the client receiving end;

[0047] The client receiving end performs cluster analysis on the received facial features and samples the pre-rendered results to replace the real-time performance of the video call.

[0048] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0049] The embodiments of the present invention provide an ultra-low-bandwidth video calling system and method based on neural radiation fields. By implementing a facial reconstruction model within a three-dimensional reconstruction model and using the model to establish a one-to-one correspondence between facial expression features and facial images, the system transforms the original transmission of facial videos into the transmission of facial features. Furthermore, through cluster analysis and feature compression, the system ultimately achieves ultra-low-bandwidth, near-real-time facial video calling.

[0050] The dynamic neural radiation field provided by the embodiment of the present invention represents the three-dimensional scene with the human face as the main body in an implicit way, rather than using an explicit reconstruction method. It can achieve better reconstruction effects on the scene lighting and other details, while also having the advantage of infinite resolution rendering. Based on the original neural radiation field, it uses facial expression features and facial posture as conditional inputs to achieve dynamic control of the reconstructed facial expression and facial orientation, ultimately achieving high-fidelity face reconstruction.

[0051] The facial feature compression module provided by the embodiment of the present invention takes into account the fact that facial expression features in vector form still have redundancy in their representation and uses compression methods such as lossless entropy coding and residual quantization coding to further compress the facial expression features for transmission, thereby further reducing the transmission bandwidth.

[0052] The facial feature cluster analysis module provided by the embodiment of the present invention takes into account for the first time the clustering characteristics of facial expressions distributed over time during a call, and based on this, constructs a video call facial expression feature database. It further meets the needs of real-time video call transmission by replacing the sampling and pre-rendering results. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0054] Figure 1 This is a structural block diagram of an ultra-low bandwidth video call transmission system based on neural radiation field in one embodiment of the present invention;

[0055] Figure 2 for Figure 1 The structural block diagram is further illustrated in the schematic diagram;

[0056] Figure 3 This is an overall framework diagram of a 3D digital face reconstruction module in a preferred embodiment of the present invention;

[0057] Figure 4 A network structure diagram of a dynamic neural radiation field in a preferred embodiment of the present invention;

[0058] Figure 5 The figure is a flowchart of an ultra-low bandwidth video call transmission method based on neural radiation field in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0059] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0060] Reference Figure 1 and Figure 2 As shown, an embodiment of an ultra-low bandwidth video call transmission system based on neural radiation field proposed by the present invention includes:

[0061] A 3D digital face reconstruction module that extracts facial expression features from input facial images and implicitly controls facial expressions using these features as input to achieve 3D reconstruction.

[0062] Video call demonstration module, which shows the real-time video call process, including the server sending end and the client receiving end. The server sending end transmits the facial expression features to the client receiving end.

[0063] The facial feature compression module is connected to the server sending end to compress the transmitted facial expression features;

[0064] The facial feature clustering analysis module is connected to the client receiving end to perform cluster analysis, sample pre-rendering and result replacement on the received facial expression features to achieve real-time video calls.

[0065] In this embodiment, a three-dimensional digital face reconstruction module extracts facial features from facial images for ultra-low-bandwidth video call transmission; is used to implicitly represent faces, realize face reconstruction, generate faces with new postures and expressions, and drive faces, and directly control the generation of facial expressions by inputting facial expression features; a facial feature clustering analysis module realizes the real-time transmission of the facial features for ultra-low-bandwidth video calls; is used to solve the problem of non-real-time video calls caused by the need to render facial features at the receiving end when transmitting facial features instead of facial images for ultra-low-bandwidth video calls; a facial feature compression module is used to further compress the facial features transmitted in ultra-low-bandwidth video call transmission; and a video call demonstration module is used to demonstrate the near real-time ultra-low-bandwidth video call transmission process.

[0066] In a preferred embodiment of the present invention, a preferred structure of a three-dimensional digital face reconstruction module is provided, referring to Figure 3 As shown, the three-dimensional digital face reconstruction module includes a face tracking unit, a background extraction unit, a dynamic neural radiation field unit and a face stereo rendering unit; the face tracking unit estimates the face posture and facial expression features in a given face image; the background extraction unit decouples the face and background in a given face image to obtain the target face and background respectively; the dynamic neural radiation field unit dynamically and implicitly represents the scene with the target face as the main body; the face stereo rendering unit renders the scene with the target face represented by the dynamic neural radiation field unit as the main body to generate a face image.

[0067] It should be noted that the facial posture and facial expression features extracted by the facial tracking unit serve as part of the input of the dynamic neural radiation field unit; the target face and background extracted by the background extraction unit are used for the calculation of the loss by the facial stereo rendering unit in the training phase and the generation of facial images in the testing phase.

[0068] Reference Figure 4 , is a network structure diagram of the dynamic neural radiation field in a preferred embodiment. The dynamic neural radiation field model takes the head posture P, facial expression features δ and learnable latent code γ as conditional parameters to restrict, and uses a multi-layer perceptron (MLP) to represent the dynamic neural radiation field D of the face. Θ Its essence is that an input is the position p of a point in three-dimensional space, composed of (x, y, z), and the two-dimensional viewing direction The function is composed of (θ, φ) and uses the above three parameters as conditional input. After the input passes through the MLP, the voxel density σ corresponding to the point in space and the radiation color RGB related to the viewing direction of the point can be output, which is used in the above-mentioned face stereo rendering model. The whole function can be expressed as: D Θ(x, y, z, σ, φ; P, δ, γ) = (RGB, σ). In the present invention, the multi-layer perceptron can be understood as a multi-layer deep neural network model, which is used to represent the dynamic neural radiation field of the face. For the convenience of description, it is denoted as D Θ .

[0069] Furthermore, the network processing process of the dynamic neural radiation field unit is:

[0070] S1, the facial expression features δ and facial posture P obtained by the facial tracking unit are processed by the embedding network (EmbeddingMLP and Attention Net) to obtain the processing results;

[0071] S2, combines the processing results in S1 with the latent code to obtain the encoded features;

[0072] S3, inputs the features encoded by S2 into 8 sequentially arranged fully connected layer networks with 256 channels;

[0073] S4, takes the position p of a point in three-dimensional space as the input of the fifth fully connected layer network with 256 channels;

[0074] In S5, all fully connected layer networks use ReLu as the activation function. The last fully connected layer network outputs the voxel density σ and a 256-dimensional feature vector. The 256-dimensional feature vector is compared with the viewing direction. Combined, the input is fed into a 128-channel fully connected layer to output the radiance color RGB pixel value.

[0075] In this embodiment, the embedding network (Embedding MLP) and the attention network (Attention Net) are used to perform windowed embedding and smoothing processing on facial features, making the rendered face video smoother and more natural.

[0076] In this embodiment, the learnable latent code γ is used to compensate for and reconcile the errors in facial expression estimation and face pose estimation, thereby increasing their relevance.

[0077] In another preferred embodiment of the present invention, an implementation process of a face stereo rendering unit is provided. The face stereo rendering unit simulates the sampling process of each pixel of each frame of the face image as a camera ray passes through it, accumulates the voxel density and RGB obtained by the dynamic neural radiation field model, and obtains the final image pixel value C, that is, the value of a pixel (of the face image) obtained by stereo rendering along a camera ray at a certain camera perspective.

[0078] In a preferred embodiment, the process of obtaining the pixel value of the final image is specifically expressed as follows:

[0079] Given a camera ray The camera center is c and the viewing direction is The nearest boundary and the farthest boundary are z near and z far In the case of , the estimated pixel value C can be expressed as:

[0080] Among them RGB Θ (·) and σ Θ (·) is the dynamic neural radiation field model D Θ The corresponding point on the camera ray is calculated by head pose P, expression feature δ and learnable hidden code γ. T(t) represents the distance from z along the camera ray. near The cumulative transmittance from t to t is expressed as:

[0081] In this embodiment, the face stereo rendering unit fully considers the information of the face in three-dimensional space based on its characteristics of simulating three-dimensional space rays, so as to obtain more realistic and accurate pixel values ​​of the face image under a certain viewing angle.

[0082] In a preferred embodiment of the present invention, the facial feature cluster analysis module includes a cluster analysis unit, a sampling and pre-rendering unit, and a result replacement unit. The cluster analysis unit is configured to perform k-means cluster analysis on facial features by calculating the distance from a point to a cluster center and determining the optimal number of clusters. The sampling and pre-rendering unit is configured to sample the clustered feature points according to the proportion of each cluster, and to store the rendered results by pre-rendering a three-dimensional digital face reconstruction model. The result replacement unit is configured to compare the transmitted facial features with the features obtained by cluster sampling, select the pre-rendered results of the closest features, and approximately replace the rendered results of the currently transmitted facial features.

[0083] The facial feature clustering analysis module in this embodiment, provided by the embodiment of the present invention, for the first time considers the clustering characteristics of facial expressions distributed over time during a call, and based on this, constructs a video call facial expression feature database. It further replaces this method of exchanging space for time by sampling and pre-rendering results, ultimately meeting the needs of real-time video call transmission.

[0084] In a preferred embodiment of the present invention, the facial feature compression module includes a residual coding compression module and a DCT coding compression module; the residual coding compression module is used to perform further residual coding compression on the facial features; and the DCT coding compression module is used to perform further DCT coding compression on the facial features.

[0085] See also Figure 5Based on the same inventive concept, the present invention provides an ultra-low bandwidth video call transmission method based on neural radiation field, the process of which is as follows:

[0086] S100, extracting facial expression features from the image;

[0087] S200, compressing the facial expression features extracted in S100;

[0088] S300, sending the facial expression features compressed in S200 from the server sending end to the client receiving end;

[0089] S400: Cluster analysis is performed on the received facial features at the client receiving end, and the sampled pre-rendered results are used to replace the real-time performance of the video call.

[0090] In this embodiment, a three-dimensional digital face reconstruction model is first constructed by adding facial posture and expression features as conditional inputs to construct a dynamic neural radiation field; then, based on the one-to-one correspondence between facial expression features and facial images in the face reconstruction model, facial features are transmitted instead of facial videos; at the same time, cluster analysis and sampling pre-rendering results of facial features are performed to achieve real-time video calls, and facial features are further compressed; finally, combined with the above process, a near real-time ultra-low bandwidth face video call transmission process is realized, which can be demonstrated through a video call demonstration module.

[0091] This method provides a new approach to ultra-low-bandwidth video call transmission. Unlike previous methods that rely solely on images or explicitly model geometric and material features, this method uses a scene representation network to implicitly represent the face and generates a facial image from this implicit representation using stereo rendering. To achieve dynamic facial changes, the present invention adds facial expression feature input to explicitly control facial expressions. Furthermore, the optimization of the present model only requires facial video data from a single camera, eliminating the need for specialized multi-camera setups. Inspired by the model's mapping of expression features to facial images, and given the current higher bandwidth requirements for online video calls, the present invention proposes a new approach to ultra-low-bandwidth video transmission by replacing facial video output with expression feature transmission. This approach also achieves real-time video call performance and further feature compression. Ultimately, the present invention demonstrates through experiments that the model can achieve highly realistic facial actuation and that the new approach of using facial features as a replacement for transmission enables near-real-time ultra-low-bandwidth video call transmission.

[0092] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various modifications or variations within the scope of the claims without affecting the essence of the present invention. The above preferred features may be used in any combination as long as they do not conflict with each other.

Claims

1. An ultra-low bandwidth video call transmission system based on neural radiation field, characterized in that: include: A three-dimensional digital face reconstruction module, wherein the three-dimensional digital face reconstruction model extracts facial expression features from an input face image and implicitly controls facial expressions using the facial expression features as input to achieve three-dimensional reconstruction; A video call demonstration module, which displays a real-time video call process and includes a server sending end and a client receiving end. The server sending end transmits the facial expression features collected by the three-dimensional digital face reconstruction module to the client receiving end; A facial feature compression module, connected to the server sending end, compresses the transmitted facial expression features; A facial feature cluster analysis module, connected to the client receiving end, performs cluster analysis on the facial expression features compressed by the facial feature compression module, and uses the three-dimensional digital face reconstruction module to perform sampling pre-rendering and result replacement to achieve real-time video calls; The facial feature clustering analysis module, include: A facial feature acquisition unit, which collects facial videos of multiple different subjects and obtains and integrates all facial expression features; A cluster analysis unit, wherein the cluster analysis unit performs a k-means cluster analysis on the facial expression features obtained by the facial feature acquisition unit, calculates the distance between each type of facial expression feature point and its cluster center, and sums and compares the distances to ultimately determine the optimal number of clusters; A sampling and pre-rendering unit, which samples each type of facial feature points after the cluster analysis in proportion, pre-renders the sampled points using a three-dimensional digital face reconstruction model, and stores the obtained rendering results; A result replacement unit compares the transmitted target facial features with the facial features obtained by cluster sampling, selects the pre-rendering result of the feature closest to the target facial features, and approximately replaces the rendering result of the currently transmitted facial features; The facial feature acquisition unit takes into account the clustering characteristics of facial expressions distributed over time during a call, and based on this, constructs a video call facial expression feature database.

2. The ultra-low bandwidth video call transmission system based on neural radiation field according to claim 1, characterized in that: The three-dimensional digital face reconstruction module has the characteristic of matching the input face image with the facial expression features one by one, so that the facial expression features can replace the face image in transmission, thereby reducing bandwidth. The facial feature compression module compresses the transmitted facial expression features to reduce the transmission bandwidth; The facial feature cluster analysis module realizes real-time performance in the process of rendering and re-presenting the received facial expression features at the client receiving end.

3. The ultra-low bandwidth video call transmission system based on neural radiation field according to claim 1, characterized in that: The three-dimensional digital face reconstruction model includes: a face tracking unit, wherein the face tracking unit estimates a face pose and a face expression feature in a given face image; a background extraction unit, wherein the background extraction unit decouples the face and the background in a given face image to obtain a target face and the background respectively; a dynamic neural radiation field unit, wherein the dynamic neural radiation field unit dynamically and implicitly represents a scene with the target face as the main body; A face stereo rendering unit, which renders a scene with the target face represented by the dynamic neural radiation field unit as the main body to generate a face image; The facial posture and facial expression features extracted by the facial tracking unit serve as part of the input of the dynamic neural radiation field unit; The target face and background extracted by the background extraction unit are used by the face stereo rendering unit to calculate the loss in the training phase and to generate the face image in the testing phase.

4. The ultra-low bandwidth video call transmission system based on neural radiation field according to claim 3, characterized in that: The dynamic neural radiation field unit uses the facial posture P, facial expression feature δ and learnable latent code γ obtained by the facial tracking unit as conditional parameters to restrict the dynamic neural radiation field unit D of the face using a multi-layer perceptron MLP. Θ ; The input of the multi-layer perceptron MLP includes five types of parameters: a position p of a point in a three-dimensional space composed of (x, y, z), a two-dimensional viewing direction composed of (θ, φ), and a three-dimensional viewing direction composed of (θ, φ). The head pose P, facial expression features δ and learnable hidden code γ; its output is the voxel density σ corresponding to the point in three-dimensional space, and the radiation color RGB related to the viewing direction of the point, that is, D Θ (x, y, z, θ, φ; P, δ, γ) = (RGB, σ).

5. The ultra-low bandwidth video call transmission system based on neural radiation field according to claim 4, characterized in that: The radiation color RGB and voxel density σ are the dynamic neural radiation field unit D Θ The network results, the acquisition process includes: The facial expression feature δ and the facial posture P obtained by the facial tracking unit are processed by the embedding network and the attention network to obtain a processing result; Combining the processing result with the hidden code to obtain the encoded feature; Input the encoded features into 8 sequentially arranged fully connected layer networks with 256 channels; The position p of a point in three-dimensional space is used as the input of the fifth fully connected layer network with 256 channels; All fully connected layer networks use ReLu as the activation function, and the last fully connected layer network outputs the voxel density σ and a 256-dimensional feature vector; the 256-dimensional feature vector is combined with the viewing direction Combined, the input is fed into a 128-channel fully connected layer to output the radiance color RGB pixel value.

6. The ultra-low bandwidth video call transmission system based on neural radiation field according to claim 5, characterized in that: The embedding network and the attention network respectively perform window embedding and smoothing processing on the facial expression features; the learnable latent code γ compensation reconciles the errors in the facial expression features and facial posture in the facial tracking unit.

7. The ultra-low bandwidth video call transmission system based on neural radiation field according to claim 5, characterized in that: The face stereo rendering unit simulates the sampling process of camera rays passing through each pixel of each frame of the face image, accumulates the voxel density σ and radiation color RGB obtained by the dynamic neural radiation field unit, and obtains the final image pixel value C, including: Given a camera ray The camera center is c and the viewing direction is The nearest boundary and the farthest boundary are z near and z far In the case of , the estimated pixel value C is expressed as: Among them RGB Θ (·) and σ Θ (·) is the dynamic neural radiation field model D Θ The corresponding point on the camera ray is calculated by head pose P, expression feature δ and learnable hidden code γ. T(t) represents the distance from z along the camera ray. near The cumulative transmittance from t to t is expressed as:

8. The ultra-low bandwidth video call transmission system based on neural radiation field according to claim 1, characterized in that: The facial feature compression module includes: A residual coding compression unit, wherein the residual coding compression unit performs residual coding compression on facial expression features; A DCT coding compression unit, wherein the DCT coding compression unit performs DCT coding compression on facial expression features; In the video call demonstration module, the server sending end is constructed through the socket library in Python and is connected to the client receiving end, and is used to send the target facial expression features to the client receiving end; the client receiving end is constructed through the socket library in Python and is connected to the server sending end, and is used to receive the transmitted facial expression features and render the results for display.

9. A method for transmitting ultra-low bandwidth video calls based on neural radiation fields, using the ultra-low bandwidth video call transmission system based on neural radiation fields according to any one of claims 1 to 8, characterized in that: include: Extract facial expression features from images; compressing the facial expression features; Send the compressed facial expression features from the server sending end to the client receiving end; The client receiving end performs cluster analysis on the received facial features and samples the pre-rendered results to replace the real-time performance of the video call.

Images