Digital human processing method and device based on neural network, equipment and medium
Through the neural network processing semantic analysis and feature transformation of 3D data, efficient coding and low bandwidth transmission of 3D digital people are realized, solving the problem of high transmission bandwidth of 3D digital people in the prior art and improving transmission efficiency.
Patent Information
- Application Number
- CN202411072640.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-22
AI Technical Summary
The existing technology is difficult to realize efficient coding and low bandwidth transmission of 3D digital humans, resulting in great challenges in real-time application scenarios.
Using a neural network-based method, the input 3D data is semantic analysis, a three-dimensional feature grid is generated, and a compact characterization is obtained through feature transformation and compression coding technology, and then the neural network encoder is compressed and coded to generate encoding results. To realize efficient encoding and transmission of 3D digital humans.
While ensuring the quality of the rendered view, the amount of data required for 3D digital people is reduced and the efficient transmission efficiency of 3D digital people is improved.
Smart Images

Figure CN119011860A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of video transmission, and in particular to a method, device, equipment and medium for processing digital humans based on a neural network. Background Art
[0002] Digital humans refer to virtual characters with human appearance or behavior created using computer technology and artificial intelligence technology. These virtual characters have a wide range of application scenarios, such as corporate publicity, education and training, virtual actors and hosts, intelligent customer service, virtual exhibitions and travel guides, IP image customization, virtual meetings, etc. Recently, 3D digital humans have become a research hotspot. They have the advantages of high realism, strong flexibility, and high cost-effectiveness, and are expected to be widely used in entertainment, education, business, science and other fields.
[0003] Compared with 2D data, the amount of 3D data is more massive, and the bandwidth required to transmit or store 3D digital human data has increased dramatically, bringing great challenges to real-time application scenarios. It is crucial to achieve efficient encoding and low-bandwidth transmission of digital humans. How to achieve efficient encoding and transmission of 3D digital humans is a technical problem that needs to be solved urgently. Summary of the invention
[0004] The embodiments of the present disclosure at least provide a method, device, equipment and medium for processing digital humans based on a neural network.
[0005] In a first aspect, an embodiment of the present disclosure provides a digital human processing method based on a neural network, characterized in that it is applied to an encoding end and includes:
[0006] Performing semantic analysis on the input 3D data based on the neural radiation field to obtain a 3D digital human and background; wherein the semantic analysis includes: 2D and / or 3D object detection, or 2D and / or 3D semantic segmentation;
[0007] The 3D digital human is represented and generated to obtain a three-dimensional feature grid of the neural radiation field of the 3D digital human; the three-dimensional feature grid is feature transformed to obtain a compact representation of the 3D digital human; the compact representation at least includes: the three-dimensional feature grid of the neural radiation field or the decomposed features of the three-dimensional feature grid;
[0008] The compact representation is compressed and encoded by an encoder based on a neural network to obtain a first encoding result.
[0009] In an optional embodiment, the neural network-based encoder includes: a characterization sub-encoder and a two-way encoding module; wherein, the first encoding module in the two-way encoding module at least includes a super-prior encoder, a first quantization, and a first entropy encoder, and the second encoding module in the two-way encoding module at least includes a second quantization and a second entropy encoder, and the output results of the two-way encoding modules are the first encoding results.
[0010] In an optional implementation, compressing and encoding the compact representation using a neural network-based encoder to obtain a first encoding result includes:
[0011] Processing the compact representation by the representation sub-encoder to obtain hidden layer features;
[0012] Determine a priori probability distribution of the hidden layer features based on the super a priori encoder, and process the prior probability distribution through the first quantization and the first entropy encoder;
[0013] Based on the second encoding module, the hidden layer features are second quantized and entropy encoded, and the first encoding result is determined based on the output results of the first encoding module and the second encoding module.
[0014] In an optional implementation, performing feature transformation on the three-dimensional feature grid to obtain a compact representation of the 3D digital human includes:
[0015] Decomposing the three-dimensional feature grid to obtain decomposed features of the three-dimensional feature grid; wherein the decomposed features include: tensor decomposition results and / or plane representations;
[0016] The compacted representation is determined based on the three-dimensional feature grid or the decomposed features.
[0017] In an optional implementation manner, the method further includes:
[0018] Get network conditions;
[0019] The coding parameters and / or coding mode are adjusted according to the network conditions.
[0020] In an optional implementation manner, the method further includes:
[0021] Obtain the requirements of downstream tasks provided by the terminal side;
[0022] Based on the requirement, matching encoding parameters are determined and / or a bit rate control strategy is adjusted.
[0023] In an optional implementation, after the compact representation is compressed and encoded by an encoder based on a neural network to obtain a first encoding result, the method further includes:
[0024] The first encoding result is transmitted to a decoding end; wherein the first encoding result is used by the decoding end to render a reconstruction rendering result of the 3D digital human; wherein the reconstruction rendering result includes a rendering viewpoint and / or a 3D model.
[0025] In an optional implementation manner, the method further includes:
[0026] According to the quality feedback from the terminal side, it is determined whether to start a quality enhancement module, wherein the quality enhancement module is used to enhance the rendering quality of the rendering viewpoint or the 3D model.
[0027] In an optional implementation manner, the method further includes:
[0028] Acquiring external parameters, and acquiring network model parameters obtained after characterizing and generating the 3D digital human; wherein the external parameters at least include: camera parameters and driving parameters of the 3D model of the 3D digital human;
[0029] The external parameters and the network model parameters are compressed and encoded by a Metadata encoder to obtain a second encoding result, and the second encoding result is transmitted to a decoding end.
[0030] In an optional implementation manner, the method further includes:
[0031] compressing and encoding the background by a video encoder to obtain a third encoding result;
[0032] The third encoding result is transmitted to a decoding end.
[0033] In an optional implementation, the neural radiation field and the codec are trained in the following manner:
[0034] The initial neural radiation field is trained by training samples to obtain the intermediate neural radiation field;
[0035] Determining a reconstruction loss function value based on the sample label of the training sample and the prediction result of the intermediate nerve radiation field on the training sample;
[0036] The initial state of the codec is set based on the reconstruction loss function value, and the codec and the intermediate neural radiation field are trained from the initial state to obtain the trained neural radiation field and the trained codec; wherein the codec includes: an encoder based on a neural network in the encoding end to be trained and a decoder based on a neural network in the decoding end to be trained.
[0037] In an optional implementation, the training of the codec and the intermediate neural radiation field from the initial state to obtain the trained neural radiation field and the trained codec includes:
[0038] Determine a target loss function; wherein the target loss function is determined based on a reconstruction loss function value and an entropy loss function, the reconstruction loss function is an L2 norm between a rendering result output by the decoder to be trained and a ground truth value, and the entropy loss function is calculated based on an input of a second entropy encoder and an input of a first entropy encoder in the encoder to be trained;
[0039] Starting from the initial state, the codec and the intermediate neural radiation field are repeatedly iteratively trained using the target loss function to obtain a trained neural radiation field and a trained codec.
[0040] In a second aspect, the embodiment of the present disclosure provides a digital human processing method based on a neural network, which is applied to a decoding end, and includes:
[0041] Obtain a first encoding result and a second encoding result sent by an encoding end; wherein the first encoding result is obtained by the encoding end performing compression encoding on a compact representation of the 3D digital human, the compact representation is obtained by performing feature transformation on a three-dimensional feature grid of the neural radiation field of the 3D digital human, the three-dimensional feature grid is generated by representing the 3D digital human, the compact representation at least includes: a three-dimensional feature grid of the neural radiation field or decomposed features of the three-dimensional feature grid, the second encoding result is obtained by the encoding end performing compression encoding on external parameters and network model parameters obtained after performing feature transformation on the three-dimensional feature grid, the external parameters at least include: camera parameters and driving parameters of a 3D model of the 3D digital human;
[0042] Decoding the first encoding result through a decoder based on a neural network, and decoding the second encoding result to obtain a first decoding result and a second decoding result respectively;
[0043] Reconstruct and render according to the first decoding result and the second decoding result to obtain the rendering viewpoint and / or 3D model of the 3D digital human, and display the rendering viewpoint and / or 3D model through the terminal side.
[0044] In an optional implementation manner, the decoding the first encoding result by a decoder based on a neural network to obtain a first decoding result includes:
[0045] Decoding the output result of the first encoding module in the first encoding result by a first entropy decoder to obtain a first entropy decoding result, and decoding the first entropy decoding result by a super a priori decoder to obtain a super a priori decoding result;
[0046] Based on the super a priori decoding result, construct an entropy model of a second entropy decoder, and decode the output result of the second encoding module in the first encoding result by the second entropy decoder based on the entropy model to obtain a second entropy decoding result;
[0047] The second entropy decoding result is decoded by a representation sub-decoder to obtain the first decoding result.
[0048] In an optional implementation manner, the method further includes:
[0049] After obtaining the third encoding result transmitted by the encoding end, the third encoding result is decoded by a video decoder to obtain a third decoding result; wherein the third encoding result is obtained by encoding the background obtained by the encoding end after semantic analysis of 3D data based on the neural radiation field.
[0050] In an optional implementation manner, reconstructing and rendering according to the first decoding result and the second decoding result to obtain the rendering viewpoint and / or 3D model of the 3D digital human includes:
[0051] Reconstruction and rendering are performed according to the first decoding result, the second decoding result and the third decoding result to obtain the rendering viewpoint and / or the 3D model of the 3D digital human.
[0052] In an optional implementation manner, the decoding the second encoding result to obtain a second decoding result includes:
[0053] The second encoding result is decoded by a Metadata decoder to obtain a second decoding result.
[0054] In a third aspect, the embodiment of the present disclosure provides a digital human processing device based on a neural network, which is arranged at an encoding end and includes:
[0055] A semantic analysis module, used for performing semantic analysis on the input 3D data based on the neural radiation field to obtain a 3D digital human and background; wherein the semantic analysis may adopt 2D and / or 3D object detection, or 2D and / or 3D semantic segmentation;
[0056] A representation generation module, used to generate a representation of the 3D digital human to obtain a three-dimensional feature grid of the neural radiation field of the 3D digital human;
[0057] A feature transformation module, used for performing feature transformation on the three-dimensional feature grid to obtain a compact representation of the 3D digital human; wherein the compact representation at least includes: the three-dimensional feature grid of the neural radiation field or the decomposed features of the three-dimensional feature grid;
[0058] The compression coding module is used to compress and encode the compact representation through an encoder based on a neural network to obtain a first coding result.
[0059] In a fourth aspect, an embodiment of the present disclosure provides a digital human processing device based on a neural network, which is arranged at a decoding end and includes:
[0060] An acquisition module is used to acquire a first encoding result and a second encoding result sent by an encoding end; wherein, the first encoding result is obtained by the encoding end performing compression encoding on a compact representation of the 3D digital human, the compact representation is obtained by performing feature transformation on a three-dimensional feature grid of the neural radiation field of the 3D digital human, the three-dimensional feature grid is generated by representing the 3D digital human, the compact representation at least includes: a three-dimensional feature grid of the neural radiation field or decomposed features of the three-dimensional feature grid, the second encoding result is obtained by the encoding end performing compression encoding on external parameters and network model parameters obtained after performing feature transformation on the three-dimensional feature grid, the external parameters at least include: camera parameters and driving parameters of a 3D model of the 3D digital human;
[0061] A decoding module, used to decode the first encoding result through a decoder based on a neural network, and to decode the second encoding result, to obtain a first decoding result and a second decoding result respectively;
[0062] A reconstruction and rendering module is used to perform reconstruction and rendering according to the first decoding result and the second decoding result to obtain the rendering viewpoint and / or 3D model of the 3D digital human, and display the rendering viewpoint and / or 3D model through the terminal side.
[0063] In a fifth aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps in any possible implementation of the first aspect or the second aspect are performed.
[0064] In a sixth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any possible implementation of the first aspect or the second aspect are executed.
[0065] In the seventh aspect, the embodiments of the present disclosure further provide a computer program product, characterized in that the computer program product is stored in a storage medium, and the program product is executed by at least one processor to perform the steps in any possible implementation of the above-mentioned first aspect or second aspect.
[0066] Based on the above research, the embodiments of the present disclosure provide a method, device, equipment and medium for processing digital humans based on neural networks. In the embodiments of the present disclosure, first, semantic analysis is performed on the input 3D data based on the neural radiation field to obtain a 3D digital human and a background; wherein the semantic analysis includes: 2D and / or 3D object detection, or 2D and / or 3D semantic segmentation; then, a representation is generated for the 3D digital human to obtain a three-dimensional feature grid of the neural radiation field of the 3D digital human; the three-dimensional feature grid is feature transformed to obtain a compact representation of the 3D digital human; wherein the compact representation includes at least: a three-dimensional feature grid of the neural radiation field or a decomposed feature of the three-dimensional feature grid; finally, the compact representation is compressed and encoded by an encoder based on a neural network to obtain a first encoding result.
[0067] In the above implementation, by performing representation generation and feature transformation on the 3D digital human obtained after semantic analysis of the 3D data based on the neural radiation field, and performing compression encoding processing on the transformed compact representation, efficient encoding of the 3D digital human can be achieved while ensuring the quality of the rendering view, reducing the amount of data required to transmit the 3D digital human, thereby improving the efficient transmission of the 3D digital human.
[0068] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0069] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can also be obtained based on these drawings without creative work.
[0070] Figure 1 A flowchart of a digital human processing method based on a neural network provided by an embodiment of the present disclosure is shown;
[0071] Figure 2A schematic diagram of an encoding process of an encoder based on a neural network provided in an embodiment of the present disclosure is shown;
[0072] Figure 3 A schematic diagram of the structure of a characterization sub-encoder provided in an embodiment of the present disclosure is shown;
[0073] Figure 4 A schematic diagram of the structure of a super-a priori encoder provided by an embodiment of the present disclosure is shown;
[0074] Figure 5 A flowchart showing another method for processing a digital human based on a neural network provided by an embodiment of the present disclosure is shown;
[0075] Figure 6 A schematic diagram of a decoding process of a neural network-based decoder provided in an embodiment of the present disclosure is shown;
[0076] Figure 7 A schematic diagram of the structure of a super-a priori decoder provided by an embodiment of the present disclosure is shown;
[0077] Figure 8 A schematic diagram of the structure of a characterization sub-decoder provided in an embodiment of the present disclosure is shown;
[0078] Fig. 9 A schematic diagram of a digital human processing system based on a neural network provided by an embodiment of the present disclosure is shown;
[0079] Fig.10 A schematic diagram of a digital human processing device based on a neural network provided by an embodiment of the present disclosure is shown;
[0080] Fig.11 A schematic diagram showing another digital human processing device based on a neural network provided by an embodiment of the present disclosure;
[0081] Fig.12 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0082] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.
[0083] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0084] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.
[0085] According to research, digital humans refer to virtual characters with human appearance or behavior created using computer technology and artificial intelligence technology. These virtual characters have a wide range of application scenarios, such as corporate publicity, education and training, virtual actors and hosts, intelligent customer service, virtual exhibitions and travel guides, IP image customization, virtual meetings, etc. Recently, 3D digital humans have become a research hotspot. They have the advantages of high realism, strong flexibility, and high cost-effectiveness, and are expected to be widely used in entertainment, education, business, science and other fields.
[0086] Compared with 2D data, the amount of 3D data is more massive, and the bandwidth required to transmit or store 3D digital human data has increased dramatically, bringing great challenges to real-time application scenarios. It is crucial to achieve efficient encoding and low-bandwidth transmission of digital humans. How to achieve efficient encoding and transmission of 3D digital humans is a technical problem that needs to be solved urgently.
[0087] Based on the above research, the present disclosure provides a method, device, equipment and medium for processing digital human based on neural network. In the embodiment of the present disclosure, first, semantic analysis is performed on the input 3D data based on neural radiation field to obtain 3D digital human and background; wherein the semantic analysis includes: 2D and / or 3D object detection, or 2D and / or 3D semantic segmentation; then, the 3D digital human is represented and generated to obtain a three-dimensional feature grid of the neural radiation field of the 3D digital human; the three-dimensional feature grid is feature transformed to obtain a compact representation of the 3D digital human; wherein the compact representation includes at least: a three-dimensional feature grid of the neural radiation field or a decomposed feature of the three-dimensional feature grid; finally, the compact representation is compressed and encoded by an encoder based on a neural network to obtain a first encoding result.
[0088] In the above implementation, by performing representation generation and feature transformation on the 3D digital human obtained after semantic analysis of the 3D data based on the neural radiation field, and performing compression encoding processing on the transformed compact representation, efficient encoding of the 3D digital human can be achieved while ensuring the quality of the rendering view, reducing the amount of data required to transmit the 3D digital human, thereby improving the efficient transmission of the 3D digital human.
[0089] To facilitate understanding of this embodiment, a digital human processing method based on a neural network disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the method provided in the embodiment of the present disclosure is generally an electronic device with certain computing capabilities. In some possible implementations, the digital human processing method based on a neural network can be implemented by a processor calling a computer-readable instruction stored in a memory.
[0090] See also Figure 1 FIG. 1 is a flowchart of a method for processing a digital human based on a neural network provided by an embodiment of the present disclosure, the method comprising steps S101 to S103, wherein:
[0091] S101: Perform semantic analysis on input 3D data based on neural radiation fields to obtain 3D digital humans and backgrounds; wherein the semantic analysis includes: 2D and / or 3D object detection, or 2D and / or 3D semantic segmentation.
[0092] In the disclosed embodiment, the input signal (3D data) can be semantically analyzed by the semantic analysis module, so as to decompose the 3D data into two-dimensional background data (i.e., background) and 3D digital human. The semantic analysis can be 2D and / or 3D object detection, or 2D and / or 3D semantic segmentation.
[0093] Neural radiance field (NeRF) takes the position coordinates and observation direction of the sampling point in three-dimensional space as input, and uses a multi-layer perceptron (MLP) to predict the radiation information of the sampling point, where the radiation information includes volume density, color information, shadow and other information. In other words, the essence of the neural radiation field is a function that maps any 3D position (x, y, z) and observation direction d to its volume density σ and view-dependent color c (color information), supporting differentiable rays for volume rendering. Here, a 3D grid with multi-channel features can be used to model the NeRF function.
[0094] S102: Generate a representation of the 3D digital human to obtain a three-dimensional feature grid of the neural radiation field of the 3D digital human; perform feature transformation on the three-dimensional feature grid to obtain a compact representation of the 3D digital human; wherein the compact representation at least includes: the three-dimensional feature grid of the neural radiation field or the decomposed features of the three-dimensional feature grid.
[0095] In the disclosed embodiment, the characterization generation module can generate a characterization of the relevant data of the 3D digital human output by the foreground and background analysis module to obtain a three-dimensional feature grid; then, the three-dimensional feature grid is subjected to feature transformation to obtain a compact characterization of the 3D digital human.
[0096] In specific implementation, the representation generation specifically includes the following steps: decomposing the relevant data of the 3D digital human into neural radiation fields and external parameters; wherein the external parameters at least include: camera parameters and driving parameters of the 3D model of the 3D digital human, etc.; then, converting the above-decomposed neural radiation fields into a voxel-based three-dimensional feature grid, that is, a 3D grid (Grid) form. Finally, by performing feature transformation on the three-dimensional feature grid, a more compact representation of the 3D digital human, that is, a compact representation, is obtained.
[0097] Here, a tensor decomposition algorithm can be used to decompose the neural radiation field to obtain a compact representation and network model parameters, wherein the network model parameters can be understood as model parameters of a multilayer perceptron (MLP) corresponding to the neural radiation field. In the disclosed embodiment, the amount of data in the compact representation is lower than the amount of data in the neural radiation field.
[0098] S103: compress and encode the compact representation using a neural network-based encoder to obtain a first encoding result.
[0099] In the embodiment of the present disclosure, the compact representation can be compressed and encoded by an encoder based on a neural network to obtain a first encoding result. Here, different encoders can be used to perform compression encoding processing for different types of feature data in the compact representation.
[0100] In the above implementation, by performing representation generation and feature transformation on the 3D digital human obtained after semantic analysis of the 3D data based on the neural radiation field, and performing compression encoding processing on the transformed compact representation, efficient encoding of the 3D digital human can be achieved while ensuring the quality of the rendering view, reducing the amount of data required to transmit the 3D digital human, thereby improving the efficient transmission of the 3D digital human.
[0101] In an optional embodiment, in the above Figure 1 Based on the described embodiment, the method further comprises the following steps:
[0102] Step S11: compressing and encoding the background by a video encoder to obtain a third encoding result;
[0103] Step S12: Transmitting the third encoding result to a decoding end.
[0104] In the disclosed embodiment, after acquiring the input signal (3D data), the encoding end can perform semantic analysis on the input signal through the foreground and background analysis module to obtain a semantic analysis result. The input signal can be separated into foreground and background through the semantic analysis, thereby separating the 3D digital human and the background. The background can be 2D background data and / or material data of the 3D digital human.
[0105] Here, it should be noted that if the input signal does not contain background, the 3D digital human can be directly analyzed through semantic analysis. After the 3D digital human is obtained through semantic analysis, the 3D digital human can be processed for representation generation to obtain a compact representation.
[0106] In the embodiment of the present disclosure, when the background is obtained based on semantic analysis, the background can be compressed and encoded by a video encoder to obtain a third encoding result; then, the third encoding result can be transmitted to a decoding end.
[0107] Here, the following video encoders can be used to compress and encode the background, such as H.264, H.265, H.266, AVS3, AV1, VP9, etc. At the same time, a video encoder based on a neural network can also be used to compress and encode the background. No specific limitation is made here, and it is subject to what can be implemented.
[0108] Through the above processing method, the separation of 3D digital people and background can be achieved. Different methods can be used for encoding processing for different types of signals. Therefore, the above separation method provides a technical basis for data compression.
[0109] In an optional implementation, the above step performs feature transformation on the three-dimensional feature grid to obtain a compact representation of the 3D digital human, specifically comprising the following steps:
[0110] Step S21: decomposing the three-dimensional feature grid to obtain decomposition features of the three-dimensional feature grid; wherein the decomposition features include: tensor decomposition results and / or plane representations;
[0111] Step S22: Determine the compact representation based on the three-dimensional feature grid or the decomposed features.
[0112] From the above description, we can see that the essence of the neural radiation field is a function that maps any 3D position (x, y, z) and observation direction d to its volume density σ and view-dependent color c (color). Therefore, a regular feature grid with multi-channel features (per voxel) can be used to model such a function.
[0113] In the disclosed embodiment, the relevant data of the 3D digital human can be decomposed into a neural radiation field and external parameters; wherein the external parameters include at least: camera parameters and driving parameters of the 3D model of the 3D digital human, etc.; then, the decomposed neural radiation field is converted into a voxel-based three-dimensional feature grid.
[0114] Next, the three-dimensional feature grid can be transformed, so that the three-dimensional feature grid representation of NeRF can be further decomposed into compact representations. The compact representations applicable to the technical solution of the present disclosure mainly include but are not limited to the following three types:
[0115] 1) Grid representation (representation of the above three-dimensional feature grid): The NeRF model is represented in the form of a 3D grid (three-dimensional feature grid).
[0116] Here, Grid representation refers to representing the NeRF model through polygonal grids, which is a very important way to represent the NeRF model.
[0117] 2) Vector-matrix representation (the above tensor decomposition result): Based on the Grid representation, the NeRF model is decomposed into multiple 1D vectors and 2D matrices.
[0118] Here, the vector-matrix representation uses 1D vectors and 2D matrices to represent the NeRF model. First, the scene can be modeled as a 4D tensor, where each item in the tensor corresponds to a voxel, and each voxel can represent volume density and multi-dimensional feature information; based on the 4D tensor, the tensor decomposition technology is used to decompose it into multiple low-rank tensors, that is, multiple one-dimensional vectors and two-dimensional matrices. Compared with the initial Grid representation of NeRF, the matrix-vector representation is more compact, with a smaller model size, higher rendering quality, and faster rendering speed.
[0119] In the disclosed embodiment, the tensor decomposition algorithm may adopt CANDECOMP / PARAFAC (CP) decomposition or vector-matrix (VM) decomposition method; in addition, other tensor decomposition algorithms may also be adopted.
[0120] Taking vector-matrix representation as an example, the goal of decomposition is to decompose the neural radiation field into more compact components for scene modeling. For any 3D tensor It can be decomposed into multiple matrices and vectors for representation. The above tensor can be decomposed into:
[0121]
[0122] Among them, for the rth component of each tensor mode, the factorization vectors of the three tensor modes are defined as is the matrix factor. The superscripts 1, 2, and 3 represent the tensor mode. The three tensor modes naturally correspond to the three spatial axes X, Y, and Z. Therefore, the three-dimensional scene can be directly represented by X, Y, and Z. Under this premise, for most scenes, R1=R2=R3=R is considered, so the above formula can be further simplified to the following formula:
[0123]
[0124] In the disclosed embodiment, the high-dimensional data is reduced in dimension by using a tensor decomposition algorithm, and a three-dimensional tensor can be decomposed into multiple low-rank tensor components, i.e., vector and matrix expressions. In the actual processing process, the four-dimensional tensor can be represented by low-rank tensor components, thereby reducing the storage space of this part of the data and improving the operation speed.
[0125] Typically, a 3D grid can be used To model the above function mapping to volume density σ and view-dependent color c, the 3D grid has multi-channel features per voxel. Here, the network can be divided into a geometric grid according to the feature channel and The two can model the volume density σ and color c respectively.
[0126] By definition, the geometric grid is a 3D tensor, the appearance grid is a 4D tensor, where I, J, and K represent the feature resolution of the X, Y, and Z axes, respectively. In the matrix-vector compact representation of the NeRF neural radiation field, we can convert the 3D geometric tensor It can be factored into:
[0127]
[0128] On the other hand, the tensor representing the appearance color has an additional pattern corresponding to the dimension of the feature channel. Compared with the XYZ pattern, this pattern usually has lower dimensions, that is, the rank of the pattern is lower. Therefore, the vector is defined separately (with b r denoted) is used to represent the low-rank mode in the decomposition. Under this premise, It can be factored into the following expression:
[0129]
[0130] 3) Plane representation: The Grid representation is further decomposed into 6 planes for representation.
[0131] Here, one plane can be used to represent an image. Similarly, three planes can be used to represent a static three-dimensional scene, while a dynamic three-dimensional scene requires three static planes plus the time dimension t to represent it. That is, six planes can be used to uniformly represent dynamic and static scenes, of which three planes are used for static scenes and three planes for dynamic scenes. A dynamic scene can be understood as a plane composed of any two dimensions of the three-dimensional coordinates x, y, and z and the time dimension t, three spatial planes Pxy, Pyz, and Pxz, and three temporal planes Pxt, Pyt, and Pzt. The above six-plane representation method has high reconstruction fidelity and low memory usage in reconstructing static and dynamic scenes. It can achieve 1000 times compression based on the NeRF Grid representation method, which is very competitive.
[0132] Among the three neural radiation field representation methods introduced above, Grid representation is an uncompressed representation method, and vector matrix representation and plane representation are compact representation methods of NeRF models after optimization. It should be noted that the present invention can also match and adapt to various other representation methods based on NeRF model compression, not limited to the above three, and no specific limitation is made here.
[0133] After the feature transformation step is completed, the result of the feature transformation step (for example, a three-dimensional feature grid or decomposed features of a three-dimensional feature grid) can be sent to a neural network-based encoder for encoding. Here, the process of the neural network-based encoder is as follows: Figure 2As shown, Figure 2 As shown, the neural network-based encoder includes: a characterization sub-encoder and two-way encoding modules; wherein, the first encoding module in the two-way encoding modules at least includes a super-prior encoder, a first quantization, and a first entropy encoder, and the second encoding module in the two-way encoding modules at least includes a second quantization and a second entropy encoder, and the output results of the two-way encoding modules are the first encoding results.
[0134] like Figure 2 As shown in the figure, the compact representation output after feature transformation is the input of the encoder based on the neural network. First, the compact representation passes through the representation sub-encoder, and then is divided into two paths. One path is sent to the probability estimation module to pass through the super prior encoder and the first quantization respectively, and then entropy encoding is performed through the first entropy encoder; the other path is directly subjected to the second quantization and then entropy encoding is performed through the second entropy encoder. The outputs of the two paths are combined as the output of the entire representation encoder, that is, the first encoding result; finally, the first encoding result is encapsulated with the encoding results output by the video encoder and the metadata encoder and sent to the network transmission module, so as to transmit the above encoding results to the decoding end through the network transmission module.
[0135] In an optional implementation, the step S103 performs compression encoding processing on the compact representation through an encoder based on a neural network to obtain a first encoding result, which specifically includes the following steps:
[0136] Step S31: Processing the compact representation through the representation sub-encoder to obtain hidden layer features;
[0137] Step S32: determining a priori probability distribution of the hidden layer features based on the super a priori encoder, and processing the priori probability distribution through the first quantization and the first entropy encoder;
[0138] Step S33: Based on the second encoding module, perform second quantization and entropy encoding on the hidden layer features, and determine the first encoding result based on the output results of the first encoding module and the second encoding module.
[0139] In the embodiment of the present disclosure, the input of the representation sub-encoder is the compact representation of the neural radiation field, and the output is the hidden layer feature. Here, a neural network can be used as the representation sub-encoder. For example, the structure of the representation sub-encoder can be Figure 3 In addition, other neural network-based encoders or neural networks that can obtain hidden layer feature representations of input graphs can also be used, which are not specifically limited here and are subject to implementation.
[0140] After obtaining the hidden layer features, the hidden layer features can be processed by the probability estimation module. The role of the probability estimation module is to capture the hidden information of the hidden layer features, thereby assisting the parameter generation of the entropy model and improving the mismatch between the entropy model estimation and the actual distribution of the hidden layer.
[0141] like Figure 2 As shown, the probability estimation module is mainly divided into two parts: a super-a priori encoder and a first quantization. Here, the super-a priori encoder can be used to determine the prior probability distribution of the hidden layer features, and then the prior probability distribution is quantized by the first quantization, and then the quantization result is entropy encoded by the first entropy encoder, so as to obtain the output result of the first encoding module.
[0142] Entropy coding relies on a priori probability model (entropy model) of quantization representation shared by the encoder and decoder. General entropy encoders can estimate the priori probability model by themselves, but there is still room for improvement, for example, removing statistical dependence. In order to achieve better entropy decoding effect, the technical solution disclosed in this disclosure uses a probability estimation module to estimate the priori probability model. Among them, the super priori encoder can use a neural network to complete the encoding, Figure 4 This is an example network structure, and other neural network encoders based on deep learning can also be used.
[0143] In the disclosed embodiment, the hidden layer features representing the output of the sub-encoder can also enter the second encoding module for processing. In specific implementation, the hidden layer features can be second quantized, and the quantization results can be entropy encoded by the second entropy encoder to obtain the output result of the second encoding module; finally, the first encoding result can be determined based on the output results of the first encoding module and the second encoding module.
[0144] In the disclosed embodiment, by quantizing the parameters representing the output of the sub-encoder or the super-a priori encoder, the number of parameter quantization bits can be reduced, which is beneficial to reducing the model parameter size and memory consumption. The disclosed technical solution can be applied to various quantization methods, such as setting a quantization matrix for quantization; or quantizing by learning quantization parameters through neural network training. The quantization method is not specifically limited here, and it is based on what can be implemented.
[0145] It can be seen from the above description that in the embodiment of the present disclosure, the result of the compact representation of the neural radiation field after passing through the representation sub-encoder is sent to the probability estimation module and the second quantization in two ways, and the quantization results of the two ways are sent to two entropy encoders for further compression encoding. Entropy coding is a scheme for lossless data compression that is independent of the specific characteristics of the medium. Common entropy coding methods include: Shannon coding, Huffman coding, and arithmetic coding, all of which can be used here without specific restrictions. For example, the technical solution of the present disclosure can use arithmetic coding, and use a simple non-adaptive binary arithmetic encoder to implement entropy coding. The elements in the two outputs are independently converted to their binary integer representations, and arithmetic coding is performed from the most significant bit to the least significant bit.
[0146] In an optional embodiment, Figure 1 On the basis of the corresponding embodiment, the method further includes:
[0147] Step S41: obtaining network conditions;
[0148] Step S42: adjusting the coding parameters and / or coding mode according to the network conditions.
[0149] Here, the encoding parameter and / or encoding mode is data used by the encoder during the process of performing compression encoding on the compact representation.
[0150] In a specific implementation, network conditions may be obtained, wherein the network conditions are transmission conditions of a network transmission module between the encoding end and the decoding end; and then, encoding parameters and / or encoding modes are adjusted according to the network conditions; wherein the encoding parameters and / or encoding modes are information for performing the encoding process.
[0151] In the disclosed embodiment, before the compression and compaction representation, the network condition of the network transmission module can also be obtained, for example, the network transmission speed of the network transmission module can be determined. Then, the encoding parameters and / or encoding mode can be adjusted according to the network condition, wherein the encoding parameters and / or encoding mode are the parameters required for performing the encoding process in the above steps.
[0152] For example, the coding parameters that match the network conditions can be determined; wherein the coding parameters can be parameters such as coding parameters and bit rates. Here, the mapping relationship between the network conditions and the coding parameters can be preset, for example, the value range of the network conditions, and the parameter value of the coding parameters corresponding to the value range can be set. For another example, the conditional value points of the network conditions, and the parameter values of the coding parameters corresponding to the conditional value points can be set. For another example, a mapping function can be constructed, wherein the mapping function is used to indicate the mapping relationship between the network conditions and the coding parameters. Here, the network conditions can be substituted into the mapping function to obtain the coding parameters that match the network conditions.
[0153] At this time, the coding parameters corresponding to the network conditions, ie, the coding parameters and / or the coding mode, may be determined according to the mapping relationship described above.
[0154] In an embodiment of the present disclosure, after determining the coding parameters and / or coding mode that match the network conditions, the results of the quantized super-prior encoder and the characterization sub-encoder can be encoded according to the coding parameters and / or coding mode to obtain a first coding result.
[0155] Through the above processing method, the encoding parameters can be dynamically adjusted according to the network conditions of the network transmission module to achieve efficient transmission of the 3D digital human.
[0156] In an optional embodiment, Figure 1 On the basis of the corresponding embodiment, the method further includes:
[0157] Step S51, obtaining the requirements of the downstream tasks provided by the terminal side;
[0158] Step S52: determining matching encoding parameters and / or adjusting bit rate control strategy based on the requirement.
[0159] In the embodiments of the present disclosure, the terminal side's requirements for the rendering viewpoint and / or 3D model of the 3D digital human can also be obtained, that is, requirements such as playback resolution, frame rate, accuracy of 3D data, playback network status (for example, playback through traffic, or playback by accessing a wireless network), and playback environment.
[0160] At this time, matching encoding parameters and / or bit rate control strategies may be determined according to the requirements of the terminal side for the 3D digital human.
[0161] Here, the requirement includes at least one sub-requirement of a requirement dimension. For each sub-requirement of each dimension, a corresponding requirement value point and a requirement value range can be set. A mapping relationship between the requirement value point and the encoding parameter and the bit rate control strategy can be set, and a mapping relationship between the requirement value range and the encoding parameter and the bit rate control strategy can be set.
[0162] In specific implementation, the requirement value range of each sub-requirement, the parameter value of the coding parameter corresponding to the requirement value range, and the corresponding rate control strategy can be set. For another example, the requirement value point of each sub-requirement, the parameter value of the coding parameter corresponding to the requirement value point, and the corresponding rate control strategy can be set. For another example, a mapping function can be constructed, where the mapping function is used to indicate the mapping relationship between the sub-requirement and the coding parameter and the rate control strategy. Here, the requirement value of each sub-requirement can be substituted into the mapping function to obtain matching coding parameters and rate control strategies.
[0163] At this time, the encoding parameters and / or the rate control strategy that match the network condition can be determined according to the mapping relationship described above. Afterwards, the encoding process can be performed according to the matching encoding parameters and / or the rate control strategy to obtain the first encoding result.
[0164] Through the above processing method, the encoding parameters and / or the bit rate control strategy can be dynamically adjusted according to the real-time transmission status of the network transmission module to achieve efficient transmission of the 3D digital human.
[0165] In an optional implementation, after the compact representation is compressed and encoded by an encoder based on a neural network to obtain a first encoding result, the method further includes:
[0166] The first encoding result is transmitted to a decoding end; wherein the first encoding result is used by the decoding end to render a reconstruction rendering result of the 3D digital human; wherein the reconstruction rendering result includes a rendering viewpoint and / or a 3D model.
[0167] In the embodiment of the present disclosure, after the encoding end obtains the first encoding result through encoding, the encoding end may transmit the first encoding result to the decoding end; then, the decoding end may decode the first encoding result to obtain a first decoding result of the first encoding result. Finally, the rendering viewpoint and / or 3D model of the 3D digital human may be reconstructed and rendered based on the first decoding result, and the rendering viewpoint and / or 3D model may be displayed through the terminal side.
[0168] In an optional embodiment, the method further comprises:
[0169] According to the quality feedback from the terminal side, it is determined whether to start a quality enhancement module, wherein the quality enhancement module is used to enhance the rendering quality of the rendering viewpoint or the 3D model.
[0170] Here, a quality enhancement module may be set at the edge node, wherein the quality enhancement module is used to perform quality enhancement processing on the rendered viewpoint and / or 3D model. The terminal side may feed back quality feedback of the rendered viewpoint and / or 3D model to the quality enhancement module. After being activated, the quality enhancement module may perform quality enhancement processing on the rendered viewpoint and / or 3D model, and feed back the rendered viewpoint and / or 3D model after quality enhancement processing to the terminal side for display.
[0171] Here, the quality feedback can be determined based on the playback effect of the rendering viewpoint and / or 3D model on the terminal side at the historical moment. The quality feedback can be automatically generated by the terminal side, or the user can input the quality feedback on the terminal side; wherein, the quality feedback can be a numerical value or a text description, and the information type of the quality feedback is not specifically limited here. For example, the quality feedback can be: general, good, bad, very good, etc.
[0172] After obtaining the quality feedback, it can be determined whether to perform quality enhancement processing on the rendering viewpoint and / or 3D model according to the quality feedback; for example, if the quality feedback is: not good, it is determined that the rendering viewpoint and / or 3D model needs to be enhanced. At this time, the rendering viewpoint and / or 3D model can be enhanced in a specified enhancement manner.
[0173] Here, an enhancement mode that matches the quality feedback may be determined as the designated enhancement mode; in addition, an enhancement mode preset by the user may also be determined as the designated enhancement mode.
[0174] In an embodiment of the present disclosure, after obtaining quality feedback from the terminal side, it is also possible to detect whether a quality enhancement function is turned on. If it is detected that the quality enhancement function is turned on, a step of determining whether to perform quality enhancement processing on the rendering viewpoint and / or 3D model based on the quality feedback is executed.
[0175] In an optional embodiment, the method further comprises:
[0176] Step S61, obtaining external parameters and network model parameters obtained after characterizing and generating the 3D digital human; wherein the external parameters at least include: camera parameters and driving parameters of the 3D model of the 3D digital human;
[0177] Step S62: compress and encode the external parameters and the network model parameters through a Metadata encoder to obtain a second encoding result, and transmit the second encoding result to a decoding end.
[0178] From the above description, we can know that the representation generation mentioned above can convert 3D digital humans into neural radiation fields and external parameters. The neural radiation field is sent to the representation encoder to complete the encoding process, while the external parameters need to be sent to the Metadata encoder. Metadata contains network model parameters such as MLP of the neural radiation field, camera parameters (pose, internal and external parameters, etc.), drive parameters and other data. The Metadata encoder is required to be a lossless encoder, such as Huffman coding or arithmetic coding. This encoder has no other restrictions and can also use other lossless compression methods.
[0179] At this time, after obtaining the external parameters and the network model parameters, the Metadata encoder can be used to compress and encode the external parameters and the network model parameters to obtain a second encoding result, and the second encoding result of the parameters is transmitted to the decoding end.
[0180] See also Figure 5 FIG. 5 is a flowchart of another method for processing a digital human based on a neural network provided by an embodiment of the present disclosure. The method is applied to a decoding end. The method includes steps S501 to S503, wherein:
[0181] S501: Obtain a first encoding result and a second encoding result sent by an encoding end; wherein, the first encoding result is obtained by the encoding end performing compression encoding on a compact representation of the 3D digital human, the compact representation is obtained by performing feature transformation on a three-dimensional feature grid of the neural radiation field of the 3D digital human, the three-dimensional feature grid is generated by representing the 3D digital human, the compact representation at least includes: a three-dimensional feature grid of the neural radiation field or a decomposed feature of the three-dimensional feature grid, the second encoding result is obtained by the encoding end performing compression encoding on external parameters and network model parameters obtained after performing feature transformation on the three-dimensional feature grid, the external parameters at least include: camera parameters and driving parameters of the 3D model of the 3D digital human.
[0182] Here, the determination of the first encoding result and the second encoding result is as described in the above embodiment and will not be described in detail here.
[0183] S502: Decoding the first encoding result through a decoder based on a neural network, and decoding the second encoding result to obtain a first decoding result and a second decoding result respectively;
[0184] S503: Reconstruct and render according to the first decoding result and the second decoding result to obtain a rendering viewpoint and / or a 3D model of the 3D digital human, and display the rendering viewpoint and / or the 3D model through the terminal side.
[0185] In the above implementation, by performing representation generation and feature transformation on the 3D digital human obtained after semantic analysis of the 3D data based on the neural radiation field, and performing compression encoding processing on the transformed compact representation, efficient encoding of the 3D digital human can be achieved while ensuring the quality of the rendering view, reducing the amount of data required to transmit the 3D digital human, thereby improving the efficient transmission of the 3D digital human.
[0186] In an optional implementation, the above step S502 decodes the first encoding result through a decoder based on a neural network to obtain a first decoding result, which specifically includes the following steps:
[0187] Step S71, decoding the output result of the first encoding module in the first encoding result by a first entropy decoder to obtain a first entropy decoding result, and decoding the first entropy decoding result by a super a priori decoder to obtain a super a priori decoding result;
[0188] Step S72, constructing an entropy model of a second entropy decoder based on the super a priori decoding result, and decoding the output result of the second encoding module in the first encoding result by the second entropy decoder based on the entropy model to obtain a second entropy decoding result;
[0189] Step S73: decoding the second entropy decoding result by using the representation sub-decoder to obtain the first decoding result.
[0190] In the embodiments of the present disclosure, Figure 6 As shown, the first coding result transmitted by the network transmission module is also divided into two paths according to the setting of the encoder based on the neural network. First, the output result of the first coding module is sent to the first entropy decoder for entropy decoding, and the decoded result (first entropy decoding result) is sent to the super priori decoder; the output of the super priori decoder enters the second entropy decoder, which allows the decoders to share the entropy model. After constructing the entropy model of the second entropy decoder, the hidden layer features are decompressed based on the correct entropy model. That is, the output result of the second coding module is sent to the second entropy decoder for decoding, and the decoded parameters are sent to the representation sub-decoder, which is output after decoding to obtain the first decoding result.
[0191] Here, the first entropy decoder and the second entropy decoder adopt a lossless compression coding method, and the bit stream generated by it can restore data without distortion through entropy decoding. The corresponding entropy coding scheme is the inverse transformation of the above entropy coding scheme.
[0192] Super a priori decoding is the inverse process of super a priori encoding. After super a priori decoding, the a priori probability model (entropy model) of the quantized representation shared by the second entropy encoder and the second entropy decoder can be obtained, which helps to improve the decoding effect of the second entropy decoder. The super a priori decoder can also use a neural network to complete the decoding process. Figure 7 This is an example network structure, and other neural network video decoders based on deep learning can also be used.
[0193] Here, the process of sending the result of the second entropy decoder (i.e., the second entropy decoding result) to the representation sub-decoder for decoding can be understood as the inverse transformation of the representation sub-encoder. The representation sub-decoder can also use a neural network to complete the decoding process. Figure 8 This is an example network structure, and other decoders based on deep learning neural networks can also be used.
[0194] In an optional embodiment, the method further comprises the following steps:
[0195] After obtaining the third encoding result transmitted by the encoding end, the third encoding result is decoded by a video decoder to obtain a third decoding result; wherein the third encoding result is obtained by encoding the background obtained by the encoding end after semantic analysis of 3D data based on the neural radiation field.
[0196] In the embodiment of the present disclosure, the encoding end may perform compression encoding processing on the background through a video encoder to obtain a third encoding result, and transmit the third encoding result to the decoding end. After obtaining the third encoding result, the decoding end may perform decoding processing on the third encoding result through a video decoder to obtain a third decoding result.
[0197] Here, the video decoder and the video encoder are of the same type, and the video decoder is the inverse process of the video encoding; wherein, the third encoding result is decoded, and the third decoding result obtained may be the decoded 2D background data and / or material data. Finally, the first decoding result, the second decoding result, and the third decoding result may be reconstructed and rendered to obtain the rendering viewpoint and / or 3D model of the 3D digital human.
[0198] In an optional implementation, the above step decodes the second encoding result to obtain a second decoding result, including:
[0199] The second encoding result is decoded by a Metadata decoder to obtain a second decoding result.
[0200] In the disclosed embodiment, after obtaining the second encoding result, the decoding end can decode the second encoding result through the Metadata decoder to obtain the second decoding result. Here, the Metadata decoder is consistent with the Metadata encoder technology route, and the Metadata decoder is the inverse process of Metadata encoding. The first decoding result output by the Metadata decoder contains network model parameters such as MLP, camera parameters (pose, internal and external parameters, etc.), drive parameters and other data.
[0201] In the embodiment of the present disclosure, there are neural radiation fields and multiple neural network codecs. The following example illustrates the training method of the neural radiation field and neural network codec parameters. In addition to the training method in the technical solution of the present disclosure, other neural network parameter training methods can also be used without specific restrictions. The neural radiation field and codec can be trained in the following manner:
[0202] First, the initial neural radiation field is trained through training samples to obtain the intermediate neural radiation field;
[0203] Secondly, based on the sample labels of the training samples and the prediction results of the intermediate neural radiation field on the training samples, a reconstruction loss function value is determined; wherein the reconstruction loss function is the L2 norm between the rendering result output by the decoding end to be trained and the ground truth value;
[0204] Finally, the initial state of the codec is set based on the reconstruction loss function value, and the codec and the intermediate neural radiation field are trained from the initial state to obtain the trained neural radiation field and the trained codec; wherein the codec includes: an encoder based on a neural network in the encoding end to be trained and a decoder based on a neural network in the decoding end to be trained.
[0205] In the embodiment of the present disclosure, a multi-stage training method can be used to train the neural radiation field and the neural network codec parameters. The multi-stage training method can be roughly divided into the following steps:
[0206] (1) The NeRF neural radiation field (initial neural radiation field) is pre-trained iteratively multiple times through training samples to obtain a compact representation of the corresponding scene and multi-layer perceptron MLP parameters (i.e., intermediate neural radiation field). Then, the reconstruction loss function value is determined based on the prediction results of the sample label and the intermediate neural radiation field for the training sample.
[0207] (2) Preheating the encoder and decoder of the above neural network by reconstructing the loss function value to set the initial state of the encoder and decoder, thereby providing a better starting point for the formal training of the randomly initialized encoder and decoder.
[0208] (3) Through the target loss function, the neural network codec and the intermediate neural radiation field are jointly trained for multiple iterations to obtain the trained neural radiation field and the trained codec. If the quantization process in the codec link is completed by a neural network, then the quantization network is iteratively trained according to the parameters of the joint training in step (3); if a non-neural network method is used for quantization, this step can be omitted.
[0209] In the embodiment of the present disclosure, the step of training the codec and the intermediate neural radiation field from the initial state to obtain the trained neural radiation field and the trained codec specifically includes:
[0210] First, determine a target loss function; wherein the target loss function is determined based on a reconstruction loss function value and an entropy loss function, the reconstruction loss function is an L2 norm between a rendering result output by the decoder to be trained and a ground truth value, and the entropy loss function is calculated based on an input of a second entropy encoder and an input of a first entropy encoder in the encoder to be trained;
[0211] Then, starting from the initial state, the codec and the intermediate neural radiation field are repeatedly iteratively trained through the target loss function to obtain a trained neural radiation field and a trained codec.
[0212] The disclosed technical solution uses the weighted sum of the reconstruction loss function and the entropy loss function as the final target loss function, as shown in the formula:
[0213] L final =L recon +λ entropy L entropy #; Among them, the reconstruction loss function L recon It is defined as the L2 norm between the rendering result and the ground truth, as shown in the formula:
[0214] Where r represents the ray traveling from the camera origin through the pixel, represents the set of all rays in a single forward pass, represents the rendered color of the corresponding light, and C(r) represents the ground truth color of the corresponding light.
[0215] The entropy loss function is defined as the quantized implicit encoding (the output after characterization sub-encoder and quantization, which is also the input of the second entropy encoder), and the output of the probability estimation module (Input of the first entropy encoder) The entropy of the probability estimate is shown in the formula:
[0216]
[0217] It is considered as quantized implicit encoding bit rate, quantizing the potential code The sum of the bit rates of . Quantized implicit coding and It can be expressed as:
[0218] and in, is the probability estimate of the corresponding variable.
[0219] The following will be combined Fig. 9 A detailed introduction to the digital human processing system based on neural network is given. Fig. 9 As shown, the digital human processing system includes an encoding end, a decoding end and a terminal side.
[0220] At the encoding end, the input signal is obtained, which may include the 3D digital human, the corresponding 2D background data and / or material data, wherein the background or material is not necessary. Therefore, the input will first undergo semantic analysis to decouple the digital human and the background or material. Then, the 3D digital human enters the representation generation to generate a three-dimensional feature grid of the neural radiation field of the 3D digital human; then, the three-dimensional feature grid is subjected to feature transformation to obtain the compact representation and network model parameters of the 3D digital human. Here, if the compact representation does not contain the decomposition features of the three-dimensional feature grid, the result after the representation generation (i.e., the three-dimensional feature grid) is no longer processed by feature transformation, and is directly sent to the neural network-based encoder for compression encoding. The compact representation will enter the neural network-based encoder for compression; and the network model parameters and external parameters (camera parameters, drive parameters, etc.) as Metadata will enter the Metadata encoder for encoding. The background or material, etc. enter the video encoder for compression. Afterwards, the encoding results of the video encoder, the neural network-based encoder, and the Metadata encoder are encapsulated and distributed to the decoding end through the network transmission module.
[0221] here, Fig. 9 The probability estimation module is as follows Figure 2 The probability estimation module shown, Fig. 9 The first quantization in the probability estimation module is Figure 2 Quantization of the probability estimation module, Fig. 9 The middle entropy encoder 1 is the first entropy encoder mentioned above. Fig. 9 The middle entropy encoder 2 is the second entropy encoder mentioned above. Fig. 9 The quantization between the representation sub-encoder and the entropy encoder 2 is Figure 2 The second quantization in . Fig. 9 The medium entropy decoder 1 is Figure 6The first entropy decoder in Fig. 9 The medium entropy decoder 2 is Figure 6 The second entropy decoder. Fig. 9 The processing process of the neural network-based encoder and the neural network-based decoder is as described in the above implementation manner and will not be described in detail here.
[0222] Here, the network condition of the network transmission module is fed back to the encoding end in the form of signal A, and the encoding end adjusts the encoding parameters and / or encoding method based on the network condition.
[0223] At the decoding end, the received code stream will be initially parsed. For example, the 2D background data or material data stream is decoded by the video decoder, the representation stream transmitted by the representation encoder is decoded by the neural network-based decoder, and the metadata data stream is decoded by the metadata decoder. Then, the decoding results of the video decoder, the neural network-based decoder, and the metadata decoder are input to the reconstruction rendering module to synthesize the final output, thereby presenting the rendering viewpoint and / or 3D model of the 3D digital human on the terminal.
[0224] Here, based on the signal C of the quality feedback from the terminal side, it can be selected whether to start the quality enhancement module (i.e., the quality enhancement module in the above embodiment) to enhance the quality of the rendered viewpoint or 3D model. In addition to the above process, the terminal side can also feed back the needs of the terminal side to the encoding end through the signal B based on the downstream task, so as to better select the encoding parameters and adjust the bit rate control strategy.
[0225] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.
[0226] Based on the same inventive concept, the embodiments of the present disclosure also provide a neural network-based digital human processing device corresponding to the neural network-based digital human processing method. Since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to the above-mentioned neural network-based digital human processing method in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0227] Reference Fig.10 FIG. 1 is a schematic diagram of a digital human processing device based on a neural network provided by an embodiment of the present disclosure, wherein the device comprises: a semantic analysis module 10, a representation generation module 20, a feature transformation module 30, and a compression coding module 40; wherein,
[0228] A semantic analysis module, used for performing semantic analysis on the input 3D data based on the neural radiation field to obtain a 3D digital human and background; wherein the semantic analysis may adopt 2D and / or 3D object detection, or 2D and / or 3D semantic segmentation;
[0229] A representation generation module, used to generate a representation of the 3D digital human to obtain a three-dimensional feature grid of the neural radiation field of the 3D digital human;
[0230] A feature transformation module, used for performing feature transformation on the three-dimensional feature grid to obtain a compact representation of the 3D digital human; wherein the compact representation at least includes: the three-dimensional feature grid of the neural radiation field or the decomposed features of the three-dimensional feature grid;
[0231] The compression coding module is used to compress and encode the compact representation through an encoder based on a neural network to obtain a first coding result.
[0232] In one possible implementation, the neural network-based encoder includes: a characterization sub-encoder and a two-way encoding module; wherein the first encoding module in the two-way encoding module at least includes a super-prior encoder, a first quantization, and a first entropy encoder, and the second encoding module in the two-way encoding module at least includes a second quantization and a second entropy encoder, and the output results of the two-way encoding modules are the first encoding results.
[0233] In a possible implementation manner, the compression encoding module is further used to:
[0234] Processing the compact representation by the representation sub-encoder to obtain hidden layer features;
[0235] Determine a priori probability distribution of the hidden layer features based on the super a priori encoder, and process the prior probability distribution through the first quantization and the first entropy encoder;
[0236] Based on the second encoding module, the hidden layer features are second quantized and entropy encoded, and the first encoding result is determined based on the output results of the first encoding module and the second encoding module.
[0237] In a possible implementation manner, the feature transformation module is further used to:
[0238] Decomposing the three-dimensional feature grid to obtain decomposed features of the three-dimensional feature grid; wherein the decomposed features include: tensor decomposition results and / or plane representations;
[0239] The compacted representation is determined based on the three-dimensional feature grid or the decomposed features.
[0240] In a possible implementation manner, the device is also used for:
[0241] Get network conditions;
[0242] The coding parameters and / or coding mode are adjusted according to the network conditions.
[0243] In a possible implementation manner, the device is also used for:
[0244] Obtain the requirements of downstream tasks provided by the terminal side;
[0245] Based on the requirement, matching encoding parameters are determined and / or a bit rate control strategy is adjusted.
[0246] In a possible implementation manner, the device is also used for:
[0247] After the compact representation is compressed and encoded by a neural network-based encoder to obtain a first encoding result, the first encoding result is transmitted to a decoding end; wherein the first encoding result is used by the decoding end to render a reconstructed rendering result of the 3D digital human; wherein the reconstructed rendering result includes a rendering viewpoint and / or a 3D model.
[0248] In a possible implementation manner, the device is also used for:
[0249] According to the quality feedback from the terminal side, it is determined whether to start a quality enhancement module, wherein the quality enhancement module is used to enhance the rendering quality of the rendering viewpoint or the 3D model.
[0250] In a possible implementation manner, the device is also used for:
[0251] Acquiring external parameters, and acquiring network model parameters obtained after characterizing and generating the 3D digital human; wherein the external parameters at least include: camera parameters and driving parameters of the 3D model of the 3D digital human;
[0252] The external parameters and the network model parameters are compressed and encoded by a Metadata encoder to obtain a second encoding result, and the second encoding result is transmitted to a decoding end.
[0253] In a possible implementation manner, the device is also used for:
[0254] compressing and encoding the background by a video encoder to obtain a third encoding result;
[0255] The third encoding result is transmitted to a decoding end.
[0256] In a possible implementation manner, the apparatus is further used to: train the neural radiation field and the codec in the following manner:
[0257] The initial neural radiation field is trained by training samples to obtain the intermediate neural radiation field;
[0258] Determine a reconstruction loss function value based on the sample label of the training sample and the prediction result of the intermediate neural radiation field on the training sample; wherein the reconstruction loss function is the L2 norm between the rendering result output by the decoding end to be trained and the ground truth value;
[0259] The initial state of the codec is set based on the reconstruction loss function value, and the codec and the intermediate neural radiation field are trained from the initial state to obtain the trained neural radiation field and the trained codec; wherein the codec includes: an encoder based on a neural network in the encoding end to be trained and a decoder based on a neural network in the decoding end to be trained.
[0260] In a possible implementation manner, the device is also used for:
[0261] Determine a target loss function; wherein the target loss function is determined based on the reconstruction loss function value and an entropy loss function, and the entropy loss function is calculated based on the input of the second entropy encoder and the input of the first entropy encoder in the encoder to be trained;
[0262] Starting from the initial state, the codec and the intermediate neural radiation field are repeatedly iteratively trained using the target loss function to obtain a trained neural radiation field and a trained codec.
[0263] Reference Fig.11 FIG. 1 is a schematic diagram of another digital human processing device based on a neural network provided by an embodiment of the present disclosure, wherein the device comprises: an acquisition module 50, a decoding module 60, and a reconstruction and rendering module 70; wherein:
[0264] An acquisition module is used to acquire a first encoding result and a second encoding result sent by an encoding end; wherein, the first encoding result is obtained by the encoding end performing compression encoding on a compact representation of the 3D digital human, the compact representation is obtained by performing feature transformation on a three-dimensional feature grid of the neural radiation field of the 3D digital human, the three-dimensional feature grid is generated by representing the 3D digital human, the compact representation at least includes: a three-dimensional feature grid of the neural radiation field or decomposed features of the three-dimensional feature grid, the second encoding result is obtained by the encoding end performing compression encoding on external parameters and network model parameters obtained after performing feature transformation on the three-dimensional feature grid, the external parameters at least include: camera parameters and driving parameters of a 3D model of the 3D digital human;
[0265] A decoding module, used to decode the first encoding result through a decoder based on a neural network, and to decode the second encoding result, to obtain a first decoding result and a second decoding result respectively;
[0266] A reconstruction and rendering module is used to reconstruct and render according to the first decoding result, the second decoding result and the third decoding result to obtain the rendering viewpoint and / or 3D model of the 3D digital human, and display the rendering viewpoint and / or 3D model through the terminal side.
[0267] In a possible implementation manner, the decoding module is specifically configured to:
[0268] Decoding the output result of the first encoding module in the first encoding result by a first entropy decoder to obtain a first entropy decoding result, and decoding the first entropy decoding result by a super a priori decoder to obtain a super a priori decoding result;
[0269] Based on the super a priori decoding result, construct an entropy model of a second entropy decoder, and decode the output result of the second encoding module in the first encoding result by the second entropy decoder based on the entropy model to obtain a second entropy decoding result;
[0270] The second entropy decoding result is decoded by a representation sub-decoder to obtain the first decoding result.
[0271] In a possible implementation manner, the device is also used for:
[0272] After obtaining the third encoding result transmitted by the encoding end, the third encoding result is decoded by a video decoder to obtain a third decoding result; wherein the third encoding result is obtained by encoding the background obtained by the encoding end after semantic analysis of 3D data based on the neural radiation field.
[0273] In a possible implementation manner, the device is also used for:
[0274] Reconstruction and rendering are performed according to the first decoding result, the second decoding result and the third decoding result to obtain the rendering viewpoint and / or the 3D model of the 3D digital human.
[0275] In a possible implementation manner, the decoding module is specifically used to: decode the second encoding result through a Metadata decoder to obtain a second decoding result.
[0276] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0277] Corresponds to Figure 1 The digital human processing method based on neural network in the present disclosure also provides an electronic device 700, such as Fig.12 FIG. 7 is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present disclosure, including:
[0278] Processor 71, memory 72, and bus 73; memory 72 is used to store execution instructions, including internal memory 721 and external memory 722; the internal memory 721 is also called internal memory, which is used to temporarily store the operation data in the processor 71 and the data exchanged with the external memory 722 such as a hard disk. The processor 71 exchanges data with the external memory 722 through the internal memory 721. When the electronic device 700 is running, the processor 71 communicates with the memory 72 through the bus 73, so that the processor 71 executes the following instructions:
[0279] Performing semantic analysis on the input 3D data based on the neural radiation field to obtain a 3D digital human and background; wherein the semantic analysis includes: 2D and / or 3D object detection, or 2D and / or 3D semantic segmentation;
[0280] The 3D digital human is represented and generated to obtain a three-dimensional feature grid of the neural radiation field of the 3D digital human; the three-dimensional feature grid is feature transformed to obtain a compact representation of the 3D digital human; the compact representation at least includes: the three-dimensional feature grid of the neural radiation field or the decomposed features of the three-dimensional feature grid;
[0281] The compact representation is compressed and encoded by an encoder based on a neural network to obtain a first encoding result.
[0282] Or execute the following command:
[0283] Obtain a first encoding result and a second encoding result sent by an encoding end; wherein the first encoding result is obtained by the encoding end performing compression encoding on a compact representation of the 3D digital human, the compact representation is obtained by performing feature transformation on a three-dimensional feature grid of the neural radiation field of the 3D digital human, the three-dimensional feature grid is generated by representing the 3D digital human, the compact representation at least includes: a three-dimensional feature grid of the neural radiation field or decomposed features of the three-dimensional feature grid, the second encoding result is obtained by the encoding end performing compression encoding on external parameters and network model parameters obtained after performing feature transformation on the three-dimensional feature grid, the external parameters at least include: camera parameters and driving parameters of a 3D model of the 3D digital human;
[0284] Decoding the first encoding result through a decoder based on a neural network, and decoding the second encoding result to obtain a first decoding result and a second decoding result respectively;
[0285] Reconstruct and render according to the first decoding result and the second decoding result to obtain the rendering viewpoint and / or 3D model of the 3D digital human, and display the rendering viewpoint and / or 3D model through the terminal side.
[0286] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the digital human processing method based on a neural network described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.
[0287] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the digital human processing method based on a neural network described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.
[0288] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).
[0289] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.
[0290] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0291] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0292] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0293] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.
Claims
1. A digital human processing method based on a neural network, characterized in that: Applied to the encoding end, including: Performing semantic analysis on the input 3D data based on the neural radiation field to obtain a 3D digital human and background; wherein the semantic analysis includes: 2D and / or 3D object detection, or 2D and / or 3D semantic segmentation; The 3D digital human is represented and generated to obtain a three-dimensional feature grid of the neural radiation field of the 3D digital human; the three-dimensional feature grid is feature transformed to obtain a compact representation of the 3D digital human; the compact representation at least includes: the three-dimensional feature grid of the neural radiation field or the decomposed features of the three-dimensional feature grid; The compact representation is compressed and encoded by an encoder based on a neural network to obtain a first encoding result.
2. The method according to claim 1, characterized in that The neural network-based encoder includes: a characterization sub-encoder and a two-way encoding module; wherein, the first encoding module in the two-way encoding module at least includes a super-prior encoder, a first quantization, and a first entropy encoder, and the second encoding module in the two-way encoding module at least includes a second quantization and a second entropy encoder, and the output results of the two-way encoding modules are the first encoding results.
3. The method according to claim 2, characterized in that The step of compressing and encoding the compact representation by using a neural network-based encoder to obtain a first encoding result includes: Processing the compact representation by the representation sub-encoder to obtain hidden layer features; Determine a priori probability distribution of the hidden layer features based on the super a priori encoder, and process the prior probability distribution through the first quantization and the first entropy encoder; Based on the second encoding module, the hidden layer features are second quantized and entropy encoded, and the first encoding result is determined based on the output results of the first encoding module and the second encoding module.
4. The method according to claim 1, characterized in that The step of performing feature transformation on the three-dimensional feature grid to obtain a compact representation of the 3D digital human includes: Decomposing the three-dimensional feature grid to obtain decomposed features of the three-dimensional feature grid; wherein the decomposed features include: tensor decomposition results and / or plane representations; The compacted representation is determined based on the three-dimensional feature grid or the decomposed features.
5. The method according to claim 1, characterized in that The method further comprises: Get network conditions; The coding parameters and / or coding mode are adjusted according to the network conditions.
6. The method according to claim 1, characterized in that The method further comprises: Obtain the requirements of downstream tasks provided by the terminal side; Based on the requirement, matching encoding parameters are determined and / or a bit rate control strategy is adjusted.
7. The method according to claim 1, characterized in that After the compact representation is compressed and encoded by a neural network-based encoder to obtain a first encoding result, the method further includes: The first encoding result is transmitted to a decoding end; wherein the first encoding result is used by the decoding end to render a reconstruction rendering result of the 3D digital human; wherein the reconstruction rendering result includes a rendering viewpoint and / or a 3D model.
8. The method according to claim 1, characterized in that The method further comprises: According to the quality feedback from the terminal side, it is determined whether to start a quality enhancement module, wherein the quality enhancement module is used to enhance the rendering quality of the rendering viewpoint or the 3D model.
9. The method according to claim 1, characterized in that: The method further comprises: Acquiring external parameters, and acquiring network model parameters obtained after characterizing and generating the 3D digital human; wherein the external parameters at least include: camera parameters and driving parameters of the 3D model of the 3D digital human; The external parameters and the network model parameters are compressed and encoded by a Metadata encoder to obtain a second encoding result, and the second encoding result is transmitted to a decoding end.
10. The method according to claim 1, characterized in that The method further comprises: compressing and encoding the background by a video encoder to obtain a third encoding result; The third encoding result is transmitted to a decoding end.
11. The method according to claim 1, characterized in that: The neural radiance field and encoder-decoder are trained by: The initial neural radiation field is trained by training samples to obtain the intermediate neural radiation field; Determine a reconstruction loss function value based on the sample label of the training sample and the prediction result of the intermediate neural radiation field on the training sample; wherein the reconstruction loss function is the L2 norm between the rendering result output by the decoding end to be trained and the ground truth value; The initial state of the codec is set based on the reconstruction loss function value, and the codec and the intermediate neural radiation field are trained from the initial state to obtain the trained neural radiation field and the trained codec; wherein the codec includes: an encoder based on a neural network in the encoding end to be trained and a decoder based on a neural network in the decoding end to be trained.
12. The method according to claim 11, characterized in that The step of training the codec and the intermediate neural radiation field from the initial state to obtain a trained neural radiation field and a trained codec comprises: Determine a target loss function; wherein the target loss function is determined based on the reconstruction loss function value and an entropy loss function, and the entropy loss function is calculated based on the input of the second entropy encoder and the input of the first entropy encoder in the encoder to be trained; Starting from the initial state, the codec and the intermediate neural radiation field are repeatedly iteratively trained using the target loss function to obtain a trained neural radiation field and a trained codec.
13. A digital human processing method based on neural network, characterized in that: Applied to the decoding end, including: Obtain a first encoding result and a second encoding result sent by an encoding end; wherein the first encoding result is obtained by the encoding end performing compression encoding on a compact representation of the 3D digital human, the compact representation is obtained by performing feature transformation on a three-dimensional feature grid of the neural radiation field of the 3D digital human, the three-dimensional feature grid is generated by representing the 3D digital human, the compact representation at least includes: a three-dimensional feature grid of the neural radiation field or decomposed features of the three-dimensional feature grid, the second encoding result is obtained by the encoding end performing compression encoding on external parameters and network model parameters obtained after performing feature transformation on the three-dimensional feature grid, the external parameters at least include: camera parameters and driving parameters of a 3D model of the 3D digital human; Decoding the first encoding result through a decoder based on a neural network, and decoding the second encoding result to obtain a first decoding result and a second decoding result respectively; Reconstruct and render according to the first decoding result and the second decoding result to obtain the rendering viewpoint and / or 3D model of the 3D digital human, and display the rendering viewpoint and / or 3D model through the terminal side.
14. The method according to claim 13, characterized in that The first encoding result is decoded by a decoder based on a neural network to obtain a first decoding result, including: Decoding the output result of the first encoding module in the first encoding result by a first entropy decoder to obtain a first entropy decoding result, and decoding the first entropy decoding result by a super a priori decoder to obtain a super a priori decoding result; Based on the super a priori decoding result, construct an entropy model of a second entropy decoder, and decode the output result of the second encoding module in the first encoding result by the second entropy decoder based on the entropy model to obtain a second entropy decoding result; The second entropy decoding result is decoded by a representation sub-decoder to obtain the first decoding result.
15. The method according to claim 13, characterized in that The method further comprises: After obtaining the third encoding result transmitted by the encoding end, the third encoding result is decoded by a video decoder to obtain a third decoding result; wherein the third encoding result is obtained by encoding the background obtained by the encoding end after semantic analysis of 3D data based on the neural radiation field.
16. The method according to claim 15, characterized in that The reconstructing and rendering according to the first decoding result and the second decoding result to obtain the rendering viewpoint and / or the 3D model of the 3D digital human includes: Reconstruction and rendering are performed according to the first decoding result, the second decoding result and the third decoding result to obtain the rendering viewpoint and / or the 3D model of the 3D digital human.
17. The method according to claim 13, characterized in that The decoding process of the second encoding result to obtain a second decoding result includes: The second encoding result is decoded by a Metadata decoder to obtain a second decoding result.
18. A digital human processing device based on a neural network, characterized in that: Set on the encoding end, including: A semantic analysis module, used for performing semantic analysis on the input 3D data based on the neural radiation field to obtain a 3D digital human and background; wherein the semantic analysis may adopt 2D and / or 3D object detection, or 2D and / or 3D semantic segmentation; A representation generation module, used to generate a representation of the 3D digital human to obtain a three-dimensional feature grid of the neural radiation field of the 3D digital human; A feature transformation module, used for performing feature transformation on the three-dimensional feature grid to obtain a compact representation of the 3D digital human; wherein the compact representation at least includes: the three-dimensional feature grid of the neural radiation field or the decomposed features of the three-dimensional feature grid; The compression coding module is used to compress and encode the compact representation through an encoder based on a neural network to obtain a first coding result.
19. A digital human processing device based on a neural network, characterized in that: Set at the decoding end, including: An acquisition module is used to acquire a first encoding result and a second encoding result sent by an encoding end; wherein, the first encoding result is obtained by the encoding end performing compression encoding on a compact representation of the 3D digital human, the compact representation is obtained by performing feature transformation on a three-dimensional feature grid of the neural radiation field of the 3D digital human, the three-dimensional feature grid is generated by representing the 3D digital human, the compact representation at least includes: a three-dimensional feature grid of the neural radiation field or decomposed features of the three-dimensional feature grid, the second encoding result is obtained by the encoding end performing compression encoding on external parameters and network model parameters obtained after performing feature transformation on the three-dimensional feature grid, the external parameters at least include: camera parameters and driving parameters of a 3D model of the 3D digital human; A decoding module, used to decode the first encoding result through a decoder based on a neural network, and to decode the second encoding result, to obtain a first decoding result and a second decoding result respectively; A reconstruction and rendering module is used to perform reconstruction and rendering according to the first decoding result and the second decoding result to obtain the rendering viewpoint and / or 3D model of the 3D digital human, and display the rendering viewpoint and / or 3D model through the terminal side.
20. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of digital human processing based on a neural network as described in any one of claims 1 to 17 are performed.
21. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of digital human processing based on a neural network as claimed in any one of claims 1 to 17.
22. A computer program product, characterized in that The computer program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of digital human processing based on a neural network as claimed in any one of claims 1 to 18.
Citation Information
Cited By
Six-plane feature representation enhanced super-resolution dynamic nerve radiation field method
CN119624775A
3D digital human and scene processing method and apparatus, and device, medium and product
WO2026145145A1
