Human body prior-based digital human global neural rendering method and system

Abstract

The application discloses a human body prior-based digital human global neural rendering method and system, which comprises the following steps: step one, acquiring human body posture samples and digital human asset data, and constructing a sample library and an asset set; step two, establishing a cross-skeleton joint correspondence relationship, and generating a unified joint mapping table; step three, constructing a parameterized human body model, and extracting features to generate a human body prior vertex vocabulary set; step four, encoding and mapping posture parameters and cross-attention calculating with prior vocabularies to obtain a human body structure vocabulary set; step five, encoding camera perspective geometry buffer data, inputting a direct light network to obtain a direct light component; step six, encoding light source perspective panoramic buffer data, inputting an indirect light network to obtain an indirect light component; step seven, fusing the direct light component and the indirect light component; and step eight, synthesizing and processing to obtain a digital human skin area rendering result. The application realizes realistic rendering of a digital human under complex illumination and dynamic postures.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and artificial intelligence, and in particular to a method and system for global neural mapping of digital humans based on human priors. Background Technology

[0002] With the widespread application of digital human technology in virtual reality, film and television production and human-computer interaction, neural network-based digital human rendering methods have gradually become a research hotspot. In particular, in terms of realistic skin rendering and complex lighting modeling, existing technologies mostly use neural rendering methods based on neural radiation fields or images for modeling, but they still have obvious limitations in scenarios involving changes in human body structure and the complexity of lighting transmission.

[0003] Existing technologies typically rely on geometric buffer information and lighting features from the camera's perspective to directly regress pixels, lacking explicit modeling of prior information about human structure. This can easily lead to deviations in structural consistency in rendering results when human posture changes or the digital human's form alters. Furthermore, traditional methods often rely on local pixel features for lighting calculations, making it difficult to effectively establish the correlation between human structure and lighting distribution, resulting in unstable skin detail rendering under complex posture conditions. Additionally, for modeling indirect lighting, most methods do not incorporate panoramic light propagation information, failing to accurately characterize multiple scattering and global lighting distribution characteristics, leading to deficiencies in lighting continuity and realism in rendering results. Moreover, existing methods often fail to uniformly handle joint correspondences between different skeletal structures, making it difficult to achieve a unified expression of posture information in multi-form digital human scenes, thus limiting the generalization ability and applicability of the methods.

[0004] Therefore, how to provide a method and system for global neural mapping of digital humans based on human priors is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a global neural rendering method and system for digital humans based on human priors. This invention achieves explicit modeling and consistent cross-skeleton expression of human structure by constructing human prior vertex lexicons and unified joint mappings. It also enhances structural stability under dynamic poses by generating structural lexicons through pose encoding. In lighting modeling, human structure is introduced to participate in direct and indirect lighting decoding, and combined with panoramic observation information and pixel query mechanisms, the correlation modeling of lighting and structure is achieved. By fusing direct and indirect lighting and performing synthesis processing, the lighting continuity and realism of the digital human skin region are improved.

[0006] The global neural mapping method for digital humans based on prior knowledge of the human body according to embodiments of the present invention includes the following steps: Step 1: Obtain and process human posture sample data to build a human posture sample library; Step 2: Acquire multi-form digital human asset data and process it uniformly to construct a multi-form digital human asset collection; Step 3: Based on the aforementioned multi-form digital human asset set, establish cross-skeleton joint correspondence, perform mapping processing on the joints, and generate a unified joint mapping table; Step 4: Construct a parameterized human body model based on the multi-form digital human asset set, extract vertex positions, vertex normals and subsurface scattering parameters under standard pose, and generate a set of prior vertex lexical terms for the human body; Step 5: Obtain the human pose parameters of the current frame, encode the human pose parameters and map them based on the unified joint mapping table, perform cross-attention calculation with the human prior vertex lexical set and perform weighted processing in combination with skinning weights to obtain the human structure lexical set; Step 6: Acquire camera view geometry buffer data and encode it to form a pixel query term set. Input the pixel query term set, the human body structure term set, and the light source features into the direct light decoding network to obtain the direct light component. Step 7: Acquire panoramic buffer data of the light source perspective and encode it to form a panoramic observation term set. Input the panoramic observation term set and the human body structure term set into the indirect light decoding network to obtain the indirect light component. Step 8: The direct light component and the indirect light component are fused and synthesized to obtain the rendering result of the digital human skin region.

[0007] Optionally, step one includes: Step 1: Obtain and process human posture sample data to build a human posture sample library; Step 2: Acquire multi-form digital human asset data and process it uniformly to construct a multi-form digital human asset collection; Step 3: Based on the aforementioned multi-form digital human asset set, establish cross-skeleton joint correspondence, perform mapping processing on the joints, and generate a unified joint mapping table; Step 4: Construct a parameterized human body model based on the multi-form digital human asset set, extract vertex positions, vertex normals and subsurface scattering parameters under standard pose, and generate a set of prior vertex lexical terms for the human body; Step 5: Obtain the human pose parameters of the current frame, encode the human pose parameters and map them based on the unified joint mapping table, perform cross-attention calculation with the human prior vertex lexical set and perform weighted processing in combination with skinning weights to obtain the human structure lexical set; Step 6: Acquire camera view geometry buffer data and encode it to form a pixel query term set. Input the pixel query term set, the human body structure term set, and the light source features into the direct light decoding network to obtain the direct light component. Step 7: Acquire panoramic buffer data of the light source perspective and encode it to form a panoramic observation term set. Input the panoramic observation term set and the human body structure term set into the indirect light decoding network to obtain the indirect light component. Step 8: The direct light component and the indirect light component are fused and synthesized to obtain the rendering result of the digital human skin region.

[0008] Optionally, step two includes: Acquire multi-form digital human asset data, which includes human body mesh data, skeletal structure data, and material parameter data; The human body mesh data is subjected to topology verification processing, and meshes that do not meet the preset topology structure are reconstructed. The skeletal structure data is adjusted at the joint level, and the number of joints is standardized. The material parameter data is then subjected to numerical normalization. The processed human body mesh data, skeletal structure data, and material parameter data are integrated to construct a collection of digital human assets in multiple forms.

[0009] Optionally, step three includes: Based on the aforementioned multi-form digital human asset set, joint sets are extracted from different skeletal structures; Name matching and spatial position correspondence analysis were performed on joints in different skeletal structures to establish joint correspondences. The target joint set is determined according to the preset standard skeleton structure, and the correspondence of the joints is rearranged. Based on the rearranged joint correspondence, the joints in each skeleton structure are mapped to obtain a unified joint representation. The unified joint representation is stored to generate a unified joint mapping table.

[0010] Optionally, step four includes: A parameterized human body model is constructed based on the multi-form digital human asset set. The parameterized human body model includes a human body mesh structure and a skeletal driving relationship corresponding to the human body mesh structure. Convert the parametric human model to a standard pose; Traverse all vertices of the parameterized human body model and extract the three-dimensional spatial coordinates, vertex normal vector, and corresponding subsurface scattering parameters of each vertex. The three-dimensional spatial coordinates, vertex normal vector, and subsurface scattering parameters of each vertex are combined to generate vertex words; All vertex lexies are aggregated to generate a human prior vertex lexygen set.

[0011] Optionally, step five includes: Obtain the human body posture parameters of the current frame, including the spatial pose parameters of each joint; The human posture parameters are encoded to generate a set of joint lexical terms; Based on the unified joint mapping table, the set of joint lexical units is mapped to obtain a unified joint representation; Perform cross-attention calculation between the unified joint representation and the human prior vertex lexicon set; Based on the skinning weights from vertices to joints in the parameterized human model, the cross-attention calculation results are weighted. The weighted results are aggregated to generate a set of human body structure lexical terms.

[0012] Optionally, step six includes: Acquire camera view geometry buffer data, which includes pixel position, pixel normal and depth information; The camera view geometry buffer data is encoded to generate a set of pixel query terms; Obtain the light source features and then vectorize the light source features; The pixel query term set, the human body structure term set, and the light source features are input into the direct optical decoding network. In the direct optical decoding network, cross-attention calculation is performed on the pixel query term set and the human body structure term set, and feature fusion processing is performed in combination with the light source features; The fused result is decoded to obtain the direct optical component.

[0013] Optionally, step seven includes: Acquire panoramic buffer data from the light source perspective, which includes radiation information in the panoramic direction and corresponding spatial distribution information. The panoramic buffer data of the light source perspective is sampled and processed, and the panoramic space is divided according to a preset resolution to obtain multiple observation units; The radiation information and spatial distribution information in each observation unit are encoded to generate corresponding panoramic observation lexical units, forming a panoramic observation lexical set. The panoramic observation lexical set and the human body structure lexical set are input into the indirect optical decoding network; In the indirect optical decoding network, self-attention calculation is performed on the panoramic observation lexical set, and the self-attention calculation result is cross-attention calculation with the human body structure lexical set; The cross-attention calculation results are decoded to obtain the indirect optical component.

[0014] Optionally, step eight includes: Obtain the direct light component and the indirect light component; The direct light component and the indirect light component are spatially aligned so that they correspond in the same pixel coordinate system. The aligned direct light component and indirect light component are fused pixel by pixel to obtain the fused illumination component. Obtain human body surface property parameters, including surface reflection properties and subsurface scattering related parameters; Based on the fused illumination components and the human body surface attribute parameters, a pixel-level skin rendering result is obtained; The skin rendering results of all pixels are combined and processed to obtain the rendering result of the digital human skin area.

[0015] According to an embodiment of the present invention, a digital human global neural mapping system based on human prior knowledge includes: The human posture sample library construction module is used to acquire and process human posture sample data to build a human posture sample library. The multi-form digital human asset construction module is used to acquire multi-form digital human asset data and process it in a unified manner to construct a multi-form digital human asset collection. The cross-skeleton joint mapping module is used to establish cross-skeleton joint correspondences based on the multi-form digital human asset set, perform joint mapping processing, and generate a unified joint mapping table. The human body prior construction module is used to construct a parameterized human body model based on the multi-morphological digital human asset set, extract vertex positions, vertex normals and subsurface scattering parameters under standard pose, and generate a human body prior vertex lexicon set. The pose-aware human structure encoding module is used to obtain human pose parameters in the current frame, encode the human pose parameters and map them based on the unified joint mapping table, perform cross-attention calculation with the human prior vertex lexicon set and perform weighted processing in combination with skinning weights to obtain the human structure lexicon set. The direct light decoding module is used to acquire camera view geometry buffer data and encode it to form a set of pixel query terms. The set of pixel query terms, the set of human body structure terms and light source features are input into the direct light decoding network to obtain direct light components. The indirect light decoding module is used to acquire panoramic buffer data of the light source perspective and encode it to form a panoramic observation word set. The panoramic observation word set and the human body structure word set are input into the indirect light decoding network to obtain the indirect light component. The result synthesis module is used to fuse and synthesize the direct light component and the indirect light component to obtain the rendering result of the digital human skin region.

[0016] The beneficial effects of this invention are: This invention addresses the problems of missing human structural information, difficulty in uniformly representing different skeletal structures, and insufficient structural consistency under pose changes in existing digital human rendering methods by constructing a parameterized human body model and a set of human body prior vertices. Combined with pose parameter encoding and cross-skeleton joint mapping mechanisms, it proposes a structural encoding strategy based on explicit human structure modeling and unified joint representation, significantly improving the structural consistency and expression stability of digital humans of different forms under dynamic pose conditions. In the lighting modeling stage, human structure lexics are introduced into the direct light decoding process. Through cross-attention calculation between pixel query lexics and human structure lexics, pixel-level lighting information and human structure lexics are integrated. The correlation modeling of body structure information effectively enhances the structural perception ability of the illumination distribution in the skin area. In the indirect light modeling stage, panoramic buffer data from the light source perspective is introduced to construct panoramic observation terms. Through self-attention calculation and cross-attention calculation of human body structure terms, the coupling relationship between multi-directional light propagation information and human body structure is established, improving the expressive ability of multiple scattering and global illumination. In the result synthesis stage, by fusing the direct light component and the indirect light component and combining them with human body surface attributes, the rendering result contains both direct and indirect light contributions, thereby achieving a digital human skin area rendering effect with structural consistency and illumination continuity. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the structure of the global neural mapping system for digital humans based on human priors proposed in this invention; Figure 2 This is a schematic diagram of the global neural mapping method for digital humans based on human priors proposed in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figures 1-2 The global neural mapping method for digital humans based on prior knowledge of the human body includes the following steps: Step 1: Obtain and process human posture sample data to build a human posture sample library; Step 2: Acquire multi-form digital human asset data and process it uniformly to construct a multi-form digital human asset collection; Step 3: Establish cross-skeleton joint correspondence based on the multi-form digital human asset set, perform mapping processing on the joints, and generate a unified joint mapping table; Step 4: Construct a parameterized human body model based on a multi-form digital human asset set, extract vertex positions, vertex normals and subsurface scattering parameters under standard pose, and generate a set of prior vertex lexical terms for the human body. Step 5: Obtain the human pose parameters of the current frame, encode the human pose parameters and map them based on a unified joint mapping table, perform cross-attention calculation with the human prior vertex lexicon set and perform weighted processing in combination with skinning weights to obtain the human structure lexicon set; Step 6: Acquire camera view geometry buffer data and encode it to form a pixel query term set. Input the pixel query term set, human body structure term set and light source features into the direct light decoding network to obtain the direct light component. Step 7: Acquire panoramic buffer data of the light source perspective and encode it to form a panoramic observation term set. Input the panoramic observation term set and the human body structure term set into the indirect light decoding network to obtain the indirect light component. Step 8: The direct light component and the indirect light component are fused and synthesized to obtain the rendering result of the digital human skin region.

[0020] In this embodiment, step one includes: Acquire human posture sample data, which includes the spatial pose parameters of each joint of the human body. The human posture sample data is standardized to convert human posture data from different sources into joint representations under a unified skeletal structure. The standardized human posture data is filtered to remove abnormal posture data; The filtered human posture data are classified and organized, and grouped according to the type of human movement; The grouped human posture data are stored and processed to construct a human posture sample library.

[0021] In this implementation, the human posture sample data includes the three-dimensional spatial pose parameters of each joint of the human body and the corresponding time series information. The acquired human posture sample data is preprocessed, and skeleton structure alignment processing is performed on human posture data from different sources to uniformly map various skeleton structures to a preset standard skeleton structure, thereby obtaining unified joint representation data. The unified joint representation data is then cleaned to remove data with missing joints, abnormal joint coordinates, and interrupted posture continuity. The cleaned human posture data is then classified and organized according to action categories to form multi-category posture data sets. Each category of posture data set is stored to establish a human posture sample library for subsequent posture parameter encoding and cross-skeleton joint mapping processes.

[0022] In this embodiment, step two includes: Acquire multi-form digital human asset data, which includes human body mesh data, skeletal structure data, and material parameter data; Perform topology verification on the human body mesh data, and reconstruct meshes that do not meet the preset topology. The skeletal structure data is adjusted at the joint level, and the number of joints is standardized. Perform numerical normalization on the material parameter data; The processed human body mesh data, skeletal structure data, and material parameter data are integrated to construct a collection of digital human assets in multiple forms.

[0023] In this implementation, the multi-form digital human asset data includes human mesh data, skeletal structure data, and material parameter data of different body types and resolutions. Topological structure analysis is performed on the human mesh data to identify vertex connections and facet structures. Meshes with inconsistent topologies are reconstructed to ensure consistent vertex connections across digital human models. Joint-level parsing is performed on the skeletal structure data, rearranging joints according to a preset skeleton hierarchy and standardizing the number of joints to ensure consistency in joint quantity and hierarchy across skeletal structures. Numerical range adjustment is performed on the material parameter data, and normalization calculations are performed on material parameters from different sources. The processed human mesh data, skeletal structure data, and material parameter data are then integrated to form a multi-form digital human asset set.

[0024] In this embodiment, step three includes: Based on a collection of multi-form digital human assets, joint sets are extracted from different skeletal structures; Name matching and spatial position correspondence analysis were performed on joints in different skeletal structures to establish joint correspondences. The target joint set is determined according to the preset standard skeleton structure, and the joint correspondence is rearranged. Based on the rearranged joint correspondence, the joints in each skeleton structure are mapped to obtain a unified joint representation. The unified joint representation is stored to generate a unified joint mapping table.

[0025] In this implementation, based on a multi-form digital human asset set, skeletal structure data corresponding to each digital human model is extracted. Joint sets and joint hierarchy relationships are then parsed from the skeletal structure data. Name matching is performed on joint sets in different skeletal structures, establishing initial correspondences for joints with consistent names. For joints with inconsistent names, a corresponding analysis is performed based on their positional relationship within the skeletal hierarchy and their three-dimensional spatial coordinates. A target joint set is determined according to a preset standard skeletal structure. The established joint correspondences are rearranged to ensure the joint order in each skeletal structure matches the target joint set. Based on the rearranged joint correspondences, a mapping process is performed on the joints in each skeletal structure, converting the joint representations of different skeletal structures into a unified joint representation. The unified joint representations are stored, generating a unified joint mapping table.

[0026] In this embodiment, step four includes: A parametric human body model is constructed based on a collection of multi-form digital human assets. The parametric human body model includes a human body mesh structure and the skeletal driving relationship corresponding to the human body mesh structure. Convert the parametric human model to a standard pose; Traverse all vertices of the parameterized human body model and extract the three-dimensional spatial coordinates, vertex normal vector, and corresponding subsurface scattering parameters of each vertex. The three-dimensional spatial coordinates, vertex normal vector, and subsurface scattering parameters of each vertex are combined to generate vertex words; All vertex lexies are aggregated to generate a human prior vertex lexygen set.

[0027] In this implementation, a parametric human model is constructed based on a multi-morphological digital human asset set. The parametric human model includes a human mesh structure and the corresponding skeletal driving relationship. The parametric human model is adjusted to a preset standard pose, and all vertices in the human mesh structure are traversed. For each vertex, the corresponding three-dimensional spatial coordinates and vertex normal vector are extracted, and the subsurface scattering parameters associated with that vertex are read from the multi-morphological digital human asset set. The three-dimensional spatial coordinates, vertex normal vectors, and subsurface scattering parameters are combined to form the feature representation of the corresponding vertex. The feature representations of all vertices are aggregated to obtain a set of prior human vertex terms, which are used for subsequent pose-aware human structure encoding.

[0028] In this embodiment, step five includes: Obtain the human pose parameters of the current frame, which include the spatial pose parameters of each joint. Human posture parameters are encoded to generate a set of joint lexical terms; A unified joint representation is obtained by mapping the set of joint lexical units based on a unified joint mapping table. Cross-attention computation is performed between the unified joint representation and the human prior vertex lexicon set; Based on the skinning weights from vertices to joints in the parametric human model, the cross-attention calculation results are weighted. The weighted results are aggregated to generate a set of human body structure lexical terms.

[0029] In this implementation, human posture parameters include rotation and translation parameters of each joint. The human posture parameters are vectorized and encoded, converting the posture parameters of each joint into corresponding joint feature vectors to form a joint terminology set. Based on a unified joint mapping table, the joint feature vectors in the joint terminology set are indexed and rearranged to obtain a unified joint representation. The unified joint representation and the human prior vertex terminology set are input into a cross-attention calculation process, using the human prior vertex terms as query terms and joint features as key terms. Based on the pre-stored skinning weights from vertices to joints in the parameterized human model, the attention calculation results are weighted vertex-by-vertex. The weighted results are then aggregated to obtain a human structure terminology set, which is used in the subsequent lighting decoding process.

[0030] In this embodiment, step six includes: Acquire camera view geometry buffer data, which includes pixel position, pixel normal and depth information; Encode the camera view geometry buffer data to generate a set of pixel query terms; Obtain the light source features and represent them in a vectorized form; Input the pixel query term set, the human body structure term set, and the light source features into the direct optical decoding network; In the direct optical decoding network, cross-attention calculation is performed on the pixel query term set and the human body structure term set, and feature fusion processing is performed in combination with light source features; The fused result is decoded to obtain the direct optical component.

[0031] In this implementation, the camera viewpoint geometric buffer data includes pixel position, pixel normal, and depth information. The camera viewpoint geometric buffer data is encoded, converting the geometric attributes of each pixel into corresponding pixel query terms, forming a set of pixel query terms. Light source features are acquired and vectorized. The set of pixel query terms, the set of human structure terms, and the light source features are input into a direct light decoding network. In the direct light decoding network, cross-attention calculation is performed using pixel query terms as query items and human structure terms as key items, combined with feature fusion processing based on light source features to obtain an intermediate feature representation. The intermediate feature representation is then decoded to obtain the direct light component. This implementation introduces human structure terms into the cross-attention calculation during the direct light decoding process, establishing a correlation between pixel-level illumination calculation and human structure information, thereby improving the structural consistency of the illumination results for the digital human skin region.

[0032] In this embodiment, step seven includes: Acquire panoramic buffer data from the light source's perspective. This data includes radiation information along the panoramic direction and the corresponding spatial distribution information. The panoramic buffer data from the light source perspective is sampled and processed, and the panoramic space is divided according to a preset resolution to obtain multiple observation units; The radiation information and spatial distribution information in each observation unit are encoded to generate corresponding panoramic observation lexical units, forming a panoramic observation lexical set. Input the panoramic observation lexical set and the human body structure lexical set into the indirect optical decoding network; In the indirect optical decoding network, self-attention calculation is performed on the panoramic observation lexical set, and the self-attention calculation result is cross-attention calculation with the human body structure lexical set. The cross-attention calculation results are decoded to obtain the indirect optical component.

[0033] In this embodiment, the panoramic buffer data from the light source perspective includes radiation information covering the panoramic direction and corresponding spatial distribution information. The panoramic buffer data is divided according to a preset angular resolution to form multiple observation areas. The radiation information and spatial distribution information of each observation area are encoded to generate corresponding panoramic observation terms, forming a panoramic observation term set. The panoramic observation term set is input into an indirect optical decoding network. Self-attention calculation is performed on the panoramic observation term set in the indirect optical decoding network, and the self-attention calculation result is cross-attention calculated with the human body structure term set to obtain an intermediate feature representation. The intermediate feature representation is then decoded to obtain the indirect light component. This embodiment, by introducing panoramic observation terms and performing self-attention and cross-attention calculations during the indirect optical decoding process, couples multi-directional light propagation information with human body structure information, thereby improving the spatial consistency of the indirect light component.

[0034] In this embodiment, step eight includes: Obtain the direct and indirect light components; Spatial alignment is performed on the direct light component and the indirect light component so that they correspond in the same pixel coordinate system; The aligned direct light component and indirect light component are fused pixel by pixel to obtain the fused illumination component. Obtain human body surface property parameters, including surface reflection properties and subsurface scattering related parameters; Pixel-level skin rendering results are obtained by merging the illumination components and human surface attribute parameters. The skin rendering results of all pixels are combined and processed to obtain the rendering result of the digital human skin area.

[0035] In this implementation, pixel-level alignment is performed on the direct and indirect light components to ensure they correspond in the same image coordinates. The aligned direct and indirect light components are then fused pixel-by-pixel to obtain a fused illumination component. Human surface attribute parameters, including surface reflection and subsurface scattering parameters, are acquired. Pixel-by-pixel compositing is performed based on the fused illumination component and the human surface attribute parameters to obtain the rendering result of the skin region. The rendering results of each pixel are then combined to obtain the rendering result of the digital human skin region. This implementation improves the continuity and realism of the lighting in the digital human skin region by fusing the direct and indirect light components at the pixel level and combining them with compositing processing based on human surface attribute parameters, ensuring that the lighting result includes both direct light contributions and multiple scattering contributions.

[0036] A global neural mapping system for digital humans based on prior knowledge includes: The human posture sample library construction module is used to acquire and process human posture sample data to build a human posture sample library. The multi-form digital human asset construction module is used to acquire multi-form digital human asset data and process it in a unified manner to construct a multi-form digital human asset collection. The cross-skeleton joint mapping module is used to establish cross-skeleton joint correspondences based on a collection of multi-form digital human assets, perform joint mapping processing, and generate a unified joint mapping table. The Human Prior Construction Module is used to construct a parameterized human model based on a multi-morphological digital human asset set, extracting vertex positions, vertex normals, and subsurface scattering parameters under standard poses, and generating a human prior vertex lexicon set. The pose-aware human structure encoding module is used to obtain the human pose parameters of the current frame, encode the human pose parameters and map them based on a unified joint mapping table, perform cross-attention calculation with the human prior vertex word set and perform weighted processing in combination with skin weights to obtain the human structure word set. The direct light decoding module is used to acquire camera view geometry buffer data and encode it to form a set of pixel query terms. The set of pixel query terms, human body structure terms and light source features are input into the direct light decoding network to obtain the direct light components. The indirect light decoding module is used to acquire panoramic buffer data of the light source perspective and encode it to form a panoramic observation word set. The panoramic observation word set and the human body structure word set are input into the indirect light decoding network to obtain the indirect light component. The result synthesis module is used to fuse and synthesize the direct and indirect light components to obtain the rendering result of the digital human skin region.

[0037] Example 1: To verify the feasibility of this invention in practice, it was applied to a virtual digital human film production scenario. In this scenario, the digital human needs to perform facial and body actions under different lighting environments and complex motion conditions, such as continuous motion performance in indoor multi-light source environments, outdoor natural light environments, and dynamic light source change scenarios. Traditional methods in such scenarios typically rely on image-based neural rendering or local illumination regression. When the human posture changes significantly or the lighting direction changes, problems such as discontinuous lighting in skin areas, blurred details, and structural distortion easily occur, especially in the shoulder, neck, and joint areas. This invention addresses the above problems by introducing a priori human structure and global illumination modeling mechanisms to achieve stable, high-quality rendering effects under dynamic posture and complex lighting conditions.

[0038] In practical applications, multiple sets of real human motion data are first collected, covering various movement forms such as walking, head turning, and arm raising, and a human posture sample library is constructed based on the collected data. Simultaneously, digital human models of various body types are acquired, including digital human assets with different heights, body shapes, and facial features. Their mesh structures, skeletal structures, and material parameters are uniformly processed to construct a multi-form digital human asset set. Based on this, correspondences are established for joints of different skeletal structures, forming a unified joint mapping table, enabling the expression of posture parameters between different digital humans within a unified framework.

[0039] A parametric human model is constructed based on multi-morphological digital human assets. Under standard pose conditions, the spatial position, normal direction, and subsurface scattering parameters of each vertex are extracted to generate a priori vertex lexicon set. During actual rendering, the human pose parameters of the current frame are acquired, encoded, and mapped to a unified joint representation. Cross-attention calculation is then performed with the priori vertex lexicon set, and weighted by skinning weights to obtain a human structure lexicon set. This process allows the human structure information to dynamically adjust with pose changes, thereby maintaining structural consistency.

[0040] In the illumination calculation stage, geometric buffer data from the camera's perspective is acquired and encoded into a set of pixel query terms. Simultaneously, light source features are incorporated and input into a direct light decoding network along with human structure terms to obtain the direct light component. Conversely, panoramic buffer data from the light source's perspective is acquired and encoded into a set of panoramic observation terms. This set, along with human structure terms, is input into an indirect light decoding network to obtain the indirect light component. Finally, the direct and indirect light components are fused and combined with human surface attributes for synthesis, resulting in the final rendering of the digital human skin region.

[0041] In summary, this invention can effectively solve the problems of insufficient human body structure representation, incomplete lighting modeling, and weak multi-pose generalization ability in existing technologies in practical applications. It can achieve stable structure and realistic lighting in digital human rendering in complex scenes, and has good engineering application value.

[0042] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for global neural mapping of digital humans based on prior knowledge of the human body, characterized in that, Includes the following steps: Step 1: Obtain and process human posture sample data to build a human posture sample library; Step 2: Acquire multi-form digital human asset data and process it uniformly to construct a multi-form digital human asset collection; Step 3: Based on the aforementioned multi-form digital human asset set, establish cross-skeleton joint correspondence, perform mapping processing on the joints, and generate a unified joint mapping table; Step 4: Construct a parameterized human body model based on the multi-form digital human asset set, extract vertex positions, vertex normals and subsurface scattering parameters under standard pose, and generate a set of prior vertex lexical terms for the human body; Step 5: Obtain the human pose parameters of the current frame, encode the human pose parameters and map them based on the unified joint mapping table, perform cross-attention calculation with the human prior vertex lexical set and perform weighted processing in combination with skinning weights to obtain the human structure lexical set; Step 6: Acquire camera view geometry buffer data and encode it to form a pixel query term set. Input the pixel query term set, the human body structure term set, and the light source features into the direct light decoding network to obtain the direct light component. Step 7: Acquire panoramic buffer data of the light source perspective and encode it to form a panoramic observation term set. Input the panoramic observation term set and the human body structure term set into the indirect light decoding network to obtain the indirect light component. Step 8: The direct light component and the indirect light component are fused and synthesized to obtain the rendering result of the digital human skin region.

2. The digital human global neural mapping method based on human priors as described in claim 1, characterized in that, Step one includes: Acquire human posture sample data, which includes spatial pose parameters of each joint of the human body; The human posture sample data is standardized to convert human posture data from different sources into joint representations under a unified skeletal structure. The standardized human posture data is filtered to remove abnormal posture data; The filtered human posture data are classified and organized, and grouped according to the type of human movement; The grouped human posture data are stored and processed to construct a human posture sample library.

3. The global neural mapping method for digital humans based on human priors as described in claim 1, characterized in that, Step two includes: Acquire multi-form digital human asset data, which includes human body mesh data, skeletal structure data, and material parameter data; The human body mesh data is subjected to topology verification processing, and meshes that do not meet the preset topology structure are reconstructed. The skeletal structure data is adjusted at the joint level, and the number of joints is standardized. The material parameter data is then subjected to numerical normalization. The processed human body mesh data, skeletal structure data, and material parameter data are integrated to construct a collection of digital human assets in multiple forms.

4. The global neural mapping method for digital humans based on human priors as described in claim 1, characterized in that, Step three includes: Based on the aforementioned multi-form digital human asset set, joint sets are extracted from different skeletal structures; Name matching and spatial position correspondence analysis were performed on joints in different skeletal structures to establish joint correspondences. The target joint set is determined according to the preset standard skeleton structure, and the correspondence of the joints is rearranged. Based on the rearranged joint correspondence, the joints in each skeleton structure are mapped to obtain a unified joint representation. The unified joint representation is stored to generate a unified joint mapping table.

5. The digital human global neural mapping method based on human priors according to claim 1, characterized in that, Step four includes: A parameterized human body model is constructed based on the multi-form digital human asset set. The parameterized human body model includes a human body mesh structure and a skeletal driving relationship corresponding to the human body mesh structure. Convert the parametric human model to a standard pose; Traverse all vertices of the parameterized human body model and extract the three-dimensional spatial coordinates, vertex normal vector, and corresponding subsurface scattering parameters of each vertex. The three-dimensional spatial coordinates, vertex normal vector, and subsurface scattering parameters of each vertex are combined to generate vertex words; All vertex lexies are aggregated to generate a human prior vertex lexygen set.

6. The global neural mapping method for digital humans based on human priors according to claim 1, characterized in that, Step five includes: Obtain the human body posture parameters of the current frame, including the spatial pose parameters of each joint; The human posture parameters are encoded to generate a set of joint lexical terms; Based on the unified joint mapping table, the set of joint lexical units is mapped to obtain a unified joint representation; Perform cross-attention calculation between the unified joint representation and the human prior vertex lexicon set; Based on the skinning weights from vertices to joints in the parameterized human model, the cross-attention calculation results are weighted. The weighted results are aggregated to generate a set of human body structure lexical terms.

7. The global neural mapping method for digital humans based on human priors according to claim 1, characterized in that, Step six includes: Acquire camera view geometry buffer data, which includes pixel position, pixel normal and depth information; The camera view geometry buffer data is encoded to generate a set of pixel query terms; Obtain the light source features and then vectorize the light source features; The pixel query term set, the human body structure term set, and the light source features are input into the direct optical decoding network. In the direct optical decoding network, cross-attention calculation is performed on the pixel query term set and the human body structure term set, and feature fusion processing is performed in combination with the light source features; The fused result is decoded to obtain the direct optical component.

8. The digital human global neural mapping method based on human priors according to claim 1, characterized in that, Step seven includes: Acquire panoramic buffer data from the light source perspective, which includes radiation information in the panoramic direction and corresponding spatial distribution information. The panoramic buffer data of the light source perspective is sampled and processed, and the panoramic space is divided according to a preset resolution to obtain multiple observation units; The radiation information and spatial distribution information in each observation unit are encoded to generate corresponding panoramic observation lexical units, forming a panoramic observation lexical set. The panoramic observation lexical set and the human body structure lexical set are input into the indirect optical decoding network; In the indirect optical decoding network, self-attention calculation is performed on the panoramic observation lexical set, and the self-attention calculation result is cross-attention calculation with the human body structure lexical set; The cross-attention calculation results are decoded to obtain the indirect optical component.

9. The digital human global neural mapping method based on human priors according to claim 1, characterized in that, Step eight includes: Obtain the direct light component and the indirect light component; The direct light component and the indirect light component are spatially aligned so that they correspond in the same pixel coordinate system. The aligned direct light component and indirect light component are fused pixel by pixel to obtain the fused illumination component. Obtain human body surface property parameters, including surface reflection properties and subsurface scattering related parameters; Based on the fused illumination components and the human body surface attribute parameters, a pixel-level skin rendering result is obtained; The skin rendering results of all pixels are combined to obtain the rendering result of the digital human skin area.

10. A global neural mapping system for digital humans based on human priors, executing the global neural mapping method for digital humans based on human priors as described in any one of claims 1 to 9, characterized in that, include: The human posture sample library construction module is used to acquire and process human posture sample data to build a human posture sample library. The multi-form digital human asset construction module is used to acquire multi-form digital human asset data and process it in a unified manner to construct a multi-form digital human asset collection. The cross-skeleton joint mapping module is used to establish cross-skeleton joint correspondences based on the multi-form digital human asset set, perform joint mapping processing, and generate a unified joint mapping table. The human body prior construction module is used to construct a parameterized human body model based on the multi-morphological digital human asset set, extract vertex positions, vertex normals and subsurface scattering parameters under standard pose, and generate a human body prior vertex lexicon set. The pose-aware human structure encoding module is used to obtain human pose parameters in the current frame, encode the human pose parameters and map them based on the unified joint mapping table, perform cross-attention calculation with the human prior vertex lexicon set and perform weighted processing in combination with skinning weights to obtain the human structure lexicon set. The direct light decoding module is used to acquire camera view geometry buffer data and encode it to form a set of pixel query terms. The set of pixel query terms, the set of human body structure terms and light source features are input into the direct light decoding network to obtain direct light components. The indirect light decoding module is used to acquire panoramic buffer data of the light source perspective and encode it to form a panoramic observation word set. The panoramic observation word set and the human body structure word set are input into the indirect light decoding network to obtain the indirect light component. The result synthesis module is used to fuse and synthesize the direct light component and the indirect light component to obtain the rendering result of the digital human skin region.

Images