Real-time rendering methods, devices and computer equipment for digital humans

Abstract

This disclosure provides a method, apparatus, and computer device for real-time rendering of digital humans, relating to the field of image processing technology, particularly real-time rendering and digital human rendering. The implementation scheme includes: collecting multiple real-time computing power parameters from a terminal; normalizing these parameters into a computing power index vector; determining whether the terminal's computing power change exceeds a preset threshold based on the computing power index vector; when the terminal's computing power change exceeds the preset threshold, inputting the computing power index vector into a trained reinforcement learning model to determine the target rendering mode for the digital human; and switching the digital human's rendering mode from the current rendering mode to the target rendering mode. This provides a generalized rendering scheme that can evaluate available computing power in real-time and dynamically adjust the digital human rendering level to meet the rendering requirements of high frame rates, low latency, and scalability.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to the fields of real-time rendering of digital humans, computer graphics, A / B motion-driven systems and heterogeneous computing power scheduling, and especially to a method, apparatus, computer device, computer-readable storage medium and computer program product for real-time rendering of digital humans. Background Technology

[0002] With the widespread application of digital human technology in scenarios such as virtual live streaming, virtual customer service, interactive AI assistants, and immersive experiences, its rendering engine needs to meet increasingly stringent performance requirements, mainly including: high frame rate (usually 30-60 FPS), low latency (the entire process from input to output needs to be completed within tens of milliseconds), and good scalability, so as to be able to adapt to multiple terminal platforms such as mobile, PC, cloud, and XR.

[0003] However, existing rendering solutions still have several key problems in digital human rendering applications, which restrict their efficient deployment and stable operation in multi-terminal and dynamic load scenarios: First, there are significant differences in computing power among different terminals; second, the real-time available computing power of the system is subject to millisecond-level interference from various factors such as background tasks, operating system scheduling, concurrent AI inference tasks, and thermal throttling, resulting in significant fluctuations; third, existing rendering solutions usually rely on developers to manually pre-configure different image quality presets, lacking a mechanism to automatically adjust rendering complexity and output quality based on real-time computing power.

[0004] Therefore, there is an urgent need for a general-purpose rendering solution that can assess available computing power in real time and dynamically adjust the rendering level of digital humans to meet the rendering requirements of high frame rate, low latency, and scalability. Summary of the Invention

[0005] This disclosure provides a method, apparatus, computer device, computer-readable storage medium, and computer program product for real-time rendering of digital humans.

[0006] According to one aspect of this disclosure, a real-time rendering method for a digital human is provided. The method includes: collecting multiple real-time computing power parameters of a terminal, wherein the terminal includes a mobile terminal, a personal computer terminal, a cloud terminal, or an extended reality device terminal; normalizing the multiple real-time computing power parameters into a computing power index vector; determining whether the computing power change of the terminal exceeds a preset change threshold based on the computing power index vector; when the computing power change of the terminal exceeds the preset change threshold, inputting the computing power index vector into a trained reinforcement learning model to determine the target rendering mode of the digital human; wherein the reinforcement learning model is trained by: defining the state space of the reinforcement learning model, the state space including multiple computing power index state vectors corresponding to the computing power states of multiple terminals; defining the action space of the reinforcement learning model, the action space including at least two rendering modes; defining the reward function of the reinforcement learning model, the reward function being used to evaluate actions based on rendering quality scores; initializing the policy network of the reinforcement learning model; optimizing the policy network of the reinforcement learning model based on the reward function to obtain the trained reinforcement learning model; and switching the rendering mode of the digital human from the current rendering mode to the target rendering mode.

[0007] In some embodiments, the method further includes determining the acquisition time interval for acquiring multiple real-time computing power parameters according to the type of the terminal, and determining whether the computing power change of the terminal exceeds a preset change threshold based on the computing power index vector, including: acquiring a first computing power index vector at a first time; acquiring a second computing power index vector at a second time, wherein the second time is separated from the first time by at least one acquisition time interval; and determining whether the computing power change of the terminal exceeds the preset change threshold based on a comparison between the difference between the first computing power index vector and the second computing power index vector and the preset change threshold.

[0008] In some embodiments, the method further includes: calculating a comprehensive computing power score based on multiple real-time computing power parameters; and inputting the computing power index vector and the comprehensive computing power score into a trained reinforcement learning model to determine the target rendering mode of the digital human.

[0009] In some embodiments, the reward function is configured to determine a rendering quality score based on frame rate stability, rendering latency, and rendering visual quality, and the reward function is configured to reward actions with high rendering quality scores and / or penalize actions with low rendering quality scores.

[0010] In some embodiments, the target rendering mode includes the following four rendering modes: a first rendering mode for implementing basic driven rendering; a second rendering mode for implementing rendering of the first rendering mode as well as facial expression and local texture rendering; a third rendering mode for implementing rendering of the second rendering mode as well as secondary action rendering; and a fourth rendering mode for implementing rendering of the third rendering mode as well as high-precision appearance rendering.

[0011] In some embodiments, switching the rendering mode of the digital human from the current rendering mode to the target rendering mode includes: determining the switching time required to switch from the current rendering mode to the target rendering mode; inserting a preset animation frame during the switching time; or inserting multiple transition frames during the switching time, wherein the current rendering parameters of the current rendering mode and the target rendering parameters of the target rendering mode are interpolated to obtain the rendering parameters of each frame of the multiple transition frames.

[0012] According to another aspect of this disclosure, a real-time rendering apparatus for a digital human is provided. The apparatus includes: a parameter acquisition unit configured to acquire multiple real-time computing power parameters of a terminal, wherein the terminal includes a mobile terminal, a personal computer terminal, a cloud terminal, or an extended reality device terminal; a vector generation unit configured to normalize the multiple real-time computing power parameters into a computing power index vector; a computing power determination unit configured to determine whether the computing power change of the terminal exceeds a preset change threshold based on the computing power index vector; and a rendering mode determination unit configured to input the computing power index vector into a trained reinforcement learning model when the computing power change of the terminal exceeds the preset change threshold, so as to determine the rendering mode of the digital human. The target rendering mode is defined as follows: A reinforcement learning model is trained by: defining the state space of the reinforcement learning model, which includes multiple computing power index state vectors corresponding to multiple terminal computing power states; defining the action space of the reinforcement learning model, which includes at least two rendering modes; defining the reward function of the reinforcement learning model, which is used to evaluate actions based on rendering quality scores; initializing the policy network of the reinforcement learning model; optimizing the policy network of the reinforcement learning model based on the reward function to obtain the trained reinforcement learning model; and a rendering mode switching unit configured to switch the rendering mode of the digital human from the current rendering mode to the target rendering mode.

[0013] According to another aspect of this disclosure, a computer device is provided, comprising: at least one processor; and a memory storing a computer program thereon, wherein the computer program, when executed by the at least one processor, causes the at least one processor to perform the digital human real-time rendering method as described above.

[0014] According to another aspect of this disclosure, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, causes the processor to perform the digital human real-time rendering method as described above.

[0015] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, causes the processor to perform the digital human real-time rendering method as described above.

[0016] According to one or more embodiments of this disclosure, real-time monitoring and evaluation of computing power changes can be performed during the rendering process. Based on the evaluation results, an appropriate rendering mode can be determined, and the rendering mode can be automatically switched according to the determined rendering mode to ensure a smooth transition. This enables stable frame rate, low latency output, and visual consistency of the rendered digital human on various devices, even when there are large differences in computing power on different terminals or when real-time computing power fluctuates greatly.

[0017] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0018] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of this disclosure. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0019] Figure 1 This is a flowchart illustrating a real-time rendering method for digital humans according to an exemplary embodiment.

[0020] Figure 2 This is a schematic diagram illustrating the process of training a reinforcement learning model according to an exemplary embodiment.

[0021] Figure 3 This is a schematic diagram illustrating a computing power evaluation process according to an exemplary embodiment.

[0022] Figure 4 This is a schematic diagram illustrating the rendering mode switching process according to an exemplary embodiment.

[0023] Figure 5 This is a schematic block diagram illustrating a real-time rendering apparatus for a digital human according to an exemplary embodiment.

[0024] Figure 6 This is a block diagram illustrating an exemplary computer device that can be applied to an exemplary embodiment. Detailed Implementation

[0025] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0027] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0028] In related technologies, real-time rendering of digital humans is often based on traditional rendering schemes using skeletal animation and rendering pipelines, or neural rendering schemes (such as NeRF, NRG, neural textures, etc.). However, regardless of the rendering scheme used, the following technical problems are faced: First, the application scenarios of digital humans require them to seamlessly switch between multiple terminals such as Web, APP, mini-program, and smart hardware; however, the computing power of each type of terminal varies greatly. For example, the CPU and GPU performance of mobile terminals is relatively low and easily affected by temperature; extended reality (XR) devices are easily limited by power consumption; and although personal computers (PCs) and cloud terminals have high computing power, they may run multiple AI models in parallel, resulting in drastic fluctuations in computing power depending on the task. Second, the real-time available computing power is highly dynamic. Background tasks (such as other user processes or system services running concurrently with the digital human application in the operating system), system scheduling, AI inference, and operating system hot throttling can all affect the available computing power in milliseconds. Finally, rendering parameters rely on manual configuration and cannot be adaptively adjusted according to computing power during the rendering process.

[0029] Due to the aforementioned issues, existing rendering solutions struggle to maintain stable frame rates, low latency output, and visual consistency across various terminal devices.

[0030] To address this, embodiments of this disclosure provide a real-time rendering method for digital humans that can solve the problems of large differences in computing power between different terminals and strong fluctuations in real-time computing power. By evaluating computing power resources in real time, the method determines and automatically switches rendering modes based on the computing power evaluation results, thereby enabling digital humans to smoothly switch between different rendering modes when terminal computing power changes, and maintaining a consistent visual experience across terminal environments.

[0031] Figure 1 This is a flowchart illustrating a real-time rendering method for digital humans according to an exemplary embodiment.

[0032] like Figure 1 As shown, the real-time rendering method for digital humans includes steps S110, S120, S130, S140, and S150.

[0033] In step S110, multiple real-time computing power parameters of the terminal are collected, wherein the terminal includes a mobile terminal, a personal computer terminal, a cloud terminal, or an extended reality device terminal.

[0034] In the examples, the terminal can be the operating terminal for various digital human application scenarios. The terminal can be a mobile terminal such as a smartphone or tablet. The main computing power constraints faced by mobile terminals are the limited computing power caused by low-to-mid-range GPUs, limited memory bandwidth, battery power, susceptibility to overheating triggering thermal throttling, and numerous background tasks interfering with the system. The terminal can also be an extended reality device terminal such as AR glasses or VR headsets. The main computing power constraint faced by extended reality device terminals is their extremely high power consumption. The terminal can also be a personal computer terminal such as a desktop or laptop computer. Its main computing power constraints are the wide range of computing power due to hardware configuration and the large dynamic changes in system load. The terminal can also be a cloud terminal, which possesses powerful computing power, but the dynamically shared computing power environment makes its real-time available computing power unstable.

[0035] It can be seen that different types of terminals are subject to different computing power constraints. In order to assess the computing power status of various terminals, it is first necessary to collect multiple real-time computing power parameters.

[0036] In the example, multiple real-time computing power parameters can include CPU utilization or utilization trend, GPU utilization, video memory usage, rendering time, AI model inference time, memory bandwidth, heat dissipation and power consumption status, etc. CPU utilization can be the average CPU utilization, which is the weighted average of the non-idle time of all CPU cores within a sampling window. CPU utilization can also be described as CPU utilization variance, indicating the fluctuation range of system CPU utilization within the same sampling window. CPU utilization trend refers to the direction and slope of CPU utilization change between at least two sampling windows. GPU utilization can be the percentage of time the 3D graphics rendering engine in the GPU spends executing core graphics tasks such as shaders and rasterization. Video memory usage can be the amount of data used by the current rendering task in the GPU's dedicated memory (VRAM). Rendering time can be the total time (in milliseconds) required to complete all rendering work for one frame. AI model inference time can be the time spent by the AI ​​model necessary to drive the digital human (such as voice-driven facial expressions, vision-driven, text generation, etc.) to perform a forward computation on the terminal. Memory bandwidth refers to the rate at which data is transferred between the CPU, GPU, and between them. Heat dissipation and power consumption status can be comprehensively evaluated using factors such as battery percentage, whether the device is connected to a power source, and the overall heat dissipation status. By selecting multiple dimensions of real-time computing power parameters, the real-time computing power status of various types of terminals can be covered.

[0037] In step S120, multiple real-time computing power parameters are normalized into a computing power index vector. As mentioned above, the selected real-time computing power parameters of multiple dimensions have different dimensions and vastly different numerical ranges. If decisions are made directly based on the collected raw parameters, the computing power evaluation logic would be extremely complex and difficult to maintain. Therefore, normalization maps each parameter to a standard range of [0, 1], where 0 represents "computing power is extremely abundant / no pressure" and 1 represents "computing power is extremely strained / reached its limit." This unifies all key computing power parameters into dimensionless values ​​of the same scale, allowing them to be compared, weighted, and calculated in the same mathematical space. Furthermore, the absolute performance values ​​of different terminals vary drastically, and the normalization process requires calibration of each terminal to obtain the average computing power and / or the calibration value of the computing power ceiling for each terminal. Thus, the real-time computing power parameters collected during normalization are all normalized based on the calibration value of the terminal, making the computing power evaluation method applicable to various types of terminals.

[0038] In step S130, it is determined whether the change in the computing power of the terminal exceeds a preset change threshold based on the computing power index vector.

[0039] In some embodiments, the time interval for collecting the plurality of real-time computing power parameters is first determined. For example, depending on the type of terminal, the time interval can be selected as any fixed value within the range of 10-50 ms.

[0040] Based on this data collection time interval, the change in computing power can be determined by comparing two computing power index vectors that are separated by at least one data collection time interval.

[0041] Figure 3 This is a schematic diagram illustrating a computing power evaluation process according to an exemplary embodiment. Figure 3 As shown, step S130 further includes steps S310, S320 and S330.

[0042] In step S310, the first moment can be the moment before the digital human real-time rendering task begins, the start moment, or a moment after a certain interval following the start.

[0043] In step S320, the second time point is at least one data acquisition time interval from the first time point. Through steps S310 and S320, it is possible to obtain computing power index vectors at least two different time points, thereby enabling the acquisition of real-time trends in computing power.

[0044] In step S330, based on the comparison between the difference between the first computing power index vector and the second computing power index vector and the preset change threshold, it is determined whether the change in the computing power of the terminal exceeds the preset change threshold.

[0045] To better illustrate the computing power assessment process, the following will detail a specific implementation of a real-time rendering computing power assessment for a digital human.

[0046] In this specific embodiment, the computing power index vector A is a computing power index vector [a1, a2, a3, a4] including four elements: average CPU utilization, GPU utilization, rendering time, and heat dissipation and power consumption status. At the first moment 10ms after the start of the real-time rendering task, the first computing power index vector A1 = [0.2, 0.5, 0.4, 0.3]. At the second moment 10ms after the first moment (i.e., after one sampling time interval), the second computing power index vector A2 = [0.5, 0.7, 0.6, 0.8]. The preset change threshold can be a threshold vector B = [0.1, 0.2, 0.2, 0.2]. The comparison between the difference between the first and second computing power index vectors and the preset change threshold can be determined by judging whether the difference between any element of the first and second computing power index vectors exceeds the value of the element corresponding to the threshold vector B. If the difference of any element exceeds the value of the element corresponding to the threshold vector B, it is determined that the computing power change exceeds the preset threshold. Alternatively, if the difference between each element exceeds the value of the element corresponding to the threshold vector B, then the computing power change is determined to exceed the preset threshold. Or, if the magnitude of the difference vector between the first and second computing power index vectors is greater than the magnitude of the threshold vector B, then the computing power change is determined to exceed the preset threshold.

[0047] Continue to refer to Figure 1 Based on the evaluation results of computing power, in step S140, when the change in the computing power of the terminal exceeds the preset change threshold, the computing power index vector is input into the trained reinforcement learning model to determine the target rendering mode of the digital human.

[0048] In some embodiments, the target rendering mode includes the following four rendering modes: a first rendering mode for implementing basic driven rendering; a second rendering mode for implementing rendering of the first rendering mode as well as facial expression and local texture rendering; a third rendering mode for implementing rendering of the second rendering mode as well as secondary action rendering; and a fourth rendering mode for implementing rendering of the third rendering mode as well as high-precision appearance rendering.

[0049] In the example, the four rendering modes are divided according to rendering complexity. The first rendering mode includes rendering of skeletons and basic lighting; the second rendering mode further includes rendering of facial expressions / local textures; the third rendering mode further includes rendering of secondary actions, hair / cloth, and micro-expressions; and the fourth rendering mode further includes rendering of high-precision appearance, global illumination, and physically simulated rendering.

[0050] In traditional skeletal animation + rendering pipeline solutions, since various rendering parameters such as resolution, shadows, and textures can be adjusted separately, the hierarchical division of target rendering modes can directly correspond to different combinations of rendering parameters. In the example, the rendering parameters covered by the first to fourth rendering modes increase sequentially. However, for neural rendering solutions, different target rendering modes or rendering levels correspond to different model architectures or computational precisions. For example, the first rendering mode might be a lightweight neural network with fewer layers and narrower feature channels, or its output image might have a lower feature network resolution, or it might have lower computational precision.

[0051] Figure 2 This is a schematic diagram illustrating the process of training a reinforcement learning model according to an exemplary embodiment. Figure 2 As shown, the process of training a reinforcement learning model may include steps S210, S220, S230, S240, and S250.

[0052] In step S210, the state space of the reinforcement learning model is defined, which includes multiple computing power index state vectors A corresponding to the computing power states of multiple terminals.

[0053] In the example, the state space S can be the set of all possible discrete computing power indicator state vectors. For the computing power indicator vectors mentioned above, which include four elements: average CPU utilization, GPU utilization, rendering time, and heat dissipation and power consumption, multiple intervals can be divided for each element. For example, dividing each element into 4 intervals can yield 256 discrete computing power indicator state vectors.

[0054] In step S220, the action space of the reinforcement learning model is defined, and the action space includes at least two rendering modes.

[0055] In the example, rendering modes can be divided into at least two levels by considering factors such as rendering complexity and rendering quality. This allows for the use of a higher-complexity, higher-quality rendering mode when system computing power is sufficient, and a lower-complexity, lower-quality rendering mode when system computing power is limited, in order to maintain the frame rate. The rendering modes in the action space can correspond to a set of target rendering modes.

[0056] In step S230, a reward function for the reinforcement learning model is defined, which is used to evaluate actions based on the rendering quality score.

[0057] In the example, the rendering quality score is a score that includes frame rate stability, rendering latency, and rendering visual quality. This reward function helps the reinforcement learning model select a rendering mode that maintains high visual quality while ensuring smoothness in a dynamic computing environment.

[0058] Based on this rendering quality score, the reward function is configured to score actions according to their rendering quality score. The reward function can be configured to reward actions that improve rendering visual quality or achieve high image quality. It can also be configured to penalize actions that cause low frame rate stability, high latency, or stuttering.

[0059] In steps S240 and S250, the policy network of the reinforcement learning model is initialized; the policy network is then optimized based on the reward function to obtain the trained reinforcement learning model. Specifically, during training, the agent can sample and select actions for the rendering mode by observing the computational power index state vector and sampling according to the probability distribution output by the policy network. After executing the action, the system monitors indicators such as rendering quality, latency, and frame rate stability, and calculates the immediate reward based on the reward function. This experience, including state, action, reward, and new state, is stored in the experience replay buffer. During training, the agent can periodically sample batch data from the experience replay buffer for learning, thereby gradually causing the policy network to tend to select action sequences that can obtain higher cumulative rewards. Finally, the trained reinforcement learning model is obtained.

[0060] It should be understood that when the terminal's computing power changes do not exceed the preset change threshold, there is no need to recalculate a new rendering mode to adapt to the dynamic changes in computing power. Instead, the previous collection time interval or the preset rendering mode is maintained.

[0061] In some embodiments, the real-time rendering method for digital humans further includes: calculating a comprehensive computing power score based on multiple real-time computing power parameters; and inputting the computing power index vector and the comprehensive computing power score into a trained reinforcement learning model to determine the target rendering mode of the digital human.

[0062] In the example, the comprehensive computing power score can be formed using linear weights, decision trees, or small neural networks. By pre-compiling a comprehensive computing power score, a priori knowledge input regarding computing power assessment is provided to the reinforcement learning model, which helps the agent focus its attention on valuable state-action associations, significantly shortening convergence time and improving policy stability. Furthermore, although the computing power index vector has been normalized, its high dimensionality and potential redundancy and noise mean that directly using it as state input may lead to overfitting the model to parameter distributions specific to a particular terminal or scenario, limiting its cross-terminal generalization ability. The comprehensive computing power score, on the other hand, forms a more universal and abstract feature.

[0063] Continue to refer to Figure 1 In step S150, the rendering mode of the digital human is switched from the current rendering mode to the target rendering mode.

[0064] Figure 4This is a schematic diagram illustrating the rendering mode switching process according to an exemplary embodiment. Figure 4 As shown, step S150 further includes steps S410 and S420.

[0065] In step S410, the switching time required to switch from the current rendering mode to the target rendering mode is determined. Specifically, this switching time can be a pre-set fixed value, such as 10ms, 30ms, or other values. The switching time can also be determined based on the difference in rendering complexity between the current rendering mode and the target rendering mode.

[0066] In step S420, a preset animation frame is inserted during the switching time; or multiple transition frames are inserted during the switching time, wherein the current rendering parameters of the current rendering mode and the target rendering parameters of the target rendering mode are interpolated to obtain the rendering parameters of each frame of the multiple transition frames.

[0067] To eliminate screen flickering and abrupt transitions caused by layer switching and provide users with a better visual experience, smooth transitions can be achieved by inserting preset animation frames or multiple transition frames during the switching time. Preset animation frames can be pre-rendered offline animation clips, such as a clip of a digital human waving, which can be inserted directly during the switching time without real-time rendering, thus preventing stuttering. Transition frames, on the other hand, can be calculated by interpolating various rendering parameters (such as graphics attribute parameters, lighting parameters, etc.) of the current and target rendering modes, allowing the animation to transition smoothly through multiple transition frames.

[0068] The real-time rendering method for digital humans in this embodiment of the disclosure senses changes in computing power through cross-terminal computing power index vectors, and divides the rendering into at least two rendering modes based on rendering complexity, thereby enabling the method to autonomously select the rendering mode under different computing power conditions. Throughout the entire digital human rendering task, the method can continuously monitor changes in computing power and adjust the rendering mode according to the acquisition time interval, thereby allowing the digital human image to maintain high image quality and low latency at all times.

[0069] The real-time rendering method for digital humans according to the embodiments of this disclosure has the following beneficial effects:

[0070] 1. Strong cross-terminal applicability. The embodiments of this disclosure propose a generalized method for describing computing power resource indicators. Whether it is a mobile terminal, extended reality device, desktop workstation or cloud GPU, computing power can be evaluated through a unified computing power indicator vector, realizing the adaptation of a single set of rendering logic to multiple terminals and reducing development and maintenance costs.

[0071] 2. Dynamic Adaptive Rendering. This method can assess the available computing power in real time and dynamically select the most suitable rendering mode based on changes in computing power. This maintains a stable frame rate even in scenarios with fluctuating computing power, reduces stuttering or frame drops during the rendering of digital humans, and improves the user experience.

[0072] 3. Optimize resource utilization and energy consumption. By adjusting the rendering mode under different computing power conditions to adjust the rendering complexity, the power consumption and heat generation of terminals with weak computing power can be reduced, while making full use of high computing power equipment to render high-precision images, thus achieving the optimal balance between computing power and energy consumption.

[0073] 4. Visual continuity and smoothness. The rendering mode switching adopts a smooth transition mechanism (interpolation, lighting gradation, smooth animation state) to avoid screen flickering or abrupt changes, ensuring visual continuity and a natural interactive experience for digital humans.

[0074] 5. Highly scalable and upgradable in the future. The rendering modes and decision logic can be flexibly expanded. For example, more complex rendering modes, AI models, or special effects modules can be added without affecting the existing adaptive framework, enabling long-term upgrades and functional expansion.

[0075] 6. Improve system robustness. Under different terminal environments, computing power fluctuations, or sudden loads, the system can automatically adjust the rendering strategy, reducing the need for human intervention and improving the overall robustness and stability of real-time rendering of digital humans.

[0076] Embodiments of this disclosure also provide a real-time rendering apparatus for digital humans.

[0077] Figure 5 This is a schematic block diagram illustrating a real-time rendering apparatus for a digital human according to an exemplary embodiment.

[0078] like Figure 5 As shown, in some embodiments, the digital human real-time rendering device 500 includes a parameter acquisition unit 510, a vector generation unit 520, a computing power determination unit 530, a rendering mode determination unit 540, and a rendering mode switching unit 550.

[0079] The parameter acquisition unit 510 is configured to acquire multiple real-time computing power parameters of the terminal, wherein the terminal includes a mobile terminal, a personal computer terminal, a cloud terminal, or an extended reality device terminal.

[0080] The vector generation unit 520 is configured to normalize multiple real-time computing power parameters into a computing power index vector.

[0081] The computing power determination unit 530 is configured to determine whether the change in the computing power of the terminal exceeds a preset change threshold based on the computing power index vector.

[0082] The rendering mode determination unit 540 is configured to input a computing power index vector into a trained reinforcement learning model when the computing power change of the terminal exceeds a preset change threshold, in order to determine the target rendering mode of the digital human. The reinforcement learning model is trained as follows: defining the state space of the reinforcement learning model, which includes multiple computing power index state vectors corresponding to multiple terminal computing power states; defining the action space of the reinforcement learning model, which includes at least two rendering modes; defining the reward function of the reinforcement learning model, which is used to evaluate actions based on rendering quality scores; initializing the policy network of the reinforcement learning model; and optimizing the policy network of the reinforcement learning model based on the reward function to obtain the trained reinforcement learning model.

[0083] The rendering mode switching unit 550 is configured to switch the rendering mode of the digital human from the current rendering mode to the target rendering mode.

[0084] While specific functions have been discussed above with reference to specific units, it should be noted that the functions of the units discussed herein may be divided into multiple units, and / or at least some functions of multiple units may be combined into a single unit. The specific unit performing an action discussed herein includes the specific unit itself performing the action, or alternatively, the specific unit calling or otherwise accessing another component or unit that performs the action (or performs the action in conjunction with the specific unit). Therefore, a specific unit performing an action may include the specific unit performing the action itself and / or another unit that performs the action, called or otherwise accessed by the specific unit.

[0085] It should also be understood that the above regarding Figure 5 The described units can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuit. For example, these units can be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes one or more components in a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0086] According to one aspect of this disclosure, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0087] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0088] According to one aspect of this disclosure, a computer program product is also provided, which includes a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0089] In the following text, combined with Figure 6 Illustrative examples describing such computer devices, non-transitory computer-readable storage media, and computer program products.

[0090] Figure 6 An example configuration of a computer device 600 that can be used to implement the methods described herein is shown.

[0091] Computer device 600 can be a variety of different types of devices. Examples of computer device 600 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on.

[0092] Computer device 600 may include at least one processor 602, memory 604, multiple communication interfaces 606, display device 608, other input / output (I / O) devices 610, and one or more mass storage devices 612 capable of communicating with each other, such as via system bus 614 or other suitable connections.

[0093] Processor 602 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 602 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 602 may be configured to acquire and execute computer-readable instructions stored in memory 604, mass storage device 612, or other computer-readable media, such as program code of operating system 616, program code of application program 618, program code of other program 620, etc.

[0094] Memory 604 and mass storage device 612 are examples of computer-readable storage media for storing instructions that are executed by processor 602 to perform the various functions described above. For example, memory 604 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 612 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 604 and mass storage device 612 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 602 as a specific machine configured to perform the operations and functions described in the examples herein.

[0095] Multiple programs may be stored on mass storage device 612. These programs include operating system 616, one or more application programs 618, other programs 620, and program data 622, and they may be loaded into memory 604 for execution. Examples of such application programs or program modules may include computer program logic (e.g., computer program code or instructions) of the methods described herein and / or other embodiments described herein.

[0096] Although Figure 6 The operating system 616, one or more application programs 618, other programs 620, and program data 622 are illustrated as being stored in the memory 604 of the computer device 600. However, the operating system 616, one or more application programs 618, other programs 620, and program data 622 may also be implemented using any form of computer-readable medium accessible by the computer device 600. As used herein, "computer-readable medium" includes at least two types of computer-readable media: computer-readable storage media and communication media.

[0097] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by computer devices. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms. Computer-readable storage media as defined herein do not include communication media.

[0098] One or more communication interfaces 606 are used for exchanging data with other devices, such as via a network, direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces, near field communication (NFC) interfaces, etc. Communication interface 606 can facilitate communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 606 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0099] In some examples, a display device 608, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 610 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0100] The technologies described herein can be supported by these various configurations of computer device 600, and are not limited to specific examples of the technologies described herein. For example, the functionality can also be implemented wholly or partially on a “cloud” using a distributed system. A cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the cloud’s hardware (e.g., servers) and software resources. Resources may include applications and / or data that can be used when performing computational processing on a server remote from computer device 600. Resources may also include services provided via the Internet and / or via subscriber networks such as cellular or Wi-Fi networks. The platform can abstract resources and functionality to connect computer device 600 to other computer devices. Therefore, the implementation of the functionality described herein can be distributed throughout the cloud. For example, the functionality can be implemented partly on computer device 600 and partly through a platform that abstracts the functionality of the cloud.

Claims

1. A method for real-time rendering of digital humans, characterized in that, The method includes: The acquisition terminal collects multiple real-time computing power parameters, wherein the terminal includes a mobile terminal, a personal computer terminal, a cloud terminal, or an extended reality device terminal. The multiple real-time computing power parameters are normalized into a computing power index vector; Based on the computing power index vector, determine whether the change in the computing power of the terminal exceeds a preset change threshold; When the computing power of the terminal changes beyond the preset change threshold, the computing power index vector is input into a trained reinforcement learning model to determine the target rendering mode of the digital human; wherein the reinforcement learning model is trained in the following manner: Define the state space of the reinforcement learning model, which includes multiple computing power index state vectors corresponding to the computing power states of multiple terminals; Define the action space of the reinforcement learning model, wherein the action space includes at least two rendering modes; Define a reward function for the reinforcement learning model, wherein the reward function is used to evaluate actions based on rendering quality scores; Initialize the policy network of the reinforcement learning model; The policy network of the reinforcement learning model is optimized based on the reward function to obtain the trained reinforcement learning model; wherein, The reward function is configured to determine the rendering quality score based on frame rate stability, rendering latency, and rendering visual quality, and the reward function is configured to reward actions with high rendering quality scores and / or penalize actions with low rendering quality scores. During training, the agent observes the computing power index state vector and samples the action of the rendering mode according to the probability distribution output by the policy network. After the agent executes the action, it monitors the frame rate stability, the rendering latency, and the rendering visual quality, and calculates an immediate reward based on the reward function; and Experiences including states, actions, rewards, and new states are stored in an experience replay buffer. During training, the agent periodically samples batch data from the experience replay buffer to learn, so that the policy network gradually tends to select action sequences that can obtain higher cumulative rewards, and finally obtains the trained reinforcement learning model. Switch the rendering mode of the digital human from the current rendering mode to the target rendering mode; The method further includes determining the collection time interval for collecting the plurality of real-time computing power parameters based on the type of the terminal, and Determining whether the computing power change of the terminal exceeds a preset change threshold based on the computing power index vector includes: Obtain the first computing power index vector at the first moment; Obtain the second computing power index vector at a second time point, wherein the second time point is at least one of the acquisition time intervals from the first time point; Based on a comparison between the difference between the first computing power index vector and the second computing power index vector and the preset change threshold, it is determined whether the computing power change of the terminal exceeds the preset change threshold; and Switching the rendering mode of the digital human from the current rendering mode to the target rendering mode includes: Determine the switching time required to switch from the current rendering mode to the target rendering mode; During the switching time, multiple transition frames are inserted, wherein the current rendering parameters of the current rendering mode and the target rendering parameters of the target rendering mode are interpolated to obtain the rendering parameters of each of the multiple transition frames; and The method further includes: Calculate the comprehensive computing power score based on the aforementioned multiple real-time computing power parameters; The computing power index vector and the comprehensive computing power score are input into the trained reinforcement learning model to determine the target rendering mode of the digital human.

2. The method according to claim 1, characterized in that, The target rendering mode includes the following four rendering modes: The first rendering mode is used to implement basic driver rendering; The second rendering mode is used to implement the rendering of the first rendering mode as well as the rendering of facial expressions and local textures; The third rendering mode is used to implement the rendering of the second rendering mode and the rendering of secondary actions; The fourth rendering mode is used to implement the rendering of the third rendering mode and high-precision appearance rendering.

3. A real-time rendering device for digital humans, characterized in that, The device includes: The parameter acquisition unit is configured to acquire multiple real-time computing power parameters of the terminal, wherein the terminal includes a mobile terminal, a personal computer terminal, a cloud terminal, or an extended reality device terminal. The vector generation unit is configured to normalize the plurality of real-time computing power parameters into a computing power index vector. The computing power determination unit is configured to determine whether the change in the computing power of the terminal exceeds a preset change threshold based on the computing power index vector. A rendering mode determination unit is configured to input the computing power index vector into a trained reinforcement learning model when the computing power change of the terminal exceeds the preset change threshold, so as to determine the target rendering mode of the digital human; wherein the reinforcement learning model is trained in the following manner: Define the state space of the reinforcement learning model, which includes multiple computing power index state vectors corresponding to the computing power states of multiple terminals; Define the action space of the reinforcement learning model, wherein the action space includes at least two rendering modes; Define a reward function for the reinforcement learning model, wherein the reward function is used to evaluate actions based on rendering quality scores; Initialize the policy network of the reinforcement learning model; The policy network of the reinforcement learning model is optimized based on the reward function to obtain the trained reinforcement learning model; wherein The reward function is configured to determine the rendering quality score based on frame rate stability, rendering latency, and rendering visual quality, and the reward function is configured to reward actions with high rendering quality scores and / or penalize actions with low rendering quality scores. During training, the agent observes the computing power index state vector and samples the action of the rendering mode according to the probability distribution output by the policy network. After the agent executes the action, it monitors the frame rate stability, the rendering latency, and the rendering visual quality, and calculates an immediate reward based on the reward function; and Experiences including states, actions, rewards, and new states are stored in an experience replay buffer. During training, the agent periodically samples batch data from the experience replay buffer to learn, so that the policy network gradually tends to select action sequences that can obtain higher cumulative rewards, and finally obtains the trained reinforcement learning model. The rendering mode switching unit is configured to switch the rendering mode of the digital human from the current rendering mode to the target rendering mode; and The acquisition interval determination unit is configured to determine the acquisition time interval for acquiring the plurality of real-time computing power parameters based on the type of the terminal, and Determining whether the computing power change of the terminal exceeds a preset change threshold based on the computing power index vector includes: Obtain the first computing power index vector at the first moment; Obtain the second computing power index vector at the second time point, wherein the second time point is at least one acquisition time interval from the first time point; and wherein The rendering mode switching unit is further configured to switch the rendering mode of the digital human from the current rendering mode to the target rendering mode in the following manner: Determine the switching time required to switch from the current rendering mode to the target rendering mode; During the switching time, multiple transition frames are inserted, wherein the current rendering parameters of the current rendering mode and the target rendering parameters of the target rendering mode are interpolated to obtain the rendering parameters of each of the multiple transition frames. The rendering mode determination unit is further configured to calculate a comprehensive computing power score based on the multiple real-time computing power parameters; and input the computing power index vector and the comprehensive computing power score into the trained reinforcement learning model to determine the target rendering mode of the digital human.

4. A computer device, characterized in that, The computer device includes: At least one processor; A memory having a computer program stored thereon, wherein, when executed by the at least one processor, the computer program causes the at least one processor to perform the method of any one of claims 1-2.

5. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method according to any one of claims 1-2.

6. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1-2.

Images