Digital human generation and streaming transmission method and system

Abstract

The invention relates to the technical field of image generation, and particularly provides a digital human generation and streaming transmission method and system, and the method and system are based on an ER-NERF algorithm, and comprise the following steps: S1, carrying out the depth estimation and optical radiation field reconstruction of multi-view image data, so as to generate a realistic 2.5 D digital human model; s2, in a data transmission stage, the system pushes the preprocessed model parameters and texture information to a user terminal in a streaming media format in real time; and S3, at a user terminal, the system performs real-time analysis and reconstruction on the received data stream through a decoding module, and dynamic rendering and interactive display of the digital human are completed. Compared with the prior art, the method has the advantages that the terminal computing pressure can be reduced, cross-platform real-time interaction and efficient rendering are ensured, and the application requirements of virtual content production, remote interaction and immersive experience are met.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and specifically provides a digital human generation and streaming transmission method and system. Background Art

[0002] A digital human refers to a virtual character generated based on computer vision and graphics technology, which is widely used in fields such as entertainment, education, and social networking. Its core technologies include 3D reconstruction, expression capture, bone binding, and rendering optimization. Traditional methods rely on multi-camera arrays and high-performance computing devices, with high costs and limited real-time performance. In recent years, the introduction of deep learning models such as Neural Radiance Fields (NeRF) has enabled the reconstruction of high-precision digital human images from a small amount of perspective data, significantly reducing hardware dependence and computational complexity, and laying a solid foundation for real-time applications and streaming media distribution.

[0003] How to solve the problems of high cost, long latency, and hardware dependence in the traditional digital human generation process in the prior art remains a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0004] In view of the above deficiencies of the prior art, the present invention provides a digital human generation and streaming transmission method with strong practicability.

[0005] A further technical task of the present invention is to provide a digital human generation and streaming transmission system with reasonable design, safety, and applicability.

[0006] The technical solution adopted by the present invention to solve its technical problems is as follows:

[0007] A digital human generation and streaming transmission method, based on the ER-NERF algorithm, has the following steps:

[0008] S1. Perform depth estimation and light radiation field reconstruction on multi-view image data to generate a realistic 2.5D digital human model;

[0009] S2. In the data transmission stage, the system pushes the preprocessed model parameters and texture information to the user terminal in real time in a streaming media format;

[0010] S3. At the user terminal, the system performs real-time parsing and reconstruction on the received data stream through a decoding module to complete the dynamic rendering and interactive display of the digital human.

[0011] Further, in step S1, combining deep learning and graphics rendering technologies, the model is trained and encoded in the cloud to generate a digital human image with fine surface texture and realistic light and shadow effects, and at the same time, the model parameters are light-weight compressed for storage and transmission.

[0012] Furthermore, in step S2, a dynamic adaptive transmission mechanism is adopted to adjust the data distribution strategy in real time according to the network bandwidth and the performance of the terminal device. Meanwhile, through incremental transmission optimization, the response speed and resource utilization rate of the system are improved.

[0013] Furthermore, in step S3, during the model reconstruction process, the graphics processing unit (GPU) is combined for accelerated rendering and texture mapping to perform realistic facial expression and body movement performance.

[0014] A digital human generation and streaming transmission system. First, depth estimation and light field reconstruction are performed on multi-view image data to generate a realistic 2.5D digital human model. Then, in the data transmission stage, the system pushes the preprocessed model parameters and texture information to the user terminal in real time in a streaming media format. Finally, at the user terminal, the system performs real-time parsing and reconstruction on the received data stream to complete the dynamic rendering and interactive display of the digital human.

[0015] Furthermore, after generating a realistic 2.5D digital human model, combining deep learning and graphics rendering technologies, the model is trained and encoded in the cloud to generate a digital human image with fine surface texture and realistic light and shadow effects. At the same time, lightweight compression of model parameters is performed for storage and transmission.

[0016] Furthermore, after being pushed to the user terminal in real time, a dynamic adaptive transmission mechanism is adopted to adjust the data distribution strategy in real time according to the network bandwidth and the performance of the terminal device. Meanwhile, through incremental transmission optimization, the response speed and resource utilization rate of the system are improved.

[0017] Furthermore, after completing the dynamic rendering and interactive display of the digital human, during the model reconstruction process, the graphics processing unit (GPU) is combined for accelerated rendering and texture mapping to perform realistic facial expression and body movement performance.

[0018] Compared with the prior art, a digital human generation and streaming transmission method and system of the present invention has the following prominent beneficial effects:

[0019] Through the improved ER-NERF algorithm, the present invention significantly improves the accuracy and realism of digital human generation, and realizes high-fidelity 2.5D digital human reconstruction. By adopting a lightweight model encoding and incremental data transmission mechanism, the data transmission and storage costs are effectively reduced, and at the same time, the end-to-end delay of the system is significantly reduced, ensuring real-time performance and smooth interaction. Compared with traditional multi-camera arrays and high-performance computing solutions, this method has lower hardware requirements and has strong adaptability and wide application range.

[0020] In addition, through the adaptive streaming transmission strategy, the present invention can dynamically adjust the transmission rate and model rendering accuracy, adapt to different network environments and terminal devices, and improve the stability of the user experience and the service coverage. This solution is applicable to application scenarios that require high interactivity, such as distance education, virtual performances, and online social interactions, providing technical support and innovative breakthroughs for the large-scale application and commercial deployment of virtual human technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0022] FIG. Figure 1 is a schematic flowchart of a digital human generation and streaming transmission method. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] In order to enable those skilled in the art of the present technology to better understand the solution of the present invention, the following will further elaborate on the present invention in combination with specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0024] The following gives a best embodiment:

[0025] As Figure 1 shown, a digital human generation and streaming transmission method in this embodiment, based on the ER-NERF algorithm, has the following steps:

[0026] S1. Perform depth estimation and light radiation field reconstruction on multi-view image data to generate a realistic 2.5D digital human model;

[0027] By combining deep learning and graphics rendering technologies, the model is trained and encoded in the cloud to generate a digital human image with fine surface texture and realistic light and shadow effects, while performing lightweight compression of the model parameters to ensure efficient storage and transmission.

[0028] S2. In the data transmission stage, the system pushes the preprocessed model parameters and texture information to the user terminal in real time in a streaming media format;

[0029] Adopt a dynamic adaptive transmission mechanism to adjust the data distribution strategy in real time according to the network bandwidth and terminal device performance, ensuring low-latency and high-quality digital human rendering effects in complex network environments. At the same time, through incremental transmission optimization, avoid the repeated sending of redundant data, and further improve the system's response speed and resource utilization rate.

[0030] S3. On the user terminal, the system performs real-time parsing and reconstruction of the received data stream through a decoding module to complete the dynamic rendering and interactive display of the digital human.

[0031] During the model reconstruction process, combine the graphics processing unit (GPU) to accelerate rendering and texture mapping to achieve realistic facial expressions and limb movement performances, meeting the requirements of various application scenarios such as virtual meetings and distance education.

[0032] The above methods are applicable to scenarios such as AR / VR, virtual hosts, and distance education, providing high-quality and low-latency solutions for the production and distribution of digital human content.

[0033] Based on the above method, in a digital human generation and streaming transmission system in this embodiment, first, perform depth estimation and light field reconstruction on multi-view image data to generate a realistic 2.5D digital human model. Then, in the data transmission stage, the system pushes the preprocessed model parameters and texture information to the user terminal in real time in a streaming media format. Finally, on the user terminal, the system performs real-time parsing and reconstruction of the received data stream through a decoding module to complete the dynamic rendering and interactive display of the digital human.

[0034] After generating a realistic 2.5D digital human model, combine deep learning and graphics rendering technologies. The model is trained and encoded in the cloud to generate a digital human image with fine surface texture and realistic lighting effects. At the same time, perform lightweight compression of the model parameters for storage and transmission.

[0035] After pushing to the user terminal in real time, adopt a dynamic adaptive transmission mechanism to adjust the data distribution strategy in real time according to the network bandwidth and terminal device performance. At the same time, through incremental transmission optimization, improve the system's response speed and resource utilization rate.

[0036] After completing the dynamic rendering and interactive display of the digital human, during the model reconstruction process, combine the graphics processing unit GPU to accelerate rendering and texture mapping for realistic facial expressions and limb movement performances.

[0037] The improved ER-NERF algorithm significantly improves the accuracy and realism of digital human generation, and achieves high-fidelity 2.5D digital human reconstruction. The lightweight model encoding and incremental data transmission mechanism effectively reduce the data transmission and storage costs, while significantly reducing the end-to-end delay of the system, ensuring real-time performance and interactive fluency. Compared with traditional multi-camera arrays and high-performance computing solutions, this method has lower hardware requirements and is more adaptable and widely applicable.

[0038] Through the adaptive streaming media transmission strategy, the present invention can dynamically adjust the transmission rate and model rendering accuracy, adapt to different network environments and terminal devices, and improve the stability of user experience and service coverage. This solution is suitable for application scenarios that require high interactivity, such as distance education, virtual performances, and online social networking, and provides technical support and innovative breakthroughs for the large-scale application and commercial deployment of virtual human technology.

[0039] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0040] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating and streaming digital lives, characterized in that, Based on the ER-NERF algorithm, it has the following steps: S1. Perform depth estimation and light field reconstruction on multi-view image data to generate a realistic 2.5D digital human model; S2. In the data transmission stage, the system pushes the preprocessed model parameters and texture information to the user terminal in real time in the form of a streaming media; S3. At the user terminal, the system performs real-time parsing and reconstruction on the received data stream through a decoding module to complete the dynamic rendering and interactive display of the digital human.

2. The digital life generation and streaming method according to claim 1, wherein In step S1, by combining deep learning and graphics rendering technologies, the model is trained and encoded in the cloud to generate a digital human image with fine surface texture and realistic lighting effects. At the same time, lightweight compression of the model parameters is performed for storage and transmission.

3. A method for digital life generation and streaming transmission according to claim 1, characterized in that In step S2, a dynamic adaptive transmission mechanism is adopted to adjust the data distribution strategy in real time according to the network bandwidth and terminal device performance. At the same time, through incremental transmission optimization, the response speed and resource utilization rate of the system are improved.

4. A method for digital life generation and streaming transmission according to claim 1, characterized in that, In step S3, during the model reconstruction process, GPU acceleration rendering and texture mapping are combined to perform realistic facial expressions and limb movement performances.

5. A digital life generation and streaming system, characterized in that, First, perform depth estimation and light field reconstruction on multi-view image data to generate a realistic 2.5D digital human model. Then, in the data transmission stage, the system pushes the preprocessed model parameters and texture information to the user terminal in real time in the form of a streaming media. Finally, at the user terminal, the system performs real-time parsing and reconstruction on the received data stream through a decoding module to complete the dynamic rendering and interactive display of the digital human.

6. The digital human generation and streaming system according to claim 5, wherein After generating a realistic 2.5D digital human model, by combining deep learning and graphics rendering technologies, the model is trained and encoded in the cloud to generate a digital human image with fine surface texture and realistic lighting effects. At the same time, lightweight compression of the model parameters is performed for storage and transmission.

7. A digital life generation and streaming system according to claim 6, characterized in that, After being pushed to the user terminal in real time, a dynamic adaptive transmission mechanism is adopted to adjust the data distribution strategy in real time according to the network bandwidth and terminal device performance. At the same time, through incremental transmission optimization, the response speed and resource utilization rate of the system are improved.

8. A digital life generation and streaming system according to claim 7, wherein, After completing the dynamic rendering and interactive display of the digital human, during the model reconstruction process, GPU acceleration rendering and texture mapping are combined to perform realistic facial expressions and limb movement performances.

Images