A dynamic 3D video automated generation system based on runtime rendering environment

Abstract

This application discloses a dynamic 3D video automated generation system based on a runtime rendering environment, comprising: a logic control platform for decoupling 3D scene elements into dynamic modules and constructing the scene in real time in response to parameterized instructions during standalone application runtime; a video stream processing module for decoding external videos to generate dynamic media textures; an ambient lighting matching module for capturing texture visual features in real time and generating ambient lighting parameters to achieve dynamic matching of light and shadow between foreground objects and background videos; and a rendering scheduler for taking over the rendering thread during video export, freezing the game's main thread and switching to offline rendering mode, outputting high-quality image sequences and compositing the video. This system enables non-professional users to generate 3D videos with offline-level image quality and realistic lighting matching with a single click in standalone applications, significantly lowering the barrier to professional video production.

Description

Technical Field

[0001] This application belongs to the fields of computer graphics, human-computer interaction and multimedia processing technology. Specifically, it relates to a technology for automatically generating dynamic 3D video when running as an independent application outside of an integrated development environment. Background Technology

[0002] This application belongs to the fields of computer graphics, human-computer interaction and multimedia processing technology. Specifically, it relates to a technology for automatically generating dynamic 3D video when running as an independent application outside of an integrated development environment. Summary of the Invention

[0003] With the explosive growth of the digital media content industry, 3D motion graphics are increasingly being used in advertising, large-screen events, and social media videos. However, there are significant technological gaps in the current production process of high-quality 3D videos.

[0004] On the one hand, professional-grade 3D production tools (such as Maya, Cinema4D, and Unreal Engine editors) are powerful but complex to operate, rely on large integrated development environments, and have high hardware requirements, making them difficult for ordinary end-users to master. The market lacks a solution that encapsulates core rendering capabilities into a lightweight, standalone application that can drive the underlying graphics pipeline through simple interaction.

[0005] On the other hand, in existing game engine-based applications, scene assets are typically pre-baked during the packaging stage, with fixed lighting information. At runtime, users cannot import external dynamic video streams as environmental backgrounds, nor can they achieve real-time physical matching of scene lighting with video content. This results in a disconnect between the foreground 3D objects and the background video lighting, leading to poor visual realism.

[0006] Furthermore, to maintain the interactive frame rate, the real-time rendering engine uses simplified anti-aliasing and low-precision shadows by default. When exporting video, it lacks an automated thread scheduling mechanism, making it impossible to seamlessly switch to a high-sampling-rate offline rendering mode at runtime and automatically complete video compositing.

[0007] Therefore, a solution to the above problems is urgently needed.

[0008] This application aims to address the aforementioned problems in the prior art and provide a system, method, and medium capable of automatically and with high quality generating dynamic 3D videos.

[0009] According to a first aspect of the embodiments of this application, a dynamic 3D video automated generation system based on a runtime rendering environment is provided, comprising: The logic control platform is configured to receive parameterized instructions from the user interaction layer during the runtime of an independent application, and decouple different constituent elements in the 3D scene into independent dynamic modules. The runtime dynamic scene layer communicates with the logic control platform. The logic control platform responds to the parameterized instructions, calls the underlying graphics interface in real time, and dynamically constructs and assembles the dynamic modules in memory to generate the three-dimensional scene. The video stream processing module is configured to decode externally imported video streams and generate dynamic media textures; An ambient lighting matching module is configured to capture the visual features of the dynamic media texture in real time, generate ambient lighting parameters based on the visual features, and apply the ambient lighting parameters to foreground 3D objects in the 3D scene so that the lighting and shadows of the foreground 3D objects dynamically match the video stream; and The rendering scheduler is configured to take over the real-time rendering thread of the standalone application during the video export phase, freeze the game's main thread, and control the engine to perform rendering tasks in offline rendering mode to generate a high-quality image sequence, which is then finally composited into a video file.

[0010] Optionally, the logic control platform is configured to abstract the three-dimensional entities, background environment, and camera trajectory in the three-dimensional scene into independent dynamic modules; wherein, constructing the three-dimensional entities includes: responding to text input commands by calling a font parsing library to dynamically generate a three-dimensional glyph mesh in memory.

[0011] Optionally, the ambient lighting matching module is configured to: extract the average chromaticity and brightness information of the dynamic media texture in each frame, generate spherical harmonic lighting coefficients as ambient lighting parameters through a spherical harmonic convolution algorithm, and inject the spherical harmonic lighting coefficients into the global shader buffer to drive the physically based rendering material reflection of the foreground 3D object.

[0012] Optionally, the rendering scheduler is configured to freeze the game's main thread using a mutex lock, suspend unnecessary logic updates, instantiate a high-fidelity rendering queue, and forcibly control the engine's clock step size; wherein, the offline rendering mode includes spatial and temporal supersampling of each frame.

[0013] Optionally, the rendering scheduler is further configured to: perform rendering warm-up before rendering each frame to stabilize the historical buffer for temporal anti-aliasing; and after rendering is completed, invoke an external codec through a system process call interface to synthesize the image sequence into a video file in a standard encoding format.

[0014] According to another aspect of the present invention, a method for automatically generating dynamic 3D video based on a runtime rendering environment is also provided, comprising: During the runtime of the standalone application, parameterized instructions are received through the logic control platform, and different constituent elements in the 3D scene are decoupled into independent dynamic modules. In response to the parameterized instructions, the underlying graphics interface is invoked in real time to dynamically construct and assemble the dynamic modules in memory to generate the 3D scene; Decode externally imported video streams to generate dynamic media textures; The system captures the visual features of the dynamic media texture in real time, generates ambient lighting parameters based on these features, and applies these parameters to foreground 3D objects in the 3D scene so that the lighting and shadows of the foreground 3D objects dynamically match the video stream; and During the video export stage, the real-time rendering thread of the standalone application is taken over, the main game thread is frozen, and the engine is controlled to perform rendering tasks in offline rendering mode to generate a high-quality image sequence, which is then finally synthesized into a video file.

[0015] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which computer-executable instructions are stored, which, when executed by a processor, implement the above-described method.

[0016] The above-mentioned technical solution of this application has the following beneficial technical effects: The method described in this application, through the construction of a runtime dynamic decoupling and reassembly mechanism, a real-time IBL matching method based on video streams, and a rendering pipeline takeover and automated compositing process, enables the automated output of offline-quality 3D videos in a standalone application using only simple parameterized commands. This invention significantly lowers the barrier to entry for professional 3D video production, allowing even non-professional users to generate dynamic 3D text videos with cinematic lighting effects with a single click. Attached Figure Description

[0017] Figure 1 is a system overall logical architecture diagram in one embodiment of this application.

[0018] Figure 2 is a flowchart of the runtime rendering pipeline takeover process in one embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to specific embodiments and accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of this application. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0020] The accompanying drawings illustrate layer structure diagrams according to embodiments of this application. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0021] Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] In the description of this application, it should be noted that the terms "first", "second", and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0023] Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0024] Example 1: System Overall Architecture Reference Figure 1 This embodiment provides a dynamic 3D video automated generation system based on a runtime rendering environment. The system can be developed based on a real-time 3D engine (such as Unreal Engine 5) and packaged as a standalone executable program (such as a .exe or .app file) for Windows or macOS platforms. The system is logically divided into four layers: 1. UI / UX layer As the system's input front end, this layer adopts an overlay-style user interface design, including text input boxes, font file selectors, video path pickers, and rendering parameter configuration panels. This layer receives user operation commands (such as text content "Hello World", background video path, output resolution, etc.) and encapsulates them into standard data packets to send to the logic control platform.

[0025] 2. Logic Control Platform This is the core hub of the system, responsible for parsing UI commands and scheduling underlying engine resources. In this embodiment, the logic control platform specifically includes three managers: Title Generation Manager: Listens for text input events. When a new string is received, it first cleans up the old 3D objects in the current scene, and then calls the underlying geometry generation algorithm (for example, extracting glyph outlines from font files through the FreeType font parsing library and building a 3D mesh) to dynamically generate a new 3D glyph mesh in memory.

[0026] Background and Lighting Manager: Responsible for decoding control of the video stream and updating parameters of the lighting components. It maintains a media player instance and manages the loading, playback, looping, and texture resource lifecycle of user-specified videos.

[0027] Render Scheduler: Responsible for taking over control of the engine during the video export stage and managing the rendering task queue. Its specific workflow will be detailed later.

[0028] 3. Runtime Dynamic Scene Layer This layer is not pre-defined in the level file, but is a collection of entities dynamically constructed at runtime by the logic control platform. Its core components include: Dynamic Media Textures: Controlled by the Background and Lighting Manager, these textures receive YUV / RGB image data decoded from user video and update them in real time.

[0029] IBL Skylight Component: This component automatically binds the aforementioned dynamic media texture to its source texture and enables the "Real-time Capture" property. This forces the rendering thread to sample and integrate the texture every frame, generating lighting probe data for diffuse and specular reflections.

[0030] 3D character entities: A 3D glyph mesh created by the title generation manager and assigned a preset PBR material (e.g., metallicity close to 1.0). This material maximizes the reflection of ambient lighting information provided by the IBL skylight component, allowing the light and shadow of the foreground text to change dynamically in real time with the background video content. For example, if the background video is a neon-lit street scene, the 3D text surface will reflect the corresponding colored light and shadow.

[0031] Virtual camera: The system's built-in virtual camera dynamically sets 3D character entities as tracking targets and automatically keeps them in the center of the frame.

[0032] 4. Rendering output layer Once the user confirms the export, the rendering scheduler takes over the process, sending the rendering task to the engine pipeline. The generated images are temporarily stored in the image sequence cache, and finally synthesized into the final MP4 video by an external transcoder (such as FFmpeg).

[0033] Example 2: Rendering Pipeline Takeover Process Reference Figure 2To ensure high-definition offline output quality, the rendering scheduler in this embodiment executes the following strict timing sequence: Step S1 (Freeze the game's main thread): When the user clicks the "Export" button, the system first freezes the game's main thread using a mutex, pausing all unnecessary logic updates (such as character AI and physics animation) to prevent unexpected displacement of scene objects during rendering. Simultaneously, it saves a snapshot of the current camera position, lighting parameters, and other state information.

[0034] Step S2 (Building the offline rendering configuration): The system instantiates a high-fidelity rendering pipeline configuration and injects the following parameters: Anti-aliasing: Spatial sampling is set to 1, and temporal sampling is set to 8. Before generating a final frame, the engine will jitter the camera frustum 8 times at the subpixel level and then weight and average the results of these 8 renderings to eliminate jagged edges and smooth noise.

[0035] Output configuration: Specify the output format as a lossless image sequence (such as .JPG, .PNG or .EXR), and the output path points to a temporary directory.

[0036] Step S3 (Initialize rendering task and warm up): Create a rendering task and submit it to the system. Force the rendering clock step to be locked to the reciprocal of the target video frame rate (e.g., 1 / 30th of a second for a 30fps video). Perform render warm-up: pre-render 3-5 frames before outputting each valid frame, but do not save them, to allow the temporal anti-aliasing history buffer to stabilize and ensure that there is no transition noise in the first few frames.

[0037] Step S4 (Frame-by-Frame High-Fidelity Rendering Loop): Call the rendering interface to start the offline rendering pipeline.

[0038] S4-1: The engine performs spatial and temporal supersampling within the current frame to generate a single, alias-free, high-detail image.

[0039] S4-2: Write the generated image to a temporary directory on disk.

[0040] S4-3: Determine if the total number of frames has been reached. If not, proceed to the next frame and repeat S4-1 to S4-3.

[0041] Step S5 (Release the game thread): After the rendering loop ends, the mutex lock of the game's main thread is released, restoring the program's real-time interactive response state.

[0042] Step S6 (Use an external transcoder to synthesize the video): The external FFmpeg encoding / decoding tool is invoked via a system process call interface (such as System.Diagnostics.Process). FFmpeg reads all image sequences in the temporary directory and can optionally merge user-provided audio streams, encoding them into H.264 or H.265 MP4 files. After encoding is complete, the system automatically opens the output folder and resumes normal updates of the game's main thread.

[0043] Through the above specific implementation methods, the present invention enables users to automatically generate 3D dynamic videos with professional-grade lighting and shadow matching effects and offline-grade image quality simply by entering text and selecting a background video in the packaged standalone application, greatly reducing the creative threshold.

[0044] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0045] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0046] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A dynamic three-dimensional video automated generation system based on a runtime rendering environment, characterized by, include: The logic control platform is configured to receive parameterized instructions from the user interaction layer during the runtime of an independent application, and decouple different constituent elements in the 3D scene into independent dynamic modules. The runtime dynamic scene layer communicates with the logic control platform. The logic control platform responds to the parameterized instructions, calls the underlying graphics interface in real time, and dynamically constructs and assembles the dynamic modules in memory to generate the three-dimensional scene. The video stream processing module is configured to decode externally imported video streams and generate dynamic media textures; An ambient lighting matching module is configured to capture the visual features of the dynamic media texture in real time, generate ambient lighting parameters based on the visual features, and apply the ambient lighting parameters to the foreground 3D object in the 3D scene so that the light and shadow of the foreground 3D object dynamically matches the video stream. as well as The rendering scheduler is configured to take over the real-time rendering thread of the standalone application during the video export phase, freeze the game's main thread, and control the engine to perform rendering tasks in offline rendering mode to generate a high-quality image sequence, which is then finally composited into a video file.

2. The system according to claim 1, characterized in that, The logic control platform is configured to abstract the 3D entities, background environment, and camera trajectory in the 3D scene into independent dynamic modules; wherein, constructing the 3D entities includes: responding to text input commands by calling a font parsing library to dynamically generate a 3D glyph mesh in memory.

3. The system according to claim 1, characterized in that, The ambient lighting matching module is configured to: extract the average chromaticity and brightness information of the dynamic media texture in each frame, generate spherical harmonic lighting coefficients as ambient lighting parameters through a spherical harmonic convolution algorithm, and inject the spherical harmonic lighting coefficients into the global shader buffer to drive the physically based rendering material reflection of the foreground 3D object.

4. The system according to claim 1, characterized in that, The rendering scheduler is configured to freeze the game's main thread using a mutex lock, suspend unnecessary logic updates, instantiate a high-fidelity rendering queue, and forcibly control the engine's clock step size; wherein, the offline rendering mode includes spatial and temporal supersampling of each frame.

5. The system according to claim 4, characterized in that, The rendering scheduler is also configured to: perform rendering warm-up before rendering each frame to stabilize the historical buffer for temporal anti-aliasing; and after rendering is completed, invoke an external codec through a system process call interface to synthesize the image sequence into a video file in a standard encoding format.

6. A method for automatically generating dynamic 3D video based on a runtime rendering environment, characterized in that, include: During the runtime of the standalone application, parameterized instructions are received through the logic control platform, and different constituent elements in the 3D scene are decoupled into independent dynamic modules. In response to the parameterized instructions, the underlying graphics interface is invoked in real time to dynamically construct and assemble the dynamic modules in memory to generate the 3D scene; Decode externally imported video streams to generate dynamic media textures; The system captures the visual features of the dynamic media texture in real time, generates ambient lighting parameters based on the visual features, and applies the ambient lighting parameters to the foreground 3D objects in the 3D scene so that the lighting and shadows of the foreground 3D objects dynamically match the video stream. as well as During the video export stage, the real-time rendering thread of the standalone application is taken over, the main game thread is frozen, and the engine is controlled to perform rendering tasks in offline rendering mode to generate a high-quality image sequence, which is then finally synthesized into a video file.

7. The method according to claim 6, characterized in that, The step of decoupling different constituent elements in a 3D scene into independent dynamic modules includes: abstracting the 3D entities, background environment, and camera trajectory in the 3D scene into independent dynamic modules; wherein, constructing the 3D entities includes: responding to text input commands by calling a font parsing library to dynamically generate a 3D glyph mesh in memory.

8. The method according to claim 6, characterized in that, Generating the ambient lighting parameters includes: extracting the average chromaticity and brightness information of the dynamic media texture for each frame, and generating spherical harmonic lighting coefficients as the ambient lighting parameters through a spherical harmonic convolution algorithm; applying the ambient lighting parameters includes: injecting the spherical harmonic lighting coefficients into the global shader buffer to drive the physically based rendering material reflection of the foreground 3D object.

9. The method according to claim 6, characterized in that, Taking over the real-time rendering thread and controlling the engine to execute rendering tasks in offline rendering mode includes: freezing the game's main thread using a mutex lock, pausing unnecessary logic updates, instantiating a high-fidelity rendering queue, and forcibly controlling the engine's clock step size; wherein, the offline rendering mode includes spatial and temporal supersampling of each frame; the method also includes: performing rendering warm-up before rendering each frame to stabilize the historical buffer for temporal anti-aliasing; after rendering is completed, invoking an external codec through a system process call interface to synthesize the image sequence into a video file in a standard encoding format.

10. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that, When the computer-executable instructions are executed by a processor, they implement the method of any one of claims 6 to 9.

Images