Digital human and multi-source video material virtual-real fusion intelligent clipping generation system

Abstract

The application discloses the field of video fusion technology, and particularly relates to an intelligent editing generation system for virtual-real fusion of digital people and multi-source video materials, which comprises a material access module, a digital people rendering module, a basic video synthesis module, a film output module, a time sequence semantic cross-domain alignment module, a layered virtual-real depth fusion correction module, a multi-agent narrative linkage editing decision module, a virtual-real fusion closed loop iteration optimization module and a cross-modal isolation cache scheduling module. The application has the beneficial effects that six types of heterogeneous data, including real scene shots, audio, scripts, digital people skeletons, facial expressions and subtitles, can be synchronously analyzed and a unified time sequence semantic graph can be constructed, the problems that traditional tools can only process single type of data, multi-material time sequence dislocation and digital people shape action drift, manual frame-by-frame correction is required, and batch production workload is huge can be solved, manual correction operation is reduced, and long video batch production efficiency is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of video fusion technology, specifically to an intelligent editing and generation system that integrates digital humans with multi-source video materials in a virtual-real manner. Background Technology

[0002] Digital humans, also known as virtual digital humans, are anthropomorphic digital entities constructed in virtual space using various digital technologies such as computer graphics rendering, motion capture, speech synthesis, and multimodal large models. They have a complete human-like appearance, adapted facial micro-expressions and body movements, can match speech to complete lip-sync, and have intelligent interactive capabilities such as semantic understanding, real-time dialogue, and emotional feedback.

[0003] Digital humans come in various visual styles, such as 2D cartoon and 3D realistic, and are different from ordinary static animated characters. They can link multiple video materials to complete real-time rendering output and are widely used in film and television editing, virtual live streaming, government customer service, science popularization and explanation and other scenarios. They are also the core virtual carrier in the production of virtual and real integrated video. Multi-source video materials are a general term for various video materials of different sources, formats and carriers in the video production scenario.

[0004] However, existing editing and generation systems can only process one type of data, either visual or audio, and cannot simultaneously analyze six types of heterogeneous information, including scripts, subtitles, multi-camera live footage, digital human skeletons, and facial expressions. They lack a unified global temporal modeling method, resulting in frequent problems such as inconsistent timestamps of multi-camera footage, drifting of digital human long-take movements, misalignment of lip movements and dialogue, and asynchrony between visuals and emotions. Moreover, there is no dedicated mechanism for resolving temporal conflicts, and when temporal breaks occur, they can only be corrected manually by dragging and dropping frame by frame, which is extremely labor-intensive when producing long videos in batches.

[0005] To address this, we propose an intelligent editing and generation system that integrates digital humans with multi-source video footage in a virtual-real fusion manner. Summary of the Invention

[0006] In view of the problems existing in the above and / or existing intelligent editing and generation systems that integrate digital humans with multi-source video materials, this invention is proposed.

[0007] Therefore, the purpose of this invention is to provide an intelligent editing and generation system that integrates digital humans with multi-source video materials. By improving existing editing and generation systems, this invention can solve the problems mentioned above. Existing editing and generation systems can only process one type of data, either video or audio, and cannot simultaneously analyze six types of heterogeneous information, including scripts, subtitles, multi-camera live footage, digital human skeletons, and facial expressions. They also lack a unified global temporal modeling method, resulting in frequent problems such as inconsistent timestamps of multi-camera materials, drifting of digital human movements in long shots, misalignment of lip movements and dialogue, and asynchrony between video and emotions. Furthermore, there is no dedicated temporal conflict resolution mechanism, and when temporal breaks occur, manual frame-by-frame dragging and correction is required, leading to an extremely large workload when producing long videos in batches.

[0008] To address the aforementioned technical problems, according to one aspect of the present invention, the present invention provides the following technical solution:

[0009] The intelligent editing and generation system that integrates digital humans with multi-source video footage includes a material access module, a digital human rendering module, a basic video compositing module, a final output module, a temporal semantic cross-domain alignment module, a layered virtual-real deep fusion correction module, a multi-agent narrative linkage editing decision module, a virtual-real fusion closed-loop iterative optimization module, and a cross-modal isolation cache scheduling module.

[0010] The material access module is used to import multi-source video materials, including live-action footage, virtual CG, dynamic graphics and text, and live streams.

[0011] The digital human rendering module loads high-poly digital human assets and outputs standardized digital human video streams and human body geometric parameters.

[0012] The basic video compositing module completes the basic layer overlay operation;

[0013] The output module supports batch export of multiple format videos.

[0014] The temporal semantic cross-domain alignment module incorporates a multimodal feature fusion encoder and an adaptive sliding temporal window unit. It synchronously and in parallel parses six types of heterogeneous data: spatiotemporal features of real-scene shots, audio spectrum emotion tags, text script storyboard timing instructions, digital human SMPL-X skeletal motion sequences, facial blendshape expression sequences, and subtitle semantic vectors. It constructs a unified spatiotemporal global semantic map, performs foreground human instance segmentation, dense depth map extraction, and ambient lighting vector extraction on real-scene frames, and synchronously outputs full-body geometric depth and local facial weight parameters for digital humans. Based on the global semantic map, it performs cross-domain temporal matching, dynamically corrects digital human motion delays and lip-sync shifts, and achieves pixel-level and audio frame-level dual alignment of digital human full-body motion, micro-expressions, lip-syncs with multi-source real-scene materials. It outputs a temporal alignment fusion base with depth annotations. At the same time, it incorporates a temporal conflict resolution unit, which automatically generates temporal compensation offsets for timeline misalignment of multi-camera materials and drift of digital human long-shot actions, eliminating temporal breaks across materials.

[0015] As a preferred embodiment of the intelligent editing and generation system for the virtual-real fusion of digital humans and multi-source video materials described in this invention, the layered virtual-real depth fusion correction module integrates a 2D video pixel correction branch and a 3D Gaussian splashing differentiable rendering dual parallel computing unit. Based on the fusion base, it divides four rendering channels: an independent foreground digital human layer, a real-scene interaction middle layer, a static background layer, and a dynamic light and shadow layer. Each layer is equipped with an independent lighting color temperature adaptive matching unit, an edge multi-level feathering anti-aliasing unit, a virtual-real space occlusion reconstruction unit, and a global environment projection generation unit. It constructs a two-way occlusion judgment matrix based on the digital human's real-time depth map and the real-scene dense depth map, dynamically redraws real-scene objects occluded by the digital human and digital human body parts occluded by the real scene, and introduces a spatiotemporal coupling dynamic fusion weight coefficient. The weight is adaptively updated in real time with the lens depth of field, the mood of the picture, and the material flow rate, and simultaneously generates a soft boundary virtual-real projection that fits the real-scene ground and obstacles, thus eliminating the four major virtual-real fusion defects of digital human feet being suspended, edge clipping, light and shadow separation, and depth of field disconnection from the bottom layer.

[0016] The multi-agent narrative linkage editing decision-making module comprises four collaborative agents: a narrative director agent, a virtual-real fusion quality assessment agent, a digital human on-screen scheduling agent, and an audio emotion matching agent. It has a built-in layered narrative rule library, a shot language constraint library, and an emotion grading threshold library. After inputting the complete script text, emotion trajectory curve, and final film length constraints, the four agents collaboratively traverse all time-series aligned virtual-real fusion segments, automatically locating key narrative nodes such as story turning points, dialogue climaxes, empty shot transitions, and information broadcasts. The four-dimensional joint scoring mechanism simultaneously calculates the narrative smoothness score, virtual-real fusion distortion score, digital human screen proportion score, and audio-visual emotion matching score, automatically filtering out low-scoring flawed segments, adaptively adjusting the playback duration of each segment, and the frequency of digital human panoramic / half-body / close-up appearances. Based on the emotion difference between consecutive shots and the virtual-real screen proportion, it automatically matches differentiated dynamic transitions, outputs a preliminary virtual-real fusion editing sequence with layered annotations, and sends shot composition instructions to the digital human rendering module, simultaneously adjusting the digital human rendering resolution, movement amplitude, and facial expression intensity.

[0017] The virtual-real fusion closed-loop iterative optimization module incorporates a multi-dimensional distortion quantization detection unit, a parameter backpropagation update unit, and an iterative convergence judgment unit. It automatically extracts five types of quantification indicators from the initial edit sequence: inter-frame jitter, temporal offset error, virtual-real edge distortion, light and shadow color difference, and occlusion redraw incompleteness. These indicators are compared with preset graded distortion thresholds. If any indicator exceeds the limit, a global parameter correction vector is generated and pushed back to the temporal semantic cross-domain alignment module and the hierarchical virtual-real deep fusion correction module to update the matching window, fusion weight, and occlusion redraw parameters. After each iteration, a complete edit sequence is regenerated and distortion is detected a second time. The loop terminates when all indicators meet the convergence threshold, and the optimal edit sequence is pushed to the final output module.

[0018] As a preferred embodiment of the intelligent editing and generation system for virtual-real fusion of digital humans and multi-source video materials described in this invention, the cross-modal isolation cache scheduling module is equipped with four physically isolated cache pools: a high-speed cache partition for real-scene materials, an independent cache partition for digital human geometric textures, a temporary cache partition for intermediate fusion frames, and a cache partition for editing decision features. The module dynamically allocates read and write bandwidth according to the current task type. For short video real-time generation tasks, the read and write priority of the digital human cache is increased. For long-form batch editing tasks, the cache capacity of real-scene materials is expanded. The cache pool has a built-in automatic expiration feature cleanup unit to avoid system lag caused by massive amounts of material and intermediate rendering frames occupying video memory, thereby achieving parallel and non-blocking scheduling of multiple tasks.

[0019] As a preferred embodiment of the intelligent editing and generation system for digital humans and multi-source video materials that integrates virtual and real elements as described in this invention, the material access module has a built-in multi-source heterogeneous material standardization preprocessing subunit, which includes five processing branches: automatic frame rate unification, audio track separation and noise reduction, lens boundary AI detection, material lighting pre-sampling, and batch rendering of dynamic frames. It uniformly converts input materials with different encodings, resolutions, and acquisition devices into the system's standard time sequence format, while automatically removing blurry, overexposed, and invalid frames, reducing computational redundancy in the backend modules.

[0020] As a preferred embodiment of the intelligent editing and generation system for digital humans and multi-source video materials that integrates virtual and real elements as described in this invention, the adaptive sliding temporal window unit of the temporal semantic cross-domain alignment module is equipped with a three-segment dynamic adjustment logic. The window is reduced to improve sub-pixel alignment accuracy for dialogue-intensive narrative segments, and the window is enlarged to reduce computing power consumption for environmental empty shot segments. The sliding superimposed window is used to avoid temporal jumps for long shot continuous narrative segments. A matching temporal conflict resolution subunit is provided to eliminate the problem of screen jumps caused by the asynchronous timestamps of multi-camera materials through dynamic interpolation compensation.

[0021] As a preferred embodiment of the intelligent editing and generation system for the virtual-real fusion of digital humans and multi-source video materials described in this invention, the 3D Gaussian splash differentiable rendering unit of the layered virtual-real depth fusion correction module outputs a three-dimensional texture map of the digital human's normal, depth, and reflectivity in real time, which serves as the input basis for the bidirectional occlusion judgment matrix. The virtual-real space occlusion reconstruction unit adopts layered depth sorting redrawing, distinguishing between two types of scene differentiation redrawing strategies: long-term occlusion of static objects and temporary occlusion of dynamic materials, thereby improving the realism of interaction between the digital human and moving real-world materials such as dynamic sand and gravel and pedestrians.

[0022] As a preferred embodiment of the intelligent editing and generation system that integrates digital humans with multi-source video materials as described in this invention, the multi-agent narrative linkage editing decision module is equipped with a hierarchical dynamic transition material library, which includes four groups of transition resources: emotional calm, conflict climax, information switching, and empty shot transition. When the emotional difference between the preceding and following segments exceeds a preset threshold, a strong visual dynamic transition is automatically invoked. For emotionally coherent segments, a fade-in / fade-out low-interference transition is used. At the same time, a character-specific mask transition can be adapted according to whether the digital human is on screen.

[0023] As a preferred embodiment of the intelligent editing and generation system for the virtual-real fusion of digital humans and multi-source video materials described in this invention, the multi-agent narrative linkage editing decision module also has a built-in final cut duration constraint control subunit. When the total editing duration exceeds the customer's limit, the intelligent agent automatically prioritizes compressing the duration of empty shots without key information, while retaining the core dialogue of the digital human and the climax of the story, thus completing the duration compression without destroying the narrative logic.

[0024] As a preferred embodiment of the intelligent editing and generation system for virtual-real fusion of digital humans and multi-source video materials described in this invention, the virtual-real fusion closed-loop iterative optimization module sets three sets of distortion thresholds for mild, moderate, and severe distortion. Mild distortion only involves local fine-tuning of the fusion weight, moderate distortion involves synchronously correcting the timing alignment window, and severe distortion involves globally updating the lighting matching parameters of the digital human. This hierarchical iteration reduces meaningless full-process re-rendering and lowers the computational power consumption of iteration.

[0025] As a preferred embodiment of the intelligent editing and generation system for the virtual-real fusion of digital humans and multi-source video materials described in this invention, the cross-modal isolation cache scheduling module is equipped with a cache preloading and prediction subunit. Based on the narrative editing sequence, it predicts the real-world materials and digital human model textures that need to be called in the next stage, and loads them into the corresponding independent cache partitions in advance, eliminating the delay in material retrieval and ensuring that short videos are generated in real time without lag.

[0026] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0027] 1. In this invention, by setting up a cross-domain alignment module for temporal semantics, six types of heterogeneous data, namely real-scene shots, audio, scripts, digital human skeletons, facial expressions, and subtitles, can be parsed simultaneously and a unified temporal semantic graph can be constructed. Combined with an adaptive sliding temporal window and a temporal conflict resolution unit, it solves the problems of traditional tools being able to process only a single type of data, temporal misalignment of multiple materials, and drift of digital human mouth movements, which require manual frame-by-frame correction and result in a huge workload for batch production. The system can automatically compensate for temporal deviations, reduce manual correction operations, and significantly improve the efficiency of batch production of long videos.

[0028] 2. In this invention, by adopting a dual parallel rendering architecture of 2D pixel correction and 3D Gaussian splashing, coupled with a bidirectional occlusion reconstruction and dynamic fusion weight mechanism, it can adaptively match light and shadow and generate projections that fit the real scene. This solves the image distortion problems such as digital human floating, clipping, and light and shadow fragmentation from the root. There is no need to manually modify the mask frame by frame. The visual realism of the virtual and real images is improved. Multi-dimensional joint scoring automatically identifies narrative nodes and matches and adapts transitions. When the duration exceeds the limit, the core segments of the digital human are retained first. This overcomes the shortcomings of traditional editing that relies on fixed templates, manual segment-by-segment adjustments, and easy deletion of key content. It reduces the degree of manual editing intervention and truly realizes fully automatic editing driven by narrative logic.

[0029] 3. In this invention, by setting a closed-loop iterative optimization module for virtual-real fusion with hierarchical threshold control, five types of image distortion indicators are automatically detected and front-end parameters are updated in reverse hierarchical manner. This eliminates the need for complete re-rendering each time, solving the problems of traditional open-loop compositing where a complete rerun of the entire process is required upon discovering defects, resulting in high computational consumption and long modification cycles. It reduces the cost of iterative optimization. The invention also features four physically isolated cache partitions and pre-loading prediction and automatic cleanup units. It dynamically allocates read and write bandwidth to distinguish between two types of tasks: real-time generation of short videos and batch rendering of long films. This solves the problems of mixed caching of materials in ordinary software, bandwidth contention during read and write operations, stuttering in short videos, and memory overflow and crashes in long films. The invention also enhances the stability of multi-task parallel operation. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the process of the present invention;

[0031] Figure 2 This is a schematic diagram of the temporal semantic cross-domain alignment of the present invention;

[0032] Figure 3 This is a schematic diagram illustrating the graded distortion threshold determination method of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0034] This invention provides an intelligent editing and generation system that integrates digital humans with multi-source video materials. It features a cross-domain temporal semantic alignment module that can simultaneously analyze six types of heterogeneous data, including real-world shots, audio, scripts, digital human skeletons, facial expressions, and subtitles, and construct a unified temporal semantic graph. Combined with an adaptive sliding temporal window and a temporal conflict resolution unit, it solves the problems of traditional tools that can only process single types of data, have temporal misalignment of multiple materials, and have lip-sync drift in digital human images. This requires manual frame-by-frame correction and results in a huge workload for batch production. The system can automatically compensate for temporal deviations, reduce manual correction operations, and significantly improve the efficiency of batch production of long videos.

[0035] Please see Figure 1-3 This intelligent editing and generation system integrates digital humans with multi-source video footage, including a material input module, a digital human rendering module, a basic video compositing module, a final output module, a temporal semantic cross-domain alignment module, a layered virtual-real deep fusion correction module, a multi-agent narrative linkage editing decision-making module, a virtual-real fusion closed-loop iterative optimization module, and a cross-modal isolation cache scheduling module. The material input module imports multi-source video footage, including live-action footage, virtual CG, dynamic graphics, and real-time live streams. The digital human rendering module loads high-poly digital human assets and outputs standardized digital human video streams and human geometric parameters. The basic video compositing module completes basic layer overlay operations. The final output module supports batch export of multiple format final outputs.

[0036] The temporal semantic cross-domain alignment module incorporates a multimodal feature fusion encoder and an adaptive sliding temporal window unit. It simultaneously and in parallel parses six types of heterogeneous data: spatiotemporal features of real-world shots, audio spectrum emotion tags, text script storyboard timing instructions, digital human SMPL-X skeletal motion sequences, facial blendshape expression sequences, and subtitle semantic vectors. This constructs a unified spatiotemporal global semantic map. For real-world frames, it performs foreground human instance segmentation, dense depth maps, and joint extraction of ambient lighting vectors. For digital humans, it simultaneously outputs full-body geometric depth and local facial weight parameters. Based on the global semantic map, it performs cross-domain temporal matching, dynamically correcting digital human motion delays and lip-sync shifts. This achieves pixel-level and audio frame-level dual-resolution matching of digital human full-body movements, micro-expressions, lip-sync with multi-source real-world footage. The system realigns and outputs a time-aligned fusion base with depth annotations. It also has a built-in time conflict resolution unit that automatically generates time compensation offsets for timeline misalignment in multi-camera footage and motion drift in long shots of digital humans, eliminating time discontinuities across footage. By setting up a cross-domain time semantic alignment module, it can simultaneously parse six types of heterogeneous data, including live-action footage, audio, script, digital human skeleton, facial expressions, and subtitles, and construct a unified time semantic graph. Combined with an adaptive sliding time window and a time conflict resolution unit, it solves the problems of traditional tools that can only process single types of data, time misalignment in multiple footage, and motion drift in digital human mouth shapes, requiring manual frame-by-frame correction and resulting in a huge workload for batch production. The system can automatically compensate for time deviations, reduce manual correction operations, and significantly improve the efficiency of batch production of long videos.

[0037] The layered virtual-real depth fusion correction module integrates a 2D video pixel correction branch and a 3D Gaussian splash differentiable rendering dual parallel computing unit. Based on the fusion base, it divides four rendering channels: an independent foreground digital human layer, a real-scene interaction middle layer, a static background layer, and a dynamic light and shadow layer. Each layer is equipped with an independent lighting color temperature adaptive matching unit, an edge multi-level feathering anti-aliasing unit, a virtual-real space occlusion reconstruction unit, and a global environment projection generation unit. It constructs a two-way occlusion judgment matrix based on the digital human's real-time depth map and the real-scene dense depth map, dynamically redraws real-scene objects occluded by the digital human and digital human body parts occluded by the real scene, and introduces a spatiotemporal coupling dynamic fusion weight coefficient. The weight is adaptively updated in real time with the lens depth of field, the mood of the picture, and the material flow rate, and simultaneously generates a soft boundary virtual-real projection that fits the real-scene ground and obstacles, eliminating the four major virtual-real fusion defects of digital human feet being suspended, edge clipping, light and shadow separation, and depth of field disconnection from the bottom layer.

[0038] The multi-agent narrative linkage editing decision-making module comprises four collaborative agents: the narrative director agent, the virtual-real fusion quality assessment agent, the digital human on-camera scheduling agent, and the audio emotion matching agent. It has a built-in layered narrative rule library, a camera language constraint library, and an emotion grading threshold library. After inputting the complete script text, emotion trajectory curve, and final cut length constraints, the four agents collaboratively traverse all time-series aligned virtual-real fusion segments, automatically locating key narrative nodes such as story turning points, dialogue climaxes, empty shot transitions, and information broadcasts. The four-dimensional joint scoring mechanism simultaneously calculates the narrative smoothness score, virtual-real fusion distortion score, digital human screen proportion score, and audio-visual emotion matching score, automatically filtering out low-scoring flawed segments, adaptively adjusting the playback duration of each segment, and the frequency of digital human panoramic / half-body / close-up appearances. Based on the emotion difference between consecutive shots and the virtual-real screen proportion, it automatically matches differentiated dynamic transitions, outputs a preliminary virtual-real fusion editing sequence with layered annotations, and sends shot composition instructions to the digital human rendering module, simultaneously adjusting the digital human rendering resolution, movement amplitude, and facial expression intensity.

[0039] The virtual-real fusion closed-loop iterative optimization module incorporates a multi-dimensional distortion quantization detection unit, a parameter backpropagation update unit, and an iterative convergence judgment unit. It automatically extracts five types of quantification indicators from the initial edit sequence: inter-frame jitter, temporal offset error, virtual-real edge distortion, light and shadow color difference, and occlusion redraw incompleteness. These indicators are compared with preset graded distortion thresholds. If any indicator exceeds the limit, a global parameter correction vector is generated and pushed back to the temporal semantic cross-domain alignment module and the hierarchical virtual-real deep fusion correction module to update the matching window, fusion weight, and occlusion redraw parameters. After each iteration, a complete edit sequence is regenerated and distortion is detected a second time. The loop terminates when all indicators meet the convergence threshold, and the optimal edit sequence is pushed to the final output module.

[0040] The cross-modal isolation cache scheduling module has four physically isolated cache pools: a high-speed cache partition for real-scene materials, an independent cache partition for digital human geometric textures, a temporary cache partition for intermediate blended frames, and a cache partition for editing decision features. It dynamically allocates read and write bandwidth according to the current task type. For short video real-time generation tasks, the read and write priority of digital human cache is increased, and for long film batch editing tasks, the cache capacity of real-scene materials is expanded. The cache pool has a built-in automatic expiration feature cleanup unit to avoid system lag caused by massive materials and intermediate rendering frames occupying video memory, and to achieve parallel and non-blocking scheduling of multiple tasks. The material access module has a built-in multi-source heterogeneous material standardization preprocessing subunit, which includes five processing branches: automatic frame rate unification, audio track separation and noise reduction, lens boundary AI detection, material lighting presampling, and batch rendering of image and text dynamic frames. It uniformly converts input materials with different encodings, different resolutions, and different acquisition devices into the system's standard time sequence format, while automatically removing blurry, overexposed, invalid and waste frames to reduce the computational redundancy of backend modules.

[0041] The adaptive sliding temporal window unit of the temporal semantic cross-domain alignment module has a three-segment dynamic adjustment logic. The window is reduced to improve sub-pixel alignment accuracy for dialogue-intensive narrative segments, the window is enlarged to reduce computing power consumption for environmental empty shot segments, and the sliding overlay window is used for long shot continuous narrative segments to avoid temporal jumps. It is equipped with a temporal conflict resolution sub-unit, which eliminates the screen jump problem caused by the asynchronous timestamps of multi-camera materials through dynamic interpolation compensation. The 3D Gaussian splash differentiable rendering unit of the layered virtual and real depth fusion correction module outputs the digital human normal, depth, and reflectivity 3D texture in real time as the input basis for the bidirectional occlusion judgment matrix. The virtual and real space occlusion reconstruction unit adopts layered depth sorting redrawing, distinguishes between two types of scenes: long-term occlusion of static objects and temporary occlusion of dynamic materials, and improves the realism of interaction between dynamic sand, pedestrians and other moving real scene materials and digital humans.

[0042] The multi-agent narrative linkage editing decision module is equipped with a hierarchical dynamic transition material library, which includes four groups of transition resources: emotional calm, conflict climax, information switching, and empty shot transition. When the emotional difference between the preceding and following segments exceeds a preset threshold, a strong visual dynamic transition is automatically invoked. For emotionally coherent segments, a fade-in / fade-out low-interference transition is used. At the same time, a character-specific mask transition can be adapted according to whether the digital human is on screen. The multi-agent narrative linkage editing decision module also has a built-in sub-unit for controlling the final cut length. When the total editing length exceeds the client's limit, the agents automatically prioritize compressing the length of empty shots without key information, while retaining the digital human's core dialogue and the story's climax segments, completing the length compression without disrupting the narrative logic.

[0043] The virtual-real fusion closed-loop iterative optimization module sets three sets of distortion thresholds: mild, moderate, and severe. Mild distortion only involves local fine-tuning of the fusion weights, moderate distortion involves synchronous correction of the timing alignment window, and severe distortion involves global updates of the digital human's lighting matching parameters. This hierarchical iteration reduces meaningless full-process re-rendering and lowers the computational power consumption of iteration. The cross-modal isolation cache scheduling module is equipped with a cache preloading and prediction subunit. Based on the narrative editing sequence, it predicts the real-scene materials and digital human model textures that need to be called in the next stage and loads them into the corresponding independent cache partitions in advance, eliminating the delay in material retrieval and ensuring that short videos are generated in real time without lag.

[0044] The workflow of this invention is as follows: The operator first imports various types of materials, such as live-action footage, virtual CG, dynamic graphics, and live streams, into the material access module. Multiple preprocessing branches within the material access module work simultaneously to unify the resolution, encoding, and frame rate of all materials, automatically separate and reduce noise tracks, detect shot boundaries, and simultaneously filter out blurry or abnormally exposed invalid images. Graphic materials are batch-rendered into standard dynamic frames, and all materials are uniformly converted into the system's standard timing format, completing the pre-processing standardization of materials and reducing the computational burden on subsequent modules. The preprocessed materials are sent to the cross-modal isolation cache scheduling module. The system allocates corresponding cache partitions according to the current task type. For short video tasks, digital human textures are prioritized for loading into the high-speed partition, while for feature film tasks, the storage space for live-action materials is expanded. At the same time, based on the subsequent editing sequence, the textures of materials to be used are pre-loaded in advance to avoid retrieval lag. Idle intermediate rendering frames and feature data are automatically released from video memory by the cleanup unit.

[0045] Standardized materials are fed into the temporal semantic cross-domain alignment module. The encoder inside the temporal semantic cross-domain alignment module synchronously reads six types of heterogeneous data: spatiotemporal information of real-world shots, audio emotion spectrum, script storyboard, SMPL skeletal motion of digital humans, facial blendshape expressions, and subtitle vectors. This constructs a unified global temporal semantic map. At the same time, depth maps and lighting vectors are extracted from the real-world shots, and the geometric parameters of the digital human's entire body are read synchronously. The system automatically switches the sliding window size according to the density of the plot. For dialogue-intensive segments, the window is shrunk to achieve subpixel alignment, and the window is enlarged for empty shots to save computing power. For multi-camera time misalignment and digital human long shot drift, the temporal conflict resolution unit automatically calculates and compensates for the offset, corrects motion delay and lip-sync misalignment issues, and outputs a temporal alignment fusion base with depth information.

[0046] The fusion substrate is sent to the layered virtual-real depth fusion correction module. The module simultaneously starts 2D pixel correction and 3D Gaussian splash dual rendering units, and divides the digital foreground, real scene interaction, static background and dynamic light and shadow into four independent channels. It builds a two-way occlusion judgment matrix based on the depth maps of digital human and real scene, distinguishes between static objects and dynamic pedestrians / materials and adopts differentiated redrawing logic, dynamically adjusts the fusion weight and generates soft projection that fits the ground, automatically corrects color temperature and edge jaggedness, and completely eliminates image defects such as floating, clipping, and depth of field disconnection.

[0047] All corrected real and virtual segments enter the multi-agent narrative linkage editing decision module. The four agents of narrative director, quality assessment, digital scheduling, and audio-visual matching work together to locate key nodes such as story climax, dialogue, and empty shots based on the script, emotional curve, and final length constraints. The system scores segments from four dimensions: narrative fluency, integration quality, digital human appearance ratio, and audio-visual matching. Low-scoring flawed segments are automatically removed. The system matches corresponding transitions based on the emotional difference between shots and automatically adjusts the playback duration of each segment. If the total duration exceeds the limit, empty shots without information are compressed first, while core dialogues of digital humans are retained. Simultaneously, the system sends shot size instructions such as wide shots and close-ups to the digital human rendering module to adjust the resolution and movement range of the digital humans, and outputs a fully annotated initial editing sequence.

[0048] The initial sequence is fed into the virtual-real fusion closed-loop iterative optimization module. The system automatically detects five types of quantitative indicators: frame jitter, temporal offset, edge distortion, color difference, and occlusion / incompleteness. It is then graded and processed according to three levels of distortion thresholds: mild, moderate, and severe. If an indicator exceeds the standard, a parameter correction vector is generated and pushed back to the temporal alignment and fusion correction modules to update the configuration. After each round of optimization, a complete segment is regenerated and re-detected. Once all indicators meet the standards, the iteration is considered to have converged, and the optimal final sequence is sent to the final output module for batch export of final videos in multiple formats. The entire process runs in a loop, and the caching module synchronously schedules various materials and intermediate data throughout the process, ensuring stable operation in both long-form batch editing and short-video real-time generation scenarios.

[0049] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. An intelligent editing and generation system that integrates digital humans with multi-source video footage, characterized in that: It includes a material access module, a digital human rendering module, a basic video compositing module, a final output module, a temporal semantic cross-domain alignment module, a layered virtual-real deep fusion correction module, a multi-agent narrative linkage editing decision module, a virtual-real fusion closed-loop iterative optimization module, and a cross-modal isolation cache scheduling module; The material access module is used to import multi-source video materials, including live-action footage, virtual CG, dynamic graphics and text, and live streams. The digital human rendering module loads high-poly digital human assets and outputs standardized digital human video streams and human body geometric parameters. The basic video compositing module completes the basic layer overlay operation; The output module supports batch export of multiple format videos. The temporal semantic cross-domain alignment module incorporates a multimodal feature fusion encoder and an adaptive sliding temporal window unit. It synchronously and in parallel parses six types of heterogeneous data: spatiotemporal features of real-scene shots, audio spectrum emotion tags, text script storyboard timing instructions, digital human SMPL-X skeletal motion sequences, facial blendshape expression sequences, and subtitle semantic vectors. It constructs a unified spatiotemporal global semantic map, performs foreground human instance segmentation, dense depth map extraction, and ambient lighting vector extraction on real-scene frames, and synchronously outputs full-body geometric depth and local facial weight parameters for digital humans. Based on the global semantic map, it performs cross-domain temporal matching, dynamically corrects digital human motion delays and lip-sync shifts, and achieves pixel-level and audio frame-level dual alignment of digital human full-body motion, micro-expressions, lip-syncs with multi-source real-scene materials. It outputs a temporal alignment fusion base with depth annotations. At the same time, it incorporates a temporal conflict resolution unit, which automatically generates temporal compensation offsets for timeline misalignment of multi-camera materials and drift of digital human long-shot actions, eliminating temporal breaks across materials.

2. The intelligent editing and generation system for digital human and multi-source video material fusion according to claim 1, characterized in that, The layered virtual-real depth fusion correction module integrates a 2D video pixel correction branch and a 3D Gaussian splash differentiable rendering dual parallel computing unit. Based on the fusion base, it divides four rendering channels: an independent foreground digital human layer, a real-scene interaction middle layer, a static background layer, and a dynamic light and shadow layer. Each layer is equipped with an independent lighting color temperature adaptive matching unit, an edge multi-level feathering anti-aliasing unit, a virtual-real space occlusion reconstruction unit, and a global environment projection generation unit. It constructs a two-way occlusion judgment matrix based on the digital human's real-time depth map and the real-scene dense depth map, dynamically redraws real-scene objects occluded by the digital human and digital human body parts occluded by the real scene, and introduces a spatiotemporal coupling dynamic fusion weight coefficient. The weight is adaptively updated in real time with the lens depth of field, the mood of the picture, and the material flow rate, and simultaneously generates a soft boundary virtual-real projection that fits the real-scene ground and obstacles, eradicating the four major virtual-real fusion defects of digital human feet being suspended, edge clipping, light and shadow fragmentation, and depth of field disconnection from the bottom layer. The multi-agent narrative linkage editing decision-making module comprises four collaborative agents: a narrative director agent, a virtual-real fusion quality assessment agent, a digital human on-screen scheduling agent, and an audio emotion matching agent. It has a built-in layered narrative rule library, a shot language constraint library, and an emotion grading threshold library. After inputting the complete script text, emotion trajectory curve, and final film length constraints, the four agents collaboratively traverse all time-series aligned virtual-real fusion segments, automatically locating key narrative nodes such as story turning points, dialogue climaxes, empty shot transitions, and information broadcasts. The four-dimensional joint scoring mechanism simultaneously calculates the narrative smoothness score, virtual-real fusion distortion score, digital human screen proportion score, and audio-visual emotion matching score, automatically filtering out low-scoring flawed segments, adaptively adjusting the playback duration of each segment, and the frequency of digital human panoramic / half-body / close-up appearances. Based on the emotion difference between consecutive shots and the virtual-real screen proportion, it automatically matches differentiated dynamic transitions, outputs a preliminary virtual-real fusion editing sequence with layered annotations, and sends shot composition instructions to the digital human rendering module, simultaneously adjusting the digital human rendering resolution, movement amplitude, and facial expression intensity. The virtual-real fusion closed-loop iterative optimization module incorporates a multi-dimensional distortion quantization detection unit, a parameter backpropagation update unit, and an iterative convergence judgment unit. It automatically extracts five types of quantification indicators from the initial edit sequence: inter-frame jitter, temporal offset error, virtual-real edge distortion, light and shadow color difference, and occlusion redraw incompleteness. These indicators are compared with preset graded distortion thresholds. If any indicator exceeds the limit, a global parameter correction vector is generated and pushed back to the temporal semantic cross-domain alignment module and the hierarchical virtual-real deep fusion correction module to update the matching window, fusion weight, and occlusion redraw parameters. After each iteration, a complete edit sequence is regenerated and distortion is detected a second time. The loop terminates when all indicators meet the convergence threshold, and the optimal edit sequence is pushed to the final output module.

3. The intelligent editing and generation system for digital human and multi-source video material fusion according to claim 2, characterized in that, The cross-modal isolation cache scheduling module has four physically isolated cache pools: a high-speed cache partition for real-scene materials, an independent cache partition for digital human geometric textures, a temporary cache partition for intermediate fusion frames, and a cache partition for editing decision features. It dynamically allocates read and write bandwidth according to the current task type. For short video real-time generation tasks, the read and write priority of digital human cache is increased, and for long film batch editing tasks, the cache capacity of real-scene materials is expanded. The cache pool has a built-in automatic expiration feature cleanup unit to avoid system lag caused by massive materials and intermediate rendering frames occupying video memory, and to achieve parallel and non-blocking scheduling of multiple tasks.

4. The intelligent editing and generation system for digital human and multi-source video material fusion according to claim 3, characterized in that, The material access module has a built-in multi-source heterogeneous material standardization preprocessing subunit, which includes five processing branches: automatic frame rate unification, audio track separation and noise reduction, lens boundary AI detection, material lighting presampling, and batch rendering of dynamic frames. It uniformly converts input materials with different encodings, resolutions, and acquisition devices into the system's standard time sequence format, while automatically removing blurry, overexposed, invalid, and waste frames, reducing the computational redundancy of the backend module.

5. The intelligent editing and generation system for digital human and multi-source video material fusion according to claim 4, characterized in that, The adaptive sliding temporal window unit of the temporal semantic cross-domain alignment module has a three-segment dynamic adjustment logic: the window is reduced to improve sub-pixel alignment accuracy for dialogue-intensive narrative segments, the window is enlarged to reduce computing power consumption for environmental empty shot segments, and the sliding superimposed window is used for long shot continuous narrative segments to avoid temporal jumps. It is equipped with a temporal conflict resolution subunit, which eliminates the screen jump problem caused by the asynchronous timestamps of multi-camera materials through dynamic interpolation compensation.

6. The intelligent editing and generation system for digital human and multi-source video material fusion according to claim 5, characterized in that, The 3D Gaussian splash differentiable rendering unit of the layered virtual-real depth fusion correction module outputs the digital human's normal, depth, and reflectivity three-dimensional texture in real time as the input basis for the bidirectional occlusion judgment matrix. The virtual-real space occlusion reconstruction unit adopts layered depth sorting redrawing to distinguish between two types of scenes: long-term occlusion of static objects and temporary occlusion of dynamic materials. This differentiates the redrawing strategies and improves the realism of interaction between the digital human and moving real-world materials such as dynamic sand and gravel and pedestrians.

7. The intelligent editing and generation system for digital human and multi-source video material fusion according to claim 6, characterized in that, The multi-agent narrative linkage editing decision module is equipped with a hierarchical dynamic transition material library, which includes four groups of transition resources: emotional calm, conflict climax, information switching, and empty shot transition. When the emotional difference between the preceding and following segments exceeds a preset threshold, a strong visual dynamic transition is automatically invoked. For emotionally coherent segments, a fade-in / fade-out low-interference transition is used. At the same time, a character-specific mask transition can be adapted according to whether the digital human is on screen.

8. The intelligent editing and generation system for digital human and multi-source video material fusion according to claim 7, characterized in that, The multi-agent narrative linkage editing decision module also has a built-in final cut duration constraint control subunit. When the total editing duration exceeds the customer's limit, the agent automatically prioritizes compressing the duration of empty shots without key information, while retaining the core dialogue of the digital human and the climax of the story, thus completing the duration compression without destroying the narrative logic.

9. The intelligent editing and generation system for digital human and multi-source video material fusion according to claim 8, characterized in that, The virtual-real fusion closed-loop iterative optimization module sets three sets of distortion thresholds: mild, moderate, and severe. Mild distortion only involves local fine-tuning of the fusion weights, moderate distortion involves synchronously correcting the timing alignment window, and severe distortion involves globally updating the lighting matching parameters of the digital human. This hierarchical iteration reduces meaningless full-process re-rendering and lowers the computational power consumption of iteration.

10. The intelligent editing and generation system for digital human and multi-source video material fusion according to claim 9, characterized in that, The cross-modal isolation cache scheduling module is equipped with a cache preloading and prediction subunit. Based on the narrative editing sequence, it predicts the real-life materials and digital human model textures that need to be called in the next stage and loads them into the corresponding independent cache partitions in advance, eliminating the delay in material retrieval and ensuring that short videos are generated in real time without lag.

Images