A video generation method and system based on the stitching of first and last frames from multiple shots
By employing techniques such as frame stitching, parallel thread pool scheduling, and adaptive audio duration alignment, the problems of discontinuous footage, low efficiency, poor audio quality, and unstable final product in multi-camera video generation have been solved. This enables efficient and controllable multi-camera video generation, suitable for content production such as science popularization and corporate promotion.
Patent Information
- Application Number
- CN202611309329.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-27
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multi-lens video generation technologies suffer from problems in automated mass production scenarios, such as inconsistent shot transitions, low generation efficiency, inaccurate audio duration alignment, incomplete task interruption control, and difficulty in guaranteeing finished product quality, thus failing to meet the demands for efficient, high-quality, and controllable production.
By employing techniques such as frame-to-frame stitching, thread pool parallel scheduling, adaptive audio duration alignment, dual-version output of both hard and soft subtitles, and automatic quality review, we achieve natural continuity of visual elements between shots, parallel processing of speech synthesis and video generation, hierarchical alignment of audio duration, automatic generation and quality verification of subtitles, ensuring flexible control of the generation process and reliability of the final product.
By seamlessly connecting the first and last frames, visual jumps during camera transitions are eliminated, improving generation efficiency, ensuring audio quality, enabling second-level task cancellation, reducing maintenance costs, ensuring product consistency, and adapting to the mass production needs of multi-camera videos.
Smart Images

Figure CN123123670A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence video generation technology, and in particular relates to a video generation method and system based on the stitching of first and last frames of multiple shots. Background Technology
[0002] With the rapid development of deep learning technology, text-based and image-based video technologies based on diffusion models have been gradually implemented, providing automated generation capabilities for short videos, science popularization, and corporate promotional content. However, in the mass production of multi-camera continuous narrative videos, existing technologies still have significant adaptation defects and engineering pain points, failing to meet the demands for efficient, high-quality, and controllable automated production.
[0003] Firstly, the lack of a seamless transition mechanism between shots results in poor visual continuity. Most current mainstream image-generated video models employ a single-shot independent output architecture, lacking the ability to automatically connect multiple shots. Multi-shot video production typically involves generating each shot independently and then manually splicing them in post-production. This lack of connection and transmission in the subject matter, lighting, and color tone between shots leads to content gaps and visual abrupt changes at transition points, resulting in insufficient narrative smoothness. General video splicing tools can only physically stitch the beginning and end of video files; they lack frame-level content transition logic, failing to address the root cause of the transition gap problem.
[0004] Secondly, the generation process is executed serially, resulting in low overall production efficiency. In traditional multi-camera video generation architectures, video generation and speech synthesis are processed serially. Speech synthesis can only be started after all video segments have been generated. The total generation time is simply the sum of the video generation time and the speech synthesis time. In multi-camera scenarios, the production cycle is long, the system's computing resources are underutilized, and it is difficult to adapt to the efficiency requirements of large-scale batch production.
[0005] Third, the audio-visual duration alignment method is simplistic, resulting in significant damage to audio quality. Existing audio-visual duration alignment methods mostly adopt a single proportional speed adjustment scheme, directly adjusting the overall speed of the audio according to the duration ratio. When the speed adjustment is too large, problems such as distortion of human voice tone and disjointed speech rate can easily occur. It is difficult to balance the accuracy of duration matching and the naturalness of speech, and it cannot adapt to the different duration matching needs of multiple shots, thus impairing the auditory experience of explanatory videos.
[0006] Fourth, the task interruption control capability is weak, resulting in significant loss of computing resources. Existing open-source generation tools generally lack fine-grained task interruption mechanisms. When users cancel a generation task, they can usually only terminate the front-end scheduling process, but cannot simultaneously stop the model inference task that is being executed in the background, resulting in the continuous invalidation of GPU computing power. If the task is terminated by forcibly ending the process, it is easy to cause problems such as memory leaks and temporary cache file residues, resulting in high operation and maintenance costs.
[0007] Fifth, the output format of the finished product is limited, and the quality control capability is insufficient. Existing automated generation solutions mostly output only a single video file, and subtitle generation and format packaging rely heavily on manual post-processing, resulting in low efficiency for mass production. Furthermore, there is a lack of automated quality verification throughout the entire process, requiring manual sampling to check the quality of the finished product. It is difficult to identify problems such as missing files, weak or silent audio, and garbled or misaligned subtitles in batches, leading to high costs for manual quality control and difficulty in ensuring the consistency of the finished product.
[0008] In summary, existing multi-camera video generation technologies have not been systematically optimized for automated mass production scenarios, and cannot simultaneously meet the comprehensive requirements of high video continuity, high production efficiency, high audio quality, controllable tasks, and high-quality finished products, thus hindering the practical application of AIGC videos in industrial content production scenarios. Summary of the Invention
[0009] The purpose of this invention is to address the aforementioned technical problems by providing a video generation method and system based on the stitching of first and last frames from multiple shots.
[0010] In view of this, the present invention provides a video generation method based on the stitching of first and last frames of multiple shots, comprising the following steps: Receive a video generation request containing configuration information for multiple shots. The configuration information for each shot includes at least the scene generation prompt, target duration, and narration text. For the first shot, the text-to-image model is invoked to generate prompts and the first frame image based on the scene of that shot; for the second and subsequent shots, the last frame image of the video already generated for the previous shot is extracted and used as the first frame input for the current shot's image-to-video task. The image-to-video generation task is executed sequentially according to the shot order. The first frame image of the current shot and the image generation prompt are used as input parameters to pass into the image-to-video generation model to generate a single-shot video clip. While the image-generated video task is executed serially, the speech synthesis task for all shots is started in parallel based on the thread pool, so that the total speech synthesis time and the total video generation time partially overlap in time. The actual duration of the synthesized audio for each shot is obtained and compared with the target duration of the corresponding shot. Adaptive duration alignment processing is performed in stages according to the degree of duration difference: if the duration difference is within the preset tolerance range, the original audio speed is maintained; if the actual audio duration is shorter than the target duration and exceeds the tolerance range, silence is added to the end of the audio to the target duration; if the actual audio duration is longer than the target duration and exceeds the tolerance range, the audio is accelerated at a speed ratio not exceeding the preset upper limit. The video clips from all shots are merged with the aligned audio track to generate a complete video file.
[0011] Furthermore, the step of extracting the last frame image of the video already generated from the previous shot specifically includes: The audio and video processing tool locates the position near the end of the previous shot's video file; a single frame image is extracted from this position; the extracted image is saved as an image file in a specified format, and used as the first frame input for the current shot's video generation task.
[0012] Furthermore, in the step of accelerating the audio at a speed ratio not exceeding a preset upper limit, when the total speed ratio exceeds the single-level speed threshold, the total speed ratio is broken down into multi-level speed parameters, and the speed is applied step by step through a chain-like multi-level speed change to reduce the sound quality loss caused by a single-level large speed change.
[0013] Furthermore, it also includes steps for quickly canceling tasks: Configure a separate cancellation event flag for each video generation task; During the execution loop of the image-generated video task, the status of the cancellation event flag is periodically checked; When a cancellation event is detected, the interruption interface of the image-based video inference service is synchronously invoked to terminate the currently executing model inference task.
[0014] Furthermore, it also includes the subtitle generation step: Accumulate the duration of each shot in sequence, and calculate the start and end timestamps of the subtitles corresponding to each segment of narration text. The narration text is automatically wrapped according to the preset single-line character limit, and the total number of lines in a single subtitle does not exceed the preset line number threshold. Based on the calculated timestamp and the narration text after line breaks, a standard SRT format subtitle file is generated.
[0015] Furthermore, the step of generating a complete video file specifically includes: First, generate a video with embedded soft subtitle tracks; Based on the aforementioned soft-subtitle version of the video, an attempt was made to burn hard subtitles; If an error occurs during the hard subtitle burning process, the video will be automatically downgraded to a soft subtitle version as the final output; if the hard subtitle burning is successful, both hard subtitle and soft subtitle versions will be output simultaneously.
[0016] Furthermore, each shot is configured with an independent random seed for video generation and a seed for the first frame image generation; When no externally specified seed parameter is received, the video generates a random seed by combining a preset baseline value with the lens number to generate a deterministic value. The seed generated from the first frame image uses a preset fixed default value. When an externally specified seed parameter is received, the corresponding generation task is executed with the specified seed parameter to ensure that the generation results can be reproduced under the same input parameters.
[0017] Furthermore, it also includes an automated output quality review step: Verify that the final video file, audio file, and subtitle file are complete and exist. Verify that the number of subtitles in the subtitle file matches the number of shots, and check for any encoding anomalies in the subtitle text; The root mean square energy value of each shot's audio is calculated based on a fixed duration window to determine whether there are weak or no audio. Summarize all test results and generate a quality review report.
[0018] A video generation system based on the stitching of first and last frames from multiple shots includes: The request receiving and parsing module is used to receive multi-lens video generation requests and parse them to obtain the configuration information of each lens; The first frame generation module is used to call the text-to-image model to generate the first frame image of the first shot; The last frame extraction module is used to extract the last frame image of the video generated from the previous shot, and use it as the first frame input for the next shot's generated video task. The video generation module is used to sequentially execute the image generation tasks of each shot in the order of the shots, and generate single-shot video clips. The parallel speech synthesis module is used to perform speech synthesis tasks for all shots in parallel based on a thread pool while the video generation module performs tasks serially. The audio alignment module is used to compare the actual duration of the synthesized audio of each shot with the target duration, and perform adaptive duration alignment processing according to the degree of duration difference. The video merging module is used to merge video clips from all shots with the aligned audio track to generate a complete video file.
[0019] Furthermore, it also includes: The task cancellation control module is used to maintain an independent cancellation event flag for each generated task and synchronously call the interruption interface of the graph-generated video inference service when a cancellation request is detected. The subtitle generation module is used to calculate subtitle timestamps and generate subtitle files in standard SRT format; The output quality review module is used to perform file integrity verification, subtitle compliance verification, and audio signal strength detection, and output the quality review results.
[0020] The beneficial effects of this invention are: By employing a serial generation mechanism that seamlessly connects the first and last frames, the last frame of the previous shot is automatically extracted as the starting input for the next shot. This ensures a natural and continuous transmission of visual elements, lighting, and color tones between shots, eliminating visual abrupt changes at the source of generation and guaranteeing the visual smoothness of multi-shot narratives. This effect can be quantitatively verified by comparing the differences in shot transitions between independently generated splicing schemes and this scheme, and statistically analyzing the proportion of visual abrupt changes.
[0021] By using a thread pool to schedule the speech synthesis tasks of all shots in parallel, the total time spent on speech synthesis partially overlaps with the total time spent on sequentially executed image-to-video generation. This avoids the accumulation of time spent in sequential execution, improves the system's computing power utilization without increasing hardware investment, and shortens the overall generation time of multi-shot videos, adapting to the efficiency requirements of mass production. This effect can be quantitatively verified by statistically comparing the total generation time of a purely sequential architecture with that of this solution for the same number of shots.
[0022] A tiered, amplitude-limited adaptive duration alignment strategy is employed. Based on the degree of difference between the audio duration and the target duration, different processing methods are used: maintaining the original speed, adding silence at the end, and amplitude-limited acceleration, prioritizing the naturalness of the speech. Simultaneously, a chain-like multi-level speed-changing mechanism is used, breaking down large speed changes into multiple small speed changes to reduce the sound quality degradation caused by single-level speed changes. While ensuring audio-visual duration matching, speech distortion is controlled within a range imperceptible to the human ear. This effect can be verified through objective speech quality evaluation indicators and subjective listening tests.
[0023] Based on a collaborative mechanism between an independent cancellation event flag and the inference service interruption interface, the cancellation status is periodically checked during the generation loop. Upon detecting a cancellation request, the underlying model inference is synchronously terminated. This achieves a second-level cancellation response without forcibly terminating the process, promptly releasing GPU computing resources while avoiding issues such as memory leaks and temporary cache residues, thus improving resource utilization and system maintenance convenience. This effect can be verified by testing the time interval from issuing the cancellation command to inference stopping and memory release.
[0024] The system employs a dual-version output of both soft and hard subtitles and an automatic downgrade mechanism. First, a basic version with an embedded soft subtitle track is generated, and then the hard subtitle is burned. If the hard subtitle burning fails, the system automatically downgrades to output a soft subtitle version, ensuring that the task always produces a usable output and avoiding the failure of the entire task due to a partial failure. At the same time, the dual-version finished products can adapt to the subtitle requirements of different playback scenarios, improving the system's operational stability and scenario adaptability.
[0025] Through an automatic subtitle generation mechanism, timestamps are accumulated based on shot duration and automatically formatted according to preset rules to generate standardized SRT subtitle files, eliminating the need for manual adjustments to the timeline and layout. Combined with an automatic end-to-end output quality review mechanism, it can automatically verify file integrity, subtitle compliance, and audio signal strength, intercepting substandard outputs in advance, significantly reducing the workload of manual post-processing and quality inspection, and improving the consistency of finished products and production efficiency in mass production scenarios.
[0026] It adopts a mechanism that separates the video generation seed from the first frame image seed. By default, a deterministic seed is generated based on the baseline value and the shot number to ensure consistent generation results under the same configuration. At the same time, it supports external explicit specification of seed parameters to achieve stable reproduction of generation results under the same input, adapting to the needs of scenarios such as content debugging and effect replication. Attached Figure Description
[0027] Figure 1 This is a schematic diagram of the overall system architecture of the present invention; Figure 2 This is a schematic diagram of the multi-lens first and last frame stitching process of the present invention; Figure 3 This is a flowchart of the audio duration adaptive alignment process of the present invention; Figure 4 This is a schematic diagram illustrating the rapid task cancellation state transition of the present invention; Figure 5 This is a flowchart of the dual-version packaging process for hardware and software subtitles in this invention; Figure 6 Output a quality review flowchart for this invention. Detailed Implementation
[0028] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below with reference to embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0029] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.
[0030] In scenarios such as popular science explanations, corporate promotion, and short video content production, the demand for automated mass production of multi-camera continuous narrative videos is growing. Existing image-generated video technologies are mostly based on a single-camera independent generation architecture, which has several adaptation defects when directly applied to multi-camera mass production scenarios: First, each shot starts its generation task independently, resulting in a lack of continuity in the subject matter, lighting, and color tone of consecutive shots, leading to noticeable visual jumps and poor image coherence after splicing; second, video generation and speech synthesis use a serial execution mode, with the total generation time being the sum of the time spent in each step, resulting in long generation cycles and low production efficiency in multi-camera scenarios; third, audio-visual duration alignment often uses a single proportional speed adjustment method, which causes severe voice distortion when the speed adjustment is too large, making it difficult to adapt to the differentiated duration matching requirements of multiple shots; fourth, the generation task lacks a fine-grained interruption mechanism, and canceling the operation only terminates the front-end scheduling, while the back-end model inference continues to run, resulting in ineffective waste of computing resources; fifth, the output format of the finished product is limited, lacking an automated quality verification process, leading to high manual quality inspection costs in mass production scenarios. To effectively address the shortcomings of traditional multi-camera video generation methods, this invention proposes a video generation method based on the stitching of the first and last frames of multiple shots.
[0031] This method is deployed in a video generation server system. The system uniformly schedules the underlying AI model capabilities through a workflow orchestration layer. The text-to-image model uses the FLUX series models, the image-to-video model uses the Wan series models, and the speech synthesis uses a mature end-to-end Chinese speech synthesis model. All of the above basic models are existing mature open-source models, and their internal network structures and training processes are existing technologies. This invention integrates and schedules the model capabilities at the workflow orchestration and process control level without involving any improvement to the internal structure of the models. All scheduling rules and processing logic are clear and reproducible. The system as a whole includes a request receiving and parsing module, a first frame generation module, a last frame extraction module, a video generation module, a parallel speech synthesis module, an audio duration alignment module, a subtitle generation module, and a video merging module. It is also equipped with a task cancellation control module and an output quality review module, forming a complete automated closed loop from request access to finished product output.
[0032] All thresholds, rules, and parameters used in this method have been pre-calibrated and verified. All parameters have clear setting criteria and can be adjusted according to the deployment environment. There are no black-box parameters.
[0033] The first category is duration alignment parameters, including duration tolerance, single-shot speed limit, overall speed limit, and single-level speed threshold. The calibration process involves selecting Chinese speech samples of different lengths and speeds, testing the clarity and naturalness of the voice at different speed ratios, and determining the single-level speed threshold and single-shot speed limit with the constraint that there is no significant distortion to the human ear. Combining the audio-visual synchronization accuracy requirements of multi-shot video, the duration tolerance range is determined. Finally, the overall speed limit in the synthesis stage serves as a safety net, further controlling the total duration deviation based on single-shot alignment, balancing sound quality and synchronization accuracy. In this embodiment, the duration tolerance is ±0.5 seconds, the single-shot audio speed limit is 1.1 times, the overall audio speed limit is 1.3 times, and the single-level speed threshold is 2.0 times.
[0034] The second category is subtitle layout parameters, including the maximum number of characters per line and the maximum number of lines per subtitle segment. The calibration process is as follows: combining the subtitle display area size of mainstream video frames, statistically analyzing the readability of subtitles with different character counts, and determining the maximum number of characters per line and the maximum number of lines to ensure a good viewing experience while covering most of the narration text length. In this embodiment, the maximum number of characters per line is set to 20, and a maximum of 2 lines are retained per subtitle segment.
[0035] The third category is the quality review threshold, which includes the audio weak sound energy threshold and encoding verification rules. The calibration process is as follows: collect audio samples of three types: normal speech, weak sound, and silence; calculate the root mean square energy value under a fixed duration window; and determine the energy threshold with the goal of distinguishing normal speech from weak / silent sounds. The subtitle encoding verification is based on whether there are garbled characters or substitution characters to ensure text readability.
[0036] The fourth category is seed generation rules, which include the video generation seed baseline value and the default seed value for the first frame image. The calibration process is as follows: a fixed baseline constant is set, and a deterministic seed is generated in combination with the shot number to ensure the consistency of the generated results under the same input; at the same time, it supports external input of custom seeds to meet the needs of effect reproduction.
[0037] Upon receiving a video generation request containing configuration information for multiple shots, the request receiving and parsing module first parses the configuration parameters of each shot. The configuration information for each shot includes scene generation prompts, target duration, and Chinese narration text. Optional configurations include an independent random seed, a first frame image seed, and negative prompts. After parsing, the system simultaneously initiates the serial image-to-video generation link and the parallel speech synthesis generation link. Each stage of the entire process has clearly defined input, processing logic, and output results.
[0038] The first step is the generation of a serial video by stitching together the first and last frames: For the first shot, the first frame generation module calls the text-to-image model to generate a prompt word based on the shot's visuals, creating the first frame image. This image is then stored according to a preset path and serves as the starting frame input for the first shot's image-to-video generation task. For the second and subsequent shots, the last frame extraction module extracts the last frame image of the previously generated video and uses it as the first frame input for the current shot's image-to-video generation task. Last frame extraction is implemented using audio / video processing tools, locating the position 0.1 seconds before the end of the previous shot's video file, extracting a single frame image, saving it in a specified format, and then inputting it into the current shot's generation task.
[0039] Each shot is executed strictly sequentially according to its number. A workflow template variable replacement method is used to batch inject parameters such as the first frame image path, on-screen cues, negative cues, and random seeds into the image-generated video workflow template before submitting it to the inference service for execution. Each shot's output segment is named according to a unified rule, including the task identifier, shot number, and shot identifier, facilitating subsequent stitching. A mechanism that uses the last frame of a preceding shot as the first frame of a subsequent shot ensures continuous transmission of image elements between shots, eliminating image gaps caused by independent generation.
[0040] The second step is parallel speech synthesis scheduling: The system utilizes a thread pool to launch Chinese speech synthesis tasks for all shots in parallel, decoupling it from the serially executed image-to-video task in terms of time. Upon completion of request parsing, a thread pool is created, submitting an independent speech synthesis task for the narration text of each shot, and performing the synthesis processing in parallel. After all video clips for all shots are generated, the results of the parallel speech synthesis are collected uniformly and then proceed to the subsequent audio alignment stage. This mechanism ensures that the total time spent on speech synthesis partially overlaps with the total time spent on video generation, rather than simply adding them together, effectively compressing the overall generation cycle.
[0041] Step 3: Adaptive audio duration limiting and alignment: First, the actual duration of the synthesized audio for each shot is obtained and compared with the target duration of the corresponding shot. The processing is then categorized according to the degree of difference: if the duration difference is within a preset tolerance range, the original audio speed is maintained, and only format transcoding is performed; if the actual audio duration is shorter than the target duration and exceeds the tolerance range, silence is added to the end of the audio until the target duration is reached, without changing the speech speed; if the actual audio duration is longer than the target duration and exceeds the tolerance range, the audio is accelerated at a speed increase not exceeding a preset upper limit.
[0042] When the total speed ratio exceeds the single-level speed threshold, it is broken down into multiple speed parameters. In acceleration mode, the maximum speed ratio per level does not exceed 2.0, and in deceleration mode, the minimum speed ratio per level is not less than 0.5. Speed changes are applied step by step through a chain-like multi-level speed change method to reduce the audio quality loss caused by large single-level speed changes. In the final video synthesis stage, the system further compares the total duration of the merged audio and the merged video. If there is a discrepancy, the overall audio is slightly speed-adjusted to ensure consistency between the total audio and video duration.
[0043] Step 4: Automatic generation of SRT subtitles: The duration of each shot is accumulated sequentially, and the start and end timestamps of the subtitles corresponding to each narration text are calculated in turn. The timestamp format follows the SRT standard and is accurate to the millisecond level. At the same time, the narration text is automatically wrapped according to the preset single-line character limit to control the total number of lines of a single subtitle to not exceed the preset threshold. Any lines exceeding the limit are not added, ensuring that the display length of a single subtitle is controllable. Finally, a standard SRT format subtitle file is generated.
[0044] Step 5: Dual-version packaging of software and hardware subtitles with downgrade and fault tolerance: First, all video clips, aligned audio tracks, and subtitle tracks are merged to generate a soft-subtitle version of the video with an embedded soft-subtitle track. Subtitles are encapsulated as an independent track, allowing playback devices to switch them on and off as needed. Next, a hardware subtitle burning process is attempted based on the soft-subtitle version video, using subtitle filters to burn the subtitles pixel-by-pixel into the screen. If an error occurs during the hardware subtitle burning process, the soft-subtitle version video is automatically downgraded and output as the final product, ensuring a usable result. If the burning is successful, both the hardware and soft-subtitle versions of the video are output simultaneously, catering to the needs of different playback scenarios.
[0045] Step 6: Quickly cancel control of the task: The system configures an independent cancellation event flag for each video generation task and maintains a global cancellation event dictionary to manage the cancellation status of each task. During the execution loop of the image-generated video task, the status of the cancellation event flag is checked every approximately 0.5 seconds. When a cancellation request is received, the cancellation event flag for the corresponding task is set. When a cancellation event is detected, the current scheduling process is immediately terminated, and the interrupt interface of the image-generated video inference service is simultaneously called to terminate the currently executing model inference task. This achieves rapid response to cancellation commands, timely release of computing resources, avoids forced process termination, and prevents memory leaks and cache file residue.
[0046] Step 7: Automatic output quality review: After generation, the entire process is automatically validated, performing three checks sequentially: First, file integrity check, verifying the completeness of the final video, audio, and subtitle files, and identifying any missing files. Second, subtitle compliance check, verifying the number of subtitles matches the number of shots, and checking for garbled characters in the subtitle text to ensure a one-to-one correspondence between subtitles and shots and readability. Third, audio signal strength check, calculating the root mean square energy value of the audio for each shot based on a fixed duration window, comparing it with a preset threshold to determine if there are any weak or silent sounds, and identifying any abnormalities in speech synthesis. After all checks are completed, a quality review report is generated, marking any anomalies.
[0047] This method is specifically optimized for automated batch production of multi-camera continuous narrative videos, as follows: 1. Multi-camera narrative scene adaptation: To address the pain points of abrupt scene changes and weak narrative continuity after multi-camera video splicing, a first and last frame connection mechanism is designed to achieve a natural transition of visual elements between shots, ensure visual narrative continuity, and adapt to the production needs of long video storyboards.
[0048] 2. Adaptation to batch production efficiency: To address the pain points of long generation cycles and low computing power utilization in batch production scenarios, a parallel and decoupled scheduling architecture for speech synthesis and video generation is designed to fully utilize system computing resources, compress overall generation time, and improve batch production efficiency.
[0049] 3. Narration audio quality adaptation: In view of the high requirements for the naturalness of voice in explanatory videos, a graded amplitude limiting audio alignment strategy is designed to prioritize the preservation of the original speech rate and apply amplitude limiting speed change only when necessary. Combined with chain-style multi-level speed change, the audio quality is reduced and the listening experience of the finished product is guaranteed.
[0050] 4. Unattended Operation and Maintenance Adaptation: For unattended batch production scenarios, a hard subtitle burning degradation fault tolerance mechanism and an automatic quality review mechanism are designed to avoid task failure caused by local faults, while automatically intercepting unqualified outputs and reducing manual operation and maintenance costs.
[0051] 5. Computing power management and adaptation: Considering the precious nature of GPU computing power resources, a fine-grained task cancellation mechanism is designed to terminate invalid tasks in a timely manner, release computing power resources, and improve resource utilization.
[0052] All the technical effects of this method can be verified through corresponding experiments, as follows: 1. Enhance visual continuity: The first and last frame stitching mechanism can effectively eliminate screen jumps at shot transitions, ensuring the continuity of the subject, lighting, and color tone between shots. This can be verified by comparing the differences between independently generated stitching schemes and this scheme at shot transitions and statistically analyzing the proportion of visual jumps.
[0053] 2. Improve overall generation efficiency: The parallel decoupling scheduling architecture of speech synthesis and video generation can effectively compress the overall generation cycle; this can be verified by statistically comparing the total generation time of the serial architecture and this solution under the same number of shots.
[0054] 3. Reduce audio speed variation distortion: The graded limiting speed variation and chain-type multi-level speed variation mechanism can control the speed variation range within a range where there is no obvious distortion in the human ear; the difference in sound quality between the unlimited speed variation scheme and this scheme can be verified by subjective listening tests and objective speech quality assessments.
[0055] 4. Improve task cancellation response speed: Based on the collaborative mechanism of event flags and inference interruption interface, a second-level response to cancellation commands can be achieved; this can be verified by testing the time interval from the issuance of the cancellation command to the inference stopping and the release of video memory.
[0056] 5. Improve system fault tolerance and finished product qualification rate: The automatic degradation mechanism for hard subtitles can avoid task failure due to local faults, and automatic quality review can intercept unqualified outputs in advance; the success rate of tasks and the qualification rate of finished products under abnormal scenarios can be verified.
[0057] 6. Ensure reproducible results: The dual-separation seed configuration mechanism can stably reproduce the generated results under specified seed parameters; the consistency of the output results can be verified by generating the test multiple times with the same parameters and comparing them.
[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.
Claims
1. A video generation method based on the stitching of first and last frames from multiple shots, characterized in that, Includes the following steps: Receive a video generation request containing configuration information for multiple shots. The configuration information for each shot includes at least the scene generation prompt, target duration, and narration text. For the first shot, the text-to-image model is invoked to generate prompts and the first frame image based on the scene of that shot; for the second and subsequent shots, the last frame image of the video generated from the previous shot is extracted and used as the first frame input for the current shot's image-to-video task. The image-to-video generation task is executed sequentially according to the shot order. The first frame image of the current shot and the image generation prompt are used as input parameters to pass into the image-to-video generation model to generate a single-shot video clip. While the image-generated video task is executed serially, the speech synthesis task for all shots is started in parallel based on the thread pool, so that the total speech synthesis time and the total video generation time partially overlap in time. The actual duration of the synthesized audio for each shot is obtained and compared with the target duration of the corresponding shot. Adaptive duration alignment is performed in stages according to the degree of duration difference: if the duration difference is within the preset tolerance range, the original speech rate of the audio is maintained; if the actual duration of the audio is shorter than the target duration and exceeds the tolerance range, silence is added to the end of the audio to the target duration. If the actual duration of the audio is longer than the target duration and exceeds the tolerance range, the audio will be accelerated at a speed ratio not exceeding the preset upper limit. The video clips from all shots are merged with the aligned audio track to generate a complete video file.
2. The video generation method based on the stitching of first and last frames of multiple shots according to claim 1, characterized in that, The step of extracting the last frame image of the video generated from the previous shot specifically includes: The audio and video processing tool locates the position near the end of the previous shot's video file; a single frame image is extracted from this position; the extracted image is saved as an image file in a specified format, and used as the first frame input for the current shot's video generation task.
3. The video generation method based on the stitching of first and last frames of multiple shots according to claim 1, characterized in that, In the step of accelerating audio at a speed ratio not exceeding a preset upper limit, when the total speed ratio exceeds the single-level speed threshold, the total speed ratio is broken down into multi-level speed parameters, and the speed is applied step by step through a chain-like multi-level speed change to reduce the sound quality loss caused by a large single-level speed change.
4. The video generation method based on the stitching of first and last frames of multiple shots according to claim 1, characterized in that, It also includes steps for quickly canceling tasks: Configure a separate cancellation event flag for each video generation task; During the execution loop of the image-generated video task, the status of the cancellation event flag is periodically checked; When a cancellation event is detected, the interruption interface of the image-based video inference service is synchronously invoked to terminate the currently executing model inference task.
5. The video generation method based on the stitching of first and last frames of multiple shots according to claim 1, characterized in that, It also includes the subtitle generation process: Accumulate the duration of each shot in sequence, and calculate the start and end timestamps of the subtitles corresponding to each segment of narration text. The narration text is automatically wrapped according to the preset single-line character limit, and the total number of lines in a single subtitle does not exceed the preset line number threshold. Based on the calculated timestamp and the narration text after line breaks, a standard SRT format subtitle file is generated.
6. The video generation method based on the stitching of first and last frames of multiple shots according to claim 5, characterized in that, The steps for generating a complete video file specifically include: First, generate a video with embedded soft subtitle tracks; Based on the aforementioned soft-subtitle version of the video, an attempt was made to burn hard subtitles; If an error occurs during the hard subtitle burning process, the soft subtitle version of the video will be automatically downgraded and output as the final video file; if the hard subtitle burning is successful, both the hard subtitle version and the soft subtitle version of the video will be output simultaneously.
7. The video generation method based on the stitching of first and last frames of multiple shots according to any one of claims 1 to 6, characterized in that, Each shot is configured with an independent random seed for video generation and a seed for generating the first frame image. When no externally specified seed parameter is received, the video generates a random seed by combining a preset baseline value with the lens number to generate a deterministic value. The seed generated from the first frame image uses a preset fixed default value. When an externally specified seed parameter is received, the corresponding generation task is executed with the specified seed parameter to ensure that the generation results can be reproduced under the same input parameters.
8. The video generation method based on the stitching of first and last frames of multiple shots according to any one of claims 1 to 6, characterized in that, It also includes an automatic output quality review step: Verify that the final video file, audio file, and subtitle file are complete and exist. Verify that the number of subtitles in the subtitle file matches the number of shots, and check for any encoding anomalies in the subtitle text; The root mean square energy value of each shot's audio is calculated based on a fixed duration window to determine whether there are weak or no audio. Summarize all test results and generate a quality review report.
9. A video generation system based on the stitching of first and last frames from multiple shots, characterized in that, include: The request receiving and parsing module is used to receive multi-lens video generation requests and parse them to obtain the configuration information of each lens; The first frame generation module is used to call the text-to-image model to generate the first frame image of the first shot; The last frame extraction module is used to extract the last frame image of the video generated from the previous shot, and use it as the first frame input for the next shot's generated video task. The video generation module is used to sequentially execute the image generation tasks of each shot in the order of the shots, and generate single-shot video clips. The parallel speech synthesis module is used to perform speech synthesis tasks for all shots in parallel based on a thread pool while the video generation module performs tasks serially. The audio alignment module is used to compare the actual duration of the synthesized audio of each shot with the target duration, and perform adaptive duration alignment processing according to the degree of duration difference. The video merging module is used to merge video clips from all shots with the aligned audio track to generate a complete video file.
10. The video generation system based on the stitching of first and last frames of multiple shots according to claim 9, characterized in that, Also includes: The task cancellation control module is used to maintain an independent cancellation event flag for each generated task and synchronously call the interruption interface of the graph-generated video inference service when a cancellation request is detected. The subtitle generation module is used to calculate subtitle timestamps and generate subtitle files in standard SRT format; The output quality review module is used to perform file integrity verification, subtitle compliance verification, and audio signal strength detection, and output the quality review results.
