Audio timing driven AIGC full-automatic sheet generation method and system

Abstract

The application discloses an audio timing driving AIGC full-automatic film generation method and system, and belongs to the technical field of fusion media short video automation production. The application constructs a full-link automatic film pipeline without manual intervention, realizes strong and weak beat grading and emotion paragraph division through multi-dimensional audio analysis of a local adaptive threshold, solves the pain point of material shortage through three-level material scheduling and AIGC intelligent completion of full-link quality control, combines beat-driven editing, emotion-adaptive color grading system and industrial abnormal fault tolerance mechanism, and finally outputs 4K films meeting the broadcasting standards. The drumstick alignment accuracy of the application reaches more than 98%, the production efficiency of a single short film is improved by more than 95%, and the application can be widely applied to the industrial high-quality batch production of short films in the fields of tourism and government affairs.

Description

Technical Field

[0001] This invention belongs to the fields of automated production of converged media short videos, intelligent editing of AI promotional videos, and the cross-disciplinary technology of computer vision and digital media processing. Specifically, it relates to an integrated, fully automated video production method and system for industrialized automated production scenarios of short videos of urban cultural tourism scenery, government promotional short videos, radio and television new media columns, and short drama trailers. This method is driven by audio temporal characteristics and combined with AIGC (Artificial Intelligence Generated Content) technology for material completion, automatic editing, and emotional color grading. The output meets the broadcasting standards of radio and television new media and the offline large-screen playback specifications. Background Technology

[0002] With the full arrival of the era of converged media, the demand for short video content in areas such as government publicity, urban cultural tourism promotion, brand marketing, and short drama derivatives is growing exponentially. In particular, content with a high degree of standardization, such as cultural tourism promotional videos, government report short films, and city image films, often needs to be produced in batches within a short period of time. Traditional video editing is highly dependent on manual labor. Editors need to manually complete the entire process, including music timing, material selection, shot splicing, transition design, and color grading. A single 60-second professional short film usually requires a senior editor to spend more than 4 hours to complete. Mass production results in high labor costs, long delivery cycles, and the quality of the final product is affected by the individual skill level of the editor, making it difficult to guarantee consistency.

[0003] To improve editing efficiency, mainstream editing software on the market has successively launched auxiliary functions such as automatic drum beat matching and one-click editing. Some existing technologies have also conducted research on audio-driven automatic editing solutions, realizing basic automatic shot switching and clip splicing through audio beat detection, which has reduced the manual threshold of editing to a certain extent.

[0004] However, after in-depth research and actual production line testing by the applicant, the existing technology generally suffers from the following multi-dimensional technical shortcomings, making it impossible to achieve truly fully automated, industrialized, large-scale production:

[0005] First, the audio analysis dimensions are limited, and the ability to layer rhythm and classify emotions is insufficient. Most existing technologies only achieve binary classification detection of "presence or absence of drum beats," failing to classify the strength of beats or identify emotional sections such as climaxes, interludes, intros, and outros. This directly results in extremely simplistic visual language, relying solely on hard cuts and transitions throughout. It cannot match different visual impacts to rhythmic intensity or adjust the editing rhythm according to the music's mood, leading to a stiff, mechanical, and unprofessional viewing experience. Furthermore, existing drum beat detection algorithms often use fixed global thresholds, making them unsuitable for different musical styles and background music with varying BPMs. Accuracy fluctuates significantly across different scenarios, such as light music, electronic music, and orchestral music, generally falling below 90%, often requiring manual correction for precise timing.

[0006] Secondly, the reliance on manual material preparation, lacking an automatic completion mechanism, prevents the formation of a closed-loop production process. Existing automatic editing solutions all presuppose a "sufficient material library," only able to arrange and combine existing materials. If the material for a corresponding scene is insufficient in duration or completely missing, the entire production process is interrupted, requiring manual supplementation through filming or material retrieval before it can continue. In customized short film production scenarios such as cultural tourism and government affairs, the shortage of specific scene materials is a frequent problem. This means that so-called "automatic editing" can only replace part of the splicing work, failing to achieve a fully automated process from audio input to final output, and thus falling fundamentally short of true fully automated production.

[0007] Third, the audio-visual linkage is limited to the editing point level, lacking visual matching and color grading linkage at the emotional dimension. Current audio-visual linkage technology is limited to "cutting shots at drum beats," failing to deeply bind musical emotions with visual color grading and effects. The final product often suffers from accurate rhythmic timing but chaotic color style: climaxes lack visual tension, lyrical sections appear harsh, and color grading styles are disjointed across different sections. Post-production color correction is still required, increasing workload and making it difficult to ensure consistency in emotional expression.

[0008] Fourth, the output standards do not match the needs of professional broadcasting and cannot be directly uploaded. Existing automatic editing tools are mostly designed for individual users of self-media, and their output parameters are mostly general network playback specifications. The resolution, bitrate, encoding format, and color space do not comply with the broadcasting standards. For professional broadcasting scenarios such as government new media, urban cultural and tourism screens, and TV channel programs, the automatically generated finished products cannot be used directly. They must undergo secondary transcoding, color correction, and format packaging, which increases the process steps and is prone to problems such as image quality loss and audio-visual desynchronization.

[0009] Fifth, the functional modules are fragmented, lacking an industrial-scale fault-tolerance mechanism and failing to form a complete production pipeline. Most existing technologies focus only on single-point editing stages, failing to connect the entire chain of "audio parsing—style matching—material supply—automatic editing—color grading output," with multiple stages still requiring manual coordination and parameter settings. Furthermore, existing solutions lack fallback mechanisms for high-frequency anomalies in industrial production, such as audio parsing failures, AIGC generation anomalies, and encoding bitrate overflows. In scenarios involving the mass production of hundreds of short films, the workload for manual troubleshooting and correction remains enormous, hindering industrialized, assembly-line-style content production.

[0010] In summary, existing automatic editing technologies only achieve single-point efficiency improvements and fail to address the core pain points of high-quality, fully automated, standardized output and robust mass production. They are insufficient to meet the needs of industrialized mass production of converged media short films, and there is clear room for technological improvement. Summary of the Invention

[0011] To address the aforementioned shortcomings of existing technologies, the present invention aims to provide an audio timing-driven AIGC fully automated video production method and system. This system constructs a fully automated pipeline without human intervention, encompassing "audio layered analysis—style mapping and matching—intelligent material scheduling—AIGC completion generation—beat-driven editing—emotion-adaptive color grading—standardized output." The invention achieves strong and weak beat classification and emotional segmentation through multi-dimensional audio timing analysis with local adaptive thresholds. It addresses the pain point of material shortage through three-level tagged material scheduling and AIGC intelligent completion with full-link quality control. It enhances the professionalism of the final product through emotionally linked graded effects and a color grading system. A supporting industrialized anomaly tolerance mechanism ensures the stability of mass production, ultimately outputting a 4K final product that meets broadcast standards. While maintaining a drum beat alignment accuracy rate of over 98%, this invention increases the production efficiency of a single short film by over 95%, achieving industrialized, unmanned, and high-quality mass production of converged media short films.

[0012] To achieve the above-mentioned objectives, the present invention employs the following technical solution:

[0013] 1. A fully automated audio timing-driven AIGC video generation method

[0014] The overall process comprises five core steps: multi-dimensional audio analysis and emotional segmentation, audio-visual style mapping rule matching, intelligent scheduling of storyboard material library and AIGC material completion, beat-driven automated editing, emotion-adaptive color grading, and final output. The specific implementation methods for each step are as follows:

[0015] S1. Audio Multidimensional Analysis and Emotional Segmentation: This step takes 48kHz sampling rate, 16-bit depth, WAV format stereo background music as input and outputs an audio timing marker sequence with millisecond-level timestamps, including strong drum beat timestamps, weak beat timestamps, climax start and end markers, interlude start and end markers, and regular segment markers. The specific processing flow is as follows:

[0016] (1) Audio standardization preprocessing: First, the input audio is standardized and preprocessed: the peak normalization algorithm is used to map the audio amplitude to the range of [-1,1] to eliminate the interference of volume difference between different sound sources on energy calculation; if the input audio sampling rate is not 48kHz, it is uniformly converted to 48kHz through the polyphase filter bank resampling algorithm to ensure the consistency of parameters in subsequent calculations.

[0017] (2) Short-Time Fourier Transform and Frame Energy Calculation: The audio is subjected to a Short-Time Fourier Transform (STFT) using a Hanning window. The default window length is 50ms, corresponding to 2400 sampling points at a sampling rate of 48kHz, and the frame shift is 20ms, corresponding to 960 sampling points. This parameter combination takes into account both time resolution and frequency resolution, and is suitable for most mainstream promotional video background music in the 100-140BPM range. The formula for calculating STFT is:

[0018]

[0019] in, It is an audio time-domain signal. For the Hanning window function, For frame indexing For frequency index, The number of sampling points corresponding to the window length. This represents the number of sampling points corresponding to the frame shift.

[0020] The single-frame spectral energy is calculated for each frame of STFT results using the following formula:

[0021]

[0022] in, For the first Frame number STFT complex amplitude value at each frequency point.

[0023] (3) Grading of strong and weak drum beats

[0024] A local adaptive thresholding method is used for drum beat classification to address the poor adaptability of global thresholding to audio with a large dynamic range. The mean local energy is calculated for the 50 frames before and after the current frame (101 frames in total). Set adaptive threshold coefficient The value range is [2.5, 4.0], and it is adapted according to the music style: higher values ​​are used for strong electronic and rock music, and lower values ​​are used for soothing orchestral and folk music. The grading rules are as follows:

[0025] when Furthermore, if the local maximum condition is met, the frame is determined to be a strong drum beat frame, and the corresponding timestamp is recorded.

[0026] when When the frame is identified as a weak beat frame, the corresponding timestamp is recorded.

[0027] when When the frame is in the normal frame, it is determined to be a normal frame and no special marking is made.

[0028] To avoid misjudging consecutive high-energy frames, a minimum interval threshold of 30ms is set: if the time interval between two strong drumbeats is less than 30ms, only the frame with higher energy is retained to ensure the accuracy of drumbeat detection.

[0029] (4) Emotional paragraph division

[0030] Melodic change rate is calculated based on Mel-frequency cepstral coefficients (MFCCs) to segment musical mood segments: First, 13-dimensional MFCC features are extracted. Then, the L2 norm distance between the MFCC features of two adjacent frames is calculated as the single-frame feature change rate. The calculation formula is as follows:

[0031]

[0032] in, Let i be the 13-dimensional MFCC feature vector of the i-th frame. This represents the L2 norm of a vector.

[0033] The average rate of change within the window is calculated using a sliding window of 3 seconds. The paragraph determination rule is as follows:

[0034] When the average MFCC change rate over 3 consecutive seconds is greater than 0.75, it is marked as the start point of the climax; when the average change rate over 2 consecutive seconds is less than 0.3, it is marked as the end point of the climax.

[0035] The low-frequency energy ratio of the entire audio is statistically analyzed. When the low-frequency energy below 80Hz accounts for more than 80% of the total energy and the energy fluctuation is less than 10%, the continuous interval is marked as an interlude. The intro and outro are uniformly classified as interludes for processing.

[0036] The section that is neither a climax nor an interlude is marked as a regular verse.

[0037] (5) Adjustable parameters

[0038] In this step, the sliding window length can be adjusted from 40 to 60 ms, and the frame shift can be adjusted from 15 to 25 ms. For fast-paced music with high BPM (above 140), a 40 ms window length and a 15 ms frame shift are used to improve temporal resolution. For soothing music with low BPM (below 90), a 60 ms window length and a 25 ms frame shift are used to improve frequency resolution, ensuring high detection accuracy under different music styles.

[0039] S2, Audio-visual style mapping rule matching

[0040] This step pre-builds and stores a structured audio-visual mapping template library, which by default includes templates in three vertical categories: traditional Chinese style, technology, and fresh and clean. Each template category includes three configurations: rhythm effect parameters, transition styles, and basic color grading preferences. The system automatically loads the corresponding mapping rules based on the style template selected by the user, matching different audio timing events with corresponding visual effects. The general mapping rules are as follows, with the three template categories differentiated by detailed effects configurations:

[0041] (1) Strong drumbeat mapping touch

[0042] When the strong drum beat timestamp is reached, a 0.2-second hard-cut transition is executed: the image linearly zooms from 1.0x to 1.2x, completing the camera switch at the peak of the drum beat, and then linearly shrinks back to 1.0x, creating a striking "zoom-cut-shrink" effect. Differentiated configurations: A traditional Chinese style template overlays a 1-frame ink wash flash effect; a tech template overlays a 1-frame tech wireframe sweep effect; a fresh template overlays a 1-frame soft light flash effect.

[0043] (2) Weak beat mapping

[0044] When a weak beat timestamp is triggered, a slight screen vibration effect is superimposed: the vibration amplitude is 0.5% of the screen width, a sine displacement curve is used, and the vibration lasts for 0.1 seconds. This enhances the sense of rhythm without disrupting the visual appeal and is suitable for scenarios such as weak beats in choruses and background beats.

[0045] (3) Climax paragraph mapping

[0046] When the scene reaches its climax, a radial particle blur effect layer is overlaid: 200-300 particles, default transparency 30%, particles diffuse outward from the center of the screen, and their movement speed is positively correlated with the audio energy of the current frame; simultaneously, the image sharpness is increased by 15%, enhancing visual tension. Differentiated configurations: warm gold particles for the traditional Chinese style template; ice blue particles for the technological template; and soft white particles for the refreshing template.

[0047] (4) Interlude mapping

[0048] When entering the interlude, a 0.8x slow motion stretch is applied, and the camera transitions use a 0.3s fade-in / fade-out transition to create a soothing and gentle visual rhythm, forming a clear rhythmic contrast with the climax.

[0049] The template library adopts an extensible architecture, supporting users to add custom style templates. Custom templates can independently configure all special effects parameters, transition styles, and color grading preferences.

[0050] S3, Intelligent scheduling of storyboard material library and AIGC material completion

[0051] This step enables tagged management and intelligent scheduling of footage, and automatically calls the AIGC model to generate supplementary footage when footage is scarce. End-to-end quality control ensures the seamless integration of generated footage with live-action footage, completely resolving workflow interruptions caused by insufficient footage. The specific process is as follows:

[0052] (1) Construction of a tag-based material library

[0053] A pre-built structured script material library is provided, with each piece of material accompanied by multi-dimensional tags, including: scene type tags (ancient architecture, city night scene, natural landscape, office scene, food close-up, etc.), style type tags (traditional Chinese style, technology, fresh style, etc.), shot duration, resolution, color space, and shot movement (push, pull, pan, tilt, stationary). The material library supports batch import and automatic tag recognition, and can automatically tag materials through AI pre-identification.

[0054] (2) Material matching and sufficiency judgment

[0055] Based on the storyboard and segment division of the current short film, match corresponding scene and style footage to each segment in chronological order, and calculate the total usable time of the matched footage for each segment. The judgment rule is:

[0056] If the total duration of the matched footage is greater than or equal to 120% of the required duration of the segment, it is considered that there is sufficient footage, and the footage is arranged for editing according to the camera movement logic and the order of the frame composition.

[0057] If the total duration of the matched material is between 80% and 120% of the required duration of the segment, it is determined that the material is basically sufficient, and the duration can be adjusted by slow motion or fast motion.

[0058] If the total duration of the matched material is less than 80% of the required duration of the paragraph, or if there is no matched material, it is determined that there is a material shortage, and the AIGC material completion process is triggered.

[0059] (3) AIGC intelligent completion generation

[0060] After completion is triggered, the system performs end-to-end generation and standardization processing, as follows:

[0061] ① Contextual feature extraction: Extract scene labels, dominant color (by extracting the dominant color of the color histogram through K-Means clustering), and motion vectors (by optical flow) from adjacent shots before and after the missing shot to ensure the contextual consistency of the generated material.

[0062] ② Standardized prompt word construction: The prompt words are constructed in the structure of "[style description] + [scene content] + [lighting and shadow atmosphere] + [image quality requirements] + negative constraint words". By default, character avoidance weight is added to the negative constraint words to avoid generating character distortion images.

[0063] ③ Controlled Generation: Calls the locally deployed StableDiffusionXL image generation model, combines it with the ControlNet depth model to control the composition of the screen, can load the project's exclusive LoRA model to align the visual style, and generate static empty shot materials with a resolution of 1024×576. The generation frame rate is set to 1 frame every 2 seconds, and the total number of generated frames is calculated based on the missing duration.

[0064] ④ Multi-step standardized post-processing:

[0065] Color space unification: The generated materials are converted from sRGB color space to Rec.709 through ICC configuration files to maintain consistency with the color space of the original materials in the material library and avoid color deviation;

[0066] Resolution Upgrade: The Real-ESRGAN super-resolution algorithm is used to upgrade the footage to the target resolution, supporting adaptive resolutions of 1024P, 1080P, and 4K to ensure that the image quality matches the original footage.

[0067] Dynamic processing: The static sequence is interpolated into a 25fps dynamic video clip using the optical flow interpolation algorithm (RIFE) to simulate the effect of slow camera movement or camera movement, thus avoiding the stiffness of static images.

[0068] S4, Beat-Driven Automated Editing

[0069] This step uses the audio timing sequence as the sole time axis, performing editing and effects compositing frame by frame. A three-level duration adaptation strategy covers all scene material adaptation, ensuring precise alignment of all visual effects with audio beats and segments. The specific processing logic is as follows:

[0070] (1) Synchronized editing of strong drum beats

[0071] Five frames before each strong drum beat timestamp, a scaling animation is initiated, linearly enlarging the image from 1.0x to 1.2x. The camera then switches at the peak of the drum beat (frame 5), and linearly shrinks back to its original size for five frames afterward. The total animation duration is 10 frames (0.4 seconds). This design ensures that the visual impact peak perfectly coincides with the auditory drum beat peak, with a timing error of less than 1ms, achieving precise timing.

[0072] (2) Weak beat effect superposition

[0073] Three frames before and after each weak beat timestamp, a sinusoidal displacement is applied to the image in both horizontal and vertical directions, with a maximum displacement of 2 pixels, to create a micro-vibration effect. This enhances the rhythm and breathing feel of the image without causing visual discomfort.

[0074] (3) Three-level duration adaptive adaptation

[0075] To address the difference between the length of source material and the required length of paragraphs, a tiered adaptation strategy is adopted:

[0076] ① Footage length > Demand length: Use a saliency detection algorithm to calculate the visual focus area of ​​each frame of the video, select the continuous segment that contains the most complete visual focus and has the smallest motion amplitude for cropping and retention, and avoid cropping out the main body of the image;

[0077] ②The length of the material is 80%~120% of the required length: The TimeStretch time stretching algorithm based on optical flow is adopted. Intermediate frames are generated by RIFE frame interpolation. While maintaining the smooth movement of the main body of the picture, the video frame sequence is non-linearly stretched / compressed. The stretching coefficient is dynamically calculated according to the ratio of the audio length of the current segment to the original length of the material.

[0078] ③ Footage duration < 80% of the required duration: After completing the AIGC footage, multiple footage are spliced ​​together. If there are still gaps, spare empty footage is called in a loop. A Ping-Pong loop mode with forward and reverse directions is used. The loop connection is achieved by fading in and out with 0.3 seconds to achieve a smooth transition and avoid abrupt changes.

[0079] For different emotional segments, additional adaptation rules are implemented: Climax segments prioritize fast-paced shot transitions, with each shot lasting 1-2 seconds, coupled with frequent, intense drumbeat transitions to enhance the fast-paced viewing experience; interludes prioritize 0.8x slow motion to create a soothing atmosphere. Global duration calibration: Using the total audio duration as the final benchmark, the playback speed of each segment is dynamically fine-tuned to ensure the final cut matches the audio duration perfectly, with a time error not exceeding one frame.

[0080] (4) Special effects layer synthesis

[0081] Particle effects, vibration effects, transition effects, etc., are layered and composited, with the effect layers blended using an alpha channel to ensure that the color and detail of the original image are not destroyed. Transition areas use an alpha blending formula to achieve smooth transitions.

[0082]

[0083] in, The transition coefficient decreases linearly from 1.0 to 0. , These are the pixel values ​​for the front and rear cameras, respectively.

[0084] S5, Mood-Adaptive Color Grading and Final Output

[0085] This step deeply integrates musical mood with color grading strategies, enabling differentiated color grading and smooth transitions between different segments. An error tolerance mechanism ensures the stability of mass production, ultimately outputting a finished product file that meets broadcast standards.

[0086] (1) Adaptive color correction for different segments

[0087] Based on the mood paragraph markers, load the corresponding color scheme for different paragraphs:

[0088] Climax Section: A high-saturation, high-contrast LUT color curve is applied, increasing saturation by 20%~25%, contrast by 15%~20%, darkening shadows by 10%, and brightening highlights by 5%, enhancing visual impact. Stylized color shifts are also applied: a warm red shift is applied to the traditional Chinese style template, a cool blue shift is applied to the tech template, and the fresh template maintains its natural color tone.

[0089] Interlude / lyrical passage: Reduce saturation by 10%~15%, overlay a cyan-orange softening filter, shift dark areas towards cyan and bright areas towards orange, reduce overall contrast by 10%, and create a soft and soothing visual atmosphere.

[0090] Normal verses: A neutral color grading scheme is used, with only minor adjustments to contrast and saturation of less than 5% to maintain a natural look.

[0091] Transition processing: At the junction of the climax and the interlude, a 1-second linear transition interval for color grading parameters is set to achieve a smooth switch of color tone and avoid obvious color abrupt changes.

[0092] (2) Standardized output of finished products

[0093] The video is encapsulated using H.264 encoding with a High4:2:2 profile. The default output resolution is 3840×2160 (4K UHD), the frame rate is 25fps, and the bitrate is continuously adjustable from 10 to 25Mbps, with a default setting of 20Mbps. It supports both CBR (Constant Bitrate) and restricted VBR (Video Bitrate) control strategies. Audio uses AAC encoding at a bitrate of 320kbps, maintaining a sampling rate of 48kHz, and the output file format is standard MP4. The system supports scene-based adaptive bitrate: automatically adapting to 15Mbps when selecting the "Broadcast New Media" scene, 25Mbps when selecting the "Offline Large Screen" scene, and 10Mbps when selecting the "Network Transmission" scene. All output parameters conform to the broadcast and playback specifications of the corresponding scene.

[0094] (3) Industrialization anomaly tolerance mechanism

[0095] A three-tiered fallback strategy is implemented to address high-frequency anomaly scenarios in mass production:

[0096] Audio parsing fault tolerance: If the input audio has no obvious beat features (such as pure human speech), the system automatically downgrades to semantic sentence segmentation driven mode, calls the ASR speech recognition model to extract semantic pause points as camera switching nodes, and ensures that the process is not interrupted.

[0097] AIGC generation fault tolerance: If the AIGC model response times out (>30 seconds) or the generation quality is substandard, the system automatically extracts alternative materials of the same style from the pre-set general empty lens library and marks them on the timeline with labels to be manually replaced, ensuring the continuous operation of the overall pipeline;

[0098] Encoding overflow tolerance: If a sudden change in local frame motion is detected, causing the instantaneous bitrate to exceed the threshold, the system automatically increases the proportion of B-frames in the GOP structure, fine-tunes the QP quantization parameters, and forces the bitrate to be suppressed within the allowable range of the specification.

[0099] 2. An Audio Timing-Driven AIGC Fully Automated Video Generation System

[0100] Corresponding to the above method, this system includes five functional units, and the functions and data flow of each unit are as follows:

[0101] The audio multidimensional parsing unit is responsible for receiving input audio files, performing preprocessing, STFT transformation, frame energy calculation, strong and weak drum beat determination, MFCC feature extraction, emotional segment division, and finally outputting a standardized time-series marker sequence.

[0102] Audio-visual mapping storage unit: Used to store audio-visual mapping templates of various styles, providing template query, loading, adding and modifying interfaces, and providing special effect parameter configuration for the editing unit;

[0103] AIGC Material Scheduling and Completion Unit: Responsible for the tagging management of the material library, material matching and retrieval, and sufficiency judgment. When there is a shortage, it calls the image generation model to complete controlled generation and multi-step standardized preprocessing, and outputs a sequence of usable materials to the editing unit.

[0104] Automatic Beat Editing Unit: Based on the timing marker sequence, it performs shot scheduling, transition compositing, special effects overlay, slow motion processing, and duration calibration, and outputs the composited video sequence.

[0105] Mood Color Grading Output Unit: Responsible for loading the corresponding LUT and color grading parameters, performing segmented color grading and transition processing, with an exception tolerance mechanism, and finally completing the encoding and encapsulation to output a standard finished product file.

[0106] 3. Supplement the topic

[0107] The present invention also includes a computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor executes the computer program to implement the steps of the fully automated film generation method described above; and a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the fully automated film generation method described above.

[0108] Compared with the prior art, the present invention has the following significant technical advancements and beneficial effects:

[0109] 1. Layered high-precision audio analysis with industry-leading rhythm alignment accuracy. This invention breaks through the limitations of traditional binary drum beat detection, employing a local adaptive threshold algorithm to classify strong and weak beats, and combining MFCC feature change rate to achieve multi-emotional segment division. Adjustable window parameters allow for adaptation to various musical styles and background music. Actual testing shows that drum beat alignment accuracy reaches over 98%, and segment division accuracy exceeds 96%, far surpassing existing general-purpose automatic editing tools. Simultaneously, different intensity beats are matched with differentiated visual effects, resulting in a layered rhythm in the final cut, significantly enhancing professionalism and avoiding the harshness of simple, hard cuts.

[0110] 2. End-to-end unmanned closed-loop system with controllable AIGC completion quality. This invention deeply integrates AIGC material generation into the automatic editing pipeline, constructing a complete closed-loop mechanism of "material scheduling—missing material classification judgment—controlled intelligent completion—standardized post-processing—continued editing." Through contextual feature extraction, ControlNet composition control, LoRA style alignment, and multi-step post-processing, it ensures the integration of generated materials with actual footage, completely solving the problem of process interruption caused by material shortages. From inputting background music to outputting the final film, the entire process requires no manual intervention. The generation time for a single 60-second short film can be controlled within 10 minutes, improving efficiency by more than 95% compared to traditional manual editing, making it especially suitable for scenarios involving the mass production of hundreds of short films in the same series.

[0111] 3. Emotion-Linked Color Grading and Hierarchical Effects System for Professional and Consistent Visual Quality in Finished Work. This invention introduces the dimension of musical emotion into the automatic editing process. By identifying emotional segments and matching them with dedicated color grading LUTs and hierarchical visual effects, smooth transitions between segments are achieved, solving the pain points of messy color tones and mismatched emotions in traditional automatic editing. The generated finished work has a consistent style and accurate emotional expression, achieving professional-grade visual effects without the need for manual secondary color correction, significantly raising the quality ceiling of automatically edited content.

[0112] 4. Broadcast-grade standardized output, directly usable in multiple scenarios. This invention strictly follows broadcast specifications in designing output parameters, supporting 4K resolution, professional encoding formats, and multiple adaptive bitrates. The output files can be directly used in professional scenarios such as broadcast new media releases, offline large-screen playback, and television channel broadcasts, without the need for secondary transcoding and format adjustments, thus bridging the last mile from automatic production to professional broadcasting.

[0113] 5. Modular and scalable architecture with strong industrial robustness. This invention adopts a modular design, where style templates, special effects parameters, and output specifications are all configurable, supporting the customization and expansion of new style templates and material tags; it is equipped with a three-level exception tolerance mechanism to cope with various abnormal scenarios in batch production. Furthermore, the overall solution is built on common open-source components such as FFmpeg, OpenCV, and open-source image generation models, allowing enterprises to quickly deploy and build their own content production platform, adapting to the short film production needs of multiple fields such as culture and tourism, government affairs, enterprises, and food. Attached Figure Description

[0114] Figure 1 This is a block diagram of the overall architecture of the audio timing-driven AIGC fully automatic video generation system described in this invention, showing the composition of the five functional units and the data flow relationship;

[0115] Figure 2 This is a schematic diagram of the overall process of the audio timing-driven AIGC fully automatic video generation method described in this invention, showing the complete step chain from audio input to video output;

[0116] Figure 3 This is a flowchart of the detailed processing steps of the audio multi-dimensional timing analysis in this invention, showing the complete sub-process from preprocessing to output timing markers;

[0117] Figure 4 This is a flowchart illustrating the judgment and execution process of intelligent material scheduling and AIGC completion in this invention, showing the branch logic of material matching, sufficiency judgment, and completion generation. Detailed Implementation

[0118] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are only used to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0119] Example 1: Fully Automated Generation of Promotional Videos for Chinese-Style Urban Cultural Tourism

[0120] This embodiment uses a 60-second traditional Chinese style urban cultural tourism promotional video as an application scenario to verify the fully automatic generation effect of the present invention in the cultural tourism promotion scenario.

[0121] 1. Operating environment configuration

[0122] Hardware server configuration: AMD EPYC7B12 64-core CPU, NVIDIA T4 16GB graphics card, 32GB DDR4 memory, 1TB NVMe solid-state drive;

[0123] Operating system: CentOS 7.9 64-bit;

[0124] Dependencies: FFmpeg 5.1.2 audio and video processing library, OpenCV 4.5.5 computer vision library, StableDiffusionXL 1.0 image generation model, Librosa 0.10.0 audio processing library, RIFE optical flow frame interpolation model, and Python 3.9 runtime environment.

[0125] 2. Input parameter settings

[0126] Input audio: 48kHz sampling rate, 16bit stereo WAV format Chinese style orchestral background music, total duration 60s, BPM=120, including intro, two verses, two choruses, and outro structure.

[0127] Material Library: A localized material library with tags, containing four categories of traditional Chinese style materials: ancient building eaves, ancient city street scenes, city night scenes, and landscape scenery. The total usable time is 42 seconds, including 15 seconds of ancient building materials, 18 seconds of city night scenes, and 9 seconds of landscape scenery.

[0128] Style Template: Chinese Style Template;

[0129] Output configuration: 4K (3840×2160) resolution, 20Mbps bitrate, H.264 High4:2:2 encoding, MP4 container.

[0130] 3. Complete execution process

[0131] (1) Multi-dimensional temporal analysis of audio

[0132] First, the input audio is preprocessed using a multi-dimensional audio analysis unit: peak normalization is performed to uniformly map the audio amplitude to the [-1,1] interval, eliminating differences in sound source volume. A Hanning window is used for short-time Fourier transform, with a window length of 50ms (2400 sampling points) and a frame shift of 20ms (960 sampling points), calculating the spectral energy frame by frame. A local adaptive thresholding method is employed, calculating the local energy mean using a 101-frame sliding window. A threshold coefficient λ=3.0 is set according to the style of traditional Chinese orchestral music, and strong and weak drum beats are determined according to rules. The calculation identifies 120 strong drum beats and 120 weak beats, evenly distributed throughout the entire track; after filtering with a 30ms minimum interval, there are no duplicate or misjudged frames. 13-dimensional MFCC features are extracted, and the average feature change rate is calculated using a 3s sliding window. Two climax sections are identified: 16s~26s and 38s~50s; the remaining intervals are marked as interludes (including the intro, verse, and outro). The final output is a timing mark sequence with millisecond precision, containing all drum timestamps and segment start and end marks, which serves as the sole time reference for subsequent editing.

[0133] (2) Audio-visual style mapping loading

[0134] Loading parameters corresponding to the Chinese style template into the audio-visual mapping storage unit:

[0135] Strong drum beats: 0.2s zoom-in hard cut, maximum magnification 1.2x, overlaid with 1 frame of ink-wash flash white;

[0136] Weak beat: 0.5% width sinusoidal micro-vibration, lasting 0.1s;

[0137] Climax: Warm gold radial particles, 250 particles, 30% opacity, 15% sharpening;

[0138] Interlude: 0.8x slow motion, 0.3s fade-in / fade-out transition.

[0139] (3) Material scheduling and AIGC completion

[0140] The AIGC material scheduling and completion unit matches materials according to the needs of the storyboard: This promotional video is divided into three segments, corresponding to three types of scenes: ancient architecture, city night scene, and landscape scenery. Each segment is required to be 20 seconds long.

[0141] Ancient architecture scene: 15 seconds of available material, accounting for 75% of the requirement. If less than 80%, the completion will be triggered, and 5 seconds of material needs to be added.

[0142] Urban night scene: 18 seconds of usable footage, accounting for 90% of the requirements, is considered basically sufficient, and will be adapted to slow motion later;

[0143] Landscape scene: 9 seconds of available material, accounting for 45% of the requirement, triggering completion, requiring 11 seconds of additional material.

[0144] After triggering the AIGC completion process, the system extracts the warm yellow tone features of adjacent shots and dynamically generates prompt words:

[0145] Ancient building completion prompts: "Chinese-style ancient architecture, flying eaves and brackets, ancient city skyline, soft sunlight, cinematic quality, high-definition realism, no people, empty shots"; negative prompts: "people, faces, hands, text, watermark, blur and distortion".

[0146] The prompts for completing the landscape description are: "Traditional Chinese landscape, green mountains and clouds, winding rivers, morning mist, cinematic light and shadow, high-definition realism, empty shots, no people"; negative words are the same as above.

[0147] Using a local SD-XL model and ControlNet depth-controlled mapping, static images were generated at a density of 1 frame every 2 seconds. Three frames were generated for the ancient architecture, and six frames for the landscape, all at a resolution of 1024×576. After generation, a three-step normalization process was performed: sRGB was converted to the Rec.709 color space using an ICC profile; Real-ESRGAN super-resolution was used to upscale to 4K resolution; and the RIFE optical flow frame interpolation algorithm was used to generate 25fps dynamic video to simulate a slow zoom-in effect. The final result was a 5-second segment with completed ancient architecture and a 12-second segment with completed landscape, fully meeting the segment length requirements, and with a color style highly consistent with the actual footage.

[0148] (4) Beat-driven automated editing

[0149] The beat-based automatic editing unit uses the timing marker sequence as its main axis to perform video compositing frame by frame:

[0150] Strong drum beat processing: All 120 strong drum beats are executed synchronously with a "5-frame zoom--slice-5-frame zoom-back" animation, with the slice point perfectly aligned with the peak frame of the drum beat, and the time error is less than 1ms;

[0151] Weak beat processing: All weak beat frames are superimposed with a 2-pixel sinusoidal micro-vibration, with a smooth transition of 3 frames before and after;

[0152] Climax: Overlay a warm gold radial particle layer, with particle movement speed dynamically changing with frame energy; the higher the energy, the faster the speed; single shot duration is controlled at around 1.5 seconds, combined with dense transitions of strong drum beats;

[0153] Interlude: The city night scene footage was stretched to 20 seconds using 0.8x slow motion, and optical flow frame interpolation was used to ensure smoothness; a 0.3-second fade-in and fade-out transition was used at the junction of the supplementary footage and the original footage;

[0154] Global calibration: The final total video duration is perfectly aligned with the audio, with an error of 0 frames.

[0155] (5) Mood-adaptive color correction and output

[0156] The mood color grading output unit loads color grading configurations by paragraph:

[0157] Climax: Load a high-saturation LUT with Chinese style, increase saturation by 22%, increase contrast by 18%, darken shadows by 10%, and overlay a warm red bias (R channel +5, G channel -2, B channel -3).

[0158] Interlude: Saturation reduced by 12%, overlaid with a cyan-orange softening filter, contrast reduced by 10%;

[0159] At paragraph transitions: Set a 1-second linear transition interval, with smooth gradient color parameters and no color abrupt changes.

[0160] The final encoding and encapsulation are: H.264 High 4:2:2 encoding, 3840×2160 resolution, 20Mbps bitrate, 25fps, audio AAC 320kbps, and the output file size is about 150MB, which fully complies with the broadcasting standards of new media.

[0161] The test results verified that 50 sets of different styles of traditional Chinese cultural tourism background music were tested in batches using the same configuration. The statistical results are as follows: the accuracy rate of strong drum beat alignment is 98.2% on average, the accuracy rate of emotional segment division is 96.5%, and all finished videos can be published directly without manual adjustment of beats and color correction; the average generation time of a single short video is 8.7 minutes, which is about 96.4% more efficient than the average 4 hours of traditional manual editing.

[0162] Example 2: Fully Automated Generation of Brand Promotional Videos for Technology Companies

[0163] This embodiment verifies the adaptability of the present invention in the context of promotional videos for technology companies.

[0164] 1. Input and Configuration

[0165] Input audio: Tech-style electronic music, 48kHz WAV format, 90s in length, BPM=140;

[0166] Material Library: Contains four categories of technology materials: urban technology landmarks, office scenes, chip circuits, and data visualization, with a total duration of 65 seconds;

[0167] Style Template: Technology Template;

[0168] Output configuration: 4K resolution, 25Mbps bitrate, suitable for offline large-screen playback scenarios.

[0169] 2. Parameter adaptation and adjustment

[0170] For high-tempo electronic music at 140 BPM, the STFT window length was adjusted to 40 ms and the frame shift was 15 ms to improve temporal resolution; the threshold coefficient λ was set to 3.5 to filter out false positives for weak energy.

[0171] 3. Generation effect

[0172] After processing, the accuracy rate of strong drum beat recognition reached 97.8%, and the climax segment was accurately divided. Under the technology template, the strong drum beat was superimposed with a wireframe sweep effect, and the climax segment was superimposed with ice blue particles. The color grading adopted a cool blue high-contrast style. For the missing material, a technology-themed urban aerial shot and a data visualization aerial shot were automatically generated. The cyberpunk technology style was aligned with the LoRA model, and the integration with the live-action material reached more than 92%. The final output was a 4K video with a bitrate of 25Mbps, which can be directly used for large-screen playback at corporate press conferences. The single video generation time is about 12 minutes.

[0173] Example 3: Fully Automated Generation of Promotional Short Videos for Refreshing Food

[0174] This embodiment verifies the adaptability of the present invention in the context of refreshing food short videos.

[0175] 1. Input and Configuration

[0176] Input audio: Fresh folk background music, 48kHz WAV format, 30s in length, BPM=90;

[0177] Footage library: Contains three categories of refreshing footage: food close-ups, pastoral scenery, and ingredients, with a total duration of 18 seconds;

[0178] Style Template: Fresh and clean template;

[0179] Output configuration: 1080P resolution, 10Mbps bitrate, suitable for new media network dissemination.

[0180] 2. Parameter adaptation and adjustment

[0181] For soothing folk songs at 90 BPM, the STFT window length was adjusted to 60 ms and the frame shift to 25 ms to improve frequency resolution accuracy; the threshold coefficient λ was set to 2.8 to ensure sensitivity in weak beat recognition.

[0182] 3. Generation effect

[0183] After processing, the drum beat alignment accuracy is 98.5%, and the slow motion effect of the interlude is natural; the fresh template uses soft light flash white and white particles, and the color grading is soft and natural; the missing material parts are automatically generated to create fresh food and pastoral empty shots; the final output is a 1080P, 10Mbps finished product with small file size and clear picture quality, which can be directly published on short video platforms, and the generation time of a single video is about 5 minutes.

[0184] Example 4: Verification of Industrial-Scale Fault-Tolerant Operation

[0185] This embodiment verifies the fault tolerance capability under mass production scenarios. Three types of abnormal scenarios are simulated:

[0186] 1. Input pure human voice government report audio, without obvious drum beats: The system automatically triggers ASR semantic recognition, uses semantic pauses as cut points, and generates the finished video normally without interruption.

[0187] 2. AIGC interface timeout of 35 seconds: The system automatically calls the same style material from the general empty lens library to fill the missing paragraph, and marks the position to be replaced in the output log. The overall production pipeline runs normally.

[0188] 3. High-speed motion camera encoding bitrate overflow: The system automatically adjusts the number of B frames and QP value to suppress the peak bitrate to within the 25Mbps threshold, and the output file meets the broadcast specifications without screen tearing or stuttering.

[0189] After 100 sets of abnormal stress tests, the overall system stability rate reached 99.2%, fully meeting the robustness requirements for industrial mass production.

[0190] Example 5: Implementation of a Computer-Readable Storage Medium

[0191] This embodiment provides a computer-readable storage medium, which is a non-volatile storage medium, specifically any one of ROM, RAM, magnetic disk, optical disk, USB flash drive, and SD card. The storage medium stores a computer program, which, after being loaded and executed by a processor, implements all the method steps S1 to S5 of this invention. This storage medium can be sold independently or bundled with system hardware, facilitating the industrialization and application of this invention.

[0192] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fully automated audio timing-driven AIGC (AIGC) video generation method, characterized in that: Includes the following steps: S1. Audio Multidimensional Analysis and Emotional Segmentation: After standardizing the input background music, the short-time Fourier transform is performed to calculate the spectral energy of a single frame. The local adaptive threshold method is used to classify the strong and weak drum beats. At the same time, based on the rate of change of the Mel frequency cepstral coefficient, three emotional segments are divided into climax, interlude and normal verse. The audio time-series marker sequence with millisecond-level timestamps is output. S2, Audio-visual style mapping rule matching: Load the preset structured audio-visual mapping template, and match different audio timing events with the corresponding transition styles, special effects parameters and color grading tendencies. The template contains at least three vertical domain configurations: Chinese style, technology, and fresh. S3, Intelligent scheduling of storyboard material library and AIGC material completion: Match the tagged material library by paragraph, and make a three-level sufficiency judgment based on the ratio of the total material duration to the paragraph's required duration. When it is determined that the material is in short supply, AIGC intelligent completion is automatically triggered to generate supplementary material of the same style and perform standardized post-processing. S4, Beat-Driven Automated Editing: Using the audio timing marker sequence as the sole time axis, it synchronously executes strong drum beat editing, weak beat effect overlay, and segment duration adaptive adaptation to complete shot splicing and effect layer compositing. S5, Adaptive Color Grading and Final Output: Performs segmented color grading and smooth transitions based on emotional segment markers, with an industrial-grade fault tolerance mechanism, and finally encodes and outputs a final output file that meets broadcast standards.

2. The fully automated audio timing-driven AIGC video generation method according to claim 1, characterized in that: In step S1, the formula for calculating the short-time Fourier transform is: in, It is an audio time-domain signal. For the Hanning window function, For frame index, For frequency index, The number of sampling points corresponding to the window length. This represents the number of sampling points corresponding to the frame shift. The formula for calculating the spectral energy of a single frame is: in, For the first Frame number The short-time Fourier transform complex amplitude values ​​at each frequency point; the local adaptive thresholding method specifically involves calculating the local energy mean of 101 frames before and after the current frame. Set threshold coefficient The value range is [2.5, 4.0]. And when the local maximum condition is met, it is determined to be a strong drum beat frame. It was determined to be a weak beat frame.

3. The audio timing-driven AIGC fully automatic video generation method and system according to claim 2, characterized in that: In step S1, the formula for calculating the single-frame feature change rate based on Mel frequency cepstral coefficients is as follows: in, Let be the 13-dimensional Mel-frequency cepstral coefficient eigenvector of the i-th frame. The L2 norm of the vector is represented; the average feature change rate within the window is calculated using a sliding window of 3 seconds, and the division of the three emotional segments into climax, interlude, and regular verse is completed by combining the proportion of low-frequency energy below 80Hz.

4. The fully automated audio timing-driven AIGC video generation method according to claim 1, characterized in that: In step S3, the three-level sufficiency judgment rule is as follows: if the total duration of the matched material is greater than or equal to 120% of the duration required for the segment, it is considered that the material is sufficient; if it is between 80% and 120%, it is considered that the material is basically sufficient; and if it is less than 80%, it is considered that the material is insufficient. AIGC intelligent completion sequentially performs context feature extraction, standardized prompt word construction, controlled image generation, color space unification, super-resolution enhancement, and optical flow dynamic frame interpolation processing to ensure that the color space and image quality style of the generated material are consistent with those of the actual shooting material.

5. The fully automated audio timing-driven AIGC video generation method according to claim 1, characterized in that: In step S4, the transition area of ​​the special effects layer is smoothly connected using an Alpha blending formula, the calculation formula of which is: in, The transition coefficient decreases linearly from 1.0 to 0. , These are the pixel values ​​corresponding to the coordinates of the front and rear shots, respectively; the segment duration adaptive adaptation adopts a three-level strategy: when the material duration is longer than the required duration, saliency detection and cropping adaptation is used; when the duration is 80%~120% of the required duration, optical flow time stretching adaptation is used; when the duration is less than 80% of the required duration, material completion splicing and loop filling adaptation are used.

6. The fully automated audio timing-driven AIGC video generation method according to claim 1, characterized in that: The industrial anomaly tolerance mechanism in step S5 includes a three-level fallback strategy: audio parsing fault tolerance, which automatically downgrades to semantic segmentation-driven mode when the input audio has no obvious beat features; AIGC generation fault tolerance, which calls a pre-set general empty mirror library to replace the filling when an anomaly is generated; and encoding overflow fault tolerance, which automatically adjusts the GOP structure and quantization parameters when the instantaneous bit rate exceeds the threshold.

7. An audio timing-driven AIGC fully automated video generation system, characterized in that, include: The audio multi-dimensional analysis unit is used to receive input audio, perform preprocessing, spectrum transformation, strong and weak drum beat classification and emotional segmentation, and output a standardized time-series marker sequence. The audio-visual mapping storage unit is used to store audio-visual mapping templates for various vertical fields, and provides interfaces for querying, loading and extending special effects parameters and color grading configurations; The AIGC material scheduling and completion unit is used for tagged material library management, material matching and retrieval and sufficiency judgment. When materials are scarce, it automatically performs AIGC completion generation and standardization post-processing to output a sequence of available materials. The beat-based automatic editing unit is used to perform beat-matching, effects overlay, and duration adaptive adaptation with the time-marked sequence as the main axis, and to complete shot splicing and effects layer compositing. The emotional color grading output unit is used to perform segmented emotional color grading and smooth transitions, with an auxiliary fault tolerance mechanism, and completes encoding and encapsulation to output finished video files that comply with broadcast standards.

8. The audio timing-driven AIGC fully automated video generation system according to claim 7, characterized in that, The audio multidimensional analysis unit has a built-in short-time Fourier transform calculation module, a local adaptive threshold determination module, and a Mel frequency cepstral coefficient extraction module. It supports an adjustable window length of 40~60ms and an adjustable frame shift of 15~25ms, which can adapt to background music of different styles and BPMs. The drum beat alignment accuracy is no less than 98%.

9. The audio timing-driven AIGC fully automated video generation system according to claim 7, characterized in that, The AIGC material scheduling and completion unit has a built-in context feature extraction submodule, a controlled generation submodule, and a multi-step post-processing submodule; the emotion color grading output unit has a built-in segmented LUT loading module and a multi-scene adaptive bitrate encoding module, and the output parameters conform to the broadcast new media and offline large screen broadcast specifications.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the audio timing-driven AIGC fully automatic video generation method according to any one of claims 1 to 6.

Images