An AIGC card point clipping method and system

Abstract

The application provides an AIGC card point editing method and system, solves the problems of audio and picture splitting, hard transition, lack of global planning and inability to complete when the material is insufficient caused by the dependence of the prior art on discrete beat point editing. The method comprises the following steps: constructing an audio energy field and a visual energy field based on background audio and video material respectively; establishing a two-field coupled partial differential equation, solving a globally optimal editing path, obtaining an editing point sequence and a segment allocation scheme; for the interval with a matching error exceeding a threshold, calling an AIGC to generate a supplementary video segment, and generating a transition frame sequence according to the visual energy field jump amplitude at the editing point; dynamically adjusting the playing speed according to the ratio of the energy field time gradient; finally, splicing and fusing the variable speed segment, the transition frame and the supplementary segment to output the card point video. The application realizes global optimization editing of continuous field coupling, and improves the card point precision and visual fluency.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and video processing technology, specifically an AIGC beat-sync editing method and system. Background Technology

[0002] Traditional video beat-sync editing primarily relies on audio beat detection technology. This involves extracting discrete beat points from the background music and then forcibly aligning the video clips' transitions to these beat moments to achieve audio-visual synchronization. Many existing automatic beat-sync editing tools and professional plugins employ this method. This approach first performs time-frequency analysis on the audio to identify energy peaks as beat points. Then, it arranges video clips sequentially, ensuring that the transitions between each clip precisely fall on these beat points. To enhance the sense of rhythm, users can also manually adjust clip lengths or add transition effects.

[0003] However, existing technologies have the following significant shortcomings: First, they rely solely on discrete beat points for hard cuts, ignoring the continuous dynamic energy changes in music, such as crescendo, diminuendo, rests, and vibrato. This results in edited videos that, while always aligned, lack an intrinsic connection between visual movement rhythm and the emotional fluctuations of the music, leading to a disconnect between "auditory excitement and visual flatness" or "auditory soothing and visual disorder." Second, beat detection algorithms experience a significant drop in accuracy when dealing with complex and rhythmically varied music (such as jazz, classical, and live recordings), failing to guarantee precise beat matching. Third, video transitions often involve hard cuts or fixed template transitions, such as fade-in / fade-out and slides, without considering the continuity of movement between preceding and following frames, easily creating a sense of visual jumpiness and affecting the viewing experience. Fourth, existing methods typically splice together footage in sequence or randomly, failing to dynamically allocate footage with different movement styles based on the overall structure of the music, resulting in a monotonous "slideshow" effect in the final product. Fifth, when the amount or content of materials provided by users cannot meet the energy distribution requirements of music, existing technologies lack effective means of supplementation, and users can only manually supplement materials or accept imperfect editing results.

[0004] In summary, existing technologies urgently need an intelligent beat-matching editing method that can deeply couple the continuous dynamic energy of audio with the visual motion energy of video, achieve global optimized editing planning, and automatically generate supplementary content when there is insufficient material. Summary of the Invention

[0005] This application provides an AIGC beat-matching editing method and system, which solves the technical problems in the prior art that result in the disconnect between audio rhythm and visual motion, abrupt transitions, inability to adapt to complex musical expression changes, lack of global editing planning, and lack of effective means of completion when existing materials cannot match the audio energy distribution, due to relying solely on discrete beat points for hard video cutting.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, an AIGC beat-sync editing method is provided, including: Based on the acquired background audio signal to be edited and video clips, an audio energy field and a visual energy field are constructed respectively; the energy field is used to characterize the energy distribution of audio or video at different spatiotemporal locations; Establish a coupled partial differential equation between the audio energy field and the visual energy field, solve the globally optimal editing path based on the coupled partial differential equation, and obtain the editing point sequence and the video segment allocation scheme corresponding to each time interval; For the time intervals existing in the editing path, if the matching error between the visual energy field and the audio energy field exceeds a preset threshold, the AIGC generative model is called to generate supplementary video segments, and at each editing point, a transition frame sequence is generated according to the jump amplitude of the visual energy field before and after. Based on the ratio of the temporal gradient of the audio energy field to that of the visual energy field, the playback speed of the video segment is dynamically adjusted, and the dynamically adjusted video segment, the transition frame sequence, and the supplementary video segment are spliced ​​and merged to output a beat-matched edited video.

[0007] Based on the above technical solutions, the AIGC beat-matching editing method provided in this application models audio and video as continuous spatiotemporal energy fields rather than discrete beat points, enabling more nuanced capture of the dynamic changes in musical rhythm and the energy distribution of visual motion. This overcomes the auditory-visual disconnect caused by traditional beat-matching editing that relies solely on hard cuts at beat moments. By establishing coupled partial differential equations between the audiovisual energy fields and solving for the globally optimal editing path, global coordination of editing point positions and segment allocation is achieved, rather than greedy local alignment, thereby improving the overall rhythmic coherence and narrative rationality of the editing result. Furthermore, by controlling the intervals where the matching error exceeds a threshold... This application utilizes an AIGC generative model to supplement video clips, filling in gaps where existing materials cannot match the audio energy distribution. This avoids the rhythmic discontinuity caused by forced splicing when materials are insufficient, a common problem in traditional methods. By generating transition frame sequences at each editing point based on the amplitude of changes in the visual energy field before and after, smooth transitions between shots are achieved, matching the music energy and improving the naturalness and visual fluency of transitions. By dynamically adjusting the video playback speed based on the ratio of the temporal gradients of the audio and visual energy fields, real-time tracking of the rhythm of the visual movement to the instantaneous energy changes in the music is achieved, ensuring that the beat-matching is not only reflected in time alignment but also in the synchronization of the movement rhythm. In summary, this application upgrades audio-driven editing from discrete event triggering to global optimization with continuous field coupling. Combined with AIGC generative completion and dynamic speed adjustment, it significantly improves the rhythmic accuracy, visual fluency, and creative intelligence of beat-matched videos.

[0008] Furthermore, the process of constructing the audio energy field includes: The constant Q transform method is used to extract the constant Q transform spectrum of the background audio signal, and the complex spectrum S(f,t) of the background audio signal is obtained; where F is the frequency and t is the time. Calculate the energy distribution function at each time t based on the complex spectrum. ; The energy distribution function is compressed along the frequency axis to a one-dimensional latent coordinate x by a one-dimensional convolutional encoder to obtain the intermediate field q(x,t); wherein the one-dimensional convolutional encoder consists of multiple convolutional layers and a global normalization layer, and the training of the one-dimensional convolutional encoder adopts self-supervised reconstruction loss. Gaussian smoothing and exponential decay in the time domain are applied to the intermediate field to obtain the audio energy field ρ. a (x,t); where the x-axis of the audio energy field is discretized into multiple equally spaced intervals.

[0009] Furthermore, the process of constructing the visual energy field includes: Extract frame sequences from video clips at a preset sampling frequency; Multidimensional features are extracted from each frame in the frame sequence; the multidimensional features include dense optical flow amplitude field, optical flow curl field, attention heatmap, and scene background difference map; wherein, the dense optical flow amplitude field is obtained by RAFT optical flow algorithm, the optical flow curl field is obtained by calculating the curl of the dense optical flow length, the attention heatmap is obtained based on visual neural network, and the scene background difference map is obtained by normalizing the brightness of the current frame and the previous frame, then performing absolute difference and thresholding; The multidimensional features are concatenated along the channel dimension and then input into a two-dimensional convolutional encoder. A pre-trained variational autoencoder is then used to compress the output of the two-dimensional convolutional encoder to a one-dimensional variable x. The encoder output of the variational autoencoder is the mean and variance. Then, the product of random noise following a standard normal distribution and the variance is added to the mean and sampled to obtain the one-dimensional variable x. Using the one-dimensional variable x(t) obtained from sampling at each time step as the kernel center, a Gaussian function is used to estimate the conditional kernel density, thus obtaining the visual energy field.

[0010] Furthermore, the process of acquiring the dense optical flow amplitude field is as follows: For each frame in the video footage and the next adjacent frame, the RAFT optical flow estimation network algorithm is used to calculate the optical flow vector field (u(x,y), v(x,y)), where u and v represent the optical flow components in the horizontal and vertical directions, respectively. The optical flow amplitude is calculated for each pixel position (x,y) to obtain an amplitude map of the same size as the original frame, namely a dense optical flow amplitude field, which is used to characterize the intensity of local motion in the image.

[0011] Furthermore, the process of obtaining the optical flow curl field is as follows: After obtaining the dense optical flow vector field (u,v), its curl is calculated. ; The optical flow curl field is obtained; where / x and / y represents the partial derivatives with respect to the x and y directions of the image space coordinates, respectively, and is numerically approximated using the Sobel operator; the curl field reflects the local rotation intensity of the flow field and is used to characterize complex motion patterns such as object rotation and vortex in the image.

[0012] Furthermore, the process of obtaining the attention heatmap is as follows: A pre-trained visual Transformer model is used to process each frame image I(t), and the average attention weight of the [CLS] token to all image block tokens in the last self-attention module is extracted. The average attention weights are rearranged according to the spatial position of the image blocks and upsampled to the original image size to obtain an attention heatmap; the value of each pixel in the attention heatmap represents the importance of that region to the overall scene semantic understanding, and is used to identify the moving subject region in the image.

[0013] Furthermore, the process of obtaining the scene background difference map is as follows: Histogram equalization is performed on consecutive frames I(t) and I(t-1) of the video footage; Calculate the absolute difference map between the two equalized frames. ; The absolute difference map is Gaussian blurred and binarized to obtain the scene background difference map D(t); the scene background difference map marks the distribution of the moving foreground region and the static background region, which is used to help determine whether there is a significant scene change or subject movement in the picture.

[0014] Furthermore, the coupled partial differential equation is: ;in, For visual energy field, Let be the audio energy field, t be time, x be a one-dimensional feature coordinate, and u be the energy flow velocity vector obtained by least-squares fitting of the optical flow sequence of the video clip. Energy dispersion represents the flux change of visual energy along the x-axis. is the coupling coefficient, used to control the driving strength of the audio energy field on the visual energy field.

[0015] Furthermore, the step of solving the globally optimal clipping path based on the coupled partial differential equations specifically includes: The coupled partial differential equation is transformed into residual form: ; Construct a minimum action integral objective function based on the principle of minimum action: Where p(t) is the cut point indicator function, which takes the value 1 at the cut point and 0 otherwise; λ is the sparsity regularization coefficient, used to control the number of cut points. The timeline of the total video playback duration is discretized into discrete moments at a preset sampling rate, and each video clip is segmented according to the discrete moments to obtain a set of candidate video clips; wherein, each candidate video clip includes a start time, an end time, the index of the video clip to which it belongs, and a sequence of consecutive frames extracted from the video clip; the start time and end time of each candidate video clip are discrete time points; The placement of each candidate video segment on the time axis is regarded as a state. The minimum value of the minimum action integral objective function is solved by dynamic programming to obtain the optimal cumulative loss value at each time step. Based on the optimal cumulative loss value at each time point, starting from the termination state, the material index that minimizes the cumulative loss value at the termination time is selected as the material for the last segment, and the predecessor state recorded in the dynamic programming for each state is determined. The process is then backtracked to the starting state to obtain the optimal editing path. The optimal editing path includes the sequence of editing point times and the video material segment indexes corresponding to each time interval.

[0016] Furthermore, the specific process for obtaining the clipping point sequence and the video segment allocation scheme includes: Set the initial cumulative loss value for each video clip. If video clip i is used at the initial time and the starting frame of the video clip is aligned with time t=0, then the initial cumulative loss value is set to 0; otherwise, it is set to positive infinity. For each discrete time point and each video clip index, define the state. Indicates at discrete time Find the minimum cumulative loss after playing a frame of video clip i, and define the state transition equation as follows: ;in, This indicates that the candidate video segments extracted from video clip j are placed within the time interval. Integral of field matching error during internal time, , Let i and j be discrete time points, and i and j be the indices of video clips. To compensate for the transient penalty constant during editing, This is an indicator function; it takes the value 1 when j ≠ i, and 0 otherwise. After calculating all states, take all material indices i at the termination time T. The minimum value is recorded as the corresponding video clip index. Then from the terminated state Start repeating the backtracking operation: find the predecessor state selected during the transition of the current state. The current state time Record it as a clipping point and divide the time interval Assign an index to the current video clip segment; then update the current state to... Continue backtracking until... Until then; the final edit point sequence is obtained. And the video clip index corresponding to each time interval, i.e., the video clip allocation scheme.

[0017] Furthermore, the matching error between the visual energy field and the audio energy field The calculation formula is: ;in, and These are the start and end times of the time interval, respectively. This represents the audio energy field of the background audio at time t and feature coordinate x. To align the visual energy field of the assigned video clip i to the time interval using linear time scaling. The visual energy field afterward.

[0018] Furthermore, the step of calling the AIGC generative model to generate supplementary video clips specifically includes: A video diffusion generation model conditioned on a target visual energy field is pre-trained. The video diffusion generation model adopts a variant structure of the latent diffusion model (SVD), and the backbone network of the video diffusion generation model is a 3D UNet, with conditional vector embeddings injected into the cross-attention layer. During training, the video diffusion generation model uses video segments extracted from a natural video dataset and their corresponding visual energy fields as conditional pairs. The ideal visual energy field target within the threshold interval is calculated based on the coupled partial differential equation; the threshold interval is the time interval during which the matching error exceeds a preset threshold. ; The visual energy field target is flattened along the x-axis into a multidimensional vector sequence, and encoded into the conditional vector of the video diffusion generation model through a multilayer perceptron. Randomly sampled Gaussian noise is used as a latent representation. The latent representation is gradually denoised according to the inverse process of the Denoising Diffusion Probability Model (DDPM). At the same time, guided by the conditional vector and the first and last frame constraints, the encoder of the video diffusion generation model generates the denoised latent representation. The first frame of the first and last frames is taken as the last frame of the real segment before the left endpoint, and the last frame of the first and last frames is taken as the first frame of the real segment after the right endpoint. The decoder of the video diffusion generation model decodes the denoised latent representation into a video frame sequence to obtain a supplementary video segment; wherein, during the generation process, the first frame is forced to be aligned with the last frame of the previous real segment through optical flow constraints, and the last frame is aligned with the first frame of the subsequent real segment through optical flow constraints.

[0019] Furthermore, the step of generating a transition frame sequence based on the jump amplitude of the preceding and following visual energy fields specifically includes: At each clipping point T, obtain the last frame of the previous clip. and the corresponding visual energy field and the first frame of the next segment and the corresponding visual energy field ; Based on the visual energy field of the last frame of the preceding segment and the visual energy field of the first frame of the following segment, the jump intensity at the clipping point is calculated using the following formula: Where T+ and T- represent the instants after and before the clipping point, respectively; Set the transition frame sequence length to , where ||·|| is the L2 norm, and L is restricted to between 4 and 16 frames; A spatiotemporal diffusion model is constructed, which uses the expected visual energy fields at the last frame of the preceding segment, the first frame of the following segment, and each intermediate time point as conditional information to generate a transition frame sequence through a conditional denoising process; wherein, the formula for calculating the expected visual energy field is: τ=0,1,2,……,L; the input of the spatiotemporal diffusion model is a random Gaussian noise tensor and the conditional information, and the output is a predicted transition frame sequence.

[0020] Furthermore, the desired visual energy field is obtained by using a linear interpolation method based on the visual energy field of the last frame of the segment preceding the clipping point. The visual energy field of the first frame of the next segment The expected visual energy field corresponding to each frame is calculated from 0 to L according to the transition frame number τ. The calculation formula is: , where L is the total length of the transition frame sequence, τ=0 corresponds to the last frame of the previous segment, and τ=L corresponds to the first frame of the next segment.

[0021] Furthermore, the spatiotemporal diffusion model employs a conditional denoising process to progressively predict intermediate frames from the noise; the model's loss function includes two terms: pixel mean square error loss. and field matching loss ,in To predict frames The visual energy field extracted from it.

[0022] Furthermore, the specific method for dynamically adjusting the playback speed of video segments is as follows: The time derivatives of the audio energy field and the visual energy field of the video clip are calculated in real time during playback. Based on the time derivative, the speed adjustment coefficient is calculated using the clip function. The calculation formula is as follows: Where r(t) is the velocity adjustment coefficient at time t. The time derivative of the audio energy field is... The time derivative of the visual energy field is... This is a preset constant used to prevent division by zero; When the speed adjustment coefficient is greater than 1, the playback speed is increased by uniformly deleting video frames at time t, and the deletion ratio is 1-1 / r(t); When the speed adjustment coefficient is less than 1, the playback speed is reduced by inserting intermediate frames between adjacent frames at time t using the RIFE-based optical flow frame interpolation method, and the number of inserted frames is (1 / r(t)-1)×the original number of frames.

[0023] Furthermore, the process of obtaining the beat-synced video is as follows: According to the aforementioned editing point sequence and video segment allocation scheme, the dynamically adjusted video segments and supplementary video segments are arranged sequentially in chronological order to obtain the initial spliced ​​video sequence; At each clipping point, a sequence of transition frames is inserted between adjacent segments of the initial spliced ​​video sequence to form a continuous video sequence; The Video-LLaVA model is used to evaluate each frame in the continuous video sequence to detect inconsistencies in subject appearance, lighting abrupt changes, or visual jumps at editing points. A diffusion-based image inpainting model is used to correct video frames with inconsistencies in subject appearance, a color transfer network is used to correct video frames with lighting abrupt changes, and video frames with visual jumps are corrected by adjusting the generation parameters of transition frames or adjusting the position of editing points to obtain a refined video. The refined video is encoded and output to obtain a beat-matched edited video.

[0024] Secondly, this application provides an AIGC beat-matching editing system, comprising: an energy field construction module, an editing path planning module, a video generation module, and a fusion output module; wherein, The energy field construction module is used to construct an audio energy field and a visual energy field based on the acquired background audio signal to be edited and video clips; the energy field is used to characterize the energy distribution of audio or video at different spatiotemporal locations; The editing path planning module is used to establish the coupled partial differential equation between the audio energy field and the visual energy field, solve the globally optimal editing path based on the coupled partial differential equation, and obtain the editing point sequence and the video segment allocation scheme corresponding to each time interval. The video generation module is used to, for the time interval existing in the editing path, if the matching error between the visual energy field and the audio energy field exceeds a preset threshold, call the AIGC generative model to generate supplementary video segments, and at each editing point, generate a transition frame sequence based on the jump amplitude of the visual energy field before and after. The fusion output module is used to dynamically adjust the playback speed of the video segment according to the ratio of the temporal gradient of the audio energy field to the visual energy field, and to splice and fuse the dynamically adjusted video segment, the transition frame sequence, and the supplementary video segment to output a beat-cut video.

[0025] Compared with the prior art, the beneficial effects of this application are: First, this application improves the accuracy and robustness of audio-visual feature extraction by constructing refined audio and visual energy fields. On the audio side, a constant Q-transform combined with a one-dimensional convolutional encoder and self-supervised reconstruction loss is used to efficiently extract the multi-scale time-frequency energy distribution of music and compress it into a one-dimensional latent space. This preserves key information about rhythm and timbre while reducing subsequent computational complexity. Gaussian smoothing and temporal attenuation of the energy field effectively suppress noise interference, making the energy change curve more consistent with auditory perception. On the video side, dense optical flow amplitude field, optical flow curl field, attention heatmap, and scene background difference map are extracted simultaneously to capture the intensity of local motion, complex motion patterns such as rotation / vortex, semantically relevant subject areas, and the distinction between foreground and background. Then, a variational autoencoder is used to compress the multi-dimensional features into a one-dimensional latent variable, and a continuous visual energy field is obtained through conditional kernel density estimation. This multi-dimensional and multi-level visual feature fusion strategy enables the visual energy field to delicately reflect different types of motion dynamics and scene changes in the video.

[0026] Secondly, this application overcomes the shortcomings of traditional beat-based editing, which relies solely on discrete beat points for local alignment, by establishing coupled partial differential equations between the audiovisual energy fields and using the principle of least action to solve for the globally optimal editing path. It transforms the problem of editing point selection and material allocation into a variational optimization problem in a continuous spatiotemporal domain, and uses dynamic programming to solve it efficiently. This yields the time sequence of editing points and the material allocation scheme corresponding to each time interval, ensuring that the evolution of the visual energy field of the entire video along the time axis closely matches the driving force of the audio energy field. Simultaneously, for time intervals where matching errors still exceed a threshold after allocation, a video diffusion generation model conditioned on the target visual energy field is actively invoked to generate supplementary video segments. During the generation process, the first and last frames are forced to align with the preceding and following real segments through optical flow constraints, seamlessly filling blank segments where existing materials cannot meet the audio-visual synchronization requirements. Furthermore, at each editing point, a transition frame sequence is adaptively generated based on the jump amplitude of the preceding and following visual energy fields; the larger the jump, the more transition frames are generated. The spatiotemporal diffusion model simultaneously optimizes pixel mean square error and field matching loss, making the transition smooth and conforming to the continuous evolution law of the energy field.

[0027] Third, this application further enhances the rhythm and viewing quality of the final product by dynamically adjusting the playback speed of video segments and introducing global consistency refinement. The ratio of the temporal gradient of the audio and visual energy fields is calculated in real time to obtain a speed adjustment coefficient. Based on this, the original material is fast-forwarded or slowed down, allowing the visual motion rhythm to actively follow the instantaneous rate of audio energy changes, thus upgrading audio-visual synchronization from time alignment to motion rhythm synchronization. Finally, the speed-adjusted segments are spliced ​​with AIGC supplementary segments according to the editing point sequence and allocation scheme, a transition frame is inserted, and then the video large language model is called to evaluate the continuous video frame by frame. For defects such as inconsistencies in subject appearance, lighting jumps, or visual jumps, refinement is performed using a diffusion repair model, a color transfer network, and fine-tuning of transition frames or editing point positions, respectively, before final encoding and output. Overall, this application elevates audio-driven editing from discrete event triggering to global optimization coupled with continuous fields, and integrates generative completion, dynamic speed adjustment, and intelligent post-processing, enhancing the rhythmic accuracy, visual smoothness, and creative intelligence level of the beat-synced video. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1A system architecture diagram of an AIGC beat-matching editing system provided in this application embodiment; Figure 2 A flowchart illustrating an AIGC beat-sync editing method provided in this application embodiment; Figure 3 A flowchart illustrating another AIGC beat-sync editing method provided in this application embodiment; Figure 4 This is a flowchart illustrating another AIGC beat-sync editing method provided in an embodiment of this application. Detailed Implementation

[0030] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0031] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0032] The AIGC beat-sync editing method provided in this application embodiment can be applied to, for example... Figure 1 In an AIGC beat-sync editing system, as shown, Figure 1 As shown, the system includes: an energy field construction module, an editing path planning module, a video generation module, and a fusion output module; among which, The energy field construction module is used to construct audio energy fields and visual energy fields based on the acquired background audio signal to be edited and video clips; the energy field is used to characterize the energy distribution of audio or video at different spatiotemporal locations; The editing path planning module is used to establish the coupled partial differential equation between the audio energy field and the visual energy field, and solve the globally optimal editing path based on the coupled partial differential equation to obtain the editing point sequence and the video segment allocation scheme corresponding to each time interval. The video generation module is used to generate supplementary video clips by calling the AIGC generative model if the matching error between the visual energy field and the audio energy field exceeds a preset threshold for time intervals existing in the editing path. At each editing point, a transition frame sequence is generated based on the jump amplitude of the visual energy field before and after. The fusion output module is used to dynamically adjust the playback speed of video clips based on the ratio of the temporal gradient of the audio energy field to the visual energy field, and to splice and merge the dynamically adjusted video clips, transition frame sequences, and supplementary video clips to output a beat-matched edited video.

[0033] To address the technical problems of low efficiency and difficulty in achieving precise rhythm synchronization between audio beats and video footage in existing manual editing techniques, especially the inability to generate coherent beat-synced videos when video footage is insufficient, this application provides an AIGC beat-syncing editing method and system, the method comprising: Based on the acquired background audio signal to be edited and video clips, an audio energy field and a visual energy field are constructed respectively; the energy field is used to characterize the energy distribution of audio or video at different spatiotemporal locations; Establish a coupled partial differential equation between the audio energy field and the visual energy field, solve the globally optimal editing path based on the coupled partial differential equation, and obtain the editing point sequence and the video segment allocation scheme corresponding to each time interval; For time intervals existing in the editing path, if the matching error between the visual energy field and the audio energy field exceeds a preset threshold, the AIGC generative model is called to generate supplementary video segments, and at each editing point, a transition frame sequence is generated based on the jump amplitude of the visual energy field before and after. Based on the ratio of the temporal gradient of the audio energy field to the visual energy field, the playback speed of the video clip is dynamically adjusted, and the dynamically adjusted video clip, transition frame sequence, and supplementary video clip are spliced ​​and merged to output a beat-matched edited video.

[0034] Based on this, this application can achieve automatic and precise alignment of audio rhythm and visual content, optimize the editing path through energy field coupling, use AIGC to complete missing materials and generate smooth transitions, and finally output rhythmically synchronized and visually smooth beat-matching videos.

[0035] like Figure 2 As shown in the embodiment of this application, an AIGC beat-sync editing method is provided, including: S1. Based on the acquired background audio signal to be edited and video clips, construct audio energy field and visual energy field respectively.

[0036] The energy field is used to characterize the energy distribution of audio or video at different spatiotemporal locations. Specifically, the audio energy field maps the background audio signal onto the time axis, with each time point corresponding to an audio energy value. This energy value can be obtained by calculating the sum of squares or logarithmic energy of the amplitudes of each frequency band using short-time Fourier transform, reflecting the loudness or beat intensity of the audio at that moment. The visual energy field maps video clips to both time and space dimensions, with each pixel or image patch corresponding to a visual energy value at each moment. This energy value can be obtained by calculating the image gradient amplitude, optical flow amplitude, or visual saliency extracted based on a convolutional neural network, reflecting the intensity of motion, texture complexity, or semantic importance of the image at that spatiotemporal location. In the field of intelligent editing, transforming unstructured audio and video data into a unified mathematical energy representation can simultaneously capture local instantaneous changes and global trends, facilitating multimodal information fusion. Compared to traditional temporal waveform comparison or feature point matching methods, the energy field has better mathematical modeling capabilities for subjective feelings such as rhythm and impact.

[0037] In some implementations, when constructing the audio energy field, the background audio signal is first pre-emphasized, framed, and windowed. Then, a Fast Fourier Transform is performed on each frame to calculate the power spectral density. Next, based on the characteristics of human hearing, a Mel filter bank or an equivalent rectangular bandwidth filter bank is applied for frequency weighting. Finally, the energy of each frequency band is accumulated or logarithmically compressed to obtain a time-varying audio energy sequence. When constructing the visual energy field, grayscale images are extracted frame by frame from each video clip. The gradient magnitude of each pixel in the horizontal and vertical directions is calculated. Gaussian pyramid downsampling is then used to reduce computational complexity. The average gradient magnitude within a fixed-size pixel block is then calculated as the spatial energy of that block. For the temporal dimension, the optical flow vector magnitude of corresponding blocks between adjacent frames is calculated. Finally, the spatial and temporal energy are weighted and fused to obtain the visual energy value at each spatiotemporal location.

[0038] S2. Establish the coupled partial differential equation between the audio energy field and the visual energy field, and solve the globally optimal editing path based on the coupled partial differential equation to obtain the editing point sequence and the video segment allocation scheme corresponding to each time interval.

[0039] The coupled partial differential equation between the audio and visual energy fields is a mathematical-physical model describing the mutual constraints and evolutionary relationship between the two energy fields. In its construction, the audio energy field can be considered as the driving source term, and the visual energy field as the response field, constructing a nonlinear partial differential equation containing a first-order time derivative term, a second-order spatial diffusion term, and a coupling source term. A typical form is... Where V represents the visual energy field and A represents the audio energy field. The spatial Laplacian operator of the visual energy field represents diffusion smoothness. Let F(V) represent the linear attraction of audio to visual energy, and F(V) represent the self-driven nonlinear term of the visual energy field. Solving this equation for the optimal editing path is equivalent to finding a time-varying visual segment selection trajectory that minimizes the spatiotemporal difference between the visual and audio energy fields while satisfying the visual material availability constraint. In practice, the equation can be discretized into a dynamic programming problem or transformed into an Euler-Lagrange equation using variational methods, followed by solving for the global optimum using gradient descent or graph cut algorithms. Using coupled partial differential equations to solve for the editing path simultaneously considers temporal alignment accuracy and spatial continuity, supports constraints on different editing styles such as jumps and smoothing, and leverages stability and uniqueness analysis from partial differential equation theory to ensure the robustness of the editing scheme and avoid getting trapped in local optima.

[0040] In some implementations, finding the optimal clipping path may include the following steps: First, the audio energy field A(t) and the visual energy fields of each video clip are compared. Project onto a common timeline reference. Define the clip path as a piecewise function. This indicates that the i-th video segment selected at global time t has undergone time remapping. The following frame. Then, the following coupled partial differential equation is established as the Euler-Lagrange form of the cost functional: The first term forces visual energy to follow audio energy, the second term penalizes the time change rate of the editing path to avoid switching too quickly, and the third term penalizes the acceleration of the visual energy field to ensure a smooth transition. Next, dynamic programming is used to solve the problem: the time axis is discretized into N intervals, with each interval containing candidate video segments from a media library. The state is defined as the current time and the selected segment. The state transition cost is given by the discrete approximation of the aforementioned functional. Finally, the Viterbi algorithm is used to find the globally minimum cost path. Each transition point on this path is a cut point, and the corresponding video segments are assigned to the continuous intervals between two cut points.

[0041] S3. For time intervals existing in the editing path, if the matching error between the visual energy field and the audio energy field exceeds a preset threshold, the AIGC generative model is called to generate supplementary video segments, and at each editing point, a transition frame sequence is generated based on the jump amplitude of the visual energy field before and after.

[0042] Matching error is defined as the difference between the actual allocated visual energy field and the ideal audio energy field within a certain time interval. Specifically, it can be measured using mean squared error, mean absolute error, or dynamic time warping distance. Matching error reflects the degree to which the current video clip matches the background audio in terms of rhythm, intensity, or motion pattern. When the error exceeds a preset threshold, it means that there are not enough suitable clips in the existing video footage to meet the timing requirements of that time interval; for example, the audio needs a strong visual impact, but all the footage is smooth. In this case, an AIGC generative model is invoked to generate supplementary video clips. This model, based on an adversarial generative network or diffusion model, uses local features of the audio energy field, such as the beat envelope and spectral centroid, as well as visual features of adjacent existing video clips as input conditions to synthesize video frames that conform to the expected energy changes and semantic content. Generating supplementary video clips fills in the gaps in the footage, ensuring that the entire video has no rhythmic breaks or energy mismatches on the timeline. Transition frame sequences are used to smoothly connect two different video clips at the editing point, avoiding abrupt cuts. Because there are abrupt changes in the visual energy field before and after an edit point—for example, jumping directly from a high-energy moving image to a low-energy static image—these abrupt transitions can be uncomfortable for the human eye. By generating an intermediate frame sequence between these two points, the visual energy field gradually transitions, conforming to the persistence of vision characteristic, thereby improving the viewing experience.

[0043] In some implementations, S3 may include the following steps: First, calculate the matching error within each time interval: for each interval The visual energy field V(t) of the assigned video segment is aligned with the audio energy field A(t) in time, and the relative root mean square error is calculated. If the RMSE exceeds the threshold θ, for example, 0.3, then supplementary generation is triggered; The specific steps for calling the AIGC generative model are as follows: extract the Mel spectrogram and beat trajectory of the audio segment in the interval as conditional vectors; extract appearance and motion features from the first and last frames of adjacent video segments; input the conditional vectors and features into a pre-trained diffusion model, which starts with random noise, gradually denoises to generate a sequence of video frames, and uses a cross-attention mechanism to ensure that the generated content is aligned with the audio conditions. The length of the generated supplementary video segment is consistent with the length of the audio interval.

[0044] For each clip point in global time The visual energy fields before and after the location are denoted as follows: and jump amplitude When ΔV exceeds a preset smoothing threshold, such as 0.2, a transition frame sequence is generated. Each frame of the transition frame is obtained by solving for optical flow-guided linear interpolation or using a frame interpolation model in a generative adversarial network, so that the visual energy field changes from... Linear or sine curve change to 。

[0045] It should be noted that the AIGC generative model can be fine-tuned according to the style of the material library. For example, the style code of an existing video clip can be used as an additional condition to make the generated supplementary clip coordinated with the original material in terms of color, texture, and composition style. The generation of the transition frame sequence can be based not only on the energy jump amplitude but also on the consistency of the motion direction: if the main object in the front and back clips moves in opposite directions, steering transition frames need to be generated.

[0046] Exemplarily, assume that in a certain time interval corresponding to 1.5 seconds to 2.0 seconds, the audio energy field requires a gradually rising energy ramp, but the visual energy fields of all video clips in the existing materials are in a fast-decreasing mode, resulting in a matching error of 0.4 exceeding the threshold of 0.3. The system calls the AIGC model and generates a video clip with an increasing visual energy field based on the spectrum and rhythm of this segment of audio. At the editing point of 1.5 seconds, the previous clip is a high-energy frame of a dunk landing, and the subsequent clip is a low-energy frame before the start of the waves. The jump amplitude is 0.6, and the system generates 6 transition frames to gradually blur and dissolve the picture from the dunk action to the starting ripples of the waves.

[0047] S4. Dynamically adjust the playback speed of the video clip according to the ratio of the time gradient of the audio energy field to the visual energy field, and splice and fuse the dynamically adjusted video clip, the transition frame sequence, and the supplementary video clip to output a beat-matching edited video.

[0048] In some implementation manners, the specific steps of dynamically adjusting the playback speed of the video clip are as follows: First, calculate the time gradient of the audio energy field , that is, the change rate of the audio energy with time; and calculate the time gradient of the visual energy field of the allocated video clip ; For each editing interval, define the speed adjustment factor , and perform low-pass filtering and clipping on it to avoid excessive speed mutation. Then remap the original playback time of the video clip based on r(t): the new playback time function is , and video frames are extracted or interpolated according to the time mapping relationship during actual playback. If r(t)>1, it means that acceleration playback is required; if 0<r(t)<1, it means that deceleration playback is required; if r(t) is negative, it means that reverse playback or special effects are required.

[0049] Dynamic adjustments allow the rate of change in visual content to follow the fluctuations of the audio rhythm. For example, speeding up scene transitions or motion during periods of intense drumbeats and slowing down the visuals during calmer passages enhances the sense of audiovisual synchronization. After speed adjustments are complete, all video clips, including the original adjusted clips, supplementary clips, and transition frame sequences, are spliced ​​together in chronological order according to the cut point sequence. During splicing, adjacent clips overlap by 0.5 to 2 frames and undergo linear blending or alpha blending. Finally, the blended video track is multiplexed with the original audio track, and the output is a beat-synced video in a standard container format such as MP4 or MOV.

[0050] It should be noted that the speed adjustment factor r(t) can be constrained by the semantics of the video content: for scenes containing faces or text, the maximum acceleration is limited to 1.5x to avoid distortion; for pure landscapes or abstract scenes, acceleration up to 4x is allowed. During the splicing and fusion process, if the preceding and following segments come from different video sources, their brightness and color temperature may differ. Color matching is required first, mapping the color histogram of the later segment to the color space of the earlier segment before fusion. The resolution and frame rate of the output video should be consistent with the input video material, or resampled under user-specified parameters.

[0051] Based on the above technical solutions, this application provides an AIGC beat-matching editing method. By constructing an audio energy field and a visual energy field and establishing coupled partial differential equations, global optimization of the editing path is achieved, overcoming the drawback of traditional local matching methods that easily lead to rhythm disorder. By introducing a matching error threshold and calling an AIGC generative model to supplement missing materials, the problem of not being able to complete beat-matching editing when materials are insufficient is solved, ensuring the integrity of the video timeline. By generating a transition frame sequence based on the jump amplitude of the visual energy field at the editing point, the visual discomfort caused by abrupt cuts is eliminated, improving the smoothness of viewing. By dynamically adjusting the playback speed of video segments, the visual rhythm can actively follow the changes in audio energy, enhancing the impact and immersion of beat-matching editing. In summary, this application significantly improves the automation level, rhythm alignment accuracy, and final product quality of beat-matching editing, and can be applied to various application scenarios such as short video creation, music video production, and advertising music editing.

[0052] In one possible implementation of this application embodiment, the above-mentioned S1 can be specifically implemented by the following S101, S102 and S103, which are described in detail below: S101. Construct the audio energy field of the background audio signal.

[0053] The audio energy field is a continuous function of time t and one-dimensional feature coordinate x. The energy field is used to characterize the energy distribution density of an audio signal at different times and on different frequency-energy compression axes. Therefore, the audio energy field can capture dynamic information in music such as beat, intensity, timbre, vibrato, crescendo and diminuendo, rather than just discrete beat points.

[0054] In some implementations, the above S101 specifically includes the following sub-steps: S101-1: Perform a constant Q transform (CQT) on the background audio signal to obtain the complex spectrum S(f,t).

[0055] Assume the sampling rate of the background audio signal is The frequency is typically 44.1kHz or 48kHz. The audio signal is divided into frames, with the frame length related to the lowest frequency of the CQT band, usually around 0.1 seconds, and the frame shift is half the frame length. Then, a CQT transform is performed on each frame to obtain the complex spectrum S(f,t), where f is the frequency index corresponding to multiple frequency bands, and t is the time frame index.

[0056] S101-2: Calculate the energy distribution function at each time t .

[0057] This function represents the instantaneous energy at a given time t and frequency f. Since the human ear perceives energy in a logarithmic manner, P(f,t) can optionally be converted to decibels. However, the network can automatically learn the nonlinear mapping in subsequent compression encoding, so linear energy can also be used directly.

[0058] S101-3: The energy distribution function P(f,t) is compressed to a one-dimensional latent coordinate x along the frequency axis by a one-dimensional convolutional encoder to obtain the intermediate field q(x,t).

[0059] The input to the one-dimensional convolutional encoder is P(f,t), and its shape is (time frame number) (B bands). The encoder consists of several one-dimensional convolutional layers, each followed by batch normalization and a ReLU activation function. For example, a three-layer convolution can be used: the first layer has a kernel size of 5×1 and 32 output channels; the second layer has a kernel size of 3×1 and 16 output channels; and the third layer has a kernel size of 3×1 and 1 output channel. Then, global average pooling or direct flattening is used in the last layer to obtain a scalar value corresponding to each time frame, thereby compressing the frequency axis to one dimension. To retain more time-frequency structure, the encoder can also be designed to output multiple potential coordinates, but this application preferably compresses it to one dimension (i.e., x is a scalar) to simplify subsequent field operations.

[0060] The scalar obtained for each time frame t is denoted as ,in These are the potential coordinate values ​​corresponding to this frame. All time frames... An intermediate field q(x,t) is constructed, where the x-axis is discretized into multiple equally spaced intervals (bins). When training this one-dimensional convolutional encoder, a self-supervised reconstruction loss is employed: the compressed q(x,t) is reconstructed back to the original energy distribution function P'(f,t) through a symmetric deconvolutional network (decoder), with the loss function being the mean squared error or perceptual loss, minimizing the reconstruction error. This self-supervised training can be performed on large music datasets, enabling the encoder to retain crucial time-frequency energy variation information.

[0061] S101-4: Gaussian smoothing and exponential decay in the time domain are applied to the intermediate field q(x,t) to obtain the final audio energy field. .

[0062] Gaussian smoothing is used to eliminate high-frequency noise and local fluctuations, while exponential decay in the time domain is used to simulate the decay memory of musical energy, and the decay factor... The value ranges from 0.9 to 0.99, and is updated frame by frame. ;in, This is the smoothed field obtained after Gaussian smoothing of the intermediate field q(x,t).

[0063] Final audio energy field The x-axis is discretized into 64 equally spaced bins, and the time sampling rate is consistent with the video frame rate (e.g., 30Hz).

[0064] S102. Extract multi-dimensional features from each frame of the video clip.

[0065] The multidimensional features include a dense optical flow amplitude field, an optical flow curl field, an attention heatmap, and a scene-background difference map, which are used to characterize the intensity of local motion in the image, complex motion patterns such as rotation / vortex, the semantically focused subject area, and the distinction between foreground and background, respectively. These features depict the motion content of the video from different perspectives, providing rich visual cues for constructing a unified visual energy field.

[0066] In some implementations, the above S102 is specifically implemented through the following sub-steps: S102-1: Extract the frame sequence I(t) of the video material segment at a preset sampling frequency (e.g., 30 frames / second), where t is the time corresponding to the frame index.

[0067] S102-2: Calculate the amplitude field of dense optical flow.

[0068] For each frame I(t) and its adjacent next frame I(t+1), the RAFT optical flow estimation network is used to calculate the pixel-level optical flow vector field. , where u and v represent the optical flow components in the horizontal and vertical directions, respectively.

[0069] Then, the optical flow amplitude is calculated for each pixel position (x, y). This yields an amplitude map M(t) of the same size as the original frame, which is the dense optical flow amplitude field.

[0070] The amplitude field of this dense optical flow reflects the intensity of local motion in the image; the larger the value, the faster the motion.

[0071] S102-3: Calculate the optical flow curl field.

[0072] After obtaining the dense optical flow vector field (u,v), its curl is calculated. Partial derivatives and The Sobel operator is used for numerical approximation.

[0073] The curl field reflects the local rotation intensity of the flow field: positive values ​​indicate counterclockwise rotation, and negative values ​​indicate clockwise rotation. This feature is sensitive to complex motions such as the rotation of people or the vortexing of objects in an image.

[0074] S102-4: Calculate the attention heatmap.

[0075] A pre-trained visual Transformer model (e.g., ViT-B / 16) is used to process each frame image I(t). ViT divides the image into 16×16 image blocks, and each block is linearly projected and added to its positional code before being fed into the Transformer encoder. The average attention weight of the [CLS] token to all image block tokens is extracted from the last layer's self-attention module. The length of this weight vector is the number of image blocks (e.g., 196 blocks for a 224×224 input). This average attention weight is rearranged into a 16×14 grid according to the spatial location of the image blocks, and then upsampled to the original image size using bilinear interpolation to obtain the attention heatmap. .

[0076] The value of each pixel in this heatmap represents the importance of that region to the overall semantic understanding of the scene. It is usually focused on moving subjects (such as people and vehicles) and is used to identify moving subject areas in the image.

[0077] S102-5: Calculate the scene background difference map.

[0078] For consecutive frames and Histogram equalization is performed on each frame separately to eliminate the effects of slow changes in illumination. Then, the absolute difference map between the two equalized frames is calculated. To suppress noise, Gaussian blur is applied to the difference image. Finally, binarization is performed: the threshold is set to twice the average gray value of the difference image, pixels greater than the threshold are set to 1 (foreground), and the rest are set to 0 (background), resulting in the scene background difference image D(t).

[0079] This binary image marks the distribution of moving foreground and static background regions, helping to determine whether there are significant scene changes or subject movement in the image.

[0080] It should be noted that the calculation order of the above four features can be executed in parallel to improve efficiency; the optical flow amplitude field and the curl field share the same optical flow calculation result, avoiding redundant calculations. The visual Transformer model used in the attention heatmap can be a publicly available pre-trained model such as ViT-B / 16, and its model weights can be pre-trained on ImageNet.

[0081] S103. Construct a visual energy field based on multidimensional features.

[0082] Here, the visual energy field is a continuous function of time t and one-dimensional feature coordinates x. This is used to characterize the energy distribution density of video images along the complexity axis of different times and motion modes. Constructing the visual energy field requires fusing and compressing the various features obtained from S102, and transforming them into a continuous field through kernel density estimation, so as to make unified comparison and coupling with the audio energy field.

[0083] In some implementations, S103 specifically includes the following sub-steps: S103-1: The four feature maps obtained in S102 are concatenated along the channel dimension to form a multi-channel feature map F(t).

[0084] Specifically, the shape of the dense optical flow amplitude field M(t) is as follows: The optical flow curl field C(t) is The attention heatmap A_{att}(t) is The scene background difference map D(t) is After concatenation along the channel dimension, the shape of F(t) is as follows: If the original video frame size is large (e.g., 1920×1080), it can be downsampled to a fixed size (e.g., 224×224) using bilinear interpolation to reduce the computational load.

[0085] S103-2: Use a two-dimensional convolutional encoder to compress the multi-channel feature map into a 256-dimensional feature vector z(t).

[0086] A 2D convolutional encoder is actually a 2D convolutional neural network, also known as a "visual feature encoder." This encoder consists of several convolutional and pooling layers, ultimately obtaining a fixed-length feature vector through global average pooling. For example, using a lightweight network structure: the input is 224×224×4, passing through 3 convolutional blocks (each block contains 3×3 convolution, batch normalization, ReLU, and 2×2 max pooling), resulting in output channels of 16, 32, and 64 respectively. The final feature map size is 28×28×64, which is then globally averaged to obtain a 64×64=4096-dimensional vector, compressed to 256 dimensions by a fully connected layer. This yields the feature vector z(t) for each time step t.

[0087] S103-3: Compress z(t) into a one-dimensional variable x using a pre-trained variational autoencoder (VAE).

[0088] A variational autoencoder consists of an encoder and a decoder. The encoder maps z(t) to the mean. Sum of logarithmic variance (or directly output the variance) Both have a dimension of 1 (i.e., compressed to one dimension). One-dimensional latent variables are obtained through reparameterization techniques. ,in During VAE training, a large number of video segment features z(t) are used as training data. The loss function is ELBO (Evidence Lower Bound), which consists of reconstruction loss (reconstructing z(t) from the sampled x(t) using the decoder) and KL divergence. After pre-training, the encoder parameters are fixed for the inference phase.

[0089] S103-4: Using the one-dimensional variable x(t) sampled at each time step as the kernel center, a Gaussian kernel function is used to estimate the conditional kernel density, thus obtaining the visual energy field. .

[0090] For each time frame t, there is a sampled value x(t). Treating all x(t) from all time frames as samples from an unknown distribution, kernel density estimation is used to obtain a continuous probability density function. But here It is "energy density" rather than probability density, and usually does not require normalization.

[0091] Specifically, for each time frame t, a Gaussian kernel is placed centered on its x(t). Then calculate the contribution on the discrete bins along the x-axis: ,in It is possible to take only the current time t (i.e., instantaneous estimation) or to take a small time neighborhood for smoothing. This application preferably uses instantaneous kernel density estimation, that is, each time step is independent: Then the x-axis is discretized (64 bins) to obtain the energy distribution of each time t across the 64 bins; where the kernel width h is determined by cross-validation, for example, by taking 1 or 2 times the sample standard deviation of all x(t).

[0092] Ultimately, the visual energy field is obtained. Its dimensions are the same as those of the audio energy field (time frame number × 64).

[0093] It should be noted that compressing VAE to one dimension may result in the loss of some information, but experiments show that the one-dimensional potential space is sufficient to distinguish basic motion patterns, such as fast motion, slow motion, rotation, and static state, and is easy to align with the one-dimensional feature space of the audio energy field.

[0094] Based on the above technical solution, this application constructs audio energy fields and visual energy fields through steps S101, S102, and S103 respectively, providing a data foundation for subsequent coupling optimization, path planning, AIGC completion, and dynamic speed adjustment. Compared to directly using discrete beat points or original pixel features, energy field representation can preserve the dynamic change trend and intrinsic structure of the signal, improving the accuracy and robustness of audio-visual matching.

[0095] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, the above S2 can be implemented through the following S201, S202 and S203, which are explained in detail below: S201. Establish the coupling partial differential equation between the audio energy field and the visual energy field.

[0096] The coupled partial differential equation describes the dynamic relationship between the audio energy field and the visual energy field over time. Physically, the rate of change of the visual energy field (including local variations and energy flow along the characteristic axis) should be proportional to the attraction of the audio energy field to the visual energy field, with the proportionality coefficient being the time-varying coupling coefficient σ(t). This equation transforms the editing problem into a continuous field tracking problem, namely, finding the optimal evolution path of the visual energy field to follow the changes in the audio energy field as closely as possible.

[0097] In some implementations, the above coupled partial differential equations are specifically expressed as: ;in, The visual energy field is a function of time t and one-dimensional feature coordinates x. For the audio energy field, the domain and Same; t is time, and x is a one-dimensional feature coordinate (e.g., the axis after frequency-energy compression). This is the energy flow velocity vector obtained by least-squares fitting of the optical flow sequence from video footage clips, representing the "flow" velocity of visual energy along the x-axis. In fact, It reflects the migration rate of video motion patterns in the feature space and can be obtained by projecting the optical flow field onto the feature space; Energy divergence represents the flux change of visual energy along the x-axis, characterizing the accumulation or dissipation of energy due to motion pattern drift. The coupling coefficient, ranging from 0 to 1, is used to control the driving strength of the audio energy field on the visual energy field. The larger the visual energy field, the faster it needs to approach the audio energy field; The smaller the value, the more likely the visual energy field is to maintain its inertia.

[0098] It should be noted that the left side of the equation represents the material derivative of the visual energy field (the derivative that follows the energy flow), and the right side represents the "elastic pull" of audio on vision. When Deviation At that time, a driving force proportional to the deviation is generated at the right end, causing... Towards Close to each other. Coupling coefficient. The values ​​can be preset according to the musical section structure (intro, verse, chorus, bridge). For example, a larger value (such as 0.9) can be used for the chorus to require the visuals to strictly follow the audio rhythm; a smaller value (such as 0.3) can be used for the intro to allow the visuals more freedom.

[0099] S202. Construct a minimum action integral objective function based on coupled partial differential equations, and discretize the time axis and video footage.

[0100] The principle of least action, a variational principle in physics, states that the actual path of motion should minimize a certain action functional. Here, this application transforms the editing problem into finding an editing path—a sequence of editing points and material allocation—that minimizes the cumulative residual of the aforementioned coupled partial differential equations over the entire video duration, while minimizing the number of editing points (controlled by sparsity regularization). This ensures both global optimality of audio-visual energy matching and avoids overly fragmented editing.

[0101] In some implementations, the above operation specifically includes the following sub-steps: S202-1: Transform the coupled partial differential equations into residual form.

[0102] Define the residual function R(t) as the difference between the left and right sides; ideally, the residual should be 0. The magnitude of the residual directly reflects the degree to which the current visual energy field deviates from the coupling equation.

[0103] S202-2: Construct the minimum action integral objective function.

[0104] Based on the principle of least action, the objective function is defined as follows: ;in, T represents the total video duration; The square norm of the residuals is given, and the integration domain includes time t and the characteristic coordinates x (i.e., ...). The penalty is for audio-visual energy mismatch; p(t) is the clipping point indicator function, which takes a value of 1 at the clipping point and a value of 0 at other times. It is a discrete pulse sequence. The integral of the squared density of clip points is equivalent to a measure of the number of clip points, which is used to constrain the number of clips. This is the sparse regularization coefficient, a positive number, used to balance matching accuracy and the number of clipping points. The larger the value, the fewer clipping points are encouraged; The smaller the size, the more dense the clipping is allowed.

[0105] The objective function takes into account both the field matching quality over continuous time (the first term) and the sparsity of the clipping points (the second term).

[0106] S202-3: Time axis discretization and candidate video segment generation.

[0107] Discretize the time axis of the total video playback duration T into discrete moments at a preset sampling rate (e.g., 30Hz) to obtain a set of discrete time points. Interval between adjacent points Second.

[0108] For each original video clip (a complete video file), it is cut into several consecutive segments of exactly [duration missing] based on these discrete moments. Sub-fragments that are multiples of an integer.

[0109] Specifically, each source j can generate multiple candidate video segments, each segment determined by its start time (relative to the starting frame position of the source) and end time. For clarity, the set of candidate video segments is defined as: for each source index j, and any two discrete time points... ( This can form a candidate segment with a playback duration of The content consists of a sequence of consecutive frames extracted from source j. Each candidate segment contains the following information: start time. End time The source material index j and the corresponding consecutive frame sequence.

[0110] It should be noted that in actual dynamic programming implementations, not all candidate fragments (which are numerous) are explicitly created. Instead, the fragment content and its field matching error are obtained through on-the-fly computation during state transitions. However, for the sake of mathematical completeness, this application still considers them as candidates.

[0111] S203. Use dynamic programming to find the minimum value of the objective function and obtain the optimal editing path.

[0112] Dynamic programming (DP) is a mathematical method for optimizing multi-stage decision-making processes. In this problem, this application treats discrete time points as stages, where the state of each stage is the index of the currently playing material and the exact frame of that material that has just finished playing at the current moment (i.e., the state is aligned with the discrete time). Through state transitions, the process is recursively applied from the starting time to the ending time, calculating the minimum cumulative loss for each state, and finally backtracking to obtain the optimal path.

[0113] In some implementations, S203 specifically includes the following sub-steps: S203-1: Define state and initialize.

[0114] Suppose there are M original video clips (indexed from 1 to M). Definition This represents the minimum cumulative loss when a frame of material i has been played exactly at discrete time t (t belongs to the set of discrete points). Here, "played out" means that the current playback position of the material is aligned with time t. For the starting time t=0, this application assumes that playback can begin from the first frame of any material, therefore initialization... (For all i). If the starting frame of a clip cannot be aligned with 0 (e.g., the clip's start time is not 0), then its initial value is set to 0. .

[0115] S203-2: State transition equation (corrected).

[0116] For each discrete time (k=1,2,3...N) and for each material index i, the state transition equation is: ; Where j is the material index used in the previous state. It is the time corresponding to the previous state (i.e., the end time of the previous segment, which is also the start time of the current segment). This represents a candidate video segment extracted from source i (starting from a certain initial frame of source i, with a playback duration of (...)). Placed within the time range The integral of the field matching error within the time frame. Its specific calculation method is as follows: First, extract a continuous frame sequence from source material i, making its duration exactly [length missing]. This application typically assumes that each use of material begins from its starting frame; therefore, the extracted segment is the first frame of material i. Duration portion. Then, the visual energy field of this segment. ( (The time within the segment) is aligned to the interval by linear time scaling. After alignment Finally, the L2 integral error between the audio energy field and the aligned visual energy field within this interval is calculated: ; This is a clip transient penalty constant, typically taken as an empirical value of 0.5, to encourage fewer clips. For indicator functions, when A value of 1 indicates that an edit (source switch) has occurred, at which point a penalty is added. When j=i, the value is 0, which means that the same material will continue to be used without additional penalty.

[0117] The meaning of this transition equation is: to reach the state... Then we need to find a previous state. , and then from Start playing the clip from source i until... It also assumes the field matching error of the segment; if the previous clip j is different from the current clip i, it incurs an additional editing penalty. In this way, dynamic programming can naturally handle the continuous use of the same clip and the switching between different clips.

[0118] It should be noted that the transition equation in this application assumes that each use of footage starts from its initial frame, which simplifies the state representation (eliminating the need to record the playback progress within the footage). In actual editing, a piece of footage is usually only used in a small segment. Reusing different parts of the same footage can be achieved by reallocating different time intervals, which is equivalent to using the same footage multiple times, but starting from the beginning each time. This simplification is computationally very efficient, and the transition frames generated by subsequent AIGC can smooth out the abrupt changes at the splicing points, so it is reasonable.

[0119] Another alternative implementation is to expand the state to... ,in This represents the playback progress (frame position) within media element i. This allows playback to continue directly from the middle of the media element, but the state space will expand to O(1). The calculation of L, where L is the average number of frames in the source material, is too computationally intensive. Therefore, this application preferably adopts a simplified playback mode from the beginning.

[0120] S203-3: Calculate all states and backtrack to the optimal path.

[0121] Calculate all in ascending order of time. For each And i, we need to iterate through all possible... (From 0 to Given the previous point and all materials j, find the combination that minimizes the right-hand side and record the predecessor state. .

[0122] After calculating all states, at the termination time T, find the minimum value. Let the corresponding material index be . Then from the state Start backtracking: Based on the recorded predecessor states, obtain The current moment. Record as a clipping point (if) ), and time interval Assign to current material Then update the current state to Continue backtracking until the starting time 0 is reached. This ultimately yields the clipping point time sequence. And the material index corresponding to each time interval, i.e., the video segment allocation scheme.

[0123] Based on the above technical solution, this application optimizes the editing point position and material allocation from a global perspective by establishing coupled partial differential equations, constructing a minimum action objective function, and solving it using dynamic programming. This avoids the rhythm disorder caused by local greedy decisions in traditional methods. This method not only ensures the optimal matching of audio and video energy on the timeline but also controls the editing density through sparse regularization, making the final video both rhythmically distinct and not overly fragmented.

[0124] In one possible implementation of the embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, the above S3 can be implemented through the following S301, S302 and S303, which are explained in detail below: S301. Calculate the matching error between the visual energy field and the audio energy field in each time interval of the editing path, and determine whether it exceeds the preset threshold.

[0125] The matching error quantifies the deviation between the actual visual energy field and the desired audio energy field within an assigned video clip. If the error exceeds a threshold, it indicates that the existing material cannot meet the audio-visual synchronization requirements within that range, and the AIGC generative model needs to be invoked for supplementation; if the error is within the threshold, the original material clip is retained. The matching error is calculated based on the L2 norm integral, comprehensively considering the overall differences along both the time and feature axes.

[0126] In some implementations, for each time interval in the clip path obtained from S2 The matching error is calculated using the following formula, along with the video material index i assigned to that interval. : ;in, and These are the start and end times of the time interval, respectively, and the interval length is... ; The background audio energy field at time t and feature coordinate x is obtained from S101; To align the original visual energy field of the assigned video clip i to the time interval using linear time scaling. The visual energy field afterward.

[0127] Specifically, let the original duration of material i be... (seconds), its visual energy field is ,in This refers to the internal time of the material. This is to place it within the duration. Within the specified interval, time scaling is required: define the mapping. The aligned visual energy field is then... .

[0128] The average matching error is obtained by first integrating over the feature coordinate x, then over time, and finally dividing by the interval length for normalization.

[0129] when Greater than the preset threshold At that point, it is determined that AIGC needs to be invoked to generate supplementary video clips for that interval. Threshold It can be determined experimentally, for example, by taking 0.25.

[0130] It should be noted that the calculation of the matching error depends on the editing path and material allocation scheme already determined in S2. This is because S2 already uses the field matching error integral when solving for the globally optimal path. But that's the error calculated for path optimization, while here... This is a precise re-evaluation of the final selected path, and the two may differ slightly in numerical value (because the error in S2 may have used an approximate or simplified alignment). To ensure accuracy, this step recalculates the precise error.

[0131] S302. For time intervals where the matching error exceeds the threshold, call the AIGC generative model to generate supplementary video clips.

[0132] The AIGC generative model is used to create entirely new video content to fill gaps where existing materials cannot meet the requirements for audio-visual energy matching. This model uses the target visual energy field as a condition and generates a sequence of video frames that smoothly connect with preceding and following real segments through a diffusion process, ensuring that the final edited video completely follows the audio in terms of energy.

[0133] In some implementations, the above S302 specifically includes the following sub-steps: S302-1: Pre-train a video diffusion generation model conditioned on the target visual energy field.

[0134] This model employs a variant of the Latent Diffusion Model, specifically drawing inspiration from the Stable Video Diffusion (SVD) design. The backbone network is a 3D U-Net, capable of handling both spatial and temporal dimensions simultaneously. Conditional vector embeddings are injected into the cross-attention layers, making the generation process controllable. During training, a large number of video clips are extracted from a natural video dataset, and the visual energy field of each clip is calculated. ,Will As a condition, the original video frame sequence is used as the target, and the model is trained using the loss function of the Denoising Diffusion Probabilistic Model (DDPM). After training, the model can generate the corresponding video frame sequence based on the given target visual energy field.

[0135] S302-2: For time intervals where the matching error exceeds the threshold The ideal visual energy field target is calculated based on the coupled partial differential equations. ,in , .

[0136] The ideal target is calculated as follows: using the left endpoint of the interval... Using the visual energy field corresponding to the last frame of the previous real segment as the initial condition, and ignoring the AIGC source term, the coupled partial differential equations in S201 are numerically solved: ;in Local time within the interval (from 0 to ), For the audio energy field in the corresponding global time The value of is given. This equation describes how the visual energy field should evolve under audio-only driving conditions. Solving it using the finite difference method or spectral method yields the value of . .

[0137] S302-3: Encode the target visual energy field into a conditional vector and generate a video clip.

[0138] Will Flattened along the x-axis into a 64-dimensional vector sequence (time steps: Then, it is encoded into a conditional vector of the diffusion model through a multilayer perceptron (MLP). Simultaneously, to ensure seamless transitions between the generated segment and the preceding and following real segments, first and last frame constraints are required: the first frame is forced to be the last frame of the real segment preceding the left endpoint. The last frame is forcibly taken as the first frame of the actual segment following the right endpoint. Randomly sampled Gaussian noise is used as the latent representation, and denoising is performed stepwise following the inverse process of DDPM. In each step, the conditional vector and the constraints of the first and last frames (via additional loss terms or guidance) are injected into the denoising process. After denoising is completed, the latent representation is decoded into a sequence of video frames by a decoder to obtain supplementary video segments.

[0139] S302-4: Forced optical flow constraint alignment.

[0140] During the generation process, an additional optical flow constraint loss is added to ensure that the first frame of the generated segment is consistent with... The optical flow error between them is the smallest, and the tail frame and The optical flow error between them is minimized. Specifically, optical flow loss is defined as... This loss is added as a regularization term to the loss function of the diffusion model, thereby ensuring that the generated segments are naturally connected with the segments before and after in terms of motion.

[0141] It should be noted that AIGC generation is triggered only when the matching error exceeds a threshold, and the generated content is entirely driven by the audio energy field, thus accurately filling the audio-visual energy gap. The length of the generated video clip is consistent with the length of the target interval, and it will directly replace the originally allocated material (or insert it into the missing position). In addition, since the computational cost of the generation model is relatively large, accelerated sampling methods (such as DDIM) can be used to reduce the number of steps.

[0142] S303. At each clipping point, generate a sequence of transition frames based on the amplitude of the jump in the visual energy field before and after.

[0143] The transition frame sequence is used to smooth the transition between two different video segments, avoiding a jarring feeling caused by abrupt changes in the visual energy field. The larger the jump amplitude, the greater the difference in motion patterns between the preceding and following segments, requiring more transition frames for a smooth transition; the smaller the jump amplitude, the fewer transition frames can be used.

[0144] In some implementations, the above S303 specifically includes the following sub-steps: S303-1: Obtain the visual energy field and corresponding frames before and after the clipping point.

[0145] At each clipping point T (where T is the clipping time), obtain the last frame of the previous segment. and its corresponding visual energy field (The field distribution at the instant before the cut point), and the first frame of the next segment. and its corresponding visual energy field (The field distribution at the instant after the clipping point).

[0146] S303-2: Calculate the transition intensity and determine the number of transition frames.

[0147] The jump intensity is defined as the difference in the L2 norm of the visual energy fields before and after the transition: ;in and These represent the instants after and before the clipping point, respectively. The magnitude of the jump intensity is determined by... Quantification, It is the L2 norm (integrated over the characteristic axis x).

[0148] The length L of the transition frame sequence is set to an integer proportional to the transition intensity: Furthermore, L is limited to between 4 and 16 frames (i.e., a minimum of 4 frames and a maximum of 16 frames). When the transition intensity is extremely small, a short transition of 4 frames is used; when the transition intensity is very large, a long transition of 16 frames is used.

[0149] S303-3: Construct a spatiotemporal diffusion model to generate transition frames.

[0150] A lightweight spatiotemporal diffusion model (parameter scale less than 200M) is adopted, which is based on... , Given the expected visual energy fields at each intermediate moment, the intermediate L-1 frame is generated through a conditional denoising process (since the beginning and end are already given).

[0151] Expected visual energy field Linear interpolation is used: ;in The sequence number is the transition frame number (0 corresponds to the first frame, L corresponds to the last frame). The model input consists of a random Gaussian noise tensor and conditional information (including the first frame, last frame, and desired energy field sequence), and the output is the predicted intermediate frame sequence. .

[0152] The loss function of this spatiotemporal diffusion model includes two terms during training: Pixel mean square error loss: ,in These are realistic transition frames, obtained from the training data. Field matching loss: ,in To predict frames The visual energy field extracted from it.

[0153] By jointly optimizing these two aspects, the generated transition frame is both visually smooth and ensures continuous change of the energy field, conforming to the physical constraints of the jump amplitude.

[0154] S303-4: Insert the generated transition frame sequence into the clipping point.

[0155] Finally, the generated transition frame sequence Insert into and In between, the original hard cut is replaced, thus completing a smooth transition at that clipping point.

[0156] It should be noted that transition frame generation is only applied to cut points (i.e., where footage changes). For continuous playback of the same footage (without cut points), transition frames are not required. If the preceding and following segments come from the same footage and the video itself is continuous, the transition intensity is zero. In this case, the shortest transition frame (4 frames) can still be generated to ensure no visual stuttering. The spatiotemporal diffusion model can reuse the video diffusion model in S302, but for efficiency, a lighter dedicated model is used.

[0157] Based on the above technical solution, this application accurately evaluates the matching error of each time interval through S301, and only calls AIGC to generate supplementary video when necessary, saving computing resources; the supplementary video clips generated through S302 can perfectly match the ideal evolution trajectory of the audio energy field and are aligned with the optical flow of the preceding and following real segments; through S303, adaptive generation of transition frames makes the visual energy field at the editing point change continuously, significantly improving the smoothness and viewing quality of the beat-matching video. These three sub-steps work together to form a complete closed loop from evaluation, completion to smooth transition, greatly enhancing the intelligence and robustness of AIGC beat-matching editing.

[0158] In one possible implementation of this application embodiment, the above-mentioned S4 specifically includes the following S401 to S403: S401. Dynamically adjust the playback speed of video clips based on the ratio of the time gradient of the audio energy field to that of the visual energy field.

[0159] The purpose of dynamically adjusting the playback speed is to make the movement rhythm of the video actively follow the instantaneous change rate of the audio energy, rather than simply being stuck at the beat. When the audio energy rises rapidly (such as a drumbeat burst), the video playback speed is appropriately increased to make the visual movement more intense; when the audio energy decays slowly, the speed is appropriately slowed down to create a soothing effect. This achieves synchronization between sound and image in terms of movement rhythm.

[0160] In some implementations, for each original video segment not generated by AIGC (i.e., a segment from the media library, excluding the supplementary segment generated by S302), the following operations are performed in real time during its playback: S401-1: Calculate the time derivatives of the audio energy field and the visual energy field.

[0161] For the current time t, the time derivative is approximated using the central difference method: ; ; in = 1 / 30 second (corresponding to a frame interval of 30fps). and It is an already constructed energy field, but for real-time calculation, only the integral value on the characteristic axis x can be taken (e.g., This refers to the rate of change of total energy over time. For simplified calculations, this application uses the total energy scalar.

[0162] S401-2: Calculate the speed adjustment coefficient r(t).

[0163] The speed adjustment factor is defined as the ratio of the time derivative of audio energy to the time derivative of visual energy, and is limited to a reasonable range using the clip function. ;in, and These are the time derivatives of the total audio energy and the total visual energy, respectively; The value is a small constant to prevent division by zero; the clip function limits r(t) to the range [0.7, 1.5] to avoid visually unnatural changes caused by excessively fast or slow speed changes.

[0164] When r(t)>1, it means that the rate of increase of audio energy is higher than that of visual energy, and playback needs to be accelerated; when r(t)<1, playback needs to be slowed down.

[0165] S401-3: Perform speed change processing according to the speed adjustment coefficient.

[0166] The variable speed processing is performed in segments with fixed time windows (e.g., 0.5 seconds), with the same r(t) used in each segment (which can be the average value within the window), and linear interpolation smoothing is performed between windows.

[0167] If r(t) > 1, fast-forward processing is performed: some frames need to be deleted, with a deletion ratio of 1 - 1 / r(t). For example, if r(t) = 1.2, approximately 16.7% of the frames need to be deleted. Specifically, frame indices are selected uniformly within the window, and one frame is deleted at regular intervals, so that the remaining number of frames equals the original number of frames divided by r(t).

[0168] If r(t) < 1, slow-motion processing is required: intermediate frames need to be inserted, the number of inserted frames is... For example, if r(t) = 0.8, then 25% of the frames need to be inserted. An optical flow interpolation method based on RIFE (Real-time Intermediate Flow Estimation) is used to generate intermediate frames between adjacent frames, increasing the total number of frames to 1 / r(t) times the original number of frames.

[0169] After speed adjustment, the actual playback duration of the video clip becomes 1 / r(t) of the original duration, and the time derivative of its visual energy field is adjusted to be approximately equal to the time derivative of the audio energy field, thereby achieving motion rhythm following.

[0170] It should be noted that dynamic speed adjustment is only applied to the original footage clips. For supplementary video clips generated by AIGC, since they have already been precisely matched to the ideal visual energy field, no further speed adjustment is performed to avoid accumulating errors. Additionally, speed adjustment may cause audio and video to become asynchronous. However, in this application, the audio is background music, and the video remains time-aligned with the original audio after speed adjustment. This is because this application determines the speed adjustment based on the audio gradient, and the video's timeline is remapped after speed adjustment, ensuring synchronization with the audio track during subsequent splicing.

[0171] S402. The dynamically adjusted video segments, transition frame sequences, and supplementary video segments are spliced ​​and merged to form a continuous video sequence.

[0172] Among them, splicing and blending is to combine the original clips processed by S401, the supplementary clips generated by S302, and the transition frames generated by S303 into a complete and uninterrupted video stream in an orderly manner according to the editing point sequence and material allocation scheme obtained by S2.

[0173] In some implementations, the above S402 specifically includes the following sub-steps: S402-1: Arrange all segments in chronological order.

[0174] According to the sequence of clip points The system also assigns a media index to each time interval, sorts all segments (including dynamically adjusted original segments and AIGC-generated supplementary segments) by their start time in ascending order, forming an initial segment list. Each segment has a start time, end time, and frame sequence.

[0175] S402-2: Insert a sequence of transition frames at each clipping point.

[0176] Iterate through each clip point T_{k}, find the last frame of the segment to the left of the point and the first frame of the segment to the right of the point, and insert the corresponding transition frame sequence generated by S303 between the two. If the transition intensity between the left and right segments at the clip point is small, still insert the shortest transition frame to ensure smoothness.

[0177] S402-3: Handles minor time offsets that may occur at fragment boundaries.

[0178] Because dynamic speed adjustment can alter segment durations, the time boundaries between segments may not align precisely. Therefore, during splicing, the end time of each segment needs to be recalculated based on the actual duration after the speed change, and the positions of subsequent segments adjusted to ensure timeline continuity. If small gaps (less than one frame) occur, frame duplication can be used to fill them; if overlaps occur, a blending and overlay process (e.g., linear fade-in / fade-out) is used.

[0179] After the above operations, a continuous video sequence from time 0 to time T is obtained, in which each frame has definite image content, and a transition frame has been inserted at the clipping point.

[0180] It should be noted that the video resolution and frame rate should be kept consistent during the splicing process. If there are materials with different resolutions, they need to be uniformly scaled to the target resolution (e.g., 1920×1080) first. The resolution of the supplementary video clips generated by AIGC should be consistent with the original materials or specified by the model.

[0181] S403. Perform global consistency retouching on the spliced ​​video and encode it for output, resulting in a beat-matching edited video.

[0182] Global consistency refinement is used to eliminate inconsistencies in appearance, lighting jumps, or minor visual jumps that may exist between different source clips (original footage, AIGC generated footage), thereby improving the overall viewing quality.

[0183] In some implementations, the above S403 specifically includes the following sub-steps: S403-1: Call the video large language model for frame-by-frame evaluation.

[0184] A Video Large Language Model (VLAVA) is used to analyze each frame of the continuous video sequence obtained from S402. This model can understand the video content and detect three common defects: Inconsistent main appearance: for example, sudden changes in human faces, clothing, or objects; Lighting abrupt changes: The brightness and color temperature of the preceding and following frames are significantly different; Visual jumps at clipping points: Despite the presence of transition frames, there may still be inconsistencies in motion.

[0185] The model outputs the defect type and location (e.g., bounding box) for each frame.

[0186] S403-2: Repairing different defects.

[0187] For areas where the subject's appearance is inconsistent, a diffusion-based image inpainting model (such as StableDiffusion Inpainting) is used to locally redraw the defective area. This model uses the defective frame and adjacent frames as references to maintain the consistency of the subject's identity.

[0188] For abrupt changes in lighting, a color transfer network is used for global correction. Specifically, using the hue of the first frame (or a selected reference frame) as a baseline, the color histogram of each subsequent frame is aligned to the reference frame through a linear transformation (such as a color mapping matrix), so that the lighting changes continuously throughout the video.

[0189] For visual jumps at clipping points, transition frames can be regenerated by fine-tuning the generation parameters of the transition frames (such as increasing the number of diffusion steps or adjusting the jump intensity scaling factor) or by slightly moving the clipping point position (within ±2 frames), and then replacing the original transition segment. If the jump is minor, optical flow interpolation can also be applied directly for correction.

[0190] S403-3: Encoded output.

[0191] The refined video frame sequence is encoded according to the target format (such as MP4, encoded H.264 or H.265), and the original background audio is merged (note that the audio is not speed-changed and remains unchanged), and the final beat-matched video file is output.

[0192] It should be noted that Video-LLaVA and the diffusion repair model are existing technologies, and this application directly calls the pre-trained model.

[0193] Based on the above technical solution, this application dynamically adjusts the video playback speed through S401, ensuring that the visual motion rhythm closely follows the changes in audio energy, thus achieving audio-visual rhythm synchronization; through S402, it orderly splices variable-speed segments, supplementary segments, and transition frames according to the optimal path, ensuring the continuity of the timeline and the smoothness of the editing points; through S403, it performs global consistency refinement, eliminating visual differences between multi-source segments and significantly improving the visual quality of the final beat-matching video. These three sub-steps work together to transform the AIGC beat-matching editing method from a theoretical algorithm into a complete technical solution that can be practically deployed and outputs high quality.

[0194] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0195] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely illustrative descriptions of the application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and variations.

Claims

1. An AIGC beat-sync editing method, characterized in that, include: Based on the acquired background audio signal to be edited and video clips, an audio energy field and a visual energy field are constructed respectively; the energy field is used to characterize the energy distribution of audio or video at different spatiotemporal locations; Establish a coupled partial differential equation between the audio energy field and the visual energy field, solve the globally optimal editing path based on the coupled partial differential equation, and obtain the editing point sequence and the video segment allocation scheme corresponding to each time interval; For the time intervals existing in the editing path, if the matching error between the visual energy field and the audio energy field exceeds a preset threshold, the AIGC generative model is called to generate supplementary video segments, and at each editing point, a transition frame sequence is generated according to the jump amplitude of the visual energy field before and after. Based on the ratio of the temporal gradient of the audio energy field to that of the visual energy field, the playback speed of the video segment is dynamically adjusted, and the dynamically adjusted video segment, the transition frame sequence, and the supplementary video segment are spliced ​​and merged to output a beat-matched edited video.

2. The AIGC beat-sync editing method according to claim 1, characterized in that, The process of constructing the audio energy field includes: The constant Q transform method is used to extract the constant Q transform spectrum of the background audio signal, and the complex spectrum S(f,t) of the background audio signal is obtained; where F is the frequency and t is the time. Calculate the energy distribution function at each time t based on the complex spectrum. ; The energy distribution function is compressed along the frequency axis to a one-dimensional latent coordinate x by a one-dimensional convolutional encoder to obtain the intermediate field q(x,t); wherein the one-dimensional convolutional encoder consists of multiple convolutional layers and a global normalization layer, and the training of the one-dimensional convolutional encoder adopts self-supervised reconstruction loss. Gaussian smoothing and exponential decay in the time domain are applied to the intermediate field to obtain the audio energy field ρ. a (x,t); where the x-axis of the audio energy field is discretized into multiple equally spaced intervals.

3. The AIGC beat-sync editing method according to claim 1, characterized in that, The process of constructing the visual energy field includes: Extract frame sequences from video clips at a preset sampling frequency; Multidimensional features are extracted from each frame in the frame sequence; the multidimensional features include dense optical flow amplitude field, optical flow curl field, attention heatmap, and scene background difference map; wherein, the dense optical flow amplitude field is obtained by RAFT optical flow algorithm, the optical flow curl field is obtained by calculating the curl of the dense optical flow length, the attention heatmap is obtained based on visual neural network, and the scene background difference map is obtained by normalizing the brightness of the current frame and the previous frame, then performing absolute difference and thresholding; The multidimensional features are concatenated along the channel dimension and then input into a two-dimensional convolutional encoder. A pre-trained variational autoencoder is then used to compress the output of the two-dimensional convolutional encoder to a one-dimensional variable x. The encoder output of the variational autoencoder is the mean and variance. Then, the product of random noise following a standard normal distribution and the variance is added to the mean and sampled to obtain the one-dimensional variable x. Using the one-dimensional variable x(t) obtained from sampling at each time step as the kernel center, a Gaussian function is used to estimate the conditional kernel density, thus obtaining the visual energy field.

4. The AIGC beat-sync editing method according to claim 1, characterized in that, The coupled partial differential equation is: ;in, For visual energy field, Let be the audio energy field, t be time, x be a one-dimensional feature coordinate, and u be the energy flow velocity vector obtained by least-squares fitting of the optical flow sequence of the video clip. Energy dispersion represents the flux change of visual energy along the x-axis. is the coupling coefficient, used to control the driving strength of the audio energy field on the visual energy field.

5. The AIGC beat-sync editing method according to claim 4, characterized in that, The process of solving the globally optimal editing path based on the coupled partial differential equations specifically includes: The coupled partial differential equation is transformed into residual form: ; Construct a minimum action integral objective function based on the principle of minimum action: Where p(t) is the cut point indicator function, which takes the value 1 at the cut point and 0 otherwise; λ is the sparsity regularization coefficient, used to control the number of cut points. The timeline of the total video playback duration is discretized into discrete moments at a preset sampling rate, and each video clip is segmented according to the discrete moments to obtain a set of candidate video clips; wherein, each candidate video clip includes a start time, an end time, the index of the video clip to which it belongs, and a sequence of consecutive frames extracted from the video clip; the start time and end time of each candidate video clip are discrete time points; The placement of each candidate video segment on the time axis is regarded as a state. The minimum value of the minimum action integral objective function is solved by dynamic programming to obtain the optimal cumulative loss value at each time step. Based on the optimal cumulative loss value at each time point, starting from the termination state, the material index that minimizes the cumulative loss value at the termination time is selected as the material for the last segment, and the predecessor state recorded in the dynamic programming for each state is determined. The process is then backtracked to the starting state to obtain the optimal editing path. The optimal editing path includes the sequence of editing point times and the video material segment indexes corresponding to each time interval.

6. The AIGC beat-sync editing method according to claim 5, characterized in that, The specific process for obtaining the edit point sequence and the video segment allocation scheme includes: Set the initial cumulative loss value for each video clip. If video clip i is used at the initial time and the starting frame of the video clip is aligned with time t=0, then the initial cumulative loss value is set to 0; otherwise, it is set to positive infinity. For each discrete time point and each video clip index, define the state. Indicates at discrete time Find the minimum cumulative loss after playing a frame of video clip i, and define the state transition equation as follows: ;in, This indicates that the candidate video segments extracted from video clip j are placed within the time interval. Integral of field matching error during internal time, , Let i and j be discrete time points, and i and j be the indices of video clips. To compensate for the transient penalty constant during editing, This is an indicator function; it takes the value 1 when j ≠ i, and 0 otherwise. After calculating all states, take all material indices i at the termination time T. The minimum value is recorded as the corresponding video clip index. Then from the terminated state Begin repeating the backtracking operation: find the predecessor state selected during the transition of the current state. The current state time Record it as a clipping point and divide the time interval Assign an index to the current video clip segment; then update the current state to... Continue backtracking until... Until then; the final edit point sequence is obtained. And the video clip index corresponding to each time interval, i.e., the video clip allocation scheme.

7. The AIGC beat-sync editing method according to claim 1, characterized in that, The specific steps involved in calling the AIGC generative model to generate supplementary video clips include: A video diffusion generation model conditioned on a target visual energy field is pre-trained. The video diffusion generation model adopts a variant structure of the latent diffusion model (SVD), and the backbone network of the video diffusion generation model is a 3D UNet, with conditional vector embeddings injected into the cross-attention layer. During training, the video diffusion generation model uses video segments extracted from a natural video dataset and their corresponding visual energy fields as conditional pairs. The ideal visual energy field target within the threshold interval is calculated based on the coupled partial differential equation; the threshold interval is the time interval during which the matching error exceeds a preset threshold. ; The visual energy field target is flattened along the x-axis into a multidimensional vector sequence, and encoded into the conditional vector of the video diffusion generation model through a multilayer perceptron. Randomly sampled Gaussian noise is used as a latent representation. The latent representation is gradually denoised according to the inverse process of the Denoising Diffusion Probability Model (DDPM). At the same time, guided by the conditional vector and the first and last frame constraints, the encoder of the video diffusion generation model generates the denoised latent representation. The first frame of the first and last frames is taken as the last frame of the real segment before the left endpoint, and the last frame of the first and last frames is taken as the first frame of the real segment after the right endpoint. The decoder of the video diffusion generation model decodes the denoised latent representation into a video frame sequence to obtain a supplementary video segment; wherein, during the generation process, the first frame is forced to be aligned with the last frame of the previous real segment through optical flow constraints, and the last frame is aligned with the first frame of the subsequent real segment through optical flow constraints.

8. The AIGC beat-sync editing method according to claim 1, characterized in that, The step of generating a transition frame sequence based on the jump amplitude of the preceding and following visual energy fields specifically includes: At each clipping point T, obtain the last frame of the previous clip. and the corresponding visual energy field and the first frame of the next segment and the corresponding visual energy field ; Based on the visual energy field of the last frame of the preceding segment and the visual energy field of the first frame of the following segment, the jump intensity at the clipping point is calculated using the following formula: Where T+ and T- represent the instants after and before the clipping point, respectively; Set the transition frame sequence length to , where ||·|| is the L2 norm, and L is restricted to between 4 and 16 frames; A spatiotemporal diffusion model is constructed, which uses the expected visual energy fields at the last frame of the preceding segment, the first frame of the following segment, and each intermediate time point as conditional information to generate a transition frame sequence through a conditional denoising process; wherein, the formula for calculating the expected visual energy field is: τ=0,1,2,……,L; the input of the spatiotemporal diffusion model is a random Gaussian noise tensor and the conditional information, and the output is a predicted transition frame sequence.

9. The AIGC beat-sync editing method according to claim 1, characterized in that, The specific method for dynamically adjusting the playback speed of video clips is as follows: The time derivatives of the audio energy field and the visual energy field of the video clip are calculated in real time during playback. Based on the time derivative, the speed adjustment coefficient is calculated using the clip function. The calculation formula is as follows: Where r(t) is the velocity adjustment coefficient at time t. The time derivative of the audio energy field is... The time derivative of the visual energy field is... This is a preset constant used to prevent division by zero; When the speed adjustment coefficient is greater than 1, the playback speed is increased by uniformly deleting video frames at time t, and the deletion ratio is 1-1 / r(t); When the speed adjustment coefficient is less than 1, the playback speed is reduced by inserting intermediate frames between adjacent frames at time t using the RIFE-based optical flow frame interpolation method, and the number of inserted frames is (1 / r(t)-1)×the original number of frames.

10. An AIGC beat-sync editing system, characterized in that, include: The module includes an energy field construction module, an editing path planning module, a video generation module, and a fusion output module; among them, The energy field construction module is used to construct an audio energy field and a visual energy field based on the acquired background audio signal to be edited and video clips; the energy field is used to characterize the energy distribution of audio or video at different spatiotemporal locations; The editing path planning module is used to establish the coupled partial differential equation between the audio energy field and the visual energy field, solve the globally optimal editing path based on the coupled partial differential equation, and obtain the editing point sequence and the video segment allocation scheme corresponding to each time interval. The video generation module is used to, for the time interval existing in the editing path, if the matching error between the visual energy field and the audio energy field exceeds a preset threshold, call the AIGC generative model to generate supplementary video segments, and at each editing point, generate a transition frame sequence based on the jump amplitude of the visual energy field before and after. The fusion output module is used to dynamically adjust the playback speed of the video segment according to the ratio of the temporal gradient of the audio energy field to the visual energy field, and to splice and fuse the dynamically adjusted video segment, the transition frame sequence, and the supplementary video segment to output a beat-cut video.

Images