Video clip synthesis method and system based on AI

Abstract

The invention relates to the technical field of video data processing, in particular to an AI-based video editing synthesis method and system, and the method comprises the steps: constructing an optical flow validity index through optical flow direction consistency statistics and an amplitude variation coefficient, and recognizing a cross-fragment optical flow failure boundary; determining an effective reference frame range of each fragment on two sides of the boundary by taking an accumulated value of the inter-frame normalized optical flow amplitude as a stop condition; constructing an influence factor of each reference frame according to the time sequence neighbor component and the motion quality component; constructing a weighted covariance matrix according to the influence factors, adaptively selecting effective structure feature dimensions through eigenvalue decomposition, and respectively forming projection matrixes of fragments on two sides; and under the joint constraint of the bilateral projection matrix, minimizing the bilateral projection residual error by taking the effective reference frame count as a weighting coefficient, and generating a transition frame. According to the method, the problems of reference frame range determination, reference frame contribution quantization and effective feature extraction in the AI-based transition frame generation process in the video editing and splicing process are solved.

Description

Technical Field

[0001] This invention relates to the field of video data processing technology, specifically to an AI-based video editing and compositing method and system. Background Technology

[0002] With the popularization of short video creation, users often stitch together multiple video clips shot independently at different times, locations, and by different filmmakers, according to the beat of music. When performing real-time synthesis of the stitched sequences on mobile devices, it is often necessary to insert transition frames at the hard-cut boundaries to improve the visual continuity.

[0003] Existing video frame interpolation techniques are mainly divided into two categories: one is the traditional method based on optical flow estimation, represented by optical flow networks such as PWC-Net and RAFT. This method estimates the dense optical flow field between consecutive frames and performs bidirectional interpolation of pixels along the motion trajectory to generate intermediate frames. The other category is end-to-end AI frame interpolation networks, represented by DAIN, RIFE, and EMA-VFI. These networks directly regress the pixel values ​​of intermediate frames through convolutional neural networks or Transformer structures. Some models still use optical flow or kernel functions as intermediate representations during the inference stage.

[0004] However, in scenarios where multiple non-homogeneous handheld video clips are hard-cut and stitched together according to a beat, the resulting video editing and stitching effect is sometimes not smooth. The frames on either side of the stitching boundary originate from independent physical shooting processes, with no constraints on scene content, camera posture, or motion mode. The inter-frame optical flow field describes a spurious pixel correspondence between two independent images, and directly using it for frame interpolation will produce severe visual artifacts. In this case, transition frame generation is essentially an image generation problem with the visual structural features of the segments on both sides of the boundary as dual constraints, rather than a motion interpolation problem.

[0005] Generating transition frames requires taking several frames from both sides of the boundary as structural references. However, the effective range of the reference frames cannot be preset with a fixed window. The range of motion in video shooting varies from person to person and from scene to scene. Including far-end frames with excessively large motion ranges will introduce motion blur interference, while too few reference frames will lead to unstable statistical estimation of structural features. Therefore, in the process of using AI to edit and synthesize multiple video segments for different scenes, how to select the best reference frames for the preceding and following segments and generate transition frames based on the reference frames is an urgent problem to be solved. Summary of the Invention

[0006] (1) Technical problems to be solved

[0007] The purpose of this invention is to provide an AI-based video editing and compositing method and system to solve the problems of determining the reference frame range, quantifying the contribution of reference frames, and extracting effective features in the AI-based transition frame generation process during video editing and splicing.

[0008] (2) Technical solution

[0009] To achieve the above objectives, in one aspect, the present invention provides an AI-based video editing and compositing method, the method comprising:

[0010] The frame calculation is used to determine whether the previous segment's last frame and the next segment's first frame are cross-segment optical flow failure boundaries. If they are cross-segment optical flow failure boundaries, the number of effective reference frames for the previous and next segments is determined. The normalized optical flow amplitude series of each pair of adjacent frames is recorded. The influence factor of each frame in the effective reference frames corresponding to the previous and next segments is calculated.

[0011] The gradient and chromaticity statistics extracted from each frame in the effective reference frame are concatenated to form a local structural feature vector; the weighted covariance matrix is ​​obtained according to the influence factor and local structural feature vector of each frame in the effective reference frame of the previous and subsequent segments; the weighted covariance matrix of the previous and subsequent segments is decomposed by eigenvalue and the mean of the corresponding eigenvalue is calculated. The principal direction feature vectors of the previous and subsequent segments are selected to form the projection matrix of the previous and subsequent segments respectively.

[0012] The ratios of the number of effective reference frames in the previous segment and the number of effective reference frames in the subsequent segment to the sum of the number of effective reference frames in the previous and subsequent segments are used as the balance coefficients for the previous and subsequent segments, respectively. The balance coefficients for the previous and subsequent segments are used as the weighted residuals of the local structural feature vectors of the transition frame under the projection matrices of the previous and subsequent segments, respectively. The pixel values ​​corresponding to the local structural feature vectors that minimize the weighted residuals are recorded as constraints. The constraints are injected into the AI ​​frame interpolation network that generates the transition frames. The AI ​​frame interpolation network is a pre-trained neural network model that receives the constraints and generates transition frames accordingly.

[0013] Furthermore, the method for determining whether the calculated frame of the last frame of the preceding segment and the first frame of the following segment corresponding to the transition frame to be edited and synthesized is a cross-segment optical flow failure boundary; if it is a cross-segment optical flow failure boundary, the number of effective reference frames for the preceding and following segments is determined; and the normalized optical flow amplitude sequence of each pair of adjacent frames is recorded; the method for calculating the influence factor of each frame in the effective reference frames corresponding to the preceding and following segments is as follows:

[0014] The dense optical flow field of the inter-frame pixel displacement is calculated between the last frame of the preceding segment and the first frame of the following segment corresponding to the transition frame to be edited and synthesized. The optical flow direction angle and optical flow amplitude of each pixel are obtained through the dense optical flow field. The direction consistency statistic is calculated through the optical flow direction angle of each pixel. The amplitude variation coefficient is calculated through the mean and standard deviation of the optical flow amplitude of each pixel. The optical flow effectiveness index is obtained through the direction consistency statistic and the amplitude variation coefficient. The optical flow effectiveness index and the preset effectiveness threshold are used to determine whether it is a cross-segment optical flow failure boundary.

[0015] When a cross-segment optical flow failure boundary is determined, the optical flow amplitude between each pair of adjacent frames within the segment is calculated for each pair of adjacent frames in the preceding segment from the last frame forward and for each pair of adjacent frames in the following segment from the first frame backward. After normalization with the frame diagonal length, the amplitude is accumulated pair by pair. The number of valid reference frames for the preceding and following segments is determined when the accumulated value first reaches the preset accumulated amplitude threshold. The normalized optical flow amplitude sequence of each pair of adjacent frames is also recorded.

[0016] The frame sequence distance of each effective reference frame from the splicing boundary is taken as the temporal nearest neighbor component, and the difference between the preset cumulative amplitude threshold and the normalized optical flow amplitude corresponding to each effective reference frame is taken as the motion quality component. The product of the temporal nearest neighbor component and the motion quality component corresponding to the effective reference frames of the previous segment and the next segment is normalized respectively to obtain the influence factor of each frame in the effective reference frames corresponding to the previous segment and the next segment.

[0017] Further, the method of calculating the dense optical flow field of inter-frame pixel displacement between the last frame of the preceding segment and the first frame of the following segment corresponding to the transition frame to be edited and synthesized, obtaining the optical flow direction angle and optical flow amplitude of each pixel through the dense optical flow field; calculating the direction consistency statistic through the optical flow direction angle of each pixel, calculating the amplitude variation coefficient through the mean and standard deviation of the optical flow amplitude of each pixel; obtaining the optical flow effectiveness index through the direction consistency statistic and the amplitude variation coefficient, and determining whether it is a cross-segment optical flow failure boundary based on the optical flow effectiveness index and a preset effectiveness threshold includes:

[0018] Calculate the pixel-by-pixel dense optical flow field for the last frame of the preceding segment and the first frame of the following segment to obtain the distance between the last frame of the preceding segment and the first frame of the following segment. Horizontal optical flow components of each pixel at coordinates With vertical optical flow component , where frame height With frame width , Values ​​range from 1 to , Values ​​range from 1 to The total number of intra-frame pixels between the last frame of the preceding segment and the first frame of the following segment is: ; Calculate the optical flow direction angle of each pixel and the optical flow amplitude of each pixel ;

[0019] Traverse each pixel coordinate Until all are traversed Each pixel represents the entire dense optical flow field between the last frame of the previous segment and the first frame of the subsequent segment. Direction angle of 1 pixel Calculate the Rayleigh directional consistency statistic :

[0020] ;

[0021] Based on the entire dense optical flow field between the last frame of the preceding segment and the first frame of the following segment Optical flow amplitude per pixel Calculate the mean amplitude and amplitude standard deviation To prevent small amounts of zero positive Calculate the amplitude variation coefficient ;

[0022] when hour ,when hour ;when The splicing boundary is then marked as the cross-segment optical flow failure boundary. This is a preset validity threshold.

[0023] Furthermore, when a cross-segment optical flow failure boundary is determined, the method of calculating the optical flow amplitude between pairs of adjacent frames within the segment for each pair of adjacent frames in the preceding segment from the last frame forward and for each pair of adjacent frames in the following segment from the first frame backward, normalizing the amplitude by the frame diagonal length and accumulating them pair by pair, and determining the number of valid reference frames for the preceding and following segments when the accumulated value first reaches a preset accumulated amplitude threshold, and recording the normalized optical flow amplitude sequence of each pair of adjacent frames includes:

[0024] Calculate the frame diagonal length For the preceding segment, trace back from the last frame backwards, using the sequence number. Mark each pair of adjacent frames and traverse each pixel coordinate. Until all are traversed The pixel is calculated from the previous segment. Normalized average optical flow amplitude between adjacent frames :

[0025] ;

[0026] in and For the previous segment For adjacent frames in The horizontal and vertical optical flow components at the coordinates are obtained by pairwise accumulation. , Values ​​range from 1 to ; Determine whether it satisfies and The smallest positive integer This represents the number of valid reference frames for the preceding segment. Set a preset cumulative motion threshold;

[0027] For each subsequent segment, starting from the first frame and working backwards, calculate the segment's [number]th frame in the same way. Normalized average optical flow amplitude between adjacent frames and cumulative value , Values ​​range from 1 to ; Determine whether it satisfies and The smallest positive integer The number of effective reference frames for the subsequent segment; the normalized optical flow amplitude series of the preceding and subsequent segments are output sequentially. and .

[0028] Further, the method of taking the frame sequence distance of each effective reference frame from the splicing boundary as the temporal nearest neighbor component, and the difference between the preset cumulative amplitude threshold and the normalized optical flow amplitude corresponding to each effective reference frame as the motion quality component; and normalizing the product of the temporal nearest neighbor component and the motion quality component corresponding to the effective reference frames of the preceding and following segments to obtain the influence factor of each frame in the effective reference frames corresponding to the preceding and following segments includes:

[0029] The position number of the previous segment is Valid reference frames, in which ,when The time corresponds to the last frame; define the temporal nearest neighbor component. Motion clarity component Impact Factor for:

[0030] ;

[0031] in Values ​​range from 1 to The position number of the subsequent segment is Valid reference frames, in which ,when Corresponding to the first frame; define the temporal nearest neighbor component. Motion clarity component Impact Factor for:

[0032] ;

[0033] in Values ​​range from 1 to .

[0034] Furthermore, the method of forming a local structural feature vector by concatenating the gradient and chromaticity statistics extracted from each frame of the effective reference frame after dividing it into fixed pixel blocks includes:

[0035] Each frame image is fixed Pixel blocks are uniformly divided across the entire frame; gradient directions are calculated for each pixel within each block, using a fixed number of direction intervals. Perform directional histogram statistics and normalization to obtain each block 3D gradient orientation histogram; extract within each block Color space Component mean and The component mean is a two-dimensional chromaticity statistic; the gradient direction histogram of each block in the whole frame is sequentially concatenated with the chromaticity statistic to obtain... A local structural feature vector, where:

[0036] ;

[0037] The position number of the previous segment is The valid reference frame is extracted in the manner described above. The dimensional local structural eigenvector is denoted as The position number of the subsequent segment is The valid reference frame is extracted in the manner described above. The dimensional local structural eigenvector is denoted as .

[0038] Further, the method of obtaining the corresponding weighted covariance matrix based on the influence factors and local structural feature vectors of each frame in the effective reference frames of the preceding and following segments; decomposing the weighted covariance matrices of the preceding and following segments into eigenvalues ​​and calculating their corresponding eigenvalue mean; and selecting the principal direction feature vectors of the preceding and following segments to form the projection matrices of the preceding and following segments respectively includes:

[0039] Calculate the weighted mean vector of the preceding segments Construct the weighted covariance matrix of the first segment ;right Perform eigenvalue decomposition on a real symmetric matrix to obtain eigenvalues ​​in descending order. ; Statistical satisfaction The number of subscripts is denoted as , ,in Values ​​range from 1 to Take the front The eigenvectors are arranged in columns to form the projection matrix of the first segment. , for OK Column matrix;

[0040] Calculate the weighted mean vector of the segments Construct the weighted covariance matrix of the subsequent segments. ;right Perform eigenvalue decomposition on a real symmetric matrix to obtain eigenvalues ​​in descending order. ;

[0041] Statistical satisfaction The number of subscripts is denoted as , ,in Values ​​range from 1 to Take the front The eigenvectors are arranged in columns to form the projection matrix of the post-segment. , for OK Column matrix.

[0042] Furthermore, the normalized average optical flow amplitude between the first pair of adjacent frames inward from the last frame of the current segment. satisfy At that time, the number of valid reference frames in the previous segment will be... Set to 1, extracted from the last frame of the previous segment. The dimensional local structural eigenvector is denoted as ,by The normalized column vector obtained by dividing by its Euclidean norm is directly used as the projection matrix of the preceding segment. Effective feature dimensions of the preceding segment Set to 1, the influence factor of the last frame of the previous segment. Set to 1;

[0043] The normalized average optical flow amplitude between the first pair of adjacent frames in the subsequent segment satisfy At that time, the number of valid reference frames for the subsequent segment will be determined. Set to 1, and the first frame extracted from subsequent segments The dimensional local structural eigenvector is denoted as ,by The normalized column vector obtained by dividing by its Euclidean norm is directly used as the projection matrix of the subsequent segment. Effective feature dimensions of the later segment Set to 1, then the impact factor of the first frame of the subsequent segment. Set to 1.

[0044] Further, the ratios of the number of effective reference frames in the previous segment and the number of effective reference frames in the subsequent segment to the sum of the number of effective reference frames in the previous and subsequent segments are used as the balance coefficients for the previous and subsequent segments, respectively. These balance coefficients are used as the weighted residuals of the local structural feature vectors of the transition frame, minimizing the weighted residuals under the projection matrices of the previous and subsequent segments. The pixel values ​​corresponding to the local structural feature vectors that minimize the weighted residuals are recorded as constraints. The method of injecting these constraints into the AI ​​frame interpolation network that generates the transition frames includes:

[0045] Calculate the pre-fragment balance coefficient Balance coefficient with subsequent segments ;

[0046] Calculate the pre-fragment side Vegan weight center of mass ,in Projection matrix of the previous segment The transpose matrix; calculate the fragment side. Vegan weight center of mass ,in Projection matrix of the later segment The transpose of the matrix;

[0047] With transition frames 3D local structural feature vector Using the independent variable, minimize the objective function. :

[0048] ;

[0049] Calculate the minimum objective function When the minimum value is obtained, the corresponding A local structural feature vector, denoted as . ; the preceding pixel block The first two components serve as constraints on the gradient direction intensity distribution of each pixel block, while the last two components serve as constraints on the gradient direction intensity distribution of each pixel block. Component mean and The component mean constraint is used to reconstruct the pixel values ​​of each pixel block according to the above constraints, and these constraints are then injected into the AI ​​frame interpolation network that generates the transition frame.

[0050] Based on the same inventive concept, this invention also provides an AI-based video editing and compositing system, the system comprising:

[0051] The composite frame impact assessment module is used to determine whether the calculated frame of the last frame of the previous segment and the first frame of the next segment corresponding to the transition frame to be edited and synthesized is a cross-segment optical flow failure boundary; if it is a cross-segment optical flow failure boundary, the number of effective reference frames of the previous segment and the next segment is determined; and the normalized optical flow amplitude series of each pair of adjacent frames is recorded; and the impact factor of each frame in the effective reference frames corresponding to the previous segment and the next segment is calculated.

[0052] The composite frame feature extraction module is used to form a local structural feature vector by concatenating the gradient and chromaticity statistics extracted from each frame in the effective reference frame after dividing it into fixed pixel blocks; the module obtains the corresponding weighted covariance matrix based on the influence factors and local structural feature vectors of each frame in the effective reference frames of the preceding and following segments; the module decomposes the weighted covariance matrices of the preceding and following segments into eigenvalues ​​and calculates the mean of the corresponding eigenvalues; the module then selects the principal direction feature vectors of the preceding and following segments to form the projection matrices of the preceding and following segments, respectively.

[0053] The composite frame generation module is used to take the ratios of the number of effective reference frames in the previous segment and the number of effective reference frames in the subsequent segment to the sum of the number of effective reference frames in the previous and subsequent segments, respectively, as the balance coefficients of the previous and subsequent segments. The balance coefficients of the previous and subsequent segments are used as the weighted residuals of the local structural feature vectors of the transition frame under the projection matrices of the previous and subsequent segments, which minimize the weighted residuals. The pixel values ​​corresponding to the local structural feature vectors that minimize the weighted residuals are recorded as constraints. The constraints are injected into the AI ​​frame interpolation network that generates the transition frames. The AI ​​frame interpolation network is a pre-trained neural network model that receives the constraints and generates transition frames accordingly.

[0054] (3) Beneficial effects

[0055] Compared with the prior art, the beneficial effects of the present invention are:

[0056] This invention addresses the problem of existing frame interpolation methods being unsuitable for hard-cutting and splicing multiple video clips due to the failure of motion continuity assumptions, enabling smoother transition frame synthesis at clip boundaries. It extracts stable and interference-controlled structural references for video footage with varying motion amplitudes, eliminating the problem of excessive variance in generation quality caused by fixed-window strategies between footage with significantly different motion amplitudes.

[0057] The generation of transition frames is solved by quantitatively analyzing the features of reference frames. Based on the structural features of reference frames near the boundary, and also by appropriately referencing reference frames at the far end, the transition frames maintain visual similarity to the segments on both sides of the boundary in terms of brightness distribution, tone consistency, and spatial structure, thereby reducing the intensity of inter-frame jumps in the perceptual domain.

[0058] The adaptive selection of effective feature dimensions uses the principal direction subspace of the weighted covariance matrix as a generation constraint, ensuring that the constraint only applies to structural directions with significant variance in the reference frame set, thus avoiding contamination from the noise dimension of the video. Attached Figure Description

[0059] The above and / or other aspects of this application will become more apparent from the description of certain embodiments with reference to the accompanying drawings, in which:

[0060] Figure 1 This is a flowchart of an AI-based video editing and compositing method according to Embodiment 1 of the present invention;

[0061] Figure 2 This is a block diagram of an AI-based video editing and compositing system according to Embodiment 2 of the present invention;

[0062] Figure 3 This is a schematic diagram of video editing, compositing, and splicing based on an AI-based video editing and compositing method according to Embodiment 1 of the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Before providing examples, it's necessary to describe the application scenario of this invention. A user takes multiple video clips independently filmed at different times, locations, and by different photographers, and splices them together according to the music beat. The spliced ​​sequence is then synthesized in real-time on a mobile device, with transition frames inserted at the hard-cut boundaries to improve visual continuity. The frames on either side of the hard-cut boundary originate from independent physical shooting processes, with no constraints on scene content, camera pose, or motion mode. The inter-frame optical flow field describes the spurious pixel correspondence between two independent images; directly using it for frame interpolation will produce severe visual artifacts. Therefore, the generation of transition frames is essentially an image generation problem with the visual structural features of the clips on both sides of the boundary as dual constraints, rather than a motion interpolation problem.

[0065] Example 1: This example uses a real-time compositing scenario on a mobile music short video platform as its specific application background. The user uploads three video clips shot at 1080p resolution and 30fps. Hard-cutting and splicing are performed at 0.5 seconds, 1.2 seconds, and 2.7 seconds on the timeline according to the music beat, forming three splicing boundaries. Taking the first splicing boundary as an example, the complete method flow is described step by step. The first clip is the video segment before the splicing point, and the second clip is the video segment after the splicing point. The two videos are independently shot by different people at different locations, with independent scene content, camera movement modes, and lighting conditions, and no continuous motion constraints.

[0066] like Figure 1 As shown, this embodiment provides an AI-based video editing and compositing method, the method including:

[0067] The frame calculation is performed to determine whether the last frame of the preceding segment and the first frame of the following segment correspond to the transition frame to be edited and synthesized are cross-segment optical flow failure boundaries. If they are cross-segment optical flow failure boundaries, the number of effective reference frames for the preceding and following segments is determined. The normalized optical flow amplitude series of each pair of adjacent frames is recorded. The influence factor of each frame in the effective reference frames corresponding to the preceding and following segments is calculated.

[0068] The judgments, determinations, and calculations involved in this step are all carried out step by step according to the specific methods described in the following paragraphs. Here, we will first give an overall explanation of the input and output relationship of each sub-step: the input is the image of the last frame of the previous segment. The first frame image of the subsequent segment And the frame sequence of the preceding segment from the last frame forward. ( That is, the last frame and subsequent segments are the frame sequences from the first frame onwards. ( (i.e., the first frame); the output is the number of valid reference frames for the previous segment. Number of valid reference frames for subsequent segments Normalized optical flow amplitude sequence of each pair of adjacent frames in the preceding segment Normalized optical flow amplitude series of the subsequent segment Influence factors of each effective reference frame in the preceding segment Influence factors of each effective reference frame in the subsequent segment This is used for subsequent feature extraction and transition frame generation steps.

[0069] In this step, the specific implementation of the optical flow estimation module is not limited; a dense optical flow network based on convolutional neural networks can be used, including but not limited to PWC-Net, RAFT, FlowNet2, etc., or a traditional variational or iterative dense estimation method based on Horn-Schunck or Lucas-Kanade can be used, as long as it can output the pixel-by-pixel horizontal optical flow components between the last frame of the previous segment and the first frame of the subsequent segment. With vertical optical flow component The pixel-wise optical flow components between each pair of adjacent frames in the preceding and following segments all meet the requirements of this step. In another embodiment, sparse optical flow (such as Shi-Tomasi corner detection combined with pyramid Lucas-Kanade tracking) can also be used to obtain a sparse optical flow field, which is then extended into a dense field by bilinear interpolation. The resulting direction consistency statistics and amplitude variation coefficients have slight differences in numerical value from the fully dense estimation, but they are essentially equivalent to the dense estimation in determining the cross-segment optical flow failure boundary, and fall within the scope of protection of this invention. In yet another embodiment, the disparity map output by the depth estimation network can also be used instead of the optical flow components as a proxy metric for inter-frame motion. As long as the proxy metric can reflect the spatial distribution of pixel-level motion amplitude between adjacent frames, the effective reference frame range can be determined in subsequent steps, which is functionally equivalent to the optical flow components.

[0070] It should be noted that the core value of this step lies in distinguishing between "spurious optical flow across segments" and "real motion optical flow within segments" through optical flow statistics, and adaptively determining the effective reference frame range based on the cumulative amplitude of the real motion optical flow within segments, rather than limiting the optical flow estimation method itself; any technical means that can distinguish the above two types of optical flow and determine the reference frame range accordingly embodies the core concept of this invention.

[0071] Furthermore, the method for determining whether the calculated frame of the last frame of the preceding segment and the first frame of the following segment corresponding to the transition frame to be edited and synthesized is a cross-segment optical flow failure boundary; if it is a cross-segment optical flow failure boundary, the number of effective reference frames for the preceding and following segments is determined; and the normalized optical flow amplitude sequence of each pair of adjacent frames is recorded; the method for calculating the influence factor of each frame in the effective reference frames corresponding to the preceding and following segments is as follows:

[0072] The dense optical flow field of the inter-frame pixel displacement is calculated between the last frame of the preceding segment and the first frame of the following segment corresponding to the transition frame to be edited and synthesized. The optical flow direction angle and optical flow amplitude of each pixel are obtained through the dense optical flow field. The direction consistency statistic is calculated through the optical flow direction angle of each pixel. The amplitude variation coefficient is calculated through the mean and standard deviation of the optical flow amplitude of each pixel. The optical flow effectiveness index is obtained through the direction consistency statistic and the amplitude variation coefficient. The optical flow effectiveness index and the preset effectiveness threshold are used to determine whether it is a cross-segment optical flow failure boundary.

[0073] Last frame of the previous segment The first frame of the following clip Calculate the dense optical flow field for inter-frame pixel displacement, and obtain the optical flow direction angle of each pixel through the dense optical flow field. and optical flow amplitude ; through the optical flow direction angle of each pixel Calculate the directional consistency statistic The average optical flow amplitude of each pixel with standard deviation Calculate the amplitude variation coefficient ; through directional consistency statistics With amplitude variation coefficient Construction of optical flow efficiency index and according to Compared with the preset validity threshold The size relationship is used to determine whether the current splicing boundary belongs to the cross-segment optical flow failure boundary.

[0074] When a cross-segment optical flow failure boundary is determined, the optical flow amplitude between each pair of adjacent frames within the segment is calculated for each pair of adjacent frames in the preceding segment from the last frame forward and for each pair of adjacent frames in the following segment from the first frame backward. After normalization with the frame diagonal length, the amplitude is accumulated pair by pair. The number of valid reference frames for the preceding and following segments is determined when the accumulated value first reaches the preset accumulated amplitude threshold. The normalized optical flow amplitude sequence of each pair of adjacent frames is also recorded.

[0075] Once a cross-segment optical flow failure boundary is identified, optical flow calculations are performed pairwise for the preceding segment from the last frame forward and for the following segment from the first frame backward, using the frame diagonal length. The average optical flow amplitude between each pair of adjacent frames is normalized, and the normalized amplitude is accumulated pair by pair. The point is reached when the accumulated value first exceeds a preset accumulated amplitude threshold. As a cutoff condition, determine the number of valid reference frames for the preceding segment. Number of valid reference frames for subsequent segments And record the normalized optical flow amplitude series of the preceding and following segments.

[0076] The frame sequence distance of each effective reference frame from the splicing boundary is taken as the temporal nearest neighbor component, and the difference between the preset cumulative amplitude threshold and the normalized optical flow amplitude corresponding to each effective reference frame is taken as the motion quality component. The product of the temporal nearest neighbor component and the motion quality component corresponding to the effective reference frames of the previous segment and the next segment is normalized respectively to obtain the influence factor of each frame in the effective reference frames corresponding to the previous segment and the next segment.

[0077] The frame sequence distance of each effective reference frame splicing boundary is defined as the temporal nearest neighbor component, with a preset cumulative amplitude threshold. The difference between the normalized optical flow amplitude corresponding to the reference frame is defined as the motion quality component. The influence factor of each effective reference frame is obtained by normalizing the product of the temporal nearest neighbor component and the motion quality component.

[0078] Further, the method of calculating the dense optical flow field of inter-frame pixel displacement between the last frame of the preceding segment and the first frame of the following segment corresponding to the transition frame to be edited and synthesized, obtaining the optical flow direction angle and optical flow amplitude of each pixel through the dense optical flow field; calculating the direction consistency statistic through the optical flow direction angle of each pixel, calculating the amplitude variation coefficient through the mean and standard deviation of the optical flow amplitude of each pixel; obtaining the optical flow effectiveness index through the direction consistency statistic and the amplitude variation coefficient, and determining whether it is a cross-segment optical flow failure boundary based on the optical flow effectiveness index and a preset effectiveness threshold includes:

[0079] Calculate the pixel-by-pixel dense optical flow field for the last frame of the preceding segment and the first frame of the following segment to obtain the distance between the last frame of the preceding segment and the first frame of the following segment. Horizontal optical flow components of each pixel at coordinates With vertical optical flow component , where frame height With frame width , Values ​​range from 1 to , Values ​​range from 1 to The total number of intra-frame pixels between the last frame of the preceding segment and the first frame of the following segment is: ; Calculate the optical flow direction angle of each pixel and the optical flow amplitude of each pixel ;

[0080] Traverse each pixel coordinate Until all are traversed Each pixel represents the entire dense optical flow field between the last frame of the previous segment and the first frame of the subsequent segment. Direction angle of 1 pixel Calculate the Rayleigh directional consistency statistic :

[0081] ;

[0082] Based on the entire dense optical flow field between the last frame of the preceding segment and the first frame of the following segment Optical flow amplitude per pixel Calculate the mean amplitude and amplitude standard deviation To prevent small amounts of zero positive Calculate the amplitude variation coefficient ;

[0083] when hour ,when hour ;when The splicing boundary is then marked as the cross-segment optical flow failure boundary. This is a preset validity threshold.

[0084] Last frame of the previous segment The first frame of the following clip Perform pixel-by-pixel dense optical flow estimation; set the frame height as... Frame width is In this embodiment , Total number of pixels in the frame For coordinates Pixels at that location, Take from 1 , Take from 1 Let the horizontal optical flow component be denoted as The vertical optical flow component is Calculate the optical flow direction angle of each pixel. and the optical flow amplitude of each pixel .

[0085] In this embodiment, a RAFT network is used to... and Dense optical flow estimation is performed, outputting a pixel-by-pixel optical flow field at full resolution (1080×1920). The RAFT network, pre-trained on the Sintel and KITTI datasets, outperforms PWC-Net by approximately 15% in endpoint error (EPE) of mobile video footage. After INT8 quantization on mobile devices, it can complete optical flow estimation for a single frame pair within 35ms, meeting real-time processing requirements. In another implementation, PWC-Net, FlowNet2, SpyNet, or other dense optical flow networks can also be used. These networks all produce highly dispersed results in the optical flow amplitude distribution across segment boundary frame pairs (because the content of the two frames is unrelated), which does not substantially affect the final failure boundary determination.

[0086] Traverse all 1 pixel, with all Direction angle of 1 pixel Calculate the Rayleigh directional consistency statistic When the optical flow direction of all pixels is completely consistent When the direction is When uniformly random, The expectation approaches 0, and as Increasing the convergence speed accelerates the process. In this embodiment... =2073600, at the cross-fragment boundary The estimated variance is extremely small, and the statistical reliability is fully guaranteed.

[0087] Rayleigh statistic It is a standard quantity in circular statistics that characterizes the degree of angular concentration, corresponding to the magnitude of the resultant force of vectors on the unit circle. Its calculation can be completed by traversing only one pixel, with a computational complexity of O(n log n). Suitable for real-time processing on mobile devices. The Rayleigh statistic is equivalent to the estimator of the first moment of the von Mises distribution, and asymptotically equivalent to the maximum likelihood estimator in large samples. Therefore, any statistic that can measure the degree of angular concentration or directional randomness is similar to... They are essentially equivalent in function.

[0088] In another implementation, the directional consistency statistic can be replaced by the complement of the circular standard deviation, defined as follows: ,in To be equivalent to the resultant force modulus of this embodiment, and Since they exhibit a monotonically increasing relationship, they are functionally equivalent in determining the randomness of direction. When ( To correspond with the adjusted preset threshold, implementation must be based on... When selecting the actual range of values, it is determined as a cross-fragment failure boundary. When using this statistic, avoid... Extremely small values ​​can cause numerical overflow, which is possible. Truncating the directional consistency statistic. In another implementation, the von Mises distribution concentration parameter can be used instead of the directional consistency statistic. Maximum likelihood estimation, Solving equations numerically Received, among which , For zero-order and first-order modified Bessel functions; when ( When a preset concentration threshold (typically ranging from 0.4 to 0.8) is used, the direction is determined randomly, consistent with this embodiment. If the functions are equivalent, they are essentially the same.

[0089] According to all Optical flow amplitude per pixel Calculate the mean amplitude and amplitude standard deviation To prevent small quantities of zero or positive Calculate the amplitude variation coefficient .

[0090] In this embodiment In double-precision floating-point arithmetic, this is sufficient to prevent the denominator from being zero, while also ensuring... The actual impact of the numerical value is lower than The magnitude is negligible.

[0091] The physical meaning of optical flow is the relative dispersion of optical flow amplitude. Within the same continuous shooting scene, the overall motion between the camera and the scene makes the optical flow amplitude distribution of each pixel relatively uniform. Typically below 0.6. At cross-fragment boundaries, where the content of two frames is completely unrelated, the optical flow estimation network gives highly scattered amplitude values ​​on meaningless pixel correspondences. Typically higher than 1.5, and may reach 2.0 or higher. This occurs when there is foreground motion (such as a person walking) superimposed on a background overall translation. In the intermediate region (0.6 to 1.2), this situation occurs within frame pairs within the same segment, but... The value usually remains high (because although the optical flow directions of the foreground and background are different, they are not random), thus the overall optical flow efficiency index... It can still be distinguished from the cross-segment case. Robustness to this boundary case is demonstrated within a preset validity threshold. This has already been taken into consideration in the selection process.

[0092] As a dimensionless quantity, it is unaffected by the overall amplitude of motion between frames, reflecting only the relative uniformity of amplitude distribution. Therefore, it is applicable to both handheld jitter (small amplitude) and large-scale translation (large amplitude) scenarios, exhibiting scale independence. All statistical quantities that can measure the spatial uniformity of optical flow amplitude distribution are related to… They are functionally equivalent. In another embodiment, the amplitude variation coefficient can be replaced with the Gini coefficient. The summation symbol express and Each element independently traverses the entire frame. All combinations of pixel coordinates. A higher Gini coefficient indicates a more uneven amplitude distribution. When the value exceeds the preset Gini threshold, the amplitude is highly dispersed, compared to The determination function is essentially equivalent. In another embodiment, the amplitude variation coefficient can be replaced with excess kurtosis. The kurtosis of a normal distribution is 3, and the excess kurtosis is 0. When the amplitude distribution exhibits a heavy-tailed distribution (lumpy or long-tailed), the excess kurtosis is significantly positive. Typically, a value higher than 3.0 indicates an abnormally concentrated or extremely dispersed distribution, consistent with high... Functionally equivalent in detecting abnormal amplitude distribution. Specifically, the optical flow amplitude of each pixel... Amplitude mean , As defined above; all the coefficients of variation of the above substitution amplitudes are within the scope of protection of this invention.

[0093] when season ;when season . Requires consistent direction ( High) and uniform amplitude ( Only when both low and high conditions are met can a higher value be obtained. Value; failure to meet any one of the conditions will result in Significantly reduced. When At that point, the amplitude distribution was extremely uneven, and regardless of directional consistency, the optical flow field was unreliable; therefore, it was forced to... This is a hard truncation in the design to prevent misjudgment due to accidental directional consistency (e.g., both frames happen to be large-scale unidirectional blur) in cases of high amplitude dispersion.

[0094] when At this time, the current splicing boundary is marked as a cross-segment optical flow failure boundary; in this embodiment Experiments on a 1080p resolution mobile phone video dataset (containing 500 pairs of frames within the same segment and 500 pairs of frames across segments) show that the frame pairs within the same segment... Values ​​distributed in Within the interval, the mean is 0.68 and the standard deviation is 0.11; across frame pairs... Values ​​distributed in Within the interval, the mean is 0.03 and the standard deviation is 0.02. There is a significant gap between the two distributions. Select within range Both methods achieve a cross-segment false negative rate of 0 and a same-segment false negative rate of less than 0.4%. Achieving zero false positive and zero false negative rates under the above experimental conditions is the optimal choice. For 480p resolution footage, the endpoint error of the optical flow estimation network increases by approximately 40%, so it is recommended to... Lower the value to 0.20 to reduce the false positive rate; for 4K resolution footage, the optical flow estimation accuracy is higher, which can... Adjust it appropriately to 0.35. Those skilled in the art can estimate the network accuracy and the resolution range of the target application based on the deployed optical flow. Select a suitable parameter range All values ​​can achieve stable cross-segment optical flow failure boundary determination, and all values ​​within the above parameter range are within the protection scope of this invention.

[0095] At the first stitching boundary in this embodiment, the result estimated by the RAFT network is... , ,because Therefore ,satisfy If it is determined to be a cross-segment optical flow failure boundary, the process will proceed to the determination of subsequent valid reference frames.

[0096] In another embodiment, the combination of optical flow efficiency indices can be changed to a weighted sum:

[0097] ;

[0098] in The preset weighting parameters typically range from 0.4 to 0.6. and When one signal approaches zero while the other still has a signal, some valid information can be retained, which is beneficial for single-indicator degradation (such as between still frames). Extremely low but the scene has changed, resulting in It exhibits stronger robustness in boundary cases with extremely low values; the values ​​given when both metrics are moderate are... The value is too high, and the corresponding judgment threshold needs to be appropriately lowered. (typically) Adjusted to (60% to 70%). This alternative combination is substantially equivalent to the present embodiment in terms of function in determining cross-segment optical flow failure boundaries, and falls within the scope of protection of this invention. In another embodiment, it can be completely independent of the optical flow field and instead use a direct image distance metric: calculate the cosine distance between the last frame of the preceding segment and the first frame of the following segment in the perceptual feature space (such as the features of the third pooling layer of VGG-16). ,when ( When a preset sensing distance threshold is used, it is determined as a cross-segment failure boundary; this method avoids the optical flow estimation step, has deployment advantages on extremely resource-constrained edge devices, and is substantially equivalent to the optical flow effectiveness index method in detecting inter-frame scene discontinuities, thus falling within the scope of protection of this invention. In another embodiment, a lightweight binary classification convolutional neural network (such as a MobileNet V3 backbone network connected to a binary classification head) can be used to perform end-to-end cross-segment discrimination on the stitched image of the last frame of the previous segment and the first frame of the subsequent segment, outputting the cross-segment probability. ,when The time marker is used as the failure boundary; the network can be trained under supervision on large-scale cross-segment / same-segment frame pairs of data, and the discrimination accuracy is higher than that of the method based on manual statistics. Moreover, the inference latency can be controlled within 5ms on mobile devices. This alternative determination method is substantially equivalent to the determination method based on optical flow statistics in this embodiment in terms of the function of cross-segment optical flow failure boundary identification, and falls within the protection scope of this invention.

[0099] Furthermore, when a cross-segment optical flow failure boundary is determined, the method of calculating the optical flow amplitude between pairs of adjacent frames within the segment for each pair of adjacent frames in the preceding segment from the last frame forward and for each pair of adjacent frames in the following segment from the first frame backward, normalizing the amplitude by the frame diagonal length and accumulating them pair by pair, and determining the number of valid reference frames for the preceding and following segments when the accumulated value first reaches a preset accumulated amplitude threshold, and recording the normalized optical flow amplitude sequence of each pair of adjacent frames includes:

[0100] Calculate the frame diagonal length For the preceding segment, trace back from the last frame backwards, using the sequence number. Mark each pair of adjacent frames and traverse each pixel coordinate. Until all are traversed The pixel is calculated from the previous segment. Normalized average optical flow amplitude between adjacent frames :

[0101] ;

[0102] in and For the previous segment For adjacent frames in The horizontal and vertical optical flow components at the coordinates are obtained by pairwise accumulation. , Values ​​range from 1 to ; Determine whether it satisfies and The smallest positive integer This represents the number of valid reference frames for the preceding segment. Set a preset cumulative motion threshold;

[0103] For each subsequent segment, starting from the first frame and working backwards, calculate the segment's [number]th frame in the same way. Normalized average optical flow amplitude between adjacent frames and cumulative value , Values ​​range from 1 to ; Determine whether it satisfies and The smallest positive integer The number of effective reference frames for the subsequent segment; the normalized optical flow amplitude series of the preceding and subsequent segments are output sequentially. and .

[0104] After determining that the current boundary is a cross-segment optical flow failure boundary, the frame diagonal length is calculated. In this embodiment , Therefore:

[0105] Pixel.

[0106] The purpose of introducing the frame diagonal length as a normalization reference is to normalize the optical flow amplitude. This becomes a dimensionless ratio, eliminating the influence of frame resolution on the dimensions of the accumulated value and ensuring the same preset cumulative motion threshold. A physically consistent cutoff criterion can be provided for video footage of various resolutions from 480p to 4K. Without normalization, for 4K footage... The absolute value of the optical flow amplitude corresponding to the same motion amplitude on the same surface is approximately twice that of 1080p footage, resulting in the same... The cutoff behavior is inconsistent across different resolutions, and parameters cannot be reused across resolutions. The frame diagonal length, rather than the frame width or frame height, is used as the normalization benchmark because optical flow displacement exists in both the horizontal and vertical directions. The frame diagonal length represents the maximum possible displacement within the frame, providing the strictest normalization constraint for any direction of motion and exhibiting direction independence.

[0107] For the preceding segment, trace back from the last frame backwards, using the sequence number. Mark each pair of adjacent frames, where Corresponding to the last frame of the previous segment Its timing is immediately adjacent to the previous frame The frame pairs formed correspond and The resulting frame pairs, and so on, The increase direction is from the last frame back towards the beginning of the segment. For the first... All in adjacent frames Each pixel records its coordinates. The horizontal optical flow component at that location is The vertical optical flow component is Calculate the first segment Normalized average optical flow amplitude between adjacent frames , The physical meaning of the first segment is Between adjacent frames, the average displacement of each pixel relative to the frame's diagonal length is dimensionless and typically between 0.01 and 0.08 (corresponding to typical handheld shooting); however, if the camera is shaking violently or there are fast-moving subjects in the scene, It can exceed 0.15. Accumulate pairwise, then calculate... Determine whether it meets the requirements. and The smallest positive integer This represents the number of valid reference frames for the preceding segment. Set the preset cumulative motion threshold.

[0108] In this embodiment, , This represents the upper limit of the cumulative normalized optical flow amplitude between adjacent frames in the previous segment. When the cumulative value exceeds 15% of the frame diagonal length, it is considered that the frames to be included in the backtracking have produced sufficient viewpoint shift or motion blur, and their structural features have deviated from the visual style at the splicing boundary. Continuing to include them will not improve the quality of the transition frames, but will instead introduce interference.

[0109] Experiments on typical handheld short videos shot at 1080p and 30fps show that: Stable and effective reference frame extraction can be achieved within the range. Typical values ​​are in the range of 3 frames to Between frames; Sufficient number of reference frames ( To ensure the statistical stability of the covariance matrix estimation and the representativeness of the characteristics ( To achieve the optimal balance between (but not too much to avoid including blurry frames), for 60fps footage, the time interval between adjacent frames is halved. The typical value is about half of 30fps, which can Adjusted to 0.08 to cut off within roughly the same time window; for high-speed motion scenarios (sports events, dance). The value is large, and the cumulative value quickly exceeds [the threshold]. , It can be degraded to 1 or 2, at which point the degradation processing mechanism described in the following paragraphs is triggered; for slow-moving landscapes or fixed-camera footage, Extremely small, can The value was appropriately increased to 0.20 to incorporate more reference frames and improve the stability of the covariance matrix estimation. Those skilled in the art can adjust the value based on the material type and frame rate. Adjustment within the range All values ​​within the above range are within the protection scope of this invention.

[0110] At the first splicing boundary in this embodiment, the normalized average optical flow amplitude of each pair of adjacent frames is calculated by backtracking the preceding segments as follows: , , , , The cumulative value at each step is , , , , .satisfy and ,therefore The effective reference frame for the preceding segment is the last frame. Up to the previous frame The normalized optical flow amplitude series is , , , , .

[0111] For each subsequent segment, starting from the first frame and working backwards, calculate the segment's [number]th frame in the same way. Normalized average optical flow amplitude between adjacent frames:

[0112] ;

[0113] in and For the following segment For adjacent frames in coordinates The horizontal and vertical optical flow components at that location, The first frame of the corresponding segment The frame immediately following it in time sequence The frame pairs are formed. Accumulated pairwise, they yield... Determined to satisfy and The smallest positive integer This represents the number of valid reference frames for the subsequent segment.

[0114] In this embodiment, the latter segment is calculated pairwise. , , , The cumulative value at each step is , , , ,satisfy and ,therefore The normalized optical flow amplitude series of the subsequent segment is as follows: , , , .

[0115] Output the normalized optical flow amplitude series of the previous segment in sequence. Normalized optical flow amplitude series of the subsequent segments This is for use in subsequent impact factor calculations.

[0116] It should be noted that the above backtracking process should have a maximum backtracking frame limit set in actual engineering implementation. (Typical values ​​are 20 to 30 frames) to prevent the backtracking computation from increasing unbounded in cases of extremely long segments and very slow motion. When backtracking to... At that time, regardless of the cumulative value Does it exceed All of them Deadline. This engineering constraint does not change the core logic of the algorithm; it only limits the upper bound of the number of reference frames under extremely low-motion footage. It is an implementation-level optimization measure and does not affect the integrity of the technical solution of this invention. Similarly, subsequent segments are also set... As The upper limit.

[0117] In another embodiment, the determination of the number of effective reference frames may not use the cumulative threshold cutoff criterion, but instead use the following equivalent substitution criteria. These are all adaptive reference frame range determination methods based on intra-segment motion statistics as the cutoff basis, and are functionally equivalent to this embodiment, and are all within the protection scope of this invention.

[0118] Firstly, the cutoff criterion of the moving average rate of change is adopted: Calculate the... The width of the center window is (Typical example) =3) inside mean ,when Two consecutive steps below the preset ratio Backtracking stops when the value (typically 0.05 to 0.10) is reached, and the last step before the cutoff is taken. As This criterion can accurately capture the turning point where the motion trend tends to stabilize in segments with monotonically changing motion amplitude, and its robustness to occasional jitters in motion amplitude is better than the single-step accumulation criterion.

[0119] Secondly, an abnormal amplitude detection cutoff criterion is adopted: previously step mean with standard deviation Based on, when ( When the typical value is 2.0 to 3.0, the determination is made for the first... For frames with abnormally large movements (such as sudden camera shake in the middle of the shot), stop the backtracking and let... This criterion is specifically designed for scenarios with abrupt changes in motion rhythm (such as when a cameraman suddenly adjusts the angle during a speech), and it is more accurate than the cumulative threshold criterion in eliminating abnormal frames in such scenarios.

[0120] Third, a perceptual hash cutoff criterion is adopted: for each preceding segment, the perceptual hash (pHash) of adjacent frames is calculated frame by frame from the last frame forward, using Hamming distance. Measurement of the first Differences in content between adjacent frames; when ( Typical values ​​range from 8 to 12 bits, corresponding to a Hamming distance exceeding 12 to 19 of the total hash bits. Backtracking stops when the hash computation is much lower than that of optical flow estimation, making it suitable for deployment on edge devices with extremely limited computing resources. It is also more accurate in situations where content changes drastically (such as frequent scene changes in MV footage that still belong to the same segment).

[0121] Fourth, a structural similarity cutoff criterion is adopted: the structural similarity (SSIM) values ​​between each valid reference frame of the preceding segment and the last frame are calculated. When the SSIM falls below a preset similarity threshold (typically 0.55 to 0.70), backtracking stops, and... This criterion can more accurately exclude structurally discontinuous frames when there are significant changes in the semantic scene (such as the speaker turning around or the subject moving out of the frame) but the inter-frame optical flow amplitude is not abnormal. It is essentially equivalent to the cumulative threshold criterion in determining the effective reference frame range. =1 (meaning the SSIM of the last frame itself is calculated as 1, and the cutoff is not triggered) or When =2, the cutoff condition may be triggered for the first time. If a reference frame cannot be obtained for other reasons when the value is 1, then it degenerates into... For single-frame degradation with a value of 1, the degradation process is performed in accordance with the degradation scenario.

[0122] Fifth, a cutoff criterion based on depth feature distance is adopted: the depth feature vectors of each effective reference frame are extracted using a pre-trained visual encoder (such as DINO ViT-S / 8), and the cosine distance between each reference frame and the last frame is calculated; when the cosine distance exceeds a preset feature distance threshold, the cutoff occurs; this criterion has the strongest perception capability for semantic-level scene changes (such as shooting at the same location at different times, or sudden changes in light), and provides the most accurate range of reference frames in high-quality scenes. It is functionally equivalent to the cumulative threshold criterion and falls within the scope of protection of this invention.

[0123] In another implementation, the normalization reference may not use the frame diagonal length. Instead, the following equivalent replacement benchmark is used: frame width Frame height The larger of the two, or the square root of the frame area. Or a fixed constant (when all input video resolutions are fixed). When using different normalization benchmarks... Different numerical scales, corresponding The values ​​need to be adjusted proportionally, but the physical nature of the cutoff logic remains unchanged, and all of them fall within the protection scope of this invention.

[0124] In another embodiment, the preceding and following segments may not be determined independently. and Instead, a joint cutoff criterion is adopted: (The sum of the cumulative amplitudes of the preceding and following segments) exceeds As the overall cutoff condition, it is determined simultaneously. and This joint cutoff can maintain a relative balance in the number of reference frames on both sides when the motion amplitudes of the preceding and following segments are significantly different. It avoids a serious asymmetry in the quality of covariance matrix estimation caused by an excessive number of reference frames on one side and a very small number on the other side. It is functionally equivalent to the separate independent cutoffs.

[0125] Further, the method of taking the frame sequence distance of each effective reference frame from the splicing boundary as the temporal nearest neighbor component, and the difference between the preset cumulative amplitude threshold and the normalized optical flow amplitude corresponding to each effective reference frame as the motion quality component; and normalizing the product of the temporal nearest neighbor component and the motion quality component corresponding to the effective reference frames of the preceding and following segments to obtain the influence factor of each frame in the effective reference frames corresponding to the preceding and following segments includes:

[0126] The position number of the previous segment is Valid reference frames, in which ,when The time corresponds to the last frame; define the temporal nearest neighbor component. Motion clarity component Impact Factor for:

[0127] ;

[0128] in Values ​​range from 1 to The position number of the subsequent segment is Valid reference frames, in which ,when Corresponding to the first frame; define the temporal nearest neighbor component. Motion clarity component Impact Factor for:

[0129] ;

[0130] in Values ​​range from 1 to .

[0131] The position number of the previous segment is Valid reference frame, , Corresponding to the last frame , The corresponding valid reference frame farthest from the last frame Define temporal nearest neighbor components With motion mass component , , Calculate the impact factor .

[0132] For temporal nearest neighbor components exist Take the maximum value at the last frame. ,exist The value of the furthest valid reference frame is the minimum of 1, decreasing linearly with frame sequence distance. The last frame is the closest in time to the splicing boundary, and its visual content is closest to the time of the transition frame, so it should receive the highest temporal weight; the furthest valid reference frame is the furthest in time, and its visual content deviates from the time of the splicing boundary, so it should receive the lowest temporal weight. The linear decreasing form is intuitive in quantifying temporal proximity and requires no additional parameters, making it suitable for real-time deployment on mobile devices.

[0133] For the motion mass component exist When the value is small (smooth motion between adjacent frames, clear image, low motion blur), a larger positive value is taken. near When the motion amplitude between adjacent frames is close to the cutoff threshold and the motion blur is significant, it tends to be 0. Intuitively, reference frames with low motion blur have clearer texture structures and more reliable gradient statistics, and should contribute more to the estimation of local structural feature vectors; reference frames with significant motion blur, even if they are close to the boundary in time sequence, have reduced feature reliability due to blur, and their weights should be reduced accordingly. by As a reference standard, the normalized optical flow amplitude is... Converted into a positive metric for motion sharpness, and With consistent dimensions (both are linear quantities), the combination of two factors is naturally achieved.

[0134] right ,because And each ,have:

[0135] ;

[0136] therefore For all Strictly true. (Regarding...) , ,and , Therefore Possibly greater than .when Very close (i.e., cumulative value in) When the step is very close to the cutoff threshold, The probability is relatively high, especially in extreme cases. ,lead to To strictly guarantee nonnegativity and a positive denominator in this extreme case, the implementation will... The revised definition is:

[0137] ;

[0138] This amendment remains unchanged in the vast majority of cases. The value (because) (This usually holds true), and will only be true in extreme cases. Truncating to 0 is equivalent to setting the impact factor of that frame to zero, excluding it from the weighting. Because... corresponding denominator The result is still positive, so the normalization result is valid.

[0139] In this embodiment, , Substitute each The values ​​for each component and influence factor are calculated, and the detailed values ​​are as follows:

[0140] : , ,product ;

[0141] : , ,product ;

[0142] : , ,product ;

[0143] : , ,product ;

[0144] : , ,product .

[0145] The denominator is .therefore , , , , .

[0146] satisfy (Due to rounding error). The last frame has the largest impact factor (≈0.350), while the farthest effective reference frame has the smallest impact factor (≈0.060), reflecting the design goal that reference frames with close temporal proximity and clear motion contribute more to the generation of transition frames.

[0147] The position number of the subsequent segment is Valid reference frame, , Corresponding to the first frame Define temporal nearest neighbor components Motion mass component Calculate the impact factor .

[0148] In this embodiment, , Substitute each value:

[0149] : , ,product ;

[0150] : , ,product ;

[0151] : , ,product ;

[0152] : , ,product .

[0153] The denominator is .therefore , , , ,satisfy .

[0154] The following detailed description provides an equivalent alternative implementation method for calculating the impact factor:

[0155] In another implementation, the temporal nearest neighbor components can be replaced with an exponentially decaying form:

[0156] ;

[0157] in This is the attenuation coefficient, typically ranging from 0.3 to 0.7. The exponential attenuation makes the temporal weight of the last frame more prominent compared to the farthest frames (in linear attenuation, the weight of the last frame is equal to that of the farthest frame). The multiple increases with exponential decay. (Increases rapidly), suitable for music video scenarios with extremely strict requirements for temporal proximity, where the visual style of transition frames must be highly consistent with that of boundary frames. The temporal weights decay exponentially to constant weights (the temporal weights of each frame are equal). In the extreme case where time degradation results in only including the last frame, the linear decrease in this embodiment can be seen as a natural compromise between the two extremes. The exponentially decaying temporal nearest neighbor component and the linear decrease form are substantially equivalent in function in quantizing the temporal proximity contribution of the reference frame, and both fall within the scope of protection of this invention.

[0158] In another embodiment, the combination of the temporal nearest neighbor component and the motion mass component can be replaced by a weighted sum:

[0159] ;

[0160] in The preset weight parameters, typically ranging from 0.4 to 0.6, are used to obtain the impact factor after normalization. This avoids the strong constraint that the overall impact factor would approach zero when any component approaches zero. or When one side is extremely small but the other side is extremely large (for example, the farthest reference frame has a very low timing weight but happens to be the clearest), more effective information can be retained. This is substantially equivalent in terms of the function of comprehensively quantifying the contribution of the reference frame and falls within the scope of protection of this invention.

[0161] It is important to note that the core function of the impact factor is to assign differentiated weights to each reference frame in the construction of the weighted covariance matrix, so that reference frames closer to the boundary and with clear motion contribute more to the orientation estimation of the principal component subspace. Any weight definition method that can reflect the joint quantification of the two dimensions of "temporal proximity" and "motion clarity (or content reliability)" and meets the normalization conditions is substantially equivalent to the design of the impact factor described in this embodiment and falls within the protection scope of this invention.

[0162] The gradient and chromaticity statistics extracted from each frame in the effective reference frame are concatenated to form a local structural feature vector; the weighted covariance matrix is ​​obtained according to the influence factor and local structural feature vector of each frame in the effective reference frame of the previous and subsequent segments; the weighted covariance matrix of the previous and subsequent segments is decomposed by eigenvalue and the mean of the corresponding eigenvalue is calculated. The principal direction feature vectors of the previous and subsequent segments are selected to form the projection matrix of the previous and subsequent segments respectively.

[0163] Furthermore, the method of forming a local structural feature vector by concatenating the gradient and chromaticity statistics extracted from each frame of the effective reference frame after dividing it into fixed pixel blocks includes:

[0164] Each frame image is fixed Pixel blocks are uniformly divided across the entire frame; gradient directions are calculated for each pixel within each block, using a fixed number of direction intervals. Perform directional histogram statistics and normalization to obtain each block 3D gradient orientation histogram; extract within each block Color space Component mean and The component mean is a two-dimensional chromaticity statistic; the gradient direction histogram of each block in the whole frame is sequentially concatenated with the chromaticity statistic to obtain... A local structural feature vector, where:

[0165] ;

[0166] The position number of the previous segment is The valid reference frame is extracted in the manner described above. The dimensional local structural eigenvector is denoted as The position number of the subsequent segment is The valid reference frame is extracted in the manner described above. The dimensional local structural eigenvector is denoted as .

[0167] For all valid reference frames of the preceding and following segments, extract the local structural feature vectors of each frame in the following manner.

[0168] Each frame image is fixed The pixel blocks are evenly divided across the entire frame; in this embodiment, we take... That is, each piece is Pixels. The entire frame is divided into 100 pixels. A pixel block, in this embodiment, is A pixel block. When the height or width of the frame cannot be... When dividing, pixel regions with edges that are not a complete block can be processed in one of the following equivalent ways: First, ignore the edge margin and only process the complete pixel block; second, perform mirror filling on the edge margin to preserve the structural information of the edge region. The feature vectors obtained by the two processing methods have slightly different dimensions, but they are functionally equivalent in the construction of the covariance matrix.

[0169] Calculate the gradient direction pixel by pixel within each block, specifically based on the block coordinates. Calculate the horizontal gradient at the pixel. with vertical gradient The Sobel operator (horizontal kernel) can be used. (Vertical kernel transpose) or Prewitt operator, pixel gradient direction angle is The gradient magnitude is With a fixed number of intervals in a fixed direction Will Average score In this embodiment, there are directional intervals. The width of each directional interval is For each pixel within the block, its gradient magnitude... As weights, accumulated to their gradient direction angle The direction interval into which the pixel falls is used to obtain an amplitude-weighted gradient orientation histogram. Then, the values ​​of each interval in this histogram are normalized by dividing them by the sum of the gradient amplitudes of all pixels in the block, yielding the result for each block. Dimensionally normalized gradient direction histogram vector.

[0170] In this embodiment, an amplitude-weighted histogram is used instead of a simple counting histogram because pixels with large gradient amplitudes (corresponding to edge regions with significant texture in the image) should contribute more to the structural features of the block, while smooth pixels with gradient amplitudes close to zero (such as the sky and walls) contribute almost nothing to the structural features. The amplitude-weighted scheme makes the feature vector more concentrated on the edge information that is truly structurally significant within the block. Those skilled in the art should understand that a simple counting gradient direction histogram (which counts the gradient directions of each pixel equally) and the amplitude-weighted scheme of this embodiment are functionally equivalent in statistical description of the block-level gradient direction distribution, and both fall within the scope of protection of this invention.

[0171] Extract within each block Color space Component mean and Component mean, a two-dimensional chromaticity statistic. The input frame is from... Color space conversion The conversion relationship of color spaces is as follows:

[0172] ;

[0173] ;

[0174] ;

[0175] Select Color space rather than direct use The reason for the mean is: and The component carries hue information and is related to the luminance component. Decoupling makes the chromaticity statistics naturally robust to changes in lighting intensity; the two chromaticity components, totaling only 2 dimensions, can fully describe the hue attribute of color, resulting in dimensionality economy and effective control over the total dimensionality of the feature vector. Growth.

[0176] The gradient orientation histogram of all 8040 pixel blocks in the entire frame (for each block) The 2D chromaticity and chromaticity statistics (each block is 2D) are sequentially assembled according to the spatial scanning order of the blocks (from left to right, from top to bottom) to obtain... 3D local structural feature vector;

[0177] In this embodiment .

[0178] The position number of the previous segment is ( The valid reference frame is extracted in the manner described above. The dimensional local structural feature vector is The position number of the following segment is recorded as follows: ( The valid reference frame is extracted in the manner described above. The dimensional local structural feature vector is .

[0179] For pixel block size and direction interval number The basis for parameter selection should be fully explained to meet the requirement of full disclosure. The trade-off between spatial resolution and feature dimension: Smaller (e.g.) (When there are more blocks, the feature vector dimension is higher) Increased by approximately 4 times ( While offering finer spatial resolution, the memory usage and computational complexity of the covariance matrix increase dramatically. Larger (e.g.) )hour While reducing computational load, insufficient spatial resolution may lead to the loss of detailed structural information. Under the constraints of 1080p resolution and real-time processing on mobile devices... To achieve the optimal balance between spatial resolution and computational feasibility. All integer values ​​within the range [8, 32] can stably form effective local structural feature vectors, and corresponding adjustments are made. All parameters related to feature extraction fall within the scope of protection of this invention. Quantization precision for controlling the gradient direction: (45° accuracy) strikes a balance between directional resolution and histogram statistical stability. All integer values ​​within the range [6, 16] can stably reflect the gradient direction distribution within the block, and fall within the scope of protection of this invention.

[0180] The following detailed description provides an equivalent alternative implementation for local structural feature vector extraction. In another implementation, the gradient orientation histogram can be replaced with an oriented gradient histogram (HOG) feature, which is normalized between adjacent blocks (calculated in 2×2 block units). (Norm normalization) provides stronger robustness to inter-block illumination inhomogeneity; HOG features have the same dimension for each block. The dimension, which is substantially equivalent to the gradient direction histogram in this embodiment in describing the gradient direction distribution within the block, falls within the scope of protection of this invention.

[0181] In another implementation, the local structural feature vector can be replaced with Local Binary Pattern (LBP) texture features: LBP encoding is calculated for each pixel within each block (in... sampling points, radius Using a circular neighborhood as the standard, typically ), Histogram of LBP encoding within the statistical block (total) The 59-dimensional uniform LBP (or simplified by equivalence class) is concatenated with the chromaticity statistics to form a block-level feature vector, and the entire frame is concatenated to obtain the structural feature vector. The LBP feature is completely invariant to monotonic illumination changes and has obvious advantages in cross-segment scenes with significant differences in illumination conditions. It is substantially equivalent to the gradient direction histogram in this embodiment in representing the texture structure within the block and falls within the protection scope of this invention.

[0182] In another implementation, the local structural feature vectors can be replaced with discrete cosine transform (DCT) coefficient features: transforming each block into... After the space, to Component execution Two-dimensional DCT, taking the top left corner (typical The low-frequency DCT coefficients of up to 6) are flattened as follows: dimensional vector, and , Mean stitching; DCT coefficients are standard structural descriptors in the field of image compression, with extremely high computational efficiency (which can be accelerated by hardware). They are substantially equivalent to the gradient direction histogram in this embodiment in extracting block-level visual structural information, and are especially suitable for deployment scenarios where mobile encoding and decoding hardware has built-in DCT operation units, which fall within the protection scope of this invention.

[0183] In another embodiment, the local structural feature vector can be replaced by the depth features of a pre-trained visual encoder: the global or regional depth feature vectors of each reference frame are extracted using a pre-trained network such as CLIP visual encoder (ViT-B / 32 or ViT-L / 14), DINO (ViT-S / 8), or MobileNetV3, and the depth features of each region (corresponding to the pixel blocks in this embodiment) are spliced ​​in spatial order to form a depth local structural feature vector; the depth features have a much stronger ability to represent semantic structures (such as human outlines and object shapes) than manual features, and can further improve the semantic consistency of transition frames in materials with complex scene content; this alternative feature extraction method is substantially equivalent to the gradient direction histogram plus chromaticity statistics in this embodiment in terms of extracting local visual structural information of the frame and providing feature input for the subsequent covariance matrix construction, and is within the scope of protection of this invention.

[0184] In another embodiment, the color space of the chromaticity statistics can be replaced with... Color space Component (hue) mean and Component (saturation) mean, or Color space Component mean and Component mean; the above alternative color space and In terms of carrying hue information and decoupling from the luminance component, they are substantially equivalent and both fall within the scope of protection of this invention. In another embodiment, the order of the chromaticity statistics can be expanded from the mean (first moment) to the mean and variance (first and second moments), and each block of chromaticity statistics is expanded from 2-dimensional to 4-dimensional. The size increases accordingly, but the logical structure of the extraction process remains completely the same, and it falls within the scope of protection of this invention.

[0185] Further, the method of obtaining the corresponding weighted covariance matrix based on the influence factors and local structural feature vectors of each frame in the effective reference frames of the preceding and following segments; decomposing the weighted covariance matrices of the preceding and following segments into eigenvalues ​​and calculating their corresponding eigenvalue mean; and selecting the principal direction feature vectors of the preceding and following segments to form the projection matrices of the preceding and following segments respectively includes:

[0186] Calculate the weighted mean vector of the preceding segments Construct the weighted covariance matrix of the first segment ;right Perform eigenvalue decomposition on a real symmetric matrix to obtain eigenvalues ​​in descending order. ; Statistical satisfaction The number of subscripts is denoted as , ,in Values ​​range from 1 to Take the front The eigenvectors are arranged in columns to form the projection matrix of the first segment. , for OK Column matrix;

[0187] Calculate the weighted mean vector of the segments Construct the weighted covariance matrix of the subsequent segments. ;right Perform eigenvalue decomposition on a real symmetric matrix to obtain eigenvalues ​​in descending order. ;

[0188] Statistical satisfaction The number of subscripts is denoted as , ,in Values ​​range from 1 to Take the front The eigenvectors are arranged in columns to form the projection matrix of the post-segment. , for OK Column matrix.

[0189] Using the influence factors of each effective reference frame in the previous segment With local structural feature vectors Construct the weighted covariance matrix of the first segment, and then form the projection matrix of the first segment through eigenvalue decomposition. Calculate the weighted mean vector of the preceding segments. ; The weighted mean of the local structural feature vectors of the effective reference frames of the preceding segment represents the weighted visual structure center of the preceding segment in the feature space. The feature pairs of the last frame with higher weights are... The contribution of the far-end low-weight reference frame is the largest, while the contribution of the far-end low-weight reference frame is suppressed, making Typical visual state of frames closer to the boundary.

[0190] Construct the weighted covariance matrix of the first segment , It is a real symmetric positive semi-definite matrix that characterizes the weighted scattering structure of the effective reference frames of the previous segment in the feature space. The eigenvalues ​​reflect the weighted variance of features in each principal direction. Principal directions with larger eigenvalues ​​correspond to structural patterns with significant variance in the previous segment's reference frame set, representing the core direction of change in the visual style of that segment. Directions with eigenvalues ​​close to zero correspond to dimensions with noise or statistically insignificant changes.

[0191] right Perform eigenvalue decomposition on a real symmetric matrix to obtain eigenvalues ​​arranged in descending order. and corresponding feature vectors Calculate the mean of the eigenvalues. Statistical satisfaction The number of subscripts, denoted as And guarantee Take the front. eigenvectors Arranged column-wise, forming the projection matrix of the first segment. .

[0192] Screening criteria This is equivalent to the Kaiser criterion: retaining principal directions whose variance contribution exceeds the average level, and filtering out noise dimensions that are below the average level. This criterion does not require a pre-defined fixed target dimensionality for dimensionality reduction; it is adaptively determined by the data. This ensures that constraints only apply to structural directions with statistically significant variances within the reference frame set, thus avoiding the contamination of transition frame generation constraints by noise dimensions.

[0193] In this embodiment, due to , The rank is at most (The actual rank is 5, because the probability that the five reference frames are linearly independent in the 80400-dimensional feature space is extremely high), so there are at most 5 non-zero eigenvalues. ,the remaining Eigenvalue mean (only Sum of non-zero terms divided by The value is extremely small, therefore it satisfies The number of subscripts represents the number of non-zero eigenvalues ​​that exceeds the minimum mean. In practice, this is usually... =3 to 5. The first splicing boundary in this embodiment, calculated as follows: =4, the projection matrix of the previous segment It is an 80,400-row, 4-column matrix that realizes the projection from the 80,400-dimensional feature space to the 4-dimensional principal component subspace, with a dimensionality reduction ratio of over 20,000 times, which greatly reduces the computational cost of subsequent optimization problems.

[0194] right Explanation: Due to , At least rank Matrix (when) hour ,but Therefore (This situation corresponds to the handling of degeneracy cases, see subsequent paragraphs for details.) hour Typically, there is at least one significant non-zero eigenvalue. This is naturally true. When, due to numerical calculation reasons, all eigenvalues ​​are less than... At that time, forced removal The projection matrix is ​​constructed using the eigenvectors corresponding to the largest eigenvalue, ensuring that the algorithm can run normally under any input.

[0195] Calculate the weighted mean vector of the segments Construct the weighted covariance matrix of the subsequent segments. ,right Perform eigenvalue decomposition on a real symmetric matrix to obtain eigenvalues ​​in descending order. Calculate the mean of the eigenvalues. Statistical satisfaction The number of subscripts, denoted as And guarantee Take the front. The eigenvectors are arranged in columns to form the projection matrix of the post-segment. In this embodiment , It is an 80400-row, 3-column matrix.

[0196] because The rank is at most (much smaller) ), no need for The matrix can be directly subjected to complete eigenvalue decomposition, while the rank can be utilized. Matrix structure: The frame feature vectors (after centering) form a matrix , ,in ; through the matrix Perform eigenvalue decomposition (computational cost: ,exist (extremely fast), then map the feature vector back 3D space, complete derivation The non-zero eigenvalues ​​and eigenvectors of have a computational complexity of O(n). It is far superior to direct decomposition. ,exist material, Under these conditions, the time taken for a single decomposition is less than 50ms, meeting the real-time processing requirements of mobile devices. Those skilled in the art should understand that the above equivalent decomposition method is different from direct decomposition. The eigenvalue decomposition of a matrix is ​​mathematically equivalent and both fall within the scope of protection of this invention.

[0197] In another implementation, effective feature dimension The determination of the cumulative variance explained ratio can be changed to the cumulative variance explained ratio criterion: retain the criterion that makes the cumulative variance explained ratio... First time exceeding the preset ratio (Typically, values ​​range from 0.85 to 0.95) the minimum number of eigenvalues ​​required; this criterion is applicable when the number of reference frames is large (e.g., ...). The above (as opposed to the Kaiser criterion) can more precisely control the information retention ratio and is substantially equivalent to the Kaiser criterion in retaining the main direction and filtering noise dimensions, and thus falls within the scope of protection of this invention.

[0198] In another implementation, a fixed upper limit can be set for the effective feature dimension. (Typical values ​​are 10 to 20), with This is the dimension to be used in the final application, to prevent issues when there are many reference frames. Excessive size leads to uncontrolled optimization problems; the combination of engineering constraints and adaptive screening criteria achieves a balance between accuracy and computational efficiency, which falls within the scope of this invention.

[0199] In another implementation, instead of performing eigenvalue decomposition independently on the preceding and following segments, joint principal component analysis can be used: the effective reference frame eigenvectors of the preceding and following segments are merged, a joint weighted covariance matrix is ​​constructed using the influence factors of each frame as weights, the shared principal direction subspace is extracted, and a unified projection matrix is ​​obtained. This method allows for comparison of preceding and following segments within the same feature subspace, facilitating direct measurement of the centroid distance between the two sides during transition frame generation. It is suitable for scenarios where preceding and following segments have similar visual styles (such as when the same photographer takes photos at similar times). This method is substantially equivalent to the function of independently extracting the bilateral projection matrices in this embodiment to constrain the structural features of the transition frame, and thus falls within the scope of protection of this invention.

[0200] Furthermore, the normalized average optical flow amplitude between the first pair of adjacent frames inward from the last frame of the current segment. satisfy At that time, the number of valid reference frames in the previous segment will be... Set to 1, extracted from the last frame of the previous segment. The dimensional local structural eigenvector is denoted as ,by The normalized column vector obtained by dividing by its Euclidean norm is directly used as the projection matrix of the preceding segment. Effective feature dimensions of the preceding segment Set to 1, the influence factor of the last frame of the previous segment. Set to 1;

[0201] The normalized average optical flow amplitude between the first pair of adjacent frames in the subsequent segment satisfy At that time, the number of valid reference frames for the subsequent segment will be determined. Set to 1, and the first frame extracted from subsequent segments The dimensional local structural eigenvector is denoted as ,by The normalized column vector obtained by dividing by its Euclidean norm is directly used as the projection matrix of the subsequent segment. Effective feature dimensions of the later segment Set to 1, then the impact factor of the first frame of the subsequent segment. Set to 1.

[0202] Normalized average optical flow amplitude between the first pair of adjacent frames from the last frame of the current segment satisfy This means that the motion amplitude between the last frame of the preceding segment and its immediately preceding frame has exceeded the cumulative threshold in a single step. This situation corresponds to extremely high-speed motion or shaky shooting scenes (such as dance timing or rapid camera movements), where there is already significant motion blur between the last frame and its immediately preceding frame. Continuing to trace back to the previous reference frame will only result in a larger motion amplitude, and the normal estimation process of the covariance matrix cannot provide a meaningful main direction. The following degradation processing is performed:

[0203] Number of valid reference frames in the previous segment Set to 1, the last frame of the previous segment Extracted in the manner described above The dimensional local structural eigenvector is denoted as ,by Normalized column vector obtained by dividing by its Euclidean norm ; directly used as the projection matrix of the preceding segment ,at this time For a single-column matrix, the effective feature dimension of the first segment is... Set to 1, the influence factor of the last frame of the previous segment. Set to 1.

[0204] The normalized average optical flow amplitude between the first pair of adjacent frames in the subsequent segment satisfy At that time, perform symmetrical degradation processing: reduce the number of effective reference frames for the later segment. Set to 1, and the first frame of subsequent clips Extracted 3D local structural feature vector After normalization ; directly used as the projection matrix of the later segment Effective feature dimensions of the later segment Set to 1, then the impact factor of the first frame of the subsequent segment. Set to 1.

[0205] by As a single-column projection matrix, its effect is equivalent to constraining the projection of the transition frame in the same direction as the last frame's eigenvector, using the direction of the last frame's eigenvector as the sole principal direction. Normalization ensures that the Euclidean norm of the projection matrix column vectors is 1 (i.e., an orthonormal basis), consistent with the eigenvectors obtained from the normal process eigenvalue decomposition (which naturally satisfy the unit norm). This allows the subsequent optimization steps for generating transition frames to directly use the output of the degenerate case without special processing.

[0206] In extreme cases, if both the preceding and following segments trigger the degradation condition, then , The generation constraint of the transition frame degenerates into a two-sided constraint on the feature vector directions of the last frame of the preceding segment and the first frame of the following segment. Subsequent optimization steps can still be executed normally. The generated transition frame visually appears as a weighted mixture of the main structural feature directions of the two frames, which is still significantly better than direct hard cutting. At the first splicing boundary in this embodiment, , Neither side triggered the degradation condition, so it was processed according to the normal procedure.

[0207] In another embodiment, the degradation threshold can be set independently of... Dedicated degradation threshold (Typical value) to This decouples the degradation judgment criteria from the reference frame cutoff criteria, facilitating independent optimization for different material types; this alternative threshold setting method, along with direct reuse... The functions are equivalent and fall within the scope of protection of this invention.

[0208] In another embodiment, when At that time, it is not necessary to Forced to , and placed as (in The values ​​obtained from the normal process are incorporated into two frames (the last frame and its immediate neighbor) for reference. Normal covariance matrix construction is performed using the normalized influence factors for each frame (2 / 3 weight for the last frame, 1 / 3 weight for the immediate neighbor). This alternative degradation processing provides slightly richer structural information than a single frame in extremely high-speed motion scenes. It may be upgraded to level 2, with more stable transition frame constraints, compared to forced... The single-frame degradation processing is functionally equivalent in ensuring the robustness of the algorithm and falls within the scope of protection of this invention.

[0209] The ratios of the number of effective reference frames in the previous segment and the number of effective reference frames in the subsequent segment to the sum of the number of effective reference frames in the previous and subsequent segments are used as the balance coefficients for the previous and subsequent segments, respectively. The balance coefficients for the previous and subsequent segments are used as the weighted residuals of the local structural feature vectors of the transition frame under the projection matrices of the previous and subsequent segments, respectively. The pixel values ​​corresponding to the local structural feature vectors that minimize the weighted residuals are recorded as constraints. The constraints are injected into the AI ​​frame interpolation network that generates the transition frames. The AI ​​frame interpolation network is a pre-trained neural network model that receives the constraints and generates transition frames accordingly.

[0210] like Figure 3The diagram illustrates that the video sequence to be edited and synthesized in this embodiment includes two independent video segments: a preceding segment and a following segment. These two segments were filmed by different photographers in different scenes and at different times, and the splicing position forms a splicing boundary, which is determined to be a cross-segment optical flow failure boundary. The transition frame insertion position is located between the last frame of the preceding segment and the first frame of the following segment. In the effective reference frame selection stage, along the tracing direction of the preceding segment (tracing back frame by frame from the last frame of the preceding segment to the inside of the segment) and the tracing direction of the following segment (tracing back frame by frame from the first frame of the following segment to the inside of the segment), the normalized optical flow amplitude of adjacent frames within the segment is calculated and accumulated pair by pair. The effective reference frame range of the preceding and following segments is adaptively determined when the accumulated value first reaches a preset accumulated motion threshold. The last frame of the preceding segment is the starting frame of the effective reference frames for the preceding segment, and the first frame of the following segment is the starting frame of the effective reference frames for the following segment. By using the backtracking rules and frame segmentation shown in Figure 3, reference frames with reasonable motion amplitude and stable structural features are selected, avoiding the inclusion of far-end frames with motion blur or scene offset. Simultaneously, a sufficient number of reference frames are ensured, providing a foundation for subsequent extraction of the bilateral segment projection matrix and generation of transition frames that fit the visual features of the preceding and following segments. This fundamentally solves the problems of time flow failure, artifacts, and visual jumps in transition frames during hard cuts across segments. The pixel values ​​corresponding to the local structural feature vectors that minimize the weighted residuals are injected into the AI ​​frame interpolation network that generates the transition frames. This addresses the uncontrollability of individual parameters in traditional AI frame interpolation networks, which leads to randomness in the overall interpolated image.

[0211] Further, the ratios of the number of effective reference frames in the previous segment and the number of effective reference frames in the subsequent segment to the sum of the number of effective reference frames in the previous and subsequent segments are used as the balance coefficients for the previous and subsequent segments, respectively. These balance coefficients are used as the weighted residuals of the local structural feature vectors of the transition frame, minimizing the weighted residuals under the projection matrices of the previous and subsequent segments. The pixel values ​​corresponding to the local structural feature vectors that minimize the weighted residuals are recorded as constraints. The method of injecting these constraints into the AI ​​frame interpolation network that generates the transition frames includes:

[0212] Calculate the pre-fragment balance coefficient Balance coefficient with subsequent segments ;

[0213] Calculate the pre-fragment side Vegan weight center of mass ,in Projection matrix of the previous segment The transpose matrix; calculate the fragment side. Vegan weight center of mass ,in Projection matrix of the later segment The transpose of the matrix;

[0214] With transition frames 3D local structural feature vector Using the independent variable, minimize the objective function. :

[0215] ;

[0216] Calculate the minimum objective function When the minimum value is obtained, the corresponding A local structural feature vector, denoted as . ; the preceding pixel block The first two components serve as constraints on the gradient direction intensity distribution of each pixel block, while the last two components serve as constraints on the gradient direction intensity distribution of each pixel block. Component mean and The component mean constraint is used to reconstruct the pixel values ​​of each pixel block according to the above constraints, and these constraints are then injected into the AI ​​frame interpolation network that generates the transition frame.

[0217] This step uses the projection matrix of the preceding fragment output from the previous steps. Post-fragment projection matrix Weighted mean vector of the preceding segments The weighted mean vector of the later segments Impact factors of each effective reference frame in the preceding segment Impact factors of each effective reference frame in the subsequent segment Number of valid reference frames in the previous segment Number of valid reference frames for subsequent segments As input, the local structural feature constraint vector of the transition frame is obtained by minimizing the weighted residuals. Then The structural constraints are provided to the AI ​​frame interpolation network, which then uses them to... This step serves as a reference for generating the final transition frame pixel output. It is a crucial step in connecting the results of the aforementioned purely mathematical structural analysis with the AI ​​image generation capabilities, ensuring that the AI ​​frame interpolation network outputs visually natural transition frames consistent with the style of the preceding and following segments under clear structural constraints, rather than generating them freely without constraints.

[0218] First, calculate the balance coefficient of the preceding segment. Balance coefficient with subsequent segments ,satisfy and In this embodiment , Therefore , The structural constraint weights of the earlier segments are slightly greater than those of the later segments, which is consistent with the fact that the earlier segments have more effective reference frames and more sufficient structural feature estimation statistics.

[0219] The design basis for using the ratio of effective reference frames as the balancing coefficient is as follows: The larger the value, the higher the cumulative motion threshold of the preceding segment. The ability to incorporate more reference frames indicates that the motion of the preceding segment is relatively smooth, and the covariance matrix more accurately portrays the real visual style. Accordingly, it should be given higher weights to guide the AI ​​frame interpolation network. The smaller the value, the more intense the motion in the preceding segment and the fewer the reference frames, resulting in lower statistical reliability of the structure estimation, and the weights should be relatively reduced. This design naturally transforms the information contained in the number of reference frames into the relative strength of the bilateral constraints, without the need to introduce additional hyperparameters.

[0220] Calculate the pre-fragment side Vegan weight center of mass ,in Projection matrix of the previous segment The transpose of the matrix, The weighted center of the set of valid reference frames for the preceding segment in the principal component subspace of the preceding segment represents the coordinates of the "typical visual structure state" of the preceding segment in the low-dimensional constrained subspace. The following calculations are performed on the following segments: Vegan weight center of mass ,in Projection matrix of the later segment The transpose of the matrix, It serves as the weighting center of the set of valid reference frames for the subsequent segment in the principal component subspace of the subsequent segment. and Together, they constitute the two-sided structural constraint objectives of the AI ​​frame interpolation network, respectively characterizing the "visual gravity center" of the preceding and following segments in the low-dimensional structural subspace. The optimal transition frame feature vector obtained in the subsequent solution must achieve a weighted balance between the two gravity centers.

[0221] With transition frames 3D local structural feature vector Using the independent variable, minimize the objective function. , The first requirement is the transition frame feature vector. The projected coordinates in the principal component subspace of the previous segment, and the weighted centroid of the previous segment reference frame set. The first requirement is that the transition frame should be as close as possible to the visual style of the preceding segment, meaning that the transition frame's core structural changes should align with the visual style of the preceding segment; the second requirement is that the projection of the transition frame into the principal component subspace of the following segment should be as close as possible to the visual style of the preceding segment. near; and Adjust the relative strength of the bilateral constraints.

[0222] Regarding the objective function Find the gradient and set it to zero, then use... and By determining the orthogonality of the columns, we obtain the analytical solution for the local structural eigenvectors of the optimal transition frame:

[0223] ;

[0224] The computational complexity of the analytical solution is In this embodiment, approximately On mobile devices, this can be completed within 10ms, meeting real-time processing requirements. about The gradient is obtained as follows:

[0225] ;

[0226] because and All column vectors are orthogonal unit vectors, and and When the column vector spaces are independent, the first and second terms can be solved separately to obtain the analytical solution described above. When the column spaces of the projection matrices of the preceding and following segments overlap, the above analytical expression is an approximate optimal solution for the objective function when the constraints on both sides are independent. In practice, the overlap between the principal component subspaces on both sides is usually extremely low, and the approximation error can be ignored. It minimizes the weighted projection residuals on both sides simultaneously. The local structural feature vector, whose components simultaneously encode gradient direction distribution constraints and chromaticity mean constraints, expresses the visual structural characteristics that the transition frame should possess in a compressed form, and can be directly used as the structural guidance condition for AI frame interpolation networks.

[0227] This completes the purely mathematical solution portion of this step, and the output is now available. Next, The inherent local structural constraints are provided to the AI ​​frame interpolation network, which then completes the pixel-level transition frame synthesis.

[0228] right Each component is analyzed to provide usable constraint parameters for the AI ​​frame interpolation network. of The dimensional components are arranged in the spatial order during feature extraction, corresponding to the entire frame. Each pixel block corresponds to Dimensional components: front The dimension is the target value vector of the gradient direction histogram of this block. The first dimension represents the gradient direction distribution that the transition frame should have within the block; the last two dimensions are the gradient direction distribution of the block, respectively. Component mean target and Component mean target This characterizes the hue that a transition frame should have within that block. Each component is truncated and then renormalized. and Perform legal value range truncation (standardization) middle and The value range is [16, 240]), to ensure the physical validity of the constraint parameters. After the above processing, Transformed into a set of block-by-block structural constraint parameters , which serves as the conditional input for the AI ​​frame interpolation network.

[0229] AI frame interpolation network receiver structure constraint parameter set and the last frame of the previous segment The first frame of the following clip Generate what satisfies The transition frame pixel image is subject to the specified local structural constraints. The specific implementation of the AI ​​frame interpolation network is not limited; those skilled in the art can select an appropriate implementation method based on the computing resources, real-time requirements, and quality objectives of the deployment environment.

[0230] For example, the fundamental difference between the AI ​​frame interpolation network described in this invention and conventional video frame interpolation networks in terms of training method lies in the fact that the set of structural constraint parameters must be included during training. The constraint injection mechanism is used as an explicit conditional input to participate in network optimization, rather than simply generating data from two consecutive frames without constraints. The following section, using the cGAN architecture as an example, elaborates on the construction of the training dataset, the constraint injection mechanism, the loss function design, and the training process, which can be directly implemented by those skilled in the art.

[0231] (a) Construction of the training dataset: The training samples required by this invention are quadruples. ,in For the last frame of the previous segment, The first frame of the following segment For the set of structural constraint parameters, This is the ground truth pixel image corresponding to the transition frame.

[0232] Collect continuously shot video clips from a large-scale short video dataset. Manually set virtual splicing points at any time point in each video clip, and use the frame at that splicing point as the ground truth of the transition frame. Using the frame preceding this frame as... The next frame as ; then with For the last frame of the previous segment, For the first frame of the subsequent segment, all the aforementioned steps of this invention are executed—namely, cross-segment optical flow failure boundary determination (here, because the two frames are from consecutive shooting, the determination result is not a failure boundary, but for the purpose of constructing constraint parameters for training, the process of determining the effective reference frame is forced), and the number of effective reference frames. and Determine and normalize the optical flow amplitude series records, and the influence factor for each frame. and Calculation of local structural feature vectors and Extraction and weighted covariance matrix and Construction and eigenvalue decomposition, projection matrix and Formation and equilibrium coefficients and Calculation, weighted centroid and Calculation, objective function Minimize the solution to obtain — Ultimately of dimensional vectors by Spatial arrangement reshaped into a set of constraint parameters Complete the construction of the sample. The training set should cover typical handheld shooting styles such as slow motion, fast motion, multi-person scenes, and indoor and outdoor lighting changes. It is recommended to have no less than 50,000 sets of quadruplet samples to ensure the network's ability to generalize to different motion amplitudes and scene types.

[0233] (ii) Constraint injection mechanism in the generator: The generator adopts an encoder-decoder structure (such as U-Net), and the encoder uses... and The channel stitching image (6 channels in total) is a pixel-level input. The encoder extracts multi-scale feature maps at each layer, and the decoder upsamples them step by step to restore them to the output resolution.

[0234] Constraint parameter set Spatially possesses resolution, One channel, of which the front Each channel corresponds to the target value of the gradient direction histogram of each pixel block. The latter two channels correspond to the target Cb component of each pixel block. Target of mean value of Cr component . After bilinear upsampling to the same spatial resolution as the feature maps of each decoder layer, the feature maps are injected into each layer of the decoder via adaptive instance normalization (AdaIN): Let the number of channels in the feature map of a certain decoder layer be... Upsampled The feature map is passed through a 1×1 convolutional layer (number of input channels) Number of output channels Regression yields spatially adaptive scaling parameters. With offset parameter For the feature map of this layer Perform the AdaIN transform: ,in , for Mean and standard deviation in spatial dimension To prevent zero positive values, this operation causes the constraints to act continuously on the feature distribution of each layer of the decoder in the form of an affine transformation, rather than just being used as additional channels and image pixel concatenation inputs. This allows for fine control over the local gradient and chromaticity statistical characteristics of the generated frame.

[0235] (III) Design of training loss function, total training loss It consists of the following four items:

[0236] Pixel reconstruction loss : Generate frames With the truth value of the transition frame Between Pixel distance:

[0237] ;

[0238] This drives the network to converge to the ground truth at the pixel level. Norm comparison Norms are more robust to outlier pixel values ​​and are less prone to oversmoothing.

[0239] Perceived loss The metric space is the feature maps of the 3rd and 4th pooling layers of a pre-trained VGG-16 network (pre-trained on ImageNet with fixed parameters and not used in training).

[0240] ;

[0241] in Indicates VGG-16 The feature extraction function of the pooling layer outputs a fixed-dimensional feature map; this loss function ensures that the generated frame maintains consistency with the ground truth in terms of texture structure and semantic features, compensating for... The VGG-16 model suffers from insufficient perception of high-frequency details. As a well-known pre-trained model, the source and usage of its parameters are common knowledge in the field and will not be elaborated upon here.

[0242] Structural constraint satisfaction loss This is a loss term unique to the training process of this invention, used to explicitly quantize the set of structural constraint parameters for generating frame pairs. The degree of satisfaction. For the generated frame. Following the exact same B×B pixel block division and feature extraction process as in the aforementioned main technical solution, the actual gradient direction histogram of each block is extracted. Mean of Cb components Average value of Cr component ,and The deviation of the constraint target value for each corresponding block is calculated block by block:

[0243] ;

[0244] in This represents the total number of pixel blocks in the entire frame. , , for Corresponding pixel block The constraint target value, This is the weight for the chromaticity term (typically 0.5), used to balance the gradient direction histogram. The dimensional difference between the (2D) and chromaticity statistics. This loss term will determine whether the generated frame conforms to the local structural statistics. The "rule" is encoded as a differentiable objective, driving the network to learn precise block-by-block structural control under the guidance of AdaIN conditions; its gradient can be propagated to the generator parameters through a chain reaction of the derivatives of the pixel values ​​of the generated frames with respect to the histogram statistics and the mean of the chroma.

[0245] Combating losses Discriminator To generate frames or truth frames, and a set of constraint parameters. The encoded features are used as joint inputs, and the output is a truth score; the generator aims to deceive the discriminator.

[0246] ;

[0247] The discriminator simultaneously As a condition, it ensures that the discrimination criterion is linked to the degree of constraint satisfaction, rather than just judging the overall realism of the generated image; adversarial loss drives the high-frequency detail distribution of the generated frame to approximate that of the realistic transition frame, compensating for... and Insufficient ability to generate detailed textures.

[0248] The total training loss is a weighted sum of four terms:

[0249] ;

[0250] Typical weight values ​​are , , , ;in weight The value is set relatively high to reflect the core requirement of this invention regarding the satisfaction of constraints on the generated frame structure. In the early stages of training (the first 10 epochs), the value can be set relatively high. Set it to 0, and gradually introduce adversarial loss after the pixel reconstruction quality stabilizes, in order to avoid the discriminator gradient signal from failing due to low generation quality in the early stage of training.

[0251] (iv) Training process

[0252] Both the generator and discriminator use the Adam optimizer, with an initial learning rate of [value missing]. Decay to cosine annealing strategy The generator and discriminator are updated alternately; the generator parameters are updated after each discriminator parameter update. The batch size is 4, and the input image is randomly cropped to a resolution of 256×256 during training. Each frame is divided into 16×16=256 pixel blocks. The space dimensions are The total training iterations are 150 epochs. The network uses a fully convolutional structure. During training, a 256×256 resolution is used to reduce GPU memory usage. During inference, it can run directly at the original resolution (e.g., 1080×1920). The space size automatically becomes (In this embodiment, the resolution is 67×120×10). The network parameters do not need to be modified, and the fully convolutional structure ensures that the training and inference resolutions can be different.

[0253] After training, the structural constraint satisfaction index (i.e., ...) is used on the validation set. The validation set mean), PSNR, and SSIM were used to jointly evaluate the generation quality, in order to The validation set mean is lower than a preset threshold (typically 0.05, corresponding to the average deviation between the gradient histogram of each block and the mean chromaticity value being less than 5% of the constraint target value) as the pass standard, ensuring that the network has the ability to accurately control the local structural statistical characteristics of the generated frame under the guidance of constraints.

[0254] There are several other ways to use constraints to enable AI to survive transition frames, as follows:

[0255] The first AI generation method is a pixel generation method based on a conditional diffusion model. It uses a lightweight conditional denoising diffusion probability model (DDPM or its variant DDIM, with the backbone network being a lightweight U-Net with fewer than 10M parameters) as the AI ​​frame interpolation network. The gradient constraints and chroma constraints of each block are encoded into condition vectors, and the previous segment and the last frame are also encoded. The first frame of the following clip As a reference image input, it is used in the denoising process of the diffusion model. To guide the direction of denoising at each step, ensuring that the generated image satisfies the requirements of gradient distribution and tone in each block. The model has defined structural constraints and can be pre-trained on a large-scale video frame triplet dataset (previous segment reference frame, subsequent segment reference frame, and ground truth transition frame). To assist in joint training by combining supervised loss with perceptual loss (such as VGG perceptual loss) and pixel-level reconstruction loss, DDIM is used to accelerate sampling during inference (typically 10 to 20 steps), enabling single-frame generation within 100ms on mobile GPUs; in this approach... The function of this is to provide a conditional guiding signal. The constrained diffusion model is structurally biased towards the visual style of the preceding and following segments. Compared with the first direct constraint optimization method, the generated pixels have higher detail quality and stronger visual naturalness, which falls within the scope of protection of this invention.

[0256] The second AI generation method is based on Conditional Generative Adversarial Networks (cGANs), using Pix2Pix or its lightweight variants as the underlying architecture. and The channel mosaic image is used as the pixel-level input to the generator. After encoding, the data is used as a global conditional vector and injected into each layer of the generator through adaptive instance normalization (AdaIN). The generator outputs the pixel image of the transition frame; the discriminator takes either the generated image or the real transition frame as input, and... To determine authenticity based on conditions; during training, adversarial loss, perceptual loss, and structural constraint satisfaction loss (i.e., the actual gradient direction histogram of the generated frame and...) are used. The difference Norm, actual chromaticity mean and , The sum of squares of the differences are used as additional loss terms for joint optimization; during inference, the generator can output high-resolution transition frames in a single forward propagation with a latency of less than 50ms; in this method pass Applying local structural constraints to the generator, so that the free generation capability of cGAN is explicitly guided by the visual style analysis results of preceding and following segments, falls within the scope of protection of this invention.

[0257] The third AI generation method is a guided generation approach based on large-scale pre-trained image generation models. It uses pre-trained latent diffusion models such as StableDiffusion as a foundation, and employs adapter structures such as ControlNet or T2I-Adapter to... The structural constraints of the encoding are injected into the intermediate layers of the pre-trained model, while the last frame of the previous segment... The first frame of the following clip of Image embedding, as a cross-attention condition, guides the pre-trained model to maintain consistent tone, texture, and edge structure with the visual style of the preceding and following segments when generating transition frames; The role of this method is to transform the mathematical structure analysis results into structural guidance signals acceptable to the pre-trained model, making up for the lack of control over local spatial structure by pure prompt word guidance; this method utilizes the generation capability of large-scale pre-trained models, and the visual quality of transition frames is the highest in high-resolution scenes, making it suitable for non-real-time high-quality video post-production scenarios; during inference, a 4 to 8-step accelerated sampling strategy (such as LCM sampling) can be used, and near real-time generation can be achieved on high-performance mobile chips, which falls within the scope of protection of this invention.

[0258] In another implementation, the balance coefficient can be defined as the ratio of the square roots of the number of effective reference frames in the preceding and following segments: , .

[0259] This alternative definition makes the balance coefficients pair and The response to quantitative differences is more gradual than that of linear ratios. and When the difference is significant (e.g.) , To avoid one side's constraint weights overwhelmingly dominating the generated results, the AI ​​frame interpolation network can still retain a moderate structural influence from the other side. This is substantially equivalent to the function of the linear ratio in adjusting the constraint weights on both sides with the number of reference frames, and falls within the scope of protection of this invention.

[0260] In another implementation, the objective function can incorporate a regularization term based on the two-sided projected residual term: ;

[0261] in This is the regularization strength parameter, typically ranging from 0.01 to 0.10; this regularization term constrains... To avoid deviating too far from the linear interpolation point in the feature space of the preceding and following segments, and to prevent numerical instability in the analytical solution when the geometric relationship between the principal component subspaces of the preceding and following segments is complex, while also ensuring that the structural constraint parameters provided to the AI ​​frame interpolation network are within acceptable limits. The reasonable interpolation range of the visual styles of the preceding and following segments further ensures the visual naturalness of the AI-generated transition frames, which falls within the scope of protection of this invention.

[0262] It should be noted that all of the above generation methods are based on (or its transformed form) The structural guidance inputs for the AI ​​frame interpolation network embody the core technical concept of this invention: extracting structural constraints from reference frames of preceding and following segments using mathematical methods, and then providing these constraints to the AI ​​frame interpolation network to guide pixel synthesis, enabling the AI ​​to generate transition frames that simultaneously satisfy the visual structural requirements of both the preceding and following segments. The specific architectures of the AI ​​frame interpolation networks (constraint optimization, diffusion model, adversarial network, large-scale pre-trained model, frame interpolation network plus post-processing) vary, but all receive inputs from... All the structural guidance signals used in this invention achieve essentially the same results (reducing the intensity of perceptual jumps at hard-cut boundaries) by employing essentially the same means (minimizing bilateral structural constraints to guide AI pixel generation), realizing essentially the same functions (generating visually seamless transition frames that connect naturally with preceding and following segments), and achieving essentially the same effects. Therefore, all of these methods fall within the scope of protection of this invention. Those skilled in the art can select any of the above AI generation methods or combinations thereof based on the actual deployment environment without making any inventive effort; the resulting technical solutions all fall within the scope of protection of this invention.

[0263] Example 2: Based on the same inventive concept, such as Figure 2 As shown, this embodiment also provides an AI-based video editing and compositing system, the system comprising:

[0264] The composite frame impact assessment module is used to determine whether the calculated frame of the last frame of the previous segment and the first frame of the next segment corresponding to the transition frame to be edited and synthesized is a cross-segment optical flow failure boundary; if it is a cross-segment optical flow failure boundary, the number of effective reference frames of the previous segment and the next segment is determined; and the normalized optical flow amplitude series of each pair of adjacent frames is recorded; and the impact factor of each frame in the effective reference frames corresponding to the previous segment and the next segment is calculated.

[0265] The composite frame feature extraction module is used to form a local structural feature vector by concatenating the gradient and chromaticity statistics extracted from each frame in the effective reference frame after dividing it into fixed pixel blocks; the module obtains the corresponding weighted covariance matrix based on the influence factors and local structural feature vectors of each frame in the effective reference frames of the preceding and following segments; the module decomposes the weighted covariance matrices of the preceding and following segments into eigenvalues ​​and calculates the mean of the corresponding eigenvalues; the module then selects the principal direction feature vectors of the preceding and following segments to form the projection matrices of the preceding and following segments, respectively.

[0266] The composite frame generation module is used to take the ratios of the number of effective reference frames in the previous segment and the number of effective reference frames in the subsequent segment to the sum of the number of effective reference frames in the previous and subsequent segments, respectively, as the balance coefficients of the previous and subsequent segments. The balance coefficients of the previous and subsequent segments are used as the weighted residuals of the local structural feature vectors of the transition frame under the projection matrices of the previous and subsequent segments, which minimize the weighted residuals. The pixel values ​​corresponding to the local structural feature vectors that minimize the weighted residuals are recorded as constraints. The constraints are injected into the AI ​​frame interpolation network that generates the transition frames. The AI ​​frame interpolation network is a pre-trained neural network model that receives the constraints and generates transition frames accordingly.

[0267] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0268] Finally, it should be noted that although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An AI-based video editing and compositing method, characterized in that, The method includes: The frame calculation is used to determine whether the previous segment's last frame and the next segment's first frame are cross-segment optical flow failure boundaries. If they are cross-segment optical flow failure boundaries, the number of effective reference frames for the previous and next segments is determined. The normalized optical flow amplitude series of each pair of adjacent frames is recorded. The influence factor of each frame in the effective reference frames corresponding to the previous and next segments is calculated. The gradient and chromaticity statistics extracted from each frame in the effective reference frame are concatenated to form a local structural feature vector; the weighted covariance matrix is ​​obtained according to the influence factor and local structural feature vector of each frame in the effective reference frame of the previous and subsequent segments; the weighted covariance matrix of the previous and subsequent segments is decomposed by eigenvalue and the mean of the corresponding eigenvalue is calculated. The principal direction feature vectors of the previous and subsequent segments are selected to form the projection matrix of the previous and subsequent segments respectively. The ratios of the number of effective reference frames in the previous segment and the number of effective reference frames in the subsequent segment to the sum of the number of effective reference frames in the previous and subsequent segments are used as the balance coefficients for the previous and subsequent segments, respectively. The balance coefficients for the previous and subsequent segments are used as the weighted residuals of the local structural feature vectors of the transition frame under the projection matrices of the previous and subsequent segments, respectively. The pixel values ​​corresponding to the local structural feature vectors that minimize the weighted residuals are recorded as constraints. The constraints are injected into the AI ​​frame interpolation network that generates the transition frames. The AI ​​frame interpolation network is a pre-trained neural network model that receives the constraints and generates transition frames accordingly.

2. The method according to claim 1, characterized in that, The method for determining whether the frame between the last frame of the preceding segment and the first frame of the following segment corresponding to the transition frame to be edited and synthesized is a cross-segment optical flow failure boundary; if it is a cross-segment optical flow failure boundary, the number of effective reference frames for the preceding and following segments is determined; and the normalized optical flow amplitude series of each pair of adjacent frames is recorded; the method for calculating the influence factor of each frame in the effective reference frames corresponding to the preceding and following segments is as follows: The dense optical flow field of the inter-frame pixel displacement is calculated between the last frame of the preceding segment and the first frame of the following segment corresponding to the transition frame to be edited and synthesized. The optical flow direction angle and optical flow amplitude of each pixel are obtained through the dense optical flow field. The direction consistency statistic is calculated through the optical flow direction angle of each pixel. The amplitude variation coefficient is calculated through the mean and standard deviation of the optical flow amplitude of each pixel. The optical flow effectiveness index is obtained through the direction consistency statistic and the amplitude variation coefficient. The optical flow effectiveness index and the preset effectiveness threshold are used to determine whether it is a cross-segment optical flow failure boundary. When a cross-segment optical flow failure boundary is determined, the optical flow amplitude between each pair of adjacent frames within the segment is calculated for each pair of adjacent frames in the preceding segment from the last frame forward and for each pair of adjacent frames in the following segment from the first frame backward. After normalization with the frame diagonal length, the amplitude is accumulated pair by pair. The number of valid reference frames for the preceding and following segments is determined when the accumulated value first reaches the preset accumulated amplitude threshold. The normalized optical flow amplitude sequence of each pair of adjacent frames is also recorded. The frame sequence distance of each effective reference frame from the splicing boundary is taken as the temporal nearest neighbor component, and the difference between the preset cumulative amplitude threshold and the normalized optical flow amplitude corresponding to each effective reference frame is taken as the motion quality component. The product of the temporal nearest neighbor component and the motion quality component corresponding to the effective reference frames of the previous segment and the next segment is normalized respectively to obtain the influence factor of each frame in the effective reference frames corresponding to the previous segment and the next segment.

3. The AI-based video editing and compositing method according to claim 1, characterized in that, The method for calculating the dense optical flow field of inter-frame pixel displacement between the last frame of the preceding segment and the first frame of the following segment corresponding to the transition frame to be edited and synthesized, obtaining the optical flow direction angle and amplitude of each pixel through the dense optical flow field; calculating the direction consistency statistic through the optical flow direction angle of each pixel, calculating the amplitude variation coefficient through the mean and standard deviation of the optical flow amplitude of each pixel; obtaining the optical flow effectiveness index through the direction consistency statistic and the amplitude variation coefficient, and determining whether it is a cross-segment optical flow failure boundary based on the optical flow effectiveness index and a preset effectiveness threshold includes: Calculate the pixel-by-pixel dense optical flow field for the last frame of the preceding segment and the first frame of the following segment to obtain the distance between the last frame of the preceding segment and the first frame of the following segment. Horizontal optical flow components of each pixel at coordinates With vertical optical flow component , where frame height With frame width , Values ​​range from 1 to , Values ​​range from 1 to The total number of intra-frame pixels between the last frame of the preceding segment and the first frame of the following segment is: ; Calculate the optical flow direction angle of each pixel and the optical flow amplitude of each pixel ; Traverse each pixel coordinate Until all are traversed Each pixel represents the entire dense optical flow field between the last frame of the previous segment and the first frame of the subsequent segment. Direction angle of 1 pixel Calculate the Rayleigh directional consistency statistic : ； Based on the entire dense optical flow field between the last frame of the preceding segment and the first frame of the following segment Optical flow amplitude per pixel Calculate the mean amplitude and amplitude standard deviation To prevent small amounts of zero positive Calculate the amplitude variation coefficient ; when hour ,when hour ;when The splicing boundary is then marked as the cross-segment optical flow failure boundary. This is a preset validity threshold.

4. The AI-based video editing and compositing method according to claim 3, characterized in that, When a cross-segment optical flow failure boundary is determined, the optical flow amplitude between each pair of adjacent frames within the segment is calculated for each adjacent frame in the preceding segment from the last frame forward, and for each adjacent frame in the following segment from the first frame backward. The amplitudes are then normalized to the frame diagonal length and accumulated pair by pair. The number of valid reference frames for the preceding and following segments is determined when the accumulated value first reaches a preset accumulated amplitude threshold. The method for recording the normalized optical flow amplitude sequence for each pair of adjacent frames includes: Calculate the frame diagonal length For the preceding segment, trace back from the last frame backwards, using the sequence number. Mark each pair of adjacent frames and traverse each pixel coordinate. Until all are traversed The pixel is calculated from the previous segment. Normalized average optical flow amplitude between adjacent frames : ； in and For the previous segment For adjacent frames in The horizontal and vertical optical flow components at the coordinates are obtained by pairwise accumulation. , Values ​​range from 1 to ; Determine whether it satisfies and The smallest positive integer This represents the number of valid reference frames for the preceding segment. Set a preset cumulative motion threshold; For each subsequent segment, starting from the first frame and working backwards, calculate the segment's [number]th frame in the same way. Normalized average optical flow amplitude between adjacent frames and cumulative value , Values ​​range from 1 to ; Determine whether it satisfies and The smallest positive integer The number of effective reference frames for the subsequent segment; the normalized optical flow amplitude series of the preceding and subsequent segments are output sequentially. and .

5. The AI-based video editing and compositing method according to claim 4, characterized in that, The method of taking the frame sequence distance of each effective reference frame from the splicing boundary as the temporal nearest neighbor component, and the difference between the preset cumulative amplitude threshold and the normalized optical flow amplitude corresponding to each effective reference frame as the motion quality component; and normalizing the product of the temporal nearest neighbor component and the motion quality component corresponding to the effective reference frames of the previous and subsequent segments to obtain the influence factor of each frame in the effective reference frames corresponding to the previous and subsequent segments includes: The position number of the previous segment is Valid reference frames, in which ,when The time corresponds to the last frame; define the temporal nearest neighbor component. Motion clarity component Impact Factor for: ； in Values ​​range from 1 to The position number of the subsequent segment is Valid reference frames, in which ,when Corresponding to the first frame; define the temporal nearest neighbor component. Motion clarity component Impact Factor for: ； in Values ​​range from 1 to .

6. The AI-based video editing and compositing method according to claim 5, characterized in that, The method for forming a local structural feature vector by concatenating the gradient extracted from each frame of the effective reference frame after dividing it into fixed pixel blocks with chromaticity statistics includes: Each frame image is fixed Pixel blocks are uniformly divided across the entire frame; gradient directions are calculated for each pixel within each block, using a fixed number of direction intervals. Perform directional histogram statistics and normalization to obtain each block 3D gradient orientation histogram; extract within each block Color space Component mean and The component mean is a two-dimensional chromaticity statistic; the gradient direction histogram of each block in the whole frame is sequentially concatenated with the chromaticity statistic to obtain... A local structural feature vector, where: ； The position number of the previous segment is The valid reference frame is extracted in the manner described above. The dimensional local structural eigenvector is denoted as The position number of the subsequent segment is The valid reference frame is extracted in the manner described above. The dimensional local structural eigenvector is denoted as .

7. The AI-based video editing and compositing method according to claim 6, characterized in that, The method for obtaining the corresponding weighted covariance matrix based on the influence factors and local structural feature vectors of each frame in the effective reference frames of the preceding and following segments; decomposing the weighted covariance matrices of the preceding and following segments into eigenvalues ​​and calculating their corresponding eigenvalue mean; and selecting the principal direction feature vectors of the preceding and following segments to form the projection matrices of the preceding and following segments respectively includes: Calculate the weighted mean vector of the preceding segments Construct the weighted covariance matrix of the first segment ;right Perform eigenvalue decomposition on a real symmetric matrix to obtain eigenvalues ​​in descending order. ; Statistical satisfaction The number of subscripts is denoted as , ,in Values ​​range from 1 to Take the front The eigenvectors are arranged in columns to form the projection matrix of the first segment. , for OK Column matrix; Calculate the weighted mean vector of the segments Construct the weighted covariance matrix of the subsequent segments. ;right Perform eigenvalue decomposition on a real symmetric matrix to obtain eigenvalues ​​in descending order. ; Statistical satisfaction The number of subscripts is denoted as , ,in Values ​​range from 1 to Take the front The eigenvectors are arranged in columns to form the projection matrix of the post-segment. , for OK Column matrix.

8. The AI-based video editing and compositing method according to claim 7, characterized in that, Normalized average optical flow amplitude between the first pair of adjacent frames from the last frame of the current segment satisfy At that time, the number of valid reference frames in the previous segment will be... Set to 1, extracted from the last frame of the previous segment. The dimensional local structural eigenvector is denoted as ,by The normalized column vector obtained by dividing by its Euclidean norm is directly used as the projection matrix of the preceding segment. Effective feature dimensions of the preceding segment Set to 1, the influence factor of the last frame of the previous segment. Set to 1; The normalized average optical flow amplitude between the first pair of adjacent frames in the subsequent segment satisfy At that time, the number of valid reference frames for the subsequent segment will be determined. Set to 1, and the first frame extracted from subsequent segments The dimensional local structural eigenvector is denoted as ,by The normalized column vector obtained by dividing by its Euclidean norm is directly used as the projection matrix of the subsequent segment. Effective feature dimensions of the later segment Set to 1, then the impact factor of the first frame of the subsequent segment. Set to 1.

9. The AI-based video editing and compositing method according to claim 8, characterized in that, The method of using the ratios of the number of effective reference frames in the previous segment and the number of effective reference frames in the subsequent segment to the sum of the number of effective reference frames in the previous and subsequent segments as the balance coefficients for the previous and subsequent segments, respectively, and using the balance coefficients for the previous and subsequent segments as the weighted residuals of the local structural feature vectors of the transition frame under the projection matrices of the previous and subsequent segments to minimize the weighted residuals, and using the pixel values ​​corresponding to the local structural feature vectors that minimize the weighted residuals as constraints, and injecting the constraints into the AI ​​frame interpolation network that generates the transition frames includes: Calculate the pre-fragment balance coefficient Balance coefficient with subsequent segments ; Calculate the pre-fragment side Vegan weight center of mass ,in Projection matrix of the previous segment The transpose matrix; calculate the fragment side. Vegan weight center of mass ,in Projection matrix of the later segment The transpose of the matrix; With transition frames 3D local structural feature vector Using the independent variable, minimize the objective function. : ； Calculate the minimum objective function When the minimum value is obtained, the corresponding A local structural feature vector, denoted as . ; the preceding pixel block The first two components serve as constraints on the gradient direction intensity distribution of each pixel block, while the last two components serve as constraints on the gradient direction intensity distribution of each pixel block. Component mean and The component mean constraint is used to reconstruct the pixel values ​​of each pixel block according to the above constraints, and these constraints are then injected into the AI ​​frame interpolation network that generates the transition frame.

10. An AI-based video editing and compositing system, characterized in that, The system includes: The composite frame impact assessment module is used to determine whether the calculated frame of the last frame of the previous segment and the first frame of the next segment corresponding to the transition frame to be edited and synthesized is a cross-segment optical flow failure boundary; if it is a cross-segment optical flow failure boundary, the number of effective reference frames of the previous segment and the next segment is determined; and the normalized optical flow amplitude series of each pair of adjacent frames is recorded; and the impact factor of each frame in the effective reference frames corresponding to the previous segment and the next segment is calculated. The composite frame feature extraction module is used to form a local structural feature vector by concatenating the gradient and chromaticity statistics extracted from each frame in the effective reference frame after dividing it into fixed pixel blocks; the module obtains the corresponding weighted covariance matrix based on the influence factors and local structural feature vectors of each frame in the effective reference frames of the preceding and following segments; the module decomposes the weighted covariance matrices of the preceding and following segments into eigenvalues ​​and calculates the mean of the corresponding eigenvalues; the module then selects the principal direction feature vectors of the preceding and following segments to form the projection matrices of the preceding and following segments, respectively. The composite frame generation module is used to take the ratios of the number of effective reference frames in the previous segment and the number of effective reference frames in the subsequent segment to the sum of the number of effective reference frames in the previous and subsequent segments, respectively, as the balance coefficients of the previous and subsequent segments. The balance coefficients of the previous and subsequent segments are used as the weighted residuals of the local structural feature vectors of the transition frame under the projection matrices of the previous and subsequent segments, which minimize the weighted residuals. The pixel values ​​corresponding to the local structural feature vectors that minimize the weighted residuals are recorded as constraints. The constraints are injected into the AI ​​frame interpolation network that generates the transition frames. The AI ​​frame interpolation network is a pre-trained neural network model that receives the constraints and generates transition frames accordingly.

Images