An automatic editing method and system for fusing text, audio, and video multi-modal materials

Abstract

This invention relates to the field of multimedia editing technology, specifically to an automated editing method and system for integrating multimodal materials including text, audio, and video. The method includes the following steps: establishing a nonlinear time remapping table based on audiovisual physical anchor points; obtaining vector-driven transition strategies; calculating heatmaps to generate dynamic damped layout trajectories; and performing multi-layer frame-by-frame rendering with parameters to obtain a multimodal synthesized video stream. In this invention, audiovisual physical anchor points are extracted using Hilbert transform and second-order differential optical flow to achieve precise frame-level alignment. Audio energy gradient-driven reciprocal mapping and exponential compensation are used to give the image a sense of breathing. Transition parameters are matched with the angle of the main motion vector to eliminate visual abrupt changes. A dynamic damped layout is generated based on a visually saliency heatmap and micro-motion dead zone determination, achieving intelligent text and image avoidance and smooth following, improving the semantic coherence and visual stability of multimodal fusion, and realizing high-precision automated editing throughout the entire process.

Description

Technical Field

[0001] This invention relates to the field of multimedia editing technology, and in particular to an automated editing method and system that integrates multimodal materials including text, images, audio, and video. Background Technology

[0002] With the rapid development of multimedia technology, the application of multimodal materials such as text, images, audio, and video in content creation is becoming increasingly widespread. Whether it's short video creation, advertising production, or documentary editing, deep integration and processing of multimodal materials are required. However, existing editing workflows largely rely on manual operation. Editors need to manually select materials, adjust audio-visual synchronization, and embed text and image elements based on subjective experience. This is not only time-consuming and labor-intensive, but the editing effect is also greatly affected by personal aesthetics and technical skills. When faced with massive amounts of material, it is difficult to guarantee the accuracy and stability of audio-visual synchronization in long video streams, making it impossible to achieve efficient integration of multimodal materials and high-quality output.

[0003] While existing automated editing technologies have alleviated the burden on manual labor to some extent, they are mostly limited to simple linear splicing or template application of single-modal materials. They lack the ability to correlate and mine the deep physical features of multimodal materials such as images, text, audio, and video. Although some technologies have attempted to introduce multimodal processing, they have failed to combine audiovisual physical transient features with optical flow field motion information, resulting in insufficient feature alignment accuracy, weak adaptive ability of editing strategies, and large synchronization errors of multiple elements. Moreover, when it comes to mixed images and text, static layouts or simple linear animations are often used, lacking dynamic avoidance mechanisms based on visual salience. This can easily cause images and text to obscure key subjects or produce visual abrupt changes, seriously affecting the semantic coherence and viewing experience of the finished product.

[0004] Therefore, there is an urgent need for an automated editing technology solution that can perform deep alignment based on audiovisual physical characteristics, adaptively generate nonlinear time remapping strategies, and realize dynamic damped layout of images and text, so as to improve editing efficiency and work quality. Summary of the Invention

[0005] To address the technical problems existing in the prior art, embodiments of the present invention provide an automated editing method that integrates multimodal materials including text, images, audio, and video, comprising the following steps:

[0006] S1: Acquire audio streams and video frames, calculate the audio transient slope and the second derivative of video motion acceleration, filter the moments when the absolute value of the peak time difference between the two is less than the synchronization window threshold, and obtain the audiovisual physical anchor point set.

[0007] S2: Calculate the audio energy gradient coefficient based on the audiovisual physical anchor point set, convert it into a playback rate scaling factor using the reciprocal mapping and exponential compensation function, calculate Bessel interpolation, and establish a nonlinear time remapping sequence list.

[0008] S3: Call the nonlinear time remapping sequence list to extract the main motion vector at the connection point, calculate the absolute value of the relative angle and compare it with the direction consistency and conflict threshold to match the transition parameters, and obtain the vector-driven transition strategy index;

[0009] S4: Calculate the heat map and extract the centroid for the sequence indexed by the vector-driven transition strategy, and calculate the Euclidean distance with the dynamic damping layout coordinate trajectory of the previous frame. Based on the interval determination results of the Euclidean distance, dead zone threshold and mutation threshold, perform hold, weight or update operations to generate the dynamic damping layout coordinate trajectory of the current frame.

[0010] S5: Determine the overlay position based on the coordinate trajectory of the dynamic damping layout, and perform multi-layer frame-by-frame rendering by combining the vector-driven transition strategy index parameters and the nonlinear time remapping sequence table timing logic to obtain a multimodal synthesized video stream.

[0011] As a further aspect of the present invention, the audiovisual physical anchor point set includes a synchronization timestamp, an absolute value of the time difference, and a keyframe index;

[0012] The nonlinear time remapping sequence table includes a playback rate scaling factor, Bezier control points, and a time remapping curve.

[0013] The vector-driven transition strategy index includes motion blur direction, brightness gain, and transition duration;

[0014] The coordinate trajectory of the dynamic damping layout includes geometric centroid coordinates, Euclidean distance, and damping coefficient;

[0015] The multimodal synthesized video stream includes a video track, image and text layers, and a synthesized frame sequence.

[0016] As a further aspect of the present invention, the step of performing maintain, weight, or update operations based on the interval determination results of Euclidean distance, dead zone threshold, and abrupt change threshold to generate a dynamic damping layout coordinate trajectory includes:

[0017] If the Euclidean distance is less than or equal to the micro-motion dead zone threshold, then the dynamic damping layout coordinate trajectory value of the previous frame is used as the dynamic damping layout coordinate trajectory of the current frame.

[0018] If the Euclidean distance is greater than the micro-motion dead zone threshold and less than or equal to the mutation threshold, then a weighted average calculation is performed on the geometric centroid of the current frame and the coordinate trajectory of the dynamic damping layout of the previous frame, and the calculation result is used as the coordinate trajectory of the dynamic damping layout of the current frame.

[0019] If the Euclidean distance is greater than the mutation threshold, the geometric centroid of the current frame is directly updated to the dynamic damping layout coordinate trajectory of the current frame.

[0020] As a further aspect of the present invention, the audio transient slope is obtained by performing a Hilbert transform on the original audio stream to extract the amplitude envelope and calculating the first derivative to quantify the drastic degree of instantaneous energy change;

[0021] The audio energy gradient coefficient is a quantitative index for driving video rate mapping obtained by performing integration and subtraction on the amplitude envelope within a sliding time window and then nonlinearly weighting.

[0022] The dead zone threshold is the minimum displacement judgment limit for filtering coding noise and minor jitter. When the displacement of the center of gravity of the image does not exceed the limit, the coordinates of the previous frame are forcibly maintained.

[0023] The mutation threshold is set as the maximum displacement judgment limit for recognizing scene switching and drastic changes. When the displacement of the center of gravity of the screen exceeds this limit, the smoothing process is skipped and the coordinates are updated directly.

[0024] As a further aspect of the present invention, the step of obtaining S1 is as follows:

[0025] S101: Acquire the raw audio data stream and video frame sequence to be processed, perform Hilbert transform on the raw audio data stream to extract the amplitude envelope, calculate the first derivative value of the amplitude envelope sequence, and obtain the audio transient slope feature value;

[0026] S102: Calculate the dense optical flow field matrix based on the video frame sequence, extract the mean optical flow modulus of each frame, calculate the second derivative of the mean optical flow modulus sequence, and obtain the video motion acceleration quantization index.

[0027] S103: Call the audio transient slope feature value and the video motion acceleration quantization index, calculate the absolute value of the peak time difference between the two, compare the absolute value of the time difference with the preset synchronization window threshold, filter the moments when the absolute value of the time difference is less than the preset synchronization window threshold, and establish an audiovisual physical anchor point set.

[0028] As a further aspect of the present invention, the step of obtaining S2 is as follows:

[0029] S201: Based on the audiovisual physical anchor point set, set a look-ahead sliding time window, calculate the integral and difference values ​​of the audio amplitude envelope within the window, and obtain the audio energy gradient coefficient;

[0030] S202: Call the audio energy gradient coefficient, use the reciprocal mapping function and the exponential compensation function to calculate the numerical mapping, convert the energy gradient value into a video playback rate control quantity, and generate a playback rate scaling factor.

[0031] S203: Obtain the playback rate scaling factor sequence, calculate Bezier interpolation, reconstruct the video frame timestamps based on the interpolation results, and establish a nonlinear time remapping sequence table.

[0032] As a further aspect of the present invention, the step of obtaining S3 is as follows:

[0033] S301: Call the nonlinear time remapping sequence table to locate the connection time of adjacent video segments, extract the planar motion vectors of the end frame and the beginning frame, and establish the main motion vector set of the boundary frame;

[0034] S302: Based on the boundary frame main motion vector set, analyze the coordinate components of the two main motion vectors, calculate the angle between them in the plane coordinate system, and obtain the absolute value of the relative angle between the main motion vectors;

[0035] S303: The absolute value of the relative angle between the main motion vectors is compared with the preset direction consistency threshold and conflict threshold respectively. Based on the comparison results, the corresponding blur direction and brightness gain parameters are matched to generate a vector-driven transition strategy index.

[0036] As a further aspect of the present invention, the step of obtaining S4 is as follows:

[0037] S401: Calculate a visual saliency heatmap for the video frame sequence indexed by the vector-driven transition strategy described above, perform numerical inversion and binarization calculations on the visual saliency heatmap, extract the maximum safe connected component and calculate its geometric centroid, and obtain the coordinates of the geometric centroid of the maximum safe region.

[0038] S402: Call the geometric centroid coordinates of the maximum safe area, read the dynamic damping layout coordinate trajectory value of the previous frame, calculate the Euclidean distance between the geometric centroid coordinates of the current frame and the trajectory value of the previous frame, and obtain the Euclidean metric value of the trajectory displacement.

[0039] S403: Determine the numerical range between the Euclidean metric value of the trajectory displacement and the micro-motion dead zone threshold and the abrupt change threshold. If the metric value falls into the micro-motion dead zone range, the value is maintained. If the metric value falls into the damping smoothing range, a weighted average calculation is performed. If the metric value falls into the abrupt change update range, the value is updated to generate the dynamic damping layout coordinate trajectory of the current frame.

[0040] As a further aspect of the present invention, the acquisition step of S5 is as follows:

[0041] S501: Based on the coordinate trajectory of the dynamic damping layout, analyze the spatial mapping relationship of the graphic materials in the video frame, determine the stacking order and pixel position of each layer, and obtain the graphic layer overlay positioning parameter set;

[0042] S502: Call the vector-driven transition strategy index and the nonlinear time remapping sequence table, extract the blur direction parameter, brightness gain parameter and playback rate scaling factor, and perform weighted fusion with the spatial distance information of the image and text layer overlay positioning parameter set to generate a composite rendering control parameter matrix.

[0043] S503: Based on the composite rendering control parameter matrix, perform frame-by-frame pixel synthesis and channel mixing on the original video frame and the layer pointed to by the image and text layer overlay positioning parameter set to obtain a multimodal composite video stream.

[0044] An automated editing system that integrates multimodal materials including text, images, audio, and video, the system comprising:

[0045] The audiovisual feature anchor point extraction module collects the original audio data stream and video frame sequence, performs Hilbert transform and second-order differential calculation of the mean magnitude of the optical flow field matrix, filters the moments when the absolute value of the time difference between the audio transient slope and the peak time value of the video motion acceleration meets the preset synchronization window threshold, and establishes an audiovisual physical anchor point set.

[0046] The time remapping sequence construction module sets a look-ahead sliding time window based on the audiovisual physical anchor point set, calculates the integral and difference values ​​of the audio amplitude envelope within the window to obtain the energy gradient coefficient, calls the reciprocal mapping function and the exponential compensation function to perform numerical mapping, converts the energy gradient coefficient into a playback rate scaling factor, calculates Bessel interpolation for the playback rate scaling factor sequence, reconstructs the video frame timestamp based on the interpolation results, and establishes a nonlinear time remapping sequence table.

[0047] The vector transition strategy generation module calls the nonlinear time remapping sequence list to locate the connection point, extracts the main motion vector of the end frame and the first frame of the entry point, calculates the absolute value of the relative angle and compares it with the direction consistency threshold and the conflict threshold, matches the blur direction and brightness gain parameters, and obtains the vector-driven transition strategy index.

[0048] The dynamic damping layout calculation module calculates the visual saliency heatmap and the geometric centroid coordinates of the maximum safe area for the sequence indexed by the vector-driven transition strategy described above. It determines the Euclidean distance between the current centroid and the dynamic damping layout coordinate trajectory of the previous frame, as well as the intervals of the micro-motion dead zone threshold and the mutation threshold. Based on the determination results, it updates the values ​​and generates the dynamic damping layout coordinate trajectory.

[0049] The multimodal rendering and compositing module determines the image and text overlay position based on the dynamic damping layout coordinate trajectory. Combining the rendering parameters in the vector-driven transition strategy index and the temporal logic of the nonlinear time remapping sequence table, it performs frame-by-frame pixel compositing and channel mixing on the original video frames and image and text layers to obtain a multimodal composite video stream.

[0050] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0051] In this invention, audiovisual physical anchor points are extracted through Hilbert transform and second-order differential of optical flow field, achieving precise frame-level alignment based on the essence of the signal, thus solving the problems of rhythm misalignment and lack of physical correlation. Audio energy gradients are used to drive nonlinear time remapping, and reciprocal mapping and exponential compensation are used to convert sound energy into playback rate scaling, giving the image a breathing feel that fluctuates with the auditory experience. The transition parameters are matched with the angle of the main motion vector to eliminate abrupt changes in shot transitions in accordance with visual inertia. A dynamic damping layout is generated based on a visual saliency heatmap, and intelligent text and image avoidance and smooth following are achieved through micro-motion dead zone determination, effectively avoiding key subject occlusion and high-frequency jitter, improving the semantic coherence, visual stability, and final quality of multimodal material fusion, and realizing high-precision editing processing with full automation. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the steps of the present invention;

[0054] Figure 2 This is a system flowchart of the present invention. Detailed Implementation

[0055] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0056] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0057] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0058] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0059] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0060] Please see Figure 1 This invention provides an automated editing method for integrating multimodal materials including text, images, audio, and video, comprising the following steps:

[0061] S1: Collect the raw audio data stream and video frame sequence to be processed, perform Hilbert transform and first-order differential calculation on the raw audio data stream to extract the audio transient slope, calculate the second-order differential value of the mean magnitude of the optical flow field matrix of the video frame sequence as the video motion acceleration, and select the moments when the absolute value of the time difference between the peak value of the audio transient slope and the peak value of the video motion acceleration is less than the preset synchronization window threshold to obtain the audiovisual physical anchor point set.

[0062] S2: Based on the audiovisual physical anchor point set, a look-ahead sliding time window is set, and the energy gradient coefficient is obtained by integral and difference calculation of the audio amplitude envelope within the window. The energy gradient coefficient is converted into a playback rate scaling factor using the reciprocal mapping function and the exponential compensation function, and the Bessel interpolation of the playback rate scaling factor sequence is calculated to establish a nonlinear time remapping sequence table.

[0063] S3: Call the non-linear time remapping sequence table to determine the connection point of adjacent video segments, extract the main motion vectors of the end frame and the beginning frame of the entry point, calculate the absolute value of the relative angle between the two main motion vectors in the plane coordinate system, compare the absolute value of the relative angle with the preset direction consistency threshold and conflict threshold respectively, match the corresponding blur direction and brightness gain parameters, and obtain the vector-driven transition strategy index.

[0064] S4: Calculate the visual saliency heatmap for the video frame sequence indexed by the vector-driven transition strategy. Extract the maximum safe connected component and geometric centroid coordinates through numerical inversion and binarization. Calculate the Euclidean distance between the geometric centroid coordinates of the current frame and the coordinate trajectory of the dynamic damping layout of the previous frame. Based on the interval determination results of the Euclidean distance, the micro-motion dead zone threshold, and the mutation threshold, perform the operations of keeping the values ​​of the previous frame, calculating the weighted average, and updating to the values ​​of the current frame to generate the coordinate trajectory of the dynamic damping layout.

[0065] Based on the interval determination results of Euclidean distance, micro-motion dead zone threshold, and abrupt change threshold, the operation of maintaining the previous frame value, calculating the weighted average, and updating to the current frame value is performed to generate the dynamic damping layout coordinate trajectory, including:

[0066] If the Euclidean distance is less than or equal to the micro-motion dead zone threshold, then the dynamic damping layout coordinate trajectory value of the previous frame is used as the dynamic damping layout coordinate trajectory of the current frame.

[0067] If the Euclidean distance is greater than the micro-motion dead zone threshold and less than or equal to the abrupt change threshold, then a weighted average calculation is performed on the geometric centroid of the current frame and the coordinate trajectory of the dynamic damping layout of the previous frame, and the calculation result is used as the coordinate trajectory of the dynamic damping layout of the current frame.

[0068] If the Euclidean distance is greater than the mutation threshold, the geometric centroid of the current frame will be directly updated to the dynamic damped layout coordinate trajectory of the current frame.

[0069] S5: Determine the superposition position of the graphic layer in the video frame based on the coordinate trajectory of the dynamic damping layout. Combine the rendering parameters in the vector-driven transition strategy index and the temporal logic of the nonlinear time remapping sequence table to perform multi-layer frame-by-frame rendering and obtain a multimodal synthesized video stream.

[0070] The audiovisual physical anchor point set includes synchronization timestamps, absolute time difference values, and keyframe indexes;

[0071] The nonlinear time remapping sequence list includes playback rate scaling factor, Bezier control points, and time remapping curves;

[0072] The vector-driven transition strategy index includes motion blur direction, brightness gain, and transition duration;

[0073] The coordinate trajectory of a dynamic damping layout includes geometric centroid coordinates, Euclidean distance, and damping coefficient;

[0074] Multimodal composite video streams include video tracks, image and text layers, and composite frame sequences.

[0075] Please see Figure 1 The steps to obtain S1 are as follows:

[0076] S101: Acquire the raw audio data stream and video frame sequence to be processed, perform Hilbert transform on the raw audio data stream to extract the amplitude envelope, calculate the first derivative value of the amplitude envelope sequence, and obtain the audio transient slope feature value;

[0077] The process involves acquiring the raw audio data stream and video frame sequence to be processed. First, the audio signal processing module is initialized, setting the sampling rate to the standard 44100Hz to cover the audible frequency range, and the sampling bit depth to 16 bits to ensure dynamic range. A time window of 2048 sampling points is defined, and Hilbert transform processing is performed on the raw audio data stream within this time window. Specifically, this is achieved by constructing a frequency response... All-pass filter, where The imaginary unit, For signal frequency, The sign function represents the original real signal. Transform into imaginary part signal Combine the two to construct an analytical signal For each sampling point in the analytic signal sequence, the arithmetic square root of the sum of the squares of the real and imaginary parts is calculated using the Pythagorean theorem, i.e., the operation is performed. Obtain the instantaneous amplitude value at that moment. For example, for a certain moment If the original signal value is 0.6 and the transformed imaginary part value is 0.8, then the amplitude envelope value at that moment is 1.0. A continuous amplitude envelope sequence is formed by iterating through all sampling points within the time window. Subsequently, discrete-time difference (DDT) operations are performed on the acquired amplitude envelope sequence. To preserve data characteristics while avoiding numerical inflation caused by extremely small time steps, the specific formula for the discrete difference operator is defined as follows:

[0078] ,

[0079] in, Indicates the current sampling time. Indicates the previous sampling time, reads the current time. The amplitude envelope value compared to the previous moment The amplitude envelope value is subtracted.

[0080] If the values ​​of three consecutive points in the amplitude envelope sequence are set to 0.5, 0.8, and 0.4, then the first differential characteristic value, i.e., the discrete difference value, is 0.8-0.5=0.3, and the second differential characteristic value is 0.4-0.8=-0.4. Arrange this series of differential results in chronological order to construct a dataset that reflects the drastic instantaneous change of audio energy and obtain the audio transient slope characteristic value.

[0081] S102: Calculate the dense optical flow field matrix based on the video frame sequence, extract the mean optical flow modulus of each frame, calculate the second derivative of the mean optical flow modulus sequence, and obtain the video motion acceleration quantization index.

[0082] The dense optical flow field matrix is ​​calculated based on a video frame sequence. The video resolution is set to 1920×1080 pixels and the frame rate to 30fps. The [number]th frame of the video sequence is read... Frame image and the first The image is framed and converted into a grayscale matrix, with each pixel coordinate point in the image matrix as an example. The constraint equation formula using constant brightness is as follows:

[0083] ,

[0084] in, Indicates pixel brightness intensity. For pixel space coordinates, For time, This is the inter-frame time interval. These are the horizontal and vertical displacement vectors, respectively;

[0085] The corresponding position of the optical flow is searched in the next frame, and a dense optical flow field matrix covering 2,073,600 pixels of the entire screen is constructed. Then, for each pixel in the optical flow field matrix, vector magnitude calculation is performed. The optical flow magnitude of the given pixel is obtained. The optical flow magnitudes of all pixels in the current frame are summed, and the sum is divided by the total number of pixels to obtain the average optical flow magnitude of the frame. For example, if the sum of the displacement magnitudes of all pixels in a certain frame is 4,147,200, then the mean optical flow magnitude of that frame is 2.0. This process is repeated for multiple consecutive frames to obtain a sequence of mean optical flow magnitudes. Then, a second-order differential calculation is performed on this sequence, first calculating the first-order difference between the mean values ​​of adjacent frames. Representing the velocity change, the first-order difference between adjacent velocities is then calculated. Represents the change in acceleration, i.e., the execution of calculations. For example, the value of three consecutive frames in the optical flow mode length mean sequence Then the second-order difference result is 4.0−2×5.0+2.0=−4.0, which yields the video motion acceleration quantization index;

[0086] S103: Call the audio transient slope feature value and the video motion acceleration quantization index, calculate the absolute value of the peak time difference between the two, compare the absolute value of the time difference with the preset synchronization window threshold, filter the moments when the absolute value of the time difference is less than the preset synchronization window threshold, and establish a set of audiovisual physical anchor points.

[0087] By invoking the audio transient slope feature value and the video motion acceleration quantization index, a peak determination window of 5 data points is first set. The sequence is then iterated to select moments whose values ​​are strictly greater than their adjacent values ​​as feature peak points, and a set of audio peak timestamps is recorded. With video peak timestamp set For each audio peak timestamp, calculate the difference between it and all video peak timestamps and take the absolute value. ,in, This represents the difference between the peak timestamps of the audio and video recordings.

[0088] A preset synchronization window threshold of 40ms is set. This threshold is based on the lower limit of human nervous system's perception of the simultaneity of audiovisual stimuli, which is usually between 40ms and 80ms. This stringent standard is set to ensure accurate synchronization. The absolute values ​​of each calculated time difference are compared with 0.04s. For example, the first pair |1.05−1.08|=0.03s is considered valid if 0.03<0.04s, and the time pair (1.05s, 1.08s) is retained. If the calculated result is greater than or equal to the threshold, it is considered invalid and is removed. All time pairs that meet the conditions are stored in a list in chronological order to establish an audiovisual physical anchor point set.

[0089] Please see Figure 1 The steps to obtain S2 are as follows:

[0090] S201: Based on the audiovisual physical anchor point set, set a look-ahead sliding time window, calculate the integral and difference values ​​of the audio amplitude envelope within the window, and obtain the audio energy gradient coefficient;

[0091] Based on the audiovisual physical anchor point set, a look-ahead sliding time window is set. First, the anchor point timestamps are read, and the window length is set to 200ms to cover approximately 8820 audio sampling points. The amplitude envelope sequence within this window is then traversed. Perform weighted numerical integration, the formula is:

[0092] ,

[0093] in, The total value of the integral. These are the start and end indices of the time window, respectively. The sampling interval is... To adapt the energy normalization gain factor to the visual perception level, it is set here. ;

[0094] The integral value representing the total energy of the interval is obtained by calculation, and a normalized integral value calculated for a certain window is set. Simultaneously calculate the absolute value of the difference between the amplitude values ​​at the beginning and end of the window. Set the basic weight coefficient The coefficient is set to 0.6, based on the statistical contribution rate of energy accumulation to visual rhythm, with a trend weighting coefficient. When the value is 0.4, a non-linear weighted operation is performed. The specific formula is as follows:

[0095] ;

[0096] Where K is the audio energy gradient coefficient;

[0097] Set the difference Substitute into the calculation Repeat the above calculation process for each anchor point to generate the corresponding coefficient value sequence and obtain the audio energy gradient coefficient;

[0098] S202: Call the audio energy gradient coefficient, use the reciprocal mapping function and the exponential compensation function to calculate the numerical mapping, convert the energy gradient value into a video playback rate control quantity, and generate a playback rate scaling factor;

[0099] Call the audio energy gradient coefficient, set the baseline playback speed to 1.0x, and define the reciprocal mapping function. This is used to map high energy to slow motion, where C is the rate control constant and ϵ is the minimum value of the avoidance denominator being zero;

[0100] The rate control constant C is set to 10.0. This constant is determined based on the quotient of the statistical average value of the energy gradient coefficient (10.0) and the standard multiple of speed. A zero-prevention minimum value is also set. If the input energy gradient coefficient K is 20.0, then the reciprocal mapping value is approximately 10 / 20.001 ≈ 0.5; simultaneously, an exponential compensation function is constructed. Make fine adjustments.

[0101] in, To compensate for the strength coefficient, The attenuation coefficient;

[0102] Set compensation strength With attenuation coefficient These coefficients are based on the tolerance setting for the lowest rate in the visual fluency test, and the compensation value is calculated by substituting K=20.0. Perform product operation Next, the maximum rate threshold is set to 3.0 and the minimum rate threshold is set to 0.1. If the calculation result exceeds this range, it is forcibly truncated to the boundary value, thereby forming a series of rate values ​​that conform to the physiological characteristics of vision and generating the playback rate scaling factor.

[0103] S203: Obtain the playback rate scaling factor sequence, calculate Bezier interpolation, reconstruct video frame timestamps based on the interpolation results, and establish a non-linear time remapping sequence table;

[0104] Obtain the playback rate scaling factor sequence, calculate Bezier interpolation, and treat the playback rate scaling factor sequence as keyframe control points on the timeline. Construct cubic Bézier curve equations within the time intervals corresponding to two adjacent anchor points, for the normalized time variable. Substitute the values ​​into Bessel's formula to calculate the instantaneous playback rate factor at that moment. For each frame of the original video, a new timestamp for the first frame is set by sampling on a Bézier curve based on its original timestamp index. For the first Frame, read its original frame duration For example, 0.033s, combined with the interpolation rate factor corresponding to that frame. Perform division operation

[0105] ,

[0106] in, For the remapped first Frame display duration;

[0107] For example, if the rate factor is 0.5, then the new duration is 0.033 / 0.5 = 0.066s, and the accumulation operation is performed. The precise position of each frame on the remapped timeline is obtained. All video frames are traversed to complete the stretching and compression calculations of the timeline. The current timestamp and the corresponding frame number are stored in pairs to establish a non-linear time remapping sequence table.

[0108] Please see Figure 1 Steps to obtain S3:

[0109] S301: Call the nonlinear time remapping sequence list to locate the connection time of adjacent video segments, extract the planar motion vectors of the end frame and the beginning frame, and establish the main motion vector set of the boundary frame;

[0110] The nonlinear time remapping sequence table is invoked to locate the connection points between adjacent video segments. The sequence table is traversed to retrieve discontinuities in timestamps or scene marker indices, locking the end frame of the preceding sequence and the beginning frame of the following sequence at the corresponding time point. Full-frame motion analysis is then performed on these two frames. The image is divided into a 16×16 pixel macroblock grid, and block matching search is performed on each macroblock to calculate the displacement vector. A direction histogram is constructed by statistically analyzing the displacement vectors of all macroblocks within the entire frame. The direction interval with the largest cumulative magnitude is selected as the main motion direction. This step aims to filter out background noise. The arithmetic mean of all vectors falling within this interval is calculated. For example, for the outgoing frame, the main motion vector is obtained statistically. For the first frame at the entry point, the main motion vector is obtained. The two two-dimensional vectors after weighted averaging and denoising are stored respectively. The planar motion vectors of the end frame and the beginning frame are extracted to establish the main motion vector set of the boundary frame.

[0111] S302: Based on the boundary frame main motion vector set, analyze the coordinate components of two main motion vectors, calculate the angle between them in the plane coordinate system, and obtain the absolute value of the relative angle between the main motion vectors;

[0112] Based on the boundary frame main motion vector set, the coordinate components of the two main motion vectors are analyzed, and the exit point tail frame vector stored in the set is read. with the first frame vector of the in-point ,

[0113] in, These are the horizontal and vertical components of the vector, respectively.

[0114] First, extract its horizontal and vertical components separately, and then perform a dot product operation. And use the Pythagorean theorem to calculate the magnitudes of the two vectors respectively. and If either modulus is close to zero, the included angle is directly determined to be zero to avoid calculation errors. If the modulus is valid, the cosine of the included angle between the two vectors is calculated. Then call the inverse cosine function. Calculate the included angle in radians and convert it to an angle value, then set... The dot product is 0, and the modulus is 10. The calculation yields... This leads to the conclusion that the included angle is 90 degrees. This geometric operation is performed on the vectors at each connection moment to obtain the absolute value of the relative included angle of the main motion vector.

[0115] S303: Compare the absolute value of the relative angle between the main motion vector and the preset direction consistency threshold and conflict threshold respectively. Based on the comparison results, match the corresponding blur direction and brightness gain parameters to generate a vector-driven transition strategy index.

[0116] The absolute value of the relative angle between the main motion vectors is compared with the preset direction consistency threshold and conflict threshold, respectively. The direction consistency threshold is set as follows: This threshold is set based on the human eye's visual persistence tolerance range for continuous motion trajectories, and the conflict threshold is set as follows: This threshold is set based on the visual conflict boundary generated by the opposite movement, and the calculated relative angle value is read. And execute the segmented judgment logic, if If it is considered a smooth transition, a directional blur strategy is applied without increasing brightness; if If the angle is within the specified range, it is determined to be a conflicting connection, and a high-brightness flash white strategy is matched, setting the brightness gain parameter to 1.5 to create a visual mask; if the angle is within the specified range... and Between these points, linear interpolation is performed to construct a linear function:

[0117] ,

[0118] in, For the calculated brightness gain, The preset maximum allowable brightness gain is 0.5;

[0119] Set the current included angle to Substituting the values ​​into the calculation, the brightness gain is calculated to be 0.25. The calculated blur type, blur direction parameter, blur intensity parameter and brightness gain value are packaged and combined. Based on the comparison results, the corresponding blur direction and brightness gain parameters are matched to generate a vector-driven transition strategy index.

[0120] Please see Figure 1 Steps to obtain S4:

[0121] S401: Calculate the visual saliency heatmap for the video frame sequence indexed by the vector-driven transition strategy, perform numerical inversion and binarization calculation on the visual saliency heatmap, extract the maximum safe connected component and calculate its geometric centroid, and obtain the geometric centroid coordinates of the maximum safe region.

[0122] For video frame sequences indexed using a vector-driven transition strategy, a visual saliency heatmap is calculated. The image is converted to the LAB color space and feature maps are extracted. A grayscale heatmap is generated using linear weighting, and the weighting formula is executed.

[0123] ,

[0124] in, For significant thermal values, These are the normalized characteristic difference values ​​for channels L, a, and b, respectively. These are the weighting coefficients for each channel;

[0125] Set weight coefficients Then, numerical inversion and binarization calculations are performed on the heatmap, and the inversion operation is performed on each pixel in the matrix. A safe binarization threshold of 150 is set. This threshold is based on the average background brightness statistics of a large number of video frames (approximately 128) and is increased by about 17% to ensure a safety margin. The inverted pixel values ​​are compared with this threshold. If the value is greater than the threshold, it is marked as a safe area; otherwise, it is marked as a critical area. A binarization mask matrix is ​​constructed in this way. A connected component labeling algorithm is used to select the connected component with the largest number of pixels as the largest safe connected component. Finally, its geometric centroid is calculated. The horizontal and vertical coordinates of all pixels in the region are accumulated and the sum is divided by the total number of pixels in the region, N. For example, if the accumulated sums are 115,200,000 and 64,800,000 respectively and N=120,000, the centroid coordinates are calculated to be (960,540). The geometric centroid coordinates of the largest safe area are obtained.

[0126] S402: Call the geometric centroid coordinates of the maximum safe area, read the dynamic damping layout coordinate trajectory value of the previous frame, calculate the Euclidean distance between the geometric centroid coordinates of the current frame and the trajectory value of the previous frame, and obtain the Euclidean metric value of the trajectory displacement.

[0127] To retrieve the geometric centroid coordinates of the maximum safe area, first read the dynamic damping layout coordinate trajectory values ​​stored at the end of the previous frame's processing. and the centroid coordinates calculated in the current frame ,

[0128] in These are the smoothed coordinates from the previous frame. The centroid coordinates calculated for the current frame;

[0129] For example, if the coordinates of the previous frame are (958.5, 541.2) and the coordinates of the current frame are (965.0, 538.0), then calculate the Euclidean distance between the geometric centroid coordinates of the current frame and the trajectory values ​​of the previous frame, using the distance formula between two points:

[0130] ,

[0131] Perform the calculation and substitute the values ​​to obtain This value precisely quantifies the instantaneous displacement of the center of gravity of the image. The distance value is retained to three decimal places to obtain the Euclidean metric value of the trajectory displacement.

[0132] S403: Determine the numerical range between the Euclidean metric value of the trajectory displacement and the micro-motion dead zone threshold and the abrupt change threshold. If the metric value falls into the micro-motion dead zone range, the value is kept. If the metric value falls into the damping smoothing range, a weighted average calculation is performed. If the metric value falls into the abrupt change update range, the value is updated to generate the dynamic damping layout coordinate trajectory of the current frame.

[0133] The system determines the numerical range between the Euclidean metric value of trajectory displacement and the micro-motion dead zone threshold and abrupt change threshold. The micro-motion dead zone threshold is set to 5.0 pixels to filter video encoding noise and minor jitter, and the abrupt change threshold is set to 150.0 pixels to identify scene transitions or drastic changes. The calculated metric values ​​are then read. The value is compared with two thresholds. If the metric is less than 5.0, it is determined to fall into the micro-motion dead zone, and the output coordinates of the current frame are forced to be equal to the coordinates of the previous frame. If the metric is greater than 150.0, it is determined to fall into the abrupt update zone, and the output coordinates of the current frame are directly set to be equal to the centroid coordinates of the current frame. If the metric is between the two thresholds, it is determined to fall into the damped smoothing zone, and a weighted average calculation is performed, introducing a damping coefficient based on the system response time. Using the formula:

[0134] Perform iterative updates.

[0135] in, These are the dynamic damping layout coordinates for the final output of the current frame. The damping coefficient;

[0136] Substitute the numerical values ​​to calculate the x-coordinate The calculated final coordinates are stored and used as the reference for the next frame calculation, generating the dynamic damping layout coordinate trajectory of the current frame.

[0137] Please see Figure 1 Steps to obtain S5:

[0138] S501: Analyze the spatial mapping relationship of graphic materials in the video frame based on the coordinate trajectory of the dynamic damping layout, determine the stacking order and pixel position of each layer, and obtain the set of graphic layer overlay positioning parameters;

[0139] Based on the coordinate trajectory analysis of the dynamic damping layout, the spatial mapping relationship of the graphic materials in the video frame is analyzed, and the smooth centroid coordinates of the current frame are read. As an alignment reference point, the base resolution of the video canvas is set to 1920×1080 pixels. The metadata of the text and image assets is loaded to obtain their inherent width and height dimensions, such as 420×150. The anchor point of the text and image assets is set as their geometric center. Based on this anchor point, the coordinates of the top-left vertex of the text and image assets on the canvas are calculated, and a subtraction operation is performed. and ,in, Draw the origin point at the top left corner of the image and text material;

[0140] Next, boundary constraint determination is performed, and the safe margin threshold is set to 50 pixels. This threshold is set according to the overscan area standard of the display device. The calculated origin coordinates are compared and corrected with the canvas boundary value to ensure that the material is within the safe viewing range. At the same time, a stacking depth index is assigned to each visual element, and a rectangular bounding box data containing the coordinates of four vertices is constructed to determine the stacking order and pixel position of each layer and obtain the set of image and text layer overlay positioning parameters.

[0141] S502: Call the vector-driven transition strategy index and non-linear time remapping sequence list, extract the blur direction parameters, brightness gain parameters and playback rate scaling factor, and combine them with the spatial distance information of the image and text layer overlay positioning parameter set for weighted fusion to generate a composite rendering control parameter matrix.

[0142] The vector-driven transition strategy index and non-linear time remapping sequence table are invoked to retrieve the transition strategy data for the current frame, and the blur direction angle and blur intensity value are extracted. In addition to the brightness gain coefficient, the playback rate scaling factor is also read. Then, the spatial distance weight is calculated to achieve differentiated rendering. The Euclidean distance D is calculated using the coordinates of the center of the image layer and the optical center of the screen, and an inverse attenuation function is constructed.

[0143] ,

[0144] in, This is the distance weighting coefficient. D is the distance attenuation coefficient, and D is the Euclidean distance from the center of the image / text to the center of the screen.

[0145] Set distance attenuation coefficient It is set based on the visual attenuation characteristics of Gaussian blur. With a distance D=200, the weights are calculated. This weight is used to attenuate the blur intensity of the image layer, and the calculation is performed. This allows for maintaining the relative clarity of the foreground text while preserving the dynamic blur of the background. The adjusted blur parameters of the image and text layers, the global brightness gain, the transparency coefficient calculated based on the playback rate, and the geometric coordinate data of the layers are uniformly encapsulated to generate a composite rendering control parameter matrix.

[0146] S503: Based on the composite rendering control parameter matrix, perform frame-by-frame pixel synthesis and channel mixing on the original video frame and the layer pointed to by the image and text layer overlay positioning parameter set to obtain a multimodal composite video stream;

[0147] Based on the composite rendering control parameter matrix, the image compositing engine is initialized, the specified mapping timestamps in the matrix are read, the corresponding source frame image data is extracted from the original video stream, its color space is converted to a 32-bit floating-point linear space, pixel-level brightness adjustment is performed on the background video layer, and all pixel channel values ​​are traversed. Perform multiplication operations A brightness gain is applied, and a Gaussian kernel function is used to generate a directional blur convolution kernel based on a specified angle. This kernel is then used to perform convolution operations on the background layer to produce a motion trailing effect. Simultaneously, the image and text layers are rasterized at specified coordinates based on image and text positioning parameters, and the calculated differential blur intensity is applied. Finally, a frame-by-frame pixel blending operation is performed, using an improved dynamic alpha blending formula:

[0148] ,

[0149] in, To output pixel color, These are the color values ​​for the foreground of the image / text and the background of the video. To adjust according to playback speed Dynamically changing transparency coefficient;

[0150] The transparency coefficient is set as an inverse correlation function of the playback rate, that is, the slower the playback rate, the higher the transparency to enhance the immersive experience. For each pixel in the overlapping area, the foreground color value and the background color value are read and weighted and summed together with the calculated transparency channel to calculate the final composite pixel value. The processed frame data is quantized back to 8-bit integer format and output in sequence. The original video frame and the layer pointed to by the image and text layer overlay positioning parameter set are composited and mixed frame by frame to obtain a multimodal composite video stream.

[0151] It should be noted that all parameter values, threshold settings, weight coefficients, and specific calculation results mentioned in the above embodiments, such as sampling rate of 44100Hz, time window of 200ms, basic weight coefficient of 0.6, distance attenuation coefficient of 0.005, and specific pixel coordinate values, are exemplary assumptions set for the purpose of clearly illustrating the technical solution and logical flow of the present invention. In actual application scenarios, those skilled in the art can make adaptive adjustments or optimizations to the above parameters according to the specific hardware processing capabilities, characteristics of the input materials, and specific business needs.

[0152] Please see Figure 2 An automated editing system that integrates multimodal materials including text, images, audio, and video, comprising:

[0153] The audiovisual feature anchor point extraction module collects the original audio data stream and video frame sequence, performs Hilbert transform and second-order differential calculation of the mean magnitude of the optical flow field matrix, filters the moments when the absolute value of the time difference between the audio transient slope and the peak time value of the video motion acceleration meets the preset synchronization window threshold, and establishes an audiovisual physical anchor point set.

[0154] The time remapping sequence construction module sets a look-ahead sliding time window based on the audiovisual physical anchor point set, calculates the integral and difference values ​​of the audio amplitude envelope within the window to obtain the energy gradient coefficient, calls the reciprocal mapping function and the exponential compensation function to perform numerical mapping, converts the energy gradient coefficient into a playback rate scaling factor, calculates Bessel interpolation for the playback rate scaling factor sequence, reconstructs the video frame timestamps based on the interpolation results, and establishes a nonlinear time remapping sequence table.

[0155] The vector transition strategy generation module calls the non-linear time remapping sequence list to locate the connection point, extracts the main motion vector of the end frame and the first frame of the entry point, calculates the absolute value of the relative angle and compares it with the direction consistency threshold and conflict threshold, matches the blur direction and brightness gain parameters, and obtains the vector-driven transition strategy index.

[0156] The dynamic damping layout calculation module calculates the visual saliency heatmap and the geometric centroid coordinates of the maximum safe area for the sequence indexed by the vector-driven transition strategy. It determines the Euclidean distance between the current centroid and the dynamic damping layout coordinate trajectory of the previous frame, as well as the intervals of the micro-motion dead zone threshold and the mutation threshold. Based on the determination results, it updates the values ​​and generates the dynamic damping layout coordinate trajectory.

[0157] The multimodal rendering and compositing module determines the image and text overlay position based on the dynamic damping layout coordinate trajectory. Combining the rendering parameters in the vector-driven transition strategy index and the temporal logic of the nonlinear time remapping sequence table, it performs frame-by-frame pixel compositing and channel mixing on the original video frames and image and text layers to obtain a multimodal composite video stream.

[0158] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An automated editing method integrating multimodal materials including text, images, audio, and video, characterized in that, Includes the following steps: S1: Acquire audio streams and video frames, calculate the audio transient slope and the second derivative of video motion acceleration, filter the moments when the absolute value of the peak time difference between the two is less than the synchronization window threshold, and obtain the audiovisual physical anchor point set. S2: Calculate the audio energy gradient coefficient based on the audiovisual physical anchor point set, convert it into a playback rate scaling factor using the reciprocal mapping and exponential compensation function, calculate Bessel interpolation, and establish a nonlinear time remapping sequence list. S3: Call the nonlinear time remapping sequence list to extract the main motion vector at the connection point, calculate the absolute value of the relative angle and compare it with the direction consistency and conflict threshold to match the transition parameters, and obtain the vector-driven transition strategy index; S4: Calculate the heat map and extract the centroid for the sequence indexed by the vector-driven transition strategy, and calculate the Euclidean distance with the dynamic damping layout coordinate trajectory of the previous frame. Based on the interval determination results of the Euclidean distance, dead zone threshold and mutation threshold, perform hold, weight or update operations to generate the dynamic damping layout coordinate trajectory of the current frame. S5: Determine the overlay position based on the coordinate trajectory of the dynamic damping layout, and perform multi-layer frame-by-frame rendering by combining the vector-driven transition strategy index parameters and the nonlinear time remapping sequence table timing logic to obtain a multimodal synthesized video stream.

2. The automated editing method for integrating multimodal materials including text, images, audio, and video according to claim 1, characterized in that, The audiovisual physical anchor point set includes synchronization timestamps, absolute time difference values, and keyframe indexes; the nonlinear time remapping sequence table includes playback rate scaling factors, Bezier control points, and time remapping curves; the vector-driven transition strategy index includes motion blur direction, brightness gain, and transition duration; the dynamic damping layout coordinate trajectory includes geometric centroid coordinates, Euclidean distance, and damping coefficients; and the multimodal synthesized video stream includes video tracks, image and text layers, and synthesized frame sequences.

3. The automated editing method for integrating multimodal materials including text, images, audio, and video according to claim 1, characterized in that, The process of determining the interval based on Euclidean distance, dead zone threshold, and abrupt change threshold, performing hold, weight, or update operations to generate a dynamic damping layout coordinate trajectory includes: If the Euclidean distance is less than or equal to the micro-motion dead zone threshold, then the dynamic damping layout coordinate trajectory value of the previous frame is used as the dynamic damping layout coordinate trajectory of the current frame. If the Euclidean distance is greater than the micro-motion dead zone threshold and less than or equal to the mutation threshold, then a weighted average calculation is performed on the geometric centroid of the current frame and the coordinate trajectory of the dynamic damping layout of the previous frame, and the calculation result is used as the coordinate trajectory of the dynamic damping layout of the current frame. If the Euclidean distance is greater than the mutation threshold, the geometric centroid of the current frame is directly updated to the dynamic damping layout coordinate trajectory of the current frame.

4. The automated editing method for integrating multimodal materials including text, images, audio, and video according to claim 1, characterized in that, The audio transient slope is obtained by performing a Hilbert transform on the original audio stream to extract the amplitude envelope and calculating the first derivative to quantify the drastic degree of instantaneous energy change; The audio energy gradient coefficient is a quantitative index for driving video rate mapping obtained by performing integration and subtraction on the amplitude envelope within a sliding time window and then nonlinearly weighting. The dead zone threshold is the minimum displacement judgment limit for filtering coding noise and minor jitter. When the displacement of the center of gravity of the image does not exceed the limit, the coordinates of the previous frame are forcibly maintained. The mutation threshold is set as the maximum displacement judgment limit for recognizing scene switching and drastic changes. When the displacement of the center of gravity of the screen exceeds this limit, the smoothing process is skipped and the coordinates are updated directly.

5. The automated editing method for integrating multimodal materials including text, images, audio, and video according to claim 1, characterized in that, The steps for obtaining S1 are as follows: S101: Acquire the raw audio data stream and video frame sequence to be processed, perform Hilbert transform on the raw audio data stream to extract the amplitude envelope, calculate the first derivative value of the amplitude envelope sequence, and obtain the audio transient slope feature value; S102: Calculate the dense optical flow field matrix based on the video frame sequence, extract the mean optical flow modulus of each frame, calculate the second derivative of the mean optical flow modulus sequence, and obtain the video motion acceleration quantization index. S103: Call the audio transient slope feature value and the video motion acceleration quantization index, calculate the absolute value of the peak time difference between the two, compare the absolute value of the time difference with the preset synchronization window threshold, filter the moments when the absolute value of the time difference is less than the preset synchronization window threshold, and establish an audiovisual physical anchor point set.

6. The automated editing method for integrating multimodal materials including text, images, audio, and video according to claim 1, characterized in that, The steps for obtaining S2 are as follows: S201: Based on the audiovisual physical anchor point set, set a look-ahead sliding time window, calculate the integral and difference values ​​of the audio amplitude envelope within the window, and obtain the audio energy gradient coefficient; S202: Call the audio energy gradient coefficient, use the reciprocal mapping function and the exponential compensation function to calculate the numerical mapping, convert the energy gradient value into a video playback rate control quantity, and generate a playback rate scaling factor. S203: Obtain the playback rate scaling factor sequence, calculate Bezier interpolation, reconstruct the video frame timestamps based on the interpolation results, and establish a nonlinear time remapping sequence table.

7. The automated editing method for integrating multimodal materials including text, images, audio, and video according to claim 1, characterized in that, The steps for obtaining S3 are as follows: S301: Call the nonlinear time remapping sequence table to locate the connection time of adjacent video segments, extract the planar motion vectors of the end frame and the beginning frame, and establish the main motion vector set of the boundary frame; S302: Based on the boundary frame main motion vector set, analyze the coordinate components of the two main motion vectors, calculate the angle between them in the plane coordinate system, and obtain the absolute value of the relative angle between the main motion vectors; S303: The absolute value of the relative angle between the main motion vectors is compared with the preset direction consistency threshold and conflict threshold respectively. Based on the comparison results, the corresponding blur direction and brightness gain parameters are matched to generate a vector-driven transition strategy index.

8. The automated editing method for integrating multimodal materials including text, images, audio, and video according to claim 1, characterized in that, The steps for obtaining S4 are as follows: S401: Calculate a visual saliency heatmap for the video frame sequence indexed by the vector-driven transition strategy described above, perform numerical inversion and binarization calculations on the visual saliency heatmap, extract the maximum safe connected component and calculate its geometric centroid, and obtain the coordinates of the geometric centroid of the maximum safe region. S402: Call the geometric centroid coordinates of the maximum safe area, read the dynamic damping layout coordinate trajectory value of the previous frame, calculate the Euclidean distance between the geometric centroid coordinates of the current frame and the trajectory value of the previous frame, and obtain the Euclidean metric value of the trajectory displacement. S403: Determine the numerical range between the Euclidean metric value of the trajectory displacement and the micro-motion dead zone threshold and the abrupt change threshold. If the metric value falls into the micro-motion dead zone range, the value is maintained. If the metric value falls into the damping smoothing range, a weighted average calculation is performed. If the metric value falls into the abrupt change update range, the value is updated to generate the dynamic damping layout coordinate trajectory of the current frame.

9. The automated editing method for integrating multimodal materials including text, images, audio, and video according to claim 1, characterized in that, The steps for obtaining S5 are as follows: S501: Based on the coordinate trajectory of the dynamic damping layout, analyze the spatial mapping relationship of the graphic materials in the video frame, determine the stacking order and pixel position of each layer, and obtain the graphic layer overlay positioning parameter set; S502: Call the vector-driven transition strategy index and the nonlinear time remapping sequence table, extract the blur direction parameter, brightness gain parameter and playback rate scaling factor, and perform weighted fusion with the spatial distance information of the image and text layer overlay positioning parameter set to generate a composite rendering control parameter matrix. S503: Based on the composite rendering control parameter matrix, perform frame-by-frame pixel synthesis and channel mixing on the original video frame and the layer pointed to by the image and text layer overlay positioning parameter set to obtain a multimodal composite video stream.

10. An automated editing system integrating multimodal materials including text, images, audio, and video, characterized in that, The system is used to implement the automated editing method for fusing multimodal materials including text, images, audio, and video as described in any one of claims 1-9, and the system includes: The audiovisual feature anchor point extraction module collects the original audio data stream and video frame sequence, performs Hilbert transform and second-order differential calculation of the mean magnitude of the optical flow field matrix, filters the moments when the absolute value of the time difference between the audio transient slope and the peak time value of the video motion acceleration meets the preset synchronization window threshold, and establishes an audiovisual physical anchor point set. The time remapping sequence construction module sets a look-ahead sliding time window based on the audiovisual physical anchor point set, calculates the integral and difference values ​​of the audio amplitude envelope within the window to obtain the energy gradient coefficient, calls the reciprocal mapping function and the exponential compensation function to perform numerical mapping, converts the energy gradient coefficient into a playback rate scaling factor, calculates Bessel interpolation for the playback rate scaling factor sequence, reconstructs the video frame timestamp based on the interpolation results, and establishes a nonlinear time remapping sequence table. The vector transition strategy generation module calls the nonlinear time remapping sequence list to locate the connection point, extracts the main motion vector of the end frame and the first frame of the entry point, calculates the absolute value of the relative angle and compares it with the direction consistency threshold and the conflict threshold, matches the blur direction and brightness gain parameters, and obtains the vector-driven transition strategy index. The dynamic damping layout calculation module calculates the visual saliency heatmap and the geometric centroid coordinates of the maximum safe area for the sequence indexed by the vector-driven transition strategy described above. It determines the Euclidean distance between the current centroid and the dynamic damping layout coordinate trajectory of the previous frame, as well as the intervals of the micro-motion dead zone threshold and the mutation threshold. Based on the determination results, it updates the values ​​and generates the dynamic damping layout coordinate trajectory. The multimodal rendering and compositing module determines the image and text overlay position based on the dynamic damping layout coordinate trajectory. Combining the rendering parameters in the vector-driven transition strategy index and the temporal logic of the nonlinear time remapping sequence table, it performs frame-by-frame pixel compositing and channel mixing on the original video frames and image and text layers to obtain a multimodal composite video stream.

Images