An intelligent video clip processing system based on multi-modal data fusion

Abstract

The application discloses an intelligent video clip processing system based on multi-modal data fusion, and relates to the technical field of computer vision, and solves the problems of unclear time correspondence of pictures, audio and subtitles in existing video clips, fixed weight fusion causing sound and picture asynchronization, and inaccurate highlight selection. The system comprises a processor and a memory. When the program in the memory is executed by the processor, the following steps are implemented: receiving a video to be clipped and corresponding audio files and subtitle files; analyzing video and audio features to obtain picture switching candidate points, audio beat candidate points and spectrum mutation candidate points; determining target clip points according to the time correspondence of the picture switching candidate points and the beat or spectrum mutation candidate points; generating a clip video according to the target clip points, and rendering corresponding segment boundary identifiers on the time axis of an interactive interface for preview. The application improves the accuracy of clip points, the synchronization of sound and picture, and the precision of highlight screening through multi-modal time alignment, dynamic weight distribution and sound and picture verification linkage.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to an intelligent video editing and processing system based on multimodal data fusion. Background Technology

[0002] With the development of short videos, film and television production, digital advertising, and online media, the quantity and types of video footage collected are constantly increasing, and video editing is gradually shifting from manual operation to automated processing. Existing video editing systems typically process single or multiple types of data, such as video footage, audio, and subtitles, primarily performing operations such as video clip extraction, shot splicing, subtitle addition, audio adjustment, transition settings, and final output. Computer vision technology can identify main objects, scene boundaries, motion trajectories, color, and brightness distribution in video frames; audio processing technology can extract waveform peaks, spectral energy, loudness changes, and beat positions; and subtitles can provide sentence boundaries, keyword locations, and narrative paragraph information.

[0003] However, existing video editing methods still have problems with unclear time correspondence between various types of data: audio beat points and video switching points are prone to time deviations, subtitle boundaries and scene boundaries are difficult to correspond accurately, and there is a lack of unified correlation between changes in the main subject of the image and changes in audio energy.

[0004] Some systems use fixed weights to process video, audio, and subtitle data, which cannot adjust the participation of various data according to the feature changes of different time periods. This can easily lead to audio-visual desynchronization, semantic segment fragmentation, and inaccurate selection of highlight content. At the same time, existing systems usually rely on a single video change or a single audio change to identify video segment boundaries, making it difficult to cross-confirm audiovisual boundaries.

[0005] During transition processing, the transition type, transition intensity, and segment duration are usually determined by preset templates, failing to coordinate with the movement direction, color differences, beat position, and audio energy of adjacent segments.

[0006] Therefore, it is necessary to propose an intelligent video editing system based on multimodal data fusion to establish the time alignment relationship between video, audio and subtitle data, and to perform collaborative processing of video segment recognition, highlight selection and transition. Summary of the Invention

[0007] This invention aims to provide an intelligent video editing system based on multimodal data fusion. By performing multimodal feature analysis on the video, audio, and subtitles to be edited, and determining the target editing point based on the time correspondence between candidate points of scene switching and candidate points of audio beat or candidate points of spectral change, and by combining dynamic weight allocation and audio-visual verification linkage, it solves the problems of unclear time correspondence between video, audio, and subtitles, audio-visual asynchrony caused by fixed weight fusion, and inaccurate selection of highlight content in existing video editing.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] An intelligent video editing and processing system based on multimodal data fusion includes a processor, a memory, and a display module. The memory stores a computer program, which, when executed by the processor, performs the following functions:

[0010] Receive the video to be edited, as well as the corresponding audio file and subtitle file;

[0011] The video and audio files to be edited are subjected to feature analysis to obtain candidate points for video scene transitions, and candidate points for beats and spectral changes in audio.

[0012] Align the candidate points for scene switching with the candidate points for beat or the candidate points for spectral change on the time axis to obtain candidate clip points;

[0013] The video clips are arranged and edited based on the candidate editing points to generate an edited video. The display module is then controlled to render and display the clip boundary markers corresponding to the candidate editing points on the timeline of the editing interface for interactive preview.

[0014] Preferably, when determining candidate clipping points, the processor is specifically used to:

[0015] Calculate the time deviation between the candidate point of the scene transition and the nearest candidate point of the beat on the timeline to establish a basic alignment relationship; divide the time difference between the candidate point of the scene transition and the previous adjacent candidate point of the beat by the beat period between the adjacent candidate points of the beat to obtain the normalized phase; lock the candidate points of the scene transition whose normalized phase is within the preset phase range as phase alignment points; allocate the time offset of the unlocked candidate points of the scene transition to the editing adjustment amount before and after the point to form the beat alignment record after offset compensation, and include it in the candidate editing point set.

[0016] Preferably, when the processor executes the computer program, it is also configured to perform a joint verification function:

[0017] Calculate the boundary time deviation between the candidate point for scene switching and the candidate point for the closest spectral change in time. When the boundary time deviation is not greater than a preset deviation threshold, it is confirmed as an audiovisual consistency boundary. Calculate the time deviation between the alignment timestamp in the candidate editing point set and the boundary timestamp of the audiovisual consistency boundary. Remove items with time deviations greater than a preset verification threshold and use the remaining items as the final editing points.

[0018] Preferably, when the processor executes the computer program to perform feature parsing, it is specifically used for:

[0019] Motion vectors between adjacent video frames are extracted to output a sequence of motion amplitudes in the image. Image features of video frames are extracted to output a sequence of brightness and information content parameters in the image, and the main subject category label and corresponding subject confidence score are output. The audio waveform is segmented into frames, and candidate beat points are output based on short-time energy peaks. Candidate points of spectral abrupt changes are output based on the spectral energy difference between adjacent audio frames. The audio loudness envelope sequence and audio energy change sequence are output, and audio energy stability parameters are output based on the ratio of short-time energy peak to mean. Subtitle segment time intervals are formed according to subtitle timestamps, and narrative unit intervals are determined when the difference in keyword sets between adjacent subtitle segments is greater than a preset difference threshold. The continuous intervals divided by the candidate points of adjacent scene transitions are denoted as scene block intervals.

[0020] Preferably, when the processor executes the computer program, it is also configured to perform a dynamic weight allocation function:

[0021] The subject confidence score and the audio energy stability parameter are normalized into visual confidence parameters and audio confidence parameters, respectively. When the absolute value of the difference between the visual confidence parameter and the audio confidence parameter is not greater than a first difference threshold, the visual confidence parameter and the audio confidence parameter are normalized to generate visual weights and audio weights, respectively. When the absolute value of the difference is greater than the first difference threshold but not greater than a second difference threshold, the visual weights and audio weights are set as the visual confidence parameter and the audio confidence parameter, respectively. When the absolute value of the difference is greater than the second difference threshold, the visual weights and audio weights are swapped. Wherein, the first difference threshold is less than the second difference threshold.

[0022] Preferably, when the processor executes the computer program, it is also used to perform an audio-visual verification linkage function:

[0023] Calculate the time difference between the local peak moments of the image motion amplitude sequence and the local peak moments of the audio energy change sequence. When the time difference is greater than a preset mismatch threshold, mark the audio-visual mismatch location. Calculate the correlation coefficient between the audio loudness envelope sequence and the image brightness sequence within a sliding time window. When the correlation coefficient is less than a preset negative correlation threshold, mark the corresponding time period as a reverse linkage segment, and confirm the tonal conflict intensity using the absolute value of the correlation coefficient. In response to the existence of the audio-visual mismatch location or the tonal conflict intensity being greater than a preset conflict threshold, output a conflict prompt icon in the editing interaction interface and control the cancellation of the transition overlay effect of the corresponding video segment.

[0024] Preferably, when the processor executes the computer program, it is also configured to perform a highlight fragment extraction function:

[0025] Video segments are formed between adjacent candidate clipping points. The image information quantity parameter and the loudness parameter of the audio loudness envelope sequence of the corresponding time period are obtained for each video segment. Based on the image information quantity parameter, the loudness parameter, and the corresponding visual weight and audio weight, a joint highlight score is calculated and generated. The preset quantile value of the global joint highlight score is determined as the global highlight threshold. The video segment with the highest joint highlight score and not lower than the global highlight threshold is selected and output within a preset fixed duration sliding time window.

[0026] Preferably, when the processor executes the computer program, it is further configured to:

[0027] Calculate the absolute value of the visual weight difference between adjacent video frames. When the difference exceeds a preset threshold, perform median filtering on the visual weights of the current frame and neighboring frames. Calculate the co-occurrence count of subtitle keywords and the main category label of the screen to form a correlation sequence. Confirm the semantic focus migration position based on the reverse change in the direction of increase or decrease of the correlation sequence and the change in the main category label.

[0028] Preferably, when the processor executes the computer program, it is further configured to:

[0029] Obtain the intersection length of the narrative unit interval and the scene block interval, and mark the part with an intersection length greater than a preset duration threshold as a text scene pairing relationship; form a keyword density sequence based on the ratio of the frequency of subtitle keywords in the video clip to the duration, mark video clips with a sliding mean continuously lower than a preset density threshold as bland clips, and reduce the selection weight of the bland clips in the editing and arrangement.

[0030] Preferably, an electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the functions of an intelligent video editing and processing system based on multimodal data fusion as described in any of the preceding claims.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] This invention performs feature analysis on the synchronization of video, audio, and subtitles, and determines the target editing point based on the time correspondence between candidate points of scene transitions, candidate points of audio beats, and candidate points of spectral changes, thus aligning the three types of data on the timeline. Previously, there were often time discrepancies between audio beats and scene transitions, making it difficult to align subtitle boundaries with scene boundaries, and lacking a unified basis for changes in the subject and audio energy. This invention, through three steps—basic alignment, phase compensation, and joint verification—locks in the scene transition points that match the beat and are confirmed by spectral cross-validation as the target editing point, making the editing point location more accurate than simply looking at the video or audio.

[0033] This invention dynamically allocates visual and audio weights based on the characteristics of each time period. After normalizing the subject confidence and audio energy stability parameters, each retains its weight when they are close, and swaps their weights when the difference is too large. This allows the visuals and audio to participate in different content based on actual feature changes, avoiding audio-visual imbalance and highlight omissions caused by fixed weights.

[0034] Audio-visual verification and linkage proactively identify conflicts during the editing stage: when the time difference between the peak of motion in the image and the peak of audio energy is too large, audio-visual discrepancy is marked; when the correlation coefficient between loudness envelope and image brightness is too low, tonal conflict is determined, and corresponding transition effects are prompted or canceled. Highlight extraction uses dynamic weighting scores combining image information content and loudness parameters, and selects the best within a preset fixed-duration sliding time window according to the global quantile threshold; text scene pairing marks the intersection of narrative units and scene blocks, and segments with consistently low keyword density are downgraded, in conjunction with a semantic focus shift state machine, to ensure narrative coherence and no loss of semantic focus in the final cut.

[0035] The system also renders clip boundaries, conflict prompts, and weight inversion markers on the timeline of the interactive interface, presenting the internal calculations to the user intuitively for easy previewing; the parsing of video, audio, and subtitles is scheduled in parallel to shorten processing latency, and the alignment threshold is adaptively tightened according to the input frame rate to reduce distortion of high frame rate materials. Attached Figure Description

[0036] Figure 1 This is an architecture diagram of an intelligent video editing and processing system based on multimodal data fusion.

[0037] Figure 2 This is the main processing flowchart of the present invention;

[0038] Figure 3 Sub-flowchart for determining the target editing point of this invention;

[0039] Figure 4 This is a flowchart of the feature parsing process of the present invention;

[0040] Figure 5 This is a diagram illustrating the dynamic weight allocation and audio-visual verification linkage of the present invention.

[0041] Figure 6 This is the semantic focus transition state machine diagram of the present invention. Detailed Implementation

[0042] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0043] Equation 1: Phase switching:

[0044] ;

[0045] In the formula, For the first The time for switching candidate points on each screen, in milliseconds; The time of the candidate point of the previous adjacent beat, in milliseconds; The beat period between adjacent beat candidate points, in milliseconds; For the first The switching phase of each candidate shot is dimensionless. This parameter technically represents the relative position of the shot cut point between two adjacent audio beats. The closer the value is to 0 or 1, the closer the shot cut is to the audio beat. The closer the value is to 0.5, the closer the shot cut is to the weak beat between the beats.

[0046] Equation 2: Audiovisual Consistency Boundary Time Deviation

[0047] ;

[0048] In the formula, Distance on the time axis The time of the most recent spectral mutation candidate point, in milliseconds. Indicates a sudden change in the spectrum; This represents the boundary time deviation, in milliseconds.

[0049] Equation 3 Joint verification time deviation:

[0050] ;

[0051] In the formula, For beat alignment in the record Each aligned timestamp, in milliseconds; For the audiovisual consistency boundary, the first Each boundary timestamp, in milliseconds; The time offset between the alignment timestamp and the boundary timestamp is measured in milliseconds.

[0052] Equation 4: Normalization of visual confidence parameters and audio confidence parameters:

[0053] ;

[0054] ;

[0055] In the formula, The subject confidence score output by the target detection. The extracted audio energy stabilization parameters are both dimensionless initial evaluation values ​​with values ​​greater than zero. These are the normalized visual confidence parameters; These are the normalized audio confidence parameters. Both satisfy... .

[0056] Equation 5: Generation of visual and audio weights:

[0057] When the absolute value of the difference When the difference is not greater than the first threshold: ;

[0058] When the absolute value of the difference When the difference is greater than the second threshold: .

[0059] In the formula, For the final visual weights, The final audio weights are all dimensionless; both satisfy the following conditions: .when When the difference is greater than the first difference threshold but not greater than the second difference threshold, the visual weight and audio weight are taken from the visual confidence parameter and the audio confidence parameter, respectively. The technical significance of this normalization operation is that it maps the heterogeneous features of visual and audio, which are two different dimensions, to a unified dimensional scale, providing a comparable computational basis for resolving weight conflicts in subsequent multimodal feature fusion.

[0060] Formula 6 Highlight Joint Scoring:

[0061] ;

[0062] In the formula, The normalized value of the image information content parameter. These are the normalized values ​​of the loudness parameters of the audio loudness envelope sequence, and both range from 0 to 1. The score is for highlighting and is dimensionless.

[0063] Equation 7 Global Spectrum Threshold:

[0064] ;

[0065] In the formula, This is a set of combined highlight scores for all candidate video clips; Preset quantile values; Global highlight threshold; For quantile calculation functions, in practical implementation, existing empirical cumulative distribution functions or linear interpolation methods can be used to obtain the corresponding values. The value of the quantile.

[0066] Formula 8 Keyword Density:

[0067] ;

[0068] In the formula, This represents the frequency of the keywords appearing in the subtitles of the current video clip, expressed in times. The duration of the video segment, in seconds (system preset). The minimum cutoff value is a non-zero minimum value to prevent overflow during division by zero calculation. Keyword density, in times per second.

[0069] Equation 9: Co-occurrence count

[0070] ;

[0071] In the formula, This is the set of keywords for the current subtitle segment; For the first The set of semantic words corresponding to the main category labels of each image; This is a dimensionless co-occurrence count, representing the number of keywords that overlap between the two.

[0072] Formula 10: Time difference between audio and video mismatch

[0073] ;

[0074] In the formula, The local peak moment of the motion amplitude sequence in the image, in milliseconds; This represents the local peak moment of the audio energy change sequence, in milliseconds. The time difference between audio and video mismatch is expressed in milliseconds.

[0075] Equation 11 Correlation coefficient:

[0076] ;

[0077] In the formula, It is an audio loudness envelope sequence; This is a sequence of screen brightness. The correlation coefficient calculation function is used. In practice, the Pearson correlation coefficient can be used to calculate the degree of linear correlation between two sequences. The correlation coefficient ranges from -1 to 1. The technical effect of this formula is that, by quantifying the synchronous trend between the fluctuation of sound loudness and the change of brightness in the picture, it provides underlying data support for the system to identify and eliminate tonal conflict segments that violate human visual and auditory psychological intuition, such as sudden increases in volume but extremely dark pictures.

[0078] The following symbol conventions are used uniformly in all accompanying figures: solid arrows indicate data flow or processing flow direction, dashed arrows indicate carrier relationship (processor and memory carrying each functional module), diamonds indicate decision nodes, rectangles indicate processing nodes, and rounded rectangles indicate start and end nodes.

[0079] Reference Figure 1 As shown, this invention includes a processor, a memory, and a display module. The memory stores a computer program; the display module is used to present an interactive editing interface, and the processor controls the display module to render segment boundary markers on the timeline of the interactive interface. When the processor executes the computer program, it implements four progressive functions: receiving the video to be edited and the corresponding audio and subtitle files; performing feature analysis on the video and audio to obtain candidate points for scene transitions, beats, and spectral changes; determining the target editing point based on the time correspondence between the candidate points for scene transitions and the candidate points for beats or spectral changes; arranging the video segments according to the target editing point to generate an edited video, and rendering and displaying the segment boundary markers corresponding to the target editing point on the timeline of the interactive editing interface. The processor and memory are connected to the feature analysis node by a dashed arrow, indicating the strong support and data flow control relationship of the hardware physical entity over all functional modules. The input subtitle file is received synchronously by the processor in the functional steps and participates in the extraction of subsequent narrative unit intervals.

[0080] Reference Figure 2 As shown, the processing flow, in chronological order, is as follows: receiving the video to be edited along with the corresponding audio and subtitle files; performing feature analysis on the video and audio; obtaining candidate points for scene transitions, beats, and spectral changes; determining whether there is a temporal correspondence between the candidate points for scene transitions and the candidate points for beats or spectral changes; determining the target editing point when a temporal correspondence exists; arranging video segments based on the target editing point to generate the edited video; rendering and displaying segment boundary markers on the timeline of the editing interface; and outputting the edited video for interactive preview.

[0081] Reference Figure 3As shown, the target cut point determination method includes two stages: basic alignment and phase compensation, followed by cross-verification of the results using joint verification. In the basic alignment stage, the processor calculates the time deviation between the candidate cut point and the nearest beat candidate point on the timeline. When the time deviation is not greater than a second preset deviation threshold, the candidate cut point is marked as a basic alignment point and a basic alignment relationship is established; otherwise, the nearest beat candidate point is determined within a preset search time window, and a basic alignment relationship is established using that beat candidate point. In the phase compensation stage, the processor divides the time difference between the candidate cut point and the previous adjacent beat candidate point by the beat period between adjacent beat candidate points to obtain the cut phase. Candidate cut points whose cut phases fall within a preset phase interval are locked as phase alignment points; for unlocked candidate cut points, their time offsets are allocated to the cut adjustments before and after that point, forming a beat alignment record after offset compensation. In the joint verification stage, the processor calculates the boundary time deviation between the candidate cut point and the nearest spectral change candidate point. When the boundary time deviation is not greater than a preset deviation threshold, it is confirmed as an audiovisual consistency boundary; otherwise, it is retained as a boundary to be confirmed. The processor calculates the time deviation between the alignment timestamp in the beat alignment record and the boundary timestamp of the audiovisual consistency boundary, removes inconsistencies with time deviations greater than a preset verification threshold, and uses the remaining items as the final editing points.

[0082] Reference Figure 4 As shown, to eliminate processing latency in the multimodal data alignment process and enhance the underlying computing power scheduling of the system, the processor drives the three branches of image parsing, audio parsing, and subtitle parsing in parallel through multi-threading or heterogeneous computing architecture (e.g., scheduling high-concurrency visual pixel calculations to the graphics processing unit (GPU) and scheduling audio waveform processing and subtitle text comparison to the central processing unit (CPU)). The image parsing unit statistically analyzes the motion vectors between adjacent video frames and outputs the image motion amplitude sequence; extracts image features from video frames and outputs the image brightness sequence and image information parameters; and outputs the image subject category label and the corresponding subject confidence score. The audio parsing unit segments the audio waveform, outputs beat candidate points based on short-time energy local peaks, outputs spectral change candidate points based on the total spectral energy difference between adjacent audio frames, outputs the audio loudness envelope sequence and audio energy change sequence, and outputs audio energy stability parameters based on the ratio of short-time energy peak to short-time energy mean. The subtitle parsing unit forms subtitle segment time intervals according to subtitle timestamps, extracts the keyword set of each subtitle segment, and determines the narrative unit interval when the difference in the keyword set of adjacent subtitle segments is greater than a preset difference threshold. Example 1 uses the following parameter values: the motion amplitude statistics are performed using a sliding window of 5 consecutive frames; the audio frame length is 2048 sampling points, and the frame shift is 512 sampling points; the preset difference threshold is 0.4. The preset difference threshold is determined based on the following: in common text similarity determination, a keyword set difference value greater than 0.4 usually corresponds to a scene or topic switch.

[0083] Reference Figure 5 As shown, the dynamic weight allocation unit normalizes the subject confidence score and audio energy stability parameter to obtain visual confidence parameters and audio confidence parameters. When the absolute value of the difference between the visual confidence parameter and the audio confidence parameter is not greater than the first difference threshold, the visual weight and audio weight are generated based on the normalization calculation of the visual confidence parameter and the audio confidence parameter, respectively. When the absolute value of the difference is greater than the second difference threshold, the visual weight and audio weight are swapped. To make this underlying "black box" logic intuitively decidable, when the processor triggers the weight swap operation, it simultaneously generates and renders a specific weight inversion status indicator (such as a confidence inversion warning icon or a specific color block highlight) on the corresponding video segment track of the editing interaction interface. This allows the abnormal circuit breaker dynamics of the internal calculation to be intuitively represented by the rendering results on the user-side interface, facilitating evidence collection.

[0084] The audio-visual verification and linkage unit calculates the time difference between the local peak moments of the image motion amplitude sequence and the local peak moments of the audio energy change sequence. When the time difference is greater than a preset misalignment threshold, the audio-visual misalignment location is marked. Within a sliding time window, the correlation coefficient between the audio loudness envelope sequence and the image brightness sequence is calculated. When the correlation coefficient is less than a preset negative correlation threshold, the corresponding time period is marked as a reverse linkage segment, and the intensity of tonal conflict is confirmed by the absolute value of the correlation coefficient. When there is an audio-visual misalignment location or the intensity of tonal conflict is greater than the preset conflict threshold, a conflict prompt is output on the editing interface, and the transition overlay effect of the corresponding video segment is canceled. Example 1 uses the following parameter values: the first difference threshold is 0.1, the second difference threshold is 0.5, the preset misalignment threshold is 150 milliseconds, the preset negative correlation threshold is -0.6, and the preset conflict threshold is 0.7. The preset misalignment threshold is chosen based on the fact that the human ear's sensitivity to audio-visual synchronization is usually between 80 and 200 milliseconds, and 150 milliseconds is in the middle of this range, which can simultaneously accommodate strict and general scenarios. The preset conflict threshold is determined based on the following: when the tonal conflict intensity is above 0.7, the audience will clearly perceive a discrepancy between the picture and the audio.

[0085] The highlight extraction and text scene pairing process follows claims 7 and 9. The highlight extraction module forms video segments between adjacent target editing points, acquiring the image information quantity parameter and the loudness parameter of the corresponding audio loudness envelope sequence for each video segment. The processor calculates a joint highlight score based on the image information quantity parameter, loudness parameter, and corresponding visual and audio weights. A preset quantile value of the global joint highlight score is determined as the global highlight threshold. Within a preset fixed-duration sliding time window, the video segment with the highest joint highlight score and not lower than the global highlight threshold is selected for output. The text scene pairing unit acquires the intersection length between the narrative unit interval and the scene block interval formed by the scene switching candidate points. The portion with an intersection length greater than a preset duration threshold is marked as a text scene pairing relationship. The flat segment processing unit forms a subtitle keyword density sequence based on the ratio of the frequency of subtitle keywords in the video segment to its duration. Video segments with a sliding mean continuously lower than a preset density threshold are marked as flat segments, and the selection weight of these flat segments in the editing process is reduced. Example 1 uses the following parameter values: preset quantile value is 0.75, isochronous long window is 10 seconds, preset duration threshold is 2 seconds, and preset density threshold is 0.2. The preset quantile value is determined based on the fact that highlight segments usually account for one-quarter of all segments, and a quantile value of 0.75 is sufficient to filter out the top one-quarter of high-resolution segments.

[0086] Reference Figure 6 As shown, the semantic focus transfer method is described in the form of a state machine: the process enters the focus stable state; when the correlation sequence difference is reversed or the subject category label changes, the state switches to focus transfer; in the focus transfer state, the process confirms the transfer position and updates the correlation sequence and subject category label; when the correlation sequence returns to the stable state, the state switches back to focus stable.

[0087] Example 1 (Baseline Static Parameter Scenario) uses the following parameter values: the second preset deviation threshold is 80 milliseconds, the preset search time window is ±200 milliseconds, the preset phase interval is 0.1 to 0.9, the preset deviation threshold is 100 milliseconds, and the preset verification threshold is 120 milliseconds. The values ​​of the second preset deviation threshold, preset search time window, and preset phase interval are based on: commonly used engineering experience values ​​for beat alignment accuracy. The corresponding screen switching time resolutions for adjacent frames at 25 frames per second and 30 frames per second are 40 milliseconds and 33 milliseconds, respectively. Setting the second preset deviation threshold to twice this resolution allows for minor acquisition jitter. The preset search time window of ±200 milliseconds covers the commonly used 4 to 8 beat tolerance range for manual editing. The preset phase interval of 0.1 to 0.9 avoids locking screen switching errors at the boundary of the beat cycle. The values ​​of the preset deviation threshold and preset verification threshold are based on: audiovisual consistency typically requires the time deviation between the image and audio to not exceed 100 milliseconds (within the acceptable range for the human eye). The preset verification threshold is slightly larger than this value to retain a small tolerance.

[0088] In Example 2 (Adaptive Dynamic Configuration Scenario), the second preset deviation threshold and the preset verification threshold are not fixed values, but are dynamically calculated and generated by the processor based on the global physical attributes of the input video during the initial stage of feature parsing. For example, the processor extracts the frame rate (FPS) of the original video. When a high frame rate (such as 60 frames per second) is detected, the system automatically reduces the second preset deviation threshold to half of the corresponding proportion in Example 1 (i.e., 40 milliseconds) to achieve higher precision adaptive timeline alignment under high frame rate material and prevent cropping distortion caused by static thresholds.

[0089] The specific implementation steps of this invention are as follows:

[0090] Step 1: Multi-path raw material access and heterogeneous computing power scheduling

[0091] The processor receives the original video file to be edited, the corresponding audio file, and the subtitle file in parallel through the system architecture. The input subtitle file is received synchronously by the processor in the functional steps and participates in the extraction of subsequent narrative unit intervals. To eliminate processing latency in the multimodal data alignment process and enhance the underlying computing power scheduling of the system, the processor starts a multi-threaded or heterogeneous computing architecture, scheduling high-concurrency visual pixel calculations to be executed by the graphics processor, while scheduling audio waveform processing and subtitle text comparison to be executed by the central processing unit, driving the subsequent multimodal feature parsing branch in parallel.

[0092] Step 2: Parallel parsing of multimodal features

[0093] The processor executes the three branches of image parsing, audio parsing, and subtitle parsing in parallel:

[0094] Image Analysis: The image analysis unit statistically analyzes the motion vectors between adjacent video frames, outputting a sequence of image motion amplitudes; extracts image features from video frames, outputting a sequence of image brightness and image information parameters; and outputs the image subject category label and its corresponding subject confidence score. The image motion amplitude statistics utilize a sliding window of five consecutive frames.

[0095] Audio analysis: The audio analysis unit divides the audio waveform into frames (frame length is 2048 sampling points, frame shift is 512 sampling points), outputs beat candidate points based on local peaks of short-time energy, outputs spectral change candidate points based on the difference in total spectral energy between adjacent audio frames, outputs audio loudness envelope sequence and audio energy change sequence, and outputs audio energy stability parameters based on the ratio of short-time energy peak to short-time energy mean.

[0096] Subtitle parsing: The subtitle parsing unit forms the time interval of subtitle segments according to the subtitle timestamp and extracts the keyword set of each subtitle segment; when the difference value of the keyword set of adjacent subtitle segments is greater than the preset difference threshold (taken as 0.4), the narrative unit interval is determined.

[0097] Step 3: Basic Alignment and Phase Compensation

[0098] The processor enters the target cut point determination process, first performing a basic alignment stage: calculating the time deviation between the candidate cut point and the nearest beat candidate point on the timeline; when the time deviation is not greater than a second preset deviation threshold (80 milliseconds as described in Example 1), the candidate cut point is marked as a basic alignment point and a basic alignment relationship is established; otherwise, the nearest beat candidate point is determined within a preset search time window (±200 milliseconds as described in Example 1), and a basic alignment relationship is established using this beat candidate point. Then, the processor enters a phase compensation stage: the processor divides the time difference between the candidate cut point and the previous adjacent beat candidate point by the beat period between adjacent beat candidate points to obtain the cut phase. When the cut phase is within a preset phase range (0.1 to 0.9 as described in Example 1), the candidate cut point is locked as a phase alignment point; for unlocked candidate cut points, their time offsets are allocated to the cut adjustment amounts before and after that point, forming a beat alignment record after offset compensation.

[0099] Step 4: Multimodal Joint Verification and Target Editing Point Locking. The processor continues the joint verification phase: calculating the boundary time difference between the candidate point of scene switching and the candidate point of the most recent spectral change. When the boundary time difference is not greater than the preset deviation threshold (100 milliseconds as described in Example 1), it is confirmed as an audiovisual consistency boundary; otherwise, it is retained as a boundary to be confirmed. Subsequently, the processor calculates the time deviation between the alignment timestamp in the beat alignment record and the boundary timestamp of the audiovisual consistency boundary, eliminates inconsistencies with time deviations greater than the preset verification threshold (120 milliseconds), and finally determines the remaining items as the target editing point.

[0100] Step 5: Linking Dynamic Weight Allocation with Front-End UI Determinability

[0101] The dynamic weight allocation unit normalizes the subject confidence score and audio energy stability parameter to obtain visual confidence parameters and audio confidence parameters. When the absolute value of the difference between the visual confidence parameter and the audio confidence parameter is not greater than the first difference threshold (0.1 as described in Example 1), the visual and audio weights are generated based on the normalization calculations of the visual and audio confidence parameters. When the absolute value of the difference is greater than the second difference threshold (0.5), the inversion mechanism is triggered, and the visual and audio weights are swapped. To make this underlying logic intuitively decidable, when the processor triggers the underlying calculation of the above weight swap operation, it simultaneously generates and renders a specific weight inversion status indicator (e.g., a confidence inversion warning icon or a specific color block highlight) on the corresponding video segment track of the editing interaction interface. This allows the abnormal circuit breaker dynamics of the internal calculation to be intuitively represented by the rendering results on the user-side interface, facilitating evidence collection.

[0102] Step 6: Audio-visual conflict detection and special effects linkage control

[0103] The audio-visual verification linkage unit calculates the time difference between the local peak moments of the image motion amplitude sequence and the local peak moments of the audio energy change sequence. When the time difference is greater than a preset misalignment threshold (150 milliseconds), the audio-visual misalignment location is marked. Within a sliding time window, the correlation coefficient between the audio loudness envelope sequence and the image brightness sequence is calculated; when the correlation coefficient is less than a preset negative correlation threshold (-0.6), the corresponding time period is marked as a reverse linkage segment, and the intensity of tonal conflict is confirmed by the absolute value of the correlation coefficient. When there is an audio-visual misalignment location or the intensity of tonal conflict is greater than a preset conflict threshold (0.7), a conflict warning indicator is output on the editing interface, and the transition overlay effect of the corresponding video segment is canceled.

[0104] Step 7: Highlight fragment extraction and text scene matching

[0105] The highlight extraction module forms video segments between adjacent target editing points, acquiring the image information parameters and the loudness parameters of the audio loudness envelope sequence for each video segment. The processor calculates a joint highlight score based on the image information parameters, loudness parameters, and corresponding visual and audio weights. Then, a preset percentile value (0.75) of the global joint highlight score is determined as the global highlight threshold. Within a preset fixed-duration sliding time window (10 seconds), the video segment with the highest joint highlight score that is not lower than the global highlight threshold is output. Simultaneously, the text scene pairing unit acquires the intersection length between the narrative unit interval and the scene block interval formed by the scene switching candidate points, marking the portion with an intersection length greater than a preset duration threshold (2 seconds) as a text scene pairing relationship.

[0106] Step 8: Processing of bland segments and control of semantic focus shift

[0107] The bland segment processing unit forms a keyword density sequence based on the ratio of the frequency of subtitle keywords to their duration in video segments. Video segments with a moving average continuously below a preset density threshold (0.2) are marked as bland segments, and their selection weight in editing is reduced. The semantic focus shifting method is described in the form of a state machine: the processor enters a focus stable state; when the correlation sequence difference is reversed or the subject category label changes, the state switches to focus shifting; in the focus shifting state, the processor confirms the shifting position and updates the correlation sequence and subject category label; when the correlation sequence returns to a stable state, the state switches back to focus stable.

[0108] Step 9: Video editing, layout, interactive rendering, and final output

[0109] Finally, based on the target cut point, joint score of highlights, weight reduction of flat segments, and semantic focus shift control, the processor performs comprehensive intelligent arrangement of video segments to generate an edited video. The processor then renders and displays the segment boundary markers corresponding to the target cut point on the timeline of the editing interface, and finally outputs the edited video for interactive preview.

[0110] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. An intelligent video editing and processing system based on multimodal data fusion, characterized in that, It includes a processor, a memory, and a display module. The memory stores a computer program, which, when executed by the processor, performs the following functions: Receive the video to be edited, as well as the corresponding audio file and subtitle file; The video and audio files to be edited are subjected to feature analysis to obtain candidate points for video scene transitions, and candidate points for beats and spectral changes in audio. Align the candidate points for scene switching with the candidate points for beat or the candidate points for spectral change on the time axis to obtain candidate clip points; The video clips are arranged and edited based on the candidate editing points to generate an edited video. The display module is then controlled to render and display the clip boundary markers corresponding to the candidate editing points on the timeline of the editing interface for interactive preview.

2. The intelligent video editing and processing system based on multimodal data fusion according to claim 1, characterized in that, When determining candidate clipping points, the processor is specifically used for: Calculate the time deviation between the candidate point of screen switching and the nearest beat candidate point on the timeline to establish a basic alignment relationship; divide the time difference between the candidate point of screen switching and the previous adjacent beat candidate point by the beat period between adjacent beat candidate points to obtain the normalized phase; lock the candidate points of screen switching with the normalized phase within the preset phase range as phase alignment points. The time offset of the unlocked scene transition candidate point is allocated to the editing adjustment amount before and after that point to form a beat alignment record after offset compensation, which is then included in the candidate editing point set.

3. The intelligent video editing and processing system based on multimodal data fusion according to claim 2, characterized in that, When the processor executes the computer program, it is also used to perform joint verification functions: Calculate the boundary time deviation between the candidate point for scene switching and the candidate point for the closest spectral change in time. When the boundary time deviation is not greater than a preset deviation threshold, it is confirmed as an audiovisual consistency boundary. Calculate the time deviation between the alignment timestamp in the candidate editing point set and the boundary timestamp of the audiovisual consistency boundary. Remove items with time deviations greater than a preset verification threshold and use the remaining items as the final editing points.

4. The intelligent video editing and processing system based on multimodal data fusion according to claim 1, characterized in that, When the processor executes the computer program to perform feature parsing, it is specifically used for: Motion vectors between adjacent video frames are extracted to output a sequence of motion amplitudes in the image. Image features of video frames are extracted to output a sequence of brightness and information content parameters in the image, and the main subject category label and corresponding subject confidence score are output. The audio waveform is segmented into frames, and candidate beat points are output based on short-time energy peaks. Candidate points of spectral abrupt changes are output based on the spectral energy difference between adjacent audio frames. The audio loudness envelope sequence and audio energy change sequence are output, and audio energy stability parameters are output based on the ratio of short-time energy peak to mean. Subtitle segment time intervals are formed according to subtitle timestamps, and narrative unit intervals are determined when the difference in keyword sets between adjacent subtitle segments is greater than a preset difference threshold. The continuous intervals divided by the candidate points of adjacent scene transitions are denoted as scene block intervals.

5. The intelligent video editing and processing system based on multimodal data fusion according to claim 4, characterized in that, When the processor executes the computer program, it is also used to perform a dynamic weight allocation function: The subject confidence score and the audio energy stability parameter are normalized into visual confidence parameters and audio confidence parameters, respectively. When the absolute value of the difference between the visual confidence parameter and the audio confidence parameter is not greater than a first difference threshold, the visual confidence parameter and the audio confidence parameter are normalized to generate visual weights and audio weights, respectively. When the absolute value of the difference is greater than the first difference threshold but not greater than a second difference threshold, the visual weights and audio weights are set as the visual confidence parameter and the audio confidence parameter, respectively. When the absolute value of the difference is greater than the second difference threshold, the visual weight and the audio weight are swapped; wherein the first difference threshold is less than the second difference threshold.

6. The intelligent video editing and processing system based on multimodal data fusion according to claim 4, characterized in that, When the processor executes the computer program, it is also used to perform the audio-visual verification linkage function: Calculate the time difference between the local peak moments of the image motion amplitude sequence and the local peak moments of the audio energy change sequence. When the time difference is greater than a preset mismatch threshold, mark the audio-visual mismatch location. Calculate the correlation coefficient between the audio loudness envelope sequence and the image brightness sequence within a sliding time window. When the correlation coefficient is less than a preset negative correlation threshold, mark the corresponding time period as a reverse linkage segment, and confirm the tonal conflict intensity using the absolute value of the correlation coefficient. In response to the existence of the audio-visual mismatch location or the tonal conflict intensity being greater than a preset conflict threshold, output a conflict prompt icon in the editing interaction interface and control the cancellation of the transition overlay effect of the corresponding video segment.

7. The intelligent video editing and processing system based on multimodal data fusion according to claim 5, characterized in that, When the processor executes the computer program, it is also used to perform a highlight fragment extraction function: Video segments are formed between adjacent candidate clipping points. The image information quantity parameter and the loudness parameter of the audio loudness envelope sequence of the corresponding time period are obtained for each video segment. Based on the image information quantity parameter, the loudness parameter, and the corresponding visual weight and audio weight, a joint highlight score is calculated and generated. The preset quantile value of the global joint highlight score is determined as the global highlight threshold. The video segment with the highest joint highlight score and not lower than the global highlight threshold is selected and output within a preset fixed duration sliding time window.

8. The intelligent video editing and processing system based on multimodal data fusion according to claim 5, characterized in that, When the processor executes the computer program, it is also used to: Calculate the absolute value of the visual weight difference between adjacent video frames. When the difference exceeds a preset threshold, perform median filtering on the visual weights of the current frame and neighboring frames. Calculate the co-occurrence count of subtitle keywords and the main category label of the screen to form a correlation sequence. Confirm the semantic focus migration position based on the reverse change in the direction of increase or decrease of the correlation sequence and the change in the main category label.

9. The intelligent video editing and processing system based on multimodal data fusion according to claim 4, characterized in that, When the processor executes the computer program, it is also used to: Obtain the intersection length of the narrative unit interval and the scene block interval, and mark the part with an intersection length greater than a preset duration threshold as a text scene pairing relationship; form a keyword density sequence based on the ratio of the frequency of subtitle keywords in the video clip to the duration, mark video clips with a sliding mean continuously lower than a preset density threshold as bland clips, and reduce the selection weight of the bland clips in the editing and arrangement.

10. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, it implements the functions of an intelligent video editing and processing system based on multimodal data fusion as described in any one of claims 1 to 9.

Images