Short video intelligent editing method and system based on multi-modal analysis

Abstract

The invention discloses a short video intelligent editing method and system based on multi-modal analysis, and relates to the technical field of video editing. The method is used for improving editing efficiency and visual experience and comprises the following steps: extracting lip motion features of a character, visual saliency features of a commodity and a voice emotion intensity value from a target short video stream to form multi-modal time sequence data; afterwards, the voice stream is recorded, a product keyword timestamp is extracted, the alignment degree is calculated through dynamic time warping in combination with a visual saliency peak value, and a preliminary editing point set is generated through weighted evaluation in combination with an emotional intensity value; constructing an editing decision optimization model based on deep reinforcement learning, taking the multi-modal features as state input, adjusting the retention probability of editing points through a joint reward function, and selecting an optimal transition mode; and the lip movement and voice synchronization error before and after the editing point and the emotional and visual continuity of the transition section are analyzed, the discontinuous region is smoothed, and the edited finished product is output, so that precise short video intelligent editing is realized.

Description

Technical Field

[0001] This invention relates to the field of video editing technology, specifically to a short video intelligent editing method and system based on multimodal analysis. Background Technology

[0002] With the rapid development of short video platforms, video has become an important medium for users to obtain information, express opinions, and make purchases. Whether it's e-commerce promotion, film and television entertainment, or knowledge dissemination and social interaction, short videos are penetrating daily life at a rapid pace. Users are demanding higher quality and personalization from video content, especially in product demonstrations and character portrayals. Viewers not only pay attention to the consistency of visuals and sound but also value overall viewing smoothness and emotional engagement. To meet these demands, intelligent video editing is gradually becoming an important direction for the industry.

[0003] Most existing video editing methods rely on single-modal feature extraction, such as using only speech recognition to determine editing points or relying solely on visual detection for scene segmentation. These methods suffer from several fundamental problems in practical applications: First, the inherent noise and instability of single-modal signals can easily lead to misalignment of editing points, resulting in audio and video desynchronization. Second, existing methods lack a holistic approach to handling emotional expression and visual attention continuity, often resulting in abrupt editing transitions, incomplete product displays, or fragmented facial expressions, ultimately leading to shortcomings in both coherence and visual appeal. These issues directly impact the user's viewing experience and the effectiveness of content dissemination in short videos. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a short video intelligent editing method and system based on multimodal analysis, which solves the problems mentioned above.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a short video intelligent editing method based on multimodal analysis, comprising the following steps: S1. Extracting multimodal features from the target short video stream, including extracting temporal data of lip movement features, product visual salience features, and voice emotional intensity values; S2. Transcripting the voice stream and extracting product keyword timestamps, while determining the peak time of product display based on the product visual salience feature sequence, calculating the alignment between the peak time of product visual salience and the keyword timestamps using a dynamic time warping algorithm, and when the alignment exceeds a preset threshold, performing a weighted evaluation based on the voice emotional intensity value at that time to generate a preliminary... S3. Construct a deep reinforcement learning-based editing decision optimization model, using extracted multimodal features as state input and the initial set of editing points as candidate action space. Product recognition accuracy and speech-video synchronization are used as the joint reward function. During training, the policy network determines whether to retain or discard editing points based on feedback from the joint reward function, and selects transition methods for retained editing points. S4. Analyze the synchronization error between the lip movement trajectory before and after the editing point and the speech flow. When the error exceeds a preset deviation value, position adjustment is performed. The rate of change in emotional intensity and visual attention continuity of the editing transition segment are evaluated. Detected discontinuous regions are smoothed, and the finished editing product is output.

[0006] Furthermore, the specific process of multimodal feature extraction for the target short video stream is as follows: The movement changes of the lip region are tracked frame by frame through facial keypoint detection; the shape changes and displacement vectors of the lip contour between adjacent frames are calculated to extract the temporal sequence of the lip movement trajectory; a visual saliency detection network based on an attention mechanism analyzes the visual attractiveness distribution of the product area in the video frames to extract the product visual saliency feature sequence; the audio stream is processed by frame segmentation, and the Mel-frequency cepstral coefficients and fundamental frequency features of each frame are extracted to calculate and extract continuous temporal data of the emotional intensity value.

[0007] Furthermore, the specific process of transcribing the speech stream and extracting product keyword timestamps, while determining the peak moment of product display based on the product visual salience feature sequence, is as follows: The audio stream is converted into a text sequence through speech recognition, named entity recognition is performed in conjunction with a pre-set product dictionary database, and keywords related to the product are selected. The start and end times of each keyword are located through the timestamp information of the speech signal. Sliding window analysis is performed on the product visual salience sequence, and the effective peak moment of product display is determined by finding local maxima points and combining them with duration thresholds.

[0008] Furthermore, the alignment between the peak visual saliency of the product and the keyword timestamp is calculated using a dynamic time warping algorithm. When the alignment exceeds a preset threshold, a weighted evaluation is performed based on the emotional intensity value of the speech at that moment. The specific process for generating a preliminary set of clipping points is as follows: The optimal matching path between the peak visual saliency sequence of the product and the keyword occurrence time sequence is calculated using a dynamic time warping algorithm. The alignment between the two sequences is measured by the cumulative distance of the path. When the alignment exceeds a preset threshold, a comprehensive score is calculated by combining the emotional intensity value of the speech at that time point and using a weighted fusion strategy. The time points that meet the conditions are selected based on the score ranking to generate a preliminary set of clipping points.

[0009] Furthermore, the construction logic of the editing decision optimization model based on deep reinforcement learning is as follows: A state representation mechanism is established, and the extracted lip movement feature sequences, product visual saliency sequences, and speech emotion intensity sequences are temporally aligned and feature-concatenated to form a multi-dimensional state vector; an action space is defined, with each moment point in the initial editing point set as an optional action, and each action is associated with a set of optional transition methods; a multi-objective reward function is designed, comprehensively considering the accuracy of product recognition in the image, the synchronization consistency between lip movements and speech flow, and the smooth transition of emotions before and after editing; a policy network is constructed through a policy gradient algorithm, and a multi-layer network structure consisting of a state feature encoding layer, an attention fusion layer, and a policy output layer is used to realize the mapping from multimodal temporal features to editing decisions.

[0010] Furthermore, during training, the policy network determines whether to retain or discard clips based on feedback from the joint reward function, and selects the transition method for the retained clips as follows: The policy network receives state input composed of multimodal features and outputs the retention probability distribution and corresponding transition method preference distribution for each candidate clip; it obtains feedback from the joint reward function through interaction with the environment, which comprehensively considers product recognition accuracy, speech-video synchronization, and emotional transition smoothness, and uses this feedback as an optimization signal to update network parameters through the policy gradient algorithm; during the network parameter update process, the policy network gradually learns to evaluate the comprehensive value of each candidate clip, dynamically adjusts the retention probability distribution of each clip, and selects the corresponding transition method for the determined retained clips based on the degree of multimodal feature matching between the segments before and after the point.

[0011] Furthermore, the synchronization error between the lip movement trajectory and the speech stream before and after the clipping point is analyzed. When the error exceeds the preset deviation value, the specific process of position adjustment is as follows: Calculate the cross-correlation function between the lip movement trajectory and the Mel frequency cepstral feature of the speech stream within a set time window before and after the clipping point. Calculate the synchronization error by finding the time delay value corresponding to the peak value of the cross-correlation function. When the time delay error exceeds the preset deviation value, use a binary search algorithm in the vicinity of the clipping point to find the position with the largest cross-correlation value, and use this position as the new clipping point after adjustment.

[0012] Furthermore, the specific process for smoothing detected discontinuous regions by evaluating the rate of change of emotional intensity and the continuity of visual attention in the editing transition segment is as follows: the first-order difference of the emotional intensity sequence is calculated through a sliding window, and abnormal abrupt changes in the difference value exceeding the allowable fluctuation range are detected; at the same time, the continuity of the distribution of the visual saliency sequence before and after the editing point is analyzed to identify the visual attention interruption region; for the detected discontinuous regions, visual transition frames are generated by optical flow based on the consistency of motion vectors between adjacent frames, and the emotional intensity sequence is adaptively smoothed and filtered.

[0013] A short video intelligent editing system based on multimodal analysis includes the following modules: a multimodal feature extraction module, an editing point generation module, a decision optimization module, and a post-processing module. The multimodal feature extraction module extracts multimodal features from the target short video stream, including extracting temporal data of lip movement features, product visual saliency features, and voice emotional intensity values. The editing point generation module transcribes the voice stream and extracts product keyword timestamps. Simultaneously, it determines the peak time of product display based on the product visual saliency feature sequence, calculates the alignment between the peak time of product visual saliency and the keyword timestamps using a dynamic time warping algorithm, and when the alignment exceeds a preset threshold, it performs a weighted evaluation based on the voice emotional intensity value at that moment. The system generates a preliminary set of editing points. The decision optimization module is used to construct an editing decision optimization model based on deep reinforcement learning. It takes the extracted multimodal features as state input, the preliminary set of editing points as candidate action space, and the product recognition accuracy and speech-video synchronization as joint reward functions. During training, the policy network decides whether to retain or discard editing points based on the feedback of the joint reward function, and selects the optimal transition method for the retained editing points. The post-processing module is used to analyze the synchronization error between the lip movement trajectory before and after the editing point and the speech flow. When the error exceeds a predetermined threshold, the position is adjusted. The module evaluates the rate of change of emotional intensity and the continuity of visual attention in the editing transition segment, smooths the detected discontinuous areas, and outputs the finished editing product.

[0014] The present invention has the following beneficial effects: (1) A short video intelligent editing method based on multimodal analysis, which extracts multimodal features such as lip movement features, visual saliency features of products, and emotional intensity values ​​of speech, can capture key information in the video in all aspects, no longer limited to single-modal signal analysis, thereby effectively improving the accuracy and robustness of editing point selection. By combining speech transcription results with the peak time of product display, and using dynamic time warping algorithm to achieve alignment and weighted evaluation of multimodal information, a preliminary set of editing points that is more in line with the content logic and user focus can be generated, significantly reducing the problems of asynchronous video and speech and incomplete product display.

[0015] (2) A short video intelligent editing system based on multimodal analysis constructs an editing decision optimization model based on deep reinforcement learning, which can dynamically adjust the retention and rejection strategies of editing points during multiple rounds of training, and select appropriate transition methods for the retained editing points, so that the finished product can achieve a balance between product recognition accuracy and speech-image synchronization. By analyzing the synchronization error between lip movement trajectory and speech flow and evaluating the continuity of transition segments, the system achieves fine-grained correction and smoothing of editing points, thereby ensuring the consistency of video in terms of emotional expression, visual attention and viewing smoothness, and improving the overall viewing experience.

[0016] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0017] Figure 1 This is a flowchart of a short video intelligent editing method based on multimodal analysis according to the present invention.

[0018] Figure 2 This is a flowchart of a short video intelligent editing system based on multimodal analysis according to the present invention. Detailed Implementation

[0019] This application provides a method and system for intelligent short video editing based on multimodal analysis, which solves the problems of insufficient utilization of multimodal information, inaccurate selection of editing points, and reliance on human experience in the quality of finished products in existing short video editing technologies.

[0020] The overall concept of the solution in this application embodiment is as follows: First, multimodal feature extraction is performed on the target short video stream, including extracting the lip movement features of the person, the visual salience features of the product, and the temporal data of the emotional intensity value of the voice, so as to achieve comprehensive capture of key elements in the video content.

[0021] Subsequently, the audio stream is transcribed into text and product keyword timestamps are extracted. At the same time, the peak time of product display is determined by analyzing the product visual saliency feature sequence. The dynamic time warping algorithm is used to match and weight the two to generate a preliminary set of clipping points.

[0022] Furthermore, a deep reinforcement learning-based editing decision optimization model is constructed, which takes the aforementioned multimodal features as state input and the set of editing points as candidate action space. Under the guidance of the joint reward function, the model dynamically optimizes the retention and transition methods of editing points.

[0023] Finally, cross-modal consistency correction is performed on the optimized editing point sequence. By detecting the synchronization error between lip movements and speech flow, as well as the emotional and visual continuity of transition segments, fine adjustments are made to the editing point positions and transition methods, ultimately outputting a finished film that meets visual coherence and emotional naturalness.

[0024] Please see Figure 1 This invention provides a technical solution: a short video intelligent editing method based on multimodal analysis, comprising the following steps: S1. Extracting multimodal features from the target short video stream, including extracting temporal data of lip movement features, product visual salience features, and voice emotional intensity values; S2. Transcripting the voice stream and extracting product keyword timestamps, while determining the peak time of product display based on the product visual salience feature sequence, calculating the alignment between the peak time of product visual salience and the keyword timestamps using a dynamic time warping algorithm, and when the alignment exceeds a preset threshold, performing a weighted evaluation based on the voice emotional intensity value at that time to generate a preliminary editing point set. S3. Construct a deep reinforcement learning-based editing decision optimization model, using extracted multimodal features as state input, a preliminary set of editing points as candidate action space, and product recognition accuracy and speech-video synchronization as joint reward functions. During training, the policy network decides whether to retain or discard editing points based on feedback from the joint reward function, and selects transition methods for retained editing points; S4. Analyze the synchronization error between the lip movement trajectory before and after the editing point and the speech flow. When the error exceeds a preset deviation value, adjust the position, evaluate the rate of change of emotional intensity and visual attention continuity of the editing transition segment, smooth the detected discontinuous areas, and output the finished editing product.

[0025] In this implementation plan, S1. Simultaneously extract three key information categories from the original short video: "speech synchronization," "product visualization," and "emotional expression intensity," serving as the foundational data for subsequent multimodal alignment, candidate clipping point generation, and decision optimization. Face detection and lip region localization are performed on the video frame sequence. Lip key points or contours are tracked frame by frame to construct a temporal signal of lip movement (used to determine the correspondence between lip shape changes and speech). Visual saliency or object detection models are used to locate and score product regions in the frame, generating a sequence of product visual saliency changes over time (used to measure the visual prominence of products at each moment). The audio stream is framed and preprocessed for speech, then input into the emotion recognition module to extract the speech emotion intensity value at each moment (used to characterize the emotional fluctuations in speech expression). Lip movement features: These refer to temporal signals obtained through facial key points or pixel-level contour difference that reflect lip opening and closing and deformation; they can be used to determine the synchronization between speech and lip shape. Product visual saliency features: These are numerical sequences reflecting the degree to which a product is visually focused on in an image, typically obtained through saliency detection or object detection weighted by attention. Voice emotion intensity value: This is a quantitative indicator representing the strength (e.g., passionate, calm) of a speaker's emotional state at a given moment, often output from acoustic features via an emotion classification / regression model. S2. Establish a temporal correspondence between the semantic and visual layers, identifying candidate moments that both semantically mention the product and visually highlight it, serving as the source of initial editing points. Transcribe the speech stream into text using Automatic Speech Recognition (ASR), identify product-related keywords in the text, and record the timestamps of these keywords in the audio (keyword timestamps). Perform local maxima detection (peak detection) on the product visual saliency sequence, and determine the peak moment of product display (visual peak moment) based on rules such as saliency persistence. Input the keyword timestamp sequence and the visual peak moment sequence into a time alignment algorithm such as Dynamic Time Warping (DTW) to calculate the alignment degree between the two sequences. If the alignment exceeds a preset threshold, it indicates that the moment is salient both semantically and visually. Further weighted evaluation is performed using the corresponding emotional intensity value of the speech at that moment to improve the confidence of the candidate points, ultimately culminating in a preliminary set of clipping points. Keyword timestamp: The start and end time position of a keyword in the ASR output text within the audio stream. Product display peak moment: The time point or time period in the product visual saliency sequence that is determined to be "visually prominent" after peak detection and persistence judgment. Dynamic Time Warping (DTW): An algorithm for comparing the similarity of two time series, allowing non-linear scaling and matching of the sequences on the time axis to find the optimal alignment path (facilitating the handling of asynchronous speech rates or camera rhythms). The lip movement, saliency, and emotional sequences extracted by S1 provide input for peak detection and weighted evaluation in S2; the preliminary set of clipping points output by S2 also carries multimodal contextual information, serving as the basis for candidate actions in subsequent decision-making modules.S3. Based on the multimodal context, the "retain / discard" decision and corresponding transition methods are automatically learned and optimized on the candidate clipping point set to balance multiple objectives such as product prominence, speech-image synchronization, and visual coherence. State Representation: The lip movement sequence, product saliency sequence, and speech emotion sequence extracted in S1 are temporally aligned and feature-concatenated to form a multidimensional state vector describing the current candidate point and its preceding and following contexts. Action Space: Each candidate clipping point is used as a basic action unit. Actions include the "retain / discard" decision for that point and several transition methods (such as direct switching, cross-fade-in / fade-out, smooth visual transition, etc.) if retained. Joint Reward Function: Designed as a comprehensive evaluation signal, balancing product recognition accuracy (ensuring product information can be recognized and presented), speech-image synchronization (ensuring lip movements correspond to voice), and emotional continuity (ensuring natural emotional transitions). This reward function serves as the optimization objective of reinforcement learning, guiding the policy network to learn priority clipping strategies in different multimodal contexts. Learning Mechanism: A policy network is trained using policy gradient methods (such as proximal policy optimization), allowing it to gradually update its parameters through interaction with the environment (simulating editing and evaluating rewards), learning to give a probability distribution for high-reward decisions. Deep Reinforcement Learning: A machine learning method that enables an agent to learn decision-making policies through interaction with the environment and based on reward signals, suitable for solving sequential decision-making and long-term reward maximization problems. Policy Network: A neural network model that outputs the probability distribution of actions, used in this invention to map multimodal states to editing decisions (preservation probabilities and transition preferences). Candidate Action Space: The set of all actions available at a given moment or candidate point (including whether to retain and optional transitions). S3 uses the candidate points provided by S2 and the multimodal states of S1 as input. The policy obtained through training can directly output the optimized editing point sequence, providing a test scheme for consistency verification in S4. S4. Fine-grained verification and local correction are performed on the optimized editing point sequence output by S3 to ensure that the final cut meets the quality requirements in terms of speech-lip-sync, visual attention continuity, and emotional expression smoothness. Synchronization Error Analysis: The timing of lip movements in S1 is aligned frame-by-frame / segment-by-segment with the speech time series to calculate the synchronization error between lip movements and audio before and after the clipping point. When this error exceeds a preset synchronization threshold, clipping point fine-tuning is triggered (e.g., moving the clipping point forward or backward to a more suitable time window). Emotional and Visual Continuity Assessment: The rate of change in speech emotional intensity within the transition segment is calculated (to determine if there are abrupt changes in emotion) and the continuity of visual attention distribution is assessed (e.g., by using saliency maps or the consistency of product area coverage to determine if attention is interrupted). If discontinuities or abrupt changes are found, smoothing processing is triggered.Smoothing Strategies: Common processing methods such as content similarity matching (replacing / extending transition segments with adjacent frames or similar shots), frame interpolation, or crossfading are employed, or transition parameters are adjusted to mitigate abruptness, thereby achieving a smooth visual and emotional transition without compromising semantic expression. Synchronization Error: This refers to the offset between lip movements and speech signals on the time axis, used to quantify whether lip movements and vocalizations are aligned. Emotional Intensity Change Rate: This represents the rate at which emotional intensity changes over time; an excessively large change rate indicates abrupt emotional shifts, potentially leading to abrupt transitions. Visual Attention Continuity: This refers to whether the target of the viewer's visual attention (such as a product) maintains a high degree of continuity between adjacent shots or transition segments, often measured using saliency maps or target area overlap. S4 uses the output of S3 as input and calls upon the multimodal data from S1 for verification and fine-tuning. After verification, the final output is completed, forming a closed-loop workflow of "extraction → candidate → optimization → verification → output".

[0026] Specifically, the process of multimodal feature extraction for the target short video stream is as follows: The movement of the lip region is tracked frame-by-frame through facial keypoint detection; the shape change and displacement vector of the lip contour between adjacent frames are calculated to extract the temporal sequence of the lip movement trajectory; a visual saliency detection network based on an attention mechanism analyzes the visual attractiveness distribution of the product area in the video frames to extract the product visual saliency feature sequence; the audio stream is processed in frames, and the Mel-frequency cepstral coefficients and fundamental frequency features of each frame are extracted to calculate and extract continuous temporal data of the emotional intensity value.

[0027] In this implementation plan, the specific process of multimodal feature extraction is as follows: Shoulder motion trajectory extraction: First, the key points of the lip region are located frame by frame using a facial key point detection algorithm to form a lip contour point set. To quantify the changes in lip shape between adjacent frames, the following formula can be used: ; Parameter description: Video frames The lip contour vector is composed of the coordinates of the detected key points; The variation in lip contour between two adjacent frames indicates the intensity of lip movement. : in frames A local set of frames centered on the local average level is used to calculate the local average level. Regularization constant, used to prevent the denominator from being too small; Normalized lip movement indices are used as feature values ​​for the temporal sequence of lip movement trajectories. Through the above calculations, a time-varying lip movement trajectory sequence can be obtained for subsequent speech synchronization analysis. Product visual saliency feature extraction utilizes an attention-based visual saliency detection network to quantify the visual attractiveness of product regions in video frames, obtaining frame-by-frame saliency scores. To stably extract peak moments, smoothing and peak determination methods are introduced. A significant peak moment is determined when the following conditions are met: Parameter description: The original score output by the saliency detection network; Smoothing window radius; Smooth kernel weights, satisfying ; : The significance score after smoothing; : Range of local extrema comparison; Significance threshold; Determination method: threshold The detection rate and false alarm rate can be tested at different significance levels using a manually labeled training sample set, and the optimal inflection point can be selected using the ROC curve. Audio sentiment intensity extraction involves segmenting the audio stream into frames, calculating the Mel-frequency cepstral coefficients and fundamental frequency features for each frame, and concatenating these two values ​​as the input feature vector. The emotional intensity score is obtained by a linear mapping and a nonlinear function. Parameter description: Audio frames Mel frequency cepstral coefficients; Audio frames The fundamental frequency characteristics; : The concatenated audio feature vector; Sentiment classification weight vectors are obtained through training with labeled corpora. : Bias term; A non-linear function (Sigmoid) used to map the output to a fixed interval; The emotional intensity value is used to form a temporal sequence of emotions that changes over time. In summary, mouth movement features are used to obtain temporal indicators through keypoint tracking and normalization calculations; product visual saliency is obtained through an attention mechanism network and peak determination formula to obtain a temporal sequence and peak time; audio features are concatenated with the fundamental frequency using MFCC and then output as emotional intensity values ​​by a classification model. These steps achieve multimodal feature extraction from short videos, providing input for subsequent semantic alignment, reinforcement learning decisions, and smoothing processing.

[0028] Specifically, the process of transcribing the audio stream and extracting product keyword timestamps, while determining the peak time of product display based on the product visual salience feature sequence, is as follows: The audio stream is converted into a text sequence through speech recognition, and named entity recognition is performed in conjunction with a pre-set product dictionary database to filter out product-related keywords. The start and end times of each keyword are located through the timestamp information of the audio signal. Sliding window analysis is performed on the product visual salience sequence, and the effective peak time of product display is determined by finding local maxima points and combining them with duration thresholds.

[0029] In this implementation scheme, the extraction of speech keyword timestamps begins by transcribing the input speech stream to obtain a text sequence. Then, using a pre-set product dictionary database, named entity recognition is employed to filter out product-related keywords. For the r-th detected keyword, its corresponding start and end timestamps can be represented as: Parameter description: : The start timestamp of the r-th keyword; : The end timestamp of the r-th keyword; The start time value output by the speech recognition system; The end time value output by the speech recognition system; Keyword indexing. Through the above calculations, the time intervals of all product keywords in the audio stream can be obtained. Product visual saliency peak detection involves smoothing the product visual saliency feature sequence using a sliding window. Let the original saliency sequence be... The smoothed sequence is: A product display peak is considered valid when the following conditions are met at a given moment: Parameter description: : The original significance score of the product at time t; : The significance score after smoothing; : Sliding window radius; : Smoothing weighting coefficients, satisfying ; : Range of local extrema comparison; The duration of significance at peak time t; Duration threshold. The determination method is as follows: the detection rate and false alarm rate can be statistically analyzed at different durations using manually labeled training samples, and the inflection point of the PR curve (precision-recall curve) can be used as the optimal threshold.

[0030] Specifically, the alignment between the peak visual saliency of a product and the keyword timestamp is calculated using a dynamic time warping algorithm. When the alignment exceeds a preset threshold, a weighted evaluation is performed based on the emotional intensity value of the speech at that moment. The process for generating a preliminary set of clipping points is as follows: The optimal matching path between the peak visual saliency sequence of a product and the occurrence time sequence of the keyword is calculated using a dynamic time warping algorithm. The alignment between the two sequences is measured by the cumulative distance of the path. When the alignment exceeds a preset threshold, the emotional intensity value of the speech at that time point is further combined with a weighted fusion strategy to calculate a comprehensive score. Based on the score ranking, the time points that meet the conditions are selected to generate a preliminary set of clipping points.

[0031] In this implementation scheme, the sequence input and feature preparation first involve recording the sequence of peak visual salience times of the product as follows: Each element represents a peak moment on the video timeline; the timestamp sequence of voice keywords is denoted as... Each element represents the time position corresponding to the keyword. Dynamic Time Warping (DTW) path calculation uses the DTW algorithm to calculate the optimal matching path between two time series. Its core recursive formula is: ;in, : Indicates the significance peak. The element and keyword timestamp The cumulative distance of each element on the matching path; : Indicates time and Time difference measurement; : These are the lengths of the two sequences, respectively. The final optimal matching path length is denoted as . Alignment can be calculated using normalized cumulative distance: ;in, Alignment metrics; : The optimal cumulative distance obtained by dynamic time warping; : The length factor used for normalization. When At this point, it is determined that the two sequences have a valid alignment relationship. Threshold The determination method is as follows: F1 scores at different thresholds are calculated on the validation set, and the threshold that maximizes the F1 score is selected as the final decision threshold. Speech emotion-weighted fusion is performed at the alignment decision point. Further introduce voice emotion intensity value This results in a weighted composite score: ;in, :time Overall score; Alignment metrics; : The emotional intensity value of the voice at the corresponding moment; Weighting coefficients, satisfying Weighting coefficient determination method: A grid search is performed on the training set to select the combination of clip points with the highest weighted average recall and precision. A clip point set is generated, and a comprehensive score is given for all candidate time points. Sort the data and select those with scores exceeding a set threshold. At that moment, a preliminary set of cut points is formed. Threshold The determination method is as follows: by statistically analyzing the detection rate and false alarm rate under different scoring thresholds in the labeled data, the threshold with the largest Youden index is selected as the optimal value using the ROC curve.

[0032] Specifically, the construction logic of the editing decision optimization model based on deep reinforcement learning is as follows: A state representation mechanism is established, which temporally aligns and concatenates the extracted lip movement feature sequences, product visual saliency sequences, and speech emotion intensity sequences to form a multi-dimensional state vector; an action space is defined, where each moment point in the initial editing point set is treated as an optional action, and a set of optional transition methods is associated with each action; a multi-objective reward function is designed, comprehensively considering the accuracy of product recognition in the image, the synchronization consistency between lip movements and the speech flow, and the smooth transition of emotions before and after editing; a policy network is constructed using a policy gradient algorithm, and through a multi-layer network structure consisting of a state feature encoding layer, an attention fusion layer, and a policy output layer, the mapping from multimodal temporal features to editing decisions is realized.

[0033] In this implementation scheme, the state representation mechanism performs time alignment and feature concatenation on the lip movement feature sequence, product visual saliency sequence, and voice emotion intensity sequence obtained from the aforementioned multimodal feature extraction to form a multidimensional state vector: ;in, :time The state vector; : Lip movement feature vector; Visual saliency feature value; : Voice emotion intensity value; Feature concatenation operation. Explanation: This state vector represents the state at time [time]. The video multimodal information is used as input for policy network decision-making. The action space definition defines the initial set of clipping points. Each point in time is treated as an optional action, and each action is associated with a set of optional transitions. Note: Action space It includes options for "whether to retain the clip point" and "which transition method to choose," providing actionable decision-making options for reinforcement learning. A multi-objective reward function is designed, along with a joint reward function. To evaluate the quality of each action: ;in, Product identification accuracy index; : A measure of the synchronization consistency between lip movements and speech flow; : An indicator of the smooth transition of emotions before and after editing; : Reward weighting coefficient, satisfying Method: The overall quality evaluation of the edit (human or automatic metrics) is maximized by adjusting the weight combination on the training set. Explanation: This reward function simultaneously considers content comprehensibility, speech synchronization, and emotional continuity, providing an optimization objective for the policy network. Policy network construction and training employ the policy gradient algorithm to construct the policy network. Its structure includes: a state feature encoding layer: for the state vector Linear mapping and nonlinear activation are performed to generate a high-dimensional representation; Attention fusion layer: Utilizes an attention mechanism to dynamically weight features of each modality, highlighting key modal information; Policy output layer: Outputs the retention probability of each candidate clipping point and the corresponding transition selection probability distribution. Core policy gradient update formula: ;in, : Policy network parameter vector; The learning rate can be determined through grid search or adaptive optimization methods. At any moment Actions taken (edit point preservation and transition selection); Reward value; Policy network in state Select action The probability. Explanation: By iteratively updating the network parameters through policy gradients, the network gradually tends to choose editing strategies that yield higher cumulative rewards, thereby achieving automated, high-quality editing decisions.

[0034] Specifically, during training, the policy network determines whether to retain or discard clips based on feedback from the joint reward function, and selects the transition method for the retained clips as follows: The policy network receives state input composed of multimodal features and outputs the retention probability distribution and corresponding transition method preference distribution for each candidate clip. It obtains feedback from the joint reward function through interaction with the environment. This feedback comprehensively considers the accuracy of product recognition, the synchronization between speech and video, and the smoothness of emotional transition, and serves as an optimization signal to update the network parameters through the policy gradient algorithm. During the network parameter update process, the policy network gradually learns to evaluate the comprehensive value of each candidate clip, dynamically adjusts the retention probability distribution of each clip, and selects the corresponding transition method for the determined retained clips based on the degree of multimodal feature matching between the segments before and after the point.

[0035] In this implementation scheme, the state input and policy output are discussed. During the training process, the policy network receives the state input, which is composed of multimodal features. Output the retention probability distribution for each candidate clipping point. and corresponding transition method preference distribution : ; Parameter description: :time The state vector is obtained by concatenating lip movement features, visual saliency features, and speech emotion intensity features; : The probability distribution of retaining candidate clipping points; : Probability distribution of transition selection corresponding to candidate cut points; The policy network output function includes linear mappings and nonlinear activation layers; The network output is mapped to a probability distribution, ensuring the sum is 1. This step maps multimodal states to action probabilities, allowing the policy network to dynamically evaluate the retention priority and transition selection preference for each clip point based on the current video content. A joint reward function is fed back; during interaction with the environment, the network obtains a joint reward function. Feedback is collected to comprehensively evaluate the effectiveness of each action: Parameter description: Product identification accuracy index; :Synchronous measurement of lip movements and speech flow; The smoothness index of emotional transition before and after editing; : Reward weighting coefficient, satisfying Determination Method: By adjusting different weight combinations in the training set to optimize manually labeled or automatically evaluated metrics, corresponding weight values ​​are selected. The reward function serves as a signal for network optimization, guiding the policy network to prioritize editing decisions that simultaneously consider content, synchronization, and emotional continuity. The policy network parameter update network updates parameters using a policy gradient algorithm. Perform iterative updates to maximize the expected cumulative reward: Parameter description: : Policy network parameter vector; The learning rate can be determined through grid search or adaptive optimization methods. At any moment The actions taken include preserving cut points and selecting transitions; Policy network in state Select action The probability of; : Reward function value. This step iteratively optimizes the policy network through gradient ascent, gradually causing the network to favor editing points and transitions that yield high cumulative rewards. The policy network dynamically adjusts the retention probability distribution of each candidate editing point during training, considering both the comprehensive value evaluation of editing points and transition selection. For the selected cut points, the optimal transition method is chosen based on the degree of multimodal feature matching between the preceding and following segments. Through the above mechanism, the strategy network can achieve automated and dynamically optimized editing decisions while ensuring content coherence and viewing experience.

[0036] Specifically, the process of analyzing the synchronization error between the lip movement trajectory and the speech stream before and after the cut point, and adjusting the position when the error exceeds the preset deviation value, is as follows: Calculate the cross-correlation function between the lip movement trajectory and the Mel frequency cepstral feature of the speech stream within a set time window before and after the cut point, and calculate the synchronization error by finding the time delay value corresponding to the peak value of the cross-correlation function; when the time delay error exceeds the preset deviation value, use a binary search algorithm in the vicinity of the cut point to find the position with the largest cross-correlation value, and use this position as the new cut point after adjustment.

[0037] In this implementation plan, the synchronization error between the lip movement trajectory and the speech stream is calculated for each initial editing point. Set time windows before and after it. Internally, extract the lip movement trajectory sequence. Mel frequency cepstral coefficient sequence of the corresponding audio frame Calculate the cross-correlation function : Parameter description: : Feature sequence of the inner lip movement trajectory within the clipping point window; : MFCC feature sequence of audio frames within the clip point window; Time delay, used to calculate the correlation between lip movements and audio. Cross-correlation function value, used to measure the time delay between lip movements and audio signals. The degree of matching below; The selected time window length before and after the clipping point is used to calculate local synchronization. Explanation: The synchronization error between lip movement and the speech signal can be quantified using the cross-correlation function. Synchronization error judgment and clipping point adjustment are based on the optimal delay corresponding to the peak value of the cross-correlation function. : The synchronization error is defined as follows: and the preset deviation value Comparison: Parameter description: The time delay corresponding to the peak value of the cross-correlation function; The maximum allowable synchronization deviation value; based on the distribution of lip-movement speech synchronization errors in the labeled samples, the maximum deviation that ensures over 95% of the clipping points are synchronized between vision and speech is selected as the threshold. If the synchronization error exceeds... A binary search algorithm is used in the vicinity of the clipping point to find the location with the highest cross-correlation value. And update the clipping point to the adjusted position: Parameter description: The maximum cross-correlation position delay obtained by binary search; Adjusted editing point position. Explanation: By following the steps above, the offset of the editing point can be dynamically corrected, ensuring optimal synchronization between the lip movement trajectory and the speech flow, thus improving the naturalness of the editing transition and the viewing experience.

[0038] Specifically, the process of evaluating the rate of change of emotional intensity and the continuity of visual attention in the editing transition segment, and smoothing the detected discontinuous regions is as follows: the first-order difference of the emotional intensity sequence is calculated through a sliding window, and abnormal abrupt changes in the difference value exceeding the allowable fluctuation range are detected; at the same time, the continuity of the distribution of the visual saliency sequence before and after the editing point is analyzed to identify the visual attention interruption region; for the detected discontinuous regions, visual transition frames are generated by optical flow based on the consistency of motion vectors between adjacent frames, and the emotional intensity sequence is adaptively smoothed and filtered.

[0039] In this implementation plan, the assessment of the rate of change in emotional intensity is performed within the time window before and after the editing point. Within the emotional intensity sequence Calculate the first-order difference to reflect the rate of change in sentiment: Parameter description: : A sequence of sentiment intensity values ​​from frames adjacent to the clipping point; First-order difference of emotional intensity, used to detect anomalous abrupt changes; : The length of the time window, used for local assessment of sentiment changes; when At that time, it was determined to be a point of abnormal emotional change. Parameter description: The maximum allowable amplitude variation in emotional intensity; determination method: by statistically analyzing the distribution of emotional change amplitude during normal clip transitions in the labeled training samples, a value that covers the vast majority of normal transitions is selected as the threshold. Visual attention continuity analysis is performed on the visual saliency sequence before and after the clipping point. Perform continuity analysis to detect regions of visual attention interruption. This includes calculating the saliency difference between adjacent frames. ;like And consistently exceeds the minimum frame rate This is identified as a region of visual attention interruption. Parameter description: Visual saliency of frames adjacent to the clipping point; Visual difference between adjacent frames; : Visual continuity determination threshold; Minimum number of continuous frames is used to distinguish between brief jitter and genuine interruptions. Threshold determination method: By statistically analyzing the range of continuous fluctuations in labeled samples, a threshold suitable for covering most normal continuous frames is selected. Smoothing processing: For detected discontinuous regions, the following processing is performed: Visual smoothing: Based on the consistency of motion vectors between adjacent frames, an interpolated transition frame is generated using optical flow. : Parameter description: : Original video frame pixel values; : Start and end frames of discontinuous regions; : Generated smooth visual transition frames; Emotional smoothing: Smoothing of emotional intensity sequences Perform adaptive filtering: Parameter description: : Smoothed sentiment intensity value; Adaptive smoothing coefficient, automatically adjusted according to the amplitude of emotional fluctuations; the greater the fluctuation, the smoother the smoothing effect. The smaller the value, the stronger the smoothness; the determination method is based on the local difference amplitude. Linear mapping is applied to the [0.1, 0.9] interval. Through the aforementioned dual-channel smoothing process of visual and emotional input, discontinuities caused by abrupt changes or interruptions in attention during editing transitions can be eliminated, resulting in a natural and smooth short video editing effect.

[0040] Please see Figure 2 A short video intelligent editing system based on multimodal analysis includes the following modules: a multimodal feature extraction module, an editing point generation module, a decision optimization module, and a post-processing module. The multimodal feature extraction module extracts multimodal features from the target short video stream, including extracting temporal data of lip movement features, product visual saliency features, and voice emotional intensity values. The editing point generation module transcribes the voice stream and extracts product keyword timestamps. Simultaneously, it determines the peak time of product display based on the product visual saliency feature sequence, calculates the alignment between the peak time of product visual saliency and the keyword timestamps using a dynamic time warping algorithm, and when the alignment exceeds a preset threshold, it performs a weighted evaluation based on the voice emotional intensity value at that moment. The first module estimates and generates a preliminary set of editing points. The second module constructs an editing decision optimization model based on deep reinforcement learning. It uses the extracted multimodal features as state input, the preliminary set of editing points as candidate action space, and the product recognition accuracy and speech-video synchronization as joint reward functions. During training, the policy network decides whether to retain or discard editing points based on feedback from the joint reward function, and selects the optimal transition method for the retained editing points. The third module analyzes the synchronization error between the lip movement trajectory before and after the editing point and the speech flow. When the error exceeds a predetermined threshold, it adjusts the position, evaluates the rate of change of emotional intensity and visual attention continuity of the editing transition segment, smooths the detected discontinuous areas, and outputs the finished editing product.

[0041] In this implementation scheme, a multimodal feature extraction module is used to perform comprehensive feature analysis on the input target short video stream and extract multimodal information, including: Lip movement features: Lip contours are located frame-by-frame using facial keypoint detection, and the change amplitude of lip contours in adjacent frames is calculated to form a lip movement trajectory sequence for subsequent speech synchronization analysis. Product visual saliency features: The visual attractiveness of the product area in each frame is quantified using an attention-based visual saliency detection network, and a visual peak sequence is obtained through smoothing and peak determination methods. Speech emotion intensity values: The audio stream is processed in frames, Mel-frequency cepstral coefficients and fundamental frequency features are extracted, and an emotion discrimination model outputs a time-varying emotion intensity sequence. The multimodal time-series data output by this module provides state information for subsequent editing point generation and decision optimization. The editing point generation module is used to initially locate potential editing points. Its functions include: Speech transcription and keyword recognition: The audio stream is converted into a text sequence using speech recognition technology, and named entity recognition is performed using a product dictionary to extract product-related keywords and their start and end timestamps. Visual peak moment determination: A sliding window analysis is performed on the visual saliency sequence of the product to identify local maxima, and a duration threshold is used to determine the effective peak. Alignment and weighted scoring: The optimal matching path between the visual peak sequence and the keyword time sequence is calculated using a dynamic time warping algorithm, and the alignment degree is measured by cumulative distance. When the alignment degree exceeds a preset threshold, the speech emotion intensity value at that moment is combined for weighted fusion to calculate a comprehensive score. The preliminary cut point set output by this module provides a candidate action space for the decision optimization module. The decision optimization module constructs a deep reinforcement learning-based cut decision optimization model to realize the mapping from multimodal features to the final cut decision. Its main functions include: State representation: Temporally aligning and concatenating lip movement features, visual saliency features, and speech emotion features to form a multidimensional state vector. Action space definition: Each moment point in the preliminary cut point set is used as a selectable action, and each action is associated with a set of selectable transition methods. Reward function design: Taking into account the product recognition accuracy, lip movement and speech flow synchronization, and the smooth transition of emotion before and after cut, a joint reward function is formed. Strategy Optimization: A policy network is trained using a policy gradient algorithm. It receives state input and outputs the retention probability of each candidate cut point and its corresponding transition preference. During interaction with the environment, the network gradually optimizes the editing strategy based on joint reward feedback, achieving joint decision-making between cut point retention and transition methods. Through the learned strategy, this module can automatically select the optimal cut point and transition method, improving the content coherence and visual appeal of the final edit.The post-processing module refines the initial editing results to ensure a natural and smooth final video output. Its main functions include: Synchronization Error Adjustment: Analyzing the cross-correlation function between the lip movement trajectory before and after the editing point and the speech flow, calculating the synchronization error; if the error exceeds a preset threshold, adjusting the editing point position by searching for the location of the maximum cross-correlation value in the vicinity of the editing point. Transition Segment Smoothing: Evaluating the rate of change in emotional intensity and the continuity of visual saliency in the editing transition segments, detecting discontinuous regions using a sliding window and threshold; for detected discontinuous regions, generating visual transition frames using optical flow, and adaptively smoothing the emotional intensity sequence. After processing by this module, the final edited video can be output, ensuring visual continuity, audio smoothness, and emotional coherence.

[0042] In summary, this application has at least the following effects: A short video intelligent editing method and system based on multimodal analysis can accurately perceive lip movements, product displays, and emotional nuances in short videos through comprehensive analysis of multimodal features, providing a reliable basis for editing decisions. Utilizing dynamic time warping algorithms and weighted fusion strategies, a preliminary set of editing points is effectively generated, improving the accuracy and content relevance of these points. A decision optimization model based on deep reinforcement learning can automatically adjust the retention and discard of editing points and select the optimal transition method, enhancing the coherence and smoothness of the editing. Through synchronization error correction and transition smoothing, the continuity of lip movements, speech flow, visual attention, and emotional changes is achieved, ensuring the final edit is natural and expressive. A complete closed-loop system is constructed, from multimodal feature extraction, editing point generation, strategy optimization to post-processing, enabling intelligent and automated short video editing, improving editing efficiency and user experience.

[0043] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0045] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0046] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0047] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0048] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A short video intelligent editing method based on multimodal analysis, characterized in that, Includes the following steps: S1. Perform multimodal feature extraction on the target short video stream, including extracting temporal data of lip movement features of people, visual salience features of products, and emotional intensity values ​​of speech; S2. Transcribe the speech stream and extract product keyword timestamps. At the same time, determine the peak time of product display based on the product visual saliency feature sequence. Calculate the alignment between the peak time of product visual saliency and keyword timestamps using a dynamic time warping algorithm. When the alignment exceeds a preset threshold, perform a weighted evaluation based on the speech emotion intensity value at that time to generate a preliminary set of clipping points. S3. Construct a deep reinforcement learning-based editing decision optimization model, using the extracted multimodal features as state input, the initial set of editing points as candidate action space, and the product recognition accuracy and speech-video synchronization as joint reward functions. During training, the policy network decides whether to retain or discard editing points based on the feedback of the joint reward function, and selects transition methods for the retained editing points. S4. Analyze the synchronization error between the lip movement trajectory before and after the editing point and the speech flow. When the error exceeds the preset deviation value, adjust the position. Evaluate the rate of change of emotional intensity and visual attention continuity in the editing transition segment. Smooth the detected discontinuous areas and output the finished editing product.

2. The short video intelligent editing method based on multimodal analysis according to claim 1, characterized in that: The specific process for multimodal feature extraction from the target short video stream is as follows: By detecting facial key points, the motion changes of the lip region are tracked frame by frame. By calculating the shape changes and displacement vectors of the lip contour between adjacent frames, the temporal sequence of lip motion trajectory is extracted. A visual saliency detection network based on an attention mechanism analyzes the distribution of visual attractiveness of product regions in video frames and extracts product visual saliency feature sequences. The audio stream is segmented into frames, and the Mel frequency cepstral coefficients and fundamental frequency features of each frame are extracted. Continuous time-series data of emotional intensity values ​​are then calculated and extracted.

3. The short video intelligent editing method based on multimodal analysis according to claim 1, characterized in that: The specific process of transcribing the audio stream, extracting product keyword timestamps, and determining the peak time of product display based on the product's visual salience feature sequence is as follows: The audio stream is converted into a text sequence through speech recognition, and named entity recognition is performed in combination with a pre-set product dictionary database to filter out product-related keywords. The start and end times of each keyword are located by the timestamp information of the voice signal. A sliding window analysis is performed on the visual salience sequence of products. By finding local maxima and combining them with duration thresholds, the effective peak moments of product display are determined.

4. The short video intelligent editing method based on multimodal analysis according to claim 3, characterized in that: The alignment between the peak visual saliency of a product and the keyword timestamp is calculated using a dynamic time warping algorithm. When the alignment exceeds a preset threshold, a weighted evaluation is performed based on the emotional intensity value of the speech at that moment. The specific process for generating a preliminary set of clipping points is as follows: The optimal matching path between the peak time sequence of visual salience of goods and the time sequence of keyword occurrence is calculated by the dynamic time warping algorithm, and the alignment of the two sequences is measured by the cumulative distance of the path. When the alignment exceeds a preset threshold, the voice emotion intensity value at that time point is further combined with the weighted fusion strategy to calculate a comprehensive score. Based on the score ranking, the time points that meet the conditions are selected to generate a preliminary set of editing points.

5. The short video intelligent editing method based on multimodal analysis according to claim 1, characterized in that: The construction logic for building an editing decision optimization model based on deep reinforcement learning is as follows: A state representation mechanism is established, which aligns the extracted lip movement feature sequence, product visual saliency sequence, and voice emotion intensity sequence in time and concatenates the features to form a multi-dimensional state vector; Define the motion space, treat each moment in the initial set of cut points as an optional motion, and associate each motion with a set of optional transitions; Design a multi-objective reward function that comprehensively considers the accuracy of product recognition in the image, the synchronization and consistency between lip movements and speech flow, and the smooth transition of emotions before and after editing. A policy network is constructed using the policy gradient algorithm. Through a multi-layer network structure consisting of a state feature encoding layer, an attention fusion layer, and a policy output layer, the mapping from multimodal temporal features to editing decisions is realized.

6. The short video intelligent editing method based on multimodal analysis according to claim 5, characterized in that: During training, the policy network determines whether to retain or discard clipping points based on feedback from the joint reward function, and the specific process of selecting transition methods for retained clipping points is as follows: The policy network receives state input consisting of multimodal features and outputs the retention probability distribution and corresponding transition mode preference distribution for each candidate clipping point; By interacting with the environment, a joint reward function feedback is obtained. This feedback comprehensively considers the accuracy of product recognition, the synchronization between voice and image, and the smoothness of emotional transition. It is used as an optimization signal to update the network parameters through a policy gradient algorithm. During the network parameter update process, the policy network gradually learns to evaluate the comprehensive value of each candidate clip point, dynamically adjusts the retention probability distribution of each clip point, and selects the corresponding transition method for the clip points that are determined to be retained based on the degree of multimodal feature matching between the segments before and after the point.

7. The short video intelligent editing method based on multimodal analysis according to claim 1, characterized in that: The specific process of analyzing the synchronization error between the lip movement trajectory before and after the editing point and the speech flow, and adjusting the position when the error exceeds the preset deviation value, is as follows: The cross-correlation function between the lip movement trajectory and the Mel frequency cepstral features of the speech stream within a set time window before and after the clipping point is calculated, and the synchronization error is calculated by finding the time delay value corresponding to the peak value of the cross-correlation function. If the delay error exceeds the preset deviation value, a binary search algorithm is used in the vicinity of the cut point to find the position with the largest cross-correlation value, and this position is used as the new cut point after adjustment.

8. The short video intelligent editing method based on multimodal analysis according to claim 7, characterized in that: The specific process for evaluating the rate of change in emotional intensity and visual attention continuity during editing transitions, and smoothing detected discontinuities, is as follows: The first-order difference of the sentiment intensity sequence is calculated using a sliding window to detect abnormal abrupt changes where the difference value exceeds the allowable fluctuation range. Simultaneously, the continuity of the visual saliency sequence distribution before and after the clipping point is analyzed to identify regions where visual attention is interrupted; For the detected discontinuous regions, visual transition frames are generated using optical flow based on the consistency of motion vectors between adjacent frames, and the emotional intensity sequence is adaptively smoothed and filtered.

9. A short video intelligent editing system based on multimodal analysis, applied to the short video intelligent editing method based on multimodal analysis as described in any one of claims 1-8, characterized in that, Includes the following modules: Multimodal feature extraction module, clipping point generation module, decision optimization module, and post-processing module; The multimodal feature extraction module is used to extract multimodal features from the target short video stream, including extracting temporal data of lip movement features of people, visual salience features of goods, and emotional intensity values ​​of speech. The clipping point generation module is used to transcribe the speech stream and extract product keyword timestamps. At the same time, it determines the peak time of product display based on the product visual saliency feature sequence, calculates the alignment between the peak time of product visual saliency and keyword timestamps through a dynamic time warping algorithm, and when the alignment exceeds a preset threshold, it performs a weighted evaluation based on the speech emotion intensity value at that time to generate a preliminary clipping point set. The decision optimization module is used to construct a deep reinforcement learning-based editing decision optimization model. It takes the extracted multimodal features as state input, the preliminary set of editing points as candidate action space, and the product recognition accuracy and speech-video synchronization as joint reward function. During training, the policy network decides whether to retain or discard editing points based on the feedback of the joint reward function, and selects the optimal transition method for the retained editing points. The post-processing module is used to analyze the synchronization error between the lip movement trajectory before and after the editing point and the speech flow. When the error exceeds a predetermined threshold, the position is adjusted. The module evaluates the rate of change of emotional intensity and the continuity of visual attention in the editing transition segment, smooths the detected discontinuous areas, and outputs the finished editing product.

Images