A short video intelligent clipping method and system based on multi-modal analysis

Abstract

The application discloses a kind of short video intelligent clipping method and system based on multi-modal analysis, it is related to video editing technical field.For improving editing efficiency and visual experience, the motion feature of the lips of the person, the visual saliency feature of the commodity and the voice emotion intensity value are extracted from the target short video stream, forming multi-modal time series data;Then, the voice stream is transcribed and the product keyword timestamp is extracted, and at the same time, the visual saliency peak value is combined, the alignment degree is calculated by dynamic time warping, and the emotion intensity value is combined to evaluate the weighted, to generate a preliminary set of editing points;Construct a deep reinforcement learning-based editing decision optimization model, input the multi-modal features as the state, adjust the editing point retention probability and select the optimal transition mode with the joint reward function;Analyze the lip movement and voice synchronization error before and after the editing point and the transition section emotion and visual continuity, smooth the discontinuous area, output the editing product, and realize the precise short video intelligent editing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video clip, in particular to a short video intelligent clipping method and system based on multi-modal analysis. BACKGROUND

[0002] With the rapid development of short video platforms, videos have become an important medium for users to obtain information, express opinions and make consumption. Whether it is e-commerce promotion, film and television entertainment, or knowledge dissemination and social interaction, short videos are penetrating into daily life at a high growth rate. Users have higher requirements for the quality and individualization of video content, especially in the context of product display and character expression, audiences not only pay attention to the consistency of pictures and sounds, but also pay more attention to the overall smoothness of the show and the sense of emotional involvement. In order to meet this demand, intelligent video editing has gradually become an important direction of the industry.

[0003] Most existing video editing methods rely on single-modal feature extraction, such as using only speech recognition to determine the clip point, or only using visual detection to segment scenes. Such methods have the following substantive problems in practical application: on the one hand, due to the noise and instability of single-modal signals, it is easy to cause the clip point selection to deviate, and then the situation of voice and picture out of sync occurs; on the other hand, the existing methods lack overall consideration when dealing with emotional expression and visual attention continuity, often resulting in abrupt clip transitions, incomplete product display or fragmented character expressions, which leads to the final film lacking in coherence and watchability. These problems directly affect the user's viewing experience and content dissemination effect of short videos. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a short video intelligent clipping method and system based on multi-modal analysis, which solves the problems of the above background technology.

[0005] To achieve the above object, the application is implemented by the following technical solutions: a short video intelligent clipping method based on multi-modal analysis, comprising the following steps: S1. Multi-modal feature extraction is performed on a target short video stream, including extracting character lip movement features, product visual saliency features and time series data of voice emotion intensity values; S2. Transcription is performed on a voice stream and product keyword timestamps are extracted, and at the same time, product display peak moments are determined according to the product visual saliency feature sequence, the alignment degree between the product visual saliency peak moment and the keyword timestamp is calculated by a dynamic time warping algorithm, when the alignment degree exceeds a preset threshold, a weighted evaluation is performed in combination with the voice emotion intensity value at the moment, and a preliminary clipping point set is generated; S3. A clipping decision optimization model based on deep reinforcement learning is constructed, the extracted multi-modal features are taken as state inputs, the preliminary clipping point set is taken as a candidate action space, and product recognition accuracy and voice-picture synchronization degree are taken as a joint reward function, in the training process, a policy network determines the retention or discard of the clipping point based on the feedback of the joint reward function, and selects a transition mode for the retained clipping point; S4. The synchronization error of the lip movement trajectory and the voice stream before and after the clipping point is analyzed, when the error exceeds a preset deviation value, the position is adjusted, the emotion intensity change rate and visual attention continuity of the clipping transition section are evaluated, the detected discontinuous area is smoothed, and the clipping finished product is output.

[0006] Further, the specific process of multi-modal feature extraction on the target short video stream is as follows: the lip region motion change is tracked frame by frame through face key point detection, the shape change and displacement vector of the lip contour between adjacent frames are calculated, and the lip movement trajectory time series is extracted; the visual saliency detection network based on the attention mechanism analyzes the visual attraction distribution of the product region in the video frame, and extracts the product visual saliency feature sequence; the audio stream is subjected to frame processing, the mel frequency cepstrum coefficient and fundamental frequency feature of each frame of audio are extracted, and the continuous time series data of emotion intensity values are calculated and extracted.

[0007] Further, the specific process of transcription on the voice stream and extraction of product keyword timestamps, and determination of product display peak moments according to the product visual saliency feature sequence is as follows: the audio stream is converted into a text sequence through voice recognition, combined with a preset product dictionary database for named entity recognition, product-related keywords are screened out, and the starting and ending times of each keyword are located through the timestamp information of the voice signal; the sliding window analysis is performed on the product visual saliency sequence, the effective product display peak moment is determined by finding the local maximum value point and combining the duration threshold.

[0008] Further, the alignment between the peak moments of the visual saliency of the commodity and the keyword timestamps is calculated by the dynamic time warping algorithm, and when the alignment exceeds the preset threshold, the voice emotion intensity value at the moment is combined for weighted evaluation to generate the specific process of generating the preliminary editing point set as follows: the optimal matching path between the peak moment sequence of the visual saliency of the commodity and the keyword appearance time sequence is calculated by the dynamic time warping algorithm, and the alignment degree of the two sequences is measured by the cumulative distance of the path; when the alignment exceeds the preset threshold, the voice emotion intensity value at the time point is further combined, the comprehensive score is calculated by the weighted fusion strategy, and the time points meeting the conditions are selected according to the score ranking to generate the preliminary editing point set.

[0009] Further, the construction logic of the editing decision optimization model based on deep reinforcement learning is as follows: a state representation mechanism is established, the extracted lip movement feature sequence, the visual saliency sequence of the commodity and the voice emotion intensity sequence are time-aligned and feature-spliced to form a multi-dimensional state vector; define the action space, each time point in the preliminary editing point set as a selectable action, and associate a set of selectable transition modes for each action; design a multi-objective reward function, considering the recognition accuracy of the commodity in the picture, the synchronization consistency of the lip movement and the voice stream, and the smooth transition of the emotion before and after editing; construct a policy network through a policy gradient algorithm, realize the mapping from multi-modal time sequence features to editing decisions through the multi-layer network structure of the state feature coding layer, the attention fusion layer and the policy output layer.

[0010] Further, in the training process, the policy network decides to keep or discard the editing points based on the joint reward function feedback, and the specific process of selecting the transition mode for the kept editing points is as follows: the policy network receives the state input composed of multi-modal features, outputs the retention probability distribution of each candidate editing point and the corresponding transition mode preference distribution; the joint reward function feedback is obtained through interaction with the environment, which considers the commodity recognition accuracy, voice-picture synchronization degree and emotion transition smoothness, and is used as an optimization signal to update the network parameters through the policy gradient algorithm; during the network parameter updating process, the policy network gradually learns to evaluate the comprehensive value of each candidate editing point, dynamically adjusts the retention probability distribution of each editing point, and for the editing points determined to be kept, selects the corresponding transition mode according to the multi-modal feature matching degree of the fragments before and after the point.

[0011] Further, the specific process of adjusting the position when the error exceeds the preset deviation value is as follows: the cross-correlation function of the lip movement track and the mel-frequency cepstrum feature of the voice stream in the set time window before and after the editing point is calculated, and the synchronization error is calculated by finding the time delay value corresponding to the peak of the cross-correlation function; when the time delay value error exceeds the preset deviation value, the position where the cross-correlation value is maximum is found in the vicinity of the editing point by using the dichotomy search algorithm, and the position is taken as the new editing point after adjustment.

[0012] Further, the specific process of smoothing the detected discontinuous region by evaluating the emotional intensity change rate and visual attention continuity of the transition section of the editing is as follows: the first-order difference of the emotional intensity sequence is calculated through a sliding window, and the abnormal mutation points whose difference values exceed the allowed fluctuation range are detected; meanwhile, the distribution continuity of the visual saliency sequence before and after the editing point is analyzed, and the visual attention interruption region is identified; for the detected discontinuous region, the visual transition frame is generated by the optical flow method based on the motion vector consistency between adjacent frames, and the adaptive smoothing filtering is performed on the emotional intensity sequence.

[0013] A short video intelligent editing system based on multi-modal analysis, comprising the following modules: a multi-modal feature extraction module, an editing point generation module, a decision optimization module, and a post-processing module; the multi-modal feature extraction module is used for multi-modal feature extraction of a target short video stream, including extracting the time sequence data of the human lip movement feature, the product visual saliency feature and the voice emotional intensity value; the editing point generation module is used for transcribing the voice stream and extracting the product keyword timestamp, simultaneously determining the product display peak time according to the product visual saliency feature sequence, calculating the alignment degree between the product visual saliency peak time and the keyword timestamp through the dynamic time warping algorithm, when the alignment degree exceeds the preset threshold, performing weighted evaluation combined with the voice emotional intensity value at the time, and generating a preliminary editing point set; the decision optimization module is used for constructing an editing decision optimization model based on deep reinforcement learning, taking the extracted multi-modal features as state input, taking the preliminary editing point set as a candidate action space, and taking the product recognition accuracy and the voice-picture synchronization degree as a joint reward function, in the training process, the policy network decides to retain or discard the editing point based on the joint reward function feedback, and selects the optimal transition mode for the retained editing point; the post-processing module is used for analyzing the synchronization error of the lip movement track and the voice stream before and after the editing point, adjusting the position when the error exceeds the predetermined threshold, evaluating the emotional intensity change rate and visual attention continuity of the transition section of the editing, smoothing the detected discontinuous region, and outputting the editing product.

[0014] The present application has the following beneficial effects:

[0015] (1) A short video intelligent editing method based on multi-modal analysis, by extracting multi-modal features of character lip movement features, product visual saliency features and speech emotion intensity values, the key information in the video can be captured comprehensively, not limited to single modal signal analysis, thereby effectively improving the accuracy and robustness of the selection of editing points. Combining the speech transcription results with the peak time of product display, and using dynamic time warping algorithm to realize the alignment and weighted evaluation of multi-modal information, a preliminary editing point set that conforms to the content logic and user focus can be generated, significantly reducing the problems of asynchronous picture and speech and incomplete product display.

[0016] (2) A short video intelligent editing system based on multi-modal analysis, a deep reinforcement learning based editing decision optimization model is constructed, which can dynamically adjust the retention and discard strategy of editing points in multiple training processes, and select appropriate transition methods for retained editing points, so that the film can be considered in terms of product recognition accuracy and speech-picture synchronization. Through the analysis of the synchronization error of the lip movement trajectory and the speech stream and the continuity evaluation of the transition section, fine correction and smoothing processing of the editing points are realized, so as to ensure the consistency of the video in emotional expression, visual attention and smoothness, and improve the overall viewing experience.

[0017] Of course, implementing any product of the present application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of a short video intelligent editing method based on multi-modal analysis of the present application.

[0019] Figure 2 A flowchart of a short video intelligent editing system based on multi-modal analysis of the present application. DETAILED DESCRIPTION

[0020] The embodiments of the present application provide a short video intelligent editing method and system based on multi-modal analysis, which solves the problems of insufficient utilization of multi-modal information, inaccurate selection of editing points and dependence on artificial experience in the existing short video editing technology.

[0021] The scheme in the embodiments of the present application has the following general idea:

[0022] First, multi-modal feature extraction is performed on the target short video stream, including extracting character lip movement features, product visual saliency features and speech emotion intensity value time series data, to realize comprehensive capture of key elements in video content.

[0023] Subsequently, the voice stream is transcribed into text and product keyword timestamps are extracted, while the peak time of commodity display is determined by analyzing the sequence of commodity visual saliency features, and the dynamic time warping algorithm is used to match and weight the two, thereby generating a preliminary set of clip points.

[0024] Further, a clip decision optimization model based on deep reinforcement learning is constructed, the above-mentioned multi-modal features are taken as state inputs, and the set of clip points is taken as a candidate action space, and under the guidance of a joint reward function, the retention and transition mode of the clip points are dynamically optimized.

[0025] Finally, the optimized clip point sequence is corrected for cross-modal consistency, the synchronization error between lip movement and voice stream and the emotional and visual continuity of the transition segment are detected, the position and transition mode of the clip points are finely adjusted, and finally a film that meets the visual coherence and emotional naturalness is output.

[0026] Please refer to Figure 1 The embodiment of the present application provides a technical scheme: a short video intelligent clipping method based on multi-modal analysis, comprising the following steps: S1. multi-modal feature extraction is performed on the target short video stream, including extracting the time sequence data of the human lip movement feature, the commodity visual saliency feature and the voice emotional intensity value; S2. the voice stream is transcribed and product keyword timestamps are extracted, while the peak time of commodity display is determined according to the sequence of commodity visual saliency features, the alignment degree between the peak time of commodity visual saliency and the keyword timestamp is calculated by the dynamic time warping algorithm, when the alignment degree exceeds a preset threshold, the voice emotional intensity value at the time is combined for weighted evaluation, and a preliminary set of clip points is generated; S3. a clip decision optimization model based on deep reinforcement learning is constructed, the extracted multi-modal features are taken as state inputs, the preliminary set of clip points is taken as a candidate action space, the commodity recognition accuracy and the voice-picture synchronization degree are taken as a joint reward function, in the training process, the policy network determines the retention or discard of the clip points based on the feedback of the joint reward function, and selects the transition mode for the retained clip points; S4. the synchronization error between the lip movement trajectory before and after the clip point and the voice stream is analyzed, when the error exceeds a preset deviation value, the position is adjusted, the emotional intensity change rate and the visual attention continuity of the clip transition segment are evaluated, the detected discontinuous region is smoothed, and the clip product is output.

[0027] In this embodiment, S1. Simultaneously obtain three types of key information reflecting "synchronization of speaking", "visual degree of commodity", and "intensity of emotional expression" from the original short video as basic data for subsequent multi-modal alignment, candidate clip point generation and decision optimization. Perform face detection and locate the lip region on the video frame sequence, track the lip key points or contours frame by frame, and construct the time sequence signal of lip movement (used to determine the correspondence between mouth shape change and speech). Use visual saliency or object detection model to locate and score the commodity area in the picture, generate the sequence of commodity visual saliency changing over time (used to measure the visual prominence of the commodity at each time). Frame the audio stream and do speech preprocessing, input the emotion recognition module to extract the speech emotion intensity value at each time (used to depict the emotional ups and downs of speech expression). Lip movement feature: refers to the time sequence signal reflecting the opening and deformation of the mouth shape obtained by face key points or pixel-level contour difference; can be used to judge the synchronization of speech and mouth shape. Commodity visual saliency feature: refers to the numerical sequence reflecting the degree of visual attention focusing on the commodity in the picture, usually obtained by saliency detection or object detection and attention weighting. Speech emotion intensity value: refers to a quantitative indicator representing the strength of the speaker's speech emotion at a certain time (such as excitement, stability, etc.), which is usually output by an acoustic feature through an emotion classification / regression model. S2. Establish a time correspondence between the semantic layer and the visual layer, find out the candidate time that both "refers to the product in semantics" and "highlighted in visual display", and use it as the source of the preliminary clip point. Transcribe the speech stream into text through automatic speech recognition (ASR), and identify the keywords related to the product in the text, while recording the time stamps of these keywords in the audio (keyword time stamp). Perform local maximum value detection (peak detection) on the commodity visual saliency sequence, and determine the peak time of commodity display (visual peak time) according to the saliency persistence rule. Input the keyword timestamp sequence and visual peak time sequence into the dynamic time warping (DTW) time alignment algorithm, and calculate the alignment degree between the two sequences. If the alignment degree exceeds the preset threshold, it means that the time is outstanding in both semantics and vision; further weighted evaluation is made on the speech emotion intensity value corresponding to the time to improve the confidence of the candidate point, and finally the preliminary clip point set is collected. Keyword timestamp: the start and end time position of a keyword in the audio stream output by ASR. Commodity display peak time: the time point or time period in the commodity visual saliency sequence that is determined to be "visually highlighted" after peak detection and persistence judgment. Dynamic time warping (DTW): an algorithm for comparing the similarity of two time sequences, allowing the sequence to be nonlinearly stretched and matched on the time axis, so as to find the optimal alignment path (convenient for dealing with the problem of different speech speed or shot rhythm). The lip movement, saliency and emotion sequences extracted by S1 provide input for the peak detection and weighted evaluation of S2; the preliminary clip point set output by S2 carries multi-modal context information, which is used as the candidate action basis for the subsequent decision module.S3. Based on the multi-modal context, the "keep / discard" decision and the corresponding transition mode are automatically learned and optimized to balance the multi-dimensional goals such as product prominence, voice-picture synchronization, and visual coherence. State representation: The lip movement sequence, product saliency sequence, and speech emotion sequence extracted in S1 are time-aligned and feature-spliced to form a multi-dimensional state vector that describes the current candidate point and its context. Action space: Each candidate cut point is taken as a basic action unit, and the action includes the "keep / discard" decision for the point and several optional transition modes (such as direct switching, cross-fade, visual smooth transition, etc.) if it is kept. Joint reward function: designed as a comprehensive evaluation signal, taking into account product recognition accuracy (to ensure that product information can be recognized and presented), voice-picture synchronization degree (to ensure that lip movements correspond to the sound), and emotional continuity (to ensure that the emotional transition is natural). This reward function serves as the optimization target for reinforcement learning, guiding the policy network to learn the preferred editing strategy under different multi-modal contexts. Learning mechanism: the policy network is trained using methods such as proximal policy optimization, allowing it to update its parameters through interaction with the environment (simulating editing and evaluating rewards) and learn the probability distribution of high-reward decisions. Deep reinforcement learning: a machine learning method that allows an agent to learn decision-making strategies through interaction with the environment and based on reward signals, suitable for solving sequence decision-making and long-term reward maximization problems. Policy network: a neural network model that outputs action probability distributions, used in this invention to map multi-modal states to editing decisions (retention probability and transition preference). Candidate action space: the set of all actions available at a certain time or candidate point (including whether to keep or discard and optional transitions). S3 uses the candidate points provided by S2 and the multi-modal states from S1 as input, and the trained policy can directly output the optimized cut point sequence, providing a test solution for the consistency check in S4. S4. Fine check and local correction of the optimized cut point sequence output by S3 to ensure that the final film meets the quality requirements in terms of voice-lip synchronization, visual attention continuity, and emotional expression smoothness. Synchronization error analysis: use the lip movement timing and speech timing sequence from S1 to perform frame-by-frame / time-segment-by-time-segment time alignment and calculate the synchronization error of the lip movement-audio before and after the cut point; when the error exceeds the preset synchronization threshold, trigger the cut point fine-tuning (e.g., move the cut point forward or backward to a more appropriate time window). Emotional and visual continuity evaluation: calculate the rate of change of speech emotion intensity in the transition segment (to determine if the emotion is abrupt) and evaluate the continuity of visual attention distribution (e.g., by significant map or product area coverage consistency to determine if the attention is interrupted); if discontinuity or abruptness is found, trigger smoothing.Smooth processing strategy: adopt general processing methods such as content similarity matching (replace / extend transition section with adjacent frames or similar shots), frame interpolation or cross-fading, or adjust transition parameters to alleviate the sense of abruptness, so as to realize smooth transition of vision and emotion without destroying semantic expression. Synchronization error: refers to the offset of lip movement and voice signal on the time axis, which is used to quantify whether the lip movement and voice are aligned. Emotional intensity change rate: represents the rate of change of emotional intensity over time, and a large change rate indicates a sudden change in emotion, which may cause a sudden transition. Visual attention continuity: refers to whether the target (such as a product) of the audience's visual attention remains high continuity in adjacent shots or transition sections, which is often measured by saliency map or target area overlap. S4 takes the output of S3 as input and calls S1 to verify and fine-tune the multi-modal data. After verification, the final film is output, thereby forming a closed-loop workflow of "extraction -> candidate -> optimization -> verification -> output".

[0028] Specifically, the specific process of multi-modal feature extraction of the target short video stream is as follows: the lip region motion change is tracked frame by frame through face key point detection, the shape change and displacement vector of the lip contour between adjacent frames are calculated, and the lip motion trajectory time sequence is extracted; the visual saliency detection network based on attention mechanism analyzes the visual attraction distribution of the product region in the video frame, and extracts the product visual saliency feature sequence; the audio stream is processed frame by frame, the mel frequency cepstrum coefficient and fundamental frequency feature of each frame of audio are extracted, and the continuous time sequence data of emotional intensity value is calculated and extracted.

[0029] In this embodiment, the specific implementation process of multi-modal feature extraction is as follows: the specific process of multi-modal feature extraction of the target short video stream is as follows: the shoulder motion trajectory extraction first locates the lip region key points frame by frame through the face key point detection algorithm, and forms the lip contour point set. To quantify the shape change of the lips between adjacent frames, the following formula can be used: ; ; Parameter description: : video frame : the lip contour vector of the video frame, composed of the coordinates of the detected key points; : the change amplitude of the lip contour between adjacent frames, representing the intensity of the lip movement; : the local frame set centered on frame : the local frame set centered on frame : regularization constant, used to prevent the denominator from being too small; Normalized lip movement indices are used as feature values ​​for the temporal sequence of lip movement trajectories. Through the above calculations, a time-varying lip movement trajectory sequence can be obtained for subsequent speech synchronization analysis. Product visual saliency feature extraction utilizes an attention-based visual saliency detection network to quantify the visual attractiveness of product regions in video frames, obtaining frame-by-frame saliency scores. To stably extract peak moments, smoothing and peak determination methods are introduced. A significant peak moment is determined when the following conditions are met: Parameter description: The original score output by the saliency detection network; Smoothing window radius; Smooth kernel weights, satisfying ; : The significance score after smoothing; : Range of local extrema comparison; Significance threshold; Determination method: threshold The detection rate and false alarm rate can be tested at different significance levels using a manually labeled training sample set, and the optimal inflection point can be selected using the ROC curve. Audio sentiment intensity extraction involves segmenting the audio stream into frames, calculating the Mel-frequency cepstral coefficients and fundamental frequency features for each frame, and concatenating these two values ​​as the input feature vector. The emotional intensity score is obtained by a linear mapping and a nonlinear function. Parameter description: Audio frames Mel frequency cepstral coefficients; Audio frames The fundamental frequency characteristics; : The concatenated audio feature vector; Sentiment classification weight vectors are obtained through training with labeled corpora. : Bias term; A non-linear function (Sigmoid) used to map the output to a fixed interval; The emotional intensity value is used to form a temporal sequence of emotions that changes over time. In summary, mouth movement features are used to obtain temporal indicators through keypoint tracking and normalization calculations; product visual saliency is obtained through an attention mechanism network and peak determination formula to obtain a temporal sequence and peak time; audio features are concatenated with the fundamental frequency using MFCC and then output as emotional intensity values ​​by a classification model. These steps achieve multimodal feature extraction from short videos, providing input for subsequent semantic alignment, reinforcement learning decisions, and smoothing processing.

[0030] Specifically, the specific process of transcribing the voice stream and extracting product keyword timestamps while determining the specific peak time of the commodity display according to the commodity visual saliency feature sequence is as follows: the audio stream is converted into a text sequence through voice recognition, named entity recognition is performed in combination with a preset product dictionary database, product-related keywords are screened out, and the starting and ending times of each keyword are located through the timestamp information of the voice signal; the sliding window analysis is performed on the commodity visual saliency sequence, the local maximum value points are found, and the effective commodity display peak time is determined in combination with the duration threshold.

[0031] In the embodiment, the voice keyword timestamp extraction first transcribes the input voice stream to obtain a text sequence. In combination with a preset product dictionary database, a named entity recognition method is used to screen out product-related keywords. The starting and ending timestamps corresponding to the rth keyword can be expressed as: ; parameter description: : starting timestamp of the rth keyword : ending timestamp of the rth keyword : starting time value output by the voice recognition system : ending time value output by the voice recognition system : keyword index. Through the above calculation, the time interval of all product keywords in the voice stream can be obtained. The commodity visual saliency peak detection performs sliding window smoothing processing on the commodity visual saliency feature sequence. Let the original saliency sequence be , then the smoothed sequence is: ; when a time meets the following conditions, it is determined as an effective commodity display peak: ; parameter description: : original saliency score of time t : smoothed saliency score : sliding window radius : smoothing weight coefficient, satisfying ; : local extreme value comparison range : saliency duration length at the peak time t : duration threshold. The threshold is determined as follows: the detection rate and false alarm rate under different durations can be respectively counted through manually annotated training samples, and the inflection point of the PR curve (precision-recall curve) is used as the optimal threshold.

[0032] Specifically, the alignment between the peak time of the visual saliency of the commodity and the keyword timestamp is calculated by the dynamic time warping algorithm, when the alignment exceeds the preset threshold, the voice emotion intensity value at the time is combined for weighted evaluation, and the specific process of generating the preliminary editing point set is as follows: the optimal matching path between the peak time sequence of the visual saliency of the commodity and the keyword appearance time sequence is calculated by the dynamic time warping algorithm, and the alignment degree of the two sequences is measured by the cumulative distance of the path; when the alignment exceeds the preset threshold, the voice emotion intensity value at the time is further combined, the comprehensive score is calculated by the weighted fusion strategy, and the time point meeting the condition is selected according to the score ranking to generate the preliminary editing point set.

[0033] In the embodiment, the sequence input and feature preparation first record the peak time sequence of the visual saliency of the commodity as , wherein each element represents a peak time on the video time axis; the keyword timestamp sequence is recorded as , wherein each element represents the time position corresponding to the keyword. The dynamic time warping path calculation adopts the dynamic time warping (DTW) algorithm to calculate the optimal matching path between the two time sequences. The core recursive formula is: ; wherein, : represents the cumulative distance of the th element of the saliency peak and the th element of the keyword timestamp on the matching path; : represents the time difference between the time and ; : the lengths of the two sequences, respectively. The final optimal matching path length is recorded as . The alignment degree can be represented by the normalized cumulative distance as: ; wherein, : the alignment index; : the optimal cumulative distance obtained by dynamic time warping; : the length factor for normalization. When , it is determined that the two sequences have an effective alignment relationship. The determination method of the threshold : calculate the F1 score under different thresholds on the validation set, and select the threshold that makes the F1 score maximum as the final determination threshold. The voice emotion weighted fusion further introduces the voice emotion intensity value at the time point determined by alignment, to form a weighted comprehensive score: ; wherein, : the comprehensive score of the time ; : the alignment index; : the voice emotion intensity value corresponding to the time. Weighting coefficients, satisfying Weighting coefficient determination method: A grid search is performed on the training set to select the combination of clip points with the highest weighted average recall and precision. A clip point set is generated, and a comprehensive score is given for all candidate time points. Sort the data and select those with scores exceeding a set threshold. At that moment, a preliminary set of cut points is formed. Threshold The determination method is as follows: by statistically analyzing the detection rate and false alarm rate under different scoring thresholds in the labeled data, the threshold with the largest Youden index is selected as the optimal value using the ROC curve.

[0034] Specifically, the construction logic of the editing decision optimization model based on deep reinforcement learning is as follows: A state representation mechanism is established, which temporally aligns and concatenates the extracted lip movement feature sequences, product visual saliency sequences, and speech emotion intensity sequences to form a multi-dimensional state vector; an action space is defined, where each moment point in the initial editing point set is treated as an optional action, and a set of optional transition methods is associated with each action; a multi-objective reward function is designed, comprehensively considering the accuracy of product recognition in the image, the synchronization consistency between lip movements and the speech flow, and the smooth transition of emotions before and after editing; a policy network is constructed using a policy gradient algorithm, and through a multi-layer network structure consisting of a state feature encoding layer, an attention fusion layer, and a policy output layer, the mapping from multimodal temporal features to editing decisions is realized.

[0035] In this implementation scheme, the state representation mechanism performs time alignment and feature concatenation on the lip movement feature sequence, product visual saliency sequence, and voice emotion intensity sequence obtained from the aforementioned multimodal feature extraction to form a multidimensional state vector: ;in, :time The state vector; : Lip movement feature vector; Visual saliency feature value; : Voice emotion intensity value; Feature concatenation operation. Explanation: This state vector represents the state at time [time]. The video multimodal information is used as input for policy network decision-making. The action space definition defines the initial set of clipping points. Each point in time is treated as an optional action, and each action is associated with a set of optional transitions. Note: Action space It includes options for "whether to retain the clip point" and "which transition method to choose," providing actionable decision-making options for reinforcement learning. A multi-objective reward function is designed, along with a joint reward function. To evaluate the quality of each action: ;in, Product identification accuracy index; : A measure of the synchronization consistency between lip movements and speech flow; : An indicator of the smooth transition of emotions before and after editing; : Reward weighting coefficient, satisfying Method: The overall quality evaluation of the edit (human or automatic metrics) is maximized by adjusting the weight combination on the training set. Explanation: This reward function simultaneously considers content comprehensibility, speech synchronization, and emotional continuity, providing an optimization objective for the policy network. Policy network construction and training employ the policy gradient algorithm to construct the policy network. Its structure includes: a state feature encoding layer: for the state vector Linear mapping and nonlinear activation are performed to generate a high-dimensional representation; Attention fusion layer: Utilizes an attention mechanism to dynamically weight features of each modality, highlighting key modal information; Policy output layer: Outputs the retention probability of each candidate clipping point and the corresponding transition selection probability distribution. Core policy gradient update formula: ;in, : Policy network parameter vector; The learning rate can be determined through grid search or adaptive optimization methods. At any moment Actions taken (edit point preservation and transition selection); Reward value; Policy network in state Select action The probability. Explanation: By iteratively updating the network parameters through policy gradients, the network gradually tends to choose editing strategies that yield higher cumulative rewards, thereby achieving automated, high-quality editing decisions.

[0036] Specifically, during training, the policy network determines whether to retain or discard clips based on feedback from the joint reward function, and selects the transition method for the retained clips as follows: The policy network receives state input composed of multimodal features and outputs the retention probability distribution and corresponding transition method preference distribution for each candidate clip. It obtains feedback from the joint reward function through interaction with the environment. This feedback comprehensively considers the accuracy of product recognition, the synchronization between speech and video, and the smoothness of emotional transition, and serves as an optimization signal to update the network parameters through the policy gradient algorithm. During the network parameter update process, the policy network gradually learns to evaluate the comprehensive value of each candidate clip, dynamically adjusts the retention probability distribution of each clip, and selects the corresponding transition method for the determined retained clips based on the degree of multimodal feature matching between the segments before and after the point.

[0037] In this implementation scheme, the state input and policy output are discussed. During the training process, the policy network receives the state input, which is composed of multimodal features. Output the retention probability distribution for each candidate clipping point. and corresponding transition method preference distribution : ; Parameter description: :time The state vector is obtained by concatenating lip movement features, visual saliency features, and speech emotion intensity features; : The probability distribution of retaining candidate clipping points; : Probability distribution of transition selection corresponding to candidate cut points; The policy network output function includes linear mappings and nonlinear activation layers; The network output is mapped to a probability distribution, ensuring the sum is 1. This step maps multimodal states to action probabilities, allowing the policy network to dynamically evaluate the retention priority and transition selection preference for each clip point based on the current video content. A joint reward function is fed back; during interaction with the environment, the network obtains a joint reward function. Feedback is collected to comprehensively evaluate the effectiveness of each action: Parameter description: Product identification accuracy index; :Synchronous measurement of lip movements and speech flow; The smoothness index of emotional transition before and after editing; : Reward weighting coefficient, satisfying Determination Method: By adjusting different weight combinations in the training set to optimize manually labeled or automatically evaluated metrics, corresponding weight values ​​are selected. The reward function serves as a signal for network optimization, guiding the policy network to prioritize editing decisions that simultaneously consider content, synchronization, and emotional continuity. The policy network parameter update network updates parameters using a policy gradient algorithm. Perform iterative updates to maximize the expected cumulative reward: Parameter description: : Policy network parameter vector; The learning rate can be determined through grid search or adaptive optimization methods. At any moment The actions taken include preserving cut points and selecting transitions; Policy network in state Select action The probability of; : Reward function value. This step iteratively optimizes the policy network through gradient ascent, gradually causing the network to favor editing points and transitions that yield high cumulative rewards. The policy network dynamically adjusts the retention probability distribution of each candidate editing point during training, considering both the comprehensive value evaluation of editing points and transition selection. And the reserved cut point is determined, and an optimal transition mode is selected according to a multimodal feature matching degree of front and rear clips Through the above mechanism, the policy network can realize automatic and dynamic optimization of the cutting decision under the premise of ensuring content coherence and viewing experience.

[0038] Specifically, the synchronization error of the lip movement trajectory and the speech stream before and after the cutting point is analyzed, and when the error exceeds a preset deviation value, the specific process of position adjustment is as follows: the cross-correlation function of the lip movement trajectory and the mel-frequency cepstrum feature of the speech stream in the set time window before and after the cutting point is calculated, and the synchronization error is calculated by finding the time delay value corresponding to the peak value of the cross-correlation function; when the time delay error exceeds the preset deviation value, the position with the maximum cross-correlation value is found in the vicinity of the cutting point by using a dichotomy search algorithm, and the position is taken as the new cutting point after adjustment.

[0039] In the embodiment, the lip movement trajectory and speech stream synchronization error calculation is performed for each preliminary cutting point , and the mel-frequency cepstrum coefficient sequence of the corresponding audio frame is extracted in the set time window before and after the cutting point , and the cross-correlation function is calculated: ; the parameters are as follows: : the lip movement trajectory feature sequence in the cutting point window; : the MFCC feature sequence of the audio frame in the cutting point window; : time delay, used for sliding calculation of the correlation of the lip movement and the audio; : cross-correlation function value, used for measuring the matching degree of the lip movement and the audio signal under the time delay ; : the selected time window length before and after the cutting point, used for calculating the local synchronization; the synchronization error is quantified between the lip movement and the speech signal by using the cross-correlation function. The synchronization error judgment and cutting point adjustment calculate the optimal time delay corresponding to the peak value of the cross-correlation function: ; the synchronization error is defined as , and compared with a preset deviation value : ; the parameters are as follows: : the time delay corresponding to the peak value of the cross-correlation function; : the maximum synchronization deviation value allowed; according to the lip movement and speech synchronization error distribution of the labeled sample, the maximum deviation that can ensure that more than 95% of the cutting points are synchronized in vision and speech is selected as the threshold value. If the synchronization error exceeds , the position with the maximum cross-correlation value is found in the vicinity of the cutting point by using a dichotomy search algorithm , and the clip point is updated to the adjusted position: ; Parameter description: : The delay of the maximum cross-correlation position obtained by binary search; The adjusted clip point position. Description: Through the above steps, the offset of the clip point can be dynamically corrected, the lip movement trajectory and the speech stream are kept in optimal synchronization, and the naturalness of the clip transition and the viewing experience are improved.

[0040] Specifically, the specific process of evaluating the emotional intensity change rate and visual attention continuity of the clip transition section, and smoothing the detected discontinuous region is as follows: the first-order difference of the emotional intensity sequence is calculated through a sliding window, and abnormal mutation points whose difference values exceed the allowed fluctuation range are detected; at the same time, the distribution continuity of the visual saliency sequence before and after the clip point is analyzed, and the visual attention interruption region is identified; for the detected discontinuous region, the visual transition frame is generated based on the motion vector consistency between adjacent frames by the optical flow method, and the adaptive smoothing filtering is performed on the emotional intensity sequence.

[0041] In this embodiment, the emotional intensity change rate evaluation calculates the first-order difference of the emotional intensity sequence in the time window before and after the clip point to reflect the emotional change rate: ; Parameter description: : The emotional intensity value sequence of the clip point adjacent frames; : The first-order difference of the emotional intensity, used for detecting abnormal mutation points; : The length of the time window, used for local evaluation of emotional changes; when , it is determined as an emotional abnormal mutation point. Parameter description: : The maximum allowed emotional intensity change fluctuation value; determination method: by statistically analyzing the normal clip transition emotional change amplitude distribution in the labeled training samples, a value that can cover most of the normal transitions is selected as the threshold. Visual attention continuity analysis, the visual saliency sequence before and after the clip point is analyzed for continuity, and the visual attention interruption region is detected. Including: calculating the saliency value difference of adjacent frames; and lasting more than the minimum frame number , it is determined as a visual attention interruption region. Parameter description: : The visual saliency value of the clip point adjacent frames; : The visual difference of adjacent frames; : The visual continuity judgment threshold; Minimum number of continuous frames is used to distinguish between brief jitter and genuine interruptions. Threshold determination method: By statistically analyzing the range of continuous fluctuations in labeled samples, a threshold suitable for covering most normal continuous frames is selected. Smoothing processing: For detected discontinuous regions, the following processing is performed: Visual smoothing: Based on the consistency of motion vectors between adjacent frames, an interpolated transition frame is generated using optical flow. : Parameter description: : Original video frame pixel values; : Start and end frames of discontinuous regions; : Generated smooth visual transition frames; Emotional smoothing: Smoothing of emotional intensity sequences Perform adaptive filtering: Parameter description: : Smoothed sentiment intensity value; Adaptive smoothing coefficient, automatically adjusted according to the amplitude of emotional fluctuations; the greater the fluctuation, the smoother the smoothing effect. The smaller the value, the stronger the smoothness; the determination method is based on the local difference amplitude. Linear mapping is applied to the [0.1, 0.9] interval. Through the aforementioned dual-channel smoothing process of visual and emotional input, discontinuities caused by abrupt changes or interruptions in attention during editing transitions can be eliminated, resulting in a natural and smooth short video editing effect.

[0042] Please see Figure 2 A short video intelligent editing system based on multimodal analysis includes the following modules: a multimodal feature extraction module, an editing point generation module, a decision optimization module, and a post-processing module. The multimodal feature extraction module extracts multimodal features from the target short video stream, including extracting temporal data of lip movement features, product visual saliency features, and voice emotional intensity values. The editing point generation module transcribes the voice stream and extracts product keyword timestamps. Simultaneously, it determines the peak time of product display based on the product visual saliency feature sequence, calculates the alignment between the peak time of product visual saliency and the keyword timestamps using a dynamic time warping algorithm, and when the alignment exceeds a preset threshold, it performs a weighted evaluation based on the voice emotional intensity value at that moment. The first module estimates and generates a preliminary set of editing points. The second module constructs an editing decision optimization model based on deep reinforcement learning. It uses the extracted multimodal features as state input, the preliminary set of editing points as candidate action space, and the product recognition accuracy and speech-video synchronization as joint reward functions. During training, the policy network decides whether to retain or discard editing points based on feedback from the joint reward function, and selects the optimal transition method for the retained editing points. The third module analyzes the synchronization error between the lip movement trajectory before and after the editing point and the speech flow. When the error exceeds a predetermined threshold, it adjusts the position, evaluates the rate of change of emotional intensity and visual attention continuity of the editing transition segment, smooths the detected discontinuous areas, and outputs the finished editing product.

[0043] In this embodiment, a multi-modal feature extraction module is used for comprehensive feature analysis of the input target short video stream, extracting multi-modal information, including: character lip movement features: through face key point detection, the lip contour is located frame by frame, and the change amplitude of adjacent frame lip contour is calculated to form a lip movement trajectory sequence for subsequent speech synchronization analysis. Commodity visual saliency features: the visual saliency detection network based on attention mechanism is used to quantify the visual attraction of the commodity area of each frame, and the visual peak sequence is obtained through the smoothing and peak determination method. Speech emotion intensity value: the audio stream is processed by frame, and the mel frequency cepstrum coefficient and fundamental frequency features are extracted, and the emotion intensity sequence changing with time is output through the emotion discrimination model. The multi-modal time series data output by this module provides state information for subsequent clip point generation and decision optimization. The clip point generation module is used for preliminary positioning of potential clip points, and its functions include: speech transcription and keyword recognition: the audio stream is converted into a text sequence through speech recognition technology, and combined with the product dictionary for named entity recognition, the product-related keywords and their start and end time stamps are extracted. Visual peak time determination: sliding window analysis is performed on the commodity visual saliency sequence, and the local maximum value points are identified, and the effective peak value is judged combined with the duration threshold. Alignment and weighted scoring: the dynamic time warping algorithm is used to calculate the optimal matching path of the visual peak sequence and the keyword time sequence, and the alignment degree is measured by cumulative distance; when the alignment degree exceeds the preset threshold, the speech emotion intensity value at this time is combined for weighted fusion, and the comprehensive score is calculated. The preliminary clip point set output by this module provides a candidate action space for the decision optimization module. The decision optimization module constructs a clip decision optimization model based on deep reinforcement learning, realizing the mapping from multi-modal features to final clip decision, and its main functions include: state representation: the lip movement features, visual saliency features and speech emotion features are time-aligned and feature-spliced to form a multi-dimensional state vector. Action space definition: each time point in the preliminary clip point set is regarded as a selectable action, and each action is associated with a group of selectable transition modes. Reward function design: considering the product recognition accuracy, the synchronization degree of lip movement and speech stream, and the smooth transition of emotions before and after clipping, a joint reward function is formed. Strategy optimization: the strategy network is trained through the strategy gradient algorithm, which receives state input and outputs the retention probability of each candidate clip point and the preference of the corresponding transition mode; the network gradually optimizes the clipping strategy according to the joint reward feedback in the interaction with the environment, realizing the joint decision of clip point retention and transition mode. The strategy learned by this module can automatically select the optimal clip point and transition mode, improving the content coherence and watchability of the final product.A post-processing module is used to finely optimize the preliminary editing result to ensure the natural and smooth final output video. The main functions include: synchronization error adjustment: analyze the cross-correlation function of the lip motion trajectory and the speech stream before and after the editing point, calculate the synchronization error; if the error exceeds the preset threshold, adjust the editing point position by searching for the position of the maximum cross-correlation in the adjacent area of the editing point. Transition segment smoothing: evaluate the emotional intensity change rate and visual saliency continuity of the editing transition segment, detect discontinuous regions through sliding window and threshold; for the detected discontinuous regions, generate visual transition frames using the optical flow method, and perform adaptive smoothing processing on the emotional intensity sequence. After processing by this module, the edited product can be output, ensuring the visual continuity, audio smoothness and emotional coherence of the video.

[0044] In summary, the present application has at least the following effects:

[0045] A short video intelligent editing method and system based on multi-modal analysis can realize accurate perception of character mouth shape, product display and speech emotion in short videos through comprehensive analysis of multi-modal features, providing reliable basis for editing decisions. The dynamic time warping algorithm and weighted fusion strategy are used to effectively generate a preliminary editing point set, improving the accuracy and content relevance of the editing points. The decision optimization model based on deep reinforcement learning can automatically adjust the retention and discard of editing points, and select the optimal transition mode, enhancing the continuity and smoothness of the edited video. Through synchronization error correction and transition segment smoothing, the continuity of lip motion, speech stream, visual attention and emotional change is realized, ensuring that the edited product is natural and expressive. A complete closed-loop system is constructed from multi-modal feature extraction, editing point generation, strategy optimization to post-processing, realizing intelligent and automated editing of short videos, and improving editing efficiency and user experience.

[0046] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer-usable program code embodied in the medium.

[0047] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the one or more flowcharts and / or blocks

[0048] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the one or more flowcharts and / or blocks

[0049] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for functionally implementing the one or more flowcharts and / or blocks

[0050] While the preferred embodiments of the application have been described, additional variations and modifications can be employed by those skilled in the art. Therefore, the appended claims intend to cover all such modifications and variations as fall within the true spirit and scope of the application.

[0051] It is apparent that a number of modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A short video intelligent editing method based on multimodal analysis, characterized in that, Includes the following steps: S1. Perform multimodal feature extraction on the target short video stream, including extracting temporal data of lip movement features of people, visual salience features of products, and emotional intensity values ​​of speech; S2. Transcribe the speech stream and extract product keyword timestamps. At the same time, determine the peak time of product display based on the product visual saliency feature sequence. Calculate the alignment between the peak time of product visual saliency and keyword timestamps using a dynamic time warping algorithm. When the alignment exceeds a preset threshold, perform a weighted evaluation based on the speech emotion intensity value at that time to generate a preliminary set of clipping points. S3. Construct a deep reinforcement learning-based editing decision optimization model, using the extracted multimodal features as state input, the initial set of editing points as candidate action space, and the product recognition accuracy and speech-video synchronization as joint reward functions. During training, the policy network decides whether to retain or discard editing points based on the feedback of the joint reward function, and selects transition methods for the retained editing points. S4. Analyze the synchronization error between the lip movement trajectory before and after the editing point and the speech flow. When the error exceeds the preset deviation value, adjust the position. Evaluate the rate of change of emotional intensity and visual attention continuity in the editing transition segment. Smooth the detected discontinuous areas and output the finished editing product.

2. The short video intelligent editing method based on multimodal analysis according to claim 1, characterized in that: The specific process for multimodal feature extraction from the target short video stream is as follows: By detecting facial key points, the motion changes of the lip region are tracked frame by frame. By calculating the shape changes and displacement vectors of the lip contour between adjacent frames, the temporal sequence of lip motion trajectory is extracted. A visual saliency detection network based on an attention mechanism analyzes the distribution of visual attractiveness of product regions in video frames and extracts product visual saliency feature sequences. The audio stream is segmented into frames, and the Mel frequency cepstral coefficients and fundamental frequency features of each frame are extracted. Continuous time-series data of emotional intensity values ​​are then calculated and extracted.

3. The short video intelligent editing method based on multimodal analysis according to claim 1, characterized in that: The specific process of transcribing the audio stream, extracting product keyword timestamps, and determining the peak time of product display based on the product visual saliency feature sequence is as follows: The audio stream is converted into a text sequence through speech recognition, and named entity recognition is performed in combination with a pre-set product dictionary database to filter out product-related keywords. The start and end times of each keyword are located by the timestamp information of the voice signal. A sliding window analysis is performed on the visual salience sequence of products. By finding local maxima and combining them with duration thresholds, the effective peak moments of product display are determined.

4. The short video intelligent editing method based on multimodal analysis according to claim 3, characterized in that: The alignment between the peak visual saliency of a product and the keyword timestamp is calculated using a dynamic time warping algorithm. When the alignment exceeds a preset threshold, a weighted evaluation is performed based on the emotional intensity value of the speech at that moment. The specific process for generating a preliminary set of clipping points is as follows: The optimal matching path between the peak time sequence of visual salience of goods and the time sequence of keyword occurrence is calculated by the dynamic time warping algorithm, and the alignment of the two sequences is measured by the cumulative distance of the path. When the alignment exceeds a preset threshold, the voice emotion intensity value at that time point is further combined with the weighted fusion strategy to calculate a comprehensive score. Based on the score ranking, the time points that meet the conditions are selected to generate a preliminary set of editing points.

5. The short video intelligent editing method based on multimodal analysis according to claim 1, characterized in that: The construction logic for building an editing decision optimization model based on deep reinforcement learning is as follows: A state representation mechanism is established, which aligns the extracted lip movement feature sequence, product visual saliency sequence, and voice emotion intensity sequence in time and concatenates the features to form a multi-dimensional state vector; Define the motion space, treat each moment in the initial set of cut points as an optional motion, and associate each motion with a set of optional transitions; Design a multi-objective reward function that comprehensively considers the accuracy of product recognition in the image, the synchronization and consistency between lip movements and speech flow, and the smooth transition of emotions before and after editing. A policy network is constructed using the policy gradient algorithm. Through a multi-layer network structure consisting of a state feature encoding layer, an attention fusion layer, and a policy output layer, the mapping from multimodal temporal features to editing decisions is realized.

6. The short video intelligent editing method based on multimodal analysis according to claim 5, characterized in that: During training, the policy network determines whether to retain or discard clipping points based on feedback from the joint reward function, and the specific process of selecting transition methods for retained clipping points is as follows: The policy network receives state input consisting of multimodal features and outputs the retention probability distribution and corresponding transition mode preference distribution for each candidate clipping point; By interacting with the environment, a joint reward function feedback is obtained. This feedback comprehensively considers the accuracy of product recognition, the synchronization between voice and image, and the smoothness of emotional transition. It is used as an optimization signal to update the network parameters through a policy gradient algorithm. During the network parameter update process, the policy network gradually learns to evaluate the comprehensive value of each candidate clip point, dynamically adjusts the retention probability distribution of each clip point, and selects the corresponding transition method for the clip points that are determined to be retained based on the degree of multimodal feature matching between the segments before and after the point.

7. The short video intelligent editing method based on multimodal analysis according to claim 1, characterized in that: The specific process of analyzing the synchronization error between the lip movement trajectory before and after the editing point and the speech flow, and adjusting the position when the error exceeds the preset deviation value, is as follows: The cross-correlation function between the lip movement trajectory and the Mel frequency cepstral features of the speech stream within a set time window before and after the clipping point is calculated, and the synchronization error is calculated by finding the time delay value corresponding to the peak value of the cross-correlation function. If the delay error exceeds the preset deviation value, a binary search algorithm is used in the vicinity of the cut point to find the position with the largest cross-correlation value, and this position is used as the new cut point after adjustment.

8. The short video intelligent editing method based on multimodal analysis according to claim 7, characterized in that: The specific process for evaluating the rate of change in emotional intensity and visual attention continuity during editing transitions, and smoothing detected discontinuities, is as follows: The first-order difference of the sentiment intensity sequence is calculated using a sliding window to detect abnormal abrupt changes where the difference value exceeds the allowable fluctuation range. Simultaneously, the continuity of the visual saliency sequence distribution before and after the clipping point is analyzed to identify regions where visual attention is interrupted; For the detected discontinuous regions, visual transition frames are generated using optical flow based on the consistency of motion vectors between adjacent frames, and the emotional intensity sequence is adaptively smoothed and filtered.

9. A short video intelligent editing system based on multimodal analysis, applied to the short video intelligent editing method based on multimodal analysis as described in any one of claims 1-8, characterized in that, Includes the following modules: Multimodal feature extraction module, clipping point generation module, decision optimization module, and post-processing module; The multimodal feature extraction module is used to extract multimodal features from the target short video stream, including extracting temporal data of lip movement features of people, visual salience features of goods, and emotional intensity values ​​of speech. The clipping point generation module is used to transcribe the speech stream and extract product keyword timestamps. At the same time, it determines the peak time of product display based on the product visual saliency feature sequence, calculates the alignment between the peak time of product visual saliency and keyword timestamps through a dynamic time warping algorithm, and when the alignment exceeds a preset threshold, it performs a weighted evaluation based on the speech emotion intensity value at that time to generate a preliminary clipping point set. The decision optimization module is used to construct a deep reinforcement learning-based editing decision optimization model. It takes the extracted multimodal features as state input, the preliminary set of editing points as candidate action space, and the product recognition accuracy and speech-video synchronization as joint reward function. During training, the policy network decides whether to retain or discard editing points based on the feedback of the joint reward function, and selects the optimal transition method for the retained editing points. The post-processing module is used to analyze the synchronization error between the lip movement trajectory before and after the editing point and the speech flow. When the error exceeds a predetermined threshold, the position is adjusted. The module evaluates the rate of change of emotional intensity and the continuity of visual attention in the editing transition segment, smooths the detected discontinuous areas, and outputs the finished editing product.

Images