Recommended forecast sheet generation system and method based on multi-modal distillation sentiment analysis

Abstract

The invention discloses a multi-modal distillation sentiment analysis-based recommended advance notice generation system and method, and the system comprises a multi-modal video preprocessing module, a multi-modal video feature recognition module, and a sentiment-oriented video synthesis module. Advanced technologies in multiple fields of natural language processing NLP, computer vision CV and the like are combined, and efficient compression and performance improvement of the model are achieved through the knowledge distillation technology. By deeply analyzing emotional preferences and multi-modal information of the user, highly personalized forecast content can be generated. A multi-mode distillation sentiment analysis technology is utilized, interest points and sentiment tendencies of users can be comprehensively captured, and the accuracy of a recommendation algorithm is improved; the method can intelligently generate the preview, reduces the cost and time of manual production, and enables the content production to be more efficient.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a system and method for intelligently generating trailers for a recommendation system based on multimodal distillation sentiment analysis guidance. Background Art

[0002] In the current broadcasting and online audio-visual industry, the number of audio-visual programs (such as movies, TV series, variety shows, etc.) is increasing day by day, and users' choices of content are becoming more diverse and personalized. However, traditional recommendation systems often rely only on basic information of programs (such as type, director, actors, etc.) or simple user behavior data (such as viewing history, ratings, etc.) for recommendations, ignoring the richness and complexity of the program content itself. Especially for the special form of program content, trailers, as an important means to attract the audience's attention and stimulate their interest in watching, need to accurately capture and show the core charm of the program.

[0003] In recent years, with the rapid development of artificial intelligence technology, multimodal analysis (including images, audio, text, etc.) has shown great potential in audio-visual content processing. However, there are still some key problems to be solved in the existing technology when performing in-depth analysis of program content. Among them, the lack of sentiment classification in the recommended clip content is a significant shortcoming. Emotion, as an important part of human cognition, has a significant impact on understanding and evaluating program content. Therefore, how to effectively extract and combine emotional features based on multimodal analysis has become the key to improving the accuracy of the recommendation system and user experience.

[0004] The main problems currently exist:

[0005] Lack of emotional classification of recommended clip content: Current recommendation systems often ignore the emotional elements in program content, resulting in the inability of recommended clips to accurately convey the emotional atmosphere of the program and the emotional resonance points of the audience, affecting the attractiveness and accuracy of the recommendations.

[0006] The semantic segmentation of programs is not fine enough: For the recognition of important characters, key objects and shot boundaries in programs, the existing technology cannot achieve high-precision semantic segmentation. This makes it difficult for the recommendation system to accurately capture the core plot and highlights of the program, and it is unable to provide users with accurate personalized recommendations.

[0007] User interests do not correspond to program content semantic split labels: Current recommendation systems often lack the ability to effectively match program content semantic split labels with user interests, resulting in deviations between recommendation results and user actual needs. Summary of the invention

[0008] The technical purpose of the present invention is to provide a recommendation system trailer intelligent generation system and method based on multimodal distillation sentiment analysis guidance. In response to the problems in the above-mentioned background technology, this method combines multimodal feature extraction, sentiment analysis, semantic splitting and other technical means to achieve high-precision recognition and sentiment classification of important elements in the program, and then generate personalized trailers that meet the user's interests, thereby improving the accuracy of the recommendation system and user experience.

[0009] To achieve the above purpose, the technical solution adopted by the present invention is a recommended trailer generation system based on multimodal distillation sentiment analysis, comprising:

[0010] The multimodal video preprocessing module completes the video decoding and track division of the original video file, and splits the video file into text mode, voice mode and visual mode according to the shot; and performs data cleaning and alignment of the video mode to obtain multimodal preprocessed data;

[0011] The multimodal video feature recognition module extracts features from preprocessed data in text, speech, and visual modes and synthesizes multimodal feature vectors, performs decoupled distillation multimodal emotion recognition and classification, and performs model training.

[0012] The emotion-oriented video synthesis module retrieves and filters a set of video clips matching emotion features according to emotion-oriented features, forms a global candidate combination that satisfies the emotion similarity, and performs timing optimization and filtering to complete the emotion-oriented video synthesis. It harmonizes and optimizes the special effects of the synthesized video to generate an emotion-oriented trailer for intelligent editing of long videos.

[0013] Furthermore, the multimodal video preprocessing module includes:

[0014] Optionally, the input original video file is decapsulated using the FFmpeg library, and the audio stream and video stream therein are obtained by dividing the tracks.

[0015] Optionally, in multimodal splitting, the AI ​​Chinese speech recognition API is used to obtain the text modality, the short-time energy STE and short-time zero-crossing rate STZCR of the speech signal are used to perform voice activity detection VAD to obtain the audio modality, and the change in similarity between video frames is used to detect shot transition SBD to obtain the visual modality, and the three modalities are optimized and adjusted.

[0016] Optionally, during data cleaning and alignment, the timestamp of each fragment recorded in the segmentation log is used.

[0017] Furthermore, the multimodal video feature recognition module includes:

[0018] Optionally, multimodal feature extraction, use the OpenCV library to read the video frame sequence, and use the MTCNN face detection model with a cascade CNN structure to detect faces frame by frame, use the pre-trained FaceNet model to extract the aligned facial image features of each frame, and generate a visual feature vector; use the librosa library to read the audio waveform sequence, convert it to the frequency domain through fast Fourier transform FFT and perform Mel-scale filter group filtering, obtain VGGish features through DCT discrete cosine transform DCT, and generate an audio feature vector; pre-process the intercepted subtitle text, use the jieba word segmentation tool to split the text into word sequences, and extract the text feature vector through the pre-trained Chinese BERT model.

[0019] Optionally, in multimodal feature splicing, the video, audio, and text modal feature vectors are spliced ​​in the order of the modalities, and serialized and formatted in a format including TFRecord or HDF5 to complete the multimodal feature splicing.

[0020] Optionally, decouple distilled multimodal emotion recognition and model training, build a shared encoder and three exclusive encoders to construct a multimodal decoupled representation, and use the Adam optimizer to train the parameters of each encoder; construct a homologous graph including a graph convolutional network GCN to realize the propagation of shared features between nodes, and construct a heterologous graph of an attention network GAT to realize the distillation of exclusive features between modalities, and use the Adam optimizer to train the parameters of GCN and GAT with the weighted sum of the total classification loss and consistency loss as the goal; input the multimodal emotion feature vector into the classifier for classification, and use the cross entropy as the loss function to jointly optimize the classification loss and the loss of dual graph convolution distillation; train a multimodal emotion dataset including a training set, a validation set, and a test set to obtain an end-to-end emotion classification model including a feature extractor, a graph convolution distillation unit, and a classifier.

[0021] The intelligent generation method of trailers for recommendation systems based on multimodal distillation sentiment analysis includes the following steps:

[0022] Step S101, decode and split the original video file, split the video into three modal data: text, voice, and vision according to the shot, and ensure the multimodal data is cleaned and aligned to obtain multimodal preprocessed data.

[0023] Specifically, the multimodal video preprocessing module first decodes the original video file and separates the audio and video tracks, including:

[0024] (1) Use the FFmpeg library to decapsulate the input original video file and obtain the audio stream and video stream.

[0025] (2) The audio stream is re-encoded, converted into PCM format, and uniformly resampled into schematic 16kHz, 16-bit mono audio data and saved as a WAV format file.

[0026] (3) Re-encode the video stream, convert it to H.264 format, adjust the frame rate to 25fps, and save it as an MP4 file.

[0027] The audio and video streams after the split tracks are split into multi-modal splitting of text mode, voice mode and visual mode according to the shots. The specific steps are as follows:

[0028] (1) Split text mode: Call the AI ​​Chinese speech recognition API to transcribe the audio sentence by sentence, and obtain the machine timestamp of the speech recognition result ending with punctuation.

[0029] 1) Divide the recognition results into multiple sentences according to the punctuation positions, and record the start and end timestamps of each sentence.

[0030] 2) Record the start and end timestamps of each sentence as candidate cutpoints.

[0031] (2) Split speech modes: Use the short-time energy (STE) and short-time zero-crossing rate (STZCR) of speech signals to perform voice activity detection (VAD).

[0032] 1) The audio signal is segmented at the frame level, where each frame is schematically segmented into 20ms and the frame shift is schematically segmented into 10ms.

[0033] 2) Calculate the STE and STZCR features for each frame and use the double threshold method to make speech / non-speech judgments to obtain the start and end timestamps of the speech event.

[0034] 3) The boundaries of the speech events, that is, the speech start and end points, are marked as timestamps candidate cutpoints.

[0035] (3) Split visual modality: Detect shot transitions (SBD) using the changes in similarity between video frames.

[0036] 1) Segment the video frames and schematically extract a key frame every 0.5 seconds.

[0037] 2) The SURF feature matching method is used to compare the similarity of adjacent key frames. When the number of matching points is less than the schematic preset threshold of 50, it is determined to be a shot boundary.

[0038] 3) The timestamps of the shot boundaries are recorded as shots candidate cutpoints, where each shot has multiple dialogues and shots candidate cutpoints contains multiple candidate cutpoints.

[0039] 4) The split video clip files are saved in MP4 format.

[0040] (4) Determine the candidate cut points of other modes based on the audio cut points: final_cuts sorted by timestamp is the final cut point list.

[0041] (5) Traverse all segmented video segments: extract the start and end timestamps, end timestamps, and duration of each segment, and save them in the video segmentation log file.

[0042] The specific steps for data cleaning and alignment of the segmented video clip files and segmented log files are as follows:

[0043] (1) Extract the voice timestamp according to the candidate cutpoints timestamps of each segment recorded in the segmentation log, and use the library including but not limited to FFmpeg to extract the audio and video of the corresponding segment from the original video, and save them as independent WAV and MP4 files respectively.

[0044] (2) Traverse all the segmented video clips and calculate their duration, filter out video clips that are too short, less than 1 second, or too long, greater than 60 seconds, delete them from the segmentation log, and delete the corresponding audio, video, and text files.

[0045] (3) Perform voice activity detection (VAD) on the audio of each segment and calculate the voice proportion:

[0046]

[0047] Filter out the voice segments with a percentage below the preset threshold of 15%, determine them as invalid segments, delete them from the segmentation log, and delete the corresponding audio and video files.

[0048] (4) Using AI’s Chinese speech recognition API, transcribe the audio of each segment to obtain the text of the speech content.

[0049] (5) Post-process the speech recognition results. Use the Jieba word segmentation tool to segment the text and filter out stop words to obtain the syntactic structure information of the segment.

[0050] (6) Update the segmentation log file cut_info.json of each segment in the segmentation log, and record the final start and end time, duration, text, and keyword metadata of each segment.

[0051] Step 102, complete the feature extraction of the preprocessed data of the text modality, speech modality and visual modality respectively, and splice the three modal features into a multimodal integrated vector. Decoupled distillation is performed through the shared encoder and the exclusive encoder, combined with dual-graph convolution distillation and model training, and finally fine-grained emotion recognition is achieved through the classifier.

[0052] The steps of extracting multimodal features from preprocessed multimodal data and segmented logs include:

[0053] (1) Extract visual modal features: Perform visual modal feature extraction on the preprocessed visual modal data and segmentation logs to obtain a 128-dimensional visual feature vector.

[0054] 1) Use the OpenCV library to read the MP4 format video clip file and extract the corresponding video frame sequence according to the start timestamp and end timestamp.

[0055] 2) For the captured video frame sequence, the MTCNN face detection model is used to detect faces frame by frame: MTCNN adopts a cascaded CNN structure to simultaneously complete face detection and key point positioning.

[0056] 3) According to the key point coordinates detected by MTCNN, the face of each frame is aligned to extract highly robust facial features. Face alignment methods such as affine transformation and similarity transformation are selected, and the key points are normalized to the calibration position.

[0057] 4) Use the pre-trained FaceNet model to extract the aligned facial image features of each frame: The FaceNet model is trained using a paired triplet loss function to map the face to a 128-dimensional Euclidean space.

[0058] 5) For a shot, the facial feature vectors extracted from each frame are arranged in chronological order to form a new feature sequence, where the dimension is (N, 128), where N is the number of frames contained in a shot.

[0059] 6) Perform statistical analysis on the feature sequence in 5) in the time dimension, calculate the mean and standard deviation of each dimension of the feature in the sequence, and obtain a 256-dimensional feature vector to represent the visual modal characteristics of the entire video clip.

[0060] (2) Extracting speech modal features: Extracting speech modal features from the preprocessed speech modal data and segmented logs to obtain visual feature vectors.

[0061] 1) Use the librosa library to read the WAV format audio clip file and extract the corresponding audio waveform sequence according to the start timestamp and end timestamp.

[0062] 2) Pre-emphasize and frame the captured audio waveform sequence: Pre-emphasis is used to balance the energy of high-frequency and low-frequency components, and a first-order high-pass filter with a coefficient of 0.97 is selected. Frame division is used for short-term analysis, with the frame length set to 0.96s and the frame shift set to 1.5s.

[0063] 3) Feature extraction for each frame of pre-emphasized audio data: First, the pre-emphasized audio waveform sequence is converted to a frequency domain audio sequence by fast Fourier transform (FFT), and the Mel-scale filter bank is used to filter the Mel spectrum to obtain audio data that is closer to the auditory characteristics of the human ear. The Mel spectrum is logarithmic and discrete cosine transform (DCT) is performed to obtain the VGGish feature vector.

[0064] 4) Arrange the VGGish feature vectors extracted from each frame in chronological order to form a new feature sequence, where the dimension is (N, 128), where N is the number of frames contained in a shot.

[0065] 5) The feature sequence generated in time series is pooled in the time dimension, the mean of each dimension of the feature in the sequence is calculated, and a 128-dimensional mean vector is obtained to represent the speech modal features of the entire audio clip.

[0066] (3) Extract text modal features: perform text modal feature extraction on the preprocessed text modal data and segmented logs to obtain text feature vectors.

[0067] 1) Read the TXT format text fragment file encoded in UTF-8, and extract the corresponding subtitle text content according to the start timestamp and end timestamp.

[0068] 2) Preprocess the extracted subtitle text: Use the Jieba word segmentation tool to segment the text into word sequences. Convert the word sequences into the input format required by the BERT model, including adding [CLS] and [SEP] tags and performing WordPiece word segmentation.

[0069] The Jieba word segmentation tool is selected to optimize the Chinese corpus, considering its moderate word segmentation granularity and customized dictionary.

[0070] 3) Input the preprocessed text into the pre-trained Chinese BERT model: The BERT model is a bidirectional language model based on the Transformer structure, which effectively models the contextual information of the text.

[0071] 4) Extract the output of the last layer of Transformer of BERT as the embedding vector of each word to obtain a word embedding matrix.

[0072] 5) Pool the word embedding matrix and select the MaskAveraging method: Construct a mask matrix of the same shape based on the input tokenization sequence of BERT, where the corresponding position of the word is set to 1 and the corresponding position of the padding mark ([PAD]) is set to 0. Perform Hadamard product processing on the word embedding matrix and the mask matrix to obtain an embedding matrix without the influence of the padding mark. Solve the mean of the embedding matrix without padding in the word dimension to obtain a 768-dimensional sentence vector, which represents the text modality characteristics of the entire text segment.

[0073] The steps of synthesizing the multimodal integrated vector from the extracted multimodal feature data include:

[0074] (1) The feature vectors extracted from the three modalities of image, audio, and text are concatenated into a multimodal integrated vector in the order of image, audio, and text modalities: the 128-dimensional visual feature vector, the 128-dimensional speech feature vector, and the 768-dimensional text feature vector are sequentially synthesized to obtain a 1024-dimensional multimodal feature integrated vector.

[0075] (2) The multimodal feature integration vector and the corresponding video clip metadata are packaged into a data unit. The metadata includes video ID, start timestamp, end timestamp, duration, etc.

[0076] (3) Arrange the data units of all video segments in the order of video ID and timestamp to form a video segment set.

[0077] (4) Serialize the video clip collection in the format of TFRecord, HDF5, etc. and save it to the local disk or cloud database.

[0078] Furthermore, multimodal emotion recognition by decoupling distillation is performed on the synthesized multimodal integrated vector, including:

[0079] (1) Construct a multimodal decoupled representation for the 1024-dimensional multimodal feature integration vector.

[0080] 1) Build a shared encoder and three exclusive encoders for the synthetic multimodal integrated vector and the three modal feature vectors of image, audio, and text respectively.

[0081] The shared encoder receives a 1024-dimensional synthetic multimodal integration vector, which is mapped to a 256-dimensional shared feature space through two fully connected layers (1024->512->256); the visual exclusive encoder receives a 128-dimensional visual feature vector, which is mapped to a 128-dimensional visual exclusive feature space through two fully connected layers (128->128->128); the speech exclusive encoder receives a 128-dimensional speech feature vector, which is mapped to a 128-dimensional speech exclusive feature space through two fully connected layers (128->128->128); the text exclusive encoder receives a 768-dimensional text feature vector, which is mapped to a 512-dimensional text exclusive feature space through two fully connected layers (768->512->512).

[0082] 2) For the three exclusive encoders of vision, speech, and text modalities, an attention pooling layer is added to the last layer of each exclusive encoder: the attention pooling layer of each exclusive encoder compresses the exclusive features into a scalar weight through a fully connected layer. The exclusive features are multiplied by the corresponding scalar weights to obtain the weighted exclusive features, and the weighted exclusive features are summed in the dimensions of vision, speech, and text modal features to obtain an aggregated feature vector with a dimension of 1024.

[0083] 3) With reconstruction error as the goal, train the shared encoder and exclusive encoder: concatenate the shared feature vector and the aggregated exclusive feature vector, and reconstruct a 1024-dimensional multimodal feature vector through a fully connected layer; take the mean square error between the reconstructed multimodal feature and the original multimodal feature integration vector as the reconstruction error; use the Adam optimizer to train the parameters of each encoder with the goal of minimizing the reconstruction error.

[0084] (2) Dual graph convolution distillation is performed on the multimodal shared feature vectors, vision-exclusive feature vectors, speech-exclusive feature vectors, and text-exclusive feature vectors of the multimodal decoupled representation. Through knowledge distillation, information from different perspectives is used to promote and distill each other to obtain a more robust and effective representation.

[0085] 1) Construct homology graph and heterology graph: Each video clip is set as a node, and the shared features are used as the node feature vector to construct a homology graph of an undirected complete graph; the exclusive features extracted from each modality on each video clip are used as a node to construct a heterology graph of a bipartite graph; the edge weights of the homology graph are calculated using cosine similarity to reflect the similarity of shared features between nodes; the edge weights of the heterology graph are calculated using the attention mechanism to reflect the complementarity of exclusive features between nodes of different modalities.

[0086] 2) Apply graph convolutional network (GCN) on homology graphs to realize the propagation of shared features between modal nodes: the graph convolution operation aggregates the features of each node with the features of its neighboring nodes and updates the learnable parameters of the modal nodes.

[0087] 3) Applying the graph attention network (GAT) on heterogeneous graphs to achieve distillation of exclusive features between modalities: The graph attention operation adjusts the learnable parameters of nodes according to the attention weights between nodes.

[0088] 4) Calculate the loss on the homologous graph and heterologous graph separately: take the emotional label of each video clip as the category label of the node, on the homologous graph, input the distilled shared features into a fully connected layer for emotional classification, and calculate the cross entropy loss; on the heterologous graph, input the exclusive features of each modality after distillation into a fully connected layer for emotional classification, and calculate the average cross entropy loss.

[0089] 5) Jointly optimize the loss functions on homologous graphs and heterologous graphs: add the node classification loss of the homologous graph and the node classification loss of the heterologous graph as the total classification loss; introduce a consistency loss in addition to the total classification loss to encourage the homologous graph and the heterologous graph to reach a consensus on the node classification task; use the Adam optimizer to train the parameters of GCN and GAT with the weighted sum of the total classification loss and the consistency loss as the goal.

[0090] 6) Dual graph convolution distillation is performed on the multimodal shared feature vector, visual exclusive feature vector, speech exclusive feature vector, and text exclusive feature vector of the multimodal decoupled representation, which is similar to the role of the encoder in extracting features. The exclusive features obtained after distillation are: 128 dimensions for visual exclusive features, 128 dimensions for speech exclusive features after distillation, and 512 dimensions for text exclusive features after distillation.

[0091] (3) Multimodal sentiment classification is performed on the distilled multimodal shared feature vectors, vision-exclusive feature vectors, speech-exclusive feature vectors, and text-exclusive feature vectors. Sentiment classification uses 7 categories to ensure the fine-grainedness of sentiment recognition.

[0092] 1) Concatenate the original multimodal features with the shared features and exclusive features after distillation to synthesize a multimodal sentiment feature vector: sequentially concatenate the original multimodal features into 1024 dimensions, the shared features into 256 dimensions, the visual exclusive features into 128 dimensions, the speech exclusive features into 128 dimensions, and the text exclusive features into 512 dimensions into a new 2048-dimensional multimodal sentiment feature vector.

[0093] 2) Input the multimodal sentiment feature vector into the classifier for classification: The classifier consists of three fully connected layers (2048->1024->512->7) and a softmax layer.

[0094] 3) Using cross entropy as the loss function, calculate the difference between the predicted probability of the classifier and the true label. Jointly optimize the classification loss and the loss of dual-graph convolution distillation: add the classification loss to the total classification loss and consistency loss to obtain the total training target, and use the Adam optimizer to update the parameters of the classifier, GCN, and GAT to achieve end-to-end training.

[0095] 4) The multimodal sentiment feature vector is input into the classifier to obtain the output probability argmax of the classifier and the final sentiment classification result.

[0096] (4) The visual exclusive features, speech exclusive features, and text exclusive features are respectively used to realize three-modal independent classification, and alpha is introduced to control the loss weight to analyze the modal contribution and assist the sentiment classification of multimodal fusion.

[0097] 1) Build three independent unimodal classifiers: build two visual classifiers with fully connected layers and softmax layers; build two speech classifiers with fully connected layers and softmax layers; build three text classifiers with fully connected layers and softmax layers. The 7-category emotion extraction results include: joy / happiness, anger, sadness, surprise, disgust, fear, neutral or no emotion.

[0098] 2) Input the image, audio, and text exclusive features into the corresponding unimodal classifiers respectively, and calculate argmax to obtain the sentiment classification results under each modality. Calculate the classification cross entropy loss of the three unimodal classifiers, and add the three unimodal classification losses to obtain the total loss of modality independent classification.

[0099] 3) In the overall training objective of multimodal sentiment classification, the modality-independent classification loss is added as a weighted term: an independent weight parameter alpha is introduced to control the proportion of the modality-independent classification loss in the overall training objective.

[0100] (5) Using multimodal sentiment datasets such as training sets, validation sets, and test sets, an end-to-end sentiment classification model consisting of feature extractor, graph convolutional distillation unit, and classifier is trained.

[0101] 1) Batch training is performed on the training set: the size of each batch is set to N = 64, and N video clips are randomly sampled from the training set; the multimodal features of these N video clips are obtained through the multimodal feature extraction module; the multimodal features are sent to the dual-graph convolutional distillation module and the classifier to calculate the total training target, i.e., classification loss + distillation loss + modality-independent classification loss; the model parameters are updated through the Adam optimizer and the back-propagation algorithm to minimize the total training target.

[0102] 2) After each training epoch, evaluate the model's accuracy, F1-score and other performance on the validation set: Use the same performance indicators as those in the test phase, such as accuracy and F1-score, and record the optimal performance of the current model on the validation set and its corresponding parameter status.

[0103] 3) Set up the triggering early stopping mechanism: When a certain number of epochs is greater than or equal to 10 and the performance of the validation set no longer improves, the early stopping mechanism is triggered, training is stopped, and the optimal parameter state is used as the final model parameter; the generalization performance of the model is evaluated on the test set, and the mean and standard deviation of each indicator are calculated and recorded.

[0104] 4) If the performance of the test set is not ideal, consider adjusting hyperparameters such as learning rate, batch size, regularization coefficient, and number of graph convolution / attention layers and retraining: Use automatic parameter adjustment methods such as grid search and Bayesian optimization to find the optimal hyperparameter combination.

[0105] 5) Package the trained model and encapsulate it into a callable interface to ensure the compatibility and robustness of the interface, and give reasonable output for inputs of different formats, lengths, and qualities.

[0106] Step 103, retrieve and filter a set of video clips matching emotions according to emotion-oriented features, optimize the sequence of clips using a minimum spanning tree algorithm and breadth-first traversal to ensure emotional coherence, extract video materials for harmonization and special effects optimization, and synthesize a personalized, emotion-oriented trailer.

[0107] The target emotion category is mapped to an emotion embedding one-hot vector. The keyword list matching the user portrait is embedded using the BERT method to generate a keyword embedding vector. The video clip is extracted through multimodal features and the classifier is used to obtain the reference video emotion embedding vector. The three types of embedding vectors are fused using the fully connected layer and the attention mechanism. The weights are adaptively adjusted to obtain the comprehensive target embedding vector. The multi-source information is fully integrated to improve the representation ability of the emotion classification task, including:

[0108] (1) Map the target emotion category to an emotion embedding one-hot vector as the target emotion embedding vector: set an all-zero vector with the same number as the emotion categories, and set the elements at the corresponding positions to 1 according to the determined target emotion category sequence number; in the one-hot vector, if the target emotion category is determined to be the i-th category among the seven categories, the i-th element is 1, and the rest are set to 0.

[0109] (2) Using the BERT embedding method, each keyword in the keyword list matched by the user profile is converted into an embedding vector of fixed length: each keyword is input into the pre-trained BERT model, and the vector corresponding to the [CLS] tag in the output sequence is extracted as the embedding of the keyword. The BERT dictionary size is fixed. For keywords outside the BERT dictionary, the embedding vector of the [UNK] tag is used instead. The embedding vectors of all keywords in the list are combined into a matrix, with each row corresponding to a keyword embedding.

[0110] (3) Using the multimodal feature extraction module, extract the reference video emotion embedding vector for each reference video: For each reference video, extract one or more key segments, extract the multimodal features of these segments, input the multimodal features of the key segments into the trained multimodal emotion classifier in turn, and extract the penultimate layer output as the emotion embedding vector of the segment. Take the average of the emotion embedding vectors of multiple key segments to obtain the emotion embedding vector of the current reference video. The emotion embedding vectors of all reference videos are combined into a matrix, with each row corresponding to a reference video embedding.

[0111] (4) The target emotion embedding, keyword embedding, and reference video emotion embedding vectors are weighted averaged through attention to obtain a comprehensive target embedding vector: the three types of embedding vectors are mapped to the same dimension through a fully connected layer respectively, and a tanh activation function is applied; the three types of activated embedding vectors are concatenated and input into an attention fully connected layer, and three scalar weights are output; the three types of embedding vectors are multiplied by the corresponding scalar weights, and then summed to obtain a comprehensive target embedding vector.

[0112] Candidate segment retrieval calculates the similarity between the embedding of homologous and heterologous graph nodes and the comprehensive target embedding, and uses the attention mechanism to adaptively weighted fusion to obtain the comprehensive similarity for sorting, and finally selects the Top-K related segments to achieve multi-modal and multi-angle accurate video segment retrieval, including:

[0113] (1) Similarity calculation: Traverse all candidate segments and calculate the similarity between their node embeddings in the homologous graph and heterologous graph and the comprehensive target embedding. The homologous graph similarity is calculated by dot product. The heterologous graph similarity is calculated by calculating the dot product of the node embeddings of the three modalities of vision, speech, and text with the comprehensive target embedding and taking the average value.

[0114] (2) Comprehensive similarity fusion: For each candidate segment, the attention fully connected layer is used to learn two weights based on the homologous graph similarity and heterologous graph similarity. The two weights are used to perform a weighted average of the homologous graph similarity and heterologous graph similarity to obtain the comprehensive similarity of each candidate segment.

[0115] (3) Sorting and retrieval result selection: Sort all candidate segments according to their comprehensive similarity. Select the top-K candidate segments with the highest similarity as the final retrieval results.

[0116] For the top-K candidate segments retrieved, the comprehensive similarity is converted into pairwise similarity using Gaussian kernel function, a fully connected graph is constructed, the minimum spanning tree algorithm is applied to obtain the weight and the minimum tree structure, and a natural and smooth candidate segment sequence is generated through breadth-first traversal to achieve global optimization combination, improve the accuracy and coherence of emotion recognition and video segment retrieval, including:

[0117] (1) Convert the comprehensive similarity of the candidate segments into pairwise similarity and control the concentration of the similarity distribution: Use the Gaussian kernel function to convert the comprehensive similarity into pairwise similarity and control the concentration of the similarity distribution. For candidate segments i and j, their pairwise similarity f(d i ,d j )for:

[0118]

[0119] where d i and d j is the comprehensive similarity of the two.

[0120] (2) Construction of a fully connected undirected graph: With candidate segments as nodes and pairwise similarities as edge weights, a fully connected undirected graph is constructed, considering the relationships among all candidate segments from a global perspective.

[0121] (3) Apply the minimum spanning tree (MST) algorithm on the undirected graph to obtain the spanning tree with the minimum weight and extract the key skeleton structure: initialize an empty tree, randomly select a candidate segment as the initial node, and each time select the edge with the smallest weight from the edges connected to the current tree, and add its corresponding candidate segment to the tree. Repeat this process until all candidate segments are added to the tree.

[0122] Perform a breadth-first traversal of the minimum spanning tree to obtain a candidate segment sequence with high similarity and emotional fluctuations, ensuring global coherence and local emotional diversity: Perform a breadth-first traversal BFS on the MST to obtain a candidate segment sequence with high similarity and emotional fluctuations. Starting from the initial node, traverse the MST layer by layer, and the nodes in the same layer are arranged in the order of joining the tree, and the child nodes of the nodes in the previous layer follow closely to form a sequence.

[0123] Calculate the candidate segment sequence obtained by global candidate combination, calculate the candidate sequence emotion embedding, determine the dominant emotion; adjust the sequence order according to the dominant emotion; initialize the recommended sequence, filter the segments according to the duration threshold; output the optimized sequence, including:

[0124] (1) Calculate the average sentiment embedding vector of the candidate sequence: average the homologous node embeddings of each candidate segment in the sequence to obtain the homologous embedding at the sequence level; do a dot product between the homologous embedding and the comprehensive target embedding to obtain the average sentiment intensity of the sequence.

[0125] (2) Compare the average emotion intensity with the typical intensity value corresponding to the emotion category, and select the closest emotion category as the dominant emotion: For each emotion category, calculate its average emotion intensity in the training set as the typical intensity value, subtract the average emotion intensity of the sequence from all typical intensity values, and take the emotion category with the smallest absolute value as the dominant emotion.

[0126] (3) Adjust the time order of the sequence according to the dominant emotion so that it conforms to the law of emotional change: for positive dominant emotions, arrange the segments in ascending order according to the emotional intensity so that the emotion gradually increases; for negative dominant emotions, arrange the segments in descending order according to the emotional intensity so that the emotion gradually decreases.

[0127] (4) Initialize an empty recommendation sequence to store the final selected segments: set a duration threshold T, which represents the upper limit of the target video duration, traverse the adjusted candidate sequence, and add the segments to the recommendation sequence one by one; record the cumulative duration of the recommendation sequence. If the duration exceeds T after adding the current segment, stop traversal.

[0128] (5) The final recommendation sequence is used as the output of timing optimization and filtering.

[0129] Extract the final sequence video material, add fade-in and fade-out effects, overlay the dominant emotional color filter, and splice and export the complete video, including:

[0130] (1) According to the final sequence timestamp, extract the corresponding video material from the original video library.

[0131] (2) Add fade-in and fade-out effects at the junction of adjacent video clips to achieve a smooth transition: set the fade-in and fade-out time length t, and linearly reduce the transparency of the frame of the previous clip in the last t seconds until it is completely transparent; linearly increase the transparency of the frame of the next clip in the first t seconds until it is completely opaque, and connect the two clips after the fade-in and fade-out end to end on the timeline to achieve a smooth transition.

[0132] (3) A color filter related to the dominant emotion is superimposed on each video clip to enhance the emotional tone.

[0133] (4) All video clips with special effects added are stitched together into a complete video file in sequence and exported as a complete trailer video in MP4 format.

[0134] The present invention proposes a recommendation system trailer intelligent generation system and method based on multimodal distillation sentiment analysis, which is an important innovation of artificial intelligence technology in the field of recommendation system and content generation. It not only combines advanced technologies in multiple fields such as natural language processing (NLP) and computer vision (CV), but also realizes efficient compression and performance improvement of the model through knowledge distillation technology. This technological innovation not only promotes the development of artificial intelligence technology, but also provides new ideas and methods for research in related fields.

[0135] Beneficial effects of this application:

[0136] (1) The system is able to generate highly personalized trailer content by deeply analyzing users’ emotional preferences and multimodal information.

[0137] (2) This system uses multimodal distillation sentiment analysis technology to comprehensively capture users’ interests and emotional tendencies, thereby improving the accuracy of the recommendation algorithm.

[0138] (3) This system can intelligently generate trailers, reducing the cost and time of manual production and making content production more efficient.

[0139] (4) By generating trailers that are closely related to user emotions, the system can stimulate user engagement and discussion enthusiasm. BRIEF DESCRIPTION OF THE DRAWINGS

[0140] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0141] Figure 1 This is a schematic diagram of the intelligent trailer generation system for recommendation systems based on multimodal distillation sentiment analysis;

[0142] Figure 2 is a schematic diagram of a multimodal video preprocessing module according to an embodiment of the present application;

[0143] Figure 3 is a schematic diagram of a multimodal video feature recognition module according to an embodiment of the present application;

[0144] Figure 4 It is a schematic flowchart of a method for intelligently generating trailers for a recommendation system based on multimodal distillation sentiment analysis guidance according to an embodiment of the present application. DETAILED DESCRIPTION

[0145] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0146] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0147] With the continuous development of radio, television and audio-visual services, users' demand for content is becoming increasingly diversified and personalized. As a bridge connecting users and content, the accuracy and intelligence level of the recommendation system are directly related to user experience and platform benefits. Current recommendation systems often ignore the emotional elements in program content, resulting in the inability of recommended clips to accurately convey the emotional atmosphere of the program and the emotional resonance points of the audience, affecting the attractiveness and accuracy of the recommendation. Therefore, it is urgent to develop a recommendation system trailer intelligent generation system and method based on multimodal distillation sentiment analysis to improve user experience. Traditional recommendation systems mainly rely on users' historical behavior data or single-modal information (such as text, images, etc.), which greatly limits the breadth and depth of recommendations. The method based on multimodal distillation sentiment analysis can integrate rich information from different modalities (such as text, images, sounds, etc.), eliminate redundancy and noise through distillation technology, and extract the most critical emotional features. This multi-dimensional sentiment analysis can not only understand user needs more accurately, but also capture users' potential interests, thereby greatly improving the accuracy of the recommendation system.

[0148] In the field of entertainment content such as videos, movies, and TV series, trailers are an important means to attract users to watch. However, the production of traditional trailers often requires a lot of manpower and material resources, and it is difficult to accurately match user preferences. The recommendation system guided by multimodal distillation sentiment analysis can automatically analyze users' emotional preferences and intelligently generate trailer content that matches users' interests. This personalized trailer can not only effectively improve users' willingness to watch, but also enhance the stickiness between users and the platform.

[0149] According to one aspect of the embodiment of the present application, Figure 1 As shown, a recommendation system trailer intelligent generation system based on multimodal distillation sentiment analysis guidance may include:

[0150] The multimodal video preprocessing module completes the video decoding and track division of the original video file, and splits the video into text mode, voice mode and visual mode according to the shot, and further cleans and aligns the video mode data to obtain multimodal preprocessed data;

[0151] The multimodal video feature recognition module extracts features from preprocessed data in text, speech, and visual modes and synthesizes multimodal feature vectors, performs decoupled distillation multimodal emotion recognition and classification, and performs model training.

[0152] The emotion-oriented video synthesis module retrieves and filters a set of video clips matching emotion features according to emotion-oriented features, forms a global candidate combination that satisfies the emotion similarity, further performs timing optimization and filtering to complete the emotion-oriented video synthesis, harmonizes and optimizes special effects on the synthesized video, and generates emotion-oriented trailers for intelligent editing of long videos.

[0153] like Figure 2 As shown, in some embodiments, the multimodal video preprocessing module may include:

[0154] Optionally, the input original video file is decapsulated using the FFmpeg library, and the audio stream and the video stream therein are obtained by dividing the tracks, wherein the original video includes but is not limited to MP4 format files.

[0155] Optionally, in multimodal splitting, the text modality is obtained by using, including but not limited to, Baidu AI Chinese speech recognition API, the audio modality is obtained by using short-time energy (STE) and short-time zero-crossing rate (STZCR) of the speech signal for voice activity detection (VAD), and the visual modality is obtained by using shot transition detection (SBD) of changes in similarity between video frames, and the three modalities are optimized and adjusted.

[0156] Optionally, during data cleaning and alignment, according to the timestamp of each segment recorded in the segmentation log, video segments and segmentation logs that are too short (such as 1 second) or too long (such as more than 60 seconds) are filtered out, invalid audio segments and segmentation logs whose voice ratio is less than 15% of a preset threshold are filtered out, and the text is segmented and stop words and segmentation logs are filtered out using tools including but not limited to jieba segmentation tools, so as to complete the cleaning and alignment of video, audio, and text multimodal data.

[0157] like Figure 3 As shown, in some embodiments, the multimodal video feature recognition module may include:

[0158] Optionally, multimodal feature extraction, using including but not limited to OpenCV library to read video frame sequence, and using including but not limited to cascaded CNN structure of MTCNN face detection model to detect face frame by frame, using including but not limited to pre-trained FaceNet model to extract the aligned face image features of each frame, and generate visual feature vector; using including but not limited to librosa library to read audio waveform sequence, convert it to frequency domain through fast Fourier transform (FFT) and filter it by Mel scale filter bank, further through DCT discrete cosine transform (DCT) to obtain VGGish features, and generate audio feature vector; pre-processing the intercepted subtitle text, using including but not limited to jieba word segmentation tool to segment the text into word sequence, and extracting text feature vector through pre-trained including but not limited to Chinese BERT model.

[0159] Optionally, in multimodal feature splicing, the video, audio, and text modal feature vectors are spliced ​​in the order of the modalities, and serialized and formatted in formats including but not limited to TFRecord or HDF5 to complete the multimodal feature splicing.

[0160] Optionally, decouple distillation for multimodal emotion recognition and model training, build a shared encoder and three exclusive encoders to construct a multimodal decoupled representation, and use including but not limited to Adam optimizer to train the parameters of each encoder; construct a homologous graph including but not limited to graph convolutional network (GCN) to realize the propagation of shared features between nodes, construct a heterologous graph including but not limited to attention network (GAT) to realize the distillation of exclusive features between modalities, and use Adam optimizer to train the parameters of GCN and GAT with the weighted sum of total classification loss and consistency loss as the goal; input the multimodal emotion feature vector into the classifier for classification, and use cross entropy as the loss function to jointly optimize the classification loss and the loss of dual graph convolution distillation; train multimodal emotion datasets including but not limited to training set, validation set and test set to obtain an end-to-end emotion classification model including but not limited to feature extractor, graph convolution distillation unit and classifier.

[0161] Figure 4 It is a schematic flowchart of a method for intelligently generating trailers for a recommendation system based on multimodal distillation sentiment analysis guidance provided in an embodiment of the present application.

[0162] The intelligent generation method of trailers for recommendation systems based on multimodal distillation sentiment analysis may include the following steps:

[0163] Step S101, complete the video decoding and track division of the original video file, and split the video into text, voice, and visual tri-modal data by shot, and ensure the cleaning and alignment of the multi-modal data to obtain multi-modal pre-processed data.

[0164] In the embodiment of the present invention, the multimodal video preprocessing module first performs video decoding and audio and video track separation on the original video file, including:

[0165] (1) Decapsulate the input original video file using a library including but not limited to FFmpeg to obtain the audio stream and video stream therein.

[0166] (2) The audio stream is re-encoded, converted to a format including but not limited to PCM, and uniformly resampled to illustrative 16kHz, 16-bit mono audio data, and saved as a file including but not limited to WAV format. In some embodiments, the phonemes and vocabulary of Chinese speech are larger than those of English, and the sampling rate of speech recognition and synthesis can be appropriately reduced to balance data volume and quality.

[0167] (3) Re-encode the video stream, convert it into a format including but not limited to H.264, and uniformly adjust the frame rate to an exemplary 25fps, and save it into a file including but not limited to MP4 format. In some embodiments, the exemplary 25fps video frame rate is more consistent with the visual characteristics of the human eye and is compatible with common industry specifications.

[0168] In the embodiment of the present invention, the divided audio stream and video stream are split into a multimodal splitting of text mode, voice mode and visual mode according to the shot. The specific steps of splitting the video into multimodal data according to the shot are detailed below.

[0169] (1) Split text mode: Call including but not limited to Baidu AI Chinese speech recognition API, transcribe the audio sentence by sentence, and obtain the machine timestamp of the speech recognition result ending with punctuation.

[0170] 1) Divide the recognition results into multiple sentences according to the punctuation positions, and record the start and end timestamps of each sentence.

[0171] 2) Record the start and end timestamps of each sentence as candidate cutpoints.

[0172] (2) Split speech modes: Voice activity detection (VAD) is performed using the short-time energy (STE) and short-time zero-crossing rate (STZCR) of the speech signal.

[0173] 1) The audio signal is segmented at the frame level, where each frame is schematically segmented into 20ms and the frame shift is schematically segmented into 10ms.

[0174] 2) Calculate the STE and STZCR features for each frame and use the double threshold method to make speech / non-speech judgments to obtain the start and end timestamps of the speech event.

[0175] 3) The boundaries of the speech events, that is, the speech start and end points, are marked as timestamps candidate cutpoints.

[0176] (3) Split visual modality: Shot transition detection (SBD) is performed using the changes in similarity between video frames.

[0177] 1) Segment the video frames and schematically extract a key frame every 0.5 seconds (12 frames).

[0178] 2) The SURF feature matching method is used to compare the similarity of adjacent key frames. When the number of matching points is less than the schematic preset threshold of 50, it is determined to be a shot boundary.

[0179] 3) The timestamps of the shot boundaries are recorded as shots candidate cutpoints, where each shot has multiple dialogues and shots candidate cutpoints contains multiple candidate cutpoints.

[0180] 4) The segmented video clip files are saved in a format including but not limited to MP4.

[0181] (4) Determine candidate cutpoints for other modes based on the audio cutpoints:

[0182] The final_cuts sorted by timestamp is the final list of cut points.

[0183] (5) Traverse all the segmented video segments: extract the start and end timestamps, end timestamps, and duration of each segment, and save them in the video segmentation log file (such as cut_info.json).

[0184] Schematically, the speed and pause characteristics of Chinese speech are different from those of English. When detecting speech activity, the frame length and frame shift can be appropriately reduced. Considering the shooting characteristics of Chinese video scenes, the time interval for lens switching judgment can be shortened and the threshold can be relaxed. In the comprehensive optimization of multimodal segmentation points, the text mode can be used as the main mode to ensure semantic integrity, while appropriate reference to speech and visual information can improve the accuracy of segmentation.

[0185] In the embodiment of the present invention, the specific steps of performing data cleaning and alignment on the segmented video clip files and the segmented log files are provided.

[0186] (1) According to the candidate cutpoints timestamps of each segment recorded in the segmentation log, mainly extract the voice timestamp, use including but not limited to FFmpeg library to extract the audio and video of the corresponding segment from the original video, and save them as independent files including but not limited to WAV and MP4 files respectively.

[0187] (2) Traverse all the segmented video clips, calculate their duration, filter out video clips that are too short (e.g., less than 1 second) or too long (e.g., greater than 60 seconds), delete them from the segmentation log, and delete the corresponding audio, video, and text files.

[0188] (3) Perform voice activity detection (VAD) on the audio of each segment and calculate the voice ratio:

[0189]

[0190] The voice segments with a percentage lower than the exemplary preset threshold of 15% are filtered out and determined as invalid segments, which are deleted from the segmentation log and the corresponding audio and video files are deleted.

[0191] (4) Using a Chinese speech recognition API including but not limited to Baidu AI, transcribe the audio of each segmented segment to obtain a text of the speech content.

[0192] (5) Post-process the speech recognition results. Use tools including but not limited to the Jieba word segmentation tool to segment the text and filter stop words to obtain the syntactic structure information of the segment. The exemplary Jieba word segmentation tool and the Peking University dependency syntax analysis tool are both commonly used tools in Chinese NLP tasks and can provide richer speech information for text feature extraction.

[0193] (6) In the segmentation log, the segmentation log file cut_info.json of each segment is updated to record meta information including but not limited to the final start and end time, duration, text, keywords, etc. of each segment.

[0194] Step 102, complete the feature extraction of the preprocessed data of the text modality, speech modality and visual modality respectively, and splice the three modal features into a multimodal integrated vector. Decoupled distillation is performed through the shared encoder and the exclusive encoder, combined with dual-graph convolution distillation and model training, and finally fine-grained emotion recognition is achieved through the classifier.

[0195] In the embodiment of the present invention, multimodal feature extraction is performed on the pre-processed multimodal data and the segmented log, including:

[0196] (1) Extracting visual modal features: Performing visual modal feature extraction on the preprocessed visual modal data and segmentation logs to obtain an exemplary 128-dimensional visual feature vector.

[0197] 1) Using a library including but not limited to OpenCV to read an exemplary MP4 format video clip file, and extracting a corresponding video frame sequence according to a start timestamp and an end timestamp.

[0198] 2) For the captured video frame sequence, use a face detection model including but not limited to MTCNN to detect faces frame by frame: MTCNN uses a cascaded CNN structure to simultaneously complete face detection and key point positioning. The confidence threshold of face detection is 0.7, and detection results above the confidence threshold are retained. If multiple faces are detected in a frame, the one with the highest confidence is selected as the main face of the frame.

[0199] The MTCNN model is selected instead of traditional face detectors such as Haar and HOG. The main consideration is the model's ability to complete detection and alignment end-to-end, and its advantages such as high accuracy and fast detection speed.

[0200] 3) Align the main face of each frame according to the key point coordinates detected by MTCNN: The purpose of the exemplary face alignment includes but is not limited to eliminating factors such as posture, size, and position to extract highly robust facial features. Exemplary face alignment methods that can be selected include but are not limited to affine transformation, similarity transformation, and normalize the key points to the calibration position. The size of the exemplary aligned face image is adjusted to 160*160 to match the input requirements of the FaceNet model selected in the example.

[0201] 4) Extract the features of the aligned face images of each frame using pre-trained models including but not limited to FaceNet: The FaceNet model is trained using a pairwise triplet loss function to map faces to an exemplary 128-dimensional Euclidean space, in which the Euclidean distance between different faces is inversely proportional to their similarity, i.e., the closer the distance, the more similar the faces. The exemplary 128-dimensional feature vector output by FaceNet can well represent the semantic information of the face.

[0202] FaceNet is selected instead of other face recognition models such as DeepFace and DeepID, mainly considering its advantages in recognition accuracy and generalization ability.

[0203] 5) For a shot, the facial feature vectors extracted from each frame are arranged in chronological order to form a new feature sequence, where the dimension is (N, 128), where N is the number of frames contained in a shot.

[0204] 6) Perform statistical analysis on the feature sequence in the time dimension, calculate the mean and standard deviation of each dimension of the feature in the sequence, and obtain an exemplary 256-dimensional feature vector to represent the visual modality feature of the entire video clip. For example, performing statistics on the time dimension of the frame-level feature sequence can fully explore the dynamic change information of facial features and make the clip-level features more representative.

[0205] (2) Extracting speech modal features: performing speech modal feature extraction on the preprocessed speech modal data and segmentation logs to obtain an exemplary visual feature vector.

[0206] 1) Use, including but not limited to, librosa library to read an exemplary WAV format audio clip file, and extract a corresponding audio waveform sequence according to a start timestamp and an end timestamp.

[0207] 2) Pre-emphasize and frame the intercepted audio waveform sequence: Pre-emphasis is used to balance the energy of high-frequency and low-frequency components, and a first-order high-pass filter with a coefficient of 0.97 is selected as an example. Frame division is used for short-term analysis, and the frame length is set to 0.96s and the frame shift is set to 1.5s to ensure sufficient time resolution and a certain overlap for smooth transition.

[0208] 3) Feature extraction for each frame of pre-emphasized audio data: First, the pre-emphasized audio waveform sequence is converted to a frequency domain audio sequence by fast Fourier transform (FFT), and then the Mel-scale filter bank is used to filter the Mel spectrum to obtain audio data that is closer to the auditory characteristics of the human ear. The Mel spectrum is logarithmic and discrete cosine transformed (DCT) is performed to obtain an exemplary VGGish feature vector.

[0209] 4) Further, the VGGish feature vectors extracted from each frame are arranged in chronological order to form a new feature sequence, where the dimension is (N, 128), where N is the number of frames contained in a shot.

[0210] 5) The feature sequence generated in time sequence is pooled in the time dimension, the mean of each dimension of the feature in the sequence is calculated, and an exemplary 128-dimensional mean vector is obtained to represent the speech modal features of the entire audio clip.

[0211] (3) Extracting text modal features: Performing text modal feature extraction on the preprocessed text modal data and segmented logs to obtain an exemplary text feature vector.

[0212] 1) Read an exemplary UTF-8 encoded TXT format text segment file, and extract the corresponding subtitle text content according to the start timestamp and the end timestamp.

[0213] 2) Preprocess the extracted subtitle text: Use the jieba word segmentation tool to segment the text into word sequences. Convert the word sequences into the input format required by the BERT model, including adding [CLS] and [SEP] tags and performing WordPiece word segmentation.

[0214] The Jieba word segmentation tool was selected to optimize the Chinese corpus, mainly considering its moderate word segmentation granularity and the ability to customize the dictionary.

[0215] 3) Input the preprocessed text into the exemplary Chinese BERT model selected for pre-training: The BERT model can effectively model the contextual information of the text based on the bidirectional language model of the Transformer structure. The Chinese version of the BERT-base-uncased model is selected as an example, which contains 12 layers of Transformer and the hidden layer dimension is 768. The model input is set as a tokenized word sequence, and the output is the context-aware embedding vector corresponding to each token.

[0216] The Chinese BERT model is selected instead of traditional word embedding models such as word2vec and GloVe, mainly because the model supports modeling two-way interactions between words, and the generated embedding vector contains richer semantic information.

[0217] 4) Extract the output of the last layer of Transformer of BERT as the embedding vector of each word to obtain a word embedding matrix. For example, extract the output of the last layer of BERT instead of the output of the previous layer, mainly considering that the last layer integrates the information of all previous layers, and the final context-aware representation can be obtained.

[0218] 5) Pooling is performed on the word embedding matrix, and MaskAveraging is selected as an example: a mask matrix of the same shape is constructed based on the input tokenization sequence of BERT, in which the corresponding position of the word is set to 1 and the corresponding position of the padding mark ([PAD]) is set to 0. Further, the word embedding matrix and the mask matrix are Hadamard product processed (i.e., multiplied element by element) to obtain an embedding matrix without the influence of the padding mark. Further, the mean of the unpadding embedding matrix in the word dimension is solved to obtain an exemplary 768-dimensional sentence vector to represent the text modality characteristics of the entire text segment.

[0219] Selecting Mask Averaging instead of simple average pooling can avoid the noise introduced by padding markers such as [PAD] and make the sentence vector reflect the text content more accurately.

[0220] In an embodiment of the present invention, the extracted multimodal feature data is synthesized into a multimodal integrated vector, including:

[0221] (1) The feature vectors extracted from the three modalities of image, audio, and text are concatenated into a multimodal integrated vector in the order of image, audio, and text modalities: For example, a 128-dimensional visual feature vector, a 128-dimensional speech feature vector, and a 768-dimensional text feature vector are sequentially synthesized to obtain a 1024-dimensional multimodal feature integrated vector.

[0222] (2) The multimodal feature integration vector and the corresponding video segment metadata are packaged to form a data unit. Exemplary metadata includes but is not limited to video ID, start timestamp, end timestamp, duration, etc.

[0223] (3) Arrange the data units of all video segments in the order of video ID and timestamp to form a video segment set.

[0224] (4) Serialize the video clip collection in a format including but not limited to TFRecord, HDF5, etc., and save it to a local disk or a cloud database. For example, when saving multimodal feature data, the general and efficient TFRecord serialization format is preferred, which can improve the speed of data reading and loading and facilitate calling by different deep learning frameworks.

[0225] In an embodiment of the present invention, the multimodal emotion recognition by performing decoupled distillation on the synthesized multimodal integrated vector includes:

[0226] (1) Construct a multimodal decoupled representation for an exemplary 1024-dimensional multimodal feature integration vector.

[0227] 1) Build a shared encoder and three exclusive encoders for the synthetic multimodal integrated vector and the three modal feature vectors of image, audio, and text respectively.

[0228] Exemplarily, the shared encoder receives a 1024-dimensional synthetic multimodal integrated vector, which is mapped to a 256-dimensional shared feature space through two fully connected layers (1024->512->256); the visual exclusive encoder receives a 128-dimensional visual feature vector, which is mapped to a 128-dimensional visual exclusive feature space through two fully connected layers (128->128->128); the speech exclusive encoder receives a 128-dimensional speech feature vector, which is mapped to a 128-dimensional speech exclusive feature space through two fully connected layers (128->128->128); the text exclusive encoder receives a 768-dimensional text feature vector, which is mapped to a 512-dimensional text exclusive feature space through two fully connected layers (768->512->512).

[0229] Exemplarily, the shared encoder is used to extract common information between different modalities, and the exclusive encoder is used to extract information specific to each modality, which complement each other and help to better understand the intrinsic structure of multimodal data; the shared encoder learns the common representation of multimodal data through reconstruction tasks, but unlike traditional autoencoders, the present invention introduces modality-exclusive features as auxiliary information; the exclusive encoder highlights the features in each modality that are most relevant to the emotion recognition task through weighted aggregation to compress redundant information.

[0230] 2) For the three exclusive encoders of vision, speech, and text modalities, an attention pooling layer is connected to the last layer of each exclusive encoder: For example, the attention pooling layer of each exclusive encoder compresses the exclusive features into a scalar weight through a fully connected layer (128->1 or 512->1). The exclusive features are multiplied by the corresponding scalar weights to obtain the weighted exclusive features, and the weighted exclusive features are summed in the dimensions of vision, speech, and text modal features to obtain an aggregated feature vector, the dimension of which is 1024.

[0231] 3) With reconstruction error as the goal, train the shared encoder and the exclusive encoder: concatenate the shared feature vector and the aggregated exclusive feature vector, and reconstruct an exemplary 1024-dimensional multimodal feature vector through a fully connected layer; take the mean square error between the reconstructed multimodal feature and the original multimodal feature integration vector as the reconstruction error; with the goal of minimizing the reconstruction error, use methods including but not limited to the Adam optimizer to train the parameters of each encoder.

[0232] (2) Dual graph convolution distillation is performed on the multimodal shared feature vectors, vision-exclusive feature vectors, speech-exclusive feature vectors, and text-exclusive feature vectors of the multimodal decoupled representation. Through knowledge distillation, information from different perspectives is used to promote and distill each other to obtain a more robust and effective representation.

[0233] 1) Construct homology graph and heterology graph: Each video clip is set as a node, and the shared features are used as the node feature vector to construct a homology graph of an undirected complete graph; the exclusive features extracted from each modality on each video clip are used as a node to construct a heterology graph of a bipartite graph; the edge weights of the homology graph are calculated using cosine similarity to reflect the similarity of shared features between nodes; the edge weights of the heterology graph are calculated using the attention mechanism to reflect the complementarity of exclusive features between nodes of different modalities.

[0234] 2) Apply graph convolutional networks (GCNs) on homology graphs to achieve the propagation of shared features between modal nodes: the graph convolution operation aggregates the features of each node with the features of its neighboring nodes and updates the learnable parameters of the modal nodes.

[0235] 3) Applying Graph Attention Network (GAT) on heterogeneous graphs to achieve distillation of exclusive features between modalities: The graph attention operation adjusts the learnable parameters of nodes according to the attention weights between nodes.

[0236] 4) Calculate the loss on the homologous graph and heterologous graph separately: take the emotional label of each video clip as the category label of the node, on the homologous graph, input the distilled shared features into a fully connected layer for emotional classification, and calculate the cross entropy loss; on the heterologous graph, input the exclusive features of each modality after distillation into a fully connected layer for emotional classification, and calculate the average cross entropy loss.

[0237] 5) Jointly optimize the loss functions on the homologous graph and the heterologous graph: add the node classification loss of the homologous graph and the node classification loss of the heterologous graph as the total classification loss; introduce a consistency loss in addition to the total classification loss to encourage the homologous graph and the heterologous graph to reach a consensus on the node classification task; take the weighted sum of the total classification loss and the consistency loss as the goal, use including but not limited to the Adam optimizer to train the parameters of GCN and GAT.

[0238] The preferred consistency loss can alleviate the problem of inconsistent node representations learned by homologous graphs and heterologous graphs, and promote the two graphs to reach consensus on downstream tasks.

[0239] 6) Dual graph convolution distillation is performed on the multimodal shared feature vector, visual exclusive feature vector, speech exclusive feature vector, and text exclusive feature vector of the multimodal decoupled representation, which is similar to the role of the encoder in extracting features. The exclusive features obtained after distillation are: visual exclusive features (128 dimensions), speech exclusive features after distillation (128 dimensions), and text exclusive features after distillation (512 dimensions).

[0240] (3) Multimodal sentiment classification is performed on the distilled multimodal shared feature vectors, vision-exclusive feature vectors, speech-exclusive feature vectors, and text-exclusive feature vectors. Sentiment classification uses 7 categories to ensure the fine-grainedness of sentiment recognition.

[0241] 1) In order to simultaneously utilize the original representation and refined representation of multimodal data and give play to their complementary advantages, the original multimodal features are concatenated with the shared features and exclusive features after distillation to synthesize a multimodal sentiment feature vector: For example, the original multimodal features (1024 dimensions), shared features (256 dimensions), visual exclusive features (128 dimensions), speech exclusive features (128 dimensions), and text exclusive features (512 dimensions) are sequentially concatenated into a new 2048-dimensional multimodal sentiment feature vector.

[0242] 2) Input the multimodal sentiment feature vector into the classifier for classification: For example, the classifier consists of three fully connected layers (2048->1024->512->7) and a softmax layer. The fully connected layer completes dimensionality reduction and feature transformation, extracts high-level semantic information, and the softmax layer maps the feature space to the category space and outputs the posterior probability of each category.

[0243] Preferably, the classifier composed of three fully connected layers has strong nonlinear transformation ability and can learn the complex mapping relationship from the original space to the category space.

[0244] 3) Using cross entropy as the loss function, calculate the difference between the predicted probability of the classifier and the true label. Jointly optimize the classification loss and the loss of dual-graph convolution distillation: For example, the classification loss is weightedly added to the total classification loss and consistency loss to obtain the overall training target, and the parameters of the classifier, GCN, and GAT are updated using the Adam optimizer to achieve end-to-end training.

[0245] Preferably, the classification loss and distillation loss are jointly optimized so that the two objectives (classification and feature extraction) in the training process promote and constrain each other, which helps to obtain a more robust model. End-to-end training can realize direct backpropagation of the loss function, while adjusting the front-end feature extractor and the back-end classifier to avoid information loss caused by independent optimization.

[0246] 4) The multimodal sentiment feature vector is input into the classifier to obtain the output probability argmax of the classifier and the final sentiment classification result.

[0247] (4) The visual exclusive features (128 dimensions), speech exclusive features (128 dimensions), and text exclusive features (512 dimensions) are respectively used to realize three-modal independent classification, and alpha is introduced to control the loss weight to analyze the modal contribution and assist in the sentiment classification of multimodal fusion.

[0248] 1) Build three independent unimodal classifiers: For example, build two fully connected layers (128->64->7) and a softmax layer for visual classifiers; build two fully connected layers (128->64->7) and a softmax layer for speech classifiers; build three fully connected layers (512->256->64->7) and a softmax layer for text classifiers. The 7-category emotion extraction results include but are not limited to: joy (happy), anger, sadness, surprise, disgust, fear, neutral (or no emotion).

[0249] Preferably, the introduction of a modality-independent classification task can add a regularization term to the multimodal classification task, thereby alleviating overfitting.

[0250] 2) Input the image, audio, and text exclusive features into the corresponding unimodal classifiers respectively, and calculate argmax to obtain the sentiment classification results under each modality. Calculate the classification cross entropy loss of the three unimodal classifiers, and add the three unimodal classification losses to obtain the total loss of modality independent classification.

[0251] Preferably, the independent classification results of the three modalities can be used to compare and analyze the contributions of different modalities to the sentiment classification task, and can also be used as a reference for confidence estimation.

[0252] 3) In the overall training objective of multimodal sentiment classification, the modality-independent classification loss is added as a weighted term: For example, an independent weight parameter alpha is introduced to control the proportion of the modality-independent classification loss in the overall training objective. For example, the value of Alpha is generally much smaller than 1 to reflect the auxiliary role of the modality-independent classification task.

[0253] (5) Using multimodal sentiment datasets including but not limited to training sets, validation sets, and test sets, an end-to-end sentiment classification model including but not limited to feature extractors, graph convolutional distillation units, and classifiers is trained.

[0254] 1) Perform batch training on the training set: illustratively, the size of each batch is set to N=64, and N video clips are randomly sampled from the training set; the multimodal features of these N video clips are obtained through the multimodal feature extraction module; the multimodal features are sent to the dual-graph convolutional distillation module and the classifier to calculate the total training target (classification loss + distillation loss + modality-independent classification loss); the model parameters are updated through the Adam optimizer and the back-propagation algorithm to minimize the total training target.

[0255] 2) After each training epoch, the model's performance, including but not limited to accuracy, F1-score, etc., is evaluated on the validation set: For example, the same performance indicators, including but not limited to accuracy, F1-score, etc. as in the test phase are used, and the optimal performance of the current model on the validation set and its corresponding parameter status are recorded.

[0256] 3) Set up the triggering early stopping mechanism: For example, when the performance of the validation set no longer improves after a certain number of epochs greater than or equal to 10, the early stopping mechanism is triggered, training is stopped, and the optimal parameter state is used as the final model parameter; the generalization performance of the model is evaluated on the test set, and the mean and standard deviation of each indicator are calculated and recorded.

[0257] 4) If the performance of the test set is not ideal, consider adjusting hyperparameters including but not limited to learning rate, batch size, regularization coefficient, number of graph convolution / attention layers, and retraining: For example, automatic parameter tuning methods including but not limited to grid search and Bayesian optimization can be used to find the optimal hyperparameter combination.

[0258] 5) Package the trained model and encapsulate it into a callable interface to ensure the compatibility and robustness of the interface, and give reasonable output for inputs of different formats, lengths, and qualities.

[0259] Step 103, retrieve and filter a set of video clips matching emotions according to emotion-oriented features, optimize the sequence of clips using a minimum spanning tree algorithm and breadth-first traversal to ensure emotional coherence, extract video materials for harmonization and special effects optimization, and synthesize a personalized, emotion-oriented trailer.

[0260] In the embodiment of the present invention, the target emotion category is mapped to an emotion embedding one-hot vector, the keyword list matching the user portrait is embedded using the BERT embedding method to generate a keyword embedding vector, and the video clip is extracted through multimodal features and a classifier to obtain a reference video emotion embedding vector, and the three types of embedding vectors are fused using a fully connected layer and an attention mechanism, and the weights are adaptively adjusted to obtain a comprehensive target embedding vector, which fully integrates multi-source information and improves the representation ability of the emotion classification task, including:

[0261] (1) Map the target emotion category to an emotion embedding one-hot vector as the target emotion embedding vector: For example, set a vector of all zeros with the same number of emotion categories, and set the elements at the corresponding positions to 1 according to the determined target emotion category sequence number; for example, in the one-hot vector, if the target emotion category is determined to be the i-th category among the seven categories, the i-th element is 1, and the rest are set to 0. The seven-category emotion extraction results include but are not limited to: joy (happiness), anger, sadness, surprise, disgust, fear, and neutral (or no emotion).

[0262] (2) Using the BERT embedding method, each keyword in the keyword list matched by the user profile is converted into an embedding vector of fixed length: For example, each keyword is input into the pre-trained BERT model, and the vector corresponding to the [CLS] tag in the output sequence is extracted as the embedding of the keyword. The BERT dictionary size is fixed. For keywords outside the BERT dictionary, the embedding vector of the [UNK] tag is used instead. The embedding vectors of all keywords in the list are combined into a matrix, with each row corresponding to a keyword embedding.

[0263] Preferably, keyword embedding uses BERT to extract deep semantic information, which is more robust and flexible than discrete word vectors.

[0264] (3) Using the multimodal feature extraction module, extract the reference video emotion embedding vector for each reference video: For example, for each reference video, extract one or more key segments, extract the multimodal features of these segments, input the multimodal features of the key segments into the trained multimodal emotion classifier in turn, and extract the penultimate layer output as the emotion embedding vector of the segment. Take the average of the emotion embedding vectors of multiple key segments to obtain the emotion embedding vector of the current reference video. The emotion embedding vectors of all reference videos are combined into a matrix, with each row corresponding to a reference video embedding.

[0265] Preferably, the reference video embedding makes full use of the knowledge learned by the multimodal sentiment classifier and avoids the deviation caused by directly calculating the similarity using the original features of the video.

[0266] (4) The target emotion embedding, keyword embedding, and reference video emotion embedding vectors are weighted averaged through attention to obtain a comprehensive target embedding vector: illustratively, the three types of embedding vectors are respectively mapped to the same dimension through a fully connected layer, and a tanh activation function is applied; the three types of activated embedding vectors are concatenated and input into an attention fully connected layer, and three scalar weights are output; the three types of embedding vectors are multiplied by the corresponding scalar weights, and then summed to obtain a comprehensive target embedding vector.

[0267] In the embodiment of the present invention, candidate segment retrieval calculates the similarity between the embedding of homologous and heterologous graph nodes and the embedding of the comprehensive target, and uses the attention mechanism to adaptively weight and fuse to obtain the comprehensive similarity for sorting, and finally selects the top-K related segments to achieve multi-modal and multi-angle accurate video segment retrieval, including:

[0268] (1) Similarity calculation: For example, all candidate segments are traversed, and the similarity between their node embeddings in the homology graph and heterology graph and the comprehensive target embedding is calculated respectively. The homology graph similarity is calculated by dot product. The heterology graph similarity is calculated by calculating the dot product between the node embeddings of the three modalities of vision, speech, and text and the comprehensive target embedding, and taking the average value.

[0269] Preferably, the segment embeddings learned from homologous and heterologous graphs are used to measure the relevance of candidate segments to the target from both global structure and local details.

[0270] (2) Comprehensive similarity fusion: For each candidate segment, the attention fully connected layer is used to learn two weights based on the homologous graph similarity and heterologous graph similarity. The two weights are used to perform a weighted average of the homologous graph similarity and the heterologous graph similarity to obtain the comprehensive similarity of each candidate segment.

[0271] (3) Sorting and selection of search results: For example, all candidate segments are sorted according to the comprehensive similarity, and the top-K candidate segments with the highest similarity are selected as the final search results.

[0272] In the embodiment of the present invention, for the retrieved Top-K candidate segments, the comprehensive similarity is converted into pairwise similarity using the Gaussian kernel function, a fully connected graph is constructed, the minimum spanning tree algorithm is applied to obtain the weight and the minimum tree structure, and a natural and smooth candidate segment sequence is generated through breadth-first traversal to achieve global optimization combination, thereby improving the accuracy and coherence of emotion recognition and video segment retrieval, including:

[0273] (1) Convert the comprehensive similarity of the candidate segments into pairwise similarity and control the concentration of the similarity distribution: For example, a Gaussian kernel function is used to convert the comprehensive similarity into pairwise similarity and control the concentration of the similarity distribution. For candidate segments i and j, their pairwise similarity f(d i ,d j )for:

[0274]

[0275] where d i and d j is the comprehensive similarity of the two.

[0276] Preferably, a Gaussian kernel function is used to convert the comprehensive similarity into a pairwise similarity, which can better reflect the relative relationship between the candidate segments.

[0277] (2) Construction of a fully connected undirected graph: For example, a fully connected undirected graph is constructed with candidate segments as nodes and pairwise similarities as edge weights, considering the relationships of all candidate segments from a global perspective.

[0278] Preferably, a fully connected graph is constructed with candidate segments as nodes to facilitate global consideration of the associations between all candidate segments.

[0279] (3) Apply the minimum spanning tree (MST) algorithm (including but not limited to Prim or Kruskal algorithm) on the undirected graph to obtain the spanning tree with the minimum weight and extract the key skeleton structure: For example, initialize an empty tree, randomly select a candidate segment as the initial node, and each time select the edge with the smallest weight from the edges connected to the current tree, and add its corresponding candidate segment to the tree. Repeat the above process until all candidate segments are added to the tree.

[0280] Perform a breadth-first traversal on the minimum spanning tree to obtain a candidate segment sequence with high similarity and emotional fluctuations, ensuring global coherence and local emotional diversity: For example, perform a breadth-first traversal (BFS) on the MST to obtain a candidate segment sequence with high similarity and emotional fluctuations. Specifically, starting from the initial node, the MST is traversed layer by layer, and the nodes in the same layer are arranged in the order of joining the tree, and the child nodes of the nodes in the previous layer follow closely to form a sequence.

[0281] Preferably, BFS can give priority to traversing areas with high similarity while covering the entire graph. The resulting sequence has a certain degree of emotional diversity while ensuring similarity. The minimum spanning tree algorithm can extract a tightly connected skeleton structure from the fully connected graph, avoiding unnecessary cycles and branches. Breadth-first traversal can ensure local continuity while taking into account global fluctuations, making the final sequence more natural and smooth.

[0282] In the embodiment of the present invention, a candidate segment sequence is obtained by calculating the global candidate combination, the candidate sequence emotion embedding is calculated, and the dominant emotion is determined; the sequence order is adjusted according to the dominant emotion; the recommended sequence is initialized, and the segments are screened according to the duration threshold; and the optimized sequence is output, including:

[0283] (1) Calculate the average sentiment embedding vector of the candidate sequence: For example, average the embeddings of the homologous nodes of each candidate segment in the sequence to obtain the homologous embedding at the sequence level; do a dot product between the homologous embedding and the comprehensive target embedding to obtain the average sentiment intensity of the sequence.

[0284] Preferably, the average sentiment embedding vector takes into account all candidate segments on the one hand, reducing the noise of individual segments, and on the other hand is directly associated with the target embedding, taking into account both local and global considerations.

[0285] (2) Compare the average emotion intensity with the typical intensity value corresponding to the emotion category, and select the closest emotion category as the dominant emotion: For example, for each emotion category, calculate its average emotion intensity in the training set as the typical intensity value, subtract the average emotion intensity of the sequence from all the typical intensity values, and take the emotion category with the smallest absolute value as the dominant emotion.

[0286] Preferably, by comparing with typical intensity values, a dominant emotion can be matched for the sequence to avoid ambiguity and confusion of emotion categories.

[0287] (3) Adjust the time order of the sequence according to the dominant emotion so that it conforms to the general law of emotion change: for example, for positive dominant emotions, arrange the segments in ascending order according to the emotional intensity so that the emotion gradually increases; for negative dominant emotions, arrange the segments in descending order according to the emotional intensity so that the emotion gradually decreases.

[0288] (4) Initialize an empty recommendation sequence to store the final selected segments: For example, set a duration threshold T to represent the upper limit of the target video duration, traverse the adjusted candidate sequence, and add the segments to the recommendation sequence in sequence; record the cumulative duration of the recommendation sequence, and stop traversing if it exceeds T after adding the current segment.

[0289] (5) The final recommendation sequence is used as the output of timing optimization and filtering.

[0290] In the embodiment of the present invention, the final sequence video material is extracted, a fade-in and fade-out effect is added, a dominant emotional color filter is superimposed, and a complete video is spliced ​​and exported, including:

[0291] (1) According to the final sequence timestamp, extract the corresponding video material from the original video library.

[0292] (2) Adding fade-in and fade-out effects at the junction of adjacent video clips to achieve a smooth transition: For example, the fade-in and fade-out time length t (e.g., 1 second) is set, and in the last t seconds of the previous clip, the transparency of its frame is linearly reduced until it is completely transparent; in the first t seconds of the next clip, the transparency of its frame is linearly increased until it is completely opaque, and the two clips after the fade-in and fade-out are connected end to end on the timeline to achieve a smooth transition.

[0293] (3) Overlay color filters related to the dominant emotion on each video clip to enhance the emotional tone: For example, for positive dominant emotions, use warm-tone filters (such as orange and yellow) to render a vibrant and enthusiastic atmosphere; for negative dominant emotions, use cold-tone filters (such as blue and gray) to render a dull and depressing atmosphere; set the transparency of the filter to include but not limited to about 0.3 to avoid over-saturation of colors that affects the picture quality.

[0294] (4) All video clips with special effects added are stitched together into a complete video file in order, and exported as a complete trailer video including but not limited to MP4 format.

[0295] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0296] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0297] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0298] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0299] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0300] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A recommendation trailer generation system based on multimodal distillation sentiment analysis, characterized in that: include: The multimodal video preprocessing module completes the video decoding and track division of the original video file, and splits the video file into text mode, voice mode and visual mode according to the shot; and performs data cleaning and alignment of the video mode to obtain multimodal preprocessed data; The multimodal video feature recognition module extracts features from preprocessed data in text, speech, and visual modes and synthesizes multimodal feature vectors, performs decoupled distillation multimodal emotion recognition and classification, and performs model training. The emotion-oriented video synthesis module retrieves and filters a set of video clips matching emotion features according to emotion-oriented features, forms a global candidate combination that satisfies the emotion similarity, and performs timing optimization and filtering to complete the emotion-oriented video synthesis. It harmonizes and optimizes the special effects of the synthesized video to generate an emotion-oriented trailer for intelligent editing of long videos.

2. The recommended trailer generation system based on multimodal distillation sentiment analysis according to claim 1, characterized in that: The multimodal video preprocessing module comprises: Use the FFmpeg library to decapsulate the input original video file and obtain the audio and video streams in it by dividing the tracks; In multimodal splitting, AI Chinese speech recognition API is used to obtain text mode, short-time energy STE and short-time zero-crossing rate STZCR of speech signals are used to perform voice activity detection VAD to obtain audio mode, and shot transition SBD is used to detect the change of similarity between video frames to obtain visual mode, and the three modes are optimized and adjusted; In data cleaning and alignment, the timestamp of each fragment recorded in the segmentation log is used.

3. The recommended trailer generation system based on multimodal distillation sentiment analysis according to claim 1, characterized in that: The multimodal video feature recognition module includes: Multimodal feature extraction: use the OpenCV library to read the video frame sequence, and use the MTCNN face detection model with a cascaded CNN structure to detect faces frame by frame. Use the pre-trained FaceNet model to extract the aligned face image features of each frame to generate a visual feature vector. Use the librosa library to read the audio waveform sequence, convert it to the frequency domain through the fast Fourier transform FFT and perform Mel-scale filter bank filtering. After DCT discrete cosine transform DCT, VGGish features are obtained to generate audio feature vectors. Pre-process the intercepted subtitle text, use the jieba word segmentation tool to segment the text into word sequences, and extract text feature vectors through the pre-trained Chinese BERT model. In multimodal feature splicing, the video, audio, and text modal feature vectors are spliced ​​in the order of the modalities, and serialized and formatted in TFRecord or HDF5 formats to complete the multimodal feature splicing; Multimodal emotion recognition and model training with decoupled distillation, build a shared encoder and three exclusive encoders to construct multimodal decoupled representation, and use the Adam optimizer to train the parameters of each encoder; construct a homologous graph including the graph convolutional network GCN to realize the propagation of shared features between nodes, and construct a heterologous graph of the attention network GAT to realize the distillation of exclusive features between modalities, and use the Adam optimizer to train the parameters of GCN and GAT with the weighted sum of the total classification loss and consistency loss as the goal; input the multimodal emotion feature vector into the classifier for classification, and use the cross entropy as the loss function to jointly optimize the classification loss and the loss of dual graph convolution distillation; train a multimodal emotion dataset including training set, validation set and test set to obtain an end-to-end emotion classification model including feature extractor, graph convolution distillation unit and classifier.

4. A method for intelligently generating trailers for a recommendation system based on multimodal distillation sentiment analysis using the system of any one of claims 1 to 3, characterized in that: The following steps are involved: Step 101, decoding and splitting the original video file, and splitting the video into three modal data of text, voice, and vision according to the shot, and ensuring the cleaning and alignment of the multimodal data to obtain multimodal preprocessed data; Step 102, respectively complete the feature extraction of the preprocessed data of the text modality, the speech modality and the visual modality, and splice the three modal features into a multimodal integrated vector; perform decoupled distillation through a shared encoder and an exclusive encoder, combine dual-graph convolution distillation and perform model training, and finally realize fine-grained emotion recognition through a classifier; Step 103, retrieve and filter a set of video clips matching emotions according to emotion-oriented features, optimize the sequence of clips using a minimum spanning tree algorithm and breadth-first traversal to ensure emotional coherence, extract video materials for harmonization and special effects optimization, and synthesize a personalized, emotion-oriented trailer.

5. The method for intelligently generating trailers for recommendation systems based on multimodal distillation sentiment analysis guidance according to claim 4 is characterized in that: In step 101, the multimodal video preprocessing module first performs video decoding and audio and video track separation on the original video file, including: (1) Use the FFmpeg library to decapsulate the input original video file and obtain the audio stream and video stream; (2) re-encode the audio stream, convert it into PCM format, and uniformly resample it into 16kHz, 16-bit mono audio data, and save it as a WAV format file; (3) Re-encode the video stream, convert it to H.264 format, adjust the frame rate to 25fps, and save it as an MP4 file; The audio and video streams after the split tracks are split into multi-modal splitting of text mode, voice mode and visual mode according to the shots. The specific steps are as follows: (1) Split text mode: Call the AI ​​Chinese speech recognition API to transcribe the audio sentence by sentence, and obtain the machine timestamp of the speech recognition result ending with punctuation marks; 1) Divide the recognition results into multiple sentences according to the punctuation positions, and record the start and end timestamps of each sentence; 2) Record the start and end timestamps of each sentence as candidate cutpoints; (2) Split speech modes: Use the short-time energy (STE) and short-time zero-crossing rate (STZCR) of speech signals to perform voice activity detection (VAD); 1) Perform frame-level segmentation on the audio signal, where each frame is schematically segmented into 20ms and the frame shift is schematically segmented into 10ms; 2) Calculate the STE and STZCR features for each frame and use the double threshold method to make speech / non-speech judgments to obtain the start and end timestamps of the speech event; 3) Mark the boundaries of the speech events, i.e. the speech start and end points, as timestamp candidate cutpoints; (3) Split visual modality: Detect shot transition SBD using the change in similarity between video frames; 1) Segment the video frames and extract a key frame every 0.5 seconds; 2) Using the SURF feature matching method to compare the similarity of adjacent key frames, when the number of matching points is less than a schematic preset threshold of 50, it is determined to be a shot boundary; 3) The timestamps of the shot boundaries are recorded as shots candidate cutpoints, where each shot has multiple dialogues and shots candidate cutpoints contains multiple candidate cutpoints; 4) Save the segmented video clip files in MP4 format; (4) Determine the candidate cut points of other modes based on the audio cut points: final_cuts sorted by timestamp is the final cut point list; (5) Traverse all segmented video segments: extract the start and end timestamps, end timestamps, and duration of each segment, and save them in the video segmentation log file; The specific steps for data cleaning and alignment of the segmented video clip files and segmented log files are as follows: (1) Extract the voice timestamp according to the candidate cutpoints timestamps of each segment recorded in the segmentation log, and use the FFmpeg library to extract the audio and video of the corresponding segment from the original video and save them as independent WAV and MP4 files respectively; (2) Traverse all segmented video clips and calculate their duration, filter out video clips that are too short, less than 1 second, or too long, greater than 60 seconds, delete them from the segmentation log, and delete the corresponding audio, video, and text files; (3) Perform voice activity detection (VAD) on the audio of each segment and calculate the voice proportion: Filter out the segments with a voice ratio lower than the preset threshold of 15%, determine them as invalid segments, delete them from the segmentation log, and delete the corresponding audio and video files; (4) Using AI’s Chinese speech recognition API, transcribe the audio of each segment to obtain the text of the speech content; (5) Post-process the speech recognition results; use the Jieba word segmentation tool to segment the text and filter stop words to obtain the syntactic structure information of the segment; (6) Update the segmentation log file cut_info.json of each segment in the segmentation log, and record the final start and end time, duration, text, and keyword metadata of each segment.

6. The method for intelligently generating trailers for recommendation systems based on multimodal distillation sentiment analysis guidance according to claim 4, characterized in that: In step 102, the step of extracting multimodal features from the preprocessed multimodal data and the segmented logs includes: (1) Extract visual modal features: extract visual modal features from the preprocessed visual modal data and segmentation logs to obtain a 128-dimensional visual feature vector; 1) Use the OpenCV library to read the MP4 format video clip file and extract the corresponding video frame sequence according to the start timestamp and end timestamp; 2) For the captured video frame sequence, the MTCNN face detection model is used to detect faces frame by frame: MTCNN adopts a cascaded CNN structure to complete face detection and key point positioning at the same time; 3) According to the coordinates of the key points detected by MTCNN, the faces of each frame are aligned to extract highly robust facial features; face alignment methods including affine transformation and similarity transformation are selected, and the key points are normalized to the calibration position; 4) Use the pre-trained FaceNet model to extract the features of the aligned face images of each frame: The FaceNet model is trained using a pairwise triplet loss function to map the face to a 128-dimensional Euclidean space; 5) For a shot, the facial feature vectors extracted from each frame are arranged in chronological order to form a new feature sequence; the dimension is (N, 128), where N is the number of frames contained in a shot; 6) Performing statistical analysis on the feature sequence in 5) in the time dimension, calculating the mean and standard deviation of each dimension of the feature in the sequence, and obtaining a 256-dimensional feature vector to represent the visual modality characteristics of the entire video clip; (2) Extracting speech modal features: extracting speech modal features from the preprocessed speech modal data and segmentation logs to obtain visual feature vectors; 1) Use librosa library to read WAV format audio clip files, and extract the corresponding audio waveform sequence according to the start timestamp and end timestamp; 2) Pre-emphasize and frame the intercepted audio waveform sequence: pre-emphasize to balance the energy of high-frequency and low-frequency components, and select a first-order high-pass filter with a coefficient of 0.97; frame is used for short-time analysis, the frame length is set to 0.96s, and the frame shift is set to 1.5s; 3) Feature extraction for each frame of pre-emphasized audio data: First, the pre-emphasized audio waveform sequence is converted into a frequency domain audio sequence by fast Fourier transform (FFT), and the Mel-scale filter bank is used to filter to obtain a Mel spectrum to obtain audio data that is closer to the auditory characteristics of the human ear; the Mel spectrum is logarithmic and discrete cosine transform (DCT) is performed to obtain a VGGish feature vector; 4) Arrange the VGGish feature vectors extracted from each frame in chronological order to form a new feature sequence; the dimension is (N, 128), where N is the number of frames contained in a shot; 5) Pooling is performed on the feature sequence generated in time sequence in the time dimension, and the mean of each dimension of the feature in the sequence is calculated to obtain a 128-dimensional mean vector to represent the speech modality features of the entire audio clip; (3) Extracting text modal features: extracting text modal features from the preprocessed text modal data and segmented logs to obtain text feature vectors; 1) Read the TXT format text fragment file encoded in UTF-8, and extract the corresponding subtitle text content according to the start timestamp and end timestamp; 2) Preprocess the extracted subtitle text: Use the Jieba word segmentation tool to segment the text into word sequences; convert the word sequences into the input format required by the BERT model, including adding [CLS] and [SEP] tags, and performing WordPiece word segmentation; The Jieba word segmentation tool is selected to optimize the Chinese corpus, considering its moderate word segmentation granularity and customized dictionary; 3) Input the preprocessed text into the pre-trained Chinese BERT model: The BERT model is a bidirectional language model based on the Transformer structure, which effectively models the contextual information of the text; 4) Extract the output of the last layer of Transformer of BERT as the embedding vector of each word to obtain a word embedding matrix; 5) Pooling the word embedding matrix, select the MaskAveraging method: construct a mask matrix of the same shape according to the input tokenization sequence of BERT, in which the corresponding position of the word is set to 1, and the corresponding position of the padding mark ([PAD]) is set to 0; perform Hadamard product processing on the word embedding matrix and the mask matrix to obtain an embedding matrix without the influence of the padding mark; solve the mean of the embedding matrix without padding in the word dimension to obtain a 768-dimensional sentence vector, which represents the text modality characteristics of the entire text segment; The steps of synthesizing the multimodal integrated vector from the extracted multimodal feature data include: (1) The feature vectors extracted from the three modalities of image, audio, and text are spliced ​​into a multimodal integrated vector in the order of image, audio, and text modalities: the 128-dimensional visual feature vector, the 128-dimensional speech feature vector, and the 768-dimensional text feature vector are sequentially synthesized to obtain a 1024-dimensional multimodal feature integrated vector; (2) Packing the multimodal feature integration vector and the corresponding video segment metadata into a data unit, where the metadata includes the video ID, start timestamp, end timestamp, and duration; (3) Arrange the data units of all video clips in the order of video ID and timestamp to form a video clip set; (4) Serialize the video clip collection in TFRecord or HDF5 format and save it to the local disk or cloud database; Multimodal emotion recognition with disentangled distillation performed on the synthetic multimodal ensemble vectors, including: (1) Construct a multimodal decoupled representation of the 1024-dimensional multimodal feature integration vector; 1) Build a shared encoder and three exclusive encoders for the synthetic multimodal integrated vector and the three modal feature vectors of image, audio, and text respectively; The shared encoder receives a 1024-dimensional synthetic multimodal integrated vector, which is mapped to a 256-dimensional shared feature space through two fully connected layers (1024->512->256); the visual exclusive encoder receives a 128-dimensional visual feature vector, which is mapped to a 128-dimensional visual exclusive feature space through two fully connected layers (128->128->128); the speech exclusive encoder receives a 128-dimensional speech feature vector, which is mapped to a 128-dimensional speech exclusive feature space through two fully connected layers (128->128->128); the text exclusive encoder receives a 768-dimensional text feature vector, which is mapped to a 512-dimensional text exclusive feature space through two fully connected layers (768->512->512); 2) For the three exclusive encoders of vision, speech, and text modalities, an attention pooling layer is connected to the last layer of each exclusive encoder: the attention pooling layer of each exclusive encoder compresses the exclusive feature into a scalar weight through a fully connected layer; the exclusive feature is multiplied by the corresponding scalar weight to obtain the weighted exclusive feature, and the weighted exclusive feature is summed in the feature dimensions of vision, speech, and text modalities to obtain an aggregated feature vector, the dimension of which is 1024; 3) With reconstruction error as the goal, train the shared encoder and the exclusive encoder: concatenate the shared feature vector and the aggregated exclusive feature vector, and reconstruct a 1024-dimensional multimodal feature vector through a fully connected layer; take the mean square error between the reconstructed multimodal feature and the original multimodal feature integration vector as the reconstruction error; use the Adam optimizer to train the parameters of each encoder with the goal of minimizing the reconstruction error; (2) Perform dual-graph convolution distillation on the multimodal shared feature vectors, vision-exclusive feature vectors, speech-exclusive feature vectors, and text-exclusive feature vectors of the multimodal decoupled representation. Through knowledge distillation, information from different perspectives is used to promote and distill each other to obtain a more robust and effective representation. 1) Construct homology graph and heterology graph: Each video clip is set as a node, and the shared features are used as node feature vectors to construct a homology graph of an undirected complete graph; the exclusive features extracted from each modality on each video clip are used as a node to construct a heterology graph of a bipartite graph; the edge weights of the homology graph are calculated using cosine similarity to reflect the similarity of shared features between nodes; the edge weights of the heterology graph are calculated using the attention mechanism to reflect the complementarity of exclusive features between nodes of different modalities; 2) Applying graph convolutional networks (GCNs) on homology graphs to achieve the propagation of shared features between modal nodes: the graph convolution operation aggregates the features of each node with the features of its neighboring nodes and updates the learnable parameters of the modal nodes; 3) Applying the Graph Attention Network (GAT) on heterogeneous graphs to achieve distillation of exclusive features between modalities: the graph attention operation adjusts the learnable parameters of nodes according to the attention weights between nodes; 4) Calculate the loss on the homologous graph and heterologous graph respectively: take the emotional label of each video clip as the category label of the node, on the homologous graph, input the distilled shared features into a fully connected layer for emotional classification, and calculate the cross entropy loss; on the heterologous graph, input the exclusive features of each modality after distillation into a fully connected layer for emotional classification, and calculate the average cross entropy loss; 5) Jointly optimize the loss functions on homologous graphs and heterologous graphs: add the node classification loss of the homologous graph and the node classification loss of the heterologous graph as the total classification loss; introduce a consistency loss in addition to the total classification loss to encourage the homologous graph and the heterologous graph to reach a consensus on the node classification task; use the Adam optimizer to train the parameters of GCN and GAT with the weighted sum of the total classification loss and consistency loss as the goal; 6) Perform dual-graph convolution distillation on the multimodal shared feature vector, visual exclusive feature vector, speech exclusive feature vector, and text exclusive feature vector of the multimodal decoupled representation, which is similar to the function of extracting features by the encoder. The exclusive features obtained after distillation are: the visual exclusive feature is 128 dimensions, the speech exclusive feature after distillation is 128 dimensions, and the text exclusive feature after distillation is 512 dimensions; (3) Perform multimodal sentiment classification on the distilled multimodal shared feature vectors, vision-exclusive feature vectors, speech-exclusive feature vectors, and text-exclusive feature vectors. Sentiment classification uses 7 categories to ensure the fine granularity of sentiment recognition. 1) The original multimodal features are concatenated with the shared features and exclusive features after distillation to synthesize a multimodal sentiment feature vector: the original multimodal features are 1024-dimensional, the shared features are 256-dimensional, the visual exclusive features are 128-dimensional, the speech exclusive features are 128-dimensional, and the text exclusive features are 512-dimensional, and they are sequentially concatenated into a new 2048-dimensional multimodal sentiment feature vector; 2) Input the multimodal sentiment feature vector into the classifier for classification: the classifier consists of three fully connected layers (2048->1024->512->7) and a softmax layer; 3) Using cross entropy as the loss function, calculate the difference between the predicted probability of the classifier and the true label; jointly optimize the classification loss and the loss of dual-graph convolutional distillation: add the classification loss to the total classification loss and consistency loss to obtain the total training target, and use the Adam optimizer to update the parameters of the classifier, GCN, and GAT to achieve end-to-end training; 4) The multimodal sentiment feature vector is input into the classifier to obtain the output probability argmax of the classifier and the final sentiment classification result; (4) The visual exclusive features, speech exclusive features, and text exclusive features are respectively used for trimodal independent classification, and alpha is introduced to control the loss weight, analyze the modal contribution, and assist in multimodal fusion sentiment classification; 1) Build three independent unimodal classifiers: build two visual classifiers with fully connected layers and softmax layers; build two speech classifiers with fully connected layers and softmax layers; build three text classifiers with fully connected layers and softmax layers; the 7-category emotion extraction results include: joy / happiness, anger, sadness, surprise, disgust, fear, neutral or no emotion; 2) Input the image, audio, and text exclusive features into the corresponding unimodal classifiers respectively, calculate argmax to obtain the sentiment classification results under each modality; calculate the classification cross entropy loss of the three unimodal classifiers, and add the three unimodal classification losses to obtain the total loss of modality independent classification; 3) In the overall training objective of multimodal sentiment classification, the modality-independent classification loss is added as a weighted term: an independent weight parameter alpha is introduced to control the proportion of the modality-independent classification loss in the overall training objective; (5) Using multimodal sentiment datasets such as training sets, validation sets, and test sets, an end-to-end sentiment classification model consisting of feature extractors, graph convolutional distillation units, and classifiers is trained; 1) Perform batch training on the training set: The size of each batch is set to N = 64, and N video clips are randomly sampled from the training set; the multimodal features of these N video clips are obtained through the multimodal feature extraction module; the multimodal features are sent to the dual-graph convolutional distillation module and the classifier to calculate the total training target, i.e., classification loss + distillation loss + modality-independent classification loss; the model parameters are updated through the Adam optimizer and the back-propagation algorithm to minimize the total training target; 2) After each training epoch, evaluate the model’s accuracy, F1-score and other performance on the validation set: Use the same performance indicators as those in the test phase, such as accuracy and F1-score, and record the current model’s optimal performance on the validation set and its corresponding parameter status; 3) Set up the triggering early stopping mechanism: When the performance of the validation set no longer improves after a certain number of epochs greater than or equal to 10, the early stopping mechanism is triggered, training is stopped, and the optimal parameter state is used as the final model parameter; the generalization performance of the model is evaluated on the test set, and the mean and standard deviation of each indicator are calculated and recorded; 4) If the performance of the test set is not ideal, consider adjusting the learning rate, batch size, regularization coefficient, and graph convolution / attention layer hyperparameters and retraining: Use grid search and Bayesian optimization automatic parameter adjustment methods to find the optimal hyperparameter combination; 5) Package the trained model and encapsulate it into a callable interface to ensure the compatibility and robustness of the interface, and to give reasonable output for inputs of different formats, lengths, and qualities.

7. The method for intelligently generating trailers for recommendation systems based on multimodal distillation sentiment analysis guidance according to claim 4, characterized in that: In step 103, the target emotion category is mapped to an emotion embedding one-hot vector, the keyword list matching the user portrait is embedded using the BERT method to generate a keyword embedding vector, and the video clip is extracted through multimodal features and a classifier to obtain a reference video emotion embedding vector, and the three types of embedding vectors are fused using a fully connected layer and an attention mechanism, and the weights are adaptively adjusted to obtain a comprehensive target embedding vector, which fully integrates multi-source information and improves the representation ability of the emotion classification task, including: (1) Map the target emotion category to an emotion embedding one-hot vector as the target emotion embedding vector: set an all-zero vector with the same number of emotion categories, and set the elements at the corresponding positions to 1 according to the determined target emotion category sequence number; in the one-hot vector, if the target emotion category is determined to be the i-th category among the seven categories, the i-th element is 1, and the rest are set to 0; (2) Using the BERT embedding method, each keyword in the keyword list matched with the user profile is converted into an embedding vector of fixed length: each keyword is input into the pre-trained BERT model, and the vector corresponding to the [CLS] tag in the output sequence is extracted as the embedding of the keyword; the BERT dictionary size is fixed, and for keywords outside the BERT dictionary, the embedding vector of the [UNK] tag is used instead; the embedding vectors of all keywords in the list are combined into a matrix, with each row corresponding to a keyword embedding; (3) Using the multimodal feature extraction module, extract the reference video emotion embedding vector for each reference video: for each reference video, extract one or more key segments, extract the multimodal features of these segments, input the multimodal features of the key segments into the trained multimodal emotion classifier in sequence, and extract the penultimate layer output as the emotion embedding vector of the segment; average the emotion embedding vectors of multiple key segments to obtain the emotion embedding vector of the current reference video; form a matrix with the emotion embedding vectors of all reference videos, where each row corresponds to a reference video embedding; (4) The target emotion embedding, keyword embedding, and reference video emotion embedding vectors are weighted averaged through attention to obtain a comprehensive target embedding vector: the three types of embedding vectors are mapped to the same dimension through a fully connected layer respectively, and a tanh activation function is applied; the three types of activated embedding vectors are concatenated and input into an attention fully connected layer, and three scalar weights are output; the three types of embedding vectors are multiplied by the corresponding scalar weights, and then summed to obtain a comprehensive target embedding vector.

8. The method for intelligently generating trailers for recommendation systems based on multimodal distillation sentiment analysis guidance according to claim 7, characterized in that: Candidate segment retrieval calculates the similarity between the embedding of homologous and heterologous graph nodes and the comprehensive target embedding, and uses the attention mechanism to adaptively weighted fusion to obtain the comprehensive similarity for sorting, and finally selects the Top-K related segments to achieve multi-modal and multi-angle accurate video segment retrieval, including: (1) Similarity calculation: Traverse all candidate segments and calculate the similarity between their node embeddings in the homology graph and heterology graph and the comprehensive target embedding respectively; the homology graph similarity is obtained by calculating the dot product; the heterology graph similarity is obtained by calculating the dot product of the node embeddings of the three modalities of vision, speech, and text with the comprehensive target embedding and taking the average value; (2) Comprehensive similarity fusion: For each candidate segment, the attention fully connected layer is used to learn two weights based on the homologous graph similarity and heterologous graph similarity. The two weights are used to perform a weighted average of the homologous graph similarity and heterologous graph similarity to obtain the comprehensive similarity of each candidate segment. (3) Sorting and selection of search results: All candidate segments are sorted according to the comprehensive similarity; the top-K candidate segments with the highest similarity are selected as the final search results.

9. The method for intelligently generating trailers for recommendation systems based on multimodal distillation sentiment analysis guidance according to claim 8, characterized in that: For the top-K candidate segments retrieved, the comprehensive similarity is converted into pairwise similarity using Gaussian kernel function, a fully connected graph is constructed, the minimum spanning tree algorithm is applied to obtain the weight and the minimum tree structure, and a natural and smooth candidate segment sequence is generated through breadth-first traversal to achieve global optimization combination, improve the accuracy and coherence of emotion recognition and video segment retrieval, including: (1) Convert the comprehensive similarity of the candidate segments into pairwise similarity and control the concentration of the similarity distribution: Use the Gaussian kernel function to convert the comprehensive similarity into pairwise similarity and control the concentration of the similarity distribution. For candidate segments i and j, their pairwise similarity f(d i ,d j )for: where d i and d j is the comprehensive similarity of the two; (2) Construction of a fully connected undirected graph: With candidate segments as nodes and pairwise similarities as edge weights, a fully connected undirected graph is constructed, considering the relationships among all candidate segments from a global perspective; (3) Apply the minimum spanning tree (MST) algorithm on the undirected graph to obtain the spanning tree with the minimum weight and extract the key skeleton structure: initialize an empty tree, randomly select a candidate segment as the initial node, and each time select the edge with the smallest weight from the edges connected to the current tree, add its corresponding candidate segment to the tree, and repeat until all candidate segments are added to the tree; Perform a breadth-first traversal of the minimum spanning tree to obtain a candidate segment sequence with high similarity and emotional fluctuations, ensuring global coherence and local emotional diversity: Perform a breadth-first traversal BFS on the MST to obtain a candidate segment sequence with high similarity and emotional fluctuations; Starting from the initial node, traverse the MST layer by layer, and the nodes in the same layer are arranged in the order of joining the tree, and the child nodes of the nodes in the previous layer follow closely to form a sequence.

10. The method for intelligently generating trailers for recommendation systems based on multimodal distillation sentiment analysis guidance according to claim 9, characterized in that: Calculate the candidate segment sequence obtained by global candidate combination, calculate the candidate sequence emotion embedding, and determine the dominant emotion; adjust the sequence order according to the dominant emotion; initialize the recommended sequence, and filter the segments according to the duration threshold; Output optimized sequence, including: (1) Calculate the average sentiment embedding vector of the candidate sequence: average the embeddings of the homologous nodes of each candidate segment in the sequence to obtain the homologous embedding at the sequence level; do a dot product between the homologous embedding and the comprehensive target embedding to obtain the average sentiment intensity of the sequence; (2) Compare the average emotion intensity with the typical intensity value corresponding to the emotion category, and select the closest emotion category as the dominant emotion: For each emotion category, calculate its average emotion intensity in the training set as the typical intensity value, subtract the average emotion intensity of the sequence from all the typical intensity values, and take the emotion category with the smallest absolute value as the dominant emotion; (3) Adjust the time order of the sequence according to the dominant emotion so that it conforms to the law of emotion change: for positive dominant emotions, arrange the segments in ascending order according to the emotional intensity so that the emotion gradually increases; for negative dominant emotions, arrange the segments in descending order according to the emotional intensity so that the emotion gradually decreases; (4) Initialize an empty recommendation sequence to store the final selected segments: set a duration threshold T, which represents the upper limit of the target video duration, traverse the adjusted candidate sequence, and add the segments to the recommendation sequence in sequence; record the cumulative duration of the recommendation sequence, and stop traversing if the duration exceeds T after adding the current segment; (5) The final recommendation sequence is used as the output of timing optimization and filtering; Extract the final sequence video material, add fade-in and fade-out effects, overlay the dominant emotional color filter, and splice and export the complete video, including: (1) Extract the corresponding video material from the original video library according to the final sequence timestamp; (2) Add a fade-in and fade-out effect at the junction of adjacent video clips to achieve a smooth transition: set the fade-in and fade-out time length t, and linearly reduce the transparency of the frame of the previous clip in the last t seconds until it is completely transparent; linearly increase the transparency of the frame of the next clip in the first t seconds until it is completely opaque, and connect the two clips after the fade-in and fade-out end to end on the timeline to achieve a smooth transition; (3) superimposing a color filter related to the dominant emotion on each video clip to enhance the emotional tone; (4) All video clips with special effects added are stitched together into a complete video file in sequence and exported as a complete trailer video in MP4 format.

Images