A method and system for predicting short video tags using multimodal collaborative interaction

By using a multimodal collaborative interactive Transformer model that combines video content, user history preferences, and tag text, the problem of user personalization differences and tag interference in existing technologies is solved, achieving higher tag prediction accuracy.

CN118334549BActive Publication Date: 2026-11-17SHANGHAI JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410372491.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-29
Publication Date
2026-11-17
Estimated Expiration
2044-03-29

AI Technical Summary

Technical Problem

Existing short video tag prediction technologies ignore user individual differences and distracting information in tag text, and fail to fully learn the relationships between tags, resulting in insufficient prediction accuracy.

Method used

We employ a multimodal collaborative interaction Transformer model, combining video content, user history preferences, and tag text. Through graph attention network modeling, we learn the semantic relationships between tags and integrate user sentiment preferences and video content features to perform tag prediction.

Benefits of technology

It improves the accuracy of short video tag prediction, can adaptively learn the semantic relationships of tags, and fully considers the influence of user factors, thus improving the prediction effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118334549B_ABST
    Figure CN118334549B_ABST
Patent Text Reader

Abstract

A multimodal collaborative interaction method and system for short video tag prediction is proposed. The method constructs and inputs a short video sample dataset containing a set of short videos, a set of users who published the short videos, and a set of corresponding tags into a multimodal collaborative interaction short video tag prediction model. A multilayer perceptron prediction module collaboratively interacts the three types of information from different perspectives to obtain comprehensive features of the short videos and generate tag prediction scores. Then, using the desired true tag of the short video as the target, the feature difference loss between the predicted candidate tag and the true tag is calculated, and training is achieved iteratively. After training is complete, the trained multimodal collaborative interaction short video tag prediction model is used to predict short video samples to generate tag prediction pairs for the target video. This invention fully utilizes the specificity of user personalization preferences in tag prediction; the model can adaptively learn and update the relationship weights between tag semantics, resulting in a more comprehensive learning of tag semantics; and the model achieves higher accuracy in the field of short video tag prediction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a technology in the field of neural network applications, specifically a method and system for predicting short video tags using multimodal collaborative interaction. Background Technology

[0002] Existing short video tag prediction technologies often focus solely on the video content itself, using modal information such as images, audio, and descriptive text to perform content tag classification tasks through video understanding. This ignores the social attributes of tags and short videos, as well as the specificity of user characteristics, such as different users' preferences for different tags. Similarly, since tags themselves also contain textual attributes, and since most tags are user-generated, short video tag text often contains a lot of interfering information. Existing technologies mostly perform simple segmentation or error correction, lacking further text modeling and feature relationship filtering. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies, which fail to consider the impact of user individuality on tagging, rely solely on simple semantic understanding of text, lack sufficient modeling of semantic relationships to learn the relationships between tags, and are unable to classify and predict tags for short videos. It proposes a multimodal collaborative interactive short video tag prediction method and system. To address the potential modality gaps in videos, a non-modal Transformer model structure is employed for feature fusion. Based on video content understanding, user historical tendencies and video sentiment tendencies are modeled, fully considering the impact of user factors on prediction. Simultaneously, tags in the tag pool are modeled and learned through a graph attention network to eliminate interference from tag text and fully learn the information of existing tags. Tag prediction is then performed after comprehensively evaluating the collaborative interaction of information from three dimensions: video content understanding, user tendencies, and tag text interference.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a multimodal collaborative interaction short video tag prediction method. It constructs and inputs a short video sample dataset containing a set of short videos, a set of users who published the short videos, and a set of corresponding tags into a multimodal collaborative interaction short video tag prediction model. A multilayer perceptron prediction module collaboratively interacts the three types of information from different perspectives to obtain the comprehensive features of the short videos and generate tag prediction scores. Then, using the desired true tag of the short video as the target, it calculates the feature difference loss between the predicted candidate tags and the true tags, and trains iteratively. After training is complete, the trained multimodal collaborative interaction short video tag prediction model is used to predict short video samples to generate tag prediction pairs for the target videos.

[0006] The aforementioned short video sample dataset includes a training set, a validation set, and a test set, with the input consisting of video-user-tag triplet pairs.

[0007] This invention relates to a short video tag prediction system for implementing the above-mentioned method, comprising: a video content feature representation module, a sentiment representation auxiliary module, a tag representation module, and a multilayer perceptron prediction module, wherein: the video content feature representation module inputs the short video into a multimodal Transformer based on a multi-head attention mechanism, extracts features from the images, text, and audio of the short video sample content through a content pre-trained model, and then performs multimodal fusion through the Transformer to obtain the comprehensive content features of the short video; the sentiment representation auxiliary module takes the short video content and user information as input, obtains comprehensive information based on user history information and short video sentiment, extracts features from the images, text, and audio of the short video sample content through a pre-trained model, performs sentiment analysis through attention encoding to obtain the short video sentiment; inputs user history information into an attention network to obtain the user's interest tendency, and then fuses the two tendencies through an attention mechanism to obtain the final sentiment information; the tag representation module inputs the tag text into a graph attention neural network (GAT) for processing to obtain the semantic relationship information between tags; the multilayer perceptron prediction module uses a multilayer perceptron to perform collaborative interaction of the feature information of each dimension, and the results are passed through a prediction layer to obtain the final tag prediction score.

[0008] The content pre-trained models include: a residual neural network (ResNet) for extracting content feature vectors from video frames, a deep learning-based audio feature extraction network (VGGish) for extracting content feature vectors from audio, and a pre-trained language representation model for extracting content feature vectors from text.

[0009] The sentiment pre-training model includes: SentiBank, a concept classifier for extracting video sentiment feature vectors from video frames; VGGish, a deep learning-based audio feature extraction network for extracting features from audio and then analyzing Mel-frequency cepstral coefficients (MFCCs) to obtain audio sentiment feature vectors; and SKEP, a sentiment pre-training model based on sentiment knowledge enhancement for extracting text sentiment feature vectors from text.

[0010] Technical effect

[0011] This invention utilizes a multi-layered GAT network that automatically learns the semantic relationships between tags, along with a fusion mechanism that integrates the emotional features of the video itself (which carries user preferences) with the preferences of user information analysis. This fully leverages the specificity of user personalized preferences in tag prediction. The model can adaptively learn and update the weights between the semantic relationships of tags, resulting in a more comprehensive learning effect of tag semantics. The model has achieved higher accuracy in the field of short video tag prediction. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the overall structure of the multimodal short video tag prediction model of the present invention;

[0013] Figure 2 This is a schematic diagram of the short video content representation module of the present invention;

[0014] Figure 3 This is a schematic diagram illustrating the auxiliary module of the present invention;

[0015] Figure 4 This is a schematic diagram of the label representation module of the present invention;

[0016] Figure 5 This is a schematic diagram of the multilayer perceptron prediction module of the present invention;

[0017] Figure 6 This is the main flowchart for implementing the present invention.

[0018] Figure 7 This is a schematic diagram illustrating the tag prediction effect of a short video embodiment of the present invention. Detailed Implementation

[0019] like Figure 1 As shown in the figure, this embodiment relates to a short video tag prediction system based on multimodal collaborative interaction of user comprehensive preferences, video content, and tag text information. The system includes: a video content feature representation module, a preference representation auxiliary module, a tag representation module, and a multilayer perceptron prediction module.

[0020] like Figure 2As shown, the video content feature representation module includes: a video content feature vector extraction unit, an audio content feature vector extraction unit, a text content feature vector extraction unit, and a multimodal content feature fusion unit. Specifically: the video content feature vector extraction unit extracts features from the video frames of the input video using a pre-trained ResNet model to obtain video content feature vectors; the audio content feature vector extraction unit extracts features from the audio information of the input video using a pre-trained VGGish model to obtain audio content feature vectors; the text content feature vector extraction unit extracts features from the text information accompanying the input video using a pre-trained BERT model to obtain text content feature vectors; and the multimodal content feature fusion unit fuses the input video content features, audio content features, and text content features using an improved Transformer model to obtain the final comprehensive content representation features.

[0021] like Figure 3 As shown, the sentiment representation auxiliary module includes: a video sentiment feature vector extraction unit, an audio sentiment feature vector extraction unit, a text sentiment feature vector extraction unit, a multimodal sentiment feature fusion unit, a user sentiment feature vector extraction unit, and a weighted fusion feature unit. Specifically: the video sentiment feature vector extraction unit extracts features from the video frames of the input video using a pre-trained SentiBank model to obtain video sentiment feature vectors; the audio sentiment feature vector extraction unit extracts features from the audio information of the input video using a pre-trained VGGish model, and then obtains audio sentiment feature vectors by analyzing MFCC; the text sentiment feature vector... The extraction unit extracts features from the text information accompanying the input video using the pre-trained model SKEP to obtain a text sentiment feature vector. The multimodal sentiment feature fusion unit fuses the input video sentiment features, audio sentiment features, and text sentiment features using an improved Transformer to obtain the final comprehensive sentiment tendency representation feature. The user tendency feature vector extraction unit models the user based on historical information and obtains the user tendency representation feature through an attention mechanism. The weighted fusion feature unit performs a weighted average fusion of the input sentiment tendency representation feature and the user tendency representation feature to obtain the comprehensive tendency representation feature.

[0022] like Figure 4 As shown, the label representation module includes: a label initial matrix unit and a multi-layer GAT unit, wherein: the label initial matrix unit reconstructs edges based on the input label information by assigning weights to different semantics between labels to obtain an initial label matrix relationship as a directed graph; the multi-layer GAT unit takes the initial label matrix as input and automatically learns and updates the relationship weights during training through an attention mechanism to obtain label representation features.

[0023] like Figure 5 As shown, the multilayer perceptron prediction module uses an MLP (Multilayer Perceptron) unit to capture the nonlinear relationship between short videos, tendencies, and tags by fusing the input content representation features, tendency representation features, and tag representation features in pairs and then passing them through a hidden layer composed of fully connected layers, thereby obtaining the probability of predicting tags.

[0024] like Figure 6 As shown, this embodiment illustrates a short video tag prediction method based on the aforementioned system, comprising:

[0025] Step 1: Construct a short video data sample set: Scrape short videos and user information and hashtags for posting short videos from the Vine platform and Instagram respectively. Process, clean and filter the original Instagram dataset and Vine dataset to obtain two cleaned datasets: Instagram-MicroVideo dataset and Vine-MicroVideo dataset. Both datasets can be used to form video-user-tag triples, which can be used as input to the model.

[0026] The video-user-tag triplet includes: short videos ,user and video tags , For users Published short videos The set of tags and the set of triples in the text. This represents the video-user-tag relationship.

[0027] The cleaning and filtering process refers to the following steps: Taking the Instagram-MicroVideo dataset as an example, a web crawler script is used to first crawl short videos from the Instagram platform. For short videos, videos lacking text or audio modalities are removed during the cleaning process. For tag information, tags with too few occurrences (less than 10 times) and tags with significant spelling errors are removed through text preprocessing. For user information, users with no history or empty information are removed. Finally, the user ID-short video content-tag correspondence and user ID-tag correspondence for the filtered short videos are recorded. The filtered short video tags are used to construct a short video tag pool, resulting in the final Instagram-MicroVideo dataset, which contains 21,278 short videos and 15,620 hashtags from 6,673 users.

[0028] Step 2: Randomly extract three independent datasets from the short video data sample set obtained in Step 1 in a ratio of 8:1:1, and use them as the training set, validation set, and test set, respectively.

[0029] Step 3: Use the training set obtained in Step 2 to train the multimodal collaborative interaction short video tag prediction model, specifically including:

[0030] 3.1) As Figure 2 As shown, after the video input passes through the short video content representation module to obtain video feature vectors, audio feature vectors, and text feature vectors respectively, the improved video multimodal Transformer model fuses the vectors from different modalities to obtain the final comprehensive content representation features, specifically including:

[0031] 3.1.1) The short video is converted into a frame sequence using FFmpeg, and 50 frames are selected for each video for uniform sampling. After extracting the frames, the ResNet-34 residual neural network is used for feature extraction. After ResNet-34 feature extraction, the final frame has a video feature vector of size 1024 dimensions.

[0032] 3.1.2) The audio is also first separated into time sequences by FFmpeg, and then the audio data is sampled at a certain frequency by VGGish and filtered to obtain the spectral data. Finally, a 1024-dimensional audio feature vector is obtained.

[0033] 3.1.3) The text data is processed by the trained language representation model BERT, and through the language model with masking mechanism, a bidirectional language representation is formed, resulting in a 512-dimensional text feature vector.

[0034] The improved multimodal Transformer model described above fuses input vectors by combining inputs. Specifically, for video... Audio ,text The vector input of three modes yields... Six embeddings are used to compute the combinatorial loss. For each training sample, six output sets of tokens and embeddings are obtained, which are then used to compute the combinatorial loss. Taking creation as an example, the output tags are grouped according to the modality. and Calculate the average value respectively , Then normalize it and combine it into the final embedding vector: ,in: Let v and a be the learnable modality-specific projections, respectively. The norm is used to measure the magnitudes of their vectors. When calculating the dot product similarity, the angle between the vectors is considered. Finally, the combined loss formula is obtained as follows: .in: Loss values ​​between The weighting coefficients.

[0035] 3.2) As Figure 3 As shown, the tendency representation auxiliary module obtains tendency representation features based on video and user information, specifically including:

[0036] 3.2.1) Sentiment as a Unit: Based on the input of the short video, video sentiment feature vectors with a dimension of 1024 are extracted from the video frames using SentiBank; audio features are extracted using VGGish and then MFCC (Mel frequency cepstral coefficients) are analyzed to obtain audio sentiment feature vectors with a dimension of 512; text sentiment feature vectors with a dimension of 1024 are extracted from the text using SKEP; after obtaining the sentiment feature vectors of video, audio, and text, a multi-head multimodal attention mechanism is used to first allocate internal attention, then continue cross-modal attention learning, and finally integrate the visual, audio, and text sentiment features to obtain the sentiment vector contained in the short video content.

[0037] 3.2.2) To learn users' historical preferences, after retrieving user profiles, historical tags, and historical video information from the input information, a model is built based on the user's historical interests. Taking historical text as an example, the feature matrix is ​​obtained by retrieving the user's historical text. After setting the weights, the similarity matrix between the current text and historical text is obtained. ,in: After calculating the similarity between the text and the i-th historical text, the weight of each text is obtained as follows: ,in: and All of them are trainable parameters.

[0038] 3.2.3) After obtaining the historical tendency vector and sentiment vector, the final vector is obtained by concatenating them through an attention mechanism. Where: user preference vector User ID Embedded , This is a mapping function.

[0039] 3.3) such as Figure 4 As shown, the label representation module obtains a graph attention neural network composed of label pairs based on the input label set. Different weights are assigned according to different relationships in the label pairs to obtain the initial association matrix of the label graph. The relation matrix is ​​then updated through multiple layers of GAT, with a total of 4 layers.

[0040] The relationship between the two labels in the label pair includes: the face weights are divided into descending order:

[0041] 1. Composition relationship: When a tag A and another tag B have an inclusion relationship in the word text, it is a composition relationship.

[0042] 2. Similarity relationship: The similarity between tags is calculated using WUP similarity, and tag pairs that are above a certain similarity standard range are considered to have a similar relationship;

[0043] 3. Co-occurrence relationship: When tag A and tag B appear together in historical videos at a certain frequency, it can be considered that the two tags frequently appear at the same time.

[0044] 4. Emotional co-occurrence relationship: When the semantic emotions of label A and label B are similar, the two labels can be considered to have an emotional co-occurrence relationship.

[0045] The steps of the aforementioned multilayer GAT neural network are as follows:

[0046] 3.3.1) The initial input labels are a matrix, which can be viewed as a directed graph. For nodes... Directed edge source node Considered as its neighbor within the border, recorded as ; there will be directed edges The target node j is considered its outgoing neighbor, denoted as .

[0047] 3.3.2) Update the label node features, specifically: ,in: For the first Layer The feature vector of each node It is the first Layer weight matrix, It is the first Layer nodes To the node Attention weights It is the ReLU activation function.

[0048] 3.3.2) After each layer of GAT, the label relation matrix is ​​recalculated based on the updated labels. The formula for updating the label relation matrix is: ,in: The labels output by the previous layer GAT are a matrix. It is the Hadamard product of the elements-by-element multiplication of the matrices.

[0049] 3.4) As Figure 5 As shown, the prediction result is obtained through feature interaction using an MLP (Multilayer Perceptron), specifically including: representing content features... Tendency indicates characteristics and labels represent features The input to the Bi-Interaction layer of the MLP (Multilayer Perceptron) performs pairwise interactions. Specifically, Bi-Interaction transforms multiple embedding vectors of two elements into a single vector, which is then passed through a hidden layer composed of fully connected layers to capture the non-linear relationship between short videos, sentiments, and tags. The final hidden layer... The output is through Adjust the predicted score to The range is used to obtain the predicted probability of the label. The closer the value is to 1, the higher the predicted probability of this label.

[0050] The hidden layer outputs the predicted probability. ,in: For the first The weight matrix of the layer, Let its deviation vector be... Then it is the first The output of the hidden layer.

[0051] Step 4, Online Phase, such as Figure 7 Based on the short videos in the test set, the multimodal collaborative interaction short video tag prediction model trained in step three outputs the predicted scores of all tags in the tag pool and ranks them.

[0052] As a Top-k problem evaluation, taking recall as an example, where: ,in: This is the list of labels with high Top-K probabilities predicted by the prediction model after training. This is the actual list of tags corresponding to the video.

[0053] Through specific practical experiments, in a server environment with dual GPUs (NVIDIA A100-PCIE-40GB and Tesla V100-PCIE-32GB), the video vector dimension was 1024, the text vector dimension was 1024, and the audio vector dimension was 512. After multiple training iterations of the model, in the Top-k problem evaluation with k=5, the Recall metric was 0.6627 for the Instagram-MicroVideo dataset test set and 0.7034 for the Vine-MicroVideo dataset test set.

[0054] Compared with existing technologies, this method not only models based on user information and analyzes users' historical tendencies, but also extracts subjective emotional tendencies from the video content itself. The fusion of these two tendencies into a more personalized prediction for short video tags is more effective. It fully explores the semantic relationships between tags, proposing a comprehensive set of applicable semantic relationships as the initial prior tag matrix. A multi-layer graph attention network adaptively learns the semantic relationships between tags and automatically updates the matrix, providing a more flexible and comprehensive understanding of tag semantic relationships and improving the model's prediction accuracy. The improved Transformer structure uses a combined input approach to fuse input vectors. Unlike the existing three sets of inputs (audio, video frames, and text), it improves to six combined embeddings: (audio), (video frames), (text), (audio, video frames), (audio, text), and (video frames, text).

[0055] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A short video tag prediction method with multimodal collaborative interaction, characterized in that, A multimodal collaborative interaction short video tag prediction model is constructed and input with a short video sample dataset containing a set of short videos, a set of users who published the short videos, and a set of tags. The model uses a multilayer perceptron prediction module to collaboratively interact the three types of information from different perspectives to obtain the comprehensive features of the short videos and generate tag prediction scores. Then, with the desired real tag of the short video as the target, the feature difference loss between the predicted candidate tag and the real tag is calculated, and training is achieved through iteration. After the training is completed iteratively, the trained multimodal collaborative interaction short video tag prediction model is used to predict short video samples to generate tag prediction pairs for the target video. The aforementioned short video sample dataset includes a training set, a validation set, and a test set, with the input consisting of video-user-tag triplet pairs. The short video tag prediction method specifically includes: Step 1: Construct a short video data sample set: Scrape short videos and user information and hashtags for posting short videos from the Vine platform and Instagram respectively. Process, clean and filter the original Instagram dataset and Vine dataset to obtain two cleaned datasets, Instagram-MicroVideo dataset and Vine-MicroVideo dataset. Both datasets form triplet forms of video-user-tag pairs, which are used as input to the model. The video-user-tag triplet includes: short videos ,user and video tags , For users Published short videos The set of tags and the set of triples in the text. The relationship is video-user-tag pairs; Step 2: Randomly extract three independent datasets from the short video data sample set obtained in Step 1 in a ratio of 8:1:1, and use them as the training set, validation set, and test set, respectively. Step 3: Use the training set obtained in Step 2 to train the multimodal collaborative interaction short video tag prediction model, including: after the video input is processed by the short video content representation module to obtain video feature vector, audio feature vector and text feature vector respectively, the vectors of different modalities are fused by the improved video multimodal Transformer model to obtain the final comprehensive content representation features; Step 4, Online Phase: Based on the short videos in the test set, the multimodal collaborative interaction short video tag prediction model trained in Step 3 outputs the prediction scores of all tags in the tag pool and ranks them. The improved video multimodal Transformer model described above fuses the input vectors by combining the inputs. Specifically, for video... Audio ,text The vector input of three modes yields... Six embeddings are used to compute the combinatorial loss, resulting in six output sets of tokens and embeddings for each training sample. Taking creation as an example, the output tags are grouped according to the modality. and Calculate the average value respectively , Then normalize it and combine it into the final embedding vector: ,in: Let v and a be the learnable modality-specific projections, respectively. The norm is used to align the magnitudes of their vectors. When calculating the dot product similarity, the angle between the vectors is considered. Finally, the formula for the combined loss is: ,in: Loss values ​​between The weighting coefficients.

2. The short video tag prediction method based on multimodal collaborative interaction according to claim 1, characterized in that, Step three specifically includes: 3.1) After the video input is processed by the short video content representation module to obtain video feature vectors, audio feature vectors, and text feature vectors respectively, the vectors from different modalities are fused using an improved video multimodal Transformer model to obtain the final comprehensive content representation features, specifically including: 3.1.1) The short video is converted into a frame sequence using FFmpeg, and 50 frames are selected for each video for uniform sampling. After extracting the frames, ResNet-34 residual neural network is used for feature extraction. After ResNet-34 feature extraction, the final size of the frame is a 1024-dimensional video feature vector. 3.1.2) The audio is also first separated into time sequences by FFmpeg, and then the audio data is sampled at a certain frequency by VGGish and filtered to obtain the spectral data. Finally, a 1024-dimensional audio feature vector is obtained. 3.1.3) The text data is processed by the trained language representation model BERT, and through the language model with masking mechanism, a bidirectional language representation is formed, resulting in a 512-dimensional text feature vector; 3.2) The tendency representation auxiliary module obtains tendency representation features based on video and user information, specifically including: 3.2.1) Sentiment as a Unit: Based on the input of the short video, video sentiment feature vectors with a dimension of 1024 are extracted from the video frames using SentiBank; audio features are extracted using VGGish and then the MFCC Mel-frequency cepstral coefficients are analyzed to obtain audio sentiment feature vectors with a dimension of 512; text sentiment feature vectors with a dimension of 1024 are extracted from the text using SKEP; after obtaining the sentiment feature vectors of video, audio, and text, a multi-head multimodal attention mechanism is used to first allocate internal attention, then continue cross-modal attention learning, and finally integrate the visual, audio, and text sentiment features to obtain the sentiment vector contained in the short video content; 3.2.2) To learn users' historical preferences, after retrieving user profiles, historical tags, and historical video information from the input information, a model is built based on the user's historical interests. Taking historical text as an example, the feature matrix is ​​obtained by retrieving the user's historical text. After setting the weights, the similarity matrix between the current text and historical text is obtained. ,in: After calculating the similarity between the text and the i-th historical text, the weight of each text is obtained as follows: ,in: and All of them are trainable parameters; 3.2.3) After obtaining the historical tendency vector and sentiment vector, the final vector is obtained by concatenating them through an attention mechanism. Where: user preference vector User ID Embedded , For mapping functions; 3.3) The label representation module obtains a graph attention neural network composed of label pairs based on the input label set. Different weights are assigned according to different relationships in the label pairs to obtain the initial association matrix of the label graph. Then, the relationship matrix is ​​updated through multiple layers of GAT, specifically including: 3.3.1) The initial input labels are a matrix, which can be viewed as a directed graph. For nodes... Directed edge source node Considered as its neighbor within the border, recorded as ; there will be directed edges The target node j is considered its outgoing neighbor, denoted as ; 3.3.2) Update the label node features, specifically: ,in: For the first Layer The feature vector of each node It is the first Layer weight matrix, It is the first Layer nodes To the node Attention weights It is the activation function ReLU; 3.3.3) After each GAT layer, the label relation matrix is ​​recalculated based on the updated labels. The update formula for the label relation matrix is: ,in: The labels output by the previous layer GAT are a matrix. The Hadamard product is the element-wise multiplication of matrices. 3.4) Obtain the prediction result through feature interaction using an MLP (Multilayer Perceptron), specifically including: representing content features... Tendency indicates characteristics and labels represent features The input to the Bi-Interaction layer of the MLP (Multilayer Perceptron) performs pairwise interactions. Specifically, Bi-Interaction transforms multiple embedding vectors of two elements into a single vector, which is then passed through a hidden layer composed of fully connected layers to capture the non-linear relationship between short videos, sentiments, and tags. The final hidden layer... The output is through Adjust the predicted score to The range is used to obtain the predicted probability of the label. The closer the value is to 1, the higher the predicted probability of this label.

3. A short video tag prediction system implementing the method of claim 1 or 2, characterized in that, include: The system comprises a video content feature representation module, a sentiment representation auxiliary module, a label representation module, and a multilayer perceptron prediction module. Specifically: the video content feature representation module inputs short videos into a multimodal Transformer based on a multi-head attention mechanism. It extracts features from the images, text, and audio of the short video sample content using a pre-trained model, and then fuses these features through the Transformer to obtain the comprehensive content features of the short video; the sentiment representation auxiliary module takes short video content and user information as input, obtaining comprehensive information based on user history information and the short video's sentiment tendency. It extracts features from the images, text, and audio of the short video sample content using a pre-trained model, and performs sentiment analysis through attention encoding to obtain the short video's sentiment tendency; it inputs user history information into an attention network to obtain the user's interest tendency, and then fuses the two tendencies through an attention mechanism to obtain the final sentiment tendency information; the label representation module inputs label text into a graph attention neural network (GAT) for processing to obtain information on the semantic relationships between labels; and the multilayer perceptron prediction module uses a multilayer perceptron to collaboratively interact with the feature information from each dimension, and the results are passed through a prediction layer to obtain the final label prediction score; The content pre-training model includes: a residual neural network ResNet for extracting content feature vectors from video frames, a deep learning-based audio feature extraction network VGGish for extracting content feature vectors from audio, and a pre-trained language representation model for extracting content feature vectors from text. The aforementioned sentiment representation auxiliary module includes: SentiBank, a concept classifier for extracting video sentiment feature vectors from video frames; VGGish, a deep learning-based audio feature extraction network for extracting features from audio and then analyzing Mel-frequency cepstral coefficients (MFCCs) to obtain audio sentiment feature vectors; and SKEP, a sentiment pre-training model based on sentiment knowledge enhancement for extracting text sentiment feature vectors from text.

4. The short video tag prediction system according to claim 3, characterized in that, The video content feature representation module includes: a video content feature vector extraction unit, an audio content feature vector extraction unit, a text content feature vector extraction unit, and a multimodal content feature fusion unit. Specifically: the video content feature vector extraction unit extracts features from the video frames of the input video using a pre-trained ResNet model to obtain video content feature vectors; the audio content feature vector extraction unit extracts features from the audio information of the input video using a pre-trained VGGish model to obtain audio content feature vectors; the text content feature vector extraction unit extracts features from the text information accompanying the input video using a pre-trained BERT model to obtain text content feature vectors; and the multimodal content feature fusion unit fuses the input video content features, audio content features, and text content features using an improved Transformer to obtain the final comprehensive content representation features.

5. The short video tag prediction system according to claim 3, characterized in that, The sentiment representation auxiliary module includes: a video sentiment feature vector extraction unit, an audio sentiment feature vector extraction unit, a text sentiment feature vector extraction unit, a multimodal sentiment feature fusion unit, a user sentiment feature vector extraction unit, and a weighted fusion feature unit. Specifically: the video sentiment feature vector extraction unit extracts features from the video frames of the input video using a pre-trained SentiBank model to obtain video sentiment feature vectors; the audio sentiment feature vector extraction unit extracts features from the audio information of the input video using a pre-trained VGGish model, and then obtains audio sentiment feature vectors by analyzing MFCC; the text sentiment feature vector extraction unit… The first unit extracts features from the text information accompanying the input video using the pre-trained model SKEP to obtain a text sentiment feature vector. The second unit fuses the input video sentiment features, audio sentiment features, and text sentiment features using an improved Transformer to obtain a final comprehensive sentiment tendency representation feature. The third unit extracts user tendency feature vectors by modeling users based on historical information and then using an attention mechanism to obtain user tendency representation features. The fourth unit performs a weighted average fusion of the input sentiment tendency representation features and user tendency representation features to obtain a comprehensive tendency representation feature.

6. The short video tag prediction system according to claim 3, characterized in that, The label representation module includes: a label initial matrix unit and a multi-layer GAT unit, wherein: the label initial matrix unit reconstructs edges based on the input label information by assigning weights to different semantics between labels to obtain an initial label matrix relationship as a directed graph; the multi-layer GAT unit takes the initial label matrix as input and automatically learns and updates the relationship weights during training through an attention mechanism to obtain label representation features.

7. The short video tag prediction system according to claim 3, characterized in that, The multilayer perceptron prediction module uses an MLP (Multilayer Perceptron) unit to capture the nonlinear relationship between short videos, tendencies, and tags by fusing the input content representation features, tendency representation features, and tag representation features in pairs and then passing them through a hidden layer composed of fully connected layers, thereby obtaining the probability of predicting tags.