A multimodal short video tag recommendation method integrating emotional information

Abstract

The present invention discloses a multimodal short video tag recommendation method that integrates emotional information, belonging to the field of video processing technology. The method comprises the following steps: constructing a short video sample set; inputting the short video samples into an initial multimodal tag recommendation model based on a multi-head attention mechanism and an autoencoder, so that the model extracts features from the image, audio, and text of the short video samples to obtain content features and emotional features, and fuses these features using an attention network to obtain multiple candidate video tags; training the initial multimodal tag recommendation model with the desired video tag as the target and the difference in text features between the candidate video tags and the desired video tags as the loss to obtain a target multimodal tag recommendation model; and inputting the current short video into the target multimodal tag recommendation model to generate a target video tag. By integrating image features, audio features, and text features, the present invention can fully utilize multimodal information related to the video and effectively improve the quality of the generated video tags.

Description

Technical Field

[0001] The present invention belongs to the field of video processing technology, and more specifically, relates to a multimodal short video tag recommendation method integrating emotional information. Background Art

[0002] Driven by the development of multimedia technology, portable mobile devices, and various short video platforms, short videos, as an extension of traditional text and graphic media, have gradually become a new form of mass communication for information acquisition and social interaction. Furthermore, short videos have a limited time span and can be easily filmed and shared instantly, making them widely disseminated and reaching a massive number of users. Recommender systems were initially primarily used in the commercial sector, analyzing the connections between users and products to recommend content that matches the user. Subsequently, recommendation systems have gradually expanded to social media and news platforms by analyzing user-related information, such as past tweets, comments, and articles, to recommend content of interest to users. The subjects in recommender systems can be broadly divided into users and items, and the recommendation tasks can be roughly categorized into the following: One approach is to match a list of items that match a user using algorithms, i.e., recommending items based on similarity between items; another approach is to use similarity between users to recommend the same item to groups of users with similar characteristics; and another approach is to use reasonable modeling of user-related information to match lists of items with similar attributes. The basic idea of ​​tag recommendation originates from this category of methods.

[0003] Tags are crucial for identifying specific information and for search engines to locate key sources across diverse fields. Tags can be single words, phrases without spaces, or even any combination of words prefixed with the # symbol. They facilitate the management and categorization of tweets with diverse content. Tags also allow users to access articles with similar content and tweets shared by other users through hyperlinks. Tagging services can benefit users in their searches and help them navigate content of greater interest. Automatic tag recommendation for text and images has become a significant research topic in recent years.

[0004] Existing short video tag recommendation models struggle to accurately predict text and tags containing emotional factors after multimodal information fusion. This is particularly true in the short video field, where images are richer and more likely to reflect user emotions. However, research on incorporating emotional information into short video tag recommendation is limited, and often simply linearly superimposed on content information, resulting in low prediction accuracy. Overall, the quality of short video tags generated using existing technologies needs to be further improved. Summary of the Invention

[0005] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention provides a multimodal short video tag recommendation method that integrates emotional information. Its purpose is to fully utilize the multimodal information related to short videos by integrating image features, audio features and text features, effectively improve the quality of the generated video tags, and thus solve the technical problem of poor quality of existing short video tags.

[0006] To achieve the above objectives, according to one aspect of the present invention, a multimodal short video tag recommendation method integrating emotional information is provided, comprising:

[0007] S1: Construct a short video sample set. The label of each short video sample includes multiple corresponding platform labels, and its attributes include corresponding image features, audio features, and text features.

[0008] S2: Input the short video sample into an initial multimodal tag recommendation model based on a multi-head attention mechanism and an autoencoder, so that it extracts features from the image, audio, and text of the short video sample to obtain content features and emotional features, and fuses them using an attention network to obtain multiple candidate video tags; with the desired video tag as the target and the difference between the text features of the candidate video tags and the desired video tag as the loss, the initial multimodal tag recommendation model is trained to obtain a target multimodal tag recommendation model;

[0009] S3: Input the current short video into the target multimodal tag recommendation model to generate a target video tag.

[0010] In one embodiment, the short video sample set includes a training set, a validation set, and a test set; S2 includes:

[0011] S21: Inputting the training set into the initial multimodal tag recommendation model; the initial multimodal tag recommendation model includes: a fusion of a content feature extraction module, a sentiment feature extraction module and a tag prediction module;

[0012] S22: Using the pre-trained model in the content feature extraction module to extract features from the image modality and audio modality of the training set, then extracting text features, and using a multimodal Transformer model to fuse the image features, audio features, and text features to obtain corresponding content features;

[0013] S23: Using the pre-trained model in the emotion feature extraction module to extract features from the image modality and audio modality of the training set, then extracting text features, and then using a multi-head attention mechanism to fuse the image features, audio features, and text features to obtain corresponding emotion features;

[0014] S24: Using the label prediction module to fuse the content features, emotional features, and label text features corresponding to the training set to obtain a short video fusion feature, and generating a number of candidate video labels based on the short video fusion feature; calculating the error in text features between each candidate video label and the real video label, so as to reduce the loss through continuous iterative training;

[0015] S25: Using the verification set and the test set to respectively verify and test the initial multimodal tag recommendation model in the training process, and using the initial multimodal tag recommendation model that passes the verification and test as the target multimodal tag recommendation model.

[0016] In one embodiment, the content feature extraction module is based on a modal Transfomer structure; the sentiment feature extraction module is based on a cross-modal multi-head attention structure; and the label prediction module is based on an attention network.

[0017] In one embodiment, the content feature extraction module includes: an encoder layer, a stacked block layer and a fusion layer connected in sequence; the encoder layer is used to encode different modal information, and the stacked block layer is used to perform modal representation with an attention mechanism; the fusion layer is used to fuse cross-modal information to obtain a final content feature representation; wherein, in the stacked block layer, N stacked blocks are used for each modality to implement feature representation with an attention mechanism, and the stacked blocks include a multi-head attention mechanism, a cross-attention mechanism and two feedforward neural networks.

[0018] In one embodiment, the emotion feature extraction module performs inter-modal feature fusion on image features, audio features, and text features through a multimodal multi-head attention framework (MMFA) to obtain an emotion representation vector corresponding to the short video sample.

[0019] Among them, MMFA includes multi-head self-attention mechanism and multi-head co-attention mechanism.

[0020] In one embodiment, before S21, S2 further includes:

[0021] Through integrity checks, video samples that cannot be played normally are eliminated; video samples with a duration below a threshold, tweet information below a word count threshold, and / or missing audio channels are filtered out.

[0022] In one embodiment, the method of extracting features from the image, audio, and text of the short video sample to obtain content features and emotional features includes:

[0023] Dividing the audio data of the short video sample into audio segments according to a preset time interval T, extracting features of each audio segment, and combining them into audio features in chronological order;

[0024] Extracting a frame of image from the image data of the short video sample according to a preset number of video frames N, performing feature extraction on each frame of image, and combining them into image features in chronological order;

[0025] A vocabulary library is constructed using the tweet information and original tag information of the short video sample, and a pre-trained language model is used to represent the words in the vocabulary library as vectors and perform feature extraction to obtain text features. Among them, for conjunctions with a length greater than a length threshold, a word segmentation tool is used to perform word segmentation, and then features are obtained through the pre-trained language model, and then averaged to obtain context features.

[0026] In one embodiment, the constructing a vocabulary using tweet information and original tag information of the short video sample includes:

[0027] All tweet information and original tag information of the short video sample are counted, and they are aligned, segmented, and word frequency is counted in turn; each word is sorted in descending order according to the word frequency, and words with a frequency higher than N times are taken to construct the word library, where N is a preset ratio parameter.

[0028] In one embodiment, the method of constructing a vocabulary using the tweet information and original tag information of the short video sample further includes: filtering non-English characters in the tweet information and original tag information; performing root restoration on English words with the same root; and performing word segmentation on conjunctions whose length is greater than a length threshold to obtain several separate words.

[0029] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0030] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0031] (1) By fusing image features, audio features, and text features, the present invention can fully utilize multimodal information related to the video and effectively improve the quality of the generated video tags.

[0032] (2) The initial multimodal tag recommendation model provided by the present invention can perform feature fusion step by step and in a step-by-step manner, thereby more effectively fusing multimodal information based on the influence weight of different information on video tags and improving the quality of the final generated video tags.

[0033] (3) By performing the above-mentioned preprocessing operation on the collected video data before feature extraction, the present invention can effectively avoid errors and redundancies in the data set, thereby ensuring the training effect of the model and ultimately ensuring that the video labels generated by the model have high quality.

[0034] (4) When performing multimodal feature fusion, the present invention takes the emotional information in the video into consideration. Through the multi-task learning method, it can capture the video content information while also capturing the video emotional information, which is beneficial to the prediction of labels with emotional attributes, thereby further optimizing the quality of generated video labels. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A logical diagram of a multimodal short video tag recommendation method integrating emotional information according to an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of the structure of a content feature extraction module of an initial multimodal tag recommendation model provided by one embodiment of the present invention;

[0037] Figure 3 A schematic diagram of the structure of stacking blocks in a content feature extraction module of an initial multimodal tag recommendation model provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of the structure of the sentiment feature extraction module of the initial multimodal tag recommendation model provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0040] like Figure 1 As shown, the present invention provides a multimodal short video tag recommendation method that integrates emotional information, a model training stage and a tag prediction stage;

[0041] The model training phase includes:

[0042] (S1.1) Collect video data and corresponding tag data; by comparing the number of videos and tag quality of various short video platforms, this embodiment chooses to use Python web crawlers to collect data from the Vine platform; the specific collection method is as follows: the original Vine data set needs to be processed, and the intersection of user ID-video URL, user ID-user tweet, and user ID-tweet tag is taken. At the same time, the data containing only video or text is filtered and processed, and the data is connected with the uniqueness of user ID, so that the information contained in the final data set is video URL-text, where the text is a tweet containing a specific tag. At the same time, tags that appear less than 10 times in the entire data set are removed to reduce the randomness of the appearance of tagged videos, and some data preprocessing methods to be described below are used to clean the data. By setting up a local proxy and crawler program to crawl the binary media files in the video download address, we finally obtained 40,049 usable short videos and 1,935 different tags. At the same time, about 86% of the short videos in the dataset are distributed in about 6 seconds. Each video contains at least 1 tag and at most 21 tags. The average number of tags per video is 4.8, and the average word length of a tweet is 9.73.

[0043] (S1.2) Preprocessing the collected videos and corresponding tags; the preprocessing operation specifically includes: eliminating videos that cannot be played normally through integrity checking; the specific integrity checking tool used is the open source tool ffmpeg; eliminating videos that cannot be played normally through integrity checking; filtering out video data with short duration, short tweet information, and / or missing audio channels; In this embodiment, step (S1.2) also includes:

[0044] The tag lexicon construction method involves segmenting all videos collected during the model training phase and counting word frequencies. After sorting the words from highest to lowest frequency, the lexicon is constructed using words with a frequency greater than N times, where N is a preset ratio parameter. During lexicon construction, non-English characters in tweets and original tag information are filtered out. English words with the same root are restored to their root form. Longer conjunctions are segmented using a word segmentation tool to produce individual words.

[0045] The video data is separated into image data and audio data, and features are extracted from the image data, audio data and the corresponding user tweet data respectively; several platform labels corresponding to the short video are used as training labels, and the image features, audio features and tweet features of the short video are used as the attributes of the video to form a sample. All samples constitute a data set, and the data set is divided into a training set, a validation set and a test set.

[0046] (S2) Model training phase includes:

[0047] Separate the video data into image data and audio data, and perform feature extraction on the image data, audio data, and corresponding user tweet data. Use several platform labels corresponding to the short video as training labels, and use the image features, audio features, and tweet features of the short video as the attributes of the video to form a sample. All samples constitute a dataset, and the dataset is divided into a training set, a validation set, and a test set.

[0048] An initial multimodal tag recommendation model is established based on a multi-head attention mechanism and an autoencoder. This model extracts content and sentiment features from image, audio, and tweet comment context using different pre-trained models. The model then fuses these features together using the content and sentiment extraction modules. Based on these fused features, the model generates several video tags related to the video content and user recommendations. A multi-task learning approach enables the content and sentiment extraction modules to learn multi-task information, exploring modal information through different pre-trained models. Compared to single-task learning, the sentiment extraction module effectively integrates sentiment information from short videos while extracting content features. The attention network fuses feature vectors from different tasks, while also avoiding noise and improving the performance of the initial multimodal tag recommendation model. The model is trained, validated, and tested using training, validation, and test sets, respectively, to obtain the target multimodal tag recommendation model.

[0049] Furthermore, the initial multimodal label recommendation model includes a content feature extraction module, an emotion feature extraction module and a label prediction module; the content feature extraction module is used to extract features of short video data in image and audio modalities through a pre-trained model related to video content information, and fuse the image, audio and text modalities through a multimodal Transformer model; the emotion feature extraction module is used to extract features of short video data in image and audio modalities through a pre-trained model related to video emotion information, and fuse the image, audio and text modalities through a multi-head attention mechanism; the label prediction module is used to fuse the video content fusion features output by the content feature extraction module, the emotion fusion features output by the emotion feature extraction module and the label text features related to the video to obtain short video fusion features, and generate several candidate video labels based on the short video fusion features; the label prediction module is also used to calculate the error between the text features of the generated video label and the real video label as a loss, so as to reduce the loss through continuous iterative training.

[0050] Based on the above-mentioned initial multimodal tag recommendation model structure, the present invention can gradually and step-by-step perform feature fusion, thereby more effectively fusing multimodal information based on the influence weight of different information on video tags and improving the quality of the final generated tags.

[0051] In step (S2), feature extraction is performed on different modal features in the content feature extraction module and the emotion feature extraction module, including: for visual features, the content feature extraction module is framed using the ffmpeg open source streaming media tool, VGG16-LSTM is used to extract visual features in the video and capture the continuous information of the picture frame in time, and a slow fusion method is used to obtain a 1024-dimensional visual feature representation vector; for the emotion feature extraction module, a 3D-CNN pre-trained model is used to obtain a 512-dimensional visual feature representation; for audio features, VGGish is used to extract the feature vector of the audio feature in the content feature extraction module, the audio data is resampled at a certain frequency, the sampled data is Fourier transformed to obtain its spectrum diagram, and finally the 1024-dimensional sound spectrum data of the audio file is obtained through a filter; for the emotion feature extraction module, the Librosa sound spectrum processing library is used and input into the convolutional neural network to obtain a 300-dimensional feature vector. For text features, the Bert pre-trained language model is used to extract features from the content feature extraction module and hashtag word text to obtain a 768-dimensional embedding representation; for the sentiment feature extraction module, the pre-trained English corpus Glove model is used, and the final text vector size is 300 dimensions.

[0052] In addition, the present application establishes an automatic generation model for barrage comments based on autoencoders and autodecoders. The automatic generation model for barrage comments is used to perform time series analysis on image features, audio features, and barrage comment context features respectively, and then realize multimodal feature fusion together with the text features of the barrage comments, and generate a barrage comment related to the video content and barrage context based on the fused features; the automatic generation model for barrage comments includes: a content feature extraction module, an emotional feature extraction module, and a label prediction module; the content feature extraction module performs content information fusion, including: the encoder layer encodes different modal information, the stacked block layer performs modal representation with an attention mechanism, and the fusion layer fuses cross-modal information to obtain the final video content feature representation.

[0053] like Figure 2 As shown in the figure, it mainly consists of three parts: the encoding layer encodes different modal information, the stacking block layer performs modal representation with an attention mechanism, and the fusion layer fuses cross-modal information to obtain the final video content feature representation.

[0054] In the stacking block layer, N stacking blocks are used for each modality to implement feature representation with attention mechanism for modality information, where each stacking block structure is as follows: Figure 3 As shown in the figure, it mainly consists of four parts: Multi-Head Attention mechanism, Cross Attention mechanism and two Feedforward Neural Networks (FNN).

[0055] The Multi-Head Attention module is a key structure in the Transformer model. Building on the traditional self-attention mechanism, it adds more matrix parameters, allowing the Q, K, and V matrices to continue multiplying with multiple matrices after multiplying the original input vector. The number of matrices represents the number of heads in the multi-head attention mechanism. This allows the data stream to be divided into different subspaces, allowing the model to focus on different aspects of information. Cross-attention actually resides in the decoder in the original Transformer model. The encoder module generates the corresponding K and V matrices, which are combined with the decoder module's own Q matrix information to achieve information fusion. This information fusion between the encoder and decoder was originally used for machine translation tasks, but it can also be used to fuse information between different modalities in short videos, allowing the Transformer model to more uniformly and naturally combine different modal information. The role of the FNN is to connect the multiple subspace matrices generated by multi-head attention into a unified matrix, which is then multiplied by a parameter matrix of an information to obtain the final single modality information fusion matrix.

[0056] In the stacked block of the tth layer, the input of the same modality information comes from the output information of the t-1 layer.

[0057]

[0058]

[0059] The feature fusion method between different modalities is represented as follows, where c represents the text modality, v represents the visual modality, and a represents the video modality. Taking the calculation method of visual modality information v as an example, after the cross-attention mechanism, a fusion gate is constructed through a multi-layer perceptron and multiplied by the corresponding weight matrix to perform inter-modal feature fusion on text and audio information respectively.

[0060]

[0061]

[0062]

[0063]

[0064] Then it will Input into the feedforward neural network to get the output of the tth stacking block. In the same way, the output of the audio mode and text mode in the t layer can be obtained and

[0065]

[0066] Next, the output of the stacked block is converted into a vector of a specific length through a weight pooling operation.

[0067]

[0068]

[0069] After obtaining the feature representations related to the three modal attentions in the same way, a feature fusion gate is constructed in the fusion layer through a multi-layer perceptron, and the different modal information is multiplied with a certain weight and then added to obtain the fusion vector in the stacked block.

[0070] g v =MLP(F c , F v , F a )#(10)

[0071]

[0072] In step (S2), the emotion feature extraction module performs emotion information fusion, including:

[0073] Compared with content attribute labels, labels with emotional attributes are easier to predict. After extracting feature representations through emotional pre-training models of different modalities, the overall architecture of the module is as follows: Figure 4 As shown in the figure, the multi-modal multi-head attention framework MMFA (Multi-Modal Multi-Head Attention) is used to fuse visual, audio, and text features to obtain the emotion representation vector in the short video.

[0074] The multi-head attention mechanism allows the model to consider the interrelationships between different representation spaces to derive the final representation information. MMFA includes both multi-head self-attention and multi-head mutual-attention. The former functions similarly to content feature extraction (MTT). It uses more parameter matrices to disperse the data stream into different subspaces. The attention weight matrix information originates from the same modality and is used only to enhance the information characteristics of that modality. The latter, on the other hand, uses attention weight matrix information from different modalities to facilitate the fusion of information from different modalities. Specifically, it fuses information from different modalities pairwise, such as vision-text, text-audio, and audio-vision. Essentially, it builds on the Transformer as a pure image classification task. In the input linear layer, the linear transformation unit, which originally only contains image information, is transformed into a linear layer that contains information from both modalities. The two modal information, after being input to the attention mechanism, is then linearly superimposed to form the fused features of different modalities under the mutual-attention mechanism. Finally, the single-modality representation is combined with the mutual-modality representation and input into the softmax layer to generate sentiment attribute features.

[0075] For the multi-head self-attention module, taking the visual modality v as an example, the Q, K, and V matrices in the traditional multi-head self-attention mechanism need to be changed accordingly through the fusion of single modality information, and the entire input part is converted into visual modality information.

[0076]

[0077] Assuming that the number of linear variation layers is m, the final single-modal fusion is expressed as shown in formula (14).

[0078]

[0079] The multi-head co-attention mechanism fuses information from different modalities, which is essentially a pairwise feature fusion. Taking the visual modality v and the audio modality a as an example, the calculation method is shown in formula (15).

[0080]

[0081] Assuming that the number of linear variation layers is m, the fusion of the final fusion modalities is expressed as shown in formula (16).

[0082]

[0083] The feature representation obtained after the modal information is fused through the self-attention mechanism and the co-attention mechanism includes both the information fusion between the two modalities and the modal information itself.

[0084]

[0085] The label prediction stage includes:

[0086] For a given video-label pair (x i ,y i ), assuming that the final content feature vector Sentiment feature vector Hashtag represents vector h. Since tags can be divided into two categories based on semantic information, one of which is tags with emotional attributes and the other is tags with content attributes, the semantic classification determines that candidate tags will have different attention weights on emotional features and content features. Therefore, an attention mechanism is needed to fuse the multimodal features of the two. Formulas (18) and (19) represent the emotional attributes and content attributes after combining the original tag information.

[0087]

[0088]

[0089] In order to obtain the attention weights of the two, formula (20) and formula (21) obtain the weight distribution of different features through exponential operation, as shown below.

[0090]

[0091]

[0092] After inputting into the attention network, the fusion ratio of the two is obtained The final fusion vector is In order to evaluate the relevance score of a given short video and tag, x i，j The input is input into a multi-layer perceptron, where the representation vector of the short video and the label embedding can learn the correlation between the two through a nonlinear hidden layer, where the hidden layer of the multi-layer perceptron is expressed as formula (22).

[0093]

[0094] During training, this example treats label recommendation as a binary classification task, where if the predicted video-label pair appears in the dataset, it is a positive sample; otherwise, the given video-label pair is a negative sample. Negative sample labels of the video are selected by random sampling, and cross entropy is used as the loss function.

[0095]

[0096] According to another aspect of the present invention, a multimodal short video tag recommendation system that integrates emotional information includes: a computer-readable storage medium and a processor; the computer-readable storage medium is used to store an executable program; the processor is used to read the executable program stored in the computer-readable storage medium and execute the multimodal short video tag recommendation method that integrates emotional information provided by the present invention.

[0097] Tags are important content required for specific information and search engines to locate key sources of information in various fields. Tags can be single words, phrases without spaces, or even any combination of words prefixed with the symbol #, which can be used to manage and classify tweets with different contents. Articles with similar content and tweets shared by other users can be obtained through hyperlinks of tags. Tag services can benefit users in searches and browse content that they are more interested in. Automatic tag recommendation for text and images has become an important research topic in recent years. However, there are few models for initial multimodal tag recommendation. Although some methods have been proposed for recommendation tasks for text, images or blogs, they are not applicable to the short video field. This is because these models are designed for their respective fields, and the structure of short videos is also different from that of text and pictures. The present invention solves the above problems and can better recommend tags for short videos.

[0098] It will be easily understood by those skilled in the art that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimodal short video tag recommendation method integrating emotional information, characterized in that: include: S1: Construct a short video sample set. The label of each short video sample includes multiple corresponding platform labels, and its attributes include corresponding image features, audio features, and text features. S2: Input the short video sample into an initial multimodal tag recommendation model based on a multi-head attention mechanism and an autoencoder, so that it extracts features from the image, audio, and text of the short video sample to obtain content features and emotional features, and fuses them using an attention network to obtain multiple candidate video tags; with the desired video tag as the target and the difference between the text features of the candidate video tags and the desired video tag as the loss, the initial multimodal tag recommendation model is trained to obtain a target multimodal tag recommendation model; S3: Inputting the current short video into the target multimodal tag recommendation model to generate a target video tag; The initial multimodal tag recommendation model includes: a content feature extraction module, a sentiment feature extraction module and a tag prediction module; the content feature extraction module is based on a modal Transfomer structure; the sentiment feature extraction module is based on a cross-modal multi-head attention structure; the tag prediction module is based on an attention network; The content feature extraction module includes: an encoder layer, a stacked block layer, and a fusion layer connected in sequence; the encoder layer is used to encode information of different modalities, the stacked block layer is used to perform modal representation with an attention mechanism; the fusion layer is used to fuse cross-modal information to obtain a final content feature representation; the stacked block layer uses N stacked blocks to implement feature representation with an attention mechanism for each modality, and the stacked blocks include a multi-head attention mechanism, a cross-attention mechanism, and two feedforward neural networks; The emotion feature extraction module uses the multimodal multi-head attention framework MMFA to perform inter-modal feature fusion on image features, audio features and text features to obtain the emotion representation vector corresponding to the short video sample; MMFA includes a multi-head self-attention mechanism and a multi-head co-attention mechanism.

2. The multimodal short video tag recommendation method integrating emotional information according to claim 1, characterized in that: The short video sample set includes a training set, a validation set, and a test set; S2 includes: S21: Inputting the training set into the initial multimodal tag recommendation model; S22: Using the pre-trained model in the content feature extraction module to extract features from the image modality and audio modality of the training set, then extracting text features, and using a multimodal Transformer model to fuse the image features, audio features, and text features to obtain corresponding content features; S23: Using the pre-trained model in the emotion feature extraction module to extract features from the image modality and audio modality of the training set, then extracting text features, and then using a multi-head attention mechanism to fuse the image features, audio features, and text features to obtain corresponding emotion features; S24: Using the label prediction module to fuse the content features, emotional features, and label text features corresponding to the training set to obtain a short video fusion feature, and generating a number of candidate video labels based on the short video fusion feature; calculating the error in text features between each candidate video label and the real video label, so as to reduce the loss through continuous iterative training; S25: Using the verification set and the test set to respectively verify and test the initial multimodal tag recommendation model in the training process, and using the initial multimodal tag recommendation model that passes the verification and test as the target multimodal tag recommendation model.

3. The multimodal short video tag recommendation method integrating emotional information as claimed in claim 1, characterized in that: Before S21, S2 further includes: Through integrity checks, video samples that cannot be played normally are eliminated; video samples with a duration below a threshold, tweet information below a word count threshold, and / or missing audio channels are filtered out.

4. The multimodal short video tag recommendation method integrating emotional information as claimed in claim 1, characterized in that: The method is to extract features of the image, audio, and text of the short video sample to obtain content features and emotional features, including: Dividing the audio data of the short video sample into audio segments according to a preset time interval T, extracting features of each audio segment, and combining them into audio features in chronological order; Extracting a frame of image from the image data of the short video sample according to a preset number of video frames N, performing feature extraction on each frame of image, and combining them into image features in chronological order; A vocabulary library is constructed using the tweet information and original tag information of the short video sample, and a pre-trained language model is used to represent the words in the vocabulary library as vectors and perform feature extraction to obtain text features. Among them, for conjunctions with a length greater than a length threshold, a word segmentation tool is used to perform word segmentation, and then features are obtained through the pre-trained language model, and then averaged to obtain context features.

5. The multimodal short video tag recommendation method integrating emotional information as claimed in claim 4, characterized in that: The method of constructing a vocabulary using tweet information and original tag information of the short video sample includes: All tweet information and tag information of the short video sample are counted, and they are aligned, segmented, and word frequency is counted in turn; each word is sorted in descending order according to the word frequency, and words with a frequency higher than N times are taken to construct the word library, where N is a preset ratio parameter.

6. The multimodal short video tag recommendation method integrating emotional information as claimed in claim 5, characterized in that: The method of constructing a vocabulary using the tweet information and original tag information of the short video sample also includes: filtering non-English characters in the tweet information and the original tag information; performing root restoration on English words with the same root; and performing word segmentation on conjunctions whose length is greater than a length threshold to obtain several individual words.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images