Short video recommendation method and device based on content and interest learning

Abstract

The application discloses a short video recommendation method and device based on content and interest learning, and the method comprises the following steps: constructing a user behavior sequence, and excavating user interest features from the user behavior sequence; mapping high-dimensional one-hot coding features of basic information of the short video into low-dimensional feature vectors through an embedding layer, so as to excavate short video features; paying attention to cross-modal multi-head attention network of specific modalities, enhancing features of image modalities, and guiding learning of the short video features; guiding user interest feature learning through a time sequence interest extraction network for guiding user feature learning; predicting a probability of user clicking the short video through a click rate prediction network based on the learned short video features and user interest features, and recommending the short video based on the predicted probability. The device comprises a processor and a memory. The application can guide short video feature learning, pay attention to cross-modal multi-head attention network of specific modalities, realize mining and utilization of short video multi-modal content features, and guide short video feature learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multimedia and big data analysis, and particularly relates to a short video recommendation method and device based on content and interest learning. BACKGROUND

[0002] With short videos gradually becoming a new way for people to obtain information, short video recommendation has attracted widespread attention in the field of multimedia computing research. Reasonably and efficiently recommending favorite short videos to users is the key to the recommendation system. However, when facing a large amount of short video data, how to better select short videos that users may like from candidate short video sequences is still a difficult problem for many people.

[0003] Nowadays, deep learning has made rapid progress in various fields, and it has also been widely applied to the field of short video recommendation. The advantage of using deep learning is that it can automatically fit the relationship between users and short videos to achieve the purpose of video recommendation. SUMMARY

[0004] The present application provides a short video recommendation method and device based on content and interest learning. The present application uses the multi-modal information of short videos to obtain short video content information while guiding short video feature learning, and uses user historical behavior information to obtain user interest features while learning user features, and fuses the learned features to realize short video recommendation. Details are described below:

[0005] A short video recommendation method based on content and interest learning, the method comprises:

[0006] Constructing a user behavior sequence, and discovering user interest features from the user behavior sequence;

[0007] Mapping high-dimensional one-hot encoding features of basic information of short videos to low-dimensional feature vectors through an embedding layer, for discovering short video features;

[0008] A cross-modal multi-head attention network focusing on specific modalities enhances the features of the image modality and guides the learning of short video features;

[0009] A time-series interest extraction network that guides user feature learning is used to guide user interest feature learning;

[0010] The short video features and user interest features learned through the above are input into a click rate prediction network to predict the probability of a user clicking a short video, and the short video is recommended based on the predicted probability.

[0011] The cross-modal multi-head attention network is as follows:

[0012] The self-attention features within the learning modalities and the cross-modal attention features between the learning modalities are learned; the input short video features are used to guide the learning of the short video features in the nonlinear transformation layer;

[0013] The multi-modal features As input, x m is any kind of modal, d m is the feature dimension of the modal m, m∈{v,a,t} respectively represents the image feature, sound feature and text feature of the short video;

[0014] Each modality is divided into n blocks Linear transformation is performed on these modality blocks to map them to the same dimension;

[0015] After transformation, position encoding is performed on the corresponding features to preserve the spatial position information within each modality, and finally the input of GFCT is obtained

[0016]

[0017] wherein, The position encoding of the modal m, is the i-th block feature vector of the modal m, is the class label vector in VIT, c is the feature dimension of the GFCT input, and FC represents the full connection layer.

[0018] The cross-modal multi-head attention network comprises a multi-head self-attention module that focuses on a specific modality, and the multi-head self-attention module learns the relationship between the feature blocks within each modality from its own perspective:

[0019]

[0020] wherein, l=1,2,…,L represents the l-th layer of the FCT module, L represents an L-layer FCT module, and the input of the first layer is represents the feature representation within the modal m of the output of the l-th layer, and MSA represents the multi-head self-attention mechanism;

[0021]

[0022]

[0023] wherein, respectively represent the cross-modal interaction features of the visual modality v and the text modality t, and the sound modality a of the l-th layer, C represents the dimension of the feature, and n represents the length of the feature;

[0024] The two kinds of interaction features are used to enhance the image features to increase the expression of visual information;

[0025]

[0026] wherein, represents the l-th layer enhanced visual feature;

[0027]

[0028] wherein, represents the sound feature after FMCA, represents the text feature after FMCA.

[0029] wherein, the method further comprises: proposing a GMLP for guiding feature learning, the GMLP input is three modalities after the FMCA module short video content feature of the upper layer

[0030]

[0031] wherein, GFCT(·) represents a cross-modal multi-head attention network capable of guiding short video feature learning while focusing on specific modalities, represents the output feature of the l-th layer modal m of GFCT, represents the l-th layer content feature, represents the l-th layer short video content feature, represents the l-th layer text feature, represents the l-th layer sound feature, represents the enhanced video feature.

[0032] the:

[0033]

[0034]

[0035] wherein, W I represents a learning parameter; b represents a bias.

[0036] The time interest extraction network for guiding user feature learning is:

[0037] s + =GTIN(S,e video ,e u )

[0038] wherein, S={s1,…s t ,…,s T} represents user behavior sequence features, GTIN(·) represents a time interest extraction network for guiding user feature learning, s t represents the user behavior feature at t time, e videodenotes the embedding feature of the current video, e u denotes the embedding feature of the user, s + is the interest feature of the user for the current video.

[0039] extract the interest feature s t of each historical video τ :

[0040] s t = match(e video,t , e u )

[0041] s τ = match(e video , e u )

[0042] wherein e video,t denotes the embedding feature of the video clicked at time t, s τ denotes the interest feature constructed by the user and the current video, and match is an attention matching mechanism:

[0043]

[0044]

[0045] wherein Similary(γ k ,β) is the kth activation weight, γ k represents the kth feature, W k represents a learnable parameter, and β can be an interest feature or a user ID feature; match(γ,β) is the final matching feature, and the user's high-level temporal interest feature is extracted using an LSTM:

[0046] [h1,…,h t ,…,h T ] = LSTM([s1,s2,…,s T ])

[0047] wherein h t represents the historical interest constructed at time t, s T represents the Tth user behavior feature, the historical interest is modeled using a match network, and the final interest feature of the user for the current video is obtained.

[0048] A short video recommendation device based on content and interest learning, the device comprising: a processor and a memory, the memory storing program instructions, and the processor invoking the program instructions stored in the memory to make the device execute the method steps of any one of the first aspect.

[0049] The technical solution provided by the present application has the beneficial effects of:

[0050] 1. The present application uses a cross-modal multi-head attention network capable of guiding short video feature learning while focusing on specific modalities to realize the mining and utilization of short video multi-modal content features, and guides short video feature learning.

[0051] 2. The present application uses a time interest extraction network to realize user interest modeling, and guides user feature learning. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A flowchart of a short video recommendation method based on content and interest learning. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application clearer, the embodiments of the present application are described in further detail below.

[0054] Example 1

[0055] A short video recommendation method based on content and interest learning, the method comprising the following steps:

[0056] Step 101: Construct a user behavior sequence, and mine user interest features from the user behavior sequence;

[0057] Step 102: Map high-dimensional one-hot (one-hot encoding) features of basic information of a short video to low-dimensional feature vectors through an Embedding Layer (embedding layer) to mine short video features;

[0058] Step 103: Use a cross-modal multi-head attention network capable of guiding short video feature learning while focusing on specific modalities to enhance the features of the image modality and guide the learning of short video features;

[0059] Step 104: Use a time interest extraction network to guide user interest feature learning;

[0060] Step 105: Use the learned short video features and user interest features to predict the probability of a user clicking a short video through a click rate prediction network, and recommend short videos based on the predicted probability.

[0061] In step 101, a user behavior sequence composed of user historical clicked short videos is constructed using existing data sets, wherein each clicked video contains information such as video author, video duration, etc. The historical clicked video sequence is used to explore user click interest; and the multi-modal information (text, picture, music, etc.) of the short video is used to extract short video content features.

[0062] In step 102, an Embedding Layer is used to project the high-dimensional vector (author ID, short video duration, etc.) into a low-dimensional feature space.

[0063]

[0064] wherein V n represents the embedding matrix of the nth field, represents the embedding vector of the nth field. is the basic information of the input short video, vector represents a vector, and scalar represents a constant.

[0065] Meanwhile, the user ID and the short video ID are mapped to the low-dimensional feature space:

[0066] e u = V U U

[0067] e I = V I I

[0068] wherein U represents the user ID, I represents the short video ID, V U and V I are the mapping dictionary matrices of the user ID and the video ID, e u and e I represent the low-dimensional feature representations of the user and the short video, respectively.

[0069] The user behavior feature S = {s1, … s t , …, s T} is constructed, wherein the user behavior feature s t at time t contains all short video information features, i.e. represents the jth short video information feature of the user.

[0070] In step 103, the multi-modal information (text, image, sound, etc.) of the short video is used to obtain short video content enhanced features, while guiding the short video features to learn.

[0071]

[0072] wherein GFCT(·) represents a cross-modal multi-head attention network capable of guiding short video feature learning while focusing on specific modalities, represents the output feature of the l-th layer modal m of GFCT, represents the l-th layer content feature, represents the l-th layer short video content feature, represents the l-th layer text feature, represents the l-th layer sound feature, represents the enhanced video feature.

[0073] In the above step 104, the user's interest features are mined by using the user's historical clicked video information features (such as video author ID, work city ID, music ID, etc.), while guiding the learning of the user's features.

[0074] s + =GTIN(S,e video ,e u )

[0075] wherein S={s1,…s t ,…,s T} represents the user behavior sequence feature, GTIN(·) represents a time sequence interest extraction network capable of guiding user feature learning, s t represents the user behavior feature at t moment, e video represents the current video embedding feature, e u represents the user's embedding feature, s + is the user's interest feature for the current video.

[0076] In the above step 105, the learned short video content feature user interest feature D, user feature e u and video feature e I are fused to predict the probability of clicking the short video.

[0077]

[0078] wherein, is the final output click rate probability, Sigmoid(·) is an activation function, MLP represents a multi-layer perceptron, and Concat represents a concatenation operation.

[0079] Embodiment 2

[0080] The scheme in embodiment 1 is further introduced below in combination with specific calculation formulas, which are described in detail as follows:

[0081] Step 201: Construct a user behavior sequence composed of user historical clicked short videos to mine user interest features using existing data sets;

[0082] Wherein each clicked video contains video author, video duration and other information. Using the historical clicked video sequence feature S = {s1, … s t , …, s T} to mine user click interest s + While guiding short video learning; using the multi-modal information x m of short video to extract short video content features While guiding user feature learning. Wherein m ∈ {v, a, t}, representing text features, picture features, and music features respectively.

[0083] Step 202: Use Embedding Layer to project (author ID, short video duration, etc.) high-dimensional vector into low-dimensional feature space;

[0084]

[0085] Wherein V n represents the embedding matrix of the nth field, represents the embedding vector of the nth field. is the basic information of the input short video. In this way, after the embedding layer, the short video information feature Where J represents that the short video information feature contains J feature fields.

[0086] At the same time, map the user ID and short video ID to the low-dimensional feature space:

[0087] e u = V U U

[0088] e I = V I I

[0089] Wherein U represents the user ID and I represents the short video ID, V U and V I are the mapping dictionary matrix of user ID and video ID, e u and e I represent the low-dimensional feature representation of the user and the short video respectively.

[0090] The constructed user behavior feature S = {s1, … s t , …, s T}, the user behavior feature s t at time t contains all short video information features, that is

[0091] Step 203: using a cross-modal multi-head attention network (GFCT) that can guide short video feature learning while focusing on specific modalities to enhance the features of the image modality and guide the learning of short video features;

[0092] In GFCT, this module can not only learn intra-modal self-attention features, but also learn cross-modal attention features between modalities to strengthen the learning of a specific modality. At the same time, the short video features input to the nonlinear transformation layer of GFCT can guide the learning of short video features. The multi-modal features As input, x m can be any number of modalities, d m is the feature dimension of modality m. Here m ∈ {v, a, t} represents the image features, sound features and text features of the short video respectively. Each modality is divided into n blocks Then linearly transform these modality blocks to map them to the same dimension.

[0093] After transformation, the corresponding features are position encoded to preserve the spatial position information within each modality, and finally the input of GFCT is obtained

[0094]

[0095] where, the position encoding of modality m, is the i-th block feature vector of modality m, is the class label vector in VIT, c is the feature dimension of GFCT input, and FC represents the full connection layer.

[0096] Similar to the Transformer, GFCT also contains three main data processing modules: focus specific modality cross-modality multi-head attention module (FMCA), guide multi-layer perceptron module (GMLP), and layer normalization module (LN).

[0097] In FMCA, first use the multi-head self-attention module (MCA) to learn the relationship between the feature blocks (embedding words) within each modality from the perspective of itself:

[0098]

[0099] where l = 1, 2, …, L represents the l-th layer of the FCT module, L represents an L-layer FCT module, and the input of the first layer is represents the modal m in the l-th layer output feature representation, and MSA represents a multi-head self-attention mechanism. This part of the transformation can be regarded as an extraction process of the modal itself potential information.

[0100] At the same time, starting from the internal relationship between the modalities, a multi-head attention mechanism (MCA) is used to explore the relationship between the modalities to further propose the modal latent high-level semantic features.

[0101]

[0102]

[0103] wherein, respectively represent the cross-modal interaction features of the l-th layer visual modality v and the text modality t, and the sound modality a, C represents the dimension of the feature, and n represents the length of the feature.

[0104] The two kinds of interaction features are used to enhance the image features to increase the expression of visual information.

[0105]

[0106] wherein, represents the l-th layer enhanced visual feature. In general, the input of FMCA is multi-modal feature, and the relationship between the modalities is established by using MSA and MCA to enhance the visual modality module, and the formula is as follows:

[0107]

[0108] wherein, represents the sound feature after FMCA, represents the text feature after FMCA.

[0109] At the same time, in order to guide the short video e I to learn more advanced embedding features, a MLP capable of guiding feature learning (Guide MLP, GMLP) is proposed. The GMLP is used to replace the MLP in the Transformer to achieve the purpose of guiding learning of high-level features.

[0110]

[0111] wherein, the input of GMLP is the three modalities the short video content feature of the previous layer When in the first layer In this way, the short video content features are used to guide e I The learning of features. The output after transformation by GMLP is the enhanced modal feature of the first layer and the short visual feature after multi-modal content guided learning

[0112]

[0113]

[0114] wherein, W I represents a learning parameter; b represents a bias.

[0115] Then, the Transformer of the cross-modal multi-head multi-attention mechanism based on content guidance attention to a specific modal can be represented as:

[0116]

[0117] In GFCT, the proposed FMCA module can pay more attention to the features of the image modal, and use the internal relationship between the modes to enhance the expression of a modal feature (image modal feature). Meanwhile, the GMLP module can guide the learning of short video features using enhanced modal features.

[0118] Step 204: Use the user historical click video information features (such as video author ID, work city ID, music ID, etc.) to mine the user interest features, and guide the learning of user features. In this module, a time sequence interest extraction network (GTIN) is used:

[0119] s + = GTIN(S, e video , e u )

[0120] wherein, S = {s1, …s t , …, s T} user behavior sequence features, s t represents the user behavior feature at t time, T user behavior features, e video represents the short video ID feature, e u represents the user ID feature.

[0121] Specifically, first, the interest feature s t of each historical video is extracted, and the interest feature s τ of the current clicked video is extracted:

[0122] s t = match(e video，t , e u )

[0123] s τ = match(e video , e u )

[0124] where s t represents the user's interest features with the video built at time t, e video，t represents the embedding features of the video clicked by the user at time t, e u represents the embedding features of the user, e video represents the embedding features of the current video, s τ represents the user's interest features with the current video, and match is an attention matching mechanism.

[0125]

[0126]

[0127] where Similary(γ k , β) is the kth activation weight, γ k represents the kth feature, W k represents a learnable parameter, and β can be an interest feature or a user ID feature; match(γ, β) is the final matching feature.

[0128] Then, LSTM is used to extract the user's high-level temporal interest features:

[0129] [h1, …, h t , …, h T ] = LSTM([s1, s2, …, s T ])

[0130] where h t represents the historical interest built at time t, and s T represents the Tth user behavior feature.

[0131] Finally, the match network is used to model the historical interest, obtaining the user's final interest feature s + for the current video.

[0132] s + = match(h, s τ )

[0133] where h represents the hidden state of the historical interest feature built by LSTM.

[0134] Step 205: fuse the learned short video content features user interest features D and user features eu and video features e I Predict the probability of clicking the short video, and recommend the short video based on the predicted probability.

[0135]

[0136] wherein, is the final output click rate probability, and Sigmoid(·) is an activation function.

[0137] Embodiment 3

[0138] The feasibility of the schemes in embodiments 1 and 2 is verified below with specific examples, as described below:

[0139] The short video recommendation dataset from today's headlines is used as an implementation sample, which contains about 1962 million interaction information in total, including 70711 users, 3687157 short videos. Moreover, each short video contains its image, text, sound and some short video description information. The user behavior sequence information is constructed using these interaction information. The entire dataset is randomly divided into three subsets: 80% as the training set, 10% as the validation set, and 10% as the test set to divide the dataset, and P train , P valid and P test are used to represent them. P train , P valid are used as the training set and the validation set of the training process, respectively, and the method proposed in the application is used for training.

[0140] The detection performance of the embodiment of the application uses the average precision mean (AUC) as the evaluation index of the model. The calculation of AUC is as follows:

[0141]

[0142]

[0143] wherein, P is the number of positive samples, N is the number of negative samples, p pos is the positive sample prediction score, and p neg is the negative sample prediction score.

[0144] Embodiment 4

[0145] The short video recommendation device based on content and interest learning comprises a processor and a memory, the memory stores program instructions, and the processor calls the program instructions stored in the memory to make the device execute the following method steps:

[0146] Constructing a user behavior sequence, and discovering user interest features from the user behavior sequence;

[0147] The basic information of the short video is mapped to a low-dimensional feature vector by embedding layer to map high-dimensional one-hot encoding features to low-dimensional features for exploring short video features;

[0148] The cross-modal multi-head attention network focusing on specific modalities enhances the features of the image modality and guides the learning of short video features;

[0149] The time interest extraction network guides the learning of user interest features by guiding the learning of user features;

[0150] The short video features and user interest features learned by the above method are input into the click rate prediction network to predict the probability of user clicking the short video, and the short video is recommended based on the predicted probability.

[0151] The cross-modal multi-head attention network is as follows:

[0152] The cross-modal multi-head attention network is as follows:

[0153] The multi-modal features are mapped to the same dimension As input, x m is any kind of modality, d m is the feature dimension of modality m, m∈{v,a,t} respectively represents the image feature, sound feature and text feature of the short video;

[0154] Each modality is divided into n blocks Linear transformation is performed on these modality blocks to map them to the same dimension;

[0155] After transformation, the corresponding features are position encoded to preserve the spatial position information within each modality, and finally the input of GFCT is obtained

[0156]

[0157] wherein, The position encoding of modality m is is the i-th block feature vector of modality m, is the category label vector in VIT, c is the feature dimension of GFCT input, and FC represents the full connection layer.

[0158] The cross-modal multi-head attention network includes a multi-head self-attention module focusing on specific modalities, which learns the relationship between feature blocks within each modality from its own perspective:

[0159]

[0160] wherein, l = 1, 2, …, L represents the l-th layer of the FCT module, L represents an L-layer FCT module, the input of the first layer is represents the feature representation in the m-th modality output of the l-th layer, and MSA represents a multi-head self-attention mechanism;

[0161]

[0162]

[0163] wherein, respectively represent the cross-modal interaction features of the l-th layer of the visual modality v and the text modality t, and the sound modality a, C represents the dimension of the feature, and n represents the length of the feature;

[0164] The two kinds of interaction features are used for enhancing the image features to increase the expression of visual information;

[0165]

[0166] wherein, represents the enhanced visual feature of the l-th layer;

[0167]

[0168] wherein, represents the sound feature after the FMCA, represents the text feature after the FMCA.

[0169] wherein, the method further comprises: proposing a GMLP for guiding feature learning, the input of the GMLP is the three modalities the short video content feature of the previous layer

[0170]

[0171] wherein, GFCT(·) represents a cross-modal multi-head attention network capable of guiding short video feature learning while focusing on specific modalities, represents the output feature of the l-th layer of the modality m of the GFCT, represents the l-th layer of the content feature, represents the l-th layer of the short video content feature, represents the l-th layer of the text feature, represents the l-th layer of the sound feature, represents the enhanced video feature.

[0172] the:

[0173]

[0174]

[0175] wherein W I represents the learning parameter; and b represents the bias.

[0176] The time sequence interest extraction network for guiding user feature learning is:

[0177] s + = GTIN(S, e video , e u )

[0178] wherein S = {s1, …s t , …, s T} represents the user behavior sequence feature, GTIN(·) represents the time sequence interest extraction network for guiding user feature learning, s t represents the user behavior feature at t moment, e video represents the current video embedding feature, e u represents the user embedding feature, and s + is the user's interest feature for the current video.

[0179] The interest feature s t of each historical video and the interest feature s τ of the current clicked video are extracted:

[0180] s t = match(e video，t , e u )

[0181] s τ = match(e video , e u )

[0182] wherein e video，t represents the video embedding feature of the historical clicked video at t moment, s τ represents the interest feature constructed by the user and the current video, and match is an attention matching mechanism:

[0183]

[0184]

[0185] wherein Similary(γ k , β) is the kth activation weight, γ k represents the kth feature, W kThe β can be a user ID feature and a feature of interest; the match (γ, β) is a final matching feature, and the LSTM is used to extract a high-level time interest feature of the user:

[0186] [h1, …, h t , …, h T ] = LSTM ([s1, s2, …, s T ])

[0187] Wherein, h t represents a history interest corresponding to the t time, s T represents the Tth user behavior feature, the history interest is modeled by using the match network, and the final interest feature of the user for the current video is obtained.

[0188] The model of each device in the embodiments of the present application is not limited unless otherwise specified, and any device that can complete the above functions can be used.

[0189] Those skilled in the art can understand that the drawings are only schematic diagrams of a preferred embodiment, and the above-mentioned embodiment numbers are only for description, not representing the advantages and disadvantages of the embodiments.

[0190] The above-mentioned is only a preferred embodiment of the present application, and does not limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A short video recommendation method based on content and interest learning, characterized in that, The method comprises: constructing a user behavior sequence, and mining user interest features from the user behavior sequence; mapping high-dimensional one-hot encoding features of basic information of the short video to low-dimensional feature vectors through an embedding layer, to mine short video features; focusing on a specific modality of a cross-modal multi-head attention network to enhance image modal features, and guiding learning of short video features; guiding learning of user interest features through a time interest extraction network for learning user features; predicting a probability of a user clicking the short video through a click rate prediction network based on the learned short video features and user interest features, and recommending the short video based on the predicted probability; The cross-modal multi-head attention network is: used for learning intra-modal self-attention features and learning cross-modal attention features between modalities; and a nonlinear transformation layer is used to guide learning of short video features by inputting the short video features; extracting multi-modal features as input, is any of a plurality of modalities, is a feature dimension of modality m, respectively represent image features, sound features and text features of the short video. Split each modality into chunks , Apply linear transformation to these modality chunks, mapping to same dimensionality; After the transformation, the corresponding features are positionally encoded to preserve the spatial positional information within each modality, resulting in the input to GFCT : ； wherein, position encoding of modality m, is the modality The block feature vector, is the class label vector in VIT, c is the feature dimension of GFCT input, representing the fully connected layer; representing a cross-modal multi-head attention network capable of guiding short video feature learning while focusing on specific modalities; The cross-modal multi-head attention network comprises a multi-head self-attention module focusing on a specific modality, and the multi-head self-attention module learns a connection between feature blocks in each modality from a self perspective: ； wherein, represents the FCT module layer layer, represents a layer FCT module, the input of the first layer is , represents the layer output modal internal feature representation, represents a multi-head self-attention mechanism; ； ； wherein, , respectively represent the cross-modal interaction features of the first layer visual modality and the text modality , the sound modality , C represents the dimension of the features, and n represents the feature length; The two interaction features are used for enhancing image features to increase expression of visual information; ； wherein, represents the layer enhances visual features; ； wherein, represents a sound feature passed through the FMCA, represents a text feature passed through the FMCA; represents a multi-head self-attention module focusing on specific modalities; represents a multi-head attention mechanism across modalities; represents a layer normalization module. 2.The short video recommendation method based on content and interest learning according to claim 1, characterized in that, The method further comprises: proposing a GMLP for guided feature learning, the input of the GMLP is three modalities (X1, X2, X3) passing through the FMCA module , and an MLP in the Transformer is replaced by the GMLP; ； Wherein, the GMLP input is three modalities (audio, video, text) of the short video content , the short video content feature of the previous layer , when in the first layer , the output after the transformation of the GMLP is the first layer enhanced modality feature and the short video feature after multi-modal content guided learning : ； ； wherein, represents a learning parameter; represents a bias; Short video content features of the previous layer , ; wherein, represents a cross-modal multi-head attention network capable of guiding short video feature learning while focusing on specific modalities, represents the first layer modal output feature, represents the first layer content feature, represents the first layer short video content feature, represents the first layer text feature, represents the first layer sound feature, represents the enhanced video feature. 3.The short video recommendation method based on content and interest learning according to claim 1, characterized in that, The time interest extraction network for learning user features is: ； wherein, representing user behavior sequence features, representing a time interest extraction network guiding user feature learning, representing user behavior features at time t, representing the current video embedding feature, representing the user's embedding feature, is the user's interest feature for the current video.

4. The short video recommendation method based on content and interest learning according to claim 3, characterized in that, extracting an interest feature of each history video , and an interest feature of a current click video : ； ； wherein, represents the history clicked video embedding features at the moment, represents the interest features built by the user with the current video, and match is an attention matching mechanism: ； ； wherein, is the kth activation weight, represents the kth feature, represents the learnable parameter, may be an interest feature or a user ID feature; is the final matched feature, which extracts the user's high-level temporal interest feature using an LSTM: ； wherein, represents the historical interest corresponding to the constructed time t, represents the Tth user behavior feature, and the historical interest is modeled by using the match network to obtain the final interest feature of the user for the current video.

5. A short video recommendation device based on content and interest learning, characterized in that, The device comprises a processor and a memory, and the memory stores program instructions; the processor invokes the program instructions stored in the memory to enable the device to perform the method of any one of claims 1-4.

Images