Heterogeneous graph neural network-based short video recommendation method and system fused with multi-modal data
Through the heterogeneous graph neural network, a multi-behavior graph perception network and graph comparison learning module are built, which solves the problem of existing short video recommendation algorithms ignoring user interaction behavior and modal association, and achieves the latest update Accurate and personalized recommendations.
Patent Information
- Application Number
- CN202411211539.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-30
AI Technical Summary
Existing short video recommendation algorithms ignore the association between potential interest characteristics and different modal data in multiple interaction behaviors of users, resulting in limitations of recommendation results.
Heterogeneous graph neural network is used to fuse multimodal data and user interaction behavior, and the embedded representation of short video nodes is extracted through graph attention network and GRU encoder, and combined with time context information, a multi-behavior graph perception network and graph comparison learning module are constructed. Capture complex relationships and collaborative signals between users and short videos.
It realizes more accurate and personalized short video recommendations, improves user stay time, successfully explores the user's potential interest characteristics, and alleviates the problem of sparseness of interactive behavior.
Smart Images

Figure CN119089004A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a short video recommendation method and system based on heterogeneous graph neural networks by integrating multimodal data. Background Art
[0002] Initially, the research on short video recommendation focused on traditional recommendation system methods, such as collaborative filtering and content filtering. In the field of short video recommendation, the complex relationship between users and short videos was ignored, and it was difficult to reveal the correlation between potential user interests and short videos. With the rise of graph neural networks, researchers began to introduce them into short video recommendation. This new idea better models the relationship network between users and short videos. Graph neural networks capture the complex relationships and potential patterns in the graph by learning the representation of nodes and edges, thereby improving the accuracy and personalization of short video recommendations. In recent years, with the rapid development of deep learning technology, graph neural networks have been widely used in the field of short video recommendation. Graph neural network models trained by deep learning can better learn the representation of users and short videos. At present, scholars are committed to further exploring the application of graph neural networks in the field of short video recommendation, including modifying the model structure, introducing attention mechanisms, and other methods to improve model performance and efficiency. At the same time, they also combined short video recommendations with other fields. For example, in the feature representation of short videos, they integrated multi-modal data to fully express the potential features therein; in terms of user feature representation, they analyzed users' various interactive behaviors (such as sliding, liking, and collecting) to explore users' deeper preferences and achieve more accurate and personalized recommendations.
[0003] In summary, the technology of using heterogeneous graph neural networks to analyze user interest preferences has gone through the process of evolving from traditional recommendation systems to the introduction of graph neural networks, and combined with the development of deep learning, multimodality and other technologies, it provides a more accurate and personalized solution for short video recommendations. In the future, this technology is expected to further promote the development of the short video recommendation system field.
[0004] At present, short video recommendation algorithms are mainly based on Graph Neural Networks (GNN) to model the complex interactive relationship between users and videos, and learn the high-order connections between user-item nodes to enrich node representations. Although some studies have tried to introduce multimodal information (such as text, images, audio, etc.) to improve recommendation effects, most methods still use a simple multimodal feature addition method and lack in-depth mining of the associations between different modalities. At the same time, existing short video recommendation algorithms often ignore the potential interest features in users' various interactive behaviors (such as browsing, liking, and collecting), resulting in limitations in recommendation results. Summary of the invention
[0005] In view of the above problems, the present invention proposes a short video recommendation method and system based on heterogeneous graph neural network by integrating multimodal data. By integrating the multimodal features of short videos and mining the potential interest features in multiple user interaction behaviors and time context, the heterogeneous graph neural network is finally used to analyze user preferences. The present invention can provide users with short video content of interest more accurately, thereby allowing users to stay longer.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] On the one hand, the present invention proposes a short video recommendation method based on heterogeneous graph neural network by integrating multimodal data, comprising:
[0008] Step 1: Collect multiple user interaction behavior data on the short video platform, and collect multimodal data at the same time; and decompose the obtained user interaction behaviors into conversation sequences; the user interaction behaviors include watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data;
[0009] Step 2: Construct a heterogeneous graph G based on the session sequence n =(V,S,U,E v ,E), where V represents the short video node set, S represents the session node set, U represents the user node set, E v represents the set of transformation relations between short videos, E represents the set of relations between short videos and sessions, short videos and users, and users and sessions; constructing a multi-behavior heterogeneous graph in represents the user node set, represents the short video node set, ε represents and The set of interactive behaviors between them; and divide the multi-behavior heterogeneous graph into multiple behavioral subgraphs according to the interactive behavior type;
[0010] Step 3: Use the graph attention network to analyze the heterogeneous graph G. n The heterogeneous neighbors of short and medium video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes.
[0011] Step 4: Use the initial embedding representation of the short video node obtained by GRU aggregation, and combine it with the attention mechanism to obtain the final embedding representation of the short video node, and then obtain the initial session embedding representation, introduce the temporal context information, and combine it with the soft attention mechanism to obtain the final session embedding representation;
[0012] Step 5: construct a multi-behavior graph perception network based on GNN to capture the collaborative signals between nodes and behaviors; the multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; input multiple behavior subgraphs into the constructed multi-behavior graph perception network to obtain user representations after adding behavior embedding;
[0013] Step 6: Enhance the user representation obtained after adding behavior embedding through graph contrast learning combined with the constructed contrast loss function;
[0014] Step 7: Concatenate the final session embedding representation and the enhanced user representation, input the concatenated features into the factorization machine to obtain the corresponding predicted score, and recommend short videos to the user based on the predicted score.
[0015] Furthermore, in step 3, all types and the same number of heterogeneous neighbors are selected for each short video node in the following manner:
[0016] For all short video nodes, random walk is started from each short video node. During the random walk, it moves to the neighbors of the current node iteratively with dynamic probability according to the attention weight, controls the number of walks of each type of node, ensures that all types of nodes are obtained during the random walk, and stores all the wandering nodes in a list;
[0017] All nodes in the list are classified and the same number of nodes are selected for each modality as heterogeneous neighbors of the short video node.
[0018] Furthermore, the initial embedding of the short video node is expressed as:
[0019]
[0020] in is the initial embedding representation of the short video node v, and d is the dimension of the node embedding; is the linear transformation layer; is a concatenation operation; x is a collection of multiple modes, a t is the attention weight of modality t, and Represents a bidirectional GRU.
[0021] Furthermore, in step 4, the initial embedding representation of the short video node is obtained by aggregation in the following manner:
[0022]
[0023] Among them, f t (v) is the embedding representation of the short video node v under node type t, N t(v) represents the t-type neighborhood set of the short video node v, and f(v) is the initial embedding representation of the short video node v.
[0024] Furthermore, in step 4, the final embedding representation of the short video node is obtained in the following manner:
[0025]
[0026] In the formula
[0027]
[0028] Among them, ε v represents the final embedding representation of the short video node v, a v,t represents the importance of node type t to the short video node v, U is the attention parameter, and f t represents the importance of node type t to node v, f j is the node embedding representation.
[0029] Furthermore, in step 4, the final session embedding representation is obtained in the following manner:
[0030]
[0031] In the formula
[0032] s l =v s,n
[0033]
[0034] Among them, s h represents the final session embedding representation, s l is the local embedding of the last session, s g is the global session embedding, a v is the attention weight of different embedding representations, ν s,n is the embedding vector of the last short video in the current session, W, W1, W2 are the weights used to generate session embedding, σ represents the sigmoid function, and c is a bias term.
[0035] Furthermore, the behavior-specific embedded propagation layer performs message delivery in the following manner:
[0036]
[0037] in, represents the embedding of user node u under the behavior k of the i+1th embedding propagation layer, LeakyReLU is the optimized version of the rectified linear unit, mean is the mean function, is the set of neighbors of u under behavior k, W(i) is a layer-specific parameter, ⊙ represents the element-wise product of two vectors, represents the duration factor embedding of behavior k in the i-th embedding propagation layer, represents the embedding of all user nodes under row k in layer i, represents the embedding of all user nodes under behavior k in the i+1th embedding propagation layer, are layer-specific parameters.
[0038] Furthermore, in the cross-behavior dependency modeling module, a self-attention mechanism is used to connect the embeddings of the user node u under all behaviors, and then the coefficient of the dependency relationship between the user u under behavior k and other behaviors is calculated, thereby obtaining the enhanced embedding of the node u under behavior k, and then the mean pooling is used to integrate the embeddings of all layers to obtain the user representation after adding the behavior embedding.
[0039] Furthermore, in step 6, the contrast loss function is:
[0040]
[0041] In the formula
[0042]
[0043] in, is the contrast loss function between behaviors, It is behavior o and k p The contrast loss between is the behavior k that eliminates missed reports of specific user nodes o and k p The contrast loss between is the behavior k that eliminates the omission of specific short video nodes o and k p The contrast loss between them, FN(u) represents the underreporting of user node u, They are user node u in behavior k o and k p The embedding representation below is is the user node v in behavior k p The embedding representation below, FN(v2) represents the underreporting of short video node v2, They are short video node v1 in behavior k o and k p The embedding representation below is It is the short video node v2 in behavior k p The embedding representation below is, τ is the temperature hyperparameter in Softmax, and φ(·) represents the inner product of two vectors.
[0044] On the other hand, the present invention proposes a short video recommendation system based on heterogeneous graph neural network integrating multimodal data, comprising:
[0045] A data acquisition unit is used to collect multiple user interaction behavior data on the short video platform, and collect multimodal data at the same time; and decompose the obtained user interaction behavior into a conversation sequence; the user interaction behavior includes watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data;
[0046] Heterogeneous graph construction unit, used to construct a heterogeneous graph G according to the session sequence n =(V,S,U,E v ,E), where V represents the short video node set, S represents the session node set, U represents the user node set, E v represents the set of transformation relations between short videos, E represents the set of relations between short videos and sessions, short videos and users, and users and sessions; constructing a multi-behavior heterogeneous graph in represents the user node set, represents the short video node set, ε represents and The set of interactive behaviors between them; and divide the multi-behavior heterogeneous graph into multiple behavioral subgraphs according to the interactive behavior type;
[0047] Multimodal feature extraction and fusion unit for heterogeneous graph G through graph attention network n The heterogeneous neighbors of short and medium video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes.
[0048] The first embedding representation extraction unit is used to obtain the initial embedding representation of the short video node using GRU aggregation, and combines it with the attention mechanism to obtain the final embedding representation of the short video node, and then obtain the initial session embedding representation, introduce the temporal context information, and combine it with the soft attention mechanism to obtain the final session embedding representation;
[0049] A second embedding representation extraction unit is used to construct a multi-behavior graph perception network based on GNN to capture the synergy signal between nodes and behaviors; the multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; multiple behavior subgraphs are input into the constructed multi-behavior graph perception network to obtain a user representation after adding the behavior embedding;
[0050] A graph contrast learning unit is used to enhance the user representation obtained after adding behavior embedding through graph contrast learning combined with the constructed contrast loss function;
[0051] The short video recommendation unit is used to splice the final session embedding representation and the enhanced user representation, input the spliced features into the factor decomposition machine, obtain the corresponding predicted score, and recommend short videos to users according to the predicted score.
[0052] Compared with the prior art, the present invention has the following beneficial effects:
[0053] The present invention fully considers the multiple modal features of short videos and the dependencies between different modalities, and uses heterogeneous graph neural networks to capture short video features with finer granularity; secondly, by introducing multiple user interaction behaviors and contextual information, the detailed features of users are captured through graph perception networks, and graph comparative learning is used to alleviate the problem of sparse interactions. The potential interests of users are successfully mined, so that short videos can be recommended to users more accurately and personalized. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 This is a flow chart of a short video recommendation method based on heterogeneous graph neural network integrating multimodal data according to an embodiment of the present invention;
[0055] Figure 2 A schematic diagram of a heterogeneous isomorphic graph provided in an embodiment of the present invention;
[0056] Figure 3 A schematic diagram of a multi-behavior heterogeneous graph provided by an embodiment of the present invention;
[0057] Figure 4 A schematic diagram of a multi-behavior graph perception network structure provided by an embodiment of the present invention;
[0058] Figure 5 A schematic diagram of the structure of a multi-behavior graph comparison module provided in an embodiment of the present invention;
[0059] Figure 6 This is a schematic diagram of the architecture of a short video recommendation system based on a heterogeneous graph neural network that integrates multimodal data according to an embodiment of the present invention. DETAILED DESCRIPTION
[0060] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:
[0061] like Figure 1 As shown, a short video recommendation method based on heterogeneous graph neural network integrating multimodal data includes:
[0062] S101: Collecting multiple user interaction behavior data on the short video platform and collecting multimodal data at the same time; and decomposing the obtained user interaction behaviors into conversation sequences; the user interaction behaviors include watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data;
[0063] S102: Construct a heterogeneous graph G based on the session sequence n =(V,S,U,E v ,E), where V represents the short video node set, S represents the session node set, U represents the user node set, E v represents the set of transformation relations between short videos, E represents the set of relations between short videos and sessions, short videos and users, and users and sessions; constructing a multi-behavior heterogeneous graph in represents the user node set, represents the short video node set, ε represents and The set of interactive behaviors between them; and divide the multi-behavior heterogeneous graph into multiple behavioral subgraphs according to the interactive behavior type;
[0064] S103: Using graph attention network to analyze heterogeneous graph G n The heterogeneous neighbors of short and medium video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes.
[0065] S104: Use the initial embedding representation of the short video node obtained by GRU aggregation, combine it with the attention mechanism, obtain the final embedding representation of the short video node, and then obtain the initial session embedding representation, introduce the temporal context information, and combine it with the soft attention mechanism to obtain the final session embedding representation;
[0066] S105: constructing a multi-behavior graph perception network based on GNN to capture the collaborative signals between nodes and behaviors; the multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; inputting multiple behavior subgraphs into the constructed multi-behavior graph perception network to obtain a user representation after adding the behavior embedding;
[0067] S106: Enhance the user representation obtained after adding behavior embedding through graph contrast learning combined with the constructed contrast loss function;
[0068] S107: Concatenate the final session embedding representation and the enhanced user representation, input the concatenated features into a factorization machine to obtain a corresponding predicted score, and recommend short videos to the user according to the predicted score.
[0069] The method specifically includes:
[0070] 1. Data Preprocessing
[0071] The present invention collects a variety of user interaction behavior data on the short video platform, including viewing, liking, and collection behaviors, and also collects multimodal data, such as short video content data, audio data, and text data. In the data preprocessing stage, operations such as denoising and missing value processing are performed to ensure the accuracy and completeness of the data. Secondly, the obtained interaction behavior V is decomposed into a conversation sequence S.
[0072] 2. Construction of heterogeneous and multi-behavior heterogeneous graphs
[0073] 2.1. Heterogeneous Heterogeneous Graphs
[0074] First, the session sequence S needs to be constructed as a heterogeneous graph G n =(V,S,U,E v ,E),G n It contains three types of nodes: short video nodes, session nodes, and user nodes. V, S, and U are the corresponding sets of the three, and two types of edges: directed edges and undirected edges between two videos. v and E are the corresponding sets of the two. Figure 2 As shown. The user node set U includes: user ID, etc. The short video node set V includes: features of short videos, mainly multimodal features: 1) short video content (image frame data), 2) audio content (background music of short videos, dialogues of characters, etc.), 3) text content (description of short videos, short video tags, etc.). The session node set S includes: 1) Session ID: Each session node has a unique identifier to distinguish different sessions. 2) Context information: such as timestamp, geographic location, etc. E υ =(v s,i ,v i+1 ) indicates that the user is watching video v s,i After watching the next video v s,i+1 . E v It can represent the complex transformation relationship set between short videos. s,i ,s),(v s,i ,u),(s,u)}, respectively, representing the relationship between short video and session, short video and user, and user and session.
[0075] 2.2. Multi-behavior heterogeneous graph
[0076] The present invention represents the user node set and the short video node set as and in and Represent the number of users and the number of short videos respectively. Bipartite graph based on user-short video interaction like Figure 3 As shown in Figure 1, it contains two types of nodes: short video nodes and user nodes, as well as multiple types of edges. The edge set ε represents and In a multi-behavior scenario with k types of interactive behaviors, the entire bipartite graph According to the different types of interactive behaviors, it can be divided into k behavioral subgraphs In order to reflect the multi-behavior interaction data, a tensor is defined here If the individual elements If it approaches 1, it means that user u interacts with short video v under behavior k, otherwise it approaches 0.
[0077] 3. Multiple modal extraction and fusion module
[0078] Create G n Finally, the present invention uses a heterogeneous graph neural network that integrates multimodal information to learn item embedding, i.e., short video embedding, which has rich information and complex transformation relationships.
[0079] Firstly, a graph attention network is introduced to dynamically sample the heterogeneous neighbors of a node, selecting all modality types and the same number of heterogeneous neighbors for each node. Then, a heterogeneous content encoder is designed to aggregate the heterogeneous neighbor content of a node, fuse different modality features, and obtain the feature representation of the node.
[0080] 3.1 Heterogeneous Neighborhood Sampling
[0081] In Figure G n In the dataset, the types and numbers of these heterogeneous neighbor nodes are different, and aggregating these nodes may require different feature transformations. For example, one user watched three short videos in two sessions, and another user watched ten short videos in three sessions. Here, the sizes of the neighbor nodes of the two users are different. Selecting nodes as heterogeneous neighbors of users so that they can use the same model to aggregate these heterogeneous neighbors is an important issue.
[0082] To solve this problem, classic topological feature extraction algorithms include DeepWalk and Node2vec, but neither of these two methods considers sampling neighboring nodes based on high-order weights and node attributes, so we designed a heterogeneous neighbor sampling strategy (SWAD) based on high-order weights and node attributes. This method walks according to the weights to collect all types of neighbors for each node; fixes the sampling neighbor size of each node and selects the most frequently visited neighbors; finally, groups neighbors of the same type (with the same content features) so that type-based aggregation can be deployed.
[0083] Taking the short video node as an example, the main steps of SWAD are as follows:
[0084] S201: For all short video nodes V = {v1, v2, v3, ..., v n}, from each short video node v i Start random walk. During the random walk, it iteratively moves to the neighbors of the current node with a dynamic probability p according to the attention weight. SWAD controls the number of nodes of each type to walk, ensuring that all types of nodes are obtained during random walk, and all wandering nodes are stored in a list, namely SWAD(v);
[0085] S202: Classify all nodes in SWAD(v) and select top-k nodes for each mode t t Nodes as short video nodes v i As a feasible implementation method, in this embodiment, the same number of nodes are selected as short video nodes v for each modality t. i Heterogeneous neighbors.
[0086] 3.2 Heterogeneous Content Encoders
[0087] Heterogeneous neighbors of different modality types have different node contents. For example, a short video node contains image frames, audio, text, and other modalities. To this end, we design a GRU-based heterogeneous content encoder to capture the interaction between features, encode the different modalities of the node into an embedded representation of a fixed dimension, and aggregate all modal features of the node into the embedded representation of the node, giving it greater representation capabilities.
[0088] First, node v is node v i We represent the feature of the t-th modality in node v as (d f : feature dimension). Note that x t Different techniques can be used to pre-train different modal data. For content of different modalities, the present invention uses different methods to convert the content of different modalities into embedding vectors of the same dimension. Specifically, for image and text data, we directly use the currently best BEIT-3 (Bert Pre-Training of Image Transformers-3) to capture the features of different modalities, and for audio data, VGGish is used to extract audio features. These methods are all based on deep learning methods and can automatically extract representative features from video, audio, and text.
[0089] Then, different from previous models that linearly transform different modality features into a unified vector, we use GRU to capture deep feature interactions to obtain more powerful expression capabilities.
[0090] More specifically, different linear transformation layers are first used to transform different content features, and then GRU is used to capture the "deep" feature interaction and accumulate the expressive power of all content features. Considering the different importance of different modal information, the present invention adds image features, audio features and text features to the attention weights for fusion, and obtains a richer and more complex feature representation:
[0091] After obtaining the embedding vectors of each node attribute, the node embedding f(v) can be expressed as:
[0092]
[0093] in d is the dimension of node embedding; is the linear transformation layer; is a concatenation operation; x is a collection of multiple modes, a t are the attention weights of different modalities. and Represents a bidirectional GRU.
[0094] The formula for GRU is as follows:
[0095]
[0096]
[0097] Among them, x t Indicates the input information at the current moment; h t-1 Represents the hidden state of the previous moment. The hidden state acts as the neural network memory, which contains information about the data that the node has seen before; t represents the hidden state passed to the next moment; h represents the candidate hidden state; r t , Represents the reset gate and update gate of the t-th modal feature; σ represents the sigmoid function, through which the data can be converted to a value in the range of 0-1; tanh represents the tanh function, through which the data can be converted to a value in the range of [-1,1].
[0098] The heterogeneous content encoder described above has a simple structure and relatively low complexity (fewer parameters). It can aggregate heterogeneous content and make node embedding more expressive. In addition, it can flexibly add additional modal features, making the model easy to expand.
[0099] 4 Heterogeneous Graph Neural Network Layers
[0100] After aggregating the heterogeneous node contents of the neighbors, an embedding representation of the heterogeneous neighbors is generated. Each node has multiple types of heterogeneous neighbors. GRU can aggregate on an unordered set of neighbors. For this purpose, we design a neural network that uses GRU to aggregate nodes of the same type and learn the complex relationships between them, so that the learned type embedding f t (v) It has stronger expressive power. GRU can solve the problems of RNN’s inability to retain long-term memory and gradient in back propagation. It has a similar function to LSTM, but is simpler and easier to train than LSTM. t The calculation formula of (v) is as follows:
[0101]
[0102] The GRU model is the same as (2), except for the input and parameter settings. t (v) represents the t-type neighborhood set of the short video node v∈V.
[0103] After obtaining the type embedding of the short video node v, it is necessary to aggregate all the type embeddings into a vector, which is the final embedding of node v. However, different types of heterogeneous neighbors have different effects on node v, so the attention mechanism is introduced. The calculation formula for the importance of different types to node v is as follows:
[0104]
[0105] where f t represents the importance of node type t to node v, f j is the node embedding representation.
[0106] Where LeakyReLU is an optimized version of the rectified linear unit, is the attention parameter. After calculating the importance of each type to node v, the embedding of node v is calculated as follows:
[0107]
[0108] The resulting project node is finally embedded in ε v It not only contains the transition items between each other, but also learns the information of other types of nodes, making it more expressive.
[0109] After obtaining the item embedding representation, generate the session embedding. For session s=v s,1 ,v s,2 ,v s,3 ,...,v s,i , the pre-embedded vector is calculated as follows:
[0110]
[0111] where ν s,i is node v s,i Node embedding of ; ν0 is a zero vector with no fixed dimension; is a concatenation operation. Since the number of short videos in each session may vary, v0 needs to be added so that all embedded session vectors have the same spatial dimension.
[0112] In practical applications, short video platforms have extremely high real-time requirements for recommendation systems. Users' interests are not static. They may change over time and with the accumulation of new experiences. This is a dynamic process, which makes it difficult for existing recommendation models to accurately capture and adapt to users' latest interest status. Existing short video recommendations cannot balance users' long-term and dynamic preferences at the same time. It is also extremely important to capture users' dynamic preferences and provide users with accurate and timely recommendations.
[0113] Therefore, an attention mechanism module is added to the model to obtain a hybrid session embedding that can express long-term and dynamic preferences. In this paper, the local embedding s of the last session is first considered l , the formula is as follows:
[0114] s l =v s,n (7)
[0115] where ν s,n is the embedding vector of the last short video in the current session.
[0116] Aiming at the user's long-term interest preferences and taking into account the transformation relationship between all videos, the present invention adopts a soft attention mechanism to learn the global session embedding s g . Global Session Embeds g The formula is as follows:
[0117]
[0118] The matrix is the weight used to generate the session embedding, and c is a bias term. After obtaining the global and local embedding representations of the session, the final session embedding can be expressed as:
[0119]
[0120] where a v is the attention weight of different embedding representations. Finally, it is converted into session embedding representation through linear transformation. The linear transformation can be a fully connected layer that maps the high-dimensional features of the input to a low-dimensional space. By learning session embedding, the overall information and semantics of the session can be captured.
[0121] 5. Multi-behavior graph perception network
[0122] The present invention constructs a multi-behavior graph-aware network based on GNN to capture complex collaborative signals between nodes and behaviors. Specifically, a behavior-specific embedding propagation method is first used to obtain contextual node embeddings under each behavior subgraph. Then, these embeddings are enhanced by personalized behavioral correlations through cross-behavior dependency modeling. In order to better capture the relationship between nodes and behaviors, the present invention also introduces a new method to enhance the representation ability of node embeddings by analyzing the relationships between behaviors. This method enables the multi-behavior graph-aware network to more comprehensively understand the complex interactions between nodes and behaviors, thereby more accurately capturing collaborative signals. In this way, the model of the present invention has a stronger expressive ability in capturing the correlation between nodes and behaviors, thereby providing more accurate and reliable prediction capabilities for recommendation systems, such as Figure 4 As shown (where Sv represents the user representation introduced into the user behavior embedding).
[0123] 5.1. Behavior-specific Embedding Propagation Layer
[0124] In the multi-behavior scenario, first construct each bipartite behavior subgraph based on k types of interactive behaviors Then, embedding propagation is performed on each subgraph to obtain the representation of each node under each behavior. In order to more accurately reflect the differences between each behavior and capture the semantic characteristics of actual user preferences, the duration of users in different behaviors is different. For browsing behavior, we calculate its playback completion rate, and for behaviors such as liking and collecting, we calculate the duration of each behavior, and then use the attention mechanism to obtain the final behavior duration factor embedding. The present invention also incorporates the embedding of nodes under specific behaviors and the representation of each behavior into the message passing paradigm:
[0125]
[0126] in, represents the embedding of user node u under the behavior k of the i+1th embedding propagation layer, LeakyReLU is the optimized version of the rectified linear unit, mean is the mean function, is the set of neighbors of u under behavior type k, W (i) is layer-specific, ⊙ represents the element-wise product of two vectors, represents the duration factor embedding of behavior k in layer i, represents the embedding of all user nodes under row k in layer i, by multiplying by another layer-specific parameter To update:
[0127]
[0128] 5.2. Cross-behavior dependency modeling module
[0129] Considering that there are implicit correlations between different behaviors based on users, and this correlation varies from user to user, in order to effectively model the dependencies across behaviors, the present invention adopts a self-attention mechanism.
[0130] Specifically, the embeddings of node u under all behaviors are concatenated as Then calculate The coefficient representing the dependency between user u and other behaviors under behavior k
[0131]
[0132] in are two behavior-specific parameters, and d′ is the output dimension size. Therefore, the enhanced embedding of node u under behavior k can be easily calculated as:
[0133] e u,k =a u,k e u (13)
[0134] Since the embeddings of different layers represent different connections, the present invention uses mean pooling to integrate the embeddings of all layers as follows:
[0135]
[0136] 6. Multi-behavior graph comparison module
[0137] In actual scenarios, multi-behavior interaction data usually exhibits sparsity, which also exists in short video recommendation scenarios. At the same time, ignoring contextual information may cause the model to have an inaccurate understanding of user interests and preferences. For example, ignoring the user's interaction time may weaken the recommendation effect. In order to solve the problem of data sparsity and improve the generalization ability of the model, it is necessary to avoid over-reliance on a single interaction signal and comprehensively consider multiple different types of behavioral interactions. In addition, it is noted that sparse supervisory signals may lead to deviations in learning representations, so appropriate methods need to be adopted to extract more meaningful feature representations. Therefore, the present invention constructs a new contrastive learning paradigm, which helps to alleviate the impact of the sparsity of supervisory signals of multiple interactive behaviors and can more effectively handle the interference caused by multiple behaviors. Taking into account that the supervisory signals of multiple behaviors are richer than those in a single interactive behavior, the present invention adopts a selective contrastive learning method to transfer knowledge between multiple interactive behaviors. In this way, the problem of data sparsity can be alleviated. As Figure 5 As shown (where Sv' represents the final user representation obtained after graph contrast learning).
[0138] Specifically, the present invention selects k={1, 2, ..., K-1}, k o ≠k p , k o ,k p ∈k, for behavior k o ,k p The common practice is to treat the views of the same node as positive pairs. Treat views of different nodes as negative pairs
[0140] However, in such a recommendation setting involving various interactive behaviors, there are usually certain commonalities between the same objects, such as users having similar preferences or items having similar attributes. However, constructing negative pairs according to traditional methods may introduce a large number of missed misses, i.e., highly similar nodes, which will lead to the loss of true semantic information. Therefore, to solve this problem, the present invention adopts the Swing algorithm to identify potential missed misses by calculating similarity scores and screen them out when comparing node pairs to eliminate the impact of these missed misses. In this way, relevant semantic information can be discovered and utilized more accurately, thereby improving the performance and effectiveness of the recommendation system. More specifically, the subgraph The similarity score of two users u and υ in is calculated using the Swing algorithm as follows:
[0141]
[0142] Among them, α is the smoothing coefficient, is the set of neighbors of u under behavior type k. The final similarity score S(u,v) is the sum of each subgraph The average of the scores.
[0143] Then, here we define the missed detection of user node u as the users with the top n similarity scores, FN(u) = {v|S(u,v)∈top(S(u),n)}. Therefore, the behavior k that eliminates the specific missed detection of user node u is o and k p The contrast loss between is defined as:
[0144]
[0145] in, is the user node u in behavior k o The embedding representation below is, τ is the temperature hyperparameter in Softmax, and φ(·) represents the inner product of two vectors.
[0146] Similarly, we can eliminate the specific missed behavior k of the short video node v o and kp The contrast loss between is defined as:
[0147]
[0148] in, is the short video node v1 in behavior k o The embedding representation below is, τ is the temperature hyperparameter in Softmax, and φ(·) represents the inner product of two vectors. FN(v2) represents the missed detection of short video node v2, specifically the short video nodes with the top n similarity scores.
[0149] By analogy, when comparing losses with the project party, behavior k o and k p The contrast loss between them is:
[0150] Therefore, the final contrastive loss between behaviors is o ,k p Compare the losses and half of them:
[0151] 7. Predict user interests
[0152] The final representation of splicing users and short videos is [e k ,s h ], and pass this representation into the factorization machine:
[0153]
[0154] in represents the prediction score of the model, ω u ,ω v are the weights of user embedding representation and short video embedding representation, respectively. <v u ,v v > represents v u and v v The dot product of ω0, b u and b v They are global bias, user bias and project bias.
[0155] Based on the above embodiments, Figure 6 As shown, the present invention also proposes a short video recommendation system based on heterogeneous graph neural network integrating multimodal data, including:
[0156] A data acquisition unit is used to collect multiple user interaction behavior data on the short video platform, and collect multimodal data at the same time; and decompose the obtained user interaction behavior into a conversation sequence; the user interaction behavior includes watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data;
[0157] Heterogeneous graph construction unit, used to construct a heterogeneous graph G according to the session sequence n =(V,S,U,E v ,E), where V represents the short video node set, S represents the session node set, U represents the user node set, E v represents the set of transformation relations between short videos, E represents the set of relations between short videos and sessions, short videos and users, and users and sessions; constructing a multi-behavior heterogeneous graph in represents the user node set, represents the short video node set, ε represents and The set of interactive behaviors between them; and divide the multi-behavior heterogeneous graph into multiple behavioral subgraphs according to the interactive behavior type;
[0158] Multimodal feature extraction and fusion unit for heterogeneous graph G through graph attention network n The heterogeneous neighbors of short and medium video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes.
[0159] The first embedding representation extraction unit is used to obtain the initial embedding representation of the short video node using GRU aggregation, and combines it with the attention mechanism to obtain the final embedding representation of the short video node, and then obtain the initial session embedding representation, introduce the temporal context information, and combine it with the soft attention mechanism to obtain the final session embedding representation;
[0160] A second embedding representation extraction unit is used to construct a multi-behavior graph perception network based on GNN to capture the synergy signal between nodes and behaviors; the multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; multiple behavior subgraphs are input into the constructed multi-behavior graph perception network to obtain a user representation after adding the behavior embedding;
[0161] A graph contrast learning unit is used to enhance the user representation obtained after adding behavior embedding through graph contrast learning combined with the constructed contrast loss function;
[0162] The short video recommendation unit is used to splice the final session embedding representation and the enhanced user representation, input the spliced features into the factor decomposition machine, obtain the corresponding predicted score, and recommend short videos to users according to the predicted score.
[0163] In summary, the current short video recommendation method does not fully consider the rich multimodal information such as text, images, audio, etc. in the video and still adopts a simple multimodal feature addition method, lacking in-depth mining of the relationship between different modalities. At the same time, it also ignores the potential interest features in the user's various interactive behaviors (such as browsing, liking, collecting, etc.), resulting in limitations in the recommendation results. The present invention constructs a heterogeneous content encoder containing an attention mechanism to fuse the features of different modalities, and uses a heterogeneous graph neural network to mine the potential features of short videos. At the same time, temporal context information is introduced, and a graph-aware network is used to mine the potential features in various user behaviors. Then, graph comparison is used to reduce the impact of the sparsity of supervision signals of various interactive behaviors, and finally provide users with accurate short video recommendations.
[0164] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.
Claims
1. A short video recommendation method based on heterogeneous graph neural network integrating multimodal data, characterized in that: include: Step 1: Collect various user interaction behavior data on the short video platform and collect multimodal data at the same time; The obtained user interaction behaviors are decomposed into conversation sequences; the user interaction behaviors include watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data; Step 2: Construct a heterogeneous graph G based on the session sequence n =(V,S,U,E v ,E), where V represents the short video node set, S represents the session node set, U represents the user node set, E v represents the set of transformation relations between short videos, E represents the set of relations between short videos and sessions, short videos and users, and users and sessions; constructing a multi-behavior heterogeneous graph in represents the user node set, v represents the short video node set, and ε represents and v; and divide the multi-behavior heterogeneous graph into multiple behavior subgraphs according to the interaction behavior type; Step 3: Use the graph attention network to analyze the heterogeneous graph G. n The heterogeneous neighbors of short and medium video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes. Step 4: Use the initial embedding representation of the short video node obtained by GRU aggregation, and combine it with the attention mechanism to obtain the final embedding representation of the short video node, and then obtain the initial session embedding representation, introduce the temporal context information, and combine it with the soft attention mechanism to obtain the final session embedding representation; Step 5: construct a multi-behavior graph perception network based on GNN to capture the collaborative signals between nodes and behaviors; the multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; input multiple behavior subgraphs into the constructed multi-behavior graph perception network to obtain user representations after adding behavior embedding; Step 6: Enhance the user representation obtained after adding behavior embedding through graph contrast learning combined with the constructed contrast loss function; Step 7: Concatenate the final session embedding representation and the enhanced user representation, input the concatenated features into the factorization machine to obtain the corresponding predicted score, and recommend short videos to the user based on the predicted score.
2. According to claim 1, the short video recommendation method based on heterogeneous graph neural network integrating multimodal data is characterized in that: In step 3, all types and the same number of heterogeneous neighbors are selected for each short video node in the following manner: For all short video nodes, random walk is started from each short video node. During the random walk, it moves to the neighbors of the current node iteratively with dynamic probability according to the attention weight, controls the number of walks of each type of node, ensures that all types of nodes are obtained during the random walk, and stores all the wandering nodes in a list; All nodes in the list are classified and the same number of nodes are selected for each modality as heterogeneous neighbors of the short video node.
3. The short video recommendation method based on heterogeneous graph neural network based on fusion multimodal data according to claim 1 is characterized in that: The initial embedding of the short video node is expressed as: in is the initial embedding representation of the short video node v, and d is the dimension of the node embedding; is the linear transformation layer; is a concatenation operation; x is a collection of multiple modes, a t is the attention weight of modality t, and Represents a bidirectional GRU.
4. The short video recommendation method based on heterogeneous graph neural network by integrating multimodal data according to claim 1 is characterized in that: In step 4, the initial embedding representation of the short video node is obtained by aggregation in the following manner: Among them, f t (v) is the embedding representation of the short video node v under node type t, N t (v) represents the t-type neighborhood set of the short video node v, and f(v) is the initial embedding representation of the short video node v.
5. The short video recommendation method based on heterogeneous graph neural network based on fusion multimodal data according to claim 4 is characterized in that: In step 4, the final embedding representation of the short video node is obtained in the following manner: In the formula Among them, ε v represents the final embedding representation of the short video node v, a v,t represents the importance of node type t to the short video node v, U is the attention parameter, and f t represents the importance of node type t to node v, f j is the node embedding representation.
6. The short video recommendation method based on heterogeneous graph neural network by integrating multimodal data according to claim 1 is characterized in that: In step 4, the final session embedding representation is obtained in the following manner: In the formula s l =n s,n Among them, s h represents the final session embedding representation, s l is the local embedding of the last session, s g is the global session embedding, a v is the attention weight of different embedding representations, ν s,n is the embedding vector of the last short video in the current session, W, W1, W2 are the weights used to generate session embedding, σ represents the sigmoid function, and c is a bias term.
7. The short video recommendation method based on heterogeneous graph neural network by integrating multimodal data according to claim 1 is characterized in that: The behavior-specific embedding in the propagation layer performs message passing in the following manner: in, represents the embedding of user node u under the behavior k of the i+1th embedding propagation layer, LeakyReLU is the optimized version of the rectified linear unit, mean is the mean function, is the set of neighbors of u under behavior k, W (i) is a layer-specific parameter, ⊙ represents the element-wise product of two vectors, represents the duration factor embedding of behavior k in the i-th embedding propagation layer, represents the embedding of all user nodes under row k in layer i, represents the embedding of all user nodes under behavior k in the i+1th embedding propagation layer, are layer-specific parameters.
8. The short video recommendation method based on heterogeneous graph neural network by integrating multimodal data according to claim 1 is characterized in that: In the cross-behavior dependency modeling module, a self-attention mechanism is used to connect the embeddings of user node u under all behaviors, and then the coefficient of the dependency relationship between user u under behavior k and other behaviors is calculated to obtain the enhanced embedding of node u under behavior k, and then the mean pool is used to integrate the embeddings of all layers to obtain the user representation after adding the behavior embedding.
9. The short video recommendation method based on heterogeneous graph neural network by integrating multimodal data according to claim 1 is characterized in that: In step 6, the contrast loss function is: In the formula in, is the contrast loss function between behaviors, It is behavior o and k p The contrast loss between is the behavior k that eliminates the omission of specific user nodes o and k p The contrast loss between is the behavior k that eliminates the omission of specific short video nodes o and k p The contrast loss between them, FN(u) represents the underreporting of user node u, They are user node u in behavior k o and k p The embedding representation below is is the user node v in behavior k p The embedding representation below, FN(v2) represents the underreporting of short video node v2, They are short video node v1 in behavior k o and k p The embedding representation below is It is the short video node v2 in behavior k p The embedding representation below is, τ is the temperature hyperparameter in Softmax, and φ(·) represents the inner product of two vectors.
10. A short video recommendation system based on heterogeneous graph neural network integrating multimodal data, characterized in that: include: A data acquisition unit is used to collect multiple user interaction behavior data on the short video platform and collect multimodal data at the same time; The obtained user interaction behaviors are decomposed into conversation sequences; the user interaction behaviors include watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data; Heterogeneous graph construction unit, used to construct a heterogeneous graph G according to the session sequence n =(V,S,U,E v ,E), where V represents the short video node set, S represents the session node set, U represents the user node set, E v represents the set of transformation relations between short videos, E represents the set of relations between short videos and sessions, short videos and users, and users and sessions; constructing a multi-behavior heterogeneous graph in represents the user node set, v represents the short video node set, and ε represents and v; and divide the multi-behavior heterogeneous graph into multiple behavior subgraphs according to the interaction behavior type; Multimodal feature extraction and fusion unit for heterogeneous graph G through graph attention network n The heterogeneous neighbors of short and medium video nodes are sampled dynamically, and all types and the same number of heterogeneous neighbors are selected for each short video node; Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes. The first embedding representation extraction unit is used to obtain the initial embedding representation of the short video node using GRU aggregation, and combines it with the attention mechanism to obtain the final embedding representation of the short video node, and then obtain the initial session embedding representation, introduce the temporal context information, and combine it with the soft attention mechanism to obtain the final session embedding representation; The second embedding representation extraction unit is used to build a multi-behavior graph perception network based on GNN to capture the collaborative signals between nodes and behaviors; The multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; multiple behavior subgraphs are input into the constructed multi-behavior graph perception network to obtain a user representation after adding behavior embedding; A graph contrast learning unit is used to enhance the user representation obtained after adding behavior embedding through graph contrast learning combined with the constructed contrast loss function; The short video recommendation unit is used to splice the final session embedding representation and the enhanced user representation, input the spliced features into the factor decomposition machine, obtain the corresponding predicted score, and recommend short videos to users according to the predicted score.
Citation Information
Patent Citations
Session social recommendation method based on multi-modal cross fusion graph network
CN112685657A
Session recommendation method based on heterogeneous graph and comparative learning
CN115018573A
Multi-modal project recommendation method, system and equipment and storage medium
CN116561446A
Neural time gate time sequence enhancement session recommendation method based on graph neural network
CN117556133A
New project session recommendation method fusing heterogeneous information
CN118245686A
Cited By
Personalized recommendation method for online learning video resources based on graph neural network
CN119760170A
Online learning video resource personalized recommendation method based on graph neural network
CN119760170B
Heterogeneous graph information mining method for online learning video recommendation
CN119782580A
A heterogeneous graph information mining method for online learning video recommendation
CN119782580B
Multi-modal recommendation method based on heterogeneous graph and homomorphic graph collaborative optimization and application
CN120198206A
