Short video recommendation method and system based on heterogeneous graph neural network for fusing multi-modal data

Abstract

The application discloses a short video recommendation method and system based on a heterogeneous graph neural network by fusing multi-modal data, and aims at the problem that current short video recommendation methods do not fully consider rich text, image, audio and other multi-modal information in the video, still adopt a simple multi-modal feature adding mode, and lack deep mining of the correlation between different modes. Meanwhile, the potential interest features in various interactive behaviors (such as browsing, liking, collecting and the like) of the user are neglected, resulting in the limitation of the recommendation result. The application constructs a heterogeneous content encoder containing an attention mechanism to fuse the features of different modes, uses a heterogeneous graph neural network to mine the potential features of the short video. At the same time, the time context information is introduced, the potential features in various behaviors of the user are mined by using a graph perception network, then the graph contrast is used to reduce the influence caused by the sparseness of the various interactive behavior supervision signals, and finally accurate short video recommendation is provided for the user.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a short video recommendation method and system based on heterogeneous graph neural networks that integrates multimodal data. Background Technology

[0002] Initially, research on short video recommendation primarily focused on traditional recommendation system methods, such as collaborative filtering and content filtering. This neglected the complex relationships between users and short videos, making it difficult to reveal the relevance of potential user interests to short videos. With the rise of graph neural networks (Graph Neural Networks), researchers began to introduce them into short video recommendation. This new approach better models the relationship network between users and short videos. Graph Neural Networks learn the representations of nodes and edges to capture complex relationships and potential patterns in the graph, thereby improving the accuracy and personalization of short video recommendations. In recent years, with the rapid development of deep learning technology, Graph Neural Networks have been widely applied in short video recommendation. Graph Neural Network models trained through deep learning can better learn the representations of users and short videos. Currently, scholars are dedicated to further exploring the application of Graph Neural Networks in short video recommendation, including modifying model structures and introducing attention mechanisms to improve model performance and efficiency. Meanwhile, they also combine short video recommendation with other fields, such as integrating multiple modal data in the feature representation of short videos to fully express their potential features; and in terms of user feature representation, they analyze various user interaction behaviors (such as swiping, liking, and collecting) to uncover deeper user preferences and achieve more accurate and personalized recommendations.

[0003] In summary, the technology of using heterogeneous graph neural networks to analyze user interests and preferences has evolved from traditional recommendation systems to the introduction of graph neural networks and the integration of deep learning, multimodal technologies, and other advancements, providing a more accurate and personalized solution for short video recommendations. In the future, this technology is expected to further drive the development of the short video recommendation system field.

[0004] Currently, short video recommendation algorithms primarily rely on Graph Neural Networks (GNNs) to model the complex interactions between users and videos, learning higher-order relationships between user-item nodes to enrich node representations. While some research has attempted to incorporate multimodal information (such as text, images, and audio) to improve recommendation performance, most methods still employ simple summation of multimodal features, lacking in-depth exploration of the relationships between different modalities. Furthermore, existing short video recommendation algorithms often overlook the potential interest features within users' various interactive behaviors (such as browsing, liking, and saving), leading to limitations in the recommendation results. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a short video recommendation method and system based on heterogeneous graph neural networks that integrates multimodal data. By fusing multiple modal features of short videos and mining various user interaction behaviors and potential interest features within the temporal context, the invention ultimately utilizes heterogeneous graph neural networks to analyze user preferences. This invention can more accurately provide users with short video content that interests them, thereby increasing user engagement time.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention proposes a short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data, comprising:

[0008] Step 1: Collect various user interaction behavior data on short video platforms, and simultaneously collect multimodal data; decompose the obtained user interaction behaviors into conversation sequences; the user interaction behaviors include watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data;

[0009] Step 2: Construct a heterogeneous graph based on the session sequence Where V represents the short video node set, S represents the session node set, and U represents the user node set. This represents a set of conversion relationships between short videos. Represent the relationship sets between short videos and conversations, short videos and users, and users and conversations; construct a multi-behavior heterogeneous graph. ,in Represents a set of user nodes. Represents a set of short video nodes. express and The set of interaction behaviors between them; and the multi-behavior heterogeneous graph is divided into multiple behavior subgraphs according to the type of interaction behavior;

[0010] Step 3: Apply graph attention network to heterogeneous graphs The heterogeneous neighbors of short video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes.

[0011] Step 4: Using the initial embedding representation of the short video node obtained by GRU aggregation, combined with the attention mechanism, the final embedding representation of the short video node is obtained, and then the initial session embedding representation is obtained. Temporal context information is introduced, and combined with the soft attention mechanism, the final session embedding representation is obtained.

[0012] Step 5: Construct a multi-behavior graph perception network based on GNN to capture cooperative signals between nodes and behaviors; the multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; input multiple behavior subgraphs into the constructed multi-behavior graph perception network to obtain user representations with added behavior embeddings;

[0013] Step 6: Enhance the user representation after adding behavioral embeddings by using graph contrastive learning and the constructed contrastive loss function;

[0014] Step 7: Concatenate the final session embedding representation and the enhanced user representation, input the concatenated features into a factor decomposition machine to obtain the corresponding predicted score, and recommend short videos to users based on the predicted score.

[0015] Furthermore, in step 3, all types and the same number of heterogeneous neighbors are selected for each short video node in the following manner:

[0016] For all short video nodes, a random walk is started from each short video node. During the random walk, the current node's neighbors are moved iteratively with dynamic probability according to the attention weight. The number of walks for each type of node is controlled to ensure that all types of nodes are obtained during the random walk. All walked nodes are stored in a list.

[0017] All nodes in the list are categorized, and the same number of nodes are selected for each modality as heterogeneous neighbors of the short video node.

[0018] Furthermore, the initial embedding representation of the short video node is as follows:

[0019]

[0020] in short video nodes The initial embedding representation, where d is the dimension of the node embedding; is a linear transformation layer; ⊕ is the connection operation; x is a set of multiple modes. These are the attention weights for mode t. and This indicates a bidirectional GRU.

[0021] Furthermore, in step 4, the initial embedding representation of the short video nodes is obtained by aggregating them in the following manner:

[0022]

[0023] in, Embedded representation of short video node v under node type t This represents the t-shaped neighborhood set of a short video node v. short video nodes The initial embedding representation.

[0024] Furthermore, in step 4, the final embedded representation of the short video node is obtained in the following manner:

[0025]

[0026] In the formula

[0027]

[0028] in, This represents the final embedding representation of the short video node v. This indicates the importance of node type t to short video node v. It is an attention parameter. This indicates the importance of node type t to node v. It is a node embedding representation.

[0029] Furthermore, in step 4, the final session embedding representation is obtained in the following manner:

[0030]

[0031] In the formula

[0032]

[0033]

[0034] in, This represents the final session embedding representation. It is a partial embedding of the last session. It is a global session embedding. These are the attention weights for different embedding representations. W is the embedding vector of the last short video in the current session. These are the weights used to generate session embeddings. This represents the sigmoid function, where c is a bias term.

[0035] Furthermore, in the behavior-specific embedded propagation layer, message passing is performed in the following manner:

[0036]

[0037]

[0038] in, Let represent the embedding of user node u under behavior k in the (i+1)th embedding propagation layer. LeakyReLU is an optimized version of the rectified linear unit, and mean is the mean function. It is the nearest neighbor set of u under behavior k. These are layer-specific parameters. This represents the element-wise product of two vectors. This represents the duration factor embedding of behavior k in the i-th embedding propagation layer. This represents the embedding of all user nodes under line k in layer i. This represents the embedding of all user nodes under behavior k in the (i+1)th embedding propagation layer. These are layer-specific parameters.

[0039] Furthermore, in the cross-behavior dependency modeling module, a self-attention mechanism is used to connect the embeddings of user node u under all behaviors, and then the coefficient of the dependency relationship between user u and other behaviors under behavior k is calculated, thereby obtaining the enhanced embedding of node u under behavior k. Then, the mean pool is used to integrate the embeddings of all layers to obtain the user representation after adding the behavior embeddings.

[0040] Furthermore, in step 6, the contrastive loss function is:

[0041] ,

[0042] In the formula

[0043] ,

[0044]

[0045]

[0046] in, It is a contrastive loss function between behaviors. It is behavior The difference in loss between them It is the behavior of eliminating false alarms for specific user nodes. and The difference in loss between them It is the behavior of eliminating false alarms for specific short video nodes. and The difference in loss between them This indicates a missed report for user node u. , These are the behaviors of user node u. and Embedded representation below, It is the user node v in the behavior Embedded representation below, Represents short video nodes underreporting, , These are short video nodes In behavior and Embedded representation below, It is a short video node In behavior Embedded representation below, It is the temperature hyperparameter in Softmax. This represents the inner product of two vectors.

[0047] Another aspect of this invention proposes a short video recommendation system based on heterogeneous graph neural networks that integrates multimodal data, comprising:

[0048] The data acquisition unit is used to collect various user interaction behavior data on the short video platform, and simultaneously collect multimodal data; and decompose the obtained user interaction behaviors into conversation sequences; the user interaction behaviors include watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data;

[0049] Heterogeneous graph construction unit, used to construct heterogeneous graphs based on session sequences. Where V represents the short video node set, S represents the session node set, and U represents the user node set. This represents a set of conversion relationships between short videos. Represent the relationship sets between short videos and conversations, short videos and users, and users and conversations; construct a multi-behavior heterogeneous graph. ,in Represents a set of user nodes. Represents a set of short video nodes. express and The set of interaction behaviors between them; and the multi-behavior heterogeneous graph is divided into multiple behavior subgraphs according to the type of interaction behavior;

[0050] The multimodal feature extraction and fusion unit is used to process heterogeneous graphs through a graph attention network. The heterogeneous neighbors of short video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes.

[0051] The first embedding representation extraction unit is used to obtain the initial embedding representation of the short video node by GRU aggregation, and then to obtain the final embedding representation of the short video node by combining the attention mechanism, thereby obtaining the initial session embedding representation. Temporal context information is introduced, and combined with the soft attention mechanism, the final session embedding representation is obtained.

[0052] The second embedding representation extraction unit is used to construct a multi-behavior graph perception network based on GNN to capture cooperative signals between nodes and behaviors; the multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; multiple behavior subgraphs are input into the constructed multi-behavior graph perception network to obtain user representations with added behavior embeddings;

[0053] The graph contrastive learning unit is used to enhance the user representation after adding behavior embeddings by learning from graph contrasts and combining it with the constructed contrastive loss function.

[0054] The short video recommendation unit is used to concatenate the final session embedding representation and the enhanced user representation, input the concatenated features into a factor decomposition machine to obtain the corresponding predicted score, and recommend short videos to users according to the predicted score.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] This invention fully considers the multimodal features of short videos and the dependencies between different modalities, utilizing heterogeneous graph neural networks to capture finer-grained short video features. Secondly, by incorporating various user interaction behaviors and contextual information, it captures detailed user features through graph perceptual networks and uses graph contrastive learning to alleviate the problem of interaction sparsity. This successfully uncovers users' potential interests, thereby providing more accurate and personalized short video recommendations. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating a short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data, according to an embodiment of the present invention.

[0058] Figure 2 This is a schematic diagram of a heterogeneous heterostructure provided in an embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of a multi-behavior heterogeneous graph provided in an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of a multi-behavior graph sensing network structure provided in an embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of the structure of the multi-line graph comparison module provided in an embodiment of the present invention;

[0062] Figure 6 This is a schematic diagram of the architecture of a short video recommendation system based on heterogeneous graph neural networks that integrates multimodal data, according to an embodiment of the present invention. Detailed Implementation

[0063] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0064] like Figure 1 As shown, a short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data includes:

[0065] S101: Collect various user interaction behavior data on short video platforms, and simultaneously collect multimodal data; decompose the obtained user interaction behaviors into conversation sequences; the user interaction behaviors include watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data;

[0066] S102: Construct a heterogeneous graph based on the session sequence Where V represents the short video node set, S represents the session node set, and U represents the user node set. This represents a set of conversion relationships between short videos. Represent the relationship sets between short videos and conversations, short videos and users, and users and conversations; construct a multi-behavior heterogeneous graph. ,in Represents a set of user nodes. Represents a set of short video nodes. express and The set of interaction behaviors between them; and the multi-behavior heterogeneous graph is divided into multiple behavior subgraphs according to the type of interaction behavior;

[0067] S103: Heterogeneous graphs through graph attention networks The heterogeneous neighbors of short video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes.

[0068] S104: The initial embedding representation of the short video node obtained by GRU aggregation is combined with the attention mechanism to obtain the final embedding representation of the short video node, and then the initial session embedding representation is obtained. Temporal context information is introduced and combined with the soft attention mechanism to obtain the final session embedding representation.

[0069] S105: Construct a multi-behavior graph perception network based on GNN to capture cooperative signals between nodes and behaviors; the multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; input multiple behavior subgraphs into the constructed multi-behavior graph perception network to obtain user representations with added behavior embeddings;

[0070] S106: Enhance the user representation after adding behavior embeddings by using graph contrastive learning and combining it with the constructed contrastive loss function;

[0071] S107: Concatenate the final session embedding representation and the enhanced user representation, input the concatenated features into a factor decomposition machine to obtain the corresponding predicted score, and recommend short videos to users according to the predicted score.

[0072] The method specifically includes:

[0073] 1. Data preprocessing

[0074] This invention collects various user interaction behavior data from short video platforms, including viewing, liking, and saving behaviors, as well as multimodal data such as short video content data, audio data, and text data. In the data preprocessing stage, noise reduction and missing value handling are performed to ensure the accuracy and completeness of the data. Secondly, the obtained interaction behavior V is decomposed into a session sequence S.

[0075] 2. Construction of Heterogeneous Heterogeneous Graphs and Multi-behavioral Heterogeneous Graphs

[0076] 2.1. Heterogeneous diagram

[0077] First, the session sequence S needs to be constructed as a heterogeneous graph. , It includes three types of nodes: short video nodes, session nodes, and user nodes, with V, S, and U representing the sets corresponding to each of the three, and two types of edges: directed edges and undirected edges between two videos. Let E and E be the sets corresponding to the two sets, respectively. For example... Figure 2 As shown. The user node set U includes: user ID, etc. The short video node set V includes: short video features, mainly multimodal features: 1) short video content (image frame data), 2) audio content (background music, dialogue, etc.), 3) text content (description, tags, etc.). The session node set S includes: 1) session ID: each session node has a unique identifier used to distinguish different sessions. 2) contextual information: such as timestamps, geographical location, etc. This indicates that the user is watching a video. Then I watched the next video. . It can represent a set of complex transformation relationships between short videos. , respectively representing the relationship between short videos and conversations, short videos and users, and users and conversations.

[0078] 2.2. Multi-behavior heterogeneous graph

[0079] This invention represents the user node set and the short video node set as follows: and ,in These represent the number of users and the number of short videos, respectively. (Bibliographic diagram based on user-short video interaction) like Figure 3 As shown, it contains two types of nodes: short video nodes and user nodes, as well as various types of edges, and edge sets. express and Different interactive behaviors between them. In a multi-behavior scenario with k types of interactive behaviors, the entire bipartite graph... Based on the different types of interaction behaviors, it can be divided into k behavior subgraphs. To reflect multi-behavioral interaction data, a tensor is defined here. If individual elements If the value approaches 1, it indicates that user u interacts with short video v under behavior k; otherwise, it approaches 0.

[0080] 3. Multimodal extraction and fusion module

[0081] create Subsequently, this invention uses a heterogeneous graph neural network that integrates multiple modalities to learn item embeddings, i.e., short video embeddings, which have rich information and complex transformation relationships.

[0082] First, a graph attention network is introduced to dynamically sample the heterogeneous neighbors of a node, selecting all modality types and the same number of heterogeneous neighbors for each node. Then, a heterogeneous content encoder is designed to aggregate the heterogeneous neighbor content of the node, fusing different modal features to obtain the feature representation of the node.

[0083] 3.1 Heterogeneous Neighbor Sampling

[0084] In the figure In this context, the types and numbers of these heterogeneous neighbor nodes differ, and aggregating these nodes may require different feature transformations. For example, one user watched three short videos in two sessions, while another user watched ten short videos in three sessions. Here, the sizes of the neighbor nodes for the two users are different. Selecting nodes as heterogeneous neighbors for users so that they can be aggregated using the same model is an important issue.

[0085] To address this issue, classic topological feature extraction algorithms include DeepWalk and Node2vec. However, neither of these methods considers sampling neighboring nodes based on higher-order weights and node attributes. Therefore, we designed a heterogeneous neighbor sampling strategy (SWAD) based on higher-order weights and node attributes. This method walks according to weights to collect all types of neighbors for each node; it fixes the sampling neighbor size for each node and selects the most frequently visited neighbors; finally, it groups neighbors of the same type (with the same content features) to enable type-based aggregation.

[0086] Taking short video nodes as an example, the main steps of SWAD are as follows:

[0087] S201: For all short video nodes From each short video node Initiate a random walk. During the random walk, move to the neighbors of the current node iteratively with dynamic probability p according to the attention weight. SWAD controls the number of visits for each type of node, ensuring that all types of nodes are visited during the random walk, and stores all visited nodes in a list, i.e., SWAD(v);

[0088] S202: Classify all nodes in SWAD(v) and select for each mode t Each node serves as a short video node. Heterogeneous neighbors. As one possible implementation, in this embodiment, the same number of nodes are selected as short video nodes for each modality t. heterogeneous neighbors.

[0089] 3.2 Heterogeneous Content Encoder

[0090] Heterogeneous neighbors of different modalities have different node contents. For example, short video nodes contain modalities such as image frames, audio, and text. To address this, we design a GRU-based heterogeneous content encoder to capture the interactions between features, encode different modalities of a node into fixed-dimensional embedding representations, and aggregate all modal features of a node into the node's embedding representation, giving it greater representational power.

[0091] First, node v is a node For heterogeneous neighbors, we represent the feature of the t-th modality in node v as: (d) f (Feature dimension). Please note, Different techniques can be used for pre-training on different modalities of data. For content of different modalities, this invention uses different methods to convert the content of different modalities into embedding vectors of the same dimension. Specifically, for image and text data, we directly use BEIT-3 (BertPre-TrainingofImageTransformers-3), which currently has the best performance, to capture features of different modalities, while for audio data, we use VGGish to extract audio features. These methods are all based on deep learning and can automatically extract representative features from video, audio, and text.

[0092] Then, unlike previous models that linearly transform different modal features into a uniform vector, we use GRU to capture deep feature interactions to gain more powerful expressive power.

[0093] More specifically, different linear transformation layers are first used to transform different content features. Then, a GRU is used to capture "deep" feature interactions and accumulate the expressive power of all content features. Considering the different levels of importance of information from different modalities, this invention incorporates image features, audio features, and text features into attention weights for fusion, resulting in a richer and more complex feature representation.

[0094] After obtaining the embedding vectors of each node's attributes, the node embedding... This can be expressed as:

[0095]

[0096] in d is the dimension of node embedding; is a linear transformation layer; ⊕ is the connection operation; x is a set of multiple modes. These are the attention weights for different modalities. and This indicates a bidirectional GRU.

[0097] The formula for GRU is as follows:

[0098]

[0099] (2)

[0100]

[0101]

[0102] in, This indicates the information to be entered at the current time; This represents the hidden state from the previous time step. The hidden state acts as the neural network's memory, containing information about the data seen by previous nodes. This indicates the hidden state that will be passed on to the next time step; Indicates the candidate hidden state; , Represents the reset gate and update gate for the t-th modal feature; This represents the sigmoid function, which can transform data into values ​​in the range of 0-1. This represents the tanh function, which transforms data into values ​​in the range [-1, 1].

[0103] The heterogeneous content encoder described above has a simple structure and relatively low complexity (fewer parameters). It can aggregate heterogeneous content, making node embedding more expressive. In addition, it can flexibly add extra modal features, making model expansion convenient.

[0104] 4 Heterogeneous Graph Neural Network Layers

[0105] After aggregating the heterogeneous node content of neighbors, we generate embedding representations of heterogeneous neighbors. Each node has multiple types of heterogeneous neighbors. GRU can aggregate on unordered neighbor sets, so we design a neural network that uses GRU to aggregate nodes of the same type and learn the complex relationships between them, so that the learned type embeddings... It has stronger expressive power. GRU can solve problems in RNNs such as the inability to retain information for long periods and gradient issues in backpropagation, similar to LSTM, but simpler and easier to train. Type embedding The calculation formula is as follows:

[0106]

[0107] The GRU model is the same as equation (2), except for the input and parameter settings. Let V represent the t-shaped neighborhood set of short video node v∈V.

[0108] After obtaining the type embedding of the short video node v, it is necessary to aggregate all type embeddings into a vector, which is the node. The final embedding. However, different types of heterogeneous neighbor pairs of nodes. The impact of different types on node v varies, therefore an attention mechanism is introduced. The formulas for calculating the importance of different types to node v are as follows:

[0109]

[0110] in This indicates the importance of node type t to node v. It is a node embedding representation.

[0111] LeakyReLU is an optimized version of the rectified linear unit. These are attention parameters. The values ​​for each type of node are calculated. After the importance of nodes The embedding calculation is as follows:

[0112]

[0113] The resulting project nodes are ultimately embedded. It not only includes the transformation items between them, but also learns information about other types of nodes, making them more expressive.

[0114] After obtaining the project embedding representation, a session embedding is generated. For the session... The pre-embedded vector is calculated as follows:

[0115]

[0116] in It is a node Node embedding; It is a zero vector with no fixed dimension; For connection operations. Since the number of short videos in each session may vary, it is necessary to add... This ensures that all embedded session vectors have the same spatial dimension.

[0117] In practical applications, short video platforms place extremely high demands on the real-time performance of their recommendation systems. User interests are not static; they can change over time and with accumulated experience—a dynamic process. This makes it difficult for existing recommendation models to accurately capture and adapt to users' latest interests. Current short video recommendations cannot simultaneously balance users' long-term and dynamic preferences. Therefore, capturing users' dynamic preferences and providing accurate and timely recommendations is of paramount importance.

[0118] Therefore, an attention mechanism module is added to the model to obtain a hybrid session embedding that can express both long-term and dynamic preferences. In this invention, the local embedding of the last session is considered first. The formula is as follows:

[0119]

[0120] in It is the embedding vector of the last short video in the current session.

[0121] Based on users' long-term interests and preferences, and taking into account the transformation relationships between all videos, this invention employs a soft attention mechanism to learn global session embeddings. Global session embedding The formula is as follows:

[0122]

[0123] Where the matrix , is the weight used to generate the session embedding, and c is a bias term. After obtaining the global and local embedding representations of the session, the final session embedding can be expressed as:

[0124]

[0125] in These are the attention weights for different embedding representations. Finally, a linear transformation is used to convert them into a session embedding representation. The linear transformation can be a fully connected layer that maps the high-dimensional features of the input to a low-dimensional space. By learning the session embedding, the overall information and semantics of the session can be captured.

[0126] 5. Multi-behavior graph sensing network

[0127] This invention constructs a multi-behavior graph-aware network based on GNNs to capture complex cooperative signals between nodes and behaviors. Specifically, it first utilizes behavior-specific embedding propagation to obtain the context node embeddings under each behavior subgraph. Then, it enhances these embeddings by leveraging personalized behavior relevance through cross-behavior dependency modeling. To better capture the relationships between nodes and behaviors, this invention also introduces a novel method to strengthen the representational power of node embeddings by analyzing the interrelationships between behaviors. This method enables the multi-behavior graph-aware network to more comprehensively understand the complex interactions between nodes and behaviors, thereby capturing cooperative signals more accurately. In this way, the model of this invention has a stronger performance in capturing node-behavior correlations, thus providing more accurate and reliable prediction capabilities for recommendation systems, such as... Figure 4 As shown (where Sv represents the user representation with embedded user behavior).

[0128] 5.1. Behavior-Specific Embedded Propagation Layer

[0129] In multi-behavior scenarios, firstly, construct each bipartite behavior subgraph based on k types of interaction behaviors. Then, embedding propagation is performed on each subgraph to obtain the representation of each node under each behavior. To more accurately reflect the differences between each behavior and capture the semantic features of actual user preferences, the duration of different user behaviors varies. For browsing behavior, we calculate its playback completion rate; for behaviors such as liking and favorites, we calculate the duration of each behavior separately. Then, an attention mechanism is used to obtain the final behavior duration factor embedding. Furthermore, this invention also incorporates the embedding of nodes under specific behaviors and the representation of each behavior into the message passing paradigm:

[0130]

[0131] in, Let represent the embedding of user node u under behavior k in the (i+1)th embedding propagation layer. LeakyReLU is an optimized version of the rectified linear unit, and mean is the mean function. It is the nearest neighbor set of u under behavior type k. It is a specific layer. This represents the element-wise product of two vectors. This represents the embedding of the duration factor of behavior k in the i-th layer. This represents the embedding of all user nodes under behavior k in layer i, multiplied by another layer-specific parameter. Update:

[0132]

[0133] 5.2. Cross-behavior dependency modeling module

[0134] Considering that there are implicit relationships between different behaviors based on the user, and that these relationships vary from user to user, this invention employs a self-attention mechanism in order to effectively model cross-behavior dependencies.

[0135] Specifically, the embedding of node u under all behaviors is connected as follows: Then calculate The coefficient representing the dependency relationship between user u and other behaviors under behavior k.

[0136]

[0137] in , These are two behavior-specific parameters. This is the output dimension size. Therefore, the enhanced embedding of node u under behavior k can be easily computed as:

[0138]

[0139] Since the embeddings of different layers represent different connections, this invention uses a mean pooling method to integrate the embeddings of all layers as follows:

[0140]

[0141] 6. Multi-line graph comparison module

[0142] In real-world scenarios, multi-behavioral interaction data often exhibits sparsity, a situation also present in short video recommendation scenarios. Furthermore, ignoring contextual information can lead to inaccurate model understanding of user interests and preferences; for example, neglecting user interaction time may weaken recommendation performance. To address the data sparsity problem and improve the model's generalization ability, it is necessary to avoid over-reliance on a single interaction signal and comprehensively consider multiple different types of behavioral interactions. Moreover, noting that sparse supervision signals may lead to biased learning representations, appropriate methods are needed to extract more meaningful feature representations. Therefore, this invention constructs a novel contrastive learning paradigm that helps mitigate the impact of sparsity in supervision signals from multiple interactive behaviors and can more effectively handle interference from multiple behaviors. Considering that supervision signals from multiple behaviors are richer than those from a single interactive behavior, this invention employs a selective contrastive learning method to transfer knowledge between multiple interactive behaviors. This approach alleviates the data sparsity problem. Figure 5 As shown (where Sv' represents the final user representation obtained through graph comparison learning).

[0143] Specifically, the present invention selects Regarding behavior The subgraphs are compared to provide different semantics. A common practice is to treat views of the same node as positive pairs. Views of different nodes are treated as negative pairs. .

[0144] However, in recommendation settings involving various interactions, similar objects often share commonalities, such as users having similar preferences or items having similar attributes. However, constructing negative pairs using traditional methods can introduce a large number of missed detections—highly similar nodes—leading to the loss of genuine semantic information. Therefore, to address this issue, this invention employs the Swing algorithm to identify potential missed detections by calculating similarity scores and filtering them out during node pair comparisons to eliminate their impact. In this way, relevant semantic information can be discovered and utilized more accurately, thereby improving the performance and effectiveness of the recommendation system. More specifically, subgraphs... Two users The similarity score is calculated using the Swing algorithm as follows:

[0145] in, It is the smoothing coefficient. It is the nearest neighbor set of u under behavior type k. Final similarity score. It is each subgraph The average of the fractions.

[0146] Then, here we define a missed call for user node u as a user with the top n similarity scores. Therefore, the behavior of eliminating specific false negatives for user node u is... and The contrast loss between them is defined as:

[0147]

[0148] in, It is the behavior of user node u Embedded representation below, It is the temperature hyperparameter in Softmax. This represents the inner product of two vectors.

[0149] Similarly, eliminating specific false negatives in short video node v. and The contrast loss between them is defined as:

[0150]

[0151] in, It is a short video node In behavior Embedded representation below, It is the temperature hyperparameter in Softmax. This represents the inner product of two vectors. Represents short video nodes The missed reports specifically refer to short video nodes with the top n similarity scores.

[0152] When comparing losses with those of the project owner, the behavior... The contrast loss between them is: ;

[0153] Therefore, the contrast loss between the final actions is for each pair of different actions. The difference between the two is half the sum of the losses: .

[0154] 7. Predicting user interests and preferences

[0155] The final representation of the user and the short video is obtained by splicing together. And this representation is fed into the factorization machine:

[0156]

[0157] in This represents the model's predicted score. These are the weights of the user embedding representation and the short video embedding representation, respectively. express dot product, and These are global bias, user bias, and project bias.

[0158] Based on the above embodiments, such as Figure 6 As shown, this invention also proposes a short video recommendation system based on heterogeneous graph neural networks that integrates multimodal data, including:

[0159] The data acquisition unit is used to collect various user interaction behavior data on the short video platform, and simultaneously collect multimodal data; and decompose the obtained user interaction behaviors into conversation sequences; the user interaction behaviors include watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data;

[0160] Heterogeneous graph construction unit, used to construct heterogeneous graphs based on session sequences. Where V represents the short video node set, S represents the session node set, and U represents the user node set. This represents a set of conversion relationships between short videos. Represent the relationship sets between short videos and conversations, short videos and users, and users and conversations; construct a multi-behavior heterogeneous graph. ,in Represents a set of user nodes. Represents a set of short video nodes. express and The set of interaction behaviors between them; and the multi-behavior heterogeneous graph is divided into multiple behavior subgraphs according to the type of interaction behavior;

[0161] The multimodal feature extraction and fusion unit is used to process heterogeneous graphs through a graph attention network. The heterogeneous neighbors of short video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes.

[0162] The first embedding representation extraction unit is used to obtain the initial embedding representation of the short video node by GRU aggregation, and then to obtain the final embedding representation of the short video node by combining the attention mechanism, thereby obtaining the initial session embedding representation. Temporal context information is introduced, and combined with the soft attention mechanism, the final session embedding representation is obtained.

[0163] The second embedding representation extraction unit is used to construct a multi-behavior graph perception network based on GNN to capture cooperative signals between nodes and behaviors; the multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; multiple behavior subgraphs are input into the constructed multi-behavior graph perception network to obtain user representations with added behavior embeddings;

[0164] The graph contrastive learning unit is used to enhance the user representation after adding behavior embeddings by learning from graph contrasts and combining it with the constructed contrastive loss function.

[0165] The short video recommendation unit is used to concatenate the final session embedding representation and the enhanced user representation, input the concatenated features into a factor decomposition machine to obtain the corresponding predicted score, and recommend short videos to users according to the predicted score.

[0166] In summary, current short video recommendation methods fail to fully consider the rich multimodal information (text, images, audio, etc.) within videos and still rely on simple feature aggregation, lacking in-depth analysis of the relationships between different modalities. Furthermore, they neglect the potential interest features within various user interactions (such as browsing, liking, and saving), leading to limitations in the recommendation results. This invention constructs a heterogeneous content encoder incorporating an attention mechanism to fuse features from different modalities and uses a heterogeneous graph neural network to mine the latent features of short videos. Simultaneously, it introduces temporal context information, utilizes graph-aware networks to mine latent features from various user behaviors, and then uses graph contrast to mitigate the impact of sparsity in the supervision signals of multiple interactive behaviors, ultimately providing users with accurate short video recommendations.

[0167] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data, characterized in that, include: Step 1: Collect various user interaction behavior data on short video platforms, and collect multimodal data at the same time; The obtained user interaction behaviors are decomposed into session sequences; the user interaction behaviors include watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data; Step 2: Construct a heterogeneous graph G based on the session sequence n =(V,S,U,E) v (E), where V represents the short video node set, S represents the session node set, U represents the user node set, and E v Let E represent the set of transformation relationships between short videos, and E represent the set of relationships between short videos and conversations, short videos and users, and users and conversations; construct a multi-behavior heterogeneous graph. in Let v represent the set of user nodes, v represent the set of short video nodes, and ε represent... The set of interaction behaviors between v; and the multi-behavior heterogeneous graph is divided into multiple behavior subgraphs according to the type of interaction behavior; Step 3: Apply graph attention network to the heterogeneous graph G n The heterogeneous neighbors of short video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes, fuse different modal features, and obtain the initial embedding representation of short video nodes. Step 4: Using the initial embedding representation of the short video node obtained by GRU aggregation, combined with the attention mechanism, the final embedding representation of the short video node is obtained, and then the initial session embedding representation is obtained. Temporal context information is introduced, and combined with the soft attention mechanism, the final session embedding representation is obtained. Step 5: Construct a multi-behavior graph perception network based on GNN to capture cooperative signals between nodes and behaviors; the multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; input multiple behavior subgraphs into the constructed multi-behavior graph perception network to obtain user representations with added behavior embeddings; Step 6: Enhance the user representation after adding behavioral embeddings by using graph contrastive learning and the constructed contrastive loss function; Step 7: Concatenate the final session embedding representation and the enhanced user representation, input the concatenated features into a factor decomposition machine to obtain the corresponding predicted score, and recommend short videos to users based on the predicted score.

2. The short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data according to claim 1, characterized in that, In step 3, all types and the same number of heterogeneous neighbors are selected for each short video node in the following manner: For all short video nodes, a random walk is started from each short video node. During the random walk, the current node's neighbors are moved iteratively with dynamic probability according to the attention weight. The number of walks for each type of node is controlled to ensure that all types of nodes are obtained during the random walk. All walked nodes are stored in a list. All nodes in the list are categorized, and the same number of nodes are selected for each modality as heterogeneous neighbors of the short video node.

3. The short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data according to claim 1, characterized in that, The initial embedding of the short video node is represented as follows: in d is the initial embedding representation of the short video node v, and d is the dimension of the node embedding; It is a linear transformation layer; It is a join operation; x is a set of multiple modalities, a t These are the attention weights for mode t. and This indicates a bidirectional GRU.

4. The short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data according to claim 1, characterized in that, In step 4, the initial embedding representation of the short video nodes is obtained by aggregation in the following manner: Among them, f t (v) is the embedding representation of short video node v under node type t, N t (v) represents the t-shaped neighborhood set of short video node v, and f(v) is the initial embedding representation of short video node v.

5. The short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data according to claim 4, characterized in that, In step 4, the final embedded representation of the short video node is obtained in the following manner: In the formula Where, ε v a represents the final embedding representation of the short video node v. v,t This represents the importance of node type t to short video node v, where U is the attention parameter, and f is the value of t. t f represents the importance of node type t to node v. j It is a node embedding representation.

6. The short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data according to claim 1, characterized in that, In step 4, the final session embedding representation is obtained in the following manner: In the formula s l =n s,n Among them, s h This represents the final session embedding representation, s l It is a partial embedding of the last session, s g It is a global session embedding, a v These are the attention weights for different embedding representations, ν s,n is the embedding vector of the last short video in the current session, W, W1, W2 are the weights used to generate the session embedding, σ represents the sigmoid function, and c is a bias term.

7. The short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data according to claim 1, characterized in that, In the behavior-specific embedded propagation layer, message passing is performed in the following manner: in, Let represent the embedding of user node u under behavior k in the (i+1)th embedding propagation layer. LeakyReLU is an optimized version of the rectified linear unit, and mean is the mean function. W is the nearest neighbor set of u under behavior k. (i) These are layer-specific parameters, and ⊙ represents the element-wise product of two vectors. This represents the duration factor embedding of behavior k in the i-th embedding propagation layer. This represents the embedding of all user nodes under line k in layer i. This represents the embedding of all user nodes under behavior k in the (i+1)th embedding propagation layer. These are layer-specific parameters.

8. The short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data according to claim 1, characterized in that, In the cross-behavior dependency modeling module, a self-attention mechanism is used to connect the embeddings of user node u under all behaviors. Then, the coefficient of the dependency relationship between user u and other behaviors under behavior k is calculated, thereby obtaining the enhanced embedding of node u under behavior k. Then, the mean pool is used to integrate the embeddings of all layers to obtain the user representation after adding the behavior embeddings.

9. The short video recommendation method based on heterogeneous graph neural networks that integrates multimodal data according to claim 1, characterized in that, In step 6, the contrast loss function is: In the formula in, It is a contrastive loss function between behaviors. It is behavior k o and k p The difference in loss between them It is the behavior of eliminating false alarms for specific user nodes. o and k p The difference in loss between them It is the behavior of eliminating false alarms for specific short video nodes. o and k p The comparison loss between them, FN(u) represents the missed reports of user node u. These are user node u and behavior k. o and k p Embedded representation below, It is user node v in behavior k p The embedding representation below, where FN(v2) represents the missed detection of short video node v2, These are the short video node v1 in behavior k. o and k p Embedded representation below, Is the short video node v2 in behavior k p The embedding representation is as follows: τ is the temperature hyperparameter in Softmax, and φ(·) represents the inner product of two vectors.

10. A short video recommendation system based on heterogeneous graph neural networks that integrates multimodal data, characterized in that, include: The data acquisition unit is used to collect various user interaction behavior data on short video platforms, and also collect multimodal data; The obtained user interaction behaviors are decomposed into session sequences; the user interaction behaviors include watching, liking, and collecting behaviors; the multimodal data includes short video content data, audio data, and text data; Heterogeneous graph construction unit, used to construct heterogeneous graph G based on session sequences. n =(V,S,U,E) v (E), where V represents the short video node set, S represents the session node set, U represents the user node set, and E v Let E represent the set of transformation relationships between short videos, and E represent the set of relationships between short videos and conversations, short videos and users, and users and conversations; construct a multi-behavior heterogeneous graph. in Let v represent the set of user nodes, v represent the set of short video nodes, and ε represent... The set of interaction behaviors between v; and the multi-behavior heterogeneous graph is divided into multiple behavior subgraphs according to the type of interaction behavior; A multimodal feature extraction and fusion unit is used to process heterogeneous graphs G through a graph attention network. n The heterogeneous neighbors of short video nodes are dynamically sampled, and all types and the same number of heterogeneous neighbors are selected for each short video node. Then, a GRU-based heterogeneous content encoder is designed to aggregate the heterogeneous neighbor information of short video nodes and fuse different modal features to obtain the initial embedding representation of short video nodes. The first embedding representation extraction unit is used to obtain the initial embedding representation of the short video node by GRU aggregation, and then to obtain the final embedding representation of the short video node by combining the attention mechanism, thereby obtaining the initial session embedding representation. Temporal context information is introduced, and combined with the soft attention mechanism, the final session embedding representation is obtained. The second embedding representation extraction unit is used to construct a multi-behavior graph perception network based on GNN to capture the cooperative signals between nodes and behaviors; The multi-behavior graph perception network includes a behavior-specific embedding propagation layer and a cross-behavior dependency modeling module; multiple behavior subgraphs are input into the constructed multi-behavior graph perception network to obtain user representations with added behavior embeddings; The graph contrastive learning unit is used to enhance the user representation after adding behavior embeddings by learning from graph contrasts and combining it with the constructed contrastive loss function. The short video recommendation unit is used to concatenate the final session embedding representation and the enhanced user representation, input the concatenated features into a factor decomposition machine to obtain the corresponding predicted score, and recommend short videos to users according to the predicted score.

Images