Group video recommendation system and method fusing multi-modal information and interest similarity

Abstract

This invention discloses a group video recommendation system and method that integrates multimodal information and interest similarity. The system comprises a data preprocessing unit, a user interest segmentation and group construction unit, a video multimodal information fusion and representation unit, a dynamic user interest update unit, a user dynamic interest representation extraction unit, and a prediction result representation unit. The system segments users into groups based on their interests and utilizes social networks to learn user interaction behaviors. It mines video multimodal information to enrich semantic information, thereby improving the information richness of the recommendation system. It analyzes users' historical behavior sequences to obtain their latest interests for a more accurate understanding of user needs. A bidirectional attention mechanism is used to aggregate the preferences of group members, thereby obtaining embedded representations of groups and tasks. A multilayer perceptron is then used to generate top-K task recommendations for the groups. This invention not only improves the accuracy and interpretability of the video recommendation system but also achieves personalized recommendations.

Description

Technical Field

[0001] This invention relates to the field of group video recommendation technology, and in particular to a group video recommendation system and method that integrates multimodal information and interest similarity. Background Technology

[0002] With the development of emerging technologies such as 5G and cloud computing, group video recommendations are showing a trend towards diversification and personalization. Statistics show that by the end of 2022, global video app downloads reached 255 billion, and total time spent on social apps exceeded 20 trillion hours. Video platforms provide relevant content to group users based on their social networks, interests, and multimodal information within the videos. Leveraging these characteristics, video platforms can acquire video content from multiple perspectives and glean potential preferences from social relationships, thereby capturing diverse and personalized group interests. For example, when Bob and his friends are traveling in Nanjing, the video platform analyzes the textual and visual information in the content they watch. Based on this analysis, the platform believes they might be interested in the image modality in the video and subsequently recommends local features that match their preferences. However, when Bob returns to school, he and his classmates participate in an artificial intelligence project. The video platform might infer that they prefer the textual modality and recommend machine learning courses accordingly.

[0003] This example illustrates that group video recommendation systems must not only utilize multimodal information to extract user preference representations but also consider the dynamic changes in user interests to adapt to different scenarios and needs. Existing methods introduce knowledge graphs to process video information and enrich user preference representations, but these methods have shortcomings in mining multimodal features. Furthermore, some researchers use social networks and behavioral sequence analysis to update user interests, but these works do not comprehensively consider the dynamic changes in social influence and behavioral order. Therefore, group video recommendation models must comprehensively consider the different roles of multimodal features, social influence, and behavioral sequences. Summary of the Invention

[0004] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the present invention provides a group video recommendation system and method that integrates multimodal information and interest similarity, and realizes top-k recommendation that best matches the interest preferences of group members.

[0005] Technical solution: To achieve the above objectives, the technical solution adopted by this invention is as follows:

[0006] A group video recommendation method that integrates multimodal information and interest similarity includes the following steps:

[0007] Step 1, Data Preprocessing

[0008] The initial group video data is preprocessed to obtain preprocessed group video data.

[0009] Step 2: Segmentation of user interests and construction of groups

[0010] Based on the preprocessed group video data, user interest similarity is obtained, and then a social interest similarity graph is constructed. If the user interest similarity exceeds a specified threshold, an edge is established between users, and a graph neural network is used to capture user interactions.

[0011] Step 3: Fusion and Representation of Video Multimodal Information

[0012] A multimodal knowledge graph is used to fuse and represent multimodal information from videos based on social interest similarity graphs. First, multimodal information is extracted from the video, unifying the modalities of all entities to the same dimension to obtain initial embedding vectors. After obtaining the initial embedding vectors, recursive information propagation is performed on the nodes in the graph, and different weights are assigned to different nodes using an attention mechanism.

[0013] Step 4: Dynamically update user interests

[0014] An interest update module based on the BERT model is used to recommend videos that match the user's current interests. This module consists of an embedding layer and two or more identical transformer layers. The embedding layer aims to project all behaviors into a low-dimensional space, which serves as the input to the transformer layers. The transformer layers learn high-level interactions from the entire user behavior sequence based on the input, obtaining the user's final preference representation.

[0015] Step 5, Extraction of User Dynamic Interest Representations

[0016] A two-way attention mechanism is used to capture the group importance of members in the team and their proficiency in video types. At the same time, weights are assigned to different users according to the final user preference representation, so as to obtain the final representation of group preference.

[0017] Step 6, Prediction Results Representation

[0018] The MLP layer is used to make video recommendations to groups based on the final representation of group preferences in order to predict whether a group will click on a target item.

[0019] Preferably, the user interest similarity calculation in step 2 is as follows:

[0020]

[0021] Among them, s final This represents the similarity of user interests, where λ1 and λ2 represent the weights of the similarity, and u i u jt represents the percentage of each category viewed by users in the total videos. i t j This represents the proportion of time spent watching each type of video out of the total viewing time, where i represents user i in the group and j represents user j in the group.

[0022] The graph convolution of the L-layer of a graph neural network is:

[0023]

[0024] Among them, Q l Let D represent the node at level l, and let A be the adjacency matrix. i The degree matrix of the diagonal nodes, A i It is an adjacency matrix.

[0025] The user's embedding in the social similarity graph is obtained through convolution: The user's final expression is: in, This represents the final representation of a user in the social similarity graph. It is the expression of the user after the Lth convolution in the nth social similarity graph.

[0026] Preferably, step 3 uses an attention mechanism to assign different weights to different nodes:

[0027] Each attention layer consists of a propagation layer and an aggregation layer. The propagation layer obtains the aggregation representation h by calculating the weighted sum of its neighbors. N :

[0028]

[0029] Where π′(h,r,t) is the weight of the neighbor aggregation representation t of h, the weight of neighbor t is related to relation r, h represents the head entity in the knowledge graph, r represents the relation between the head entity and the tail entity in the knowledge graph, t represents the tail entity in the knowledge graph, and Ω represents the knowledge graph.

[0030]

[0031] in, Let be the transformation matrix. Indicates the initial embedding of entity t. Let represent the initial embedding of relation r, and T denote the matrix transpose. Relation r is trained, and the weights are normalized using the softmax function:

[0032]

[0033] Another component of the single propagation layer is updating the higher-order representation of node h by combining the aggregated representations of node h's neighbors with its original representation. The general form of this process is:

[0034] h′=f agg (h,h N )

[0035] Where h′ is the higher-order representation of h, f agg It is an aggregate function. The aggregate functions are as follows:

[0036] f agg =σ(W·concat(h,h) N )+b)

[0037] Here, concat(·,·) is the concatenation operation for two vectors, W∈R. 2d×d is a trainable weight matrix, and b is a trainable bias.

[0038] Higher-order information aggregation is performed by stacking a series of propagation layers to further aggregate neighbor information:

[0039]

[0040] in, This is the representation of m at level m-1. The weights are normalized, and then the representation of h in layer m is updated:

[0041]

[0042] Among them, e m It is the final output after convolution of the knowledge graph, h( m-1 ) is the representation of h in layer m-1.

[0043] Preferably, the method for dynamically updating user interests in step 4 is as follows:

[0044] Given a timestamp x = (e 1 ,e 2 ,...,e m The goal of the embedding layer is to project all behaviors into a low-dimensional space as input to the transformer layer, which is a sequence of sorted behaviors. Two types of embeddings are used to meet the group recommendation requirements: item embedding and location embedding.

[0045] Project embedding: This represents the d-dimensional item embedding output of a k-layer multimodal knowledge graph, which is the sequence of user interactions at time step t. Item embeddings are derived from the weight matrix W. V The learning process involves adding a CLS tag after the item sequence to obtain the user's current interests from the behavior sequence.

[0046] Location embedding: Add a location vector p to each item embedding. t The vector is derived from the learnable position matrix W. p Obtained from.

[0047] By overlaying the item embedding and the location embedding, a merged embedding representation e is obtained. t As input to the transformer layer:

[0048]

[0049] Among them, e t For the fusion of embedded representations, Let h be the weight matrix. t Embedded in the project, Let p be the weight matrix. t This is a position vector.

[0050] The entire sequence of user u's actions is mapped to a sequence from e0 to e t The embedded sequence.

[0051] The transformer layer learns high-level interactions throughout the entire user behavior sequence, including a multi-head self-attention layer and a location feedforward network.

[0052] The dot product in the multi-head self-attention layer is:

[0053]

[0054] Where Q, K, and V represent query, key, and value, respectively, and d is the dimension. The input to the multi-head self-attention layer is the fused representation E = {e1, ..., e2} of the embedding layer. t Then, it is transformed into three matrices through linear projection:

[0055] MH(E)=Concat(head1, head2,..,head n W O

[0056]

[0057] head h =Attention(Q) (h) K (h) V (h) )

[0058] The weight matrix for each head It is a learnable parameter, W O It is the output projection matrix, MH(E) is multi-head self-attention, and head h The h-th head represents the multi-head self-attention.

[0059] Following the multi-head self-attention layer is a feedforward network, which further increases the nonlinearity of the model, defined as follows:

[0060] S=Concatrow(E+Dropout(MH(E)))

[0061] F=LayerNorm(S+Dropout(Re Lu(SQ (1) +b (1) )Q (2) +b (2) )

[0062] Where S represents the intermediate layer representation, F represents the output layer representation, and W represents the intermediate layer representation. (1) b (1) W (2) b (2) For all locations, learnable parameters are shared. Concatrow is a row-connected vector technique, LayerNorm is a standard normalization layer, Dropout is a random deactivation regularization technique, and ReLU is the activation function.

[0063] Stacking multiple transformers yields the final output H of the behavioral sequence. N The last position It is fed into the intent layer to obtain the point β that the user is currently interested in. k :

[0064]

[0065] Among them, W l and b l Here are the learnable parameters for the intent layer, and LeakyReLU is the activation function. This is the first vector of user interest after processing through the neural network.

[0066] The final expression of user preferences is obtained: l k For the user's final representation, For the final representation of users in the social similarity graph, β k This represents the user's final representation after dynamic updates.

[0067] Preferably, the method for obtaining the final representation of group preferences in step 5 is as follows:

[0068] Two-way attention mechanisms include group attention mechanisms and knowledge attention mechanisms.

[0069] Group attention mechanism: reflects the relative status and contribution of each member in the group.

[0070]

[0071] in, The output is for group attention, where G is the set of groups, k is the user k in the group, and g is the group attention output. l For a group, n k This represents the user's final preference under the activation function. For X a The transpose of W a ∈R d×d ,X a ∈R d b a ∈R d It consists of learnable weight matrices and biases.

[0072] Knowledge Attention Mechanism: Reflects each member's understanding and mastery of video types.

[0073]

[0074] in, For knowledge attention output, e k For embedding vectors of the target item, l k G represents l The preference vector of each member in the group. This represents the final representation of other users within the group.

[0075] Through the attention mechanism, the final representation of group preference is as follows:

[0076]

[0077] Among them, g l It is based on group weight and knowledge weight A group representation formed by aggregating the preferences of its members, l k is the individual's final preference, g is the group, and softmax is the normalization function.

[0078] Preferably, the method for representing the prediction result in step 6 is as follows:

[0079] The entire process involves connecting groups and item embeddings, which are then fed into the MLP layer:

[0080]

[0081] in, This is the final output of the MLP layer, where MLP stands for MLP layer. It's a join operation, g l It is based on group weight and knowledge weight A group representation formed by aggregating the preferences of its members, e k Embed the vector for the target item.

[0082] The task is modeled as a binary classification problem to predict whether a group will click on the target item.

[0083]

[0084] in, G represents l Group click on candidate items e k The probability, W o and b o This represents the weights and biases of the prediction layer, and tanh is the activation function.

[0085] The final expression is:

[0086]

[0087] in, Q l Let e ​​represent the node at level l in the social similarity graph. k Embed the vector for the target item.

[0088] Preferably, it also includes loss optimization using a pairwise ranking loss function based on MSE:

[0089]

[0090] Among them, L MSE Let MSE be the loss function, n be the number of users, l be the l-th group, k be the k-th item, and O represent the training set. and g l Group in e k The actual score and predicted score for each item.

[0091] Preferably, the preprocessed group video data includes a user set U = {u1, u2, ..., u...} K}, where K is the number of users, and the item set I = {i1, i2, ..., i N}, where N is the number of items, and the group set G = {g1, g2, ..., g} L}, where L is the number of groups, and Y∈R is the user-item interaction. K×N If user u a and Project i b If interaction exists, then Y(a,b) = 1; otherwise, Y(a,b) = 0. Similarly, user-group interaction is R ∈ R K×L .

[0092] A group video recommendation system integrating multimodal information and interest similarity, employing the aforementioned group video recommendation method integrating multimodal information and interest similarity, includes a data preprocessing unit, a user interest segmentation and group construction unit, a video multimodal information fusion and representation unit, a dynamic user interest update unit, a user dynamic interest representation extraction unit, and a prediction result representation unit, wherein:

[0093] The data preprocessing unit is used to preprocess the initial group video data to obtain preprocessed group video data.

[0094] The user interest segmentation and group construction unit is used to obtain user interest similarity based on preprocessed group video data, and then construct a social interest similarity graph. If the user interest similarity exceeds a specified threshold, an edge is established between users, and a graph neural network is used to capture user interactions.

[0095] The video multimodal information fusion and representation unit is used to fuse and represent video multimodal information based on a social interest similarity graph using a multimodal knowledge graph. First, the multimodal information of the video is extracted, unifying the modalities of all entities to the same dimension to obtain an initial embedding vector. After obtaining the initial embedding vector, recursive information propagation is performed on the nodes in the graph, and an attention mechanism is used to assign different weights to different nodes.

[0096] The dynamically updated user interest unit is used to recommend videos that match the user's current interests using a BERT-based interest update module. This BERT-based interest update module includes an embedding layer and two or more identical transformer layers. The embedding layer aims to project all behaviors into a low-dimensional space, which serves as the input to the transformer layers. The transformer layers learn the high-level interactions of the entire user behavior sequence based on the input, obtaining the user's final preference representation.

[0097] The user dynamic interest representation extraction unit is used to capture the group importance of members in the team and their proficiency in video types using a two-way attention mechanism. At the same time, it assigns weights to different users according to the user's final preference representation, thereby obtaining the final representation of group preferences.

[0098] The prediction result representation unit uses an MLP layer to make video recommendations to the group based on the final representation of the group's preferences, in order to predict whether the group will click on the target item.

[0099] Compared with the prior art, the present invention has the following advantages:

[0100] Compared to existing technologies, this invention offers higher accuracy and greater interpretability. Its unique feature lies in combining knowledge graphs and the BERT model to comprehensively mine multimodal information from videos while dynamically updating user interests, thereby improving the accuracy of video recommendation systems. This novel approach not only recommends videos to groups more accurately but also understands and caters to users' personalized interests in greater detail, providing a richer and more engaging viewing experience. Attached Figure Description

[0101] Figure 1 This is an overall schematic diagram of a group video recommendation method that integrates multimodal information and interest similarity according to the present invention. Detailed Implementation

[0102] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0103] A group video recommendation method integrating multimodal information and interest similarity is proposed. It divides users into groups based on their interests and leverages social networks to learn user interaction behaviors. It mines multimodal information from videos to enrich semantic information, thereby improving the information richness of the recommendation system. It analyzes users' historical behavior sequences to obtain their latest interests, enabling a more accurate understanding of user needs. A bidirectional attention mechanism is designed to aggregate the preferences of group members, thereby obtaining embedded representations of groups and tasks. Finally, a multilayer perceptron is used to generate top-K task recommendations for the groups. Figure 1 As shown, the specific steps include:

[0104] Step 1, Data Preprocessing

[0105] The initial group video data is preprocessed to obtain preprocessed group video data. The preprocessed group video data includes a user set U = {u1, u2, ..., u...}. K}, where K is the number of users, and the item set I = {i1, i2, ..., i N}, where N is the number of items, and the group set G = {g1, g2, ..., g} L}, where L is the number of groups, and Y∈R is the user-item interaction. K×N If user u a and Project i b If interaction exists, then Y(a,b) = 1; otherwise, Y(a,b) = 0. Similarly, user-group interaction is R ∈ R K×L .

[0106] Step 2: Segmentation of user interests and construction of groups

[0107] User interest similarity is obtained based on the video types and viewing durations of users in the preprocessed group video data. A social interest similarity graph is constructed based on the user interest similarity. If the user interest similarity exceeds a specified threshold, edges are established between users, and a graph neural network is used to capture user interactions.

[0108] This invention groups users based on their interest similarity, arguing that user interest similarity is determined by two factors: genre preference and effective viewing time. The former is calculated based on the type of video the user watches, while the latter is calculated based on the viewing time. This method calculates these two aspects using cosine similarity and then weights them to obtain the final interest similarity score. Specifically, this method uses cosine similarity to calculate both aspects separately:

[0109]

[0110] Where u i ∈U and u j ∈U represents the percentage of each category watched by a user in the total videos, i represents user i in the group, and j represents user j in the group.

[0111]

[0112] Where t i and t j This indicates the percentage of total viewing time spent on each type of video.

[0113] To obtain the final interest similarity between user a and user b, the user interest similarity based on type preference and effective duration is calculated as follows:

[0114]

[0115] Among them, s final This represents the similarity of user interests, where λ1 and λ2 represent the weights of the similarity, 0 ≤ λ1, λ2 ≤ 1, and λ1 + λ2 = 1. i u j t represents the percentage of each category viewed by users in the total videos. i t j This represents the proportion of time spent watching each type of video out of the total viewing time, where i represents user i in the group and j represents user j in the group.

[0116] After obtaining the final similarity representation, a social interest similarity graph is constructed. If a specified threshold is exceeded, edges are established between users, and a graph neural network is used to capture user interactions. According to the graph convolutional network, the graph convolution of the L-layer graph neural network is:

[0117]

[0118] Among them, Q l Let D represent the node at level l, and let A be the adjacency matrix. i The degree matrix of the diagonal nodes, A i It is an adjacency matrix.

[0119] The user's embedding in the social similarity graph is obtained through convolution: The user's final expression is: in, This represents the final representation of a user in the social similarity graph. It is the expression of the user after the Lth convolution in the nth social similarity graph.

[0120] Step 3: Fusion and Representation of Video Multimodal Information

[0121] To effectively capture deep relationships and semantic information in video content, a multimodal knowledge graph is introduced. This invention fuses and represents multimodal information from the video based on a social interest similarity graph. To fully construct the knowledge graph, various modeling methods are employed to convert multimodal data into vectors. First, multimodal information from the video is extracted. For image modalities, a pre-trained CNN model aggregates the features from the fc7 layer of each frame into a feature vector. For text modalities, a multi-layer bidirectional Transformer encoder extracts semantic features. Finally, the modalities of all entities are unified to the same dimension to obtain the initial embedding vector.

[0122] After obtaining the initial embedding vector, recursive information propagation needs to be performed on the nodes in the graph, and different weights need to be assigned to different nodes using an attention mechanism. Each attention layer includes a propagation layer and an aggregation layer. The propagation layer obtains the aggregated representation h by calculating the weighted sum of its neighbors. N :

[0123]

[0124] Where π′(h,r,t) is the weight of the neighbor aggregation representation t of h, the weight of neighbor t is related to relation r, h represents the head entity in the knowledge graph, r represents the relation between the head entity and the tail entity in the knowledge graph, t represents the tail entity in the knowledge graph, and Ω represents the knowledge graph.

[0125] Achieved through an attention mechanism:

[0126]

[0127] in, Let be the transformation matrix. Indicates the initial embedding of entity t. Let represent the initial embedding of relation r, and T denote the matrix transpose. If relation r can be trained, the weights are normalized using the softmax function:

[0128]

[0129] Another component of the single propagation layer is updating the higher-order representation of node h by combining the aggregated representations of node h's neighbors with its original representation. The general form of this process is:

[0130] h′=f agg (h,h N )

[0131] Where h′ is the higher-order representation of h, f agg It is an aggregate function. The aggregate functions are as follows:

[0132] f agg =σ(W·concat(h,h) N )+b)

[0133] Here, concat(·,·) is the concatenation operation for two vectors, W∈R. 2d×d is a trainable weight matrix, and b is a trainable bias.

[0134] However, the ability of a single propagation layer to propagate information along the edge is limited. To overcome this drawback, another embodiment of the invention performs higher-order information aggregation by stacking a series of propagation layers to further aggregate neighbor information:

[0135]

[0136] in, This is the representation of m at level m-1. The weights are normalized, and then the representation of h in layer m is updated:

[0137]

[0138] Among them, e m It is the final output after convolution of the knowledge graph, h (m-1) It is the representation of h in layer m-1.

[0139] Step 4: Dynamically update user interests

[0140] To recommend videos that match the user's current interests, an interest update module based on the BERT model is used. The interest update module based on the BERT model includes an embedding layer and two or more identical transformer layers, which can learn higher-order interactions from behavioral sequences.

[0141] Given a timestamp x = (e 1 ,e 2 ,...,e m The goal of the embedding layer is to project all behaviors into a low-dimensional space as input to the transformer layer, which is a sequence of sorted behaviors. Two types of embeddings are used to meet the group recommendation requirements: item embedding and location embedding.

[0142] Project embedding: This represents the d-dimensional item embedding output of a k-layer multimodal knowledge graph, which is the sequence of user interactions at time step t. Item embeddings are derived from the weight matrix W. V The learning process involves adding a CLS tag after the item sequence to obtain the user's current interests from the behavior sequence.

[0143] Location embedding: To better capture contextual information, a location vector p is added to each item embedding. t The vector is derived from the learnable position matrix W. p Obtained from.

[0144] By overlaying the item embedding and the location embedding, a merged embedding representation e is obtained. t As input to the transformer layer:

[0145]

[0146] Among them, e t For the fusion of embedded representations, Let h be the weight matrix. t Embedded in the project, Let p be the weight matrix. t This is a position vector.

[0147] In this way, the entire sequence of user u's actions is mapped to a sequence from e0 to e t The embedded sequence.

[0148] The transformer layer learns high-level interactions throughout the entire user behavior sequence, including a multi-head self-attention layer and a location feedforward network.

[0149] The dot product in the multi-head self-attention layer is:

[0150]

[0151] Where Q, K, and V represent query, key, and value, respectively, and d is the dimension. The input to the multi-head self-attention layer is the fused representation E = {e1, ..., e2} of the embedding layer. t Then, it is transformed into three matrices through linear projection:

[0152] MH(E)=Concat(head1, head2, .., head n W O

[0153]

[0154] head h =Attention(Q) (h) K (h) V (h) )

[0155] The weight matrix for each head It is a learnable parameter, W O It is the output projection matrix, MH(E) is multi-head self-attention, and head h The h-th head represents the multi-head self-attention.

[0156] Following the multi-head self-attention layer is a feedforward network, which further increases the nonlinearity of the model, defined as follows:

[0157] S=Concatrow(E+Dropout(MH(E)))

[0158] F=LayerNorm(S+Dropout(ReLu(SQ (1) +b (1) )Q (2) )+b (2) )

[0159] Where S represents the intermediate layer representation, F represents the output layer representation, and W( 1 ), b( 1 ), W( 2 ),W( 2 ) represents the learnable parameters shared across all locations, Concatrow is a row-connected vector technique, LayerNorm is a standard normalization layer, Dropout is a random deactivation regularization technique, and ReLU is the activation function.

[0160] To better learn the evolution of interest, another embodiment of the present invention stacks multiple transformers to obtain the final output H of the behavioral sequence. N The last position It is fed into the intent layer to obtain the point β that the user is currently interested in. k :

[0161]

[0162] Among them, W l and b lHere are the learnable parameters for the intent layer, and Leaky ReLU is the activation function. This is the first vector of user interest after processing through the neural network.

[0163] The final expression of user preferences is obtained: l k For the user's final representation, For the final representation of users in the social similarity graph, β k This represents the user's final representation after dynamic updates.

[0164] Step 5, Extraction of User Dynamic Interest Representations

[0165] After obtaining the final user preference representation, weights need to be assigned to different users to obtain a group preference representation. This embodiment of the invention assumes that the weight of a member depends on two factors: their group importance within the team and their proficiency in video types. Therefore, this embodiment of the invention introduces a bidirectional attention mechanism to capture these two factors, specifically called a group attention mechanism and a knowledge attention mechanism. That is, a bidirectional attention mechanism is used to capture a member's group importance within the team and their proficiency in video types, thereby assigning weights to different users. The bidirectional attention mechanism includes both a group attention mechanism and a knowledge attention mechanism.

[0166] Group attention mechanisms reflect the relative status and contribution of each member in a group, and thus... The knowledge attention mechanism reflects each member's understanding and mastery of the video type, and obtains... By assigning weights to group members using an attention mechanism, the final representation g of group preference is obtained. l .

[0167] Group attention mechanism: reflects the relative status and contribution of each member in the group.

[0168]

[0169] in, The output is for group attention, where G is the set of groups, k is the user k in the group, and g is the group attention output. l For a group, n k This represents the user's final preference under the activation function. For X a The transpose of W a ∈R d×d ,X a ∈R d b a ∈R d It consists of learnable weight matrices and biases.

[0170] Knowledge Attention Mechanism: Reflects each member's understanding and mastery of video types.

[0171]

[0172] in, For knowledge attention output, e k For embedding vectors of the target item, l k G represents l The preference vector of each member in the group. This represents the final representation of other users within the group.

[0173] Through the attention mechanism, the final representation of group preference is as follows:

[0174]

[0175] Among them, g l It is based on group weight and knowledge weight A group representation formed by aggregating the preferences of its members, l k is the individual's final preference, g is the group, and softmax is the normalization function.

[0176] Step 6, Prediction Results Representation

[0177] To make video recommendations to groups more accurate, an MLP layer is used for video recommendations to groups. The entire process includes joining groups and embedding items, which are then fed into the MLP layer:

[0178]

[0179] in, This is the final output of the MLP layer, where MLP stands for MLP layer. It's a join operation, g l It is based on group weight and knowledge weight A group representation formed by aggregating the preferences of its members, e k Embed the vector for the target item.

[0180] The task is modeled as a binary classification problem to predict whether a group will click on the target item.

[0181]

[0182] in, G represents l Group click on candidate items e k The probability, W o and b o This represents the weights and biases of the prediction layer, and tanh is the activation function.

[0183] The final expression is:

[0184]

[0185] in, Q l Let e ​​represent the node at level l in the social similarity graph. k Embed the vector for the target item.

[0186] Step 7, Loss Optimization

[0187] In another embodiment of the present invention, a pairwise ranking loss function based on MSE is designed for optimization:

[0188]

[0189] Among them, I MSE Let MSE be the loss function, n be the number of users, l be the l-th group, k be the k-th item, and O represent the training set. and g l Group in e k The actual score and predicted score for each item.

[0190] A group video recommendation system integrating multimodal information and interest similarity, employing the aforementioned group video recommendation method integrating multimodal information and interest similarity, includes a data preprocessing unit, a user interest segmentation and group construction unit, a video multimodal information fusion and representation unit, a dynamic user interest update unit, a user dynamic interest representation extraction unit, and a prediction result representation unit, wherein:

[0191] The data preprocessing unit is used to preprocess the initial group video data to obtain preprocessed group video data.

[0192] The user interest segmentation and group construction unit is used to obtain user interest similarity based on preprocessed group video data, and then construct a social interest similarity graph. If the user interest similarity exceeds a specified threshold, an edge is established between users, and a graph neural network is used to capture user interactions.

[0193] The video multimodal information fusion and representation unit is used to fuse and represent video multimodal information based on a social interest similarity graph using a multimodal knowledge graph. First, the multimodal information of the video is extracted, unifying the modalities of all entities to the same dimension to obtain an initial embedding vector. After obtaining the initial embedding vector, recursive information propagation is performed on the nodes in the graph, and an attention mechanism is used to assign different weights to different nodes.

[0194] The dynamically updated user interest unit is used to recommend videos that match the user's current interests using a BERT-based interest update module. This BERT-based interest update module includes an embedding layer and two or more identical transformer layers. The embedding layer aims to project all behaviors into a low-dimensional space, which serves as the input to the transformer layers. The transformer layers learn the high-level interactions of the entire user behavior sequence based on the input, obtaining the user's final preference representation.

[0195] The user dynamic interest representation extraction unit is used to capture the group importance of members in the team and their proficiency in video types using a two-way attention mechanism. At the same time, it assigns weights to different users according to the user's final preference representation, thereby obtaining the final representation of group preferences.

[0196] The prediction result representation unit uses an MLP layer to make video recommendations to the group based on the final representation of the group's preferences, in order to predict whether the group will click on the target item.

[0197] This invention improves the accuracy and interpretability of video recommendation systems by combining knowledge graphs with the BERT model to mine multimodal information from videos, dynamically update user interests, and achieve personalized recommendations.

[0198] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A group video recommendation method fusing multi-modal information and interest similarity, characterized in that, Includes the following steps: Step 1, Data Preprocessing; The initial group video data is preprocessed to obtain preprocessed group video data. Step 2: Segmentation of user interests and construction of groups; Based on the preprocessed group video data, user interest similarity is obtained, and then a social interest similarity graph is constructed. If the user interest similarity exceeds a specified threshold, an edge is established between users, and a graph neural network is used to capture user interactions. Step 3: Fusion and representation of video multimodal information; Multimodal knowledge graphs are used to fuse and represent multimodal information from videos based on social interest similarity graphs. First, multimodal information from the video is extracted, and the modalities of all entities are unified to the same dimension to obtain the initial embedding vector. After obtaining the initial embedding vector, recursively propagate information to the nodes in the graph and uses an attention mechanism to assign different weights to different nodes. Step 4: Dynamically update user interests; The interest update module based on the BERT model recommends videos that match the user's current interests. The interest update module based on the BERT model includes an embedding layer and two or more identical transformer layers. The goal of the embedding layer is to project all behaviors into a low-dimensional space as the input of the transformer layer. The transformer layer learns the high-level interactions of the entire user behavior sequence based on the input to obtain the user's final preference representation. Step 5, Extraction of user dynamic interest representation; A two-way attention mechanism is used to capture the group importance of members in the team and their proficiency in video types. At the same time, weights are assigned to different users according to the final user preference representation, so as to obtain the final representation of group preference. Step 6, Representation of prediction results; The MLP layer is used to make video recommendations to groups based on the final representation of group preferences in order to predict whether a group will click on a target item. 2.The group video recommendation method of fusing multi-modal information and interest similarity according to claim 1, characterized in that, The user interest similarity is calculated in step 2 as follows: in, Indicates the similarity of user interests. Weights representing similarity This represents the percentage of each category viewed by users in the total number of videos. This indicates the percentage of total viewing time spent on each type of video. Users in the group , Users in the group ; Graph Neural Networks The graph convolution of the layer is: in, Indicates the first Layer nodes, Adjacency matrix The degree matrix of the diagonal nodes, It is an adjacency matrix; The user's embedding in the social similarity graph is obtained through convolution: The user ultimately expressed it as: ,in, This represents the final representation of a user in the social similarity graph. Is the user in the The first in the social similarity graph The expression after convolution.

3. The group video recommendation method integrating multimodal information and interest similarity according to claim 2, characterized in that, Step 3 utilizes an attention mechanism to assign different weights to different nodes: Each attention layer consists of a propagation layer and an aggregation layer; the propagation layer obtains the aggregation representation by calculating the weighted sum of its neighbors. : in, For calculation Neighbor aggregation representation Weights, neighbors Weights and Relationships Related, This represents the head entity in the knowledge graph. This represents the relationship between head entities and tail entities in a knowledge graph. This represents the tail entity in a knowledge graph. Representing a knowledge graph; ； in, Let be the transformation matrix. Representing entities The initial embedding, Representing relations The initial embedding, Represents matrix transpose; relation Trained, Use The function normalizes the weights: ; Another component of the single propagation layer is by using nodes The neighbor aggregation representation is combined with its original representation to update the node. The higher-order representation of this process is in the general form of: in yes The higher-order representation of , These are aggregate functions; the aggregate functions are as follows: in, It is a concatenation operation of two vectors. It is a trainable weight matrix. It is a trainable bias; Higher-order information aggregation is performed by stacking a series of propagation layers to further aggregate neighbor information: in, yes exist Layer representation, It's the normalized weights, then updated. exist Layer representation: in, It is the final output after convolution of the knowledge graph. yes exist Layer representation.

4. The group video recommendation method integrating multimodal information and interest similarity according to claim 3, characterized in that, The method for dynamically updating user interests in step 4 is as follows: Given a timestamp The goal of the embedding layer is to project all behaviors into a low-dimensional space as input to the transformer layer, which is used to sort the behavior sequence. Two types of embeddings are used to meet the group recommendation requirements: item embedding and location embedding. Project embedding: express Multimodal knowledge graph The project embeds the output, which is the user's time step. The sequence that interacts with it; item embeddings are from the weight matrix Learned from the process, add a CLS tag after the item sequence to obtain the user's current interests from the behavior sequence; Location embedding: Add a location vector to each item embedding. The vector is derived from the learnable position matrix. Obtained; By overlaying the item embedding and the location embedding, a merged embedding representation is obtained. As input to the transformer layer: in, For the fusion of embedded representations, This is the weight matrix. Embedded in the project, This is the weight matrix. It is a position vector; user The entire sequence of actions is mapped to a sequence from arrive Embedded sequences; The transformer layer learns high-level interactions throughout the entire user behavior sequence, including a multi-head self-attention layer and a positional feedforward network; The dot product in the multi-head self-attention layer is: in, These represent the query, key, and value, respectively. It is a dimension; the input to the multi-head self-attention layer is the fused representation of the embedding layer. Then, it is transformed into three matrices through linear projection: The weight matrix for each head These are learnable parameters. It is the output projection matrix. For the sake of the bulls' self-attention, For the first time, the attention of the bulls Size; Following the multi-head self-attention layer is a feedforward network, which further increases the nonlinearity of the model, defined as follows: in, This is represented as an intermediate layer. For output layer representation, Learnable parameters shared across all locations This is a row-join vector technique. For standard normalization layer, Regularization techniques for random inactivation. For activation functions; Stacking multiple transformers yields the final output of the behavioral sequence. The last position It is fed into the intent layer to obtain the user's current point of interest. : in, and These are the learnable parameters for the intent layer. For activation function, This is the first vector of user interests after processing through the neural network; The final expression of user preferences is obtained: , For the user's final representation, This is the final representation of users in the social similarity graph. This represents the user's final representation after dynamic updates.

5. The group video recommendation method based on the fusion of multimodal information and interest similarity according to claim 4, characterized in that, The method for obtaining the final representation of group preferences in step 5: Two-way attention mechanisms include group attention mechanisms and knowledge attention mechanisms; Group attention mechanism: reflects the relative status and contribution of each member in the group. in, For group attention output, For group collection, For users in the group , For groups, This represents the user's final preference under the activation function. for transpose, , It consists of learnable weight matrices and biases; Knowledge Attention Mechanism: Reflects each member's understanding and mastery of video types. in, For knowledge attention output, Embed a vector for the target item. express The preference vector of each member in the group. This is the final representation for other users within the group; Through the attention mechanism, the final representation of group preference is as follows: in, It is based on group weight and knowledge weight A group representation that aggregates the preferences of its members. It is an individual's final preference. For groups, This is the normalization function.

6. The group video recommendation method based on the fusion of multimodal information and interest similarity according to claim 5, characterized in that, The prediction results in step 6 are presented as follows: The entire process involves connecting groups and item embeddings, which are then fed into the MLP layer: in, This is the final output of the MLP layer. Indicates the MLP layer. It's a connection operation. It is based on group weight and knowledge weight A group representation that aggregates the preferences of its members. Embed a vector for the target item; The task is modeled as a binary classification problem to predict whether a group will click on the target item. in, express Group click candidate items The probability, and This represents the weights and biases of the prediction layer. It is an activation function; The final expression is: in, , Indicating the first in the social similarity graph Layer nodes, Embed the vector for the target item.

7. The group video recommendation method based on the fusion of multimodal information and interest similarity according to claim 6, characterized in that, It also includes loss optimization based on the pairwise ranking loss function of MSE: in, The MSE loss function is... For the number of users, For the first One group, For the first One project, Denotes the training set, where and They are respectively group in The actual score and predicted score for each item.

8. The group video recommendation method based on the fusion of multimodal information and interest similarity according to claim 7, characterized in that, The preprocessed group video data includes user sets. , It refers to the number of users and the collection of projects. , It refers to the number of projects and the collection of groups. , It refers to the number of groups and user-project interactions. If the user and projects If there is interaction, then ,otherwise Similarly, user-group interaction is .

9. A group video recommendation system that integrates multimodal information and interest similarity, characterized in that, The group video recommendation method based on the fusion of multimodal information and interest similarity as described in claim 1 includes a data preprocessing unit, a user interest segmentation and group construction unit, a video multimodal information fusion and representation unit, a dynamic user interest update unit, a user dynamic interest representation extraction unit, and a prediction result representation unit, wherein: The data preprocessing unit is used to preprocess the initial group video data to obtain preprocessed group video data; The user interest segmentation and group construction unit is used to obtain user interest similarity based on preprocessed group video data, and then construct a social interest similarity graph. If the user interest similarity exceeds a specified threshold, an edge is established between users, and a graph neural network is used to capture user interactions. The video multimodal information fusion and representation unit is used to fuse and represent video multimodal information based on social interest similarity graphs using multimodal knowledge graphs. First, the multimodal information of the video is extracted, and the modalities of all entities are unified to the same dimension to obtain an initial embedding vector. After obtaining the initial embedding vector, recursive information propagation is performed on the nodes in the graph, and different weights are assigned to different nodes using an attention mechanism. The dynamically updated user interest unit is used to recommend videos that match the user's current interests using a BERT-based interest update module. The BERT-based interest update module includes an embedding layer and two or more identical transformer layers. The goal of the embedding layer is to project all behaviors into a low-dimensional space as the input to the transformer layer. The transformer layer learns the high-level interactions of the entire user behavior sequence based on the input to obtain the user's final preference representation. The user dynamic interest representation extraction unit is used to capture the group importance of members in the team and their proficiency in video types using a two-way attention mechanism. At the same time, it assigns weights to different users according to the user's final preference representation, thereby obtaining the final representation of group preferences. The prediction result representation unit uses an MLP layer to make video recommendations to the group based on the final representation of the group's preferences, in order to predict whether the group will click on the target item.

Images