Cross-view sequence recommendation method based on multiple modes and multiple behaviors

Through the multimodal multi-behavior cross-view sequence recommendation method, combined with de-entanglement learning and multimodal attention mechanism, the problem of users' preferences for different modalities and differences in modal features is solved, and the accuracy of the recommendation system and the user's Satisfaction, effectively handles user multi-behavior interaction sequences, extracts user dynamic interest characteristics, and enhances recommendation performance.

CN119006098APending Publication Date: 2024-11-22CHONGQING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411049165.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing multimodal recommendation method fails to effectively consider the user's preference for different modalities and the characteristic differences between modalities, resulting in unsatisfactory recommendation results, and complex dynamic differences between users and products in user historical interaction sequences. The structured relationship is difficult to explore, resulting in low information utilization and inaccurate recommendation results.

Method used

Using a multimodal multi-behavior cross-view sequence recommendation method, the preprocessing module, multimodal preference extraction module and multi-behavior preference extraction module are used to capture the user in each factor through the preprocessing module, multimodal preference extraction module and multi-behavior preference extraction module, combined with de-entanglement learning and multimodal attention mechanism. The modal preferences on it and learn the synergistic relationship between multimodal and multi-behavior through the cross-view learning framework, calculate the loss function to update the model parameters.

Benefits of technology

The accuracy and user satisfaction of the recommendation system are improved, and the weights of different modal features can better reflect users' personalized preferences, effectively handle user multi-behavior interaction sequences, extract user's dynamic interest characteristics, and enhance recommendation performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006098A_ABST
    Figure CN119006098A_ABST
Patent Text Reader

Abstract

The invention belongs to the field of sequence recommendation, and relates to a multi-modal and multi-behavior-based cross-view sequence recommendation method, which comprises the following steps: a training process of a sequence recommendation model comprises the following steps: acquiring a multi-modal e-commerce platform sequence, and inputting the multi-modal e-commerce platform sequence into a preprocessing module to obtain a multi-modal feature embedding vector and a multi-behavior perception interaction sequence; inputting the multi-modal feature embedded vector into a multi-modal preference extraction module to obtain independent feature representations and multi-modal preferences of different factors in multiple modals; inputting the multi-behavior perception interaction sequence into a multi-behavior preference extraction module to obtain multi-behavior preferences; inputting independent feature representations of different factors in multiple modes, multi-mode preferences and multi-behavior preferences into a cross-view learning module to calculate a loss function value, updating model parameters according to the loss function value, and finishing model training when the loss function value is minimum; according to the method, a cross-view learning framework is adopted to learn the collaborative relationship between multiple modes and multiple behaviors, the recommendation performance is enhanced through bidirectional optimization, and the recommendation accuracy is further improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of sequence recommendation and relates to a cross-view sequence recommendation method based on multi-modal and multi-behavior. Background Art

[0002] With the development of Internet technology, recommendation systems have become an indispensable part of our lives. It predicts user preferences by analyzing users' historical interaction records, thereby recommending the most likely and relevant items to users. In recent years, with the rise of deep learning technology, a large number of methods have emerged to solve the problem of sequential recommendation, which use different neural network technologies. For example, the model based on recurrent neural network encodes sequential information from the sequence of interaction items of each user. Currently, sequence recommendation is one of the important research directions of recommendation systems. It predicts users' future behavior and preferences by analyzing the historical behavior sequence of users. However, the existing sequence recommendation system requires a large number of user-item interaction records to achieve more accurate recommendation performance. For users with fewer interaction records, the model effect is significantly reduced, resulting in data sparsity and cold start problems.

[0003] To solve these problems, researchers have introduced multimodal features (such as text, vision, etc.) and multi-behavior features (such as clicks, browsing, purchases, etc.) to improve the accuracy of recommendations. Multimodal recommendations use multiple data sources (such as text, images, audio) to supplement the interaction information between users and items, thereby improving the accuracy of recommendations. The multi-behavior sequence recommendation system not only considers the user's single behavior, but also integrates multiple behavior types (such as clicks, favorites, purchases, etc.) to more comprehensively understand user preferences.

[0004] Therefore, in e-commerce recommendation systems, combining multimodal recommendation and multi-behavior sequence modeling can more comprehensively capture user preferences and improve recommendation accuracy and user satisfaction.

[0005] In summary, although many scholars have conducted extensive research on e-commerce recommendation models and achieved quite good results, there are still some challenges:

[0006] 1. Existing multimodal recommendation methods do not consider users' preferences for different modalities and the feature differences between modalities when using different modalities, resulting in unsatisfactory recommendation results.

[0007] 2. There is a complex dynamic heterogeneous relationship structure between users and products in the user historical interaction sequence of sequential recommendation (such as page browsing, adding to favorites, purchasing, etc.). It is inherently difficult to explore the relationship dependencies behind multi-behavior user-product interactions.

[0008] 3. The feature mixing between the user multi-behavior interaction sequence features and the corresponding multimodal features causes problems of feature coupling and poor feature recognition, resulting in low information utilization and reduced accuracy of recommendation results. Summary of the invention

[0009] In order to solve the above problems in the prior art, the present invention adopts a cross-view sequence recommendation method based on multimodal and multi-behavior, including: obtaining a multimodal e-commerce platform sequence, inputting the multimodal e-commerce platform sequence into a trained sequence recommendation model, and obtaining recommended products; the sequence recommendation model includes: a preprocessing module, a multimodal preference extraction module, a multi-behavior preference extraction module and a cross-view learning module;

[0010] The training process of the sequence recommendation model includes:

[0011] S1. Obtain a multimodal e-commerce platform sequence, input the multimodal e-commerce platform sequence into a preprocessing module, and obtain a multimodal feature embedding vector and a multi-behavior perception interaction sequence;

[0012] S2, input the multimodal feature embedding vector into the multimodal preference extraction module to obtain independent feature representations and multimodal preferences of different factors in the multimodality;

[0013] S3, inputting the multi-behavior perception interaction sequence into the multi-behavior preference extraction module to obtain the multi-behavior preference;

[0014] S4. Input the independent feature representations of different factors in multimodality, multimodal preferences and multi-behavior preferences into the cross-view learning module to calculate the loss function value, update the model parameters according to the loss function value, and complete the model training when the loss function value is minimized.

[0015] The preprocessing module processes the multimodal e-commerce platform sequence by: extracting the initial text feature F from the multimodal e-commerce platform sequence using the BERT model text , extracting initial visual features F from multimodal e-commerce platform sequences through CNNs model vi , for the initial text feature F text and the initial visual features F vi Perform feature embedding respectively to obtain the text feature embedding vector E text and the visual feature embedding vector E vi ; Randomly initialize each product ID in the multimodal e-commerce platform sequence to obtain the ID feature embedding vector E ID .

[0016] The multimodal preference extraction module processes the multimodal feature embedding vector including:

[0017] S21. Process the multimodal feature embedding vector using disentanglement learning technology to obtain independent feature representations of different factors in the multimodality;

[0018] S22. Use the multimodal attention mechanism to calculate the user's modal preference on each factor based on the independent feature representations of different factors in the multimodality.

[0019] Calculating the user's modality preference on each factor involves:

[0020] According to the independent feature representations of different factors in multimodality, the user u's attention degree to the kth factor of each modality is calculated. According to the attention degree, the user's attention weight to the kth factor of each modality is calculated. According to the attention weight, the independent feature representations of different factors in multimodality are weighted and summed to obtain the user's modal preference on each factor.

[0021] The multi-behavior preference extraction module includes: behavior-aware context embedding layer, item order dependency encoding layer, multi-behavior mode aggregation layer, cross-type aggregation layer, and graph attention network;

[0022] The multi-behavior preference extraction module processes the multi-behavior perception interaction sequence including:

[0023] S31, inputting multiple behavior-aware interaction sequences into a behavior-aware context embedding layer to obtain behavior-aware product embedding;

[0024] S32, embedding the behavior-aware commodity into the input item sequence dependency encoding layer to obtain behavior-aware sequence-dependent commodity embedding;

[0025] S33, embedding the products that are dependent on the behavior perception order into the multi-behavior mode aggregation layer to obtain the user's preference for each product under each behavior type;

[0026] S34, input the user's preference for each product into the cross-type aggregation layer to obtain the cross-time embedding of each product;

[0027] S35. Input the cross-time embedding of each product into the graph attention network to obtain multi-behavioral preferences.

[0028] The multi-behavior perception interaction sequence includes: u =[(o u,1 ,p u,1 ,t u,1 ),(o u,2 ,p u,2 ,t u,2 ),…(o u,I ,p u,I ,t u,I )], where I is the length of the multi-behavior perception interaction sequence, o u,irepresents the representation of the i-th item in the interaction sequence, p u,i The embedding vector representing the behavior type of user u interacting with the i-th item, t u,i represents the interaction time between user u and the i-th product;

[0029] The behavior-aware context embedding layer processes the multi-behavior-aware interaction sequence by encoding the position of the i-th item in the multi-behavior-aware interaction sequence, obtaining the learnable position embedding of the i-th item, and converting the representation of the i-th item into u,i , learnable position embedding, behavior type embedding vector p u,i and the interaction time t u,i Perform element-wise addition to obtain the behavior-aware product embedding A i .

[0030] The item order-dependent encoding layer processes behavior-aware item embeddings including:

[0031] The behavior-aware product embedding is converted into a query vector, a key vector and a value vector. The learnable item-level correlation score between the i-th product and the rest of the products in the multi-behavior-aware interaction sequence is calculated based on the behavior-aware product embedding, the query vector and the key vector. The behavior-aware sequentially dependent product embedding of the i-th product is calculated based on the value vector, the learnable item-level correlation score and the behavior-aware product embedding.

[0032] Commodity embedding based on behavior perception order dependency by multi-behavior mode aggregation layer Processing includes:

[0033]

[0034] Among them, D c represents the preference of user u under the cth behavior type, c i is the behavior type of the i-th commodity, W c is the behavior perception conversion matrix, σ(·) is the Relu activation function, To fit the indicator function of behavior type-aware embedding propagation, when c i = c,

[0035] The cross-type aggregation layer calculates the user’s preference for each product D c Processing includes:

[0036] Adaptive attention network is used to classify users’ preferences for each product D c Calculate the attention weight of each behavior type and assign the user’s preference D for each product according to the attention weight. c Perform weighted combination to obtain time-aware multi-behavior context information Utilize multi-behavior aggregation layer to obtain time-aware multi-behavior context information Processing is performed to obtain the cross-time embedding of product i under each behavior type Will Perform weighted combination to obtain the cross-time embedding of product i, where c represents the behavior type.

[0037] The process of calculating the loss function includes: using the attention mechanism according to the multimodal preference x su and multiple behavioral preferences x Mo Calculate the multimodal interest distribution matrix B separately Mo and multi-behavior interest allocation matrix B Su ;

[0038] Calculate the loss function:

[0039]

[0040] Among them, sg is the stop gradient operation, The negative log-likelihood loss function representing multiple behavior preferences, The negative log-likelihood loss function representing multimodal preferences, represents the bidirectional loss function between multi-behavior preference and multi-modal preference, is the similarity loss function between the multimodal interest assignment matrix and the multi-behavior interest assignment matrix, λ1 and λ2 are weights, and L D Regularization loss for independent feature representation of different factors in multimodality.

[0041] Beneficial effects:

[0042] 1. The present invention adopts a cross-view learning framework to learn the collaborative relationship between multi-modality and multi-behavior, and uses bidirectional optimization to enhance the recommendation performance and further improve the accuracy of the recommendation system; 2. The present invention first ensures that the feature representations of different factors in each modality are independent of each other through disentanglement technology, and then uses a multimodal attention mechanism to capture the user's preference for different modalities on each factor, thereby dynamically adjusting the weights of different modal features to better reflect the user's personalized preferences; 3. The present invention introduces a behavior-aware context embedding layer, an item order dependency encoding layer, a multi-behavior mode aggregation layer, a cross-type aggregation layer, and a graph attention network to model the complex relationship and collaborative effect between users and products, effectively process the user's multi-behavior interaction sequence, and extract the user's dynamic interest characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A flowchart of a cross-view sequence recommendation method based on multi-modal and multi-behavior provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a multimodal preference extraction module provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a multi-behavior preference extraction module provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] like Figure 1 As shown, the present invention adopts a cross-view sequence recommendation method based on multimodal and multi-behavior, including: obtaining a multimodal e-commerce platform sequence, inputting the multimodal e-commerce platform sequence into a trained sequence recommendation model, and obtaining recommended products; the sequence recommendation model includes: a preprocessing module, a multimodal preference extraction module, a multi-behavior preference extraction module and a cross-view learning module;

[0048] The training process of the sequence recommendation model includes:

[0049] S1. Obtain a multimodal e-commerce platform sequence, input the multimodal e-commerce platform sequence into a preprocessing module, and obtain a multimodal feature embedding vector and a multi-behavior perception interaction sequence;

[0050] S2, inputting the multimodal feature embedding vector into the multimodal preference extraction module to obtain the multimodal preference;

[0051] S3, inputting the multi-behavior perception interaction sequence into the multi-behavior preference extraction module to obtain the multi-behavior preference;

[0052] S4. Input the multimodal preference and multi-behavior preference into the cross-view learning module to calculate the loss function value, update the model parameters according to the loss function value, and complete the model training when the loss function value is minimized.

[0053] Furthermore, the preprocessing module processes the multimodal e-commerce platform sequence including:

[0054] The modes of the multimodal e-commerce platform sequence include: ID, text, and vision. The vector of each mode is composed of K blocks, each of which represents a factor, such as color or material.

[0055] Calculate the embedding vectors for the following three modalities:

[0056] For the embedding of product ID: a unique ID is randomly assigned to each product and initialized with the ID to obtain the ID feature embedding vector E ID ; ID feature embedding vector E ID Used to represent the item ID and capture its features and semantic information in the model.

[0057] For the initial text feature F te Extraction: Extracting initial text features F from text documents of multimodal e-commerce platform sequences using the BERT model text .

[0058] For the initial visual feature F vi Extraction: Extract the initial visual features F from the pictures of the multimodal e-commerce platform sequence through the CNNs model vi .

[0059] For the extracted initial text and visual features, a neural network is used for further learning to convert the initial text and visual features into feature embedding vectors with better representation ability and semantic information; the text feature embedding vector can be obtained by the following formula:

[0060]

[0061] in, and is the weight matrix corresponding to the text modality, and is the bias vector corresponding to the text modality, and σ is the LeakyReLU activation function.

[0062] Similarly, the embedding of visual features can be obtained:

[0063]

[0064] in, and is the weight matrix corresponding to the visual modality, and is the bias vector corresponding to the visual modality, and σ is the LeakyReLU activation function.

[0065] Furthermore, if Figure 2 As shown, the multimodal preference extraction module processes the multimodal feature embedding vector including:

[0066] S21. Process the multimodal feature embedding vector using disentanglement learning technology to obtain independent feature representations of different factors in the multimodality;

[0067] The feature representation of each modality is divided into K blocks, each of which represents a potential factor that affects the user's preference for the product, such as color or material, that is:

[0068]

[0069]

[0070]

[0071] However, in order to avoid the entanglement of the features of different factors and cause information redundancy, the distance correlation function is used to characterize the independence of any two pairs of embeddings of different factors. That is, for the feature vector E vi , we can get:

[0072]

[0073] Among them, input is the vector E vi The factors of the kth and k′th blocks of are represented by , dCov represents the distance covariance between the two matrices, and dVar(·) is the distance variance of each matrix.

[0074] Define the regularization loss L ID , L text and L vi The feature independence of each factor in product ID, text features and visual features is achieved. Therefore, the regularization loss of disentanglement learning is:

[0075] L D =L ID +L text +L vi

[0076] S22. Use the multimodal attention mechanism to calculate the user's modal preference on each factor based on the independent feature representations of different factors in the multimodality.

[0077] After disentangling different factors of multimodal features, how to capture the user's complex modality preferences for each factor on different modalities is a problem. Therefore, the present invention adopts a multimodal attention mechanism to learn modality preferences.

[0078] Specifically, for product i, user u's attention to the kth factor of each modality is calculated as follows:

[0079]

[0080]

[0081]

[0082] Among them, M and b are the weight matrix and bias vector of the neural network respectively, is the output of the hidden layer of the neural network, w is the transformation matrix that maps the output of the hidden layer to the attention weight vector, [] means embedding multimodal features in series, Tanh is the activation function, is the attention degree of user u to the kth factor of each modality, Represent the three modal attention levels respectively.

[0083] Use the softmax function to Normalize and convert the attention weights into probability distributions; therefore, the final attention weight of the text feature of the kth factor is The calculation is as follows:

[0084]

[0085] Similarly, we can get the attention weight of the ID and visual features of product i and Right now:

[0086]

[0087]

[0088] By introducing a multimodal attention network, the user's attention weight on each factor in different modalities can be calculated, thereby reflecting the user's preference for different modalities.

[0089] Therefore, according to the calculated attention weights, the feature representations of different modalities are weighted summed to obtain the user's comprehensive multimodal preference representation on each factor. This comprehensive representation reflects the user's preference x for different modalities on each factor k in the multimodal view. Mo :

[0090]

[0091] Further, such as Figure 3 The process of capturing user multi-behavior preferences in the second part is as follows:

[0092] The multi-behavior preference extraction module includes: behavior-aware context embedding layer, item order dependency encoding layer, multi-behavior pattern aggregation layer, cross-type aggregation layer and graph attention network.

[0093] For each user u, define a multi-behavior perception interaction sequence S u =[(o u,1 ,p u,1 ,t u,1 ),(o u,2 ,p u,2,t u,2 ),…(o u,I ,p u,I ,t u,I )], where I is the length of the behavior perception interaction sequence, o u,i represents the representation of the i-th item in the interaction sequence, p u,i The embedding vector representing the behavior type of user u interacting with the i-th item, such as page browsing, adding to favorites, adding to shopping cart, and purchasing behavior in an e-commerce platform; t u,i Indicates the moment of user-product interaction; the present invention defines the behavior type to be predicted as the target behavior, and other behavior types are defined as contextual behaviors, which are used to describe the user's preference for the target behavior.

[0094] The multi-behavior preference extraction module processes the multi-behavior perception interaction sequence including:

[0095] S31, inputting multiple behavior-aware interaction sequences into a behavior-aware context embedding layer to obtain behavior-aware product embedding;

[0096] In this embedding layer, for the i-th product in the multi-behavior interaction sequence, the behavior dynamics are captured by encoding strategies, and the contextual signals of multi-behavior patterns and interaction dynamics are integrated to learn the behavior-aware product embedding A. i :

[0097]

[0098] Among them, l u,i Indicates o u,i The learnable position embedding of o in the multi-behavior perception interaction sequence u,i Position encoding is performed to obtain p u,i is the embedding vector of the interaction behavior type of product i, t u,i represents the interaction time between user and item i, Represents an element-wise addition operation.

[0099] S32, embedding the behavior-aware commodity into the input item sequence dependency encoding layer to obtain behavior-aware sequence-dependent commodity embedding;

[0100] In order to capture the dynamics of sequence transformations across different products, a sequence dependency encoder based on a self-attention mechanism is used to calculate the encoding embedding of the i-th interactive product through multi-head dot product attention, thereby learning the mutual relationship between products, including:

[0101]

[0102]

[0103] in, It is a producti and product i j The learnable item-level correlation score between the nth head, σ(·) represents the softmax activation function, is the refined representation of the product obtained by aggregation, i.e., the behavior-aware sequentially dependent product embedding to capture the relationship between different products, n∈N, represents the number of attention heads, Q n , K n 、V n They represent the query vector, key vector, and value vector of the nth head respectively, and d represents the dimension of behavior-aware product embedding.

[0104] The above behavior-aware context embedding layer and item order dependency encoding layer can both preserve the transition relations across different commodities and capture the dynamics of multiple behaviors.

[0105] S33, embedding the products that are dependent on the behavior perception order into the multi-behavior mode aggregation layer to obtain the user's preference for each product under each behavior type;

[0106] Create a bipartite graph G g = {u∪S u}, where u represents the interaction sequence S u Corresponding user nodes. In order to capture the heterogeneity of user behavior, a behavior-aware message passing scheme is designed to distinguish and aggregate interaction patterns from different behavior types. The information transmitted from the product end to the user end can be expressed in the form of:

[0107]

[0108] Among them, Polymerize means polymerization, c i is the behavior interaction type of the ith item of user node u, D c represents the preference of user node u under the cth behavior type, W c is the behavior-aware transformation matrix used to enhance the modeling of behavior-specific semantics, σ(·) is the ReLU activation function, To fit the indicator function of behavior type-aware embedding propagation, c i = c,

[0109] Behavior perception conversion matrix W c It is calculated through a multi-channel parameter learning strategy, which can effectively distinguish multiple types of interactions during message transmission. The calculation formula is as follows:

[0110]

[0111]

[0112] N basic transformations shared by all behavior types in multi-channel projection, α n is the learned weight of the nth potential channel representation, computed by a fully connected layer with transformation H and bias θ.

[0113] The purpose of multi-channel projection is to distinguish the multi-relation interactions in the process of information transmission. After behavioral embedding projection on multiple basic transformations, the developed channel-based aggregation layer can be used to encode the behavior-aware semantics, thereby retaining the inherent behavioral semantics of different types of user-item interactions.

[0114] S34, input the user's preference for each product into the cross-type aggregation layer to obtain the cross-time embedding of each product;

[0115] An adaptive attention network is used to distinguish the impact of different behavior types on the predicted target behavior and match the learning scenario with dynamic user preferences. The specific formula is as follows:

[0116]

[0117]

[0118] Among them, W a ,τ a are the transformation matrix and bias matrix of the aggregation layer, σ s is the softmax activation function, σ r is the Relu activation function, β c is the attention weight of the cth behavior type, and the sum of all behavior embeddings is calculated as the query of the attention mechanism. Therefore, the present invention explicitly retains the time-aware multi-behavior context information in the user representation

[0119] In order to simulate the influence of time-aware users on different commodities, message passing from users to commodities is performed through an embedded propagation and aggregation scheme:

[0120]

[0121]

[0122] Among them, c k is the behavior type of the kth commodity, W c is the behavior perception transformation matrix calculated by the above multi-channel parameter learning strategy, β c is the attention weight attention aggregation mechanism returned by the c-th behavior type of the above adaptive attention network, is the cross-time embedding of the c-th behavior type of product i, is the cross-temporal embedding of product i, which can capture the changes in product semantics over time.

[0123] S35. Input the cross-time embedding of each product into the graph attention network to obtain multi-behavioral preferences.

[0124] First, define a user graph to include all users (u∈U) and the items they interact with (i∈I), where Represents the connection between each user u and its interactive item i.

[0125] To learn user representations, temporal context is introduced in the message passing process of the graph attention neural architecture to allow preferences encoded with different timestamps to interact with each other in a differentiable way:

[0126]

[0127]

[0128] ε=H u T ·V 0

[0129] V 1 =H u +T

[0130]

[0131] The interaction of the zth layer of the attention neural architecture is:

[0132]

[0133]

[0134]

[0135]

[0136]

[0137] is the user embedding output by the zth layer, is the product embedding output by the zth layer, V z is the user interest preference embedding output by the zth layer, ε is the learned attention weight between user u and its interactive products, T∈R d is the time embedding in the interaction sequence, which is obtained by embedding the timestamps in the multi-behavior perception interaction sequence. u is the initial user embedding, which is initialized by the cross-temporal embedding of the connected interactive item i.

[0138] H′ u For the user embedding at the final layer, For the final product embedding, we use a multi-layer perceptron MLP to concatenate the two to obtain the multi-behavior preference:

[0139]

[0140] In order to obtain the final prediction result, the inner product is used to calculate the predictions of the multimodal view and the multi-behavior view, and they are added together to obtain the final prediction result. For the representation o of the jth candidate product in the dataset j , the similarity score s is calculated as follows u,j , and sort them to get the user's recommendation list.

[0141] s u,j =x Su T o j +x Mo T o j

[0142] Furthermore, for the third part, the process of jointly training multi-modal views and multi-behavior views using the cross-view framework is as follows:

[0143] Due to the high heterogeneity of multimodal preferences and multi-behavior preferences, it is difficult to establish a one-to-one correspondence between user preferences under the two preferences. In order to structure the synergistic relationship between preference representations, a cross-view learning framework is proposed. Specifically, a two-layer MLP is first used to transform each preference x Mo and x Su Embedded into the common embedding space, we get and

[0144]

[0145]

[0146] Among them, GELU is the activation function, and are the trainable weight matrices and biases.

[0147] Calculate the final training loss function:

[0148]

[0149] Among them, sg is the stop gradient operation, which is used to eliminate numerical effects and ensure that the importance of recommendation tasks in both views is equal; Represents the negative log-likelihood loss function for the multi-view recommendation task, which is used to measure the prediction accuracy of the model in the single-view recommendation task. represents the negative log-likelihood loss function for the multimodal view recommendation task, represents the loss function between the two-way views, which is used to measure the consistency and mutual complementarity of the model between different views. is the similarity loss function between the assignment matrices of multiple behaviors and multiple modalities, λ1 and λ2 are used as hyperparameters to adjust model training and performance, and L D represents the regularization loss for the disentanglement computation.

[0150] Represents the loss function between the bidirectional views, and the loss function is the Information Noise-Contrastive Estimation (InfoNCE) loss function, which can learn the overall semantic similarity of preference pairs, with the goal of ensuring that two preferences of the same user represent the same semantic information. Specifically, for each pair of preference representations, two preference representations belonging to the same user are regarded as positive pairs, and two preference representations belonging to different users are regarded as negative pairs. In the common embedding space, in order to narrow the distance between positive preference representations and push the distance between negative preference representations, the present invention sets a contrast loss based on bidirectional views to learn the preference synergy relationship from multimodal to multi-behavior views and from multi-behavior to multimodal views:

[0151]

[0152] Then the loss function from multimodal view to multi-behavior view is for:

[0153]

[0154] Among them, x and v represent the index of each element in the interest distribution matrix B, β is a learnable temperature parameter used to scale the logarithm, and B represents the interest distribution matrix, which is used to learn the synergistic relationship between multimodal views and multi-behavior views. Specifically, the interest distribution matrix B is the interest information distribution matrix B of the multimodal view Mo and the interest information distribution matrix B of multiple behavior views Su By comparing the similarity of the interest information distribution matrices under two views, the model can learn the correlation between the user's preferences and the feature information of the product under different views.

[0155] Loss function from multi-behavior view to multi-modal view The same logic applies.

[0156] Interest information distribution matrix B of multimodal viewsMo and the interest information distribution matrix B of multiple behavior views Su for:

[0157]

[0158]

[0159] W q , W k , W q ′ and W k ′ is the weight of the learnable linear projection, The kth feature representing multimodal preference, The nth feature representing multiple behavior preferences.

[0160] When B Mo and B Su When the dimensions of B are not uniform, Mo and B Su Pad to make its dimension uniform; then the loss function It can be expressed as:

[0161]

[0162] Where L is B Mo and B Su The dimension of , l is the index of the dimension, and cos is the cosine similarity.

[0163] Calculating the negative log-likelihood loss function for the multi-behavior recommendation task includes:

[0164]

[0165] in, is the multi-behavior preference embedded in the common embedding space according to the multi-behavior perception interaction sequence before the i+1th product, o u,i+1 is the embedding vector of the next item in the multi-behavior perception interaction sequence, and the arg max operator is used to extract Select with o u,i+1 The closest behavioral preference

[0166] Calculate user u and product representation o u,i+1 Possibility of interaction:

[0167]

[0168] Among them, t u,i Represents the current time step, and the denominator calculates the sum of the matching degrees between the user and all products.

[0169] Calculate the negative log-likelihood loss function:

[0170]

[0171] in, represents the user set, and T represents the moment set of multi-behavior perception interaction sequence.

[0172] Negative log-likelihood loss function for multimodal view recommendation tasks The same logic applies.

[0173] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-view sequence recommendation method based on multi-modal and multi-behavior, characterized in that: include: Obtain a multimodal e-commerce platform sequence, input the multimodal e-commerce platform sequence into the trained sequence recommendation model, and obtain recommended products; The sequence recommendation model includes: a preprocessing module, a multimodal preference extraction module, a multi-behavior preference extraction module and a cross-view learning module; The training process of the sequence recommendation model includes: S1. Obtain a multimodal e-commerce platform sequence, input the multimodal e-commerce platform sequence into a preprocessing module, and obtain a multimodal feature embedding vector and a multi-behavior perception interaction sequence; S2, input the multimodal feature embedding vector into the multimodal preference extraction module to obtain independent feature representations and multimodal preferences of different factors in the multimodality; S3, inputting the multi-behavior perception interaction sequence into the multi-behavior preference extraction module to obtain the multi-behavior preference; S4. Input the independent feature representations of different factors in multimodality, multimodal preferences and multi-behavior preferences into the cross-view learning module to calculate the loss function value, update the model parameters according to the loss function value, and complete the model training when the loss function value is minimized.

2. The method for cross-view sequence recommendation based on multi-modal and multi-behavior according to claim 1, characterized in that: The preprocessing module processes the multimodal e-commerce platform sequence by: extracting the initial text feature F from the multimodal e-commerce platform sequence using the BERT model text , extracting initial visual features F from multimodal e-commerce platform sequences through CNNs model vi , for the initial text feature F text and the initial visual features F vi Perform feature embedding respectively to obtain the text feature embedding vector E text and the visual feature embedding vector E vi ; Randomly initialize each product ID in the multimodal e-commerce platform sequence to obtain the ID feature embedding vector E ID .

3. The method for cross-view sequence recommendation based on multi-modal and multi-behavior according to claim 1, characterized in that: The multimodal preference extraction module processes the multimodal feature embedding vector including: S21. Process the multimodal feature embedding vector using disentanglement learning technology to obtain independent feature representations of different factors in the multimodality; S22. Use the multimodal attention mechanism to calculate the user's modal preference on each factor based on the independent feature representations of different factors in the multimodality.

4. The method for cross-view sequence recommendation based on multi-modal and multi-behavior according to claim 3, characterized in that: Calculating the user's modality preference on each factor involves: According to the independent feature representations of different factors in multimodality, the user u's attention degree to the kth factor of each modality is calculated. According to the attention degree, the user's attention weight to the kth factor of each modality is calculated. According to the attention weight, the independent feature representations of different factors in multimodality are weighted and summed to obtain the user's modal preference on each factor.

5. The method for cross-view sequence recommendation based on multi-modal and multi-behavior according to claim 1, characterized in that: The multi-behavior preference extraction module includes: behavior-aware context embedding layer, item order dependency encoding layer, multi-behavior mode aggregation layer, cross-type aggregation layer, and graph attention network; The multi-behavior preference extraction module processes the multi-behavior perception interaction sequence including: S31, inputting multiple behavior-aware interaction sequences into a behavior-aware context embedding layer to obtain behavior-aware product embedding; S32, embedding the behavior-aware commodity into the input item sequence dependency encoding layer to obtain behavior-aware sequence-dependent commodity embedding; S33, embedding the products that are dependent on the behavior perception order into the multi-behavior mode aggregation layer to obtain the user's preference for each product under each behavior type; S34, input the user's preference for each product into the cross-type aggregation layer to obtain the cross-time embedding of each product; S35. Input the cross-time embedding of each product into the graph attention network to obtain multi-behavioral preferences.

6. The method for cross-view sequence recommendation based on multi-modal and multi-behavior according to claim 5, characterized in that: The multi-behavior perception interaction sequence includes: u =[(o u,1 ,p u,1 ,t u,1 ),(o u,2 ,p u,2 ,t u,2 ),…(o u,I ,p u,I ,t u,I )], where I is the length of the multi-behavior perception interaction sequence, o u,i represents the representation of the i-th item in the interaction sequence, p u,i The embedding vector representing the behavior type of user u interacting with the i-th item, t u,i represents the interaction time between user u and the i-th product; The behavior-aware context embedding layer processes the multi-behavior-aware interaction sequence by encoding the position of the i-th item in the multi-behavior-aware interaction sequence, obtaining the learnable position embedding of the i-th item, and converting the representation of the i-th item into u,i , learnable position embedding, behavior type embedding vector p u,i and the interaction time t u,i Perform element-wise addition to obtain the behavior-aware product embedding A i .

7. The method for cross-view sequence recommendation based on multi-modal and multi-behavior according to claim 5, characterized in that: The item order-dependent encoding layer processes behavior-aware item embeddings including: The behavior-aware product embedding is converted into a query vector, a key vector and a value vector. The learnable item-level correlation score between the i-th product and the rest of the products in the multi-behavior-aware interaction sequence is calculated based on the behavior-aware product embedding, the query vector and the key vector. The behavior-aware sequentially dependent product embedding of the i-th product is calculated based on the value vector, the learnable item-level correlation score and the behavior-aware product embedding.

8. The method for cross-view sequence recommendation based on multi-modal and multi-behavior according to claim 5, characterized in that: Commodity embedding based on behavior perception order dependency by multi-behavior mode aggregation layer Processing includes: Among them, D c represents the preference of user u under the cth behavior type, c i is the behavior type of the i-th commodity, W c is the behavior perception conversion matrix, σ(·) is the Relu activation function, To fit the indicator function of behavior type-aware embedding propagation, when c i = c, 9. The method for cross-view sequence recommendation based on multi-modal and multi-behavior according to claim 5, characterized in that: The cross-type aggregation layer processes the user's preferences for each product, including: Adaptive attention network is used to classify users’ preferences for each product D c Calculate the attention weight of each behavior type and assign the user’s preference D for each product according to the attention weight. c Perform weighted combination to obtain time-aware multi-behavior context information Utilize multi-behavior aggregation layer to obtain time-aware multi-behavior context information Processing is performed to obtain the cross-time embedding of product i under each behavior type Will Perform weighted combination to obtain the cross-time embedding of product i, where c represents the behavior type.

10. The method for cross-view sequence recommendation based on multi-modal and multi-behavior according to claim 1, characterized in that: The process of calculating the loss function includes: using the attention mechanism according to the multimodal preference x su and multiple behavior preferences x Mo Calculate the multimodal interest distribution matrix B separately Mo and multi-behavior interest allocation matrix B Su ; Calculate the loss function: Among them, sg is the stop gradient operation, The negative log-likelihood loss function representing multiple behavior preferences, The negative log-likelihood loss function representing multimodal preferences, represents the bidirectional loss function between multi-behavior preference and multi-modal preference, is the similarity loss function between the multimodal interest assignment matrix and the multi-behavior interest assignment matrix, λ1 and λ2 are weights, and L D Regularization loss for independent feature representation of different factors in multimodality.

Citation Information

Cited By

  • Cross-modal content recommendation method and device, equipment, medium and product

    CN120407952A