Diffusion-based multi-modal contrast learning commodity recommendation method

Abstract

The invention discloses a multi-modal comparison learning commodity recommendation method based on diffusion. The method comprises the following steps: constructing an original interaction graph containing interaction information of a user and a commodity, and extracting multi-modal features of the commodity; an improved denoising diffusion probability model is adopted, a dynamic gating mechanism and multi-modal features are combined, and a precise modal interaction diagram is generated in the reverse denoising process through mean square error loss, user preference consistency loss and a regularization term optimization model; through cross-view comparative learning optimization, the feature robustness and the distribution uniformity are improved; based on the generated modal interaction diagram and the original interaction diagram, designing a dual-path propagation architecture with modal alignment to integrate collaborative information and modal information; and based on the parameter characteristics of the user and the project, calculating a recommendation score of the user for the project through an inner product, and outputting a recommendation result. According to the method, the de-noising diffusion probability model and the comparative learning strategy are effectively combined, and the recommendation performance and the user preference modeling ability are remarkably improved.

Description

Technical Field

[0001] This invention relates to the technical field of multimodal recommendation, and in particular to a diffusion-based multimodal contrastive learning method for product recommendation. Background Technology

[0002] Multimodal recommendation aims to integrate auxiliary information from different modalities (such as images, text, and audio) to improve the representation quality of users and products, thereby enhancing recommendation performance. In recent years, with the development of deep learning, multimodal recommendation methods have increasingly adopted graph neural networks and contrastive learning strategies to uniformly model structural information and modal features. A typical approach is to propagate features based on interaction graphs and utilize graph convolutional neural networks to capture high-order neighbor information. Simultaneously, modal information is used to supplementally model user preferences through shared embedding spaces or attention mechanisms. To further enhance the model's generalization ability and robustness, cross-modal contrastive learning is widely applied in the learning process of multimodal feature alignment and feature discrimination.

[0003] Despite significant progress in existing methods, several issues remain: First, most methods directly fuse the original interaction graph with multimodal features, failing to effectively distinguish the impact of modal information on graph structure modeling, leading to significant noise interference. Second, existing methods typically employ static structures or heuristic rules when generating modal interaction graphs, lacking flexible graph generation mechanisms and limiting the recommendation system's ability to model complex preferences. To address these issues, this invention proposes a diffusion-based multimodal contrastive learning-based product recommendation method. It introduces a denoising diffusion model to automatically generate modal interaction graphs and guides modal features to participate in the generation process through a dynamic gating mechanism. Combining a modality-aligned dual-path propagation architecture and cross-view contrastive learning optimization effectively alleviates modal noise interference and feature heterogeneity problems, thereby improving recommendation performance and model robustness. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a diffusion-based multimodal contrastive learning method for product recommendation. This method can generate interaction graphs in modal space, perform feature modeling based on users' real preferences, and improve the robustness of node features.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a diffusion-based multimodal contrastive learning-based product recommendation method, comprising the following steps:

[0006] 1) Filter and extract features from the public dataset to obtain the recommendation system dataset as the training dataset. This dataset includes the original interaction graph between users and products and the modal features of products. Based on the original interaction graph, information is propagated through the graph convolutional neural network algorithm to obtain the user ID features and product ID features.

[0007] 2) Using the recommendation system dataset obtained in step 1), train the improved denoising diffusion probability model to obtain the optimal denoising diffusion probability model. Use this optimal denoising diffusion probability model as a view generator to generate modal interaction graphs. The specific improvements to this model are as follows: during the forward denoising process of the original denoising diffusion probability model, noise is gradually added to the original interaction graph to form a noisy interaction graph; during the reverse denoising process, a dynamic gating mechanism is used to fuse the modal features of the goods into the noisy interaction graph. By gradually removing noise from the noisy interaction graph, the view generator is guided by the modal features of the goods, and then the modal interaction graph is obtained through this view generator; the user preference consistency and regularization terms of the original interaction graph and the modal interaction graph are also used as optimization terms in the model training, so that the generated modal interaction graph can reflect the user's true preferences.

[0008] 3) Use the modal interaction graph generated in step 2) and combine it with the modal features of the product to perform modal-level discrimination learning, and use the original interaction graph to perform structural-level contrastive learning to form cross-view contrastive learning optimization; after cross-view contrastive learning optimization, the distribution of user ID features and product ID features can be more uniform and robust, and the optimized ID features of users and products can be obtained.

[0009] 4) Using the optimized ID features of users and products obtained in step 3), a modal-aligned dual-path propagation architecture was designed, which integrates the collaborative information in the original interaction graph and the modal information in the modal interaction graph to obtain the final user features and the final product features.

[0010] 5) Construct a recommendation model using the final user features and final product features obtained in step 4); use the final user features and final product features as input parameters of the recommendation model, optimize the final features through Bayesian personalized ranking to obtain user parameter features and product parameter features; use the inner product of the parameter features to calculate the user's recommendation rating for the product, and give the recommendation results based on the recommendation rating from high to low.

[0011] Furthermore, in step 1), two types of entities are selected from the public dataset as users and items in the recommendation system, respectively. The interaction history between them is stored in the form of a two-dimensional sparse matrix, thereby forming the original interaction graph G∈R. U ×I Where R represents the set of real numbers, and U×I represents a matrix consisting of U users and I items, with each element indicating whether there is an interaction between a user and an item (1 if there is, 0 otherwise). Based on the resulting original interaction graph, information is propagated using a graph neural network algorithm to obtain the user's ID feature. and product ID characteristics Where u represents any user among U users, and i represents any product among I products;

[0012] For each item i that constitutes the original interaction graph, data for each modality is filtered out from the public dataset, and feature extraction is performed using a pre-trained model to obtain feature sets for I items under different modalities. in express Different modal types.

[0013] Furthermore, in step 2), the specific training method for the improved denoising diffusion probability model is as follows:

[0014] The original interaction graph is input into the improved denoising diffusion probability model, with x t Let x' represent the state of the original interaction graph at time t, where t = 1, 2, ..., T′; the initial state of the original interaction graph is x0, indicating that no time has been reached; combining the noise ∈ sampled from the standard normal distribution, the state x of the original interaction graph at any time t can be derived from the forward noise addition process of the denoising diffusion probability model. t for:

[0015]

[0016] In the formula, The overall information retention rate from time k to t is represented as follows:

[0017]

[0018] In the formula, Δ k =1-β k ,β k Let α represent the noise variance constant added during the forward noise addition process at time k. k Representative information retention range; This indicates cumulative multiplication. By multiplying the information retention range from time 1 to time t, the overall information retention rate is obtained.

[0019] After the forward noise addition process, the state x of the original interaction graph at time T′ can be obtained. T′ Let the original interaction graph after adding noise be the noisy interaction graph, then z t′ =x T′ This represents the state of the noise interaction graph at time t', where t' = T', T'-1, ..., 1; Next, we need to define the state z of the noise interaction graph. t′ Perform a reverse denoising process to restore it to the initial state x0, thereby achieving the goal of training the denoised diffusion probability model; firstly, initialize the T′ time steps in the reverse denoising process as a set of time step vectors {C}. t′The reverse denoising process can be described as follows:

[0020]

[0021] In the formula, p θ (z t'-1 |z t' ) is a conditional probability distribution, representing a distribution where z is known. t' At that time, the noise interaction graph state transitions to z. t'-1 The probability of z describes how the denoising operation of the reverse denoising process is performed; t'-1 This represents the state of the noise interaction graph at time t'-1. Indicates a normal distribution. and They represent The mean and variance matrices need to be obtained from the noise interaction graph state z at time t'. t′ Time step vector C t' Modal features of product type m Solve for I′; I′ represents an identity matrix.

[0022] According to the theoretical formula of the denoising diffusion probability model, under ideal conditions, p θ (z t'-1 |z t' The mean of the multimodal perceptron is calculated from the initial state x0, while the variance is a known constant. However, the initial state x0 is actually unknown throughout the entire inverse denoising process. Therefore, a multilayer perceptron that fuses multimodal information through a dynamic gating mechanism is designed. To fit the mean at each time step in the inverse denoising process, where Let θ represent the input parameters, where θ is a trainable parameter; the calculation process of the multilayer perceptron is as follows:

[0023]

[0024] W g =σ1(W1P m +b1)

[0025] P′ m =P m ⊙W g ,

[0026]

[0027] In the formula, P m Let m be the modal features of product type m at time t'. The projection, W gThe fusion weights are calculated through a dynamic gating mechanism. W1 and b1 represent the weights and biases of the dynamically gating linear layer, respectively, and σ1(·) represents the Sigmoid activation function. The fusion weights W1 and b1 are then used to calculate the fusion weights. g With projection P m After performing the dot product operation ⊙, the weighted modal feature projection P of the product is obtained. m Then, through the transpose of the modal features of product type m. Map back to the original dimensional space and superimpose it onto z t' The modal information-enhanced state z′ at time t' is obtained. t' This achieves the fusion of multimodal information and the reverse denoising process; ultimately, the enhanced modal information state z′ at time t' is processed. t' and the time step vector C at that moment t' Perform the splicing operation, and then complete the mean fitting calculation through a bilinear layer; W2 and W3 represent the weights of two different linear layers, b2 and b3 represent the biases of the corresponding linear layers, and tanh(·) represents the hyperbolic tangent function;

[0028] To optimize the multilayer perceptron The fitted mean μ is gradually approximated to x0, and the mean squared error is used as the loss function, which is called the mean squared error loss function. This enables the denoising diffusion probability model to generate accurate modal interaction graphs; mean squared error loss function The definition is as follows:

[0029]

[0030] In the formula, μ represents the multilayer perceptron. The obtained fitted mean, ‖·‖ 2 First, the square of the L2 norm is used; second, to further improve the generalization ability of the denoising diffusion probability model and achieve user preference alignment learning, a user preference alignment loss function is constructed.

[0031]

[0032] In the formula, feature E id Based on user ID characteristics and product ID characteristics The concatenation operation is performed to obtain cos(·), which represents the cosine similarity function. The above formula uses the cosine similarity function to force the modal features of product type m. The corresponding multimodal information and ID feature information are kept consistent, thereby achieving consistent alignment of user preferences between the original interaction graph and the noisy interaction graph;

[0033] The final joint optimization objective of the denoising diffusion probability model Defined as:

[0034]

[0035] In the formula, ρ and γ1 are the loss functions for controlling user preference alignment, respectively. The weight coefficients of the regularization term Θ1 are used; the optimal denoising diffusion probability model is used as the view generator, and the modal interaction graph G is generated using this view generator. m .

[0036] Furthermore, in step 3), in order to improve the quality of user and product ID features, a cross-view contrastive learning optimization is designed, which is divided into two parts: structural-level contrastive learning and modality-level contrastive learning.

[0037] First, along the modal interaction graph G m To disseminate information, user ID characteristics Modal features of the product Combination:

[0038]

[0039] In the formula, To aggregate modal features, and to bridge the heterogeneity of features across different modalities, modal-level contrastive learning is introduced. Achieving cross-modal optimization:

[0040]

[0041] In the formula, τ1 represents the temperature hyperparameter of modal-level contrastive learning, and a and b respectively represent Two different modes in a modality; Let x represent the set of users, where x is the positive sample user and v is the negative sample user; Let represent the set of goods, where y represents positive sample goods and j represents negative sample goods; These represent the aggregate modal features from mode a. The aggregated modal features of positive sample users and positive sample products separated from the data; These represent the aggregate modal features from mode b. The aggregated b-modal features of positive sample users and positive sample products separated from the data; These represent the aggregate modal features from mode b. The aggregated b-modal features of negative sample users and negative sample items separated from the data, where x≠v, y≠j;

[0042] User ID characteristics and product ID characteristics After performing the splicing operation, structural feature E is obtained. ego :

[0043]

[0044] The structural features are propagated layer by layer along the original interaction graph G using a graph convolutional neural network algorithm, and random noise Δ is added between the propagation layers to perturb E. ego :

[0045]

[0046] In the formula, and Let represent the structural features of the l-th and (l-1)-th layers, respectively. The sign function sign(·) controls the direction of the L2 norm ||Δ||2 of the random noise to be consistent with the structural features. η is a parameter used to control the noise amplitude, ensuring that the random noise falls on a hypersphere of radius η. To obtain global high-order features, average pooling is performed on the multi-layer propagation results.

[0047]

[0048] In the formula, L represents L propagation layers, and E avg The average structural features are represented; finally, structural-level contrastive learning is performed. Enhance the cross-layer robustness of features:

[0049]

[0050] In the formula, τ2 represents the temperature hyperparameter of structure-level contrastive learning, and log(·) and exp(·) represent the logarithmic function and the exponential function, respectively; From structural feature E ego Structural characteristics of positive sample users and positive sample products extracted from the data; This represents the structural characteristics of positive sample users and positive sample products after passing through the first propagation layer; From the average structural feature E avg The average structural characteristics of positive sample users and positive sample goods extracted from the data; From the average structural feature E avg The average structural features of negative sample users and negative sample goods extracted from the data, where x≠v, y≠j;

[0051] Through cross-view comparative learning optimization, user ID features and product ID characteristics The optimized ID feature of the user will be obtained through the gradient backpropagation mechanism of deep learning. And the optimized ID features of the product

[0052] Furthermore, in step 4), in order to effectively integrate the collaborative information in the original interaction graph and the modal information in the modal interaction graph, a modal-aligned dual-path propagation architecture is designed. First, information propagation is performed on the original interaction graph G to aggregate modal features. The modal enhancement features are obtained by weighted summation.

[0053]

[0054] In the formula, λ m These are the modal weighting coefficients; to balance the information contribution of different modes, an adaptive modal fusion strategy is designed to... Different modal enhancement features To superimpose, that is:

[0055] ω m =Softmax(W m )

[0056]

[0057] In the formula, W m It is a learnable modality fusion weight matrix, which, after activation by the Softmax activation function, yields the modality fusion weights ω. m Using this modality to fuse weights ω m Enhancement features for each modality The modality fusion features are obtained by weighted summation.

[0058] To obtain high-order neighbor features in the modal interaction graph, a multi-terminal amplified residual connection mechanism with LightGCN as the backbone is designed to simulate the multi-hop propagation process of the traditional graph convolutional neural network GCN, thereby alleviating the oversmoothing problem caused by stacking multiple propagation layers in GCN:

[0059]

[0060] H u H i =Split(H)

[0061] In the formula, δ represents the residual connection weight, Split(·) represents the segmentation operation, and H is the final feature; by segmenting the final feature H, the user's final feature H is obtained. u and the final characteristics of the product H i .

[0062] Furthermore, in step 5), the user's final feature H is used. u and the final characteristics of the product H i Construct a recommendation model, where H uand H i As input parameters to the recommendation model; using Bayesian personalized ranking. Optimize the recommendation model's ability to identify user preferences:

[0063]

[0064] In the formula, σ2(·) represents the Sigmoid activation function. Let x', y', and j' represent the training set of the recommendation model, where x', y', and j' represent positive sample users, positive sample items, and negative sample items, respectively. and These are the ratings from positive user x' for positive product y' and negative product j', respectively, calculated using the inner product:

[0065]

[0066] In the formula, The final feature H of positive sample user x' x′ The transpose of H y' H j' These are the final features of the positive sample item y' and the negative sample item j', respectively; ultimately, the overall loss function... for:

[0067]

[0068] In the formula, λ1, λ2, and γ2 represent the weight coefficients of control structure-level contrastive learning, modality-level contrastive learning, and regularization term Θ2, respectively. 2 Represents the square of the L2 norm; after the global loss function After optimization, the input parameters H of the recommendation model u and H i Optimized into user parameter feature h u and product parameter features h i ;

[0069] User parameter features h of positive sample users x' Product parameter features h of positive sample products y' Calculate the recommendation rating s of positive sample user x' for positive sample product y'. x'y' :

[0070]

[0071] In the formula, This represents the transpose of the user parameter features of positive sample users; this recommendation score will be used as the basis for the recommendation model to make recommendations to users; for each user, the recommendation model will calculate the recommendation score between them and all items they have not interacted with, and then sort the recommendation scores from high to low to obtain a set of items {c1,c2,...,c...} r}, where r represents the number of products to be recommended, c r Let r represent the r-th item, which will be used as the recommendation model to output the recommendation results to the user.

[0072] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0073] 1. This invention introduces a denoising diffusion probability model into the modal interaction graph generation task of a recommendation system, and utilizes the model's generation capability to improve the quality of the modal interaction graph and enhance the modeling of the relationship between users and products.

[0074] 2. This invention introduces a dynamic gating mechanism to fuse multimodal features. In the reverse denoising process of the denoising diffusion probability model, the fusion weights of different modes are dynamically adjusted to achieve effective guidance of modal information on the generation process.

[0075] 3. This invention designs a modality-aligned dual-path propagation architecture, which effectively integrates collaborative information and modal information to realize the propagation and fusion of modal features on the graph structure.

[0076] 4. This invention proposes an adaptive modality fusion strategy that dynamically adjusts the fusion weights based on the contribution of different modalities, thereby avoiding the problems of single modality dominance or information redundancy.

[0077] 5. This invention designs a cross-view comparative learning optimization, which can improve the model's generalization ability and cross-modal semantic alignment ability.

[0078] 6. This invention proposes a user preference consistency loss function, which preserves user preference information during the training process of the denoising diffusion probability model, thereby improving the authenticity of the modal interaction graph and the relevance of the recommendation.

[0079] 7. This invention proposes a multi-terminal amplified residual connection mechanism, which can alleviate the oversmoothing problem in graph neural networks and improve the utilization rate of high-order neighbor information.

[0080] 8. This invention supports flexible access and fusion of multiple modal data, and is applicable to multiple modal features such as images, text, and audio, with good scalability and adaptability.

[0081] In summary, this invention innovatively introduces a denoising diffusion probability model into multimodal recommendation systems, optimizing the fusion and propagation of multimodal features and alleviating the over-smoothing problem of graph neural networks. Combined with cross-view contrastive learning, it enhances the model's generalization ability. The overall method balances accuracy and computational efficiency, supports multiple modalities of data, and provides a precise, robust, and scalable solution for recommendation systems. Attached Figure Description

[0082] Figure 1 A schematic diagram of the improved denoising diffusion probability model.

[0083] Figure 2 This diagram illustrates the optimization of contrastive learning across views; the multimodal approach in the diagram uses text and images as examples, where Avg. represents average pooling and GCN represents graph convolutional neural networks.

[0084] Figure 3 This is a schematic diagram of a dual-path propagation architecture for modal alignment; the multimodal representation in the diagram uses text and images as examples. Detailed Implementation

[0085] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0086] This embodiment discloses a diffusion-based multimodal contrastive learning-based product recommendation method, which includes the following steps:

[0087] 1) Construct a recommender system dataset using a public dataset, including filtering and feature extraction of the original dataset to obtain a recommender system dataset as the training dataset.

[0088] The filtering steps for the original dataset are as follows: Select U users and I items, ensuring that each user interacts with at least 10 items, forming an interaction matrix of shape U×I. The elements of the interaction matrix are filled with 0 or 1; a 1 indicates an interaction between a user and an item, and a 0 indicates otherwise. This interaction matrix is ​​stored as a sparse matrix and used as input to the original interaction graph G∈R of the recommendation system. U×I , where R represents the set of real numbers.

[0089] The steps for multimodal feature extraction are as follows: For all I items, find the corresponding images and text from the original dataset, and use a pre-trained model to extract the image modal features and text modal features of the items respectively. The modal features are stored in the form of a dictionary, where the key is the item's ID string and the value is a tensor of the corresponding modal feature. The extracted modal features are denoted as... Where m∈{te,vi} represents the modality type, te and vi represent the text modality and the image modality respectively, and i represents any one of the I items.

[0090] 2) Using the recommendation system dataset obtained in step 1), train the improved denoising diffusion probability model to obtain the optimal denoising diffusion probability model. Use this optimal denoising diffusion probability model as a view generator to generate modal interaction graphs. The specific improvements to this model are as follows: during the forward denoising process of the original denoising diffusion probability model, noise is gradually added to the original interaction graph to form a noisy interaction graph; during the reverse denoising process, a dynamic gating mechanism is used to fuse the modal features of the goods into the noisy interaction graph. By gradually removing noise from the noisy interaction graph, the view generator is guided by the modal features of the goods, and then the modal interaction graph is obtained through this view generator; the consistency of user preferences and the regularization term between the original interaction graph and the modal interaction graph are also used as optimization terms in the model training, so that the generated modal interaction graph can reflect the user's true preferences.

[0091] like Figure 1 As shown, the improved denoising diffusion probability model consists of two parts: a forward denoising process and a backward denoising process. The forward denoising process is implemented as follows:

[0092] Let the initial state of the original interaction graph G be denoted as x0, representing the point at which no time has been reached. During the forward noise addition process, x0 will be uniformly filled with normally distributed noise at each time step, with x... t Let represent the state of the original interaction graph at any time t. The evolution of this state can be described as follows:

[0093]

[0094] In the formula, t = 1, 2, ..., T', q(x t |x t-1 () represents a probability distribution evolving from time t-1 to time t, corresponding to any noise-adding operation. This distribution is a normal distribution. The mean of this normal distribution is The variance matrix is ​​β t I1,β t ∈(0,1) represents the variance of the normally distributed noise added at each time step, and I1 represents the identity matrix. After multiple iterations, q(x) t |x t-1 From the additivity of the normal distribution, we can deduce that:

[0095]

[0096] In the formula, ∈ represents a noise sample sampled from a standard normal distribution. To simplify the formula, α is set... t =1-β t , The value of k ranges from 1 to t, representing the elapsed time t. α tThe range of information to be retained. Indicates cumulative multiplication. The cumulative product of the information retention amplitudes across all time points represents the overall information retention rate during the entire forward noise addition process. Let α... t and Substituting into the above equation, we can obtain a simplified derivation of the forward noise addition process:

[0097]

[0098] This simplified derivation allows direct derivation of the state x at any time t from the initial state x0 of the original interaction graph. t .

[0099] For the reverse denoising process, the goal is to transform the original interaction graph state x, which has been subjected to T' levels of noise, into a denoised graph. T' Restore the initial state x0 to achieve the effect of training the denoised diffusion probability model. Let the original interaction graph after adding noise be the noisy interaction graph, z. t′ =x T' Let C represent the state of the noise interaction graph at time t', where t' = T', T'-1, ..., 1. First, initialize the T' time steps in the reverse denoising process as a set of time step vectors {C}. t′ The inverse denoising process is represented as follows:

[0100]

[0101] In the formula, p θ (z t'-1 |z t' Let represent a probability distribution from the state at time t' to the state at time t'-1, corresponding to any denoising operation. This process also follows a normal distribution, but the mean and variance of this normal distribution are unknown and need to be fitted using a neural network. θ represents the trainable parameters of this neural network, and the fitted mean and variance matrices are respectively... and I′ represents an identity matrix. The input value of the neural network consists of three parts: the state z of the noisy interaction graph at time t'. t' Time step vector C t' Modal features of product type m By changing the calculation method of the original denoising diffusion probability model and incorporating the modal features of the product into the reverse denoising process, the model can perceive the user's preference for product modal information, thereby generating an accurate modal interaction graph.

[0102] Assuming the initial state x0 of the original interaction graph is known during the reverse denoising process, the expression for the denoising operation can be represented by Bayes' theorem as follows:

[0103]

[0104] In the formula, p(·) represents the conditional probability distribution. Since the above formula is a theoretical derivation, it does not require solving it using a neural network with trainable parameters θ. Based on the simplified derivation of the forward noise-adding process obtained above, the single-conditional sub-formula in the above formula can be solved:

[0105]

[0106] In the formula, I2 represents the identity matrix. Substituting the solution back into the expression for the denoising operation, we get:

[0107]

[0108] In the formula, exp(·) represents an exponential function. According to Gauss's product theorem, p(z t'-1 |z t' The expression (x, x0) also follows a normal distribution, therefore it can be expressed as:

[0109]

[0110] In the formula, and Let p(z) represent the normal distribution respectively. t'-1 |z t' The variance and mean of x0). Combining the equivalence of the two formulas above, we can solve for the expressions for the mean and variance in the reverse denoising process:

[0111]

[0112] In the formula, As can be seen from the derivation of the forward noise addition process, it is a constant; therefore, only the...

[0113] The theoretical derivation of the above denoising operations is based on the assumption that the initial state x0 of the original interaction graph is known during the reverse denoising process. However, in reality, only the three input values ​​of the neural network are known: z. t' C t' and Therefore, it needs to be solved using a neural network. To fit To accelerate the process of calculating the mean, a multilayer perceptron is used as the neural network for mean fitting. First, the modal features of the goods are utilized... The state z of the noise interaction graph at time t' t′ Project:

[0114]

[0115] In the formula, Pm This represents the calculated modal features of product type m. The projection of the image is then used. The fusion weight W is calculated using a dynamic gating mechanism. g :

[0116] W g =σ1(W1P m +b1)

[0117] In the formula, σ1(·) represents the Sigmoid activation function, and W1 and b1 represent the weights and biases of the dynamically gated linear layer, respectively. The fused weights W... g With projection P m After performing the dot product operation σ, the weighted modal feature projection P of the commodity is obtained. m ′:

[0118] P m 'P m σW g

[0119] Transpose of the modal features of product type m Map back to the original dimensional space and superimpose it onto z t' :

[0120]

[0121] In the formula, z′ t' This represents the state at time t' after modal information enhancement. Finally, the mean is fitted using a bilinear layer:

[0122]

[0123] In the formula, W2 and W3 represent the weights of two different linear layers, b2 and b3 represent the biases of the corresponding linear layers, tanh(·) represents the hyperbolic tangent function, and ‖ represents the splicing operation.

[0124] The optimization process of the denoising diffusion probability model involves three loss functions. The first is the mean squared error loss function. By minimizing the fitted mean The difference between the current state and the initial state x0 enables high-quality modal interaction graph generation.

[0125]

[0126] In the formula, μ represents the multilayer perceptron. The obtained fitted mean, ‖·‖ 2 This represents the square of the L2 norm. Secondly, to control the consistency of user preferences between the original interaction graph and the modal interaction graph, a user preference alignment loss function is designed.

[0127]

[0128] In the formula, express Different modes, E id Based on user ID characteristics and product ID characteristics The image is obtained by concatenating the images. The cosine similarity function cos(·) is used to control the alignment of user preferences on the original interaction graph and the modal interaction graph. Finally, the joint optimization objective of the denoising diffusion probability model is... Defined as:

[0129]

[0130] In the formula, ρ and γ1 are the loss functions for controlling user preference alignment, respectively. The weight coefficients of the regularization term Θ1 are also considered. The optimal denoising diffusion probability model is used as the view generator, and this view generator is used to generate the modal interaction graph G. m .

[0131] 3) To improve the quality of user and product ID characteristics, a method such as... Figure 2 The cross-view contrastive learning optimization shown is divided into two parts: structural-level contrastive learning and modal-level contrastive learning.

[0132] Modal-level contrastive learning first aggregates modal interaction graphs G using a graph convolutional neural network algorithm. m Information is used to obtain aggregate modal features.

[0133]

[0134] In the formula, m∈{te,vi}, where te and vi represent the text modality and the image modality, respectively. To bridge the heterogeneity of features across different modalities, modality-level contrastive learning is introduced. Achieving cross-modal optimization:

[0135]

[0136] In the formula, τ1 represents the temperature hyperparameter of modal-level contrastive learning. Let x represent the set of users, where x is the positive sample user and v is the negative sample user. Let represent the set of goods, where y represents positive sample goods and j represents negative sample goods. These represent the aggregate modal features from the text modality. The aggregated text modal features of positive sample users and positive sample products extracted from the data. These represent the aggregated modal features from the image modalities. The aggregated image modal features of positive sample users and positive sample products separated from the image. These represent the aggregated modal features from the image modalities. The negative sample users and negative sample items are extracted from the image, and their aggregated image modal features are used, where x≠v and y≠j. Therefore, the entire modal-level contrastive learning optimizes the aggregated modal features of users and items under different modalities, using the same aggregated modal features of users or items under both text and image modalities as a contrastive view.

[0137] In structural-level contrastive learning, to facilitate the computation of graph convolutional neural networks, the user's ID feature is first analyzed. and product ID characteristics Perform the splicing operation to obtain structural feature E. ego :

[0138]

[0139] Then, the structural features are propagated layer by layer along the original interaction graph G using a graph convolutional neural network algorithm:

[0140]

[0141] By adding random noise during layer-by-layer propagation, representation-level perturbations are achieved, thereby forming the enhanced view required for structure-level contrastive learning. This perturbation method can significantly reduce computational overhead and improve the robustness of the optimized features. The above equation describes the structural features from the (l-1)th propagation layer. Structural features up to the l-th propagation layer The derivation is as follows. `sign(·)` is a sign function that controls the direction of the L2 norm ||ε||2 of the random noise, ensuring it aligns with the direction of the structural features. `η` controls the amplitude of the noise, ensuring it falls on a hypersphere of radius `η`. This guarantees that the added random noise provides sufficient differentiation for the features without causing excessive semantic bias. To obtain global high-order features, average pooling is also performed on the multi-layer propagation results:

[0142]

[0143] In the formula, L represents L propagation layers. The average structural feature E is obtained by averaging the results of all layers. avg Finally, structural-level contrastive learning. The definition is as follows:

[0144]

[0145] In the formula, τ2 represents the temperature hyperparameter of structure-level contrastive learning, and log(·) and exp(·) represent the logarithmic function and the exponential function, respectively. From structural feature E ego The structural characteristics of positive sample users and positive sample products separated from the data. This represents the structural characteristics of positive sample users and positive sample products after passing through the first propagation layer. From the average structural feature E avg The average structural features of the positive sample users and positive sample products separated from the data. From the average structural feature E avg The average structural features of negative sample users and negative sample items extracted from the data are given, where x≠v, y≠j. Therefore, the entire structural-level contrastive learning optimizes the user and item ID features, enhancing the robustness of features across propagation layers. and As one comparative view, the average structural features serve as another.

[0146] Through cross-view comparative learning optimization, user ID features and product ID characteristics The optimized ID feature of the user will be obtained through the gradient backpropagation mechanism of deep learning. And the optimized ID features of the product

[0147] 4) To effectively integrate the collaborative information in the original interaction graph with the modal information in the modal interaction graph, a method was designed as follows: Figure 3 The modality-aligned dual-path propagation architecture is shown.

[0148] First, information propagation is performed on the original interaction graph G, combining the aggregated modal features corresponding to the text modality and the image modality. The corresponding modal enhancement features are obtained by weighted summation.

[0149]

[0150] In the formula, λ te ,λ vi These represent the text modality weighting coefficients and the image modality weighting coefficients, respectively. These represent text modality enhancement features and image modality enhancement features, respectively. To balance the information contribution of the two modalities, an adaptive modality fusion strategy is designed to superimpose the two different modality enhancement features:

[0151] ω te =Softmax(W te ),ω vi=Softmax(W vi )

[0152]

[0153] In the formula, W te W vi These are the learnable text modality fusion weight matrix and the image modality fusion weight matrix, respectively. After activation by the Softmax function, the corresponding text modality fusion weights ω are obtained. te Image modality fusion weights ω vi The modality fusion features are obtained by weighting and summing the corresponding modality enhancement features using the modality fusion weights.

[0154] Existing research indicates that higher-order propagation of features is crucial for learning graph structure representations. However, stacking multiple propagation layers introduces oversmoothing issues. To address this problem, a multi-terminal amplified residual connection mechanism based on LightGCN is designed to simulate the multi-hop propagation process:

[0155]

[0156] H u H i =Split(H)

[0157] In the formula, δ represents the residual connection weight, Split(·) denotes the segmentation operation, and H is the final feature. By segmenting the final feature H, the user's final feature Hi is obtained. u and the final characteristics of the product H i In standard weighted residual connections, the residual connection weights typically apply only to self-loop components. The above equation applies the residual connection weights δ to neighbor information simultaneously; this functional change approximates the k-hop effect. Because the coefficients of both operations are bound together, this method can more effectively capture long-range dependencies while suppressing noise. During backpropagation, the gradient through these "bound coefficients" is also scaled accordingly, helping to mitigate the vanishing gradient problem.

[0158] 5) For the final recommendation task, use the user's final feature H. u and the final characteristics of the product H i Construct a recommendation model, where H u and H i As input parameters to the recommendation model, Bayesian personalized ranking is employed. This serves as the fundamental optimization objective. The objective directly optimizes the recommendation model's ability to identify user preferences by maximizing the score difference between positive and negative samples.

[0159]

[0160] In the formula, σ2(·) represents the Sigmoid activation function. Let x', y', j' represent the training set of the recommendation model, where x', y', j' represent positive sample users, positive sample items, and negative sample items, respectively. and These are the ratings from positive user x' for positive product y' and negative product j', respectively, calculated using the inner product:

[0161]

[0162] In the formula, The final feature H of positive sample user x' x′ The transpose of H y' H j' These are the final features of the positive sample item y' and the negative sample item j', respectively. Finally, the overall loss function of the model is:

[0163]

[0164] In the formula, λ1, λ2, and γ2 represent the weight coefficients of the control structure-level contrastive learning, modality-level contrastive learning, and regularization term Θ2, respectively. 2 This represents the square of the L2 norm. After the overall loss function... After optimization, the input parameters H of the recommendation model u and H i Optimized into user parameter feature h u and product parameter features h i .

[0165] In the final recommendation process, a recommendation score is calculated using parametric features, which serves as the basis for the recommendation model to make recommendations to users. Specifically, the user parameter features h of positive sample users are used. x' Product parameter features h of positive sample products y' Calculate the recommendation rating s of positive sample user x' for positive sample product y'. x'y' :

[0166]

[0167] For each user, the recommendation model calculates a recommendation score between them and all uninteracted products using the above formula, then sorts the recommendation scores from highest to lowest, resulting in a set of products {c1, c2, ..., c...}. r}, where r represents the number of products to be recommended, c r This represents the r-th item. These items will be used as the recommendation model to output recommendations to the user.

[0168] To verify the effectiveness of this invention, two publicly available recommended benchmark datasets were selected for experiments, and the statistical results of the datasets are shown in the table.

[0169] Table 1 Dataset Details

[0170]

[0171]

[0172] The Yelp dataset contains 235,735 interactions from 37,397 users and 32,491 items. Each item has an image modality feature dimension of 512 and a text modality feature dimension of 1024. The Amazon Sports dataset contains 296,337 interactions from 35,598 users and 18,357 items. Each item has a text modality feature dimension of 1024 and an image modality feature dimension of 4096. All datasets are divided into training, validation, and test sets in a 7:2:1 ratio.

[0173] The experiment used three commonly used metrics: recall, normalized discounted cumulative gain (NDCG), and precision. A full-ranking protocol was employed to evaluate the top-K recommendation performance, with K set to 20. Specifically, the system calculates scores for all items not appearing in the training set and selects the top 20 items as the recommendation results. For simplicity, these three metrics are abbreviated as R@20, N@20, and P@20, respectively, and their specific calculation methods are as follows.

[0174] Recall (R@20): Measures the ability of recommendations to correctly match items the user actually interacted with. It is calculated as the ratio of the number of items in the top 20 predicted items that belong to the user's past interactions to the total number of items the user actually interacted with. Normalized Decline Cumulative Gain (N@20): This metric considers the ranking of recommendations and measures the quality of the distribution of relevant items in the top 20 recommendations. The higher the ranking of relevant items, the higher the score. Precision (P@20): Reflects the accuracy of recommendations, i.e., what percentage of the top 20 recommendations are items the user is genuinely interested in. It is calculated as the ratio of the number of relevant items in the top 20 to 20. These metrics allow for a comprehensive evaluation of recommendation performance, examining both the model's recognition ability and its ranking performance, thus effectively validating the practicality and superiority of the recommendation algorithm.

[0175] The experiment compared the method of this invention with three classic collaborative filtering recommendation methods—LightGCN, SGL, and SimGCL—and seven state-of-the-art multimodal recommendation methods—LATTICE, SLMRec, MGCN, BM3, LGMRec, diffmm, and SMORE. The experimental results are shown in the table, where the results of this invention are indicated in bold, the results of suboptimal methods are indicated by underlined lines, and the relative improvement represents the improvement of this method compared to the suboptimal method.

[0176] Table 2 shows the experimental results on the three datasets.

[0177]

[0178] Experimental Conclusions: The method proposed in this invention exhibits superior performance compared to existing technologies, proving its effectiveness. By leveraging the generative capability of the denoising diffusion probability model, accurate modal interaction graphs can be generated in the modal space; while the modal-aligned dual-path propagation architecture effectively alleviates the multimodal noise problem caused by directly aggregating the original interaction graphs. Furthermore, cross-view contrastive learning optimization regulates feature distribution from two complementary perspectives—structural and modal—maintaining feature diversity while achieving similarity constraints between different semantic spaces. Experimental results demonstrate that this invention significantly improves recommendation performance, has promising application prospects, and is worthy of promotion.

[0179] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A diffusion-based multimodal contrastive learning-based product recommendation method, characterized in that, Includes the following steps: 1) Filter and extract features from the public dataset to obtain the recommendation system dataset as the training dataset. This dataset includes the original interaction graph between users and products and the modal features of products. Based on the original interaction graph, information is propagated through a graph convolutional neural network algorithm to obtain the user's ID features and the product's ID features; 2) Using the recommendation system dataset obtained in step 1), train the improved denoising diffusion probability model to obtain the optimal denoising diffusion probability model. Use this optimal denoising diffusion probability model as a view generator to generate modal interaction graphs. The specific improvements to this model are as follows: during the forward denoising process of the original denoising diffusion probability model, noise is gradually added to the original interaction graph to form a noisy interaction graph; during the reverse denoising process, a dynamic gating mechanism is used to fuse the modal features of the goods into the noisy interaction graph. By gradually removing noise from the noisy interaction graph, the view generator is guided by the modal features of the goods, and then the modal interaction graph is obtained through this view generator; the user preference consistency and regularization terms of the original interaction graph and the modal interaction graph are also used as optimization terms in the model training, so that the generated modal interaction graph can reflect the user's true preferences. 3) Use the modal interaction graph generated in step 2) and combine it with the modal features of the product to perform modal-level discrimination learning, and use the original interaction graph to perform structural-level contrastive learning to form cross-view contrastive learning optimization; after cross-view contrastive learning optimization, the distribution of user ID features and product ID features can be more uniform and robust, and the optimized ID features of users and products can be obtained. 4) Using the optimized ID features of users and products obtained in step 3), a modal-aligned dual-path propagation architecture was designed, which integrates the collaborative information in the original interaction graph and the modal information in the modal interaction graph to obtain the final user features and the final product features. 5) Use the final user features and final product features obtained in step 4) to build a recommendation model; The final user features and final product features are used as input parameters for the recommendation model. The final features are optimized through Bayesian personalized ranking to obtain user parameter features and product parameter features. The inner product of the parameter features is used to calculate the user's recommendation rating for the product, and the recommendation results are given in descending order of recommendation rating.

2. The diffusion-based multimodal contrastive learning-based product recommendation method according to claim 1, characterized in that: In step 1), two types of entities are selected from the public dataset as users and items in the recommendation system, respectively. The interaction history between them is stored as a two-dimensional sparse matrix, thus forming the original interaction graph G∈R. U×I Where R represents the set of real numbers, and U×I represents a matrix consisting of U users and I items, with each element indicating whether there is an interaction between a user and an item (1 if there is, 0 otherwise). Based on the resulting original interaction graph, information is propagated using a graph neural network algorithm to obtain the user's ID feature. and product ID characteristics Where u represents any user among U users, and i represents any product among I products; For each item i that constitutes the original interaction graph, data for each modality is filtered out from the public dataset, and feature extraction is performed using a pre-trained model to obtain feature sets for I items under different modalities. in express Different modal types.

3. The diffusion-based multimodal contrastive learning-based product recommendation method according to claim 2, characterized in that: In step 2), the specific training method for the improved denoising diffusion probability model is as follows: The original interaction graph is input into the improved denoising diffusion probability model, with x t Let x' represent the state of the original interaction graph at time t, where t = 1, 2, ..., T′; the initial state of the original interaction graph is x0, indicating that no time has been reached; combining the noise ∈ sampled from the standard normal distribution, the state x of the original interaction graph at any time t can be derived from the forward noise addition process of the denoising diffusion probability model. t for: In the formula, The overall information retention rate from time k to t is represented as follows: In the formula, α k =1-β k ,β k Let α represent the noise variance constant added during the forward noise addition process at time k. k Representative information retention range; This indicates cumulative multiplication. By multiplying the information retention range from time 1 to time t, the overall information retention rate is obtained. After the forward noise addition process, the state x of the original interaction graph at time T′ can be obtained. T′ ; Let the original interaction graph after adding noise be called the noisy interaction graph, then z t′ =x T′ This represents the state of the noise interaction graph at time t', where t' = T', T'-1, ..., 1; Next, we need to define the state z of the noise interaction graph. t′ Perform a reverse denoising process to restore it to the initial state x0, thereby achieving the goal of training the denoised diffusion probability model; firstly, initialize the T′ time steps in the reverse denoising process as a set of time step vectors {C}. t′ The reverse denoising process can be described as follows: In the formula, p θ (z t'-1 |z t' ) is a conditional probability distribution, representing a distribution where z is known. t' At that time, the noise interaction graph state transitions to z. t'-1 The probability of z describes how the denoising operation of the reverse denoising process is performed; t'-1 This represents the state of the noise interaction graph at time t'-1. Indicates a normal distribution. and They represent The mean and variance matrices need to be obtained from the noise interaction graph state z at time t'. t′ Time step vector C t' Modal features of product type m Solve for I′; I′ represents an identity matrix. According to the theoretical formula of the denoising diffusion probability model, under ideal conditions, p θ (z t'-1 |z t' The mean of the multimodal perceptron is calculated from the initial state x0, while the variance is a known constant. However, the initial state x0 is actually unknown throughout the entire inverse denoising process. Therefore, a multilayer perceptron that fuses multimodal information through a dynamic gating mechanism is designed. To fit the mean at each time step in the inverse denoising process, where Let θ represent the input parameters, where θ is a trainable parameter; the calculation process of the multilayer perceptron is as follows: IN g =σ1(W1P m +b1) P m ′=P m ⊙W g , In the formula, P m Let m be the modal features of product type m at time t'. The projection, W g The fusion weights are calculated through a dynamic gating mechanism. W1 and b1 represent the weights and biases of the dynamically gating linear layer, respectively, and σ1(·) represents the Sigmoid activation function. The fusion weights W1 and b1 are then used to calculate the fusion weights. g With projection P m After performing the dot product operation ⊙, the weighted modal feature projection P of the product is obtained. m ′ Then, through the transpose of the modal features of product type m. Map back to the original dimensional space and superimpose it onto z t' The modal information-enhanced state z′ at time t' is obtained. t' This achieves the fusion of multimodal information and the reverse denoising process; ultimately, the enhanced modal information state z′ at time t' is processed. t' and the time step vector C at that moment t' Perform the splicing operation, and then complete the mean fitting calculation through a bilinear layer; W2 and W3 represent the weights of two different linear layers, b2 and b3 represent the biases of the corresponding linear layers, and tanh(·) represents the hyperbolic tangent function; To optimize the multilayer perceptron The fitted mean μ is gradually approximated to x0, and the mean squared error is used as the loss function, which is called the mean squared error loss function. This enables the denoising diffusion probability model to generate accurate modal interaction graphs; mean squared error loss function The definition is as follows: In the formula, μ represents the multilayer perceptron. The obtained fitted mean, ‖·‖ 2 First, the square of the L2 norm is used; second, to further improve the generalization ability of the denoising diffusion probability model and achieve user preference alignment learning, a user preference alignment loss function is constructed. In the formula, feature E id Based on user ID characteristics and product ID characteristics The concatenation operation is performed to obtain cos(·), which represents the cosine similarity function. The above formula uses the cosine similarity function to force the modal features of product type m. The corresponding multimodal information and ID feature information are kept consistent, thereby achieving consistent alignment of user preferences between the original interaction graph and the noisy interaction graph; The final joint optimization objective of the denoising diffusion probability model Defined as: In the formula, ρ and γ1 are the loss functions for controlling user preference alignment, respectively. The weight coefficients of the regularization term Θ1 are used; the optimal denoising diffusion probability model is used as the view generator, and the modal interaction graph G is generated using this view generator. m .

4. The diffusion-based multimodal contrastive learning-based product recommendation method according to claim 3, characterized in that: In step 3), in order to improve the quality of user and product ID features, a cross-view contrastive learning optimization is designed, which is divided into two parts: structural-level contrastive learning and modality-level contrastive learning. First, along the modal interaction graph G m To disseminate information, user ID characteristics Modal features of the product Combination: In the formula, To aggregate modal features, and to bridge the heterogeneity of features across different modalities, modal-level contrastive learning is introduced. Achieving cross-modal optimization: In the formula, τ1 represents the temperature hyperparameter of modal-level contrastive learning, and a and b respectively represent Two different modes in a modality; Let x represent the set of users, where x is the positive sample user and v is the negative sample user; Let represent the set of goods, where y represents positive sample goods and j represents negative sample goods; These represent the aggregate modal features from mode a. The aggregated modal features of positive sample users and positive sample products separated from the data; These represent the aggregate modal features from mode b. The aggregated b-modal features of positive sample users and positive sample products separated from the data; These represent the aggregate modal features from mode b. The aggregated b-modal features of negative sample users and negative sample items separated from the data, where x≠v, y≠j; User ID characteristics and product ID characteristics After performing the splicing operation, structural feature E is obtained. ego : The structural features are propagated layer by layer along the original interaction graph G using a graph convolutional neural network algorithm, and random noise Δ is added between the propagation layers to perturb E. ego : In the formula, and Let represent the structural features of the l-th and (l-1)-th layers, respectively. The sign function sign(·) controls the direction of the L2 norm ||Δ||2 of the random noise to be consistent with the structural features. η is a parameter used to control the noise amplitude, ensuring that the random noise falls on a hypersphere of radius η. To obtain global high-order features, average pooling is performed on the multi-layer propagation results. In the formula, L represents L propagation layers, and E avg The average structural features are represented; finally, structural-level contrastive learning is performed. Enhance the cross-layer robustness of features: In the formula, τ2 represents the temperature hyperparameter of structure-level contrastive learning, and log(·) and exp(·) represent the logarithmic function and the exponential function, respectively; From structural feature E ego Structural characteristics of positive sample users and positive sample products extracted from the data; This represents the structural characteristics of positive sample users and positive sample products after passing through the first propagation layer; From the average structural feature E avg The average structural characteristics of positive sample users and positive sample goods extracted from the data; From the average structural feature E avg The average structural features of negative sample users and negative sample goods extracted from the data, where x≠v, y≠j; Through cross-view comparative learning optimization, user ID features and product ID characteristics The optimized ID feature of the user will be obtained through the gradient backpropagation mechanism of deep learning. And the optimized ID features of the product 5. The diffusion-based multimodal contrastive learning-based product recommendation method according to claim 4, characterized in that: In step 4), to effectively integrate the collaborative information in the original interaction graph with the modal information in the modal interaction graph, a modal alignment dual-path propagation architecture is designed. First, information propagation is performed on the original interaction graph G to aggregate modal features. The modal enhancement features are obtained by weighted summation. In the formula, λ m These are the modal weighting coefficients; to balance the information contribution of different modes, an adaptive modal fusion strategy is designed to... Different modal enhancement features To superimpose, that is: oh m =Softmax(W m ) In the formula, W m It is a learnable modality fusion weight matrix, which, after activation by the Softmax activation function, yields the modality fusion weights ω. m Using this modality to fuse weights ω m Enhancement features for each modality The modality fusion features are obtained by weighted summation. To obtain high-order neighbor features in the modal interaction graph, a multi-terminal amplified residual connection mechanism with LightGCN as the backbone is designed to simulate the multi-hop propagation process of the traditional graph convolutional neural network GCN, thereby alleviating the oversmoothing problem caused by stacking multiple propagation layers in GCN: H u ,H i =Split(H) In the formula, δ represents the residual connection weight, Split(·) represents the segmentation operation, and H is the final feature; by segmenting the final feature H, the user's final feature H is obtained. u and the final characteristics of the product H i .

6. The diffusion-based multimodal contrastive learning-based product recommendation method according to claim 5, characterized in that: In step 5), the user's final feature H is used. u and the final characteristics of the product H i Construct a recommendation model, where H u and H i As input parameters to the recommendation model; using Bayesian personalized ranking. Optimize the recommendation model's ability to identify user preferences: In the formula, σ2(·) represents the Sigmoid activation function. Let x', y', and j' represent the training set of the recommendation model, where x', y', and j' represent positive sample users, positive sample items, and negative sample items, respectively. and These are the ratings from positive user x' for positive product y' and negative product j', respectively, calculated using the inner product: In the formula, The final feature H of positive sample user x' x′ The transpose of H y' H j' These are the final features of the positive sample item y' and the negative sample item j', respectively; ultimately, the overall loss function... for: In the formula, λ1, λ2, and γ2 represent the weight coefficients of control structure-level contrastive learning, modality-level contrastive learning, and regularization term Θ2, respectively. 2 Represents the square of the L2 norm; after the global loss function After optimization, the input parameters H of the recommendation model u and H i Optimized into user parameter feature h u and product parameter features h i ; User parameter features h of positive sample users x' Product parameter features h of positive sample products y' Calculate the recommendation rating s of positive sample user x' for positive sample product y'. x'y' : In the formula, This represents the transpose of the user parameter features of positive sample users; this recommendation score will be used as the basis for the recommendation model to make recommendations to users; for each user, the recommendation model will calculate the recommendation score between them and all items they have not interacted with, and then sort the recommendation scores from high to low to obtain a set of items {c1,c2,...,c...} r }, where r represents the number of products to be recommended, c r Let r represent the r-th item, which will be used as the recommendation model to output the recommendation results to the user.

Images