Recommendation method and device based on multi-modal semantics and behavior relation diffusion fusion

Abstract

The invention discloses a recommendation method and device based on multi-modal semantics and behavior relation diffusion fusion. The recommendation method comprises the following steps: training a diffusion model by utilizing a semantic article adjacency matrix and a behavior article adjacency matrix; inputting the semantic article adjacency matrix into the trained diffusion model to generate a diffusion perception article adjacency matrix; obtaining original total article characterization, diffusion total article characterization and collaborative user characterization; updating the initial user and article embedding; according to the updated user and article embedding, obtaining an updated collaborative user characterization and an updated diffusion total article characterization; and obtaining an article recommendation result according to the updated collaborative user characterization and the updated diffused total article characterization. According to the method, more complete and real structural information reflecting the relation between the articles can be obtained, the influence of multi-modal inherent noise is reduced, and the accuracy of personalized recommendation is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data mining, and in particular relates to a recommendation method and device based on diffusion fusion of multimodal semantics and behavioral relations. Background Art

[0002] In recent years, with the rapid development of Internet technology, the amount of information users face has exploded. How to accurately recommend interesting content to users in the massive amount of information has attracted the attention of researchers. The introduction of recommendation system provides an effective solution to this challenge.

[0003] The core of the recommendation system is to encode the historical interaction data between users and items to obtain their potential representations and establish a prediction model to evaluate the similarity between user and item representations. On this basis, the multimodal recommendation system further utilizes the multimodal information of items (such as text, pictures, etc.) to improve the quality of personalized recommendations. However, most of the existing multimodal recommendation methods construct semantic item graphs based on multimodal feature similarity, which only reflects the semantic similarity between items and is easily affected by multimodal inherent noise, and the recommendation accuracy is poor. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a recommendation method and device based on diffusion fusion of multimodal semantics and behavioral relations.

[0005] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] In a first aspect, the present invention provides a recommendation method based on diffusion fusion of multimodal semantics and behavioral relations, comprising:

[0007] The diffusion model is trained using a semantic item adjacency matrix and a behavioral item adjacency matrix to obtain a trained diffusion model; wherein the semantic item adjacency matrix is ​​constructed based on the similarity between multimodal features of items, and the behavioral item adjacency matrix is ​​constructed based on the co-occurrence frequency of items in a user-item interaction adjacency matrix;

[0008] Inputting the semantic item adjacency matrix into the trained diffusion model to generate a diffusion-aware item adjacency matrix;

[0009] Inputting the initial user embedding, the initial item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix, and the diffuse perception item adjacency matrix into a graph convolution module to obtain an original total item representation, a diffuse total item representation, and a collaborative user representation;

[0010] The initial user embedding and the initial item embedding are updated using the original total item representation, the diffused total item representation and the collaborative user representation to obtain an updated user embedding and an updated item embedding;

[0011] Inputting the updated user embedding, the updated item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix and the diffusion-aware item adjacency matrix into the graph convolution module to obtain an updated collaborative user representation and an updated diffuse total item representation;

[0012] An item recommendation result is obtained according to the updated collaborative user representation and the updated diffused total item representation.

[0013] In a second aspect, the present invention provides a recommendation device based on diffusion fusion of multimodal semantics and behavioral relations, comprising:

[0014] A model training module, used to train the diffusion model using a semantic item adjacency matrix and a behavioral item adjacency matrix to obtain a trained diffusion model; wherein the semantic item adjacency matrix is ​​constructed based on the similarity between the multimodal features of the items, and the behavioral item adjacency matrix is ​​constructed based on the co-occurrence frequency of the items in the user-item interaction adjacency matrix;

[0015] A matrix generation module, used for inputting the semantic item adjacency matrix into the trained diffusion model to generate a diffusion-aware item adjacency matrix;

[0016] A representation acquisition module, configured to input the initial user embedding, the initial item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix, and the diffuse perception item adjacency matrix into a graph convolution module to obtain an original total item representation, a diffuse total item representation, and a collaborative user representation;

[0017] A first updating module, configured to update the initial user embedding and the initial item embedding by using the original total item representation, the diffused total item representation and the collaborative user representation to obtain an updated user embedding and an updated item embedding;

[0018] A second updating module is used to input the updated user embedding, the updated item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix and the diffusion-aware item adjacency matrix into the graph convolution module to obtain an updated collaborative user representation and an updated diffuse total item representation;

[0019] The recommendation result acquisition module is used to obtain the item recommendation result according to the updated collaborative user representation and the updated diffused total item representation.

[0020] The technical solution provided by the embodiments of the present invention may have the following beneficial effects:

[0021] In the above technical solution, a diffusion model is used to learn the structure of the semantic item adjacency matrix, and the user-item interaction adjacency matrix obtained from user behavior data is used to construct the semantic item adjacency matrix. The semantic item adjacency matrix is ​​used as guiding information to participate in the diffusion process to generate a diffusion-aware item adjacency matrix that integrates multimodal semantics and user behavior information. This can obtain more complete structural information that truly reflects the relationship between items, significantly improving the accuracy of personalized recommendations. Based on the diffusion-aware total item representation and the original total item representation, the rich semantic relationships in the diffusion-aware item adjacency matrix and the semantic item adjacency matrix are fully mined to improve the accuracy of recommendations and reduce the impact of multimodal inherent noise.

[0022] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of a recommendation method based on diffusion fusion of multimodal semantics and behavioral relations provided by an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of the forward and backward processes of a diffusion model provided by an embodiment of the present invention;

[0025] Figure 3 It is a block diagram of a recommendation device based on diffusion fusion of multimodal semantics and behavioral relations provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0027] Figure 1 is a flowchart of a recommendation method based on diffusion fusion of multimodal semantics and behavioral relationships provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0028] S101. Train the diffusion model using a semantic item adjacency matrix and a behavioral item adjacency matrix to obtain a trained diffusion model; wherein the semantic item adjacency matrix is ​​constructed based on the similarity between multimodal features of items, and the behavioral item adjacency matrix is ​​constructed based on the co-occurrence frequency of items in a user-item interaction adjacency matrix.

[0029] Optionally, before S101, the method may include:

[0030] The modality similarity score matrix is ​​calculated based on the features of any modality of the item;

[0031] The connection relationship of the first k similar edges in the similarity score matrix is ​​retained to obtain the adjacency matrix of the modality;

[0032] The adjacency matrix of the modality is weighted using hyperparameters to obtain the semantic item adjacency matrix;

[0033] Obtain the item co-occurrence frequency matrix based on the user-item interaction adjacency matrix;

[0034] The item co-occurrence frequency matrix is ​​pruned, the connection relationship of the first k edges of each item is retained, and the behavior-item adjacency matrix is ​​obtained.

[0035] For example, the similarity score matrix of modality m is calculated based on the features of modality m of the item. Elements of the similarity score matrix It can be expressed as:

[0036]

[0037] in, represents the similarity score between item i and item j in modality m, Represents the similarity score matrix of items in modality m, R N×N express The dimension of N is the total number of items. and represents the pre-extracted features of item i and item j in modality m, and ||·|| represents taking the binary norm.

[0038] Keep the similarity score matrix The connection relationship of the first k similar edges of each item in , and the adjacency matrix S of mode m is obtained m , the elements of the adjacency matrix of mode m It can be expressed as:

[0039]

[0040] in, represents the elements in the adjacency matrix of mode m, Indicates that item i and item j have a connection relationship in mode m, S m ∈R N×N represents the adjacency matrix of items in mode m, Represents the similarity score matrix In the i-th row of , top-k(·) means selecting the largest k elements.

[0041] The adjacency matrix S of mode m is adjusted using hyperparameters m The weighted processing obtains the semantic item adjacency matrix S mm , the semantic item adjacency matrix can be expressed as:

[0042]

[0043] Among them, S mm ∈R N×N represents the semantic item adjacency matrix, α m is the importance score hyperparameter for mode m, is a collection of modes.

[0044] Calculate item co-occurrence frequency matrix based on user-item interaction adjacency matrix Item co-occurrence frequency matrix elements It can be expressed as:

[0045]

[0046] in, represents the co-occurrence frequency of item i and item j, represents the item co-occurrence frequency matrix, U i and U j denote the sets of users interacting with item i and item j respectively, |·| denotes the modulus value, and ∩ denotes the intersection.

[0047] Item co-occurrence frequency matrix The low-value elements in are pruned and the connection relationship of the first k edges of each item is retained to obtain the behavior-item adjacency matrix S c , the behavior-item adjacency matrix is:

[0048]

[0049] Among them, S c ∈R N×N represents the behavior-item adjacency matrix, and ρ represents the pruning threshold.

[0050] Optionally, S101 may include:

[0051] Input the semantic item adjacency matrix into the diffusion model and perform the forward noisy process to obtain the noisy item adjacency matrix;

[0052] Input the noisy item adjacency matrix, the behavior item adjacency matrix and the time step sequence into the diffusion model, perform the reverse denoising process, and obtain the denoised item adjacency matrix;

[0053] The mean square error loss function is obtained according to the semantic item adjacency matrix and the denoised item adjacency matrix, and the mean square error loss function is used for back propagation to update the diffusion model to obtain a trained diffusion model.

[0054] Understandably, Figure 2 A schematic diagram of the forward and backward processes of a diffusion model provided by an embodiment of the present invention, such as Figure 2 As shown, get any item i in the semantic item adjacency matrix S mm The item relationship vector in And randomly sample the time step corresponding to item i represents discrete uniform distribution, and T represents the maximum time step. Perform t-step forward process on the relationship vector of any item i to obtain the noisy relationship vector of item i It can be expressed as:

[0055]

[0056] in, represents the noisy relationship vector of item i, represents a Gaussian distribution, α t =1-β t , β t ∈(0,1) represents the Gaussian noise scale and satisfies β t =1-κ t / κ t-1 , the linear difference sequence κ satisfies κ=(1,κ max ,...,κ min ), κ max and κ min is a hyperparameter, and finally the adjacency matrix of the noisy items is obtained The corresponding time step sequence is t = [t 1 ,t 2 ,...,t N ].

[0057] The sine embedding method can be used to generate the time step embedding E' of the time step sequence t. The time step embedding element E' iv ∈E' can be expressed as:

[0058]

[0059] in, represents the vth dimension of the embedding of item i corresponding to the time step, represents the sinusoidal embedding of the time step sequence, D T represents the dimension of the embedding for a single time step, t i Represents the time step corresponding to item i.

[0060] The time step embedding E' is input into the first linear layer, the GELU nonlinear activation function, and the second linear layer in turn to obtain the final time step embedding The final time step embedding corresponding to item i can be expressed as:

[0061]

[0062] in,(·) T represents the transpose operation, represents the final time step embedding corresponding to item i, represents the final time step embedding, represents the time step embedding corresponding to item i, represents the first weight matrix, represents the first bias vector, represents the second weight matrix, represents the second bias vector, and GELU(·) represents the GELU nonlinear activation function.

[0063] For the noisy item adjacency matrix and the behavior-item adjacency matrix S c All are normalized and regularized to obtain the noisy item embeddings. and behavioral item embedding It can be expressed as:

[0064]

[0065] in, represents the noisy item embedding, represents behavioral item embedding, Dropout(·) represents the selected regularization layer, BN(·) represents the normalization layer, and p1 and p2 represent the dropout probability of the Dropout layer.

[0066] Embed the noisy items Behavior Item Embedding and the final time step embedding The splicing results in the diffuse splicing embedding E, which can be expressed as:

[0067]

[0068] in, represents diffuse splicing embedding.

[0069] The diffusion splicing embedding E passes through the third linear layer to obtain the denoised item adjacency matrix The denoised item relationship vector corresponding to item i can be expressed as:

[0070]

[0071] in, represents the denoised item relationship vector corresponding to item i, represents the denoised item adjacency matrix, represents the diffuse concatenation embedding corresponding to item i, represents the third weight matrix, b3∈R 1×NRepresents the third bias vector, and then obtains the denoised item adjacency matrix

[0072] Optionally, the mean square error loss function is expressed as follows:

[0073]

[0074] Among them, L diff is the mean square error loss function, N represents the total number of items, represents the diffusion loss function of item i, represents the noisy relationship vector of item i, represents the noisy item relationship of item i, R1×N represents The dimension is 1×N, represents the original item relationship vector of item i, Ε(·) represents the expectation, represents the denoised relationship vector of item i, c i Represents the behavior-item relationship vector of item i, t i represents the time step of item i, represents the hyperparameter of the time step t.

[0075] S102: Input the semantic item adjacency matrix into the trained diffusion model to generate a diffusion-aware item adjacency matrix.

[0076] Optionally, S102 may include:

[0077] Input the semantic item adjacency matrix into the trained diffusion model, perform a reverse iteration process for a preset number of steps, and obtain a diffusion perception matrix;

[0078] According to the connection relationship of the first k similar edges of each item in the diffusion perception matrix, the diffusion perception item adjacency matrix is ​​obtained.

[0079] It is understandable that t=T, As the starting condition, for 1≤t≤T, the reverse iterative process can be expressed as:

[0080]

[0081] in, Represents the behavior-item relationship vector of item i. The final output after iteration is the diffusion-perceived item relationship of any item i. The diffusion perception matrix can be obtained by splicing the diffusion perception item relationships of all items.

[0082]

[0083] in, represents the denoised item relationship vector corresponding to item i, represents the denoised item relationship vector corresponding to item i obtained by setting the diffusion condition to all zero vectors, and ω represents a hyperparameter used to control the strength of the guidance information.

[0084] According to the diffusion perception matrix The connection relationship of the first k similar edges of each item in the , and the diffusion perception item adjacency matrix S is obtained. diff , the elements of the diffusion-aware item adjacency matrix It can be expressed as:

[0085]

[0086] in, represents the elements of the diffusion-aware item adjacency matrix, Indicates that item i and item j have a connection relationship, Represents the diffuse perception matrix The i-th row of S diff ∈R N×N Represents the diffusion-aware item adjacency matrix.

[0087] S103, inputting the initial user embedding, the initial item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix and the diffuse perception item adjacency matrix into the graph convolution module to obtain the original total item representation, the diffuse total item representation and the collaborative user representation.

[0088] Optionally, S103 may include:

[0089] Input the initial item embedding and the semantic item adjacency matrix into the graph convolution module to obtain the semantic item representation;

[0090] Input the initial item embedding and the diffusion-aware item adjacency matrix into the graph convolution module to obtain the diffusion-aware item representation;

[0091] Concatenate the initial user embedding and the initial item embedding to get the initial total embedding;

[0092] Construct a symmetric interaction adjacency matrix based on the user-item interaction adjacency matrix;

[0093] The initial total embedding and the symmetric interaction adjacency matrix are input into the graph convolution module to obtain collaborative user representation and collaborative item representation;

[0094] The diffuse perceived item representation and the collaborative item representation are added together to obtain the diffuse total item representation;

[0095] The semantic item representation and the collaborative item representation are added together to obtain the original total item representation.

[0096] It can be understood that the user and item IDs are first compressed to the required embedding dimension using the embedding layer, and then the initial user embedding is obtained using the Xavier uniform initialization method. and initial item embedding D represents the initial embedding dimension, and M represents the number of users in the user set.

[0097] Embed the initial item and the semantic item adjacency matrix S mm Input to the graph convolution module to obtain semantic object representation It can be expressed as:

[0098]

[0099] in, Represents the l-th layer embedding of the item, initialized to D mm ∈R N×N Indicates S mm The diagonal matrix of L mm Indicates the number of layers of the corresponding graph convolution module.

[0100] Embed the initial item And the diffusion-aware item adjacency matrix S diff Input to the convolution module to obtain the diffuse perception object representation It can be expressed as:

[0101]

[0102] in, Represents the l-th layer embedding of the item, initialized to D diff ∈R N×N Indicates S diff The diagonal matrix of L diff Indicates the number of layers of the corresponding graph convolution module.

[0103] Concatenate the initial item embedding and the initial item embedding to get the total initialization embedding It can be expressed as:

[0104]

[0105] in, Represents the total initialization embedding.

[0106] Step 4.5: Construct a symmetric interaction adjacency matrix A based on the user-item interaction adjacency matrix A ρ , the symmetric adjacency matrix is:

[0107]

[0108] Among them, A ρ ∈R (M+N)×(M+N) represents the symmetric interaction adjacency matrix, A∈R M×N represents the interaction adjacency matrix.

[0109] Step 4.6: Embed the total initialization and the symmetric adjacency matrix A ρ Input to the graph convolution module to obtain collaborative user representation and collaborative item representation The collaborative user representation and collaborative item representation are:

[0110]

[0111] in, represents the l-th layer concatenated embedding of user items, D UI ∈R N×N Indicates A ρ The diagonal matrix of L UI Indicates the number of layers of the corresponding graph convolution module.

[0112] Step 4.7: Characterize diffuse perception items and collaborative item representation Add together to get the total diffusion item representation Representing semantic items and collaborative item representation Add together to get the original total item representation H I , the diffuse total item representation and the original total item representation are:

[0113]

[0114] in, represents the total diffusion item representation, H I ∈R N×D Represents the original total item representation.

[0115] S104: Update the initial user embedding and the initial item embedding using the original total item representation, the diffused total item representation, and the collaborative user representation to obtain an updated user embedding and an updated item embedding.

[0116] Optionally, a contrastive loss function is obtained based on the diffused total item representation and the original total item representation;

[0117] The Bayesian personalized ranking loss function is obtained based on the diffuse total item representation and collaborative user representation;

[0118] The total loss function is obtained by using the contrast loss function and the Bayesian personalized ranking loss function;

[0119] The initial user embedding and the initial item embedding are updated according to the total loss function to obtain updated user embedding and updated item embedding.

[0120] It can be understood that the total loss function is expressed as follows:

[0121]

[0122]

[0123] Among them, L represents the total loss function, L bpr represents the Bayesian personalized ranking loss function, L cl represents the contrast loss function, represents the regularization term, θ represents the model parameter, λ1 is the first preset parameter, and λ2 is the second preset parameter; D represents the data set, user u interacts with item i, and user u does not interact with item j, σ(·) represents the sigmoid activation function, represents the collaborative user representation of user u, represents the diffuse total item representation of item i, (·) T represents the transposition operation; I represents the item set, exp(·) represents the exponential function, s(·) represents the cosine similarity function, represents the positive pairs in the diffuse total item representation and the original total item representation, represents the negative pairs in the diffuse total item representation and the original total item representation, and τ represents the temperature coefficient hyperparameter.

[0124] It is worth mentioning that the contrast loss function is designed to maximize the similarity between positive pairs of item representations and minimize the similarity between negative pairs; the observable interactions between users and products are taken as positive samples, and the unobserved interactions are taken as negative samples. The Bayesian personalized ranking loss function is designed to optimize in the direction that positive samples should have higher prediction scores than negative samples.

[0125] S105, inputting the updated user embedding, the updated item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix and the diffusion-aware item adjacency matrix into the graph convolution module to obtain an updated collaborative user representation and an updated diffusion total item representation.

[0126] Optionally, S105 may include: inputting the updated item embedding and the semantic item adjacency matrix into a graph convolution module to obtain an updated semantic item representation;

[0127] Input the updated item embedding and the diffusion-aware item adjacency matrix into the graph convolution module to obtain the updated diffusion-aware item representation;

[0128] Concatenate the updated user embedding and the updated item embedding to get the updated total embedding;

[0129] The updated total embedding and the symmetric interaction adjacency matrix are input into the graph convolution module to obtain the updated collaborative user representation and the updated collaborative item representation.

[0130] The updated diffuse perception item representation and the updated collaborative item representation are added together to obtain the updated diffuse total item representation.

[0131] It is understandable that S105 may refer to the specific operations in the embodiment of S103, which will not be described in detail here.

[0132] S106. Obtain item recommendation results according to the updated collaborative user representation and the updated diffused total item representation.

[0133] In one embodiment, a simulation experiment is performed according to the present invention.

[0134] 1. Simulation experiment conditions:

[0135] Operating system: Ubuntu 20.04, Python 3.10.14; Experimental platform: Pytorch-2.3.0; Processor: Intel Core i9-13900KF; Graphics card: NVIDIA RTX 4090D; Memory: 32GB;

[0136] 2. Simulation experiment content:

[0137] Simulation experiment 1: Recall rate R@K and normalized cumulative gain N@K experiment of recommendation tasks:

[0138] It should be noted that the following experiments are all conducted in the same experimental environment. The dataset uses a classic dataset of multimodal recommendation methods. The benchmark method and the multimodal recommendation method based on diffusion fusion of item semantics and behavior relations proposed in this invention are both multimodal recommendation methods based on item graph construction.

[0139] Table 1

[0140]

[0141]

[0142] As shown in Table 1, the experimental results give a performance comparison of the baseline method and the present invention in terms of recall rate R@K and normalized cumulative gain N@K. The experimental results are as follows: The present invention outperforms the baseline method in different recommendation task performance evaluation indicators, with an improvement of 11.41% in the R@5 indicator and 13.01% in the N@5 indicator. The above experimental results verify the effectiveness of the present invention in improving personalized recommendation results by using the diffusion model to generate a diffusion-aware item adjacency matrix that integrates semantic and behavioral information.

[0143] Simulation experiment 2: ablation experiment:

[0144] It should be noted that the following experiments are all conducted in the same experimental environment. In order to study the role of each module in the present invention, each module is removed through an ablation experiment to conduct an ablation experiment.

[0145] Table 2

[0146]

[0147] As shown in Table 2, the experimental results give a performance comparison of the present invention and its two variants in terms of recall rate R@K and normalized cumulative gain N@K. Removing contrastive learning means not using contrastive loss in the total loss of the recommendation task, and removing guidance information means setting the behavior-item adjacency matrix used for diffusion guidance to an all-0 matrix. The experimental results are as follows: Removing contrastive learning leads to a decrease in performance, indicating the importance of contrastive learning methods for making full use of dual views; the present invention is superior to the method of removing guidance information, verifying the key role of the behavior-item graph that reflects the actual preferences of users in generating the item graph by the diffusion model.

[0148] Figure 3 is a block diagram of a recommendation device based on diffusion fusion of multimodal semantics and behavioral relations provided by an embodiment of the present invention, such as Figure 3 As shown, the device 300 may include:

[0149] The model training module 301 is used to train the diffusion model using the semantic item adjacency matrix and the behavioral item adjacency matrix to obtain a trained diffusion model; wherein the semantic item adjacency matrix is ​​constructed based on the similarity between the multimodal features of the items, and the behavioral item adjacency matrix is ​​constructed based on the co-occurrence frequency of the items in the user-item interaction adjacency matrix;

[0150] A matrix generation module 302 is used to input the semantic item adjacency matrix into the trained diffusion model to generate a diffusion-aware item adjacency matrix;

[0151] A representation acquisition module 303 is used to input the initial user embedding, the initial item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix and the diffuse perception item adjacency matrix into a graph convolution module to obtain an original total item representation, a diffuse total item representation and a collaborative user representation;

[0152] A first updating module 304 is used to update the initial user embedding and the initial item embedding using the original total item representation, the diffused total item representation and the collaborative user representation to obtain an updated user embedding and an updated item embedding;

[0153] The second updating module 305 is used to input the updated user embedding, the updated item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix and the diffusion-aware item adjacency matrix into the graph convolution module to obtain an updated collaborative user representation and an updated diffusion total item representation;

[0154] The recommendation result acquisition module 306 is used to obtain the item recommendation result according to the updated collaborative user representation and the updated diffused total item representation.

[0155] In the above technical solution, a diffusion model is used to learn the structure of the semantic item adjacency matrix, and the user-item interaction adjacency matrix obtained from user behavior data is used to construct the semantic item adjacency matrix. The semantic item adjacency matrix is ​​used as guiding information to participate in the diffusion process to generate a diffusion-aware item adjacency matrix that integrates multimodal semantics and user behavior information. This can obtain more complete structural information that truly reflects the relationship between items, significantly improving the accuracy of personalized recommendations. Based on the diffusion-aware total item representation and the original total item representation, the rich semantic relationships in the diffusion-aware item adjacency matrix and the semantic item adjacency matrix are fully mined to improve the accuracy of recommendations and reduce the impact of multimodal inherent noise.

[0156] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0157] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0158] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other changes to the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the term "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and "multiple" means two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0159] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A recommendation method based on diffusion fusion of multimodal semantics and behavioral relations, characterized in that: The method comprises: The diffusion model is trained using a semantic item adjacency matrix and a behavioral item adjacency matrix to obtain a trained diffusion model; wherein the semantic item adjacency matrix is ​​constructed based on the similarity between multimodal features of items, and the behavioral item adjacency matrix is ​​constructed based on the co-occurrence frequency of items in a user-item interaction adjacency matrix; Inputting the semantic item adjacency matrix into the trained diffusion model to generate a diffusion-aware item adjacency matrix; Inputting the initial user embedding, the initial item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix, and the diffuse perception item adjacency matrix into a graph convolution module to obtain an original total item representation, a diffuse total item representation, and a collaborative user representation; The initial user embedding and the initial item embedding are updated using the original total item representation, the diffused total item representation and the collaborative user representation to obtain an updated user embedding and an updated item embedding; Inputting the updated user embedding, the updated item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix and the diffusion-aware item adjacency matrix into the graph convolution module to obtain an updated collaborative user representation and an updated diffuse total item representation; An item recommendation result is obtained according to the updated collaborative user representation and the updated diffused total item representation.

2. The recommendation method based on diffusion fusion of multimodal semantics and behavioral relations according to claim 1 is characterized in that: Before the diffusion model is trained using the semantic item adjacency matrix and the behavioral item adjacency matrix to obtain the trained diffusion model, the method further includes: Calculating a similarity score matrix of any modality according to the features of any modality of the item; The connection relationship of the first k similar edges in the similarity score matrix is ​​retained to obtain the adjacency matrix of the modality; The adjacency matrix of the modality is weighted by using hyperparameters to obtain the semantic item adjacency matrix; Obtaining an item co-occurrence frequency matrix according to the user-item interaction adjacency matrix; The item co-occurrence frequency matrix is ​​pruned to retain the connection relationship of the first k edges of each item, thereby obtaining the behavior-item adjacency matrix.

3. The recommendation method based on diffusion fusion of multimodal semantics and behavioral relations according to claim 1 is characterized in that: The method of training the diffusion model using the semantic item adjacency matrix and the behavioral item adjacency matrix to obtain a trained diffusion model includes: Inputting the semantic item adjacency matrix into the diffusion model, performing a forward noisy process, and obtaining a noisy item adjacency matrix; Inputting the noisy item adjacency matrix, the behavior item adjacency matrix and the time step sequence into the diffusion model, performing a reverse denoising process, and obtaining a denoised item adjacency matrix; A mean square error loss function is obtained according to the semantic item adjacency matrix and the denoised item adjacency matrix, and the mean square error loss function is used for back propagation to update the diffusion model to obtain the trained diffusion model.

4. The recommendation method based on diffusion fusion of multimodal semantics and behavioral relations according to claim 1 is characterized in that: The step of inputting the semantic item adjacency matrix into the trained diffusion model to generate a diffusion-aware item adjacency matrix includes: Inputting the semantic item adjacency matrix into the trained diffusion model, performing a reverse iteration process of a preset number of steps, and obtaining a diffusion perception matrix; The diffusion-sensing item adjacency matrix is ​​obtained according to the connection relationship of the first k similar edges of each item in the diffusion-sensing matrix.

5. The recommendation method based on diffusion fusion of multimodal semantics and behavioral relations according to claim 1 is characterized in that: The initial user embedding, the initial item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix and the diffuse perception item adjacency matrix are input into a graph convolution module to obtain an original total item representation, a diffuse total item representation and a collaborative user representation, including: Inputting the initial item embedding and the semantic item adjacency matrix into the graph convolution module to obtain a semantic item representation; Inputting the initial item embedding and the diffusion-aware item adjacency matrix into the graph convolution module to obtain a diffusion-aware item representation; Concatenating the initial user embedding and the initial item embedding to obtain an initial total embedding; Constructing a symmetric interaction adjacency matrix according to the user-item interaction adjacency matrix; Inputting the initial total embedding and the symmetric interaction adjacency matrix into the graph convolution module to obtain collaborative user representation and collaborative item representation; Adding the diffuse perception item representation and the collaborative item representation to obtain the diffuse total item representation; The semantic item representation and the collaborative item representation are added together to obtain the original total item representation.

6. The recommendation method based on diffusion fusion of multimodal semantics and behavioral relations according to claim 5 is characterized in that: The updating of the initial user embedding and the initial item embedding by using the original total item representation, the diffused total item representation and the collaborative user representation to obtain an updated user embedding and an updated item embedding includes: Obtaining a contrast loss function according to the diffused total item representation and the original total item representation; Obtaining a Bayesian personalized ranking loss function according to the diffused total item representation and the collaborative user representation; Obtaining a total loss function using the contrast loss function and the Bayesian personalized ranking loss function; The initial user embedding and the initial item embedding are updated according to the total loss function to obtain the updated user embedding and the updated item embedding.

7. The recommendation method based on diffusion fusion of multimodal semantics and behavioral relations according to claim 6 is characterized in that: The step of inputting the updated user embedding, the updated item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix, and the diffusion-aware item adjacency matrix into the graph convolution module to obtain an updated collaborative user representation and an updated diffusion total item representation, comprising: Inputting the updated item embedding and the semantic item adjacency matrix into the graph convolution module to obtain an updated semantic item representation; Inputting the updated item embedding and the diffusion-aware item adjacency matrix into the graph convolution module to obtain an updated diffusion-aware item representation; Concatenating the updated user embedding and the updated item embedding to obtain an updated total embedding; Inputting the updated total embedding and the symmetric interaction adjacency matrix into the graph convolution module to obtain an updated collaborative user representation and an updated collaborative item representation; The updated diffusion-aware item representation and the updated collaborative item representation are added together to obtain the updated diffusion total item representation.

8. The recommendation method based on diffusion fusion of multimodal semantics and behavioral relations according to claim 3 is characterized in that: The mean square error loss function is expressed as follows: Wherein, Ldiff is the mean square error loss function, N represents the total number of items, represents the diffusion loss function of item i, represents the noisy relationship vector of the item i, represents the noisy item relationship of item i, R1×N represents The dimension is 1×N, represents the original item relationship vector of item i, Ε(·) represents the expectation, represents the denoised relationship vector of item i, ci represents the behavior item relationship vector of item i, ti represents the time step of item i, represents the hyperparameter of the time step t.

9. The recommendation method based on diffusion fusion of multimodal semantics and behavioral relations according to claim 6, characterized in that: The total loss function is expressed as follows: Wherein, L represents the total loss function, Lbpr represents the Bayesian personalized ranking loss function, Lcl represents the contrast loss function, represents the regularization term, θ represents the model parameter, λ1 is the first preset parameter, and λ2 is the second preset parameter; D represents the data set, user u interacts with item i, and user u does not interact with item j, σ(·) represents the sigmoid activation function, u∈U represents the collaborative user representation of user u, i∈I represents the diffuse total item representation of item i, (·)T represents the transposition operation; I represents the item set, exp(·) represents the exponential function, s(·) represents the cosine similarity function, i∈I represents the positive pair in the diffuse total item representation and the original total item representation, i,v∈I,i≠v denotes the negative pairs in the diffused total item representation and the original total item representation, and τ denotes the temperature coefficient hyperparameter.

10. A recommendation device based on diffusion fusion of multimodal semantics and behavioral relations, characterized in that: The device comprises: A model training module, used to train the diffusion model using a semantic item adjacency matrix and a behavioral item adjacency matrix to obtain a trained diffusion model; wherein the semantic item adjacency matrix is ​​constructed based on the similarity between the multimodal features of the items, and the behavioral item adjacency matrix is ​​constructed based on the co-occurrence frequency of the items in the user-item interaction adjacency matrix; A matrix generation module, used for inputting the semantic item adjacency matrix into the trained diffusion model to generate a diffusion-aware item adjacency matrix; A representation acquisition module, configured to input the initial user embedding, the initial item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix, and the diffuse perception item adjacency matrix into a graph convolution module to obtain an original total item representation, a diffuse total item representation, and a collaborative user representation; A first updating module, configured to update the initial user embedding and the initial item embedding by using the original total item representation, the diffused total item representation and the collaborative user representation to obtain an updated user embedding and an updated item embedding; A second updating module is used to input the updated user embedding, the updated item embedding, the semantic item adjacency matrix, the user-item interaction adjacency matrix and the diffusion-aware item adjacency matrix into the graph convolution module to obtain an updated collaborative user representation and an updated diffuse total item representation; The recommendation result acquisition module is used to obtain the item recommendation result according to the updated collaborative user representation and the updated diffused total item representation.

Images